25 July 2018
More and more smartphones are coming with some AI capabilities, including the image recognition and its classification, face recognition, and more. While these AI capabilities sound too basic, there are a few smartphones that come with a dedicated NPU for specific AI tasks. There isn’t any specific metric or test to check the artificial intelligence of the devices, like how intelligent they are over the other. But, a new benchmark app tries to fill this gap with a series of its test to check the performance of such devices.
A group of computer vision researchers from ETH Zurich has come up with an AI benchmark that assesses the performance of several neural network architectures used for common AI tasks.
The name of this benchmark is AI Benchmark and its Android app is available on Play Store to download. According to the description of the app, the benchmark will check if the smartphone has a dedicated AI chip, or is fast enough?
The AI Benchmark app measures the speed and memory requirements for several key Computer Vision AI algorithms. “Among the tested solutions are Image Classification and Face Recognition methods, Neural Networks used for Image Super-Resolution and Photo Enhancement, as well as algorithms used in autonomous driving systems. Visualization.”
The AI Benchmark app will be helpful in knowing the current state-of-the-art in various AI fields. With the help of the results, the chipmakers will be able to show the competitive insights and the developers will be able to see how fast their AI models will run on different devices. While, in the real world, not every one of us judges the phones by the benchmark results, there are a fair number of users who believe in these results. It is yet to be seen how this new benchmark app will perform. But, the researchers want their AI Benchmark app to be universally accepted metric similar to DxOMark that is used for evaluating the camera performance.
The AI Benchmark app can run on any device with Android 4.1 or higher. According to the researchers, the generated score will be the “final verdict” of the device’s AI performance.
AI Benchmark running through computer vision tasks. Image Credit: Kyle Wiggers / VentureBeat
AI researcher and developer Andrey Ignatov explains to Techcrunch that the final score is based on two main aspects – The SoC’s speed and the available RAM.
“Since there was no information about this (currently, all AI algorithms are running remotely on the servers, not on your device, except for some built-in apps integrated in phone’s firmware), we decided to develop our own tool that will clearly show the performance and capabilities of each device.
“Let’s consider two devices: one with a score of 6000 and one with a score of 200. If some AI algorithm will run on the first device for 5 seconds, then this means that on the second device this will take about 30 times longer, i.e. almost 2.5 minutes. And if we are thinking about applications like face recognition this is not just about the speed, but about the applicability of the approach: Nobody will wait 10 seconds till their phone will be trying to recognize them.
Ignatov also explained that what the team assumed in the beginning about various AI chips turned out completely opposite. For instance, the devices running Qualcomm chips weren’t the clear winners they’d imagined. According to the team’s analyzation, the much-hyped Snapdragon 845 capabilities are only for “quantized” networks that currently can’t be deployed on the phones, “thus for ‘normal’ networks you won’t get any acceleration at all.”
Also, the Huawei’s Kirin 970 CPU which is technically slower than the Snapdragon 636 chipset offered a surprisingly strong performance.
“Their integrated NPU gives almost 10x acceleration for Neural Networks, and thus even the most powerful phone CPUs and GPUs can’t compete with it,” says Ignatov. “Additionally, Huawei P20/P20 Pro are the only smartphones on the market running Android 8.1 that are currently providing AI acceleration, all other phones will get this support only in Android 9 or later.”
Here are the top 10 performers, along with their associated scores. The full list is available on the AI Benchmark website.
- Huawei P20 Pro – 6397
- OnePlus 6 – 1875
- Sony Xperia XZ2 – 1664
- Samsung Galaxy S9+ – 1494
- Razer Phone – 1470
- Samsung Galaxy S9 – 1446
- OnePlus 5T – 1440
- OnePlus 5 – 1416
- Samsung Galaxy Note 8 – 1408
- Xiaomi Mi Mix 2 – 1405
“In summary: Snapdragon — can theoretically provide good results, but are lacking the drivers; Huawei — quite outstanding results now and most probably in the nearest future; Samsung — no acceleration support now (most likely this will change soon since they are now developing their own AI Chip), but powerful CPUs; Mediatek — good results for mid-range devices, but definitely no breakthrough.”
AI Benchmark | Android | Play Store, Free