14 February 2019
In the past year or so, we have seen almost every other smartphone maker using the word artificial intelligence (AI) to market its products and lure customers. It has become the newest buzzword as all of them are claiming that their particular offering is the smartest of all but there is honestly no good way to judge that. This is where AnTuTu has decided to take the matter in control and has announced an AI benchmarking tool for smartphones.
This new tool from AnTuTu is meant to help anyone measure the AI performance of any smartphone to get an idea of how it stands against the competition. But as it has been revealed in the official blog post, the whole situation is not as easy as offering a score as it happens with general performance benchmarks. The main reason behind this is that the smartphone industry does not make use of a standard AI. All of the OEMs, chipset makers, and developers have different meaning and implementation of AI.
Qualcomm makes use of processor's DSP, Huawei relies on using dedicated NPUs, and Samsung and MediaTek offer dedicated AI chips referred to as NPUs or APUs. But this is only one part of the problem as even the software side of things have no common aspect. All these companies have their own SDKs for AI performance which include SNPE by Qualcomm, NeuroPilot by MediaTek, and TensorRT by NVIDIA. As far as Samsung is concerned, its SDK is yet to be announced publically.
But still, AnTuTu has collaborated with various tech giants to make the whole AI benchmarking process as unified as possible. It is divided into two categories called Image Classification and Object Recognition. The first one features data of about 200 pictures and is based on the Inception v3 neural networks. The second one is a 600-frame video based on the MobileNet SSD neural networks. Both of these data is fed to the SDK used in the particular device but if it is not supported, then the tool makes use of TFLite SDK for the test which AnTuTu claims to be highly unreliable.
The tool considers both the accuracy and speed of an AI to benchmark the performance. But if a particular AI is found to be completing tasks with high speed and low accuracy, it will end up getting a low score. This has been done to prevent benchmark cheating so that no one can try to fool the tool by completing all the tasks at quite quickly but by making wrong identifications.
It should be noted if there are devices using the same AI SDK, then the score difference would not be a significant one because the tool focuses only on the AI performance, not the overall performance. Since Samsung's SDK is currently unavailable and Huawei is using TFLite SDK, scores for both these devices will be on the lower side until both situations are fixed. Lastly, the exact Android version will also affect scores as Google is continuously improving native AI support and implementation. It means a device with the latest Android version is likely to score higher than the device with the older Android version even if both are using the same AI SDK.
AnTuTu's AI benchmarking tool is currently in public beta. The APK for the same can be downloaded for manual installation. It is about 33MB in size but before the first use, the user will be required to download an additional 160MB files. There is no doubt that the tool currently has a lot of caveats but that is excepted considering how vast AI is and how much difficult it is to standardize it.