16 January 2019
Google at its Made by Google event, happened on October 9 this year brought the new-gen Pixel phones and more hardware solutions. The event circulated around the AI+software+hardware together with which the company is considering new things and reaching new heights. Pixel phones have been one of the key things to Google and they will always be and it’s great to see the company bringing some crucial features to these phones which are available in some form in its other hardware options.
Take, for instance, Google Clips, a tiny camera that automatically captures the best moments for you without the need of you to hit the shutter. And, Google used this experience for developing the same capability for the Pixel 3 camera and as the company gets a success, it started calling it the Top Shot feature.
Google retained the single-lens camera approach for its third-gen Pixel phones as well and this resulted in the same appreciation it gained before due to its previous gen phones. But with the new generation, also comes some upgrades and enhancements that keep them a different offering from their predecessors. Similar is the case with the Pixel 3, the camera of which is loaded with some AI-based powers and they make things easier for the single-lens camera.
There are so many reasons to love Pixel phones, there’s one feature that truly catches the attention and this is the Top Shot. Google introduced this feature because it knew missing a perfect shot is a frustrating problem. The Top Shot feature on the Pixel 3 phones is an interesting feature and Google explains it for the curious ones.
What does the Top Shot feature on Pixel 3 phone do?
The Top Shot feature on the Pixel 3 manages to catch the moments you’d wish to capture and it does it with the help of AI. There are times when you miss a perfect shot if you don't hit the shutter at exactly the time you should, so here comes the Top Shot as a solution. The feature catches the image frames before and after the shutter press on the device and that too in real-time using the computer vision techniques and then recommends several alternative high-quality HDR+ photos to you so that you could select the best. This means the feature is doing the work for you because it’s difficult to capture some epic shots or moments in the real time.
To better understand this feature, watch the embedded video below.
The process goes behind the recommendations –
According to Google’s explanation, the Top Shot feature captures 90 images from 1.5 seconds before and after the shutter is pressed. Google explains the shutter frame is processed and saved first and the best alternative shots are saved afterward. Google gives the credit to Google Visual Core on Pixel 3 that helps in processing the top alternative shots as HDR+ images with a very small amount of extra latency.
To the uninitiated ones, Google Visual Core is a custom-designed co-processor that has been designed from scratch to deliver maximum performance at low power. It debuted on Pixel 2 phones so as to offer better HDR+ shots.
And.. how the process of recommendation works?
The recommendation is based on several qualitative features like if the subject is smiling or not and this process happens in real time. This background process also looks for some other facts as well, like lighting, open eyes, emotional expressions, and so on functional and subjection qualities. And, according to Google, a computer vision model designed by the company recognizes these factors while operating in a low-latency mode. Together with the hardware-accelerated MobileNet-based single shot detector, the feature ensures everything is happening within a set latency and power tradeoff constraints.
The blurriness of a picture is detected at a very low-level layer and additional computations are applied to detect complex objective attributes like whether the subject’s eyes are open and objective attributes like expressions of amusement or surprise. And, to further improve the process of recognizing such complex attributes, Google used and trained its more advanced models like knowledge distillation using the methods quantization and a layered Generalized Additive Model (GAM) to identify the exact causes of success or failure.
The camera shots could also include things other than the faces of the people so for such use cases, the Top Shot uses the Frame Scoring Model which includes several sub-models which help in predicting an overall quality score.
Watch how Google demonstrated the feature at its Made by Google event -
Preserving the basic things -
At last, everything needs to be combined in a system with favorable results because eventually the recommendations are going to be seen by the users and they will decide if the Top Shot feature is worth enough. Thus these predictions need to be gauged against what the users perceive as the “top shot.” For this, Google collected data from volunteers to know if the feature offers what it should and it was concluded that the Top feature had more than one good shot. Google also explains that there are many more metrics based on which Top Shot feature was also tested, these include gender, age, ethnicity. This means any process in the background of the Top Shot shouldn’t hamper the basic things of a camera shot.
So, yeah use Top Shot because without it you will miss the perfect shots if you don’t hit the shutter at exactly the right moment.