Supporting Qualcomm Snapdragon flagship, Adaps Photonics dToF technology creates a new "Director Mode" for Android

Time:2022-11-19
On November 16th, the Snapdragon 8 Gen 2 processor and "Director Mode" was officially unveiled at the Qualcomm Snapdragon Technology Summit. Director mode is created by Qualcomm in collaboration with Adaps Photonics and Arcsoft for Android.

On November 16th, the Snapdragon 8 Gen 2 processor was officially unveiled at the Qualcomm Snapdragon Technology Summit. At the Sanya Station conference, Heng Xu, senior director of product marketing at Qualcomm, demonstrated the highly anticipated "Director Mode" on the Snapdragon 8 Gen 2. This mode is created by Qualcomm in collaboration with Adaps Photonics and Arcsoft, and for the first time introduced a new "Director Mode" for Android, providing Android users with an unparalleled video shooting experience.

Heng Xu, senior director of product marketing at Qualcomm, is introducing "director mode"

 

Mobile phone perceives 2D to 3D, everyone is a great director

Today, Vlog (Video Blog) has gradually become the most important way for people to record their lives and express their personalities. A trip, a party, a game, or a cooking can all become the material for video recording life. The Vlog presentation of professional video bloggers is inseparable from excellent shooting, planning, and editing, and there is a certain creative threshold. In order to allow more ordinary consumers to shoot whenever and wherever they want, the "Mobile Director Mode" came into being.

In the "Director Mode", with the movement of the character's line of sight, Arcsoft's AI algorithm detects changes in the points of interest in the screen and guides the camera's focus and follow-focus to achieve the effect of directorial filming. At the same time, the software uses complex depth estimation algorithms to simulate optical lens blur on areas beyond the subject's focal plane, resulting in high-quality cinematic effects.

The dToF hardware provided by Adaps Photonics has made outstanding contributions to the improvement of "focus", "depth estimation" and "complex light environment". By perceiving the 3D world, it surpasses the pure vision (no ToF) of most current mobile phones scheme.

The Spot dToF is a depth sensing scheme that uses a scattered point of light to obtain direct flight time of light under the position of the scattered point. Compared with traditional PDAF (phase focusing method), Spot dToF can increase the speed of focusing by 1-2 times in low light or backlight conditions, and the focusing can be completed within 2 frames without adjusting the "bellows" action of exposure. This is critical for smooth video shooting. Additionally, based on the working principle of dToF, it can well distinguish the interference of transparent materials, such as glass, on the target distance and even achieve glass recognition, that is, the glass focus tracking function displayed in the video. Moreover, due to the high spatial resolution (greater than 24x24) achieved by Spot dToF depth measurement, advanced operations such as video focus switching and focus tracking can be performed at any position on the video screen. Existing limited point dToF products on the market have significant challenges in realizing this function. Finally, Spot dToF can also enhance depth estimation in weakly textured walls, desktops, and weak light and backlight environments.

 

Adaps Photonics' Spot dToF solution unlocks professional movie effects.

The demo hardware of the director mode is a LiDAR scanner composed of a Adaps Photonics Spot dToF chip and a module based on this chip. It is placed on a Qualcomm Reference Design (QRD) phone equipped with Snapdragon 8 Gen 2 using dedicated external circuits and enclosures. After calibration and algorithm injection by Hikvision, the QRD phone is demonstrated and experienced at the exhibition site for visitors and media. Thanks to the powerful computing power of the Snapdragon 8 Gen 2, the power consumption and delay of dToF depth calculation and director mode algorithm calculation have been greatly reduced. 

Compared with iToF solutions, dToF has lower power consumption, higher accuracy, longer outdoor distance, and glass detection functions. Adaps Photonics' high-performance Spot dToF chip adopts the most advanced 3D stacking (hybrid bonding) process, with up to 20% photon detection efficiency in the 940nm band, and frame rates of 15-60FPS. The chip's ranging range covers 0.1m to 19m, with Rx power consumption as low as 70mW, and outdoor performance of at least 6m@100klux.

 

  

Standing on the shoulders of giants, director mode logs into Android

Using Spot dToF to achieve functions similar to a LiDAR scanner first appeared in the low-light portrait mode of the iPhone 12 Pro. In 2021, the iPhone released a "cinematic mode" that helps users shoot movie-quality images in a concise way. An iPhone 13 Pro that is equipped with a LiDAR scanner has better movie mode performance even under glass, low-light, and backlit scenes.

Now, with the polishing by Adaps Photonics, Qualcomm, and Arcsoft, the "director mode" on an Android phone has been successfully released. In the future, with the 3D perception ability of Spot dToF, the weak light portrait mode, and the intelligent camera based on AI image semantic segmentation technology (segmentation), can continue to be improved on Android phones.

In their previous cooperation, Adaps Photonics and Arcsoft also completed the RGB + ToF depth completion algorithm based on neural networks, as well as the mesh based on ARCore SLAM. The depth completion algorithm provides high-resolution depth maps, which can be used to achieve video blurring effects, AR virtual-real occlusion, and even motion capture. The mesh algorithm demonstrates faster SLAM effects based on dToF and more accurate 3D scene modeling capabilities.

 

With collaborative innovation, we break through image boundaries

In the development of the director mode, Adaps Photonics has received strong support from the product marketing and R&D teams of Qualcomm China. Under the leadership of Qualcomm's product marketing department, the three parties cooperated in full sincerity and completed the effect from project approval to exhibition sample in literally just three months. This reflects China's good software and hardware ecosystem and further demonstrates Adaps Photonics' technological strength and product iteration ability. Adaps Photonics will continue to make efforts to open the "smart eye" for diversified applications such as phone 3D sensing, car-mounted LiDAR, robotics depth vision, AR/MR devices, and push the door to the 3D world.

 

Giving a product demonstration to Ms. Hou Mingjuan, the global vice president of Qualcomm, by Zang Kai, CEO of Adaps Photonics