Problem

The client developed a 3D reconstruction and measurement app for iOS and Android for wound measurement and analysis. The app worked well for iPhones and Android phones with ToF/LiDAR sensors. However, without dedicated depth sensors, the quality of depth measurements was low-quality.
Brainstorm was engaged to develop a custom algorithm to obtain depth map measurements for Android devices without depth sensors, using only camera images and accelerometer readings.

Solution

The proposed solution was to, instead of one-shot measurements, scan the environment by obtaining multiple images of the environment and aggregating all depth data.
The solution is a depth data processing pipeline that includes:

• Scanning starts from a quality frame (with plenty of depth points)
• Aggregating depth points from multiple frames
• Projecting depth points to the final camera frame for measuring.

The solution was implemented in Java, ARCore, and Raw Depth API. The most important aspect of working with the 3D reconstruction of wounds is that the algorithm needs to work well with human skin, faces, etc. For testing purposes, we perform 3D scanning of our developers in order to generate dense point clouds of their faces and arms. After that, we imported the colored point clouds into 3D software like MeshLab and performed validation of the results.

Results

Our solution managed to increase the number of supported Android devices by 100x by relaxing the requirement that the device needs to contain a ToF sensor and supporting all ARCore-supported devices (with Depth API support). Our solution managed to preserve the accuracy and robustness of 3D reconstruction and measurement results through depth data aggregation from many frames, guided and explained to the user through an intuitive UI.