Deep Depth Completion of a Single RGB-D Image

June 2019

tl;dr: Two stage solution for depth completion problem: first stage use RGB to predict surface normals. Then use depth measurement as regularization for global optimization.

Overall impression

The paper proposed that surface normals are easier to predict as they do not scale with depth. The methods is superior to previous SOTA in predicting depth with RGB only image.

The two stage pipeline is flexible but also slow. The advantage is that the first stage only depend on RGB and does not need to be retrained for different depth sensors. The second stage is a sparse optimization problem with depth constraint with real depth measurement data which can be solved on CPU (the inference time is ~1 sec thus slow).

DeepLidar proves that the depth can be done with an end-to-end solution. However it also converts lidar to image. Maybe we could combine the idea of DeepLidar with PseudoLidar.

Key ideas

Technical details