SDFLabel: Autolabeling 3D Objects With Differentiable Rendering of SDF Shape Priors

September 2020

tl;dr: Using differentiable rendering for automatic labeling.

Overall impression

Use 2D regression to predict NOCS map and a shape vector. The NOCS map can be used with lidar to extract a sparse 3D model, and the shape vector can be used with DeepSDF to decode a 3D model. The compute approximate pose with 3D matching. Then calculate 2D and 3D loss for back-propagation for refinement.

Previous work such as 3D RCNN and RoI10D uses PCA or CAE (conv auto-encoder) to predict the shape of cars. This is not end-to-end differentiable. DeepSDF enables backpropagation onto a smooth shape manifold and is more powerful.

Autolabel is still not as good as lidar labels, but very close. Closer in performance in BEV rather than 3D, but only BEV should be good enough for autonomous driving. But the 3D drop may not be real as the autolabels are tight 3D bbox as compared to KITTI3D lidar labels.

Key ideas

Technical details