PointRend: Image Segmentation as Rendering

January 2020

tl;dr: Find most uncertain points in segmentation and use both coarse RoI feature map and fine feature map to predict results.

Overall impression

The paper tells a great story about borrowing ideas from rendering to segmentation. However the idea of coarse-to-fine has been explored extensively before. The main novelty of this paper is how to save computation by non-uniform sampling.

A regular grid will invariably oversample the smooth areas while simultaneously undersample object boundaries. For semantic segmentation, we use feature map of 1/8 size of input. Or 28x28 for instance segmentation.

PointRend is a module that can be incorporated in instance/semantic segmentation frameworks to improve results.

Key ideas

Technical details