PnPNet: End-to-End Perception and Prediction with Tracking in the Loop

September 2021

tl;dr: Have tracker in the loop improves perception and prediction. Track level features is important for long term trajectory prediction.

Overall impression

MOT has two challenges: the discrete problem of data association and the continuous problem of trajectory estimation.

Previous methods with perception and prediction only uses tracking as postprocessing. The full temporal history contained in tracks is not used by detection and prediction. They usually limit the time step to 3, instead of a long-term trajectory. Their performance usually saturates with fewer than 1 second of sensor data.

PnPNet includes a tracker in the loop and thus can be trained end to end. The Hungarian matching cost function is learnable.

Key ideas

Technical details