Title:MonoDETR: Depth-guided Transformer for Monocular 3D Object Detection

Abstract:Monocular 3D object detection has long been a challenging task in autonomous driving, which requires to decode 3D predictions solely from a single 2D image. Most existing methods follow conventional 2D object detectors to localize objects based on their centers, and predict 3D attributes by neighboring features around the centers. However, only using local features is insufficient to understand the scene-level 3D spatial structures and ignore the inter-object depth relations from contextual cues. In this paper, we introduce a novel framework for Monocular DEtection with a depth-guided TRansformer, named MonoDETR. The vanilla transformer is modified to be depth-aware and the whole detection process is then guided by depth. Specifically, we represent 3D object candidates as a set of queries and adopt an attention-based depth encoder to produce non-local depth embeddings of the input image. Then, we propose a depth-guided decoder with depth cross-attention modules to conduct both inter-query and query-scene depth feature interactions. In this way, each object query estimates its 3D attributes adaptively from the depth-guided regions from the image and is no longer constrained to use only neighboring visual features. MonoDETR is an end-to-end network without extra data or NMS post-processing and achieves state-of-the-art performance on KITTI benchmark with significant gains. Extensive ablation studies demonstrate the effectiveness of our approach and its potential to serve as a transformer baseline for future monocular 3D object detection research. Code is available at this https URL.