Title:RadarRGBD A Multi-Sensor Fusion Dataset for Perception with RGB-D and mmWave Radar

Abstract:Multi-sensor fusion has significant potential in perception tasks for both indoor and outdoor environments. Especially under challenging conditions such as adverse weather and low-light environments, the combined use of millimeter-wave radar and RGB-D sensors has shown distinct advantages. However, existing multi-sensor datasets in the fields of autonomous driving and robotics often lack high-quality millimeter-wave radar data. To address this gap, we present a new multi-sensor dataset:RadarRGBD. This dataset includes RGB-D data, millimeter-wave radar point clouds, and raw radar matrices, covering various indoor and outdoor scenes, as well as low-light environments. Compared to existing datasets, RadarRGBD employs higher-resolution millimeter-wave radar and provides raw data, offering a new research foundation for the fusion of millimeter-wave radar and visual sensors. Furthermore, to tackle the noise and gaps in depth maps captured by Kinect V2 due to occlusions and mismatches, we fine-tune an open-source relative depth estimation framework, incorporating the absolute depth information from the dataset for depth supervision. We also introduce pseudo-relative depth scale information to further optimize the global depth scale estimation. Experimental results demonstrate that the proposed method effectively fills in missing regions in sensor data. Our dataset and related documentation will be publicly available at: this https URL.