Real-time pedestrian detection based on multi-modal sensor fusion

Abstract

Pedestrian detection in driving assistant system refers to obtain the 3-d coordinate of the pedestrians nearby through the information from the sensors such as RGB camera, depth camera, LiDAR or RADAR. The success implement of deep learning approaches in Computer Vision has spurred considerable progress in the field of pedestrian detection. Fast and accurate detection algorithm can supply more information for the driving assistant system. However, the overall detection performance is still limited to the characteristics of the sensor: detection based on RGB-D camera is limited to the performance of depth camera, while the detection based on complete point cloud data consumes unacceptable computing resources in case of real-time pedestrians detection. In this project, a synthesized survey is firstly conducted to find out the existing detection models. Several state-of-the-art algorithms are then evaluated and compared through Carla simulator, including algorithms which detected only based on camera data, that only based on LiDAR data and that based on both camera data and LiDAR data. Also, several tracking algorithms are compared to pass the information of detected pedestrian to fusion processor. Finally, a real-time pedestrian detection system is established to reach the goal of quick and accurate detection. The result indicates that our detection system has robust and satisfying performance on simulation scene. In addition, the system is designed to be flexible. Users can modify the configuration file to easily selected the needed plugins such as enable LiDAR detection and enable frustum detection. The content of this report includes the introduction of several state-of-the-art detection algorithms, the design of the system and the validation on simulator. Moreover, there is a discussion which focuses on the performance of each detection algorithm and the overall simulation result.