3D multi-modal perception for autonomous vehicles in urban environments

Abstract

Autonomous driving is one of the most popular research topics at present, and 3D object perception is an essential part of it. 3D perception aims to ensure the safe operation of autonomous systems in complex environments. To acquire sufficient information about the environment, autonomous vehicles are equipped with a variety of sensors, such as LiDAR, Cameras, Radars, etc. How to utilize these sensors to perform accurate 3D perception is a challenging task.

Now, Lidar-based 3D perception methods can achieve high precision, but due to their high cost, they are difficult to popularize. Camera-based methods have obvious disadvantages in depth estimation, leading to lower detection accuracy. At the same time, mainstream models based on frameworks like PyTorch and TensorFlow, due to their bulky size, complex environmental dependencies, and slow inference speed, are not suitable for deployment on edge devices with limited memory and computational power.

To address the depth estimation issue in Camera-based models, this dissertation designs a Multi-modal 3D object detection model. On the basis of Camera-based models, Lidar data supervision is introduced during the training process to enhance the model's depth estimation capability. Experiments show that the addition of Lidar supervision brings a considerable improvement in detection accuracy. To tackle the issues of PyTorch model size and inference speed, this dissertation introduces an acceleration pipeline. By utilizing ONNX transformation, TensorRT acceleration and calibration, we reduce the model size by 60% and increase the inference speed by 11 times. To address deployment environment issues, this dissertation introduces Docker and ROS to improve the usability of model deployment.