Please use this identifier to cite or link to this item:
Title: Multimodal data fusion for object detection under rainy conditions
Authors: Liu, Ting Tao
Keywords: Engineering::Computer science and engineering::Computing methodologies::Image processing and computer vision
Issue Date: 2022
Publisher: Nanyang Technological University
Source: Liu, T. T. (2022). Multimodal data fusion for object detection under rainy conditions. Final Year Project (FYP), Nanyang Technological University, Singapore.
Project: W3357-212
Abstract: In recent years, autonomous driving technology has developed rapidly because there is demand to reduce number of the death caused by serious accident. As an important part of the autonomous driving perception algorithm, object detection algorithms have been given more and more attention and impressive progress has achieved. However, under rainy conditions, pure vision-based object detection methods can be severely affected, resulting in a large number of missed and wrong detections. At the same time, radar sensors are more robust to rain than camera sensors. Therefore, this project aims to implement a Multimodal Object Detection that combines radar and camera data for object detection in rainy weather conditions. Firstly, because of the lack of public multimodal rainy datasets, we generated our own rainy dataset based on the nuScenes dataset with Cycle GAN and some of our own recorded rainy images. The rainy dataset was then used to train a Rainy Image Classifier to give a score for the rainy degree of each data frame. Then the data stream was weighted according to the scores, so that the data generator would focus more on the radar data for rainy frames. This kind of generator is the proposed Adaptive Data Generator in this project. Based on the Adaptive Data Generator, we proposed the Multimodal CRF-Net and compared it with a purely visual-based approach and CRF-Net on the rainy and non-rainy datasets. Finally, we presented and discussed the experimental results. Overall, the results show that the Multimodal CRF-Net proposed in this project performs better than pure visual-based method and CRF-Net on our generated rainy dataset. However, it should also be noted that the results of this project have some limitations and shortcomings: the overall mAP is not high enough, the epochs are all set to 10 may not be enough, the dataset may not be large enough, etc. We recommend some approaches to improve in the future work section such as building a multimodal rainy dataset, trying other training parameters, enhancing the image visibility in rain and fusing fata from more sensors.
Schools: School of Electrical and Electronic Engineering 
Fulltext Permission: restricted
Fulltext Availability: With Fulltext
Appears in Collections:EEE Student Reports (FYP/IA/PA/PI)

Files in This Item:
File Description SizeFormat 
FYP Report by LiuTingtao_April 2022.pdf
  Restricted Access
4.85 MBAdobe PDFView/Open

Page view(s)

Updated on May 17, 2024


Updated on May 17, 2024

Google ScholarTM


Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.