Multi-modal semantic segmentation in poor lighting conditions

Abstract

Semantic segmentation is a complicate dense prediction task that consumes significant computational resources, and the use of multi-modal RGB-T data makes its computational burden even more severe. This dissertation presents a novel and lightweight network for RGB-T semantic segmentation with a parameter-free feature fusion module that facilitates efficient fusion between modalities. The proposed method integrates both modalities by leveraging multi-scale features from both RGB and T domains in different feature extraction stages. Specifically, we employ a dual-encoder architecture to extract RGB-T features and fuse them with a parameter-free cross-modal attention mechanism, taking the advantage of the complementary information provided by the two modalities to improve segmentation accuracy. Besides, we further investigate the impact of different pretrained strategies on the performance of the model. We evaluate our approach on several benchmark datasets, including the MFNet and PST900 datasets. Experimental results show that our approach outperforms real-time state-of-the-art methods in the literature while showing comparable performance with state-of the-art methods that require up to 100 times the computational complexity. Our findings demonstrate the effectiveness of lightweight RGB-T model for semantic segmentation and highlight the potential of this approach for various real-world applications.