Hierarchical framework of collaborative 3D semantic mapping based on semantic segmentation

Abstract

In recent years, significant progress has been made in semantic segmentation and 3D geometric mapping. As a result, multi-robot systems are expected to operate in increasingly complex environment with intelligent ability, such as dynamic perception and active navigation. In this case, comprehensive contextual understanding of the surroundings is a critical challenge for multi-robot perception system. Attempts have been made from single robot semantic mapping, collaborative 3D geometric mapping. However, there is a gap in performing accurate and largescale semantic mapping for multiple robots. This thesis goes a step further by focusing on a hierarchical multi-robot semantic mapping framework.

Prior to semantic mapping, the semantic segmentation model is added to the robot system so that the robot can perceive the semantic information of the surrounding environment in real time at a speed of 2 Hz. Then, the thesis proposes a novel hierarchical multi-robot semantic mapping framework, where the problem is addressed in low level single robot semantic mapping and high level global semantic mapping. In the single robot semantic mapping process, Bayesian rules are used for label fusion and occupancy probability update, where the semantic information is added to the geometric map grid. High level global semantic map fusion covers decentralized map sharing and global semantic map updating. Collaborative semantic reconstruction is conducted in two scenarios, that is, NTU dataset and the KITTI dataset. The results show the high quality of the global semantic map, which demonstrates the efficiency, accuracy and versatility of 3D semantic map fusion algorithm in multi-robot missions.