Advancing 3D scene understanding through discriminative and generative learning approaches

Abstract

This thesis explores the crucial role of Computer Vision in endowing computers with general intelligence, focusing on developing algorithms that enable machines to perceive and understand their three-dimensional surroundings. The research is divided into two parts: discriminative and generative learning approaches, with three core chapters formulating 3D scene understanding. From a discriminative learning perspective, a novel approach to point cloud segmentation is devised, which is crucial for road scene perception. The proposed method processes point clouds as a whole while retaining local information, achieving high accuracy in segmenting objects from scenes despite the computational challenges of processing large input data. The generative learning approach focuses on generating entire 3D scenes from 2D images. Prior art methods in rendering 3D scenes via volumetric rendering are studied, and an end-to-end learning approach with transformers is proposed as an alternative to physics-based approaches. Novel methods to capture lighting information of scenes, inspired by modern game engines, are devised to improve rendering quality. Further investigation into new rendering methods with rasterisation of 3D Gaussian spheres is conducted, along with a different method for capturing lighting information to enhance rendering quality. The research contributes to the overarching goal of helping computers perceive and interact with the 3D world, offering numerous advantages for downstream applications such as autonomous vehicles, augmented reality, and virtual collaboration.