3D reconstruction from single images

Abstract

3D reconstruction from single images is a fundamental task in computer vision, and it has a wide range of applications, including anime films, robot object interaction, AR, VR and 3D games. Due to the task's complexity and significant information loss from 3D to 2D, traditional methods are ineffective. The use of deep learning and large-scale datasets to learn priori knowledge is a promising direction and has achieved varying degrees of success. However, 3D deep learning requires a large number of annotated 3D objects. These annotations are usually tedious and time-consuming. In view of this fact, this thesis presents a method to generate annotated 3D datasets automatically, and experiments demonstrate the effectiveness of the generated datasets for 3D deep learning tasks.

However, the generated 3D datasets can only imitate objects with basic topology. For 3D reconstruction tasks, objects with higher precision are required to capture the widely varying structures of objects in daily life. This thesis provides three methods to enhance the quality of the single-view 3D reconstruction using publicly accessible 3D datasets. - Most of the existing reconstruction methods focus too much on the reconstruction metrics, such as Chamfer Distance(CD), and neglect the visual consistency between the reconstructed 3D objects and the objects in the given image. In our first framework, we enhance the visual quality of the reconstructed shapes by emphasising the consistency between the reconstructed 3D shape and the object's boundaries and corner points in the given image. - Earlier point cloud-based reconstruction techniques could only generate point clouds with preset resolutions. Moreover, to obtain dense point clouds, previous research need to employ multistage training. We propose PushNet, which can produce point clouds with arbitrary resolutions, including very dense resolutions, in an end-to-end manner and only require sparse point clouds during training. - To improve the reconstruction quality in local areas, a two-stage reconstruction approach is proposed. We overcome the shortcomings of previous pixel-aligned reconstruction methods and produce reliable results without predicting camera parameters. Our method incorporates local information about each pixel in the first stage and focuses on global information in the second stage.