Vision-based 3D information modeling and applications

Abstract

We can infer the 3D structure of our surroundings simply by looking. It is long hoped that imaging devices can mirror such an ability, which is crucial to many computer vision tasks. This thesis is about our work on developing algorithms, as well as utilizing novel optical devices, in particular light-field cameras, to infer 3D information without active illuminations, and how such information can be used in various practical applications:

• We develop an algorithm to synthesize novel views. When we shift to a different viewpoint, certain scene points not captured in the input image will be revealed. We first infer the colors of these points, based on 3D plane notations, and then use the expanded scene points to generate the target image.

• We propose using depths calculated from light field (LF) data to remove reflections. The depths are used to roughly identify background and reflection scene points. We then reconstruct the background and the reflection layers using scene points identified.

• Finally, we utilize depth information to help identify different kinds of materials. Given images captured using a multi-camera cell phone, we estimate a depth probability map. The estimated depth probability map, together with one of the color images, are then inputted into a trained neural network to determine the material type.

The first method utilizes 3D plane notations, while the rest use depths recovered from LF data or stereo images. 3D information is crucial in accomplishing these tasks.