Extrapolation algorithms for raw depth maps captured by kinect sensor

Abstract

Depth map contains depth information of a scene that measures the distances from a viewpoint to the objects and surfaces in the scene. It has wide applications in 3D computer graphics. Microsoft Kinect sensor provides a simple and accessible way to obtain depth maps. However, almost every raw depth map captured by Kinect has some “hole” regions where the real depth values are missing. The objective of the project is to extrapolate (to predict and fill) the “holes” with proper depth values. In the report, four extrapolation algorithms are proposed with the first three implemented offline and the last one implemented in real time. The proposed algorithms use different kinds of available information and apply to different kinds of “holes” on raw depth maps. The mean value inpainting (MVI) algorithm uses the depth information on a global base. It can give smooth depth transitions but not sharp object boundaries. The similar color based median filter (SCBMF) algorithm uses both depth and color information on a local base. It can give sharp object boundaries but not smooth depth transitions. The segmentation based median filter (SBMF) algorithm is similar to the second algorithm except that the two algorithms use the color information in different ways. The spatial mode and temporal weighted mean (SMATWM) algorithm uses depth information on a local base and time information on a global base. It is applicable to small “hole” regions. It improves the visual effect of depth frames significantly in simple real-time implementation. In the future, an adaptive extrapolation algorithm and its fast algorithm can be developed to select the most suitable algorithm for different kinds of “holes” in a single depth map and to achieve real-time extrapolation.