Variance-aware learning based material recognition using stereo cameras

Abstract

Material recognition exploits rich information from surfaces, which general object recognition fails to stress on, and therefore has wide application in real-world scenarios like construction management, autonomous agent navigation and image editing of interior design. Employing deep neural networks as a robust feature extractor, recent methods successfully decode reflectance and texture cues from the angular variation of materials, with equipment like light-field cameras and robotic arms. Stereo cameras, as a two-lens system, also provide differential angular images that reveal material’s latent characteristic. Moreover, a depth map can be estimated from a stereo camera system. Inspired by the unprecedented mobilization of stereo cameras, we explore the potentiality of deploying algorithms that incorporate both differential angular information and geometry information on a common multi-camera mobile phone and benchmark their performance. The task demands a new dataset that has not been collected before. Therefore, we build a data collection tool, which is a mobile app that captures each camera’s output image from a multi-camera system and processes the images with calibration information. With this tool, we collected a mini-dataset for benchmarking. Lastly, to tackle the challenge posed by the complex natural condition in outdoor scenes and decouple confounding patterns from local invariant features, we propose a novel method that models intra-class variance with a variance encoder. The variance encoder also serves to mine hard negative samples from easy negative samples, which leads toward a more robust material recognition.