Neural radiance selector: find the best 2D representations of 3D data for CLIP based 3D tasks

Abstract

Representing the world in 3D space provides vivid texture and depth information. However, 3D datasets currently do not match the scale of 2D datasets. There is a growing trend in representing 3D data as multi-view 2D images and using large-scale 2D models, to solve 3D tasks. In this work, we present the Neural Radiance Selector, a method that automatically selects the optimal 2D representations of 3D data. Instead of indiscriminately sampling multi-view 2D images, we define the optimal 2D views as those capable of reconstructing the entire 3D scene with a conditional neural radiance field. We propose two distinct methods for 3D point cloud data and 3D implicit models to achieve faster inference. We demonstrate the efficacy of our methods in various 3D tasks, including zero-shot 3D point cloud classification, 3D implicit model classification, and language-guided NeRF editing.