4D Gaussian editing with diffusion guidance

Abstract

We explore an approach to modify 4D Gaussian reconstructions, by superimposing 2D diffusion-generated imagery with a user-chosen prompt and projection camera angle. The Gaussians are modified using Score Distillation Sampling (SDS) loss, and are graphically altered with minimal impact on their structure and motion. This project relies on the Shape of Motion technique for 4D reconstruction. Attempts to use depth-enabled diffusion models to superimpose 3D generations or to fill gaps in occluded areas resulted in a loss of scene identity. Future work could improve on integration with depth-enabled diffusion models, as well as artist-friendly user interfaces for easier control and wider accessibility.