Intrinsicdiff: extracting intrinsic image components from diffusion models

Abstract

Intrinsic image properties such as surface normal maps, depth maps, shading, and albedo provide a decomposition of scenes that is broadly useful for image editing and scene understanding. This paper proposes an approach to intrinsic image decomposition using latent diffusion models, using the generative prior of Stable Diffusion (versions 1.5, 2.1, and 3) to learn and extract these intrinsic properties. We adapt the pre-trained diffusion model via Low-Rank Adaptation (LoRA) and an extended U-Net architecture, enabling the network to output multiple intrinsic channels with minimal additional parameters. The method is trained on the DIODE dataset , augmented with pseudo ground-truth shading and albedo to provide supervision for all intrinsic components. We attempt to explore single-step (feed-forward) strategy: while multi-step diffusion generates more detailed and accurate intrinsic outputs at the cost of increased compu- tation, single-step inference offers a faster solution with slightly reduced detail. Our results demonstrate that diffusion models can be effectively repurposed for intrinsic image decomposition, highlighting a promising avenue for generative models to yield rich intrinsic representations of scenes. Furthermore, we investigate the impact of model complexity on the quality of intrinsic decomposition.