Empirical study on multi-unet diffusion models

Abstract

We explore augmenting a pre-trained diffusion model using Low-Rank Adaptation (LoRA) to rapidly train specialised Unet models for specific diffusion timesteps. By leveraging the trained model’s existing understanding of the data distribution, this approach enables efficient specialisation. We investigate the interactions between these multi-specialised Unet models, evaluate their performance, and explore various strategies for integrating their capabilities. The aim of this project is to explore if this multi-model adaptation approach can provide an efficient method for enhancing model performance by building upon an already pre-trained diffusion model.