Exemplar based image colourization using diffusion models

Abstract

Current state-of-the-art exemplar-based colourization models, such as UniColor and Deep Exemplar, have shown promising results in conditional image colourization. However, their performance heavily relies on the choice of the exemplar image, as their method relies on hint points generated through similarity matching leveraging a correspondence network. This dependency often leads to limitations in cases where the exemplar is not fully aligned with the source image, resulting in incomplete colourization and limited colour richness. In this paper, we present a novel framework that builds upon and modifies the ControlNet architecture, utilizing the generational capabilities of diffusion models for the exemplar-based image colourization task. By integrating a trainable CLIP image encoder into the ControlNet architecture and manipulating the attention layers, our approach allows for more accurate and vibrant colourizations that are consistent with the exemplar image. Furthermore, by tapping into the extensive knowledge of pretrained Stable Diffusion models, which they have acquired from training on massive datasets, we can effectively handle cases where the exemplar does not perfectly align with the source image and the missing information is filled using the prior knowledge of the diffusion model. We conduct extensive experiments on the ImageNet dataset and compare our method against state-of-the-art baselines, demonstrating its effectiveness in generating vivid, realistic, high-quality colourizations. Our results highlight the efficacy of diffusion models in the domain of conditional image colourization and open up new avenues for future research in this area.