Automatic image colorization

Abstract

Recent advancements in image colorization models—both unconditional and conditional— have achieved remarkable results, generating rich and vibrant colours. These models also allows users to guide the colorization process through input conditions such as text prompts, brush strokes, reference exemplar images, or combinations of them. Colorization has seen significantly improved output quality and color richness through breakthroughs in deep learning architectures, such as Transformers, GANs, and Stable Diffusion. However, most of these research prioritized enhancing the quality of colorization over improving inference speed. While large models like stable diffusion produce impressive results, their slowprocessing speeds remain a key limitation. In this paper, we propose a novel framework that utilizes ControlNet architecture for image colorization and then perform model distillation on it. ControlNet is trained for both unconditional and prompt-based colorization, using LLaVA to generate image prompts. The trained ControlNet model serves as a teacher in the model distillation pipeline, training a smaller, faster student model designed to deliver significantly quicker outputs with negligible performance degradation. We conduct extensive experiments on the ImageNet and COCO-stuff datasets to evaluate the proposed approach against stateof- the-art models for both unconditional and prompt-based colorization. This work aims to advance the field by offering an approach that combines vivid and vibrant colorization with fast inference speed, paving the way for more practical and efficient applications.