VScenimefy: fast and stable anime video stylization via diffusion prior distillation

Abstract

Automatic rendering anime scenes from complex real-world images is of significant practical value. Prior works, from GANs to Diffusion-based methods, have shown impressive results in image-level translation. However, their extension to video exposes issues such as weak stylization, obvious flickering, and high computational demands. The core challenges of this task lie in 1) maintaining semantic preservation for complex scenes; 2) ensuring temporal consistency; 3) capturing the unique features of anime style. In this study, we propose VScenimefy, a novel three-stage video-to-video translation pipeline leveraging pretrained diffusion model’s prior as a pseudo supervision to narrow the domain gap. A low-level alignment technique with pseudo paired data is introduced to improve stylization and ensure structural integrity. Besides, we present an optical-flow-based video finetuning framework, yielding temporally consistent and stylistically coherent videos. Our extensive experiments demonstrate the superiority of our method over state-of-the-art baselines in terms of perceptual quality and quantitative performance. Remarkably, our model runs in real-time, confirming its feasibility for practical application and highlighting its potential to empower GANs for efficient video stylization through diffusion model distillation.