Body movement mimic - video-based human body motion transfer

Abstract

Animating a still image to reflect realistic human motion is a complex yet transformative task with wide-ranging applications in entertainment, virtual reality, and digital media. While recent advances in human motion transfer models, such as MagicAnimate and AnimateAnyone, have made significant strides, they still face significant challenges. MagicAnimate for one fails to preserve the natural appearance of the reference subject due to the misalignment of body shape between the driving video’s DensePose representation of the driving video and the reference image. This often results in unnatural modifications and distorted outputs. To tackle this challenge, this project proposes training a shape-aware model that generates DensePose representations aligned with the reference image’s body shape. The model is a dual-branched U-Net architecture with cross-attention that effectively transfers textures and skeletal structure. Various experimental approaches were explored and refined to optimise the model, ultimately resulting in the best performance that fits memory and time constraints of this project. This enhanced representation is then used as the driving video within the MagicAnimate pipeline, enabling more accurate motion transfer by leveraging its ability to preserve fine details and minimise artifacts. The findings from this study contribute to advancing the robustness of human motion transfer techniques and enhancing their applicability in diverse real-world scenarios.