Adapting learning-based video retargeting to mainstream video application

Abstract

Generation-based learning methods have achieved great success in image retargeting. Additionally, utilizing both segmentation and inpainting has achieved great success in user satisfaction. However, the current limitations arise when the user wishes to manually select which part to segment, and also to make the object more realistic after the retargeting. This paper aims to address the above two limitations. We aim to explore segmentation models to facilitate a more flexible paradigm for each video frame which can help improve the accuracy of image retargeting and improve the quality of the retargeted image and additionally add more ways of relocating the object. The proposed pipeline combining the approach of the ORPVR, Segment anything 2 (SAM2) as well as the harmonizer has yielded promising results. A limitation was discovered for the object relocation algorithm and addressed accordingly with the improved widescreen to portrait mode for the object relocation algorithm to retarget images from widescreen (16:9) to (9:16). We also evaluate the components of the pipeline to see how it could be applied in a real world setting and discovered that the inpainting segment and the number of frames was the main contributing factor for the total time taken to run the proposed pipeline. We also discuss the possible limitations which was revealed on custom image retargeting dimentsions such as portrait (9:16) to widescreen (16:9) and future work to make the relocation algorithm more suitable for a wider variety of dimensions.