Deep learning methods with less supervision

Abstract

This study examines weakly supervised semantic segmentation in video datasets, eval- uating the effectiveness of various prompting strategies on the initial video frame. The prompting approaches explored include segmentation masks from initial frames, sparse point-based prompting, augmented point-based techniques, text-based object grounding, and a combination of the methods. The findings indicate that the un- derlying models effectively propagate key image features across frames, mitigating challenges such as motion blur, perspective shifts, and occlusion. Augmenting point- based prompts enhances segmentation accuracy by reducing semantic ambiguity, while text-based prompting with a grounded object detection model offers a low-annotation alternative for object segmentation and tracking. The integration of Grounding DINO and SAM2 for text based prompting also shows strong results but is heavily reliant on the precision of image-level semantic labels. Overall, the results validate the efficacy of prompt-based segmentation in weakly supervised settings and underscore its potential for generating accurate semantic masks in video analysis.