Please use this identifier to cite or link to this item:
Title: Emergent semantic segmentation: training-free dense-label-free extraction from vision-language models
Authors: Luo, Jiayun
Keywords: Computer and Information Science
Issue Date: 2024
Publisher: Nanyang Technological University
Source: Luo, J. (2024). Emergent semantic segmentation: training-free dense-label-free extraction from vision-language models. Master's thesis, Nanyang Technological University, Singapore.
Abstract: From an enormous amount of image-text pairs, large-scale vision-language models (VLMs) learn to implicitly associate image regions with words, which is vital for tasks such as image captioning and visual question answering. However, leveraging such pre-trained models for open-vocabulary semantic segmentation remains a challenge. In this thesis, we propose a simple, yet extremely effective, training-free technique, Plug-and-Play Open-Vocabulary Semantic Segmentation (PnP-OVSS) for this task. PnP-OVSS leverages a VLM with direct text-to-image cross-attention and an image-text matching loss to produce semantic segmentation. However, cross-attention alone tends to over-segment, whereas cross-attention plus GradCAM tend to under-segment. To alleviate this issue, we introduce Salience Dropout; by iteratively dropping patches that the model is most attentive to, we are able to better resolve the entire extent of the segmentation mask. PnP-OVSS does not require any neural network training and performs hyperparameter tuning without the need for any segmentation annotations, even for a validation set. PnP-OVSS demonstrates substantial improvements over comparable baselines (+29.4\% on PASCAL VOC, +13.2\% on PASCAL Context, +14.0\% mIoU on MS COCO, +2.4\% on COCO Stuff) and even outperforms most baselines that conduct additional network training on top of pretrained VLMs.
DOI: 10.32657/10356/175765
Schools: School of Computer Science and Engineering 
Rights: This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License (CC BY-NC 4.0).
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Theses

Files in This Item:
File Description SizeFormat 
Thesis_Master_of_Engineering_Luo_Jiayun_DRNTU.pdf71.06 MBAdobe PDFThumbnail

Page view(s)

Updated on Jul 22, 2024


Updated on Jul 22, 2024

Google ScholarTM




Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.