Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/171830
Title: Counterfactual samples synthesizing and training for robust visual question answering
Authors: Chen, Long
Zheng, Yuhang
Niu, Yulei
Zhang, Hanwang
Xiao, Jun
Keywords: Engineering::Computer science and engineering
Issue Date: 2023
Source: Chen, L., Zheng, Y., Niu, Y., Zhang, H. & Xiao, J. (2023). Counterfactual samples synthesizing and training for robust visual question answering. IEEE Transactions On Pattern Analysis and Machine Intelligence, 45(11), 13218-13234. https://dx.doi.org/10.1109/TPAMI.2023.3290012
Journal: IEEE Transactions on Pattern Analysis and Machine Intelligence
Abstract: Today's VQA models still tend to capture superficial linguistic correlations in the training set and fail to generalize to the test set with different QA distributions. To reduce these language biases, recent VQA works introduce an auxiliary question-only model to regularize the training of targeted VQA model, and achieve dominating performance on diagnostic benchmarks for out-of-distribution testing. However, due to the complex model design, ensemble-based methods are unable to equip themselves with two indispensable characteristics of an ideal VQA model: 1) Visual-explainable: The model should rely on the right visual regions when making decisions. 2) Question-sensitive: The model should be sensitive to the linguistic variations in questions. To this end, we propose a novel model-agnostic Counterfactual Samples Synthesizing and Training (CSST) strategy. After training with CSST, VQA models are forced to focus on all critical objects and words, which significantly improves both visual-explainable and question-sensitive abilities. Specifically, CSST is composed of two parts: Counterfactual Samples Synthesizing (CSS) and Counterfactual Samples Training (CST). CSS generates counterfactual samples by carefully masking critical objects in images or words in questions and assigning pseudo ground-truth answers. CST not only trains the VQA models with both complementary samples to predict respective ground-truth answers, but also urges the VQA models to further distinguish the original samples and superficially similar counterfactual ones. To facilitate the CST training, we propose two variants of supervised contrastive loss for VQA, and design an effective positive and negative sample selection mechanism based on CSS. Extensive experiments have shown the effectiveness of CSST. Particularly, by building on top of model LMH+SAR (Clark et al. 2019), (Si et al. 2021), we achieve record-breaking performance on all out-of-distribution benchmarks (e.g., VQA-CP v2, VQA-CP v1, and GQA-OOD).
URI: https://hdl.handle.net/10356/171830
ISSN: 0162-8828
DOI: 10.1109/TPAMI.2023.3290012
Schools: School of Computer Science and Engineering 
Rights: © 2023 IEEE. All rights reserved.
Fulltext Permission: none
Fulltext Availability: No Fulltext
Appears in Collections:SCSE Journal Articles

SCOPUSTM   
Citations 50

5
Updated on Jun 19, 2024

Page view(s)

81
Updated on Jun 22, 2024

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.