Mitigating shortcut learning and improving out-of-distribution generalization for vision and vision-language models

Vision and vision-language models have achieved impressive results on independently and identically distributed (IID) benchmarks, where training and test data share the same distribution. However, their performance often degrades significantly on out-of-distribution (OOD) data, whose distribution differs from that of the training data. A key reason for this lack of generalization is shortcut learning — the tendency of models to rely on spurious correlations or superficial features that are predictive on training data but do not reflect the underlying cause of the correct outcome. Shortcut learning arises from two main factors: (1) training data frequently contain spurious correlations that do not hold in other distributions, and (2) these correlations are easy for models to learn, as they correspond to flat minima in the loss landscape that gradient-based optimization favors. Consequently, models often prioritize non-causal patterns over robust, generalizable features.

Mitigating shortcut learning is therefore crucial for improving OOD generalization. This challenge can be addressed from two complementary perspectives. On the data side, intervention techniques such as data augmentation and synthetic data generation can break spurious correlations and highlight invariant features. On the model side, architectural designs that broaden the minima associated with causal features while suppressing those linked to shortcut features can encourage the learning of more generalizable features. Together, these approaches promote the learning of robust representations and improve OOD generalization. This thesis investigates shortcut learning as a fundamental challenge to out-of-distribution generalization in vision and vision-language models across a progression of tasks, including video recognition, video generation, and vision-language understanding. These tasks are organized from single-modal discrimination to pixel-level generation and cross-modal reasoning, enabling a systematic evaluation of shortcut mitigation strategies under increasing representational and modeling demands.

In video action recognition, we introduce data augmentation techniques aimed at mitigating shortcut learning from static visual cues. We propose StillMix, which mixes bias-inducing frames into training videos while preserving the action labels of videos. This augmentation disrupts spurious correlations between static features and action labels, thereby enhancing model generalization to OOD visual appearances.

In few-shot human action generation, we propose a combination of data augmentation and model design strategies to mitigate shortcut learning driven by appearance cues. We introduce FLASH, which creates video pairs that share the same motion but differ in appearance, and conduct feature alignment between the two videos to promote the learning of transferable motion representations. This method reduces appearance overfitting in the low-data setting and enhances motion generalization.

In vision-language compositional understanding, we develop synthetic data generation techniques to reduce reliance on coarse-grained visual-text alignments. We propose SPARCL, which generates multimodal data with subtle variations and trains the model to distinguish these subtle differences. This approach enhances the model ability to capture compositional semantics beyond surface-level correlations.

These contributions show that mitigating shortcut learning through data interventions and model designs is key to improving OOD generalization in vision and vision-language tasks.