Analytic learning in multi-modal continual test-time adaptation

Abstract

Test-Time Adaptation (TTA) aims to help pre-trained model bridge the gap between source and target datasets using only the pre-trained model and unlabelled test data. A key objective of TTA is to address domain shifts in test data caused by corruption, such as weather changes, noise, or sensor malfunctions. Multi-Modal Continual Test-Time Adaptation (MM-CTTA), an extension of TTA with better real-world applications, further allows pre-trained models to handle multi-modal inputs and adapt to continuously-changing target domains. MM-CTTA typically faces challenges including error accumulation, catastrophic forgetting, and reliability bias, with few existing approaches effectively addressing these issues in multi-modal corruption scenarios. This dissertation proposes a novel approach named Multi-modality Dynamic Analytic Adapter (MDAA), for MM-CTTA tasks. In MDAA, we introduce analytic learning into TTA for the first time by using the Analytic Classifiers (ACs) to prevent the forgetting issues. Additionally, we develop Dynamic Selection Mechanism (DSM) and Soft Pseudo-label Strategy (SPS), which enable MDAA to select and integrate reliable information from different modalities in a dynamical way, mitigating influence from error accumulation and reliability bias. Moreover, we proposed two new experiment settings to evaluate models' performance under extreme TTA cases. Extensive experiments demonstrate that MDAA can solve the challenges well in two settings, outperforms other state-of-the-art methods by 6.22% and 6.84% respectively.