Multi-view fusion for action recognition in car cabin environment

Abstract

Action recognition plays a critical role in autonomous driving systems for interpreting driver behavior. Conventional single view methods that depend on manually designed features and Convolutional Neural Networks (CNNs) frequently face challenges in handling occlusions, illumination variations, and viewpoint alterations, resulting in limited performance. This study addresses these limitations through the integration of data from multiple synchronized camera perspectives. This research adopts UniFormerV2 as the baseline architecture and investigates three distinct fusion methodologies: early fusion, late fusion, and a META (Motion and Multi-View Excitation and Temporal Aggregation) block. Early fusion merges multi-view data at the preprocessing phase, whereas late fusion aggregates processed features while preserving view-specific characteristics. The META-based framework employs motion-aware mechanisms and cross-view attention to dynamically model spatiotemporal dependencies, thereby improving inter-view information exchange and action comprehension. Benchmark evaluations conducted on the Drive&Act dataset demonstrate that multi-view models outperform single view models, particularly in complex scenarios involving occlusions or actions across different spatial regions. The META-based method achieves the highest accuracy.