NeuroEmoFlow: EEG-driven continuous emotion prediction during video viewing

Abstract

NeuroEmoFlow is a multimodal emotion prediction system for continuously monitoring and predicting emotional responses during video watching. The research addresses main limitations of conventional emotion measurement methods, in which self-report questionnaires neglect unconscious emotional processes and typical EEG analyses do not usually involve complementary data modalities. Our study introduces an integrated approach by synchronising EEG with vocal and textual data of individuals speaking aloud their thoughts while watching video stimuli. The multimodal framework significantly enhances the emotion recognition validity and consistency by integrating neurophysiological activity with speech patterns and linguistic content and, in the process, transcends individual reporting biases and single modality analysis limitations. We conducted rigorous experiments with validated affective stimuli from standard databases like the Library for Affective Videos (OpenLAV) and Stanford Affective Film Library. Video clips were selected meticulously based on content validity, emotional intensity, and cultural appropriateness. EEG data underwent rigorous preprocessing, including artifact removal through Independent Component Analysis (ICA) and Automatic Artifact Removal (AAR) methods. We employed state-of-the-art deep learning architectures like Convolutional Neural Networks (CNNs) such as EEGNet and TSception for emotion classification. In result analysis, our user questionnaires confirmed that our selected visual stimuli successfully evoked intended emotional responses in static images and dynamic video stimuli. Comparison of performance between TSception and EEGNet deep learning models showed that TSception's has achieved 55.10% average accuracy compared to EEGNet's 48.98% in three-class emotion classification(Pos,Neg and Neutral). TSception showed stable consistency in most of the test blocks, reflecting effective learning of generalizable temporal-spatial features in emotional EEG patterns. Our Leave-One Block-Out cross-validation strategy confirmed TSception's generalisation performance in different testing sessions. While acknowledging difficulties like individual variability in emotional expression and potential signal contamination, future research will entail the expansion of the EEG dataset, the inclusion of currently unused multimodal data (linguistic features and audio transcriptions), the use of advanced fusion architectures, the exploration of dimensional emotion classification, and the development of computationally efficient implementations for real-time applications.