Multimodel deception detection - are you telling a lie?

Abstract

Deception detection plays a crucial role across various fields, evolving from traditional physical polygraphs to today’s machine learning techniques to analyze deceptive behaviors. Fraud can be detected through multiple modalities, including heart rate, EEG, blood pressure, facial micro-expressions, and voice changes. This project introduces a multimodal deception detection system that utilizes two primary modalities: facial micro-expressions and voice. It integrates 2D and 3D ResNet models, trained on spectral data and video frames. Un- like most similar projects that primarily utilize Western face databases for train- ing, this project specifically focuses on deception detection among Asian populations, employing the ROSE Lab Vision2 dataset. This dataset encompasses three domains: China, India, and Malaysia. To enhance the baseline accuracy, the project employs a pre-training of multimodel using contrastive learning. Contrastive learning is employed to ascertain the correspondence between video and audio by training on the Asian Speaker dataset. This method enhances the model’s ability to discern the behavioral characteristics of Asians, and the trained weights are subsequently loaded into the fraud detection task to improve the prediction performance of the system.