Real-time facial emotion recognition with LSTM-CNN

Abstract

In the digital age of communication, video as a means of communication becomes increasingly common. In video interviews or video-based user research, the ability to recognize emotions presents valuable insights to the subject’s emotional state. While deep learning methods have been shown to perform well in the area of Facial Emotion Recognition (FER), most of these conventional methods are limited to still images and do not use temporal features across consecutive video frames. In this project, a real-time facial emotional recognition system is developed using a hybrid deep learning network. This approach uses a Convolutional Neural Network (CNN) for spatial feature extraction and a Long Short-Term Memory (LSTM) network for temporal features of consecutive frames. The subject’s emotions are predicted and displayed in real-time through a graphical display.