Emotional facial expression transfer using motion capture data

Abstract

An input-output temporal restricted Boltzmann machine is an artificial neural network that learns the probability distribution between input sequences and output sequences, and then uses the model to predict output sequences. Unlike other static models, IOTRBM can catch the details of non-linear facial movements and eliminate irrelevant temporal noise, resulting in realistic predicted sequences. This project covers pre-processing raw motion capture data of neutral face expression and happy face expression and pass them as training data and testing data to the IOTRBM. The pre-processing tasks include recovering missing data points, removing silent parts in the motion capture data, and exercising canonical time warping (CTW) to temporally align the data frames based on visemes or phonemes. After passing the data sequences to IOTRBM and training the model, natural-looking happy expression sequences have been predicted based on the neutral expression sequences.