BGaitR-Net: an effective neural model for occlusion reconstruction in gait sequences by exploiting the key pose information

Abstract

Gait recognition in the presence of occlusion is a challenging problem and the solutions proposed to date either lack robustness or depend on several unrealistic constraints. In this work, we propose a Deep Learning framework to detect and reconstruct the occluded frames in a gait sequence. Initially, occlusion detection is done using a VGG-16 network and for each frame the corresponding pose information is represented as a one-hot encoded vector. This vector is next fused with the corresponding spatial information using a Conditional Variational Autoencoder (CVAE) to obtain an effective embedding. Following this, a Bi-directional Long Short Term Memory (Bi-LSTM) is used to predict the occluded frames using the encoded vector sequence. A decoder next transforms these predicted frames back to the image space. Our proposed reconstruction model termed the Bidirectional Gait Reconstruction Network (BGaitR-Net) is formed by stacking the CVAE, Bi-LSTM, and the decoder. The CASIA-B and OU-ISIR LP datasets are used to prepare extensive gallery sets to train each of the above sub-networks and testing is done using synthetically occluded sequences from the CASIA-B data and real-occluded sequences from the TUM-IITKGP data. A thorough evaluation of our work through Dice Score and GEINet-based recognition accuracy for varying degrees of occlusion highlight the effectiveness of our model in generating frames consistent with the temporal gait pattern. Comparative study with other existing gait recognition techniques (with or without occlusion handling mechanism) and with recent Deep Learning-based video frame prediction methods emphasizes the superiority of BGaitR-Net over the others.