Data-driven based pedestrian emotion detection in human-drone interaction

Abstract

The increasing use of drones in pedestrian-dense environments presents challenges for safe and socially acceptable human-drone interaction. Traditional obstacle avoidance methods focus on geometric path planning and reactive collision avoidance, overlooking pedestrians’ emotional states. This study introduces FacialGait-Emotion (FG-Emo), a data-driven framework that enhances drone obstacle avoidance by recognizing pedestrian emotions in real time. FG-Emo integrates emotion classification through facial expression and gait emotion classification using a multimodal deep learning approach. A convolutional neural network classifies facial expressions, while a view-guided skeleton graph convolutional network (VS-GCNN) detects gait emotions. An angle-weighted fusion strategy balances facial and gait features, which are then incorporated into an artificial potential field (APF) model for emotion-aware obstacle avoidance. Evaluations using the AirSim simulation platform and real-world pedestrian video data show that FG-Emo improves classification accuracy compared to single-modality approaches. Moreover, its emotional obstacle avoidance strategy minimizes intrusions into pedestrians’ private space, enhancing human-drone interaction. This study advances drone obstacle avoidance in social settings. Future research will explore multimodal physiological signals, real-world deployment in dynamic environments, and ethical considerations related to AI-driven emotion inference.