Advancements in zero-shot learning for skeleton-based action recognition

Abstract

This paper presents an advanced approach to Zero-Shot Learning (ZSL) for skeleton-based action recognition, aiming to improve the model's ability to recognize unseen actions without extensive labeled training data. Skeleton-based action recognition has gained significant attention due to its robustness to environmental variations and compact data representation. However, traditional methods often struggle with generalization to new, unseen actions. To address this challenge, we propose a novel framework that integrates semantic embeddings with skeleton features through a disentangled latent space. Our method leverages a fine-grained formulation strategy to partition skeleton features into body parts and generates detailed text descriptions for each part using a Large Language Model (LLM). These descriptions are then aligned with skeleton features through a cross-modal alignment module, which employs Variational Autoencoders (VAEs) and adversarial training to disentangle semantic-related and semantic-irrelevant components. We further design a multi-stream classifier to effectively utilize both global and part-level features for action recognition. Extensive experiments on the NTU RGB+D and NTU RGB+D 120 datasets shows that our approach surpasses state-of-the-art methods in both 2-stream and multi-stream ZSL classification settings. Our results highlight the importance of fine-grained feature partition and semantic alignment in improving the generalization capability of skeleton-based action recognition models.