Human action recognition using LLM

Abstract

Human Action Recognition (HAR) is a prevalent task in the domain of computer vision, with widespread applications in surveillance, autonomous, and human-computer interaction. Although remarkable advancements have been made through the conventional HAR approaches of RGB- and skeleton-based recognition, such methods often face issues of generalisability, interpretability, and multimodal data fusion. This work explores a novel framework using Large Language Models (LLMs) with Chain-of-Thought (CoT) distillation in the context of HAR. By transforming action data into rich linguistic representations, we extend the ability of LLMs to reason about human actions in a way compatible with their pre-training, enhancing semantic awareness and interpretability. A two-stage methodology is followed: first, LLMs generate rationale-rich prompts for HAR; then student models are fine-tuned on these rationales while strictly following computational constraints. This work also involves multimodal CoT prompting using vision-based inputs, leading to significant improvements in classification accuracy and temporal understanding. The results show that, while text-based HAR using distilled LLMs attains modest performance due to limitations on input length, vision-based CoT prompting significantly boosts recognition effectiveness and interpretability. This work emphasizes the potential and feasibility of using LLMs in HAR and, thereby, enhancing streamlined, multimodal, and reasoning-capable systems for action recognition.