Humanoid robot motion dataset with automated labelling and human-in-the-loop refinement

Abstract

Humanoid robot motion datasets are crucial for advancing pose estimation and motion analysis. This dissertation presents a pipeline for constructing such datasets by integrating deep neural networks and a Human-In-The-Loop (HITL) refine- ment process. Videos are collected, filtered through multiple automated steps (detection, tracking, and pose estimation), and further refined via HITL re- annotation. Experiments demonstrate significant improvements in keypoint pre- diction accuracy, with the fine-tuned Real-Time Multi-Person Pose Estimation (RTMPose) model achieving a Percentage of Correct Keypoints (PCK) of 0.983. Limitations include reliance on manual review for edge cases and challenges with keypoint occlusions. Future work will focus on automated quality evalua- tors and smoothing algorithms for temporal consistency.