Man-machine cooperative method based on deep learning in flexible manufacturing system

Abstract

Human-Robot Collaboration (HRC) is essential for enhancing productivity and flexibility in smart manufacturing, which poses requirements on accurately predicting the future movements of human operators, especially the trajectories of their upper limbs. However, existing model-based studies on human manipulation prediction lacks consideration of stochasticity and variability while the emerging deep learning-based methods are demanding on data size, which yet makes real-time deployment challenging. To explore worker arm motion prediction in human-robot collaboration scenarios, a new Human Arm Motion under Manufacturing environment (HUAMM) dataset was designed. This dataset is based on real static working scenes in an automated factory and includes video sequences of various assembly line operations with and without occlusions, totaling over 10 hours. The dataset contains both color and depth information and, after preprocessing, combining the advantages of both model-based and deep learning-based methods, a method for predicting human arm motion, specifically, the position of a worker's wrist in less than 0.5 second, is proposed by hybridizing a Long Short-Term Memory (LSTM) network with an Inverse Kinematics (IK) model. Using historical coordinate sequences of the wrist joint in three-dimensional space in the past multiple frames as input, a neural network is trained to output the predicted coordinates of the wrist joint for the next frame. Then IK (Inverse Kinematics) is used to calculate the arm's motion trajectory based on the predicted wrist coordinates. As the predicted wrist coordinates are sequentially used as the input for the next prediction cycle, the prediction is realized over a sliding time window. Evaluation was conducted using both proprietary and open datasets, results demonstrated that our LSTM-IK method achieved high prediction accuracy, with an average distance error of approximately 5 cm, and can adapt to various task scenarios and individual differences. Additionally, comparison with ground truth illustrated the model's ability to handle complex motion patterns, even with partial occlusions or rapid movements.