A multisensory interaction framework for human-cyber–physical system based on graph convolutional networks

Abstract

Human-Cyber-Physical Systems (HCPS), as an emerging paradigm centered around humans, provide a promising direction for the advancement of various domains, such as intelligent manufacturing and aerospace. In contrast to Cyber-Physical Systems (CPS), the development of HCPS emphasizes the expansion of human capabilities. Humans no longer solely function as operators or agents working in collaboration with computers and machines but extend their roles to include system design and innovation management. This paper proposes a Multisensory Interaction Framework for HCPS (MS-HCPS) that leverages human senses to facilitate system creation and management. Additionally, the introduced Multisensory Graph Convolutional Network (MS-GCN) model calculates recommendation values for multiple senses, elucidating their relevance to system development. Furthermore, the effectiveness of the proposed framework and model is validated through three practical engineering scenarios. This study explores the research on multisensory interaction in HCPS from a human sensory perspective, aiming to facilitate the progress and development of HCPS across various domains.