Virtual assembly using hand tracking with Microsoft Kinect

Abstract

Virtual reality allows users to interact with objects in a virtual environment in a way like real life. Camera devices like the Microsoft Kinect are required, where the body of the users are tracked using its equipped camera sensors. Additionally, the user’s body orientation and gestures like closing and opening of the hands, can be tracked. This allows realistic interactions such as grabbing and releasing an object or changing an object’s orientation to be mapped to the user’s gestures and body orientation. In an ideal situation, tasks performed in virtual reality should take as long as in real life. However, limitations such as the lack of physical feedback, and tracking problems such as occlusion can affect the accuracy of interactions and can make tasks take longer to complete in Virtual Reality. In this project, using the Unity Engine, the interactions methods were developed to circumvent these limitations and scripted with C#, utilizing the Microsoft Kinect. A user study is carried out to determine how effective and user-friendly these interactions methods are. Users will be tasked to complete tasks created in Unity and in real-life and the completion times are recorded. These interaction methods are evaluated by the difference in completion times between the Unity and real-life variants and feedback gathered from the users’ experience.