Hand gesture-based interactive musical and visual experiential system

Abstract

There have been many systems available in the market in the past and present catered to virtual-reality and augmented reality and they have seen an increase in popularity in recent years especially in their usage for gaming and entertainment. Current and past technologies such as the Microsoft Kinect, Switch, Wii and Leap Motion have changed how we interact with the virtual environment. The Leap Motion device in particular is a sensor device that supports extremely accurate hand and finger motions as input and has been used in the areas of science and the medical field. This Final Year project aims to create an immersive interactable audio and visual environment using the Leap Motion device and Unity game engine where users are able to control and manipulate certain audio parameters whilst getting visual feedback on the screen. This report also gives an overview of the project development cycle including the project and work breakdown structure, implementation and development, issues faced and solutions implemented, as well user studies results and potential ideas for further exploration.