An augmented virtual reality approach to 3D video conferencing

Abstract

Advancement in media of communications has been studied thoroughly by researchers and its applications are used attentively by companies in the software industry. Skype, for example, was widely known and used by people as the frontier of voice and video call through internet, which helped connecting people and building relationships irrespective of distance. Not only that, many other applications including Google Hangouts, Discord, and Zoom had shown their attractions in this line of business and provided their own service along with their own unique features.

In this project, it aims on creating a new experience in video call which is known to be absent in those mentioned software applications. Particularly, this project is to develop a medium for teleconferencing using Unity3D — a real-time development platform, usually used for developing game. This project becomes the foundation of the final distribution-ready software, which includes the integration of Mixed Reality (VR and AR), multipresence, and other features that are not in this project scope.

There are three main requirements in this project: (i) Real-time audio transmission with space localisation. (ii) Interaction with surroundings and other players in an open 3D space environment. (iii) Narrowcasting, which means a player can opt for speaking to a specific player or group. Likewise, the player can subscribe to only a certain person or group, filtering out only they want to hear. Additionally, a player is also allowed to mute themselves. This project extensively uses Photon Unity Networking as a framework. Through this framework, a highly integrated cloud server — Photon Cloud, is used as the central application server, handling transmissions of all data and publishing events occurred in the environment.