Developing an accessible learning application for sign language (C)

Abstract

Hearing loss is a health issue that is affecting over 1.5 billion people worldwide, and this number is projected to increase even more to 2.5 billion by the Year 2050. In the case of children who are born deaf or suffered from hearing loss during their early development stage, statistics has shown that more than 90 per cent of these deaf children are born from hearing parents. While these children would usually learn sign language to communicate at specialised institutions, their parents might not have the necessary resources to pick up sign language themselves. This predicament often left the hearing parents at loss on how to communicate meaningfully with their children. As such, they might grow reluctant to communicate beyond necessary, which leads to deaf people feeling disconnected and isolated from their hearing family members.

In this project, the author, together with two other team members, aims to address the above issue through the development of an interactive sign language learning web application targeted at the hearing family members of deaf people. The web app, named Sign2Sign, utilises an interactive 3D avatar and real-time sign recognition AI to support conversational-focused sign language lessons. These are implemented using several Microsoft Azure technologies such as Azure App Service, Azure Cosmos DB, and Azure Machine Learning, coupled with other core technologies including MediaPipe and ThreeJS. Through providing such accessible avenue to learn sign language effectively, this project aims to help bridge the communication gap between the deaf and their hearing family members. Furthermore, this project also aims to promote inclusivity for the Deaf community by encouraging more hearing people to learn sign language.

This report discusses the author's contribution in the development of the Sign2Sign web app, which includes curating the sign language lessons, building the lessons database and implementing the lessons APIs, generating signing landmarks for sign recognition, as well as developing the UI for the lessons' quiz components.

The completed prototype, Sign2Sign, was also submitted to the Microsoft Imagine Cup 2022 competition. It successfully achieved a World Finalist result and won in the Lifestyle category of the Asia region in the World Finals.