Software development for digital Chinese medicine tongue diagnosis

Abstract

Tongue diagnosis is an important inspection method in Traditional Chinese Medicine (TCM). However, it is subjective and can be unreliable as diagnostic results can differ depending on the personal knowledge and experience of the TCM practitioner. Various computerized tongue diagnosis systems were built to standardize and remove subjectivity in tongue diagnosis. However, the majority of such systems use novel image capturing devices that are not available and accessible to most individuals. Thus, this project seeks to develop a tongue diagnosis system that makes use of the smartphone’s camera to capture the user’s tongue image. The system will first segment the tongue from the input image and perform a content-based image retrieval (CBIR) to retrieve the most similar tongue image along with its diagnosis from an image database. In addition, users will be able to register an account, log in and view their diagnosis history. The project is developed using React Native as the frontend, Flask as the backend. The system segments the tongue using U-Net and morphological operations. Given the segmented tongue image, a feature vector is extracted using a feature extractor. Two methods of extracting features were explored. The methods include using an autoencoder’s encoder or a multi-output convolutional neural network. The project then uses an approximate nearest neighbor algorithm Annoy to retrieve the most similar tongue image in the database.