Please use this identifier to cite or link to this item:
Title: Identifying psychiatric manifestations in schizophrenia and depression from audio-visual behavioural indicators through a machine-learning approach
Authors: Xu, Shihao
Yang, Zixu
Chakraborty, Debsubhra
Chua, Victoria Yi Han
Tolomeo, Serenella
Winkler, Stefan
Birnbaum, Michel
Tan, Bhing-Leet
Lee, Jimmy
Dauwels, Justin
Keywords: Science::Medicine
Engineering::Electrical and electronic engineering
Issue Date: 2022
Source: Xu, S., Yang, Z., Chakraborty, D., Chua, V. Y. H., Tolomeo, S., Winkler, S., Birnbaum, M., Tan, B., Lee, J. & Dauwels, J. (2022). Identifying psychiatric manifestations in schizophrenia and depression from audio-visual behavioural indicators through a machine-learning approach. Schizophrenia, 8(1), 92-.
Project: NMRC/CG/ 004/2013
Journal: Schizophrenia
Abstract: Schizophrenia (SCZ) and depression (MDD) are two chronic mental disorders that seriously affect the quality of life of millions of people worldwide. We aim to develop machine-learning methods with objective linguistic, speech, facial, and motor behavioral cues to reliably predict the severity of psychopathology or cognitive function, and distinguish diagnosis groups. We collected and analyzed the speech, facial expressions, and body movement recordings of 228 participants (103 SCZ, 50 MDD, and 75 healthy controls) from two separate studies. We created an ensemble machine-learning pipeline and achieved a balanced accuracy of 75.3% for classifying the total score of negative symptoms, 75.6% for the composite score of cognitive deficits, and 73.6% for the total score of general psychiatric symptoms in the mixed sample containing all three diagnostic groups. The proposed system is also able to differentiate between MDD and SCZ with a balanced accuracy of 84.7% and differentiate patients with SCZ or MDD from healthy controls with a balanced accuracy of 82.3%. These results suggest that machine-learning models leveraging audio-visual characteristics can help diagnose, assess, and monitor patients with schizophrenia and depression.
ISSN: 2754-6993
DOI: 10.1038/s41537-022-00287-z
Schools: School of Electrical and Electronic Engineering 
School of Social Sciences 
Lee Kong Chian School of Medicine (LKCMedicine) 
Organisations: Institute of Mental Health, Singapore
Rights: © The Author(s) 2022. Open Access. This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:EEE Journal Articles
LKCMedicine Journal Articles
SSS Journal Articles

Files in This Item:
File Description SizeFormat 
s41537-022-00287-z.pdf888.82 kBAdobe PDFThumbnail

Citations 50

Updated on Feb 22, 2024

Web of ScienceTM
Citations 50

Updated on Oct 28, 2023

Page view(s)

Updated on Feb 28, 2024


Updated on Feb 28, 2024

Google ScholarTM




Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.