Please use this identifier to cite or link to this item:
Title: Dialogue systems with audio context
Authors: Young, Tom
Pandelea, Vlad
Poria, Soujanya
Cambria, Erik
Keywords: Engineering::Computer science and engineering
Issue Date: 2020
Source: Young, T., Pandelea, V., Poria, S. & Cambria, E. (2020). Dialogue systems with audio context. Neurocomputing, 388, 102-109.
Project: A18A2b0046
Journal: Neurocomputing
Abstract: Research on building dialogue systems that converse with humans naturally has recently attracted a lot of attention. Most work on this area assumes text-based conversation, where the user message is modeled as a sequence of words in a vocabulary. Real-world human conversation, in contrast, involves other modalities, such as voice, facial expression and body language, which can influence the conversation significantly in certain scenarios. In this work, we explore the impact of incorporating the audio features of the user message into generative dialogue systems. Specifically, we first design an auxiliary response retrieval task for audio representation learning. Then, we use word-level modality fusion to incorporate the audio features as additional context to our main generative model. Experiments show that our audio-augmented model outperforms the audio-free counterpart on perplexity, response diversity and human evaluation.
ISSN: 0925-2312
DOI: 10.1016/j.neucom.2019.12.126
Schools: School of Computer Science and Engineering 
Rights: © 2020 Elsevier B.V. All rights reserved.
Fulltext Permission: none
Fulltext Availability: No Fulltext
Appears in Collections:SCSE Journal Articles

Citations 10

Updated on Sep 22, 2023

Web of ScienceTM
Citations 10

Updated on Sep 24, 2023

Page view(s)

Updated on Sep 26, 2023

Google ScholarTM




Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.