Please use this identifier to cite or link to this item:
Title: Discriminator-enhanced knowledge-distillation networks
Authors: Li, Zhenping
Cao, Zhen
Li, Pengfei
Zhong, Yong
Li, Shaobo
Keywords: Engineering::Electrical and electronic engineering
Issue Date: 2023
Source: Li, Z., Cao, Z., Li, P., Zhong, Y. & Li, S. (2023). Discriminator-enhanced knowledge-distillation networks. Applied Sciences, 13(14), 8041-.
Journal: Applied Sciences 
Abstract: Query auto-completion (QAC) serves as a critical functionality in contemporary textual search systems by generating real-time query completion suggestions based on a user’s input prefix. Despite the prevalent use of language models (LMs) in QAC candidate generation, LM-based approaches frequently suffer from overcorrection issues during pair-wise loss training and efficiency deficiencies. To address these challenges, this paper presents a novel framework—discriminator-enhanced knowledge distillation (Dis-KD)—for the QAC task. This framework combines three core components: a large-scale pre-trained teacher model, a lightweight student model, and a discriminator for adversarial learning. Specifically, the discriminator aids in discerning generative-level differences between the teacher and the student models. An additional discriminator score loss is amalgamated with the traditional knowledge-distillation loss, resulting in enhanced performance of the student model. Contrary to the stepwise evaluation of each generated word, our approach assesses the entire generation sequence. This method alleviates the prevalent overcorrection issue in the generation process. Consequently, our proposed framework boasts improvements in model accuracy and a reduction in parameter size. Empirical results highlight the superiority of Dis-KD over established baseline methods, with the student model surpassing the teacher model in QAC tasks for sub-word languages.
ISSN: 2076-3417
DOI: 10.3390/app13148041
Schools: School of Electrical and Electronic Engineering 
Rights: © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:EEE Journal Articles

Files in This Item:
File Description SizeFormat 
applsci-13-08041.pdf848.9 kBAdobe PDFThumbnail

Page view(s)

Updated on Jul 19, 2024


Updated on Jul 19, 2024

Google ScholarTM




Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.