Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/171765
Title: Discriminator-enhanced knowledge-distillation networks
Authors: Li, Zhenping
Cao, Zhen
Li, Pengfei
Zhong, Yong
Li, Shaobo
Keywords: Engineering::Electrical and electronic engineering
Issue Date: 2023
Source: Li, Z., Cao, Z., Li, P., Zhong, Y. & Li, S. (2023). Discriminator-enhanced knowledge-distillation networks. Applied Sciences, 13(14), 8041-. https://dx.doi.org/10.3390/app13148041
Journal: Applied Sciences 
Abstract: Query auto-completion (QAC) serves as a critical functionality in contemporary textual search systems by generating real-time query completion suggestions based on a user’s input prefix. Despite the prevalent use of language models (LMs) in QAC candidate generation, LM-based approaches frequently suffer from overcorrection issues during pair-wise loss training and efficiency deficiencies. To address these challenges, this paper presents a novel framework—discriminator-enhanced knowledge distillation (Dis-KD)—for the QAC task. This framework combines three core components: a large-scale pre-trained teacher model, a lightweight student model, and a discriminator for adversarial learning. Specifically, the discriminator aids in discerning generative-level differences between the teacher and the student models. An additional discriminator score loss is amalgamated with the traditional knowledge-distillation loss, resulting in enhanced performance of the student model. Contrary to the stepwise evaluation of each generated word, our approach assesses the entire generation sequence. This method alleviates the prevalent overcorrection issue in the generation process. Consequently, our proposed framework boasts improvements in model accuracy and a reduction in parameter size. Empirical results highlight the superiority of Dis-KD over established baseline methods, with the student model surpassing the teacher model in QAC tasks for sub-word languages.
URI: https://hdl.handle.net/10356/171765
ISSN: 2076-3417
DOI: 10.3390/app13148041
Schools: School of Electrical and Electronic Engineering 
Rights: © 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:EEE Journal Articles

Files in This Item:
File Description SizeFormat 
applsci-13-08041.pdf848.9 kBAdobe PDFThumbnail
View/Open

Page view(s)

68
Updated on Jul 19, 2024

Download(s)

18
Updated on Jul 19, 2024

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.