Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/149102
Title: Attentive embedding for document representation
Authors: Tang, Kok Foon
Keywords: Engineering::Electrical and electronic engineering
Issue Date: 2021
Publisher: Nanyang Technological University
Source: Tang, K. F. (2021). Attentive embedding for document representation. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/149102
Project: A3046-201
Abstract: With NLP reaching new and greater heights in many real-world applications, researchers are still trying to find better ways for a model to learn document representation. Moreover, most state-of-the-art NLP models have an encoder-decoder like architecture, which looks like an autoencoder architecture. Furthermore, KATE is an autoencoder that introduces a competition layer between the encoder and decoder. Hence, this project aims to use the KATE on a more complicated model to determine if KATE's usage provides a more attentive representation of documents. To investigate if KATE could improve document representation, training was implemented with 2 phases. The first phase trains the encoder-decoder models on a sentence reconstruction task, which enabled the model to learn the document representation. And the second phase, a classification task, can validate if the encoder and KATE from the first phase learned a good document representation via a classification task. The two models used for this test are 2-Layer LSTM and ALBERT. Both models were implemented and trained with and without KATE for comparison. The experiment results show that KATE helps in document representation for the 2-Layer LSTM but not for the ALBERT. Therefore, it concluded that KATE has the potential to help document representation for a simpler model like LSTM with minimal implementation cost, but not for a more complicated model like ALBERT.
URI: https://hdl.handle.net/10356/149102
Fulltext Permission: restricted
Fulltext Availability: With Fulltext
Appears in Collections:EEE Student Reports (FYP/IA/PA/PI)

Files in This Item:
File Description SizeFormat 
Attentive Embedding for Document Representation.pdf
  Restricted Access
2.32 MBAdobe PDFView/Open

Page view(s)

117
Updated on May 23, 2022

Download(s)

4
Updated on May 23, 2022

Google ScholarTM

Check

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.