Development of an API for EDU segmentation

Abstract

EDU stands for elementary discourse unit, which is a clause-like structure in a sentence. EDU segmentation, refers to determining the boundaries to split sentences into multiple EDUs. This project aims to experiment and develop EDU segmentation models. The experiments are conducted using the Rhetorical Structure Theory (RST) dataset and the model performance is evaluated using the F1-score based on the token level EDU boundaries.

The current existing research model, Segbot, has a Seq2seq model architecture using a bi-GRU encoder and GRU decoder with a pointer network to select the boundaries for EDU segmentation. To improve Segbot, we proposed replacing the bi-GRU encoder in Segbot with the generative pretrained BART encoder. This model performed at 94.5% F1-score.

Token classification for EDU segmentation based on the boundaries is also explored. This is done by finetuning pretrained models such as BERT as well as using the PosTag embeddings as additional input features.

Segbot with BART encoder yielded the highest performance and hence, the model weights would be used to develop an API Python Library in the future. This library would improve ease of usage for EDU segmentation on downstream NLP tasks, such as sentiment analysis and question answering.