Supervised news topic detection

Abstract

With the advancement of technology, there has been much improvement in the automatic recording of broadcast news by utilizing speech recognition. However the continually increasing dynamic information pool is posing challenges for efficient information retrieval techniques. This pain-point creates the need to develop systems that can automatically categorize this information under relevant topics for the purpose of easy information retrieval.

In recent years, much focus has been given to the subject of topic detection of broadcast news more through unsupervised techniques such as clustering as a few studies focusing on supervised classification techniques. In this project, we propose a simple yet effective approach for this purpose by drawing inspiration from previously conducted studies.

In this thesis, we experiment with a supervised machine learning algorithm namely Logistic Regression along with language processing techniques to automatically detect topics from broadcast news comprised in the TDT2 English corpus. We consider the input documents, as a stream of sentences and use the trained classifier to predict the topics they are associated with and accordingly assign these news documents to the most appropriate topic. This approach includes various pre-processing techniques along with feature selection and natural language processing. It can be inferred from the results obtained that the chosen model is able to detect relevant topics of new articles by adopting a simplistic topic detection approach that uses the Logistic Regression classifier while taking inspiration from conducted studies. The proposed model performs in comparison to some state-of-the-art topic classifiers.