Developing web crawler and categorization of newspaper text

Abstract

The automated categorization (or classification) of texts into predefined categories has witnessed a booming interest in the last ten years, due to the increased availability of documents in digital form on World Wide Web like online newspaper, magazines, catalogues, blogs, video transcripts, etc. Existing supervised machine-learning based text classification models available in this field faces the challenge of needing large corpus/dataset of labelled data to train the language models. An innovative approach to this problem is to utilize the already classified/categorised news articles that are easily available on the internet. For the scope of this project an English modular text crawler that can be extended to multiple languages and is capable of automatically crawling online newspaper archives, extracting new keywords, and categories is developed. The corpus is further smoothened and transformed into human-speakable forms by using appropriate language-specific normalisation techniques. The crawler has mined over 1.16GB of data ranging from 2006-2012. This normalised corpus is used to build bi-gram probability based statistical language models for each category. These single-label paradigm classifiers are then combined together to form a text classification model. A document can be assigned to multiple categories with certain degree of ranking, but in this project primary focus is on assigning the most probable category to each news article based on the lowest perplexity value (highest similarity). The classification model, built is more robust than most of its counterparts currently available. The system shows a high average accuracy rate of 99.37%, and an average precision of 98.75%, when perplexity tests were conducted with randomly chosen articles