Online text mining

Abstract

Training language model made from conversational speech is difficult due to large variation of the way conversational speech is made. Deriving the conversational speech through direct transcription is costly and impractical for large corpus. A solution to this is to utilize the text that is easily available on the Internet to improve an existing language model made from broadcast news. In this project, the author developed an automated application capable of mining for text from online source, transforming the data into human speak-able forms through normalization techniques before using them to generate language model for adaption to improve the existing language model. The system developed had mined 16 years of data mounting up to 1.69 GB of new articles text. Through smoothing technique and interpolation weight analysis, the author had improved the perplexity of existing system made from broadcast news text significantly by 48.5%.

The previous finding had shown that a larger corpus could improve the perplexity of a language model however there is a constant need to find out better ways to make use of data instead of massively crawling data across the Internet.Thus the second objective of this project is to investigate the effectiveness of improving language model based on latest data. To find out whether constantly crawling for new data is good for improving a language model or the change in perplexity is so small that it is negligible. In this report, the author had conducted 8 experiments using 10 years of past data to gather a baseline perplexity. Each of the experiments had new data added to the base model before a perplexity test is again carried out. The findings showed that despite the new data being only an average of 0.18% of the baseline’straining data. It had an average of 1.9% perplexity improvement.improvement. Therefore the author is able to conclude that new data is of high importance and should always be crawled to improve a language model if its usage patterns changes according to new data. The final experiment of this project had created language models using a variation of vocabulary sizes. Tests on them revealed that an increased in vocabulary size actually increases the perplexity.