Traffic optimized, news crawling and classification mobile solution

Abstract

With the swift moving of information technologies, the extensive usage of mobile devices is consuming large amount of network traffic. Mobile devices are also transforming modern lifestyle. Online news reading has become one of the major Internet activities on mobile equipment. However, duplicative web content results in extra mobile traffic and incurs additional cost for both consumers and network service providers. To provide improved and holistic reading experience and also save network traffic, a mobile application was developed in this project to solve the above problem. The application is able to perform duplication detection, news categorization and image compression. The solution consists of a server module and a client module. The news datasets are crawled from Wall Street Journal (WSJ) and Bloomberg. The application successfully removed all the 216 pieces of duplicative news among 1113 pieces of testing data. News categorisation achieved an accuracy of around 85% using different machine learning algorithms. A combination of classifiers was also proposed which increased the accuracy to 87%. Image compression generally achieved an efficiency of 73% in saving spaces and network traffic.