Parallel social network crawler system

Abstract

Crawling social network data can uncover interesting phenomena for a variety of usage. However it is also generally sluggish due to the fact that it requires 3rd party services over a competitive network. These services are provided at their discretion and their usage quota needs to be complied. Therefore, it resulted in the need to accelerate the retrieval process, which is the objective of this project, to exploit parallelism so as to speed up the crawling procedure.

The nature of social network data looks very much like a graph. Hence, the Breadth-First Search (BFS) graph traversal technique is revisited to explore for improvements on crawling operations. This project has chosen Google’s social networking platform called Google+ and experimented parallel crawling method based on BFS to increase throughput.

The implementation of the experimental system has performed reasonably well over the naive crawling approach, in light of external limitations like Google’s courtesy usage quota of their services. The system was able fetch more data in the same or even shorter amount of time, therefore, increasing efficiency by a few folds.

Although the project demonstrated the speed up of the crawling process, there are still rooms for improvement to further scale up the entire job. Using this as a basis, more concepts can still be used to enhance the efficiency of the system.