Please use this identifier to cite or link to this item:
Title: Parallel social network crawler system
Authors: Lim, Ivan Wei Jie.
Keywords: DRNTU::Engineering::Computer science and engineering::Computer systems organization::Computer system implementation
DRNTU::Engineering::Computer science and engineering::Computer systems organization::Performance of systems
Issue Date: 2012
Abstract: Crawling social network data can uncover interesting phenomena for a variety of usage. However it is also generally sluggish due to the fact that it requires 3rd party services over a competitive network. These services are provided at their discretion and their usage quota needs to be complied. Therefore, it resulted in the need to accelerate the retrieval process, which is the objective of this project, to exploit parallelism so as to speed up the crawling procedure. The nature of social network data looks very much like a graph. Hence, the Breadth-First Search (BFS) graph traversal technique is revisited to explore for improvements on crawling operations. This project has chosen Google’s social networking platform called Google+ and experimented parallel crawling method based on BFS to increase throughput. The implementation of the experimental system has performed reasonably well over the naive crawling approach, in light of external limitations like Google’s courtesy usage quota of their services. The system was able fetch more data in the same or even shorter amount of time, therefore, increasing efficiency by a few folds. Although the project demonstrated the speed up of the crawling process, there are still rooms for improvement to further scale up the entire job. Using this as a basis, more concepts can still be used to enhance the efficiency of the system.
Rights: Nanyang Technological University
Fulltext Permission: restricted
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Student Reports (FYP/IA/PA/PI)

Files in This Item:
File Description SizeFormat 
  Restricted Access
Main article1.67 MBAdobe PDFView/Open

Google ScholarTM


Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.