Please use this identifier to cite or link to this item:
https://hdl.handle.net/10356/48449
Title: | Parallel social network crawler system | Authors: | Lim, Ivan Wei Jie. | Keywords: | DRNTU::Engineering::Computer science and engineering::Computer systems organization::Computer system implementation DRNTU::Engineering::Computer science and engineering::Computer systems organization::Performance of systems |
Issue Date: | 2012 | Abstract: | Crawling social network data can uncover interesting phenomena for a variety of usage. However it is also generally sluggish due to the fact that it requires 3rd party services over a competitive network. These services are provided at their discretion and their usage quota needs to be complied. Therefore, it resulted in the need to accelerate the retrieval process, which is the objective of this project, to exploit parallelism so as to speed up the crawling procedure. The nature of social network data looks very much like a graph. Hence, the Breadth-First Search (BFS) graph traversal technique is revisited to explore for improvements on crawling operations. This project has chosen Google’s social networking platform called Google+ and experimented parallel crawling method based on BFS to increase throughput. The implementation of the experimental system has performed reasonably well over the naive crawling approach, in light of external limitations like Google’s courtesy usage quota of their services. The system was able fetch more data in the same or even shorter amount of time, therefore, increasing efficiency by a few folds. Although the project demonstrated the speed up of the crawling process, there are still rooms for improvement to further scale up the entire job. Using this as a basis, more concepts can still be used to enhance the efficiency of the system. | URI: | http://hdl.handle.net/10356/48449 | Schools: | School of Computer Engineering | Rights: | Nanyang Technological University | Fulltext Permission: | restricted | Fulltext Availability: | With Fulltext |
Appears in Collections: | SCSE Student Reports (FYP/IA/PA/PI) |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
SCE11-0336.pdf Restricted Access | Main article | 1.67 MB | Adobe PDF | View/Open |
Page view(s) 50
543
Updated on Feb 17, 2025
Download(s)
6
Updated on Feb 17, 2025
Google ScholarTM
Check
Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.