Please use this identifier to cite or link to this item:
Full metadata record
DC FieldValueLanguage
dc.contributor.authorNg, Alvin Keng Hian
dc.description.abstractSocial media has been increasing adopted as a mode of communication throughout the world, causing the amount of data to increase at an alarming rate and raising concerns over the management and analysis on big data. This has resulted in data/business analytics to be increasing popular as it seeks to study user’s behaviour, including popular topics and popular users as well as malicious users with bad intentions. Twitter is adopted as a tool to provide users with the latest insights to various incidents or news in the shortest time across the globe, which has also triggered a huge interest in social journalism. It also contains a URL shortening tool, limiting the word count to a maximum of 140 characters for each tweet. This allows Twitter to track user behaviour and prevent users from being targeted by malicious attackers. Big data enables Twitter to generate additional revenue through providing data analytics services to various large organizations as they are interested in the types of information or trends that are popular amongst Twitter users. In this project, Python is used as the preferred language since various libraries such as NLTK are readily available, which allows the analysis of near duplicates and spam detection to be made possible within a short period of time. Several forms of testing have been conducted to identify any potential performance and memory leaks present in the codes. Overall, the objectives of this project have been successfully accomplished on time. However, due to the many types of algorithms that are made available for near duplicate detection during the point of writing, only some of the popular algorithms have been implemented, which specifically tailors to data streaming. Various spam detection tools are looked at, which enables us to identify the types of tweets that will constitute to being identified as a spam tweet. Near duplicate and spam detection are related in such a way that spam is able to detect bots, while near duplicates are able to determine the amount of similarity or differences between tweets, ensuring that no spam has been able to escape the spam detection process unscathed.en_US
dc.format.extent72 p.en_US
dc.rightsNanyang Technological University
dc.titleNear duplicate detection on tweetsen_US
dc.typeFinal Year Project (FYP)en_US
dc.contributor.supervisorYeo Chai Kiaten_US
dc.contributor.schoolSchool of Computer Engineeringen_US
dc.description.degreeBachelor of Engineering (Computer Science)en_US
item.fulltextWith Fulltext-
Appears in Collections:SCSE Student Reports (FYP/IA/PA/PI)
Files in This Item:
File Description SizeFormat 
Final Year Report - Alvin Ng Keng Hian.pdf
  Restricted Access
3.16 MBAdobe PDFView/Open

Page view(s)

Updated on Dec 8, 2021

Download(s) 50

Updated on Dec 8, 2021

Google ScholarTM


Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.