Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/66410
Title: Near duplicate detection on tweets
Authors: Ng, Alvin Keng Hian
Keywords: DRNTU::Engineering
Issue Date: 2016
Abstract: Social media has been increasing adopted as a mode of communication throughout the world, causing the amount of data to increase at an alarming rate and raising concerns over the management and analysis on big data. This has resulted in data/business analytics to be increasing popular as it seeks to study user’s behaviour, including popular topics and popular users as well as malicious users with bad intentions. Twitter is adopted as a tool to provide users with the latest insights to various incidents or news in the shortest time across the globe, which has also triggered a huge interest in social journalism. It also contains a URL shortening tool, limiting the word count to a maximum of 140 characters for each tweet. This allows Twitter to track user behaviour and prevent users from being targeted by malicious attackers. Big data enables Twitter to generate additional revenue through providing data analytics services to various large organizations as they are interested in the types of information or trends that are popular amongst Twitter users. In this project, Python is used as the preferred language since various libraries such as NLTK are readily available, which allows the analysis of near duplicates and spam detection to be made possible within a short period of time. Several forms of testing have been conducted to identify any potential performance and memory leaks present in the codes. Overall, the objectives of this project have been successfully accomplished on time. However, due to the many types of algorithms that are made available for near duplicate detection during the point of writing, only some of the popular algorithms have been implemented, which specifically tailors to data streaming. Various spam detection tools are looked at, which enables us to identify the types of tweets that will constitute to being identified as a spam tweet. Near duplicate and spam detection are related in such a way that spam is able to detect bots, while near duplicates are able to determine the amount of similarity or differences between tweets, ensuring that no spam has been able to escape the spam detection process unscathed.
URI: http://hdl.handle.net/10356/66410
Rights: Nanyang Technological University
Fulltext Permission: restricted
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Student Reports (FYP/IA/PA/PI)

Files in This Item:
File Description SizeFormat 
Final Year Report - Alvin Ng Keng Hian.pdf
  Restricted Access
3.16 MBAdobe PDFView/Open

Page view(s)

197
Updated on Oct 21, 2021

Download(s) 50

15
Updated on Oct 21, 2021

Google ScholarTM

Check

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.