Please use this identifier to cite or link to this item:
|Title:||Forum date understanding and mining||Authors:||Sim, Edwin Wong Loong||Keywords:||DRNTU::Engineering::Computer science and engineering::Computer systems organization::Computer system implementation||Issue Date:||2014||Abstract:||With the explosive growth of data online from terabytes to petabytes, a large amount of data is being collected and warehoused. People are drowning in data and starving for knowledge. Traditional techniques such as statistics and database systems are not suitable to extract sufficient knowledge as there is an enormity of data, and high dimensionality of data. Hence, data mining technique was created to perform non- trivial extraction of implicit, previously unknown and potentially useful knowledge from large amounts of data. As computers get cheaper and more powerful, the analysis of huge amounts of data is made possible. In this project, the student was given a sample dataset, which was retrieved from a popular local forum, www.hardwarezone.com. This data was retrieved in Extensible Markup Language (XML) format and the student has to process the data to determine the communication patterns among users in forums. By using text mining, which is a process of text analytics that involves information retrieval, study of word frequency distributions and pattern recognition, the student was able to derive high- quality information from the text. The data was broken down into terms and documents where the frequency of each term is calculated against the document. Using techniques such as case folding, stop word removal, lemmatization and stemming, the student was able to clean the data and make it more efficient for analysis. A weighted formula factor term frequency–inverse document frequency (Tf-idf) was used to normalise the frequency of the term. A single pass clustering method using cosine similarity is used on the data to find out the relation of each term. From the clustered formed, the student determined a certain event occurrence on a particular day. From this project, the student has gained a higher level of knowledge of how data is processed and analyzed. It also greatly increased the interest of the student in data mining and data processing. The techniques used and learned from this project will be very helpful in the future for data analysis.||URI:||http://hdl.handle.net/10356/59103||Rights:||Nanyang Technological University||Fulltext Permission:||restricted||Fulltext Availability:||With Fulltext|
|Appears in Collections:||SCSE Student Reports (FYP/IA/PA/PI)|
Page view(s) 5146
checked on Oct 20, 2020
checked on Oct 20, 2020
Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.