Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/171942
Title: A critical study on MovieLens dataset for recommender systems
Authors: Tan, Ernest Yan Heng
Keywords: Engineering::Computer science and engineering::Information systems::Information storage and retrieval
Issue Date: 2023
Publisher: Nanyang Technological University
Source: Tan, E. Y. H. (2023). A critical study on MovieLens dataset for recommender systems. Final Year Project (FYP), Nanyang Technological University, Singapore. https://hdl.handle.net/10356/171942
Abstract: The growth in recommendation systems (RecSys) research has led to the development of many toolkits, which provide users, who may have varying levels of knowledge in the field, with the necessary tools to build, test, evaluate and benchmark different algorithms. The MovieLens datasets have garnered widespread popularity as a benchmark dataset for RecSys research, but exploratory analysis has shown that the datasets elicit certain issues such as popularity bias and data sparsity as a result. Therefore, evaluation results of baseline algorithms trained on this dataset may pick up these inherent signals present in the data, and therefore should not be generalised across other recommendation scenarios. A comprehensive and consistent experiment involving 3 Python-based Top-N recommendation toolkits: LensKit, RecPack, and daisyRec have shown that toolkits are often built with different purposes or to solve specific issues, which leads to inconsistency in implementation methodology and hence evaluation results. This can be attributed to several main factors: (1) unclear or inconsistent definition of concepts such as evaluation metrics and (2) differences in default preprocessing and splitting strategies being the most significant. The experiments also highlight the disadvantages of using a global time-aware split on the MovieLens dataset, such as eliminating unseen users which are present in the test set but not in the train set. Additionally, analysis showed that having a low absolute number of train interactions, e.g., less than 15, is detrimental to the performance of a model than having a low train to test interaction ratio, with the evaluation metrics showing relatively poorer performance on 2 out of 3 of the toolkits discussed. Lastly, this study proposes some possible improvements to the toolkits based on the issues highlighted, such as clearly defined default dataset preprocessing, fully customisable hyperparameters, and frameworks which allow for quick development of algorithms and metrics, with a possible future work of producing an actively managed, open source toolkit which can solve the problems surfaced during this study.
URI: https://hdl.handle.net/10356/171942
Schools: School of Computer Science and Engineering 
Fulltext Permission: restricted
Fulltext Availability: With Fulltext
Appears in Collections:SCSE Student Reports (FYP/IA/PA/PI)

Files in This Item:
File Description SizeFormat 
Amended_SCSE22-0878_A Critical Study on MovieLens Dataset for Recommender System.pdf
  Restricted Access
Undergraduate project report1.14 MBAdobe PDFView/Open

Page view(s)

121
Updated on Jul 19, 2024

Download(s)

21
Updated on Jul 19, 2024

Google ScholarTM

Check

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.