Please use this identifier to cite or link to this item: https://hdl.handle.net/10356/154620
Title: Evaluating human versus machine learning performance in classifying research abstracts
Authors: Goh, Yeow Chong
Cai, Xin Qing
Theseira, Walter
Ko, Giovanni
Khor, Khiam Aik
Keywords: Engineering::Mechanical engineering
Issue Date: 2020
Source: Goh, Y. C., Cai, X. Q., Theseira, W., Ko, G. & Khor, K. A. (2020). Evaluating human versus machine learning performance in classifying research abstracts. Scientometrics, 125(2), 1197-1212. https://dx.doi.org/10.1007/s11192-020-03614-2
Project: NRF2014-NRF-SRIE001-027 
Journal: Scientometrics 
Abstract: We study whether humans or machine learning (ML) classification models are better at classifying scientific research abstracts according to a fixed set of discipline groups. We recruit both undergraduate and postgraduate assistants for this task in separate stages, and compare their performance against the support vectors machine ML algorithm at classifying European Research Council Starting Grant project abstracts to their actual evaluation panels, which are organised by discipline groups. On average, ML is more accurate than human classifiers, across a variety of training and test datasets, and across evaluation panels. ML classifiers trained on different training sets are also more reliable than human classifiers, meaning that different ML classifiers are more consistent in assigning the same classifications to any given abstract, compared to different human classifiers. While the top five percentile of human classifiers can outperform ML in limited cases, selection and training of such classifiers is likely costly and difficult compared to training ML models. Our results suggest ML models are a cost effective and highly accurate method for addressing problems in comparative bibliometric analysis, such as harmonising the discipline classifications of research from different funding agencies or countries.
URI: https://hdl.handle.net/10356/154620
ISSN: 0138-9130
DOI: 10.1007/s11192-020-03614-2
Rights: © 2020 The Author(s). This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/..
Fulltext Permission: open
Fulltext Availability: With Fulltext
Appears in Collections:MAE Journal Articles
TRACS Posters and Papers

Files in This Item:
File Description SizeFormat 
s11192-020-03614-2.pdf945.34 kBAdobe PDFThumbnail
View/Open

SCOPUSTM   
Citations 20

11
Updated on Dec 2, 2022

Web of ScienceTM
Citations 20

10
Updated on Dec 6, 2022

Page view(s)

76
Updated on Dec 7, 2022

Download(s)

7
Updated on Dec 7, 2022

Google ScholarTM

Check

Altmetric


Plumx

Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.