Please use this identifier to cite or link to this item:
Title: Active learning with confidence-based answers for crowdsourcing labeling tasks
Authors: Song, Jinhua
Wang, Hao
Gao, Yang
An, Bo
Keywords: Engineering::Computer science and engineering
Issue Date: 2018
Source: Song, J., Wang, H., Gao, Y., & An, B. (2018). Active learning with confidence-based answers for crowdsourcing labeling tasks. Knowledge-Based Systems, 159, 244-258. doi:10.1016/j.knosys.2018.07.010
Journal: Knowledge-Based Systems
Abstract: Collecting labels for data is important for many practical applications (e.g., data mining). However, this process can be expensive and time-consuming since it needs extensive efforts of domain experts. To decrease the cost, many recent works combine crowdsourcing, which outsources labeling tasks (usually in the form of questions) to a large group of non-expert workers, and active learning, which actively selects the best instances to be labeled, to acquire labeled datasets. However, for difficult tasks where workers are uncertain about their answers, asking for discrete labels might lead to poor performance due to the low-quality labels. In this paper, we design questions to get continuous worker responses which are more informative and contain workers’ labels as well as their confidence. As crowd workers may make mistakes, multiple workers are hired to answer each question. Then, we propose a new aggregation method to integrate the responses. By considering workers’ confidence information, the accuracy of integrated labels is improved. Furthermore, based on the new answers, we propose a novel active learning framework to iteratively select instances for “labeling”. We define a score function for instance selection by combining the uncertainty derived from the classifier model and the uncertainty derived from the answer sets. The uncertainty derived from uncertain answers is more effective than that derived from labels. We also propose batch methods which select multiple instances at a time to further improve the efficiency of our approach. Experimental studies on both simulated and real data show that our methods are effective in increasing the labeling accuracy and achieve significantly better performance than existing methods.
ISSN: 0950-7051
DOI: 10.1016/j.knosys.2018.07.010
Rights: © 2018 Elsevier B.V. All rights reserved.
Fulltext Permission: none
Fulltext Availability: No Fulltext
Appears in Collections:SCSE Journal Articles

Google ScholarTM




Items in DR-NTU are protected by copyright, with all rights reserved, unless otherwise indicated.