TY - JOUR
T1 - Framework and Literature Analysis for Crowdsourcing's Answer Aggregation
AU - Moayedikia, Alireza
AU - Yeoh, William
AU - Ong, Kok-Leong
AU - Boo, Yee-Ling
N1 - Publisher Copyright:
© 2017, © 2017 International Association for Computer Information Systems.
PY - 2020/1/2
Y1 - 2020/1/2
N2 - This paper presents a classification framework and a systematic analysis of literature on answer aggregation techniques for the most popular and important type of crowdsourcing, i.e., micro-task crowdsourcing. In doing so, we analyzed research articles since 2006 and developed four classification taxonomies. First, we provided a classification framework based on the algorithmic characteristics of answer aggregation techniques. Second, we outlined the statistical and probabilistic foundations used by different types of algorithms and micro-tasks. Third, we provided a matrix catalog of the data characteristics for which an answer aggregation algorithm is designed. Fourth, a matrix catalog of the commonly used evaluation metrics for each type of micro-task was presented. This paper represents the first systematic literature analysis and classification of the answer aggregation techniques for micro-task crowdsourcing.
AB - This paper presents a classification framework and a systematic analysis of literature on answer aggregation techniques for the most popular and important type of crowdsourcing, i.e., micro-task crowdsourcing. In doing so, we analyzed research articles since 2006 and developed four classification taxonomies. First, we provided a classification framework based on the algorithmic characteristics of answer aggregation techniques. Second, we outlined the statistical and probabilistic foundations used by different types of algorithms and micro-tasks. Third, we provided a matrix catalog of the data characteristics for which an answer aggregation algorithm is designed. Fourth, a matrix catalog of the commonly used evaluation metrics for each type of micro-task was presented. This paper represents the first systematic literature analysis and classification of the answer aggregation techniques for micro-task crowdsourcing.
KW - Crowdsourcing
KW - Answer aggregation
KW - Classification framework
KW - Systematic literature analysis
UR - http://www.scopus.com/inward/record.url?scp=85041604694&partnerID=8YFLogxK
U2 - 10.1080/08874417.2017.1394171
DO - 10.1080/08874417.2017.1394171
M3 - Article
SN - 0887-4417
VL - 60
SP - 49
EP - 60
JO - Journal of Computer Information Systems
JF - Journal of Computer Information Systems
IS - 1
ER -