Framework and Literature Analysis for Crowdsourcing’s Answer Aggregation

ABSTRACT

This paper presents a classification framework and a systematic analysis of literature on answer aggregation techniques for the most popular and important type of crowdsourcing, i.e., micro-task crowdsourcing. In doing so, we analyzed research articles since 2006 and developed four classification taxonomies. First, we provided a classification framework based on the algorithmic characteristics of answer aggregation techniques. Second, we outlined the statistical and probabilistic foundations used by different types of algorithms and micro-tasks. Third, we provided a matrix catalog of the data characteristics for which an answer aggregation algorithm is designed. Fourth, a matrix catalog of the commonly used evaluation metrics for each type of micro-task was presented. This paper represents the first systematic literature analysis and classification of the answer aggregation techniques for micro-task crowdsourcing.

KEYWORDS:

• Crowdsourcing
• answer aggregation
• systematic literature analysis
• classification framework

Notes

¹ http://goo.gl/rzFfLw.

² www.mturk.com.

⁵ http://archive.ics.uci.edu/ml/.

⁶ http://ir.ischool.utexas.edu/square/.

⁷ One paper is repeated across different cells if it is applicable to more than one data set and data type (e.g., Yan et al.⁴⁷ belongs to Classification-Text-R, Classification-Image-R, Annotation-Text-R, and Annotation-Structured-B).

⁸ One paper is repeated within and across different cells if it uses several evaluation metrics and if it is applicable to more than one micro-tasking type (e.g., Welinder et al.⁴¹ uses both Mahalanobis distance and error rate and is applicable to both annotation and classification).