Crowdsourcing research: Data collection with Amazon’s Mechanical Turk

References

• Abshire, M., Dinglas, V. D., Cajita, M. I. A., Eakin, M. N., Needham, D. M., & Himmelfarb, C. D. (2017). Participant retention practices in longitudinal clinical research studies with high retention rates. BMC Medical Research Methodology, 17(1), 458. doi: 10.1186/s12874-017-0310-z
   Web of Science ®Google Scholar
• Alonso, O., & Lease, M. (2011). Crowdsourcing for information retrieval: Principles, methods, and applications. Proceedings of the 34th international ACM SIGIR conference on research and development in information retrieval, pp. 1299–1300. ACM. doi: 10.1145/2009916.2010170
   Google Scholar
• Arditte, K. A., Çek, D., Shaw, A. M., & Timpano, K. R. (2016). The importance of assessing clinical phenomena in mechanical Turk research. Psychological Assessment, 28(6), 684–691. doi: 10.1037/pas0000217
   PubMed Web of Science ®Google Scholar
• Barr, J., & Cabrera, L. F. (2006). AI gets a brain. Queue, 4(4), 24. doi: 10.1145/1142055.1142067
   Google Scholar
• Berinsky, A. J., Huber, G. A., & Lenz, G. S. (2012). Evaluating online labor markets for experimental research: Amazon. Com’s mechanical Turk. Political Analysis, 20(3), 351–368. doi: 10.1093/pan/mpr057
   Web of Science ®Google Scholar
• Brabham, D. C. (2010). Moving the crowd at threadless: Motivations for participation in a crowdsourcing application. Information, Communication & Society, 13(8), 1122–1145. doi: 10.1080/13691181003624090
   Web of Science ®Google Scholar
• Buhrmester, M., Kwang, T., & Gosling, S. D. (2011). Amazon’s mechanical Turk: A new source of inexpensive, yet high-quality, data? Perspectives on Psychological Science, 6(1), 3–5. doi:10.1037/e527772014-223 doi: 10.1177/1745691610393980
   Web of Science ®Google Scholar
• Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 2156–2160. doi: 10.1016/j.chb.2013.05.009
   Web of Science ®Google Scholar
• Chambers, S., Nimon, K., & Anthony-McMann, P. (2016). A primer for conducting survey research using MTurk: Tips for the field. International Journal of Adult Vocational Education and Technology (IJAVET), 7(2), 54–73. doi: 10.4018/IJAVET.2016040105
   Web of Science ®Google Scholar
• Chandler, J., Paolacci, G., Peer, E., Mueller, P., & Ratliff, K. A. (2015). Using nonnaive participants can reduce effect sizes. Psychological Science, 26(7), 1131–1139. doi:10.3758/s13428-013-0365-7 doi: 10.1177/0956797615585115
   PubMed Web of Science ®Google Scholar
• Cooper, E. A., & Farid, H. (2016). Does the sun revolve around the earth? A comparison between the general public and online survey respondents in basic scientific knowledge. Public Understanding of Science, 25(2), 146–153. doi: 10.1177/0963662514554354
   PubMed Web of Science ®Google Scholar
• Cushing, E. (2013). Amazon mechanical Turk: The digital sweatshop. Utne Reader. Retrieved from http://www.utne.com/science-and-technology/amazon-mechanical-turk-zm0z13jfzlin
   Google Scholar
• Deng, X., & Joshi, K. D. (2013). Understanding crowd workers’ perceptions of crowdsourcing career. Thirty Fourth International Conference on Information Systems, Milan. Retrieved from https://pdfs.semanticscholar.org/73ef/ab88621309fdf3d39ac2aff8c70b193c0606.pdf
   Google Scholar
• Dobson, J. (2013). Mechanical Turk: Amazon’s new underclass. Huffington Post, p. 19. Retrieved from http://www.huffingtonpost.com/julian-dobson/mechanical-turk-amazons-underclass_b_2687431.html
   Google Scholar
• Dworkin, J., Hessel, H., Gliske, K., & Rudi, J. H. (2016). A comparison of three online recruitment strategies for engaging parents. Family Relations, 65(4), 550–561. doi: 10.1111/fare.12206
   PubMed Web of Science ®Google Scholar
• Ford, J. B. (2017). Amazon’s mechanical Turk: A comment. Journal of Advertising, 46(1), 156–158. doi: 10.1080/00913367.2016.1277380
   Web of Science ®Google Scholar
• Fort, K., Adda, G., & Cohen, K. B. (2011). Amazon mechanical Turk: Gold mine or coal mine? Computational Linguistics, 37(2), 413–420. doi: 10.1162/coli_a_00057
   Web of Science ®Google Scholar
• Gardner, R. M., Brown, D. L., & Boice, R. (2012). Using Amazon’s mechanical Turk website to measure accuracy of body size estimation and body dissatisfaction. Body Image, 9(4), 532–534. doi: 10.1016/j.bodyim.2012.06.006
   PubMed Web of Science ®Google Scholar
• Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224. doi: 10.1002/bdm.1753
   Web of Science ®Google Scholar
• Goodman, J., & Paolacci, G. (2017). Crowdsumers take over. Retrieved from https://u.osu.edu/goodman/files/2016/12/JCR-MTurk-Tutorial-1dnc063.pdf
   Google Scholar
• Hauser, D. J., & Schwarz, N. (2016). Attentive turkers: MTurk participants perform better on online attention checks than do subject pool participants. BehaviorRresearch Methods, 48(1), 400–407. doi: 10.3758/s13428-015-0578-z
   PubMed Web of Science ®Google Scholar
• Henrich, J., Heine, S. J., & Norenzayan, A. (2010). The weirdest people in the world? Behavioral and Brain Sciences, 33(2–3), 61–83. doi: 10.1017/s0140525x0999152x
   PubMed Web of Science ®Google Scholar
• Hitlin, P. (2016). Research in the crowdsourcing age, a case study. Retrieved from http://assets.pewresearch.org/wpcontent/uploads/sites/14/2016/07/PI_2016.07.11_Mechanical-Turk_FINAL.pdf
   Google Scholar
• Holden, C. J., Dennie, T., & Hicks, A. D. (2013). Assessing the reliability of the M5-120 on Amazon’s mechanical Turk. Computers in Human Behavior, 29(4), 1749–1754. doi: 10.1016/j.chb.2013.02.020
   Web of Science ®Google Scholar
• Horton, J. J., Rand, D. G., & Zeckhauser, R. J. (2011). The online laboratory: Conducting experiments in a real labor market. Experimental Economics, 14(3), 399–425. doi: 10.1007/s10683-011-9273-9
   Web of Science ®Google Scholar
• Howe, J. (2006). The rise of crowdsourcing. Wired Magazine, 14(6), 1–4.
   Google Scholar
• Huff, C., & Tingley, D. (2015). “Who are these people?” Evaluating the demographic characteristics and political preferences of MTurk survey respondents. Research & Politics, 2(3), 205316801560464. doi:2053168015604648 doi: 10.1177/2053168015604648
   Web of Science ®Google Scholar
• Ipeirotis, P. G. (2010). Analyzing the Amazon mechanical Turk marketplace. XRDS: Crossroads, The ACM Magazine for Students, 17(2), 16–21. doi: 10.1145/1869086.1869094
   Google Scholar
• Irani, L. C., & Silberman, M. (2013). Turkopticon: Interrupting worker invisibility in Amazon mechanical Turk. Proceedings of the SIGCHI conference on human factors in computing systems, pp. 611–620, ACM. doi: 10.1145/2470654.2470742
   Google Scholar
• Jonason, P. K., & Luévano, V. X. (2013). Walking the thin line between efficiency and accuracy: Validity and structural properties of the dirty dozen. Personality and Individual Differences, 55(1), 76–81. doi: 10.1016/j.paid.2013.02.010
   Web of Science ®Google Scholar
• Jones, D. N., & Paulhus, D. L. (2014). Introducing the short dark triad (SD3) a brief measure of dark personality traits. Assessment, 21(1), 28–41. doi: 10.1177/1073191113514105
   PubMed Web of Science ®Google Scholar
• Kees, J., Berry, C., Burton, S., & Sheehan, K. (2017). An analysis of data quality: Professional panels, student subject pools, and Amazon’s mechanical Turk. Journal of Advertising, 46(1), 141–155. doi: 10.1080/00913367.2016.1269304
   Web of Science ®Google Scholar
• Kuek, S. C., Paradi-Guilford, C., Fayomi, T., Imaizumi, S., Ipeirotis, P., Pina, P., & Singh, M. (2015). The global opportunity in online outsourcing (No. 22284). The World Bank. Retrieved from http://documents.worldbank.org/curated/en/138371468000900555/The-global-opportunity-in-online-outsourcing
   Google Scholar
• Lapidot-Lefler, N., & Barak, A. (2015). The benign online disinhibition effect: Could situational factors induce self-disclosure and prosocial behaviors? Cyberpsychology: Journal of Psychosocial Research on Cyberspace, 9(2), article 3. doi: 10.5817/CP2015-2-3
   Google Scholar
• Lease, M., Hullman, J., Bigham, J. P., Bernstein, M. S., Kim, J., Lasecki, W., Bakhashi, S., Mitra, T., & Miller, R. C. (2013). Mechanical Turk is not anonymous. SSRN. doi: /10.2139/ssrn.2228728
   Google Scholar
• Litman, L., Robinson, J., & Rosenzweig, C. (2015). The relationship between motivation, monetary compensation, and data quality among US-and India-based workers on mechanical Turk. Behavior Research Methods, 47(2), 519–528. doi: 10.3758/s13428-014-0483-x
   PubMed Web of Science ®Google Scholar
• Mason, W., & Suri, S. (2012). Conducting behavioral research on Amazon’s mechanical Turk. Behavior Research Methods, 44(1), 1–23. doi: 10.3758/s13428-011-0124-6
   PubMed Web of Science ®Google Scholar
• Morton, E. (2015). The mechanical chess player that unsettled the world. Retrieved from http://www.slate.com/blogs/atlas_obscura/2015/08/20/the_turk_an_supposed_chess_playing_robot_was_a_hoax_that_started_an_early.html
   Google Scholar
• Paolacci, G., & Chandler, J. (2014). Inside the Turk: Understanding mechanical Turk as a participant pool. Current Directions in Psychological Science, 23(3), 184–188. doi: 10.1177/0963721414531598
   Web of Science ®Google Scholar
• Peer, E., Brandimarte, L., Samat, S., & Acquisti, A. (2017). Beyond the Turk: Alternative platforms for crowdsourcing behavioral research. Journal of Experimental Social Psychology, 70, 153–163. doi: 10.1016/j.jesp.2017.01.006
   Web of Science ®Google Scholar
• Peer, E., Paolacci, G., Chandler, J., & Mueller, P. (2012). Screening participants from previous studies on Amazon mechanical turk and qualtrics (Unpublished manuscript).
   Google Scholar
• Schleider, J. L., & Weisz, J. R. (2015). Using mechanical Turk to study family processes and youth mental health: A test of feasibility. Journal of Child and Family Studies, 24(11), 3235–3246. doi: 10.1007/s10826-015-0126-6
   Web of Science ®Google Scholar
• Schmidt, F. A. (2013, September). The good, the bad and the ugly: Why crowdsourcing needs ethics. Cloud and Green Computing (CGC), 2013 Third International Conference, pp. 131–535. doi: 10.1109/cgc.2013.89
   Google Scholar
• Schütz, E., Sailer, U., Al Nima, A., Rosenberg, P., Arntén, A. C. A., Archer, T., & Garcia, D. (2013). The affective profiles in the USA: Happiness, depression, life satisfaction, and happiness-increasing strategies. PeerJ, 1, e156. doi: 10.7717/peerj.156
   PubMed Web of Science ®Google Scholar
• Sheehan, K. B., & Pittman, M. (2016). Amazon’s mechanical Turk for academics: The HIT handbook for social science research. Irvine, CA: Melvin & Leigh.
   Google Scholar
• Siegel, J. T., Navarro, M. A., & Thomson, A. L. (2015). The impact of overtly listing eligibility requirements on MTurk: An investigation involving organ donation, recruitment scripts, and feelings of elevation. Social Science & Medicine, 142, 256–260. doi: 10.1016/j.socscimed.2015.08.020
   PubMed Web of Science ®Google Scholar
• Simons, D. J., Chabris, C. F., & de Fockert J. (2012). Common (mis) beliefs about memory: A replication and comparison of telephone and mechanical Turk survey methods. PloS One, 7(12), e51876. doi: 10.1371/journal.pone.0051876
   PubMed Web of Science ®Google Scholar
• Smith, S. M., Roster, C. A., Golden, L. L., & Albaum, G. S. (2016). A multi-group analysis of online survey respondent data quality: Comparing a regular USA consumer panel to MTurk samples. Journal of Business Research, 69(8), 3139–3148. doi: 10.1016/j.jbusres.2015.12.002
   Web of Science ®Google Scholar
• Springer, V. A., Martini, P. J., Lindsey, S. C., & Vezich, I. S. (2016). Practice-based considerations for using multi-stage survey design to reach special populations on Amazon’s mechanical Turk. Survey Practice, 9(5). Retrieved from http://www.surveypractice.org/index.php/SurveyPractice/article/view/305/html_75
   Google Scholar
• Stewart, N., Ungemach, C., Harris, A. J., Bartels, D. M., Newell, B. R., Paolacci, G., & Chandler, J. (2015). The average laboratory samples a population of 7,300 Amazon mechanical Turk workers. Judgment and Decision Making, 10(5), 479–491.
   Web of Science ®Google Scholar
• Wessling, K. S., Huber, J., & Netzer, O. (2017). MTurk Character Misrepresentation: Assessment and Solutions. Journal of Consumer Research, 44(1), 211–230. doi: 10.1093/jcr/ucx053
   Web of Science ®Google Scholar