Why don’t we care more about carelessness? Understanding the causes and consequences of careless participants

References

• Aruguete, M. S., Huynh, H., Browne, B. L., Jurs, B., Flint, E., & McCutcheon, L. E. (2019). How serious is the ‘carelessness’ problem on Mechanical Turk? International Journal of Social Research Methodology, 22, 441–449.
   Web of Science ®Google Scholar
• Bowling, N. A., Huang, J. L., Bragg, C. B., Khazon, S., Liu, M., & Blackmore, C. E. (2016). Who cares and who is careless? Insufficient effort responding as a reflection of respondent personality. Journal of Personality and Social Psychology, 111(2), 218–229.
   PubMed Web of Science ®Google Scholar
• Carr, P. B., & Steele, C. M. (2010). Stereotype threat affects financial decision-making. Psychological Science, 21(10), 1411–1416.
   PubMed Web of Science ®Google Scholar
• Casler, K., Bickel, L., & Hackett, E. (2013). Separate but equal? A comparison of participants and data gathered via Amazon’s MTurk, social media, and face-to-face behavioral testing. Computers in Human Behavior, 29(6), 2156–2160.
   Web of Science ®Google Scholar
• Castagno, M. D., Lange, K. M., & Edlund, J. E. (2017). Nesting effects across the menstrual cycle. North American Journal of Psychology, 19(1), 103–112.
   Google Scholar
• Cialdini, R., Trost, M., & Newsom, J. (1995). Preference for consistency: The development of a valid measure and the discovery of surprising behavioral implications. Journal of Personality and Social Psychology, 69, 318–328.
   Web of Science ®Google Scholar
• Curran, P. G. (2016). Methods for the detection of carelessly invalid responses in survey data. Journal of Experimental Social Psychology, 66, 4–19.
   Web of Science ®Google Scholar
• Edlund, J. E., Lange, K. M., Sevene, A. M., Umansky, J., Beck, C. D., & Bell, D. J. (2017). Participant crosstalk: Issues when using the Mechanical Turk. The Quantitative Methods for Psychology, 13(3), 174–182.
   Web of Science ®Google Scholar
• Edlund, J. E., & Nichols, A. L. (Eds.). (2019). Advanced research methods for the social and behavioral sciences. Cambridge, UK: Cambridge University Press.
   Google Scholar
• Edlund, J. E., Nichols, A. L., Okdie, B. M., Guadagno, R. E., Eno, C. A., Heider, J. D., … Wilcox, K. T. (2014). The prevalence and prevention of crosstalk: A multi-institutional study. Journal of Social Psychology, 154, 1–5.
   PubMed Web of Science ®Google Scholar
• Edlund, J. E., & Sagarin, B. J. (2014). The Mate Value scale. Personality and Individual Differences, 64, 72–77.
   Web of Science ®Google Scholar
• Furnham, A., Hyde, G., & Trickey, G. (2015). Personality and value correlates of careless and erratic questionnaire responses. Personality and Individual Differences, 80, 64–67.
   Web of Science ®Google Scholar
• Goodman, J. K., Cryder, C. E., & Cheema, A. (2013). Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making, 26(3), 213–224.
   Web of Science ®Google Scholar
• Gosling, S. D., Rentfrow, P. J., & Swann, W. B. (2003). A very brief measure of the Big-Five personality domains. Journal of Research in Personality, 37(6), 504–528.
   Web of Science ®Google Scholar
• Huang, J. L., Bowling, N. A., Liu, M., & Li, Y. (2015). Detecting insufficient effort responding with an infrequency scale: Evaluating validity and participant reactions. Journal of Business and Psychology, 30(2), 299–311.
   Web of Science ®Google Scholar
• Huang, J. L., Curran, P. G., Keeney, J., Poposki, E. M., & DeShon, R. P. (2012). Detecting and deterring insufficient effort responding to surveys. Journal of Business and Psychology, 27(1), 99–114.
   Web of Science ®Google Scholar
• John, O. P., Donahue, E. M., & Kentle, R. L. (1991). The Big Five inventory–versions 4a and 54. Berkeley, CA: University of California, Berkeley, Institute of Personality and Social Research.
   Google Scholar
• Kam, C. C. S., & Fan, X. (2017). Approaches to handling common response styles and issues in educational surveys. Oxford Research Encyclopedia of Education, Oxford University Press.
   Google Scholar
• Kam, C. C. S., & Meyer, J. P. (2015). How careless responding and acquiescence response bias can influence construct dimensionality: The case of job satisfaction. Organizational Research Methods, 18(3), 512–541.
   Web of Science ®Google Scholar
• Lennox, R. D., & Wolfe, R. N. (1984). Revision of the self-monitoring scale. Journal of Personality and Social Psychology, 46(6), 1349–1364.
   PubMed Web of Science ®Google Scholar
• Levenson, H. (1973). Multidimensional locus of control in psychiatric patients. Journal of Consulting and Clinical Psychology, 41, 397–404.
   PubMed Web of Science ®Google Scholar
• Lovett, M., Bajaba, S., Lovett, M., & Simmering, M. J. (2018). Data quality from crowdsourced surveys: A mixed method inquiry into perceptions of amazon’s mechanical turk masters. Applied Psychology, 67(2), 339–366.
   Web of Science ®Google Scholar
• Maniaci, M. R., & Rogge, R. D. (2014). Caring about carelessness: Participant inattention and its effects on research. Journal of Research in Personality, 48, 61–83.
   Web of Science ®Google Scholar
• Meade, A. W., & Craig, S. B. (2012). Identifying careless responses in survey data. Psychological Methods, 17(3), 437–455.
   PubMed Web of Science ®Google Scholar
• Meyer, J. F., Faust, K. A., Faust, D., Baker, A. M., & Cook, N. E. (2013). Careless and random responding on clinical and research measures in the addictions: A concerning problem and investigation of their detection. International Journal of Mental Health and Addiction, 11(3), 292–306.
   Web of Science ®Google Scholar
• Nichols, A. L., & Edlund, J. E. (2015). Practicing what we preach (and sometimes study): Methodological issues in laboratory research. Review of General Psychology, 19(2), 191–202.
   Web of Science ®Google Scholar
• Nichols, A. L., & Edlund, J. E. (2019). Performing research in the laboratory. In J. E. Edlund & A. L. Nichols (Eds.), Advanced research methods for the social and behavioral sciences (pp. 38–52). Cambridge, UK: Cambridge University Press.
   Google Scholar
• Nichols, A. L., & Maner, J. K. (2008). The good subject effect: Investigating participant demand characteristics. Journal of General Psychology, 135, 151–165.
   PubMed Web of Science ®Google Scholar
• Niessen, A. S. M., Meijer, R. R., & Tendeiro, J. N. (2016). Detecting careless respondents in web-based questionnaires: Which method to use? Journal of Research in Personality, 63, 1–11.
   Web of Science ®Google Scholar
• Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of Personality and Social Psychology, 46(3), 598–609.
   Web of Science ®Google Scholar
• Peytchev, A., & Peytcheva, E. (2017). Reduction of measurement error due to survey length: Evaluation of the split questionnaire design approach. Survey Research Methods, 11(4), 361–368.
   Web of Science ®Google Scholar
• Rosenberg, M. (1965). Society and the adolescent self-image. Princeton, NJ: Princeton University Press.
   Google Scholar
• Scheier, M. F., & Carver, C. S. (1985). The self-consciousness scale: A revised version for use with general populations. Journal of Applied Social Psychology, 15, 687–699.
   Web of Science ®Google Scholar
• Schmitt, N., & Stults, D. M. (1985). Factors defined by negatively keyed items: The result of careless respondents? Applied Psychological Measurement, 9(4), 367–373.
   Web of Science ®Google Scholar
• Schroder, H. S., Dawood, S., Yalch, M. M., Donnellan, M. B., & Moser, J. S. (2016). Evaluating the domain specificity of mental health–related mind-sets. Social Psychological and Personality Science, 7(6), 508–520.
   Web of Science ®Google Scholar
• Siegel, J. T., Navarro, M. A., & Thomson, A. L. (2015). The impact of overtly listing eligibility requirements on MTurk: An investigation involving organ donation, recruitment scripts, and feelings of elevation. Social Science & Medicine, 142, 256–260.
   PubMed Web of Science ®Google Scholar
• Silber, H., Danner, D., & Rammstedt, B. (2019). The impact of respondent attentiveness on reliability and validity. International Journal of Social Research Methodology, 22(2), 153–164.
   Web of Science ®Google Scholar
• Silber, H., Roßmann, J., & Gummer, T. (2018). When near means related: Evidence from three web survey experiments on inter-item correlations in grid questions. International Journal of Social Research Methodology, 21(3), 275–288.
   Web of Science ®Google Scholar
• Silber, H., Schröder, J., Struminskaya, B., Stocké, V., & Bosnjak, M. (2019). Does panel conditioning affect data quality in ego-centered social network questions? Social Networks, 56, 45–54.
   Web of Science ®Google Scholar
• Thompson, M. M., Naccarato, M. E., Parker, K. C. H., & Moskowitz, G. B. (2001). The personal need for structure and personal fear of invalidity measures: Historical perspectives, current applications, and future directions. In G. B. Moskowitz (Ed.), Cognitive social psychology: The princeton symposium on the legacy and future of social cognition (pp. 19–39). Princeton, NJ: Lawrence Erlbaum Associates Publishers.
   Google Scholar
• Tom, S. M., Fox, C. R., Trepel, C., & Poldrack, R. A. (2007). The neural basis of loss-aversion in decision-making under risk. Science, 315(5811), 515–518.
   PubMed Web of Science ®Google Scholar
• Van Sonderen, E., Sanderman, R., & Coyne, J. C. (2013). Ineffectiveness of reverse wording of questionnaire items: Let’s learn from cows in the rain. PloS One, 8(7), e68967.
   PubMed Web of Science ®Google Scholar
• Wang, R., & Krosnick, J. A. (2020). Middle alternatives and measurement validity: A recommendation for survey researchers. International Journal of Social Research Methodology, 23(2), 169–184.
   PubMed Web of Science ®Google Scholar
• Ward, M. K., & Pond III, S. B. (2015). Using virtual presence and survey instructions to minimize careless responding on internet-based surveys. Computers in Human Behavior, 48, 554–568.
   Web of Science ®Google Scholar
• Weathers, D., & Bardakci, A. (2015). Can response variance effectively identify careless respondents to multi-item, unidimensional scales? Journal of Marketing Analytics, 3(2), 96–107.
   Google Scholar
• Woods, C. M. (2006). Careless responding to reverse-worded items: Implications for confirmatory factor analysis. Journal of Psychopathology and Behavioral Assessment, 28(3), 186–191.
   Web of Science ®Google Scholar
• Zakrisson, I. (2005). Construction of a short version of the Right-Wing Authoritarianism (RWA) scale. Personality and Individual Differences, 39, 863–872.
   Web of Science ®Google Scholar