Using memes and emoji-scales in a web survey: experimental assessment of consequences for multimodal cognitive effort and data quality

References

• Albaum, G., Roster, A., & Smith, M. (2014). Topic sensitivity and research design: Effects on internet survey respondents’ motives. Asia Pacific Journal of Marketing & Logistics, 26(1), 147–161. https://doi.org/10.1108/APJML-12-2012-0127
   Google Scholar
• Bosch, O. J., & Revilla, M. (2021). Using emojis in mobile web surveys for millennials? A study in Spain and Mexico. Quality & Quantity, 55(1), 39–61. https://doi.org/10.1007/s11135-020-00994-8
   Google Scholar
• Brüggen, E., Wetzels, M., De Ruyter, K., & Schillewaert, N. (2011). Individual differences in motivation to participate in online panels: The effect on response rate and response quality perceptions. International Journal of Market Research, 53(3), 369–390. https://doi.org/10.2501/IJMR-53-3-369-390
   Web of Science ®Google Scholar
• Burke, B. (2016). Gamify: How gamification motivates people to do extraordinary things. Routledge.
   Google Scholar
• Callegaro, M. (2005, July). Origins and developments of the cognitive models of answering questions in survey research. In 1st Annual Meeting of the European Association for Survey Research, WAPOR thematic seminar on Quality criteria in survey research VI, Barcelona. University of Nebraska.
   Google Scholar
• Cechanowicz, J., Gutwin, C., Brownell, B., & Goodfellow, L. (2013, October). Effects of gamification on participation and data quality in a real-world market research domain. In L. E. Nacke, K. Harrigan, & N. Randall (Eds.), Proceedings of the First International Conference on Gameful Design, Research, and Applications (pp. 58–65). ACM.
   Google Scholar
• Couper, M. (1998). Measuring survey quality in a CASIC environment. In Proceedings of the Survey Research Methods Section of the ASA at JSM1998, Alexandria, VA. (pp. 41–49). American Statistical Association.
   Google Scholar
• Couper, M. (2002). New technologies and survey data collection: Challenges and opportunities. In International Conference on Improving Surveys, Copenhagen, Denmark (pp. 1–24).
   Google Scholar
• De Raad, B. (2000). The big five personality factors: The psycholexical approach to personality. Hogrefe & Huber Publishers.
   Google Scholar
• Deterding, S. (2015). The lens of intrinsic skill atoms: A method for gameful design. Human–Computer Interaction, 30(3–4), 294–335. https://doi.org/10.1080/07370024.2014.993471
   Web of Science ®Google Scholar
• Deviatko, I. F., Bogdanov, M. B., & Lebedev D. V. (2021). Pupil diameter dynamics as an indicator of the respondent’s cognitive load: Methodological experiment comparing CASI and P&PSI. Vestnik Rossijskogo universiteta druzhby narodov. Serija Sociologija, 21(1), 36–49. https://doi.org/10.22363/2313-2272-2021-21-1-36-49
   Google Scholar
• Deviatko, I. F., & Lebedev, D. V. (2017). Through the eyes of the interviewer, through the eyes of the respondent: Outlining a new approach towards the assessment of cognitive load during the interview. Monitoring of Public Opinion: Economic and Social Changes, 5, 1–19. https://doi.org/10.14515/monitoring.2017.5.01
   Google Scholar
• Dillman, D. A. (2000). Mail and internet surveys: The tailored design method. John Wiley & Sons.
   Google Scholar
• Fan, W., & Yan, Z. (2010). Factors affecting response rates of the web survey: A systematic review. Computers in Human Behavior, 26(2), 132–139. https://doi.org/10.1016/j.chb.2009.10.015
   Web of Science ®Google Scholar
• Fernández-Fontelo, A., Kieslich, P. J., Henninger, F., Kreuter, F., & Greven, S. (2021). Predicting question difficulty in web surveys: A machine learning approach based on mouse movement features. Social Science Computer Review, 41(1), 141–162. https://doi.org/10.1177/08944393211032950
   Web of Science ®Google Scholar
• Galesic, M., Tourangeau, R., Couper, M. P., & Conrad, F. G. (2008). Eye-tracking data: New insights on response order effects and other cognitive shortcuts in survey responding. Public Opinion Quarterly, 72(5), 892–913. https://doi.org/10.1093/poq/nfn059
   PubMed Web of Science ®Google Scholar
• Gerjets, P., Scheiter, K., & Catrambone, R. (2004). Designing instructional examples to reduce intrinsic cognitive load: Molar versus modular presentation of solution procedures. Instructional Science, 32(1), 33–58. https://doi.org/10.1023/B:TRUC.0000021809.10236.71
   Web of Science ®Google Scholar
• Groves, R. M., & Lyberg, L. (2010). Total survey error: Past, present, and future. Public Opinion Quarterly, 74(5), 849–879. https://doi.org/10.1093/poq/nfq065
   Web of Science ®Google Scholar
• Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work?–a literature review of empirical studies on gamification. In 2014 47th Hawaii International Conference on System Sciences, Hawaii. (pp. 3025–3034). IEEE. https://doi.org/10.1109/HICSS.2014.377
   Google Scholar
• Han, V., Albaum, G., Wiley, J. B., & Thirkell, P. (2009). Applying theory to structure respondents’ stated motivations for participating in web surveys. Qualitative Market Research: An International Journal, 12(4), 428–442. https://doi.org/10.1108/13522750910993338
   Google Scholar
• Harms, J., Biegler, S., Wimmer, C., Kappel, K., & Grechenig, T. (2015, September). Gamification of online surveys: Design process, case study, and evaluation. In IFIP Conference On Human-Computer Interaction (pp. 219–236). Springer, Cham.
   Google Scholar
• Hart, S. G., & Staveland, L. E. (1988). Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology (Vol. 52, pp. 139–183). North-Holland. https://doi.org/10.1016/S0166-4115(08)62386-9
   Google Scholar
• Horwitz, R., Brockhaus, S., Henninger, F., Kieslich, P. J., Schierholz, M., Keusch, F., & Kreuter, F. (2020). Learning from mouse movements: Improving questionnaires and respondents’ user experience through passive data collection. Advances in Questionnaire Design, Development, Evaluation and Testing, 403–425. https://doi.org/10.1002/9781119263685.ch16
   Google Scholar
• Horwitz, R., Kreuter, F., & Conrad, F. (2017). Using mouse movements to predict web survey response difficulty. Social Science Computer Review, 35(3), 388–405. https://doi.org/10.1177/0894439315626360
   Web of Science ®Google Scholar
• Kasanov, D. (2020). The design of the study of the psychophysiological foundations of the state of cognitive overload. Humanitarian Knowledge and Artificial Intelligence: Strategies and Innovations: 4th UrFU Youth Convention—Ekaterinburg, 2020, 878–881.
   Google Scholar
• Koen, B., Loosveldt, G., Vandenplas, C., & Stoop, I. (2018). Response rates in the European Social Survey: Increasing, decreasing, or a matter of fieldwork efforts? Survey Methods: Insights from the Field, 1–12. https://doi.org/10.13094/SMIF-2018-00003
   Google Scholar
• Krosnick, J. A., Narayan, S., & Smith, W. R. (1996). Satisficing in surveys: Initial evidence. New Directions for Evaluation, 1996(70), 29–44. https://doi.org/10.1002/ev.1033
   Google Scholar
• Lebedev, D. 2020. Paradata: definition, types, collection, and possible uses. Monitoring of Public Opinion: Economic and Social Changes, 2, 4–32. https://doi.org/10.14515/monitoring.2020.2.915
   Google Scholar
• Malhotra, N. (2008). Completion time and response order effects in web surveys. Public Opinion Quarterly, 72(5), 914–934. https://doi.org/10.1093/poq/nfn050
   Web of Science ®Google Scholar
• Mavletova, A. (2015a). A gamification effect in longitudinal web surveys among children and adolescents. International Journal of Market Research, 57(3), 413–438. https://doi.org/10.2501/IJMR-2015-035
   Web of Science ®Google Scholar
• Mavletova, A. (2015b). Web surveys among children and adolescents: Is there a gamification effect? Social Science Computer Review, 33(3), 372–398. https://doi.org/10.1177/0894439314545316
   Web of Science ®Google Scholar
• Mavletova, A., Couper, M. P., & Lebedev D. (2018). Grid and Item-by-Item Formats in PC and Mobile Web Surveys. Social Science Computer Review, 36(6), 647–668. https://doi.org/10.1177/0894439317735307
   Web of Science ®Google Scholar
• Mavletova, A., Gavrilov, K., & Tholmogorova, T. (2019). Gamifying a web survey among adolescents: Effects on understanding of risk, risk calculation, and ratio-bias. Journal of Risk Research, 22(12), 1532–1545. https://doi.org/10.1080/13669877.2018.1501592
   Web of Science ®Google Scholar
• McClain, C. A., Couper, M. P., Hupp, A. L., Keusch, F., Peterson, G., Piskorowski, A. D., & West, B. T. (2019). A typology of web survey paradata for assessing total survey error. Social Science Computer Review, 37(2), 196–213. https://doi.org/10.1177/0894439318759670
   Web of Science ®Google Scholar
• Olson, K., & Parkhurst, B. (2013). Collecting paradata for measurement error evaluations. Improving Surveys with Paradata: Analytic Uses of Process Information, 43–72. https://doi.org/10.1002/9781118596869.ch3
   Google Scholar
• Paas, F. G. (1992). Training strategies for attaining transfer of problem-solving skill in statistics: A cognitive-load approach. Journal of Educational Psychology, 84(4), 429. https://doi.org/10.1037/0022-0663.84.4.429
   Web of Science ®Google Scholar
• Phan, W. M. J., Amrhein, R., Rounds, J., & Lewis, P. (2019). Contextualizing interest scales with emojis: Implications for measurement and validity. Journal of Career Assessment, 27(1), 114–133. https://doi.org/10.1177/1069072717748647
   Web of Science ®Google Scholar
• Puleston, J. (2013, February). Online research, game on. In NCRM Conference ‘‘Web Surveys for the General Population: How, Why and When, London (pp. 25–26).
   Google Scholar
• Redline, C. D., & Lankford, C. P. (2001). Eye-movement analysis: A new tool for evaluating the design of visually administered instruments ( paper and web).
   Google Scholar
• Rentfrow, P. J., & Gosling, S. D. (2003). The do re mi’s of everyday life: The structure and personality correlates of music preferences. Journal of Personality and Social Psychology, 84(6), 1236. https://doi.org/10.1037/0022-3514.84.6.1236
   PubMed Web of Science ®Google Scholar
• Roberts, C., Gilbert, E., Allum, N., & Eisner, L. (2019). Research synthesis: Satisficing in surveys: A systematic review of the literature. Public Opinion Quarterly, 83(3), 598–626. https://doi.org/10.1093/poq/nfz035
   Web of Science ®Google Scholar
• Roster, C. A. (2021). Using memes in online surveys to engage and motivate respondents. International Journal of Market Research, 63(6), 754–771. https://doi.org/10.1177/1470785320981821
   Web of Science ®Google Scholar
• Różycka-Tran, J., Boski, P., & Wojciszke, B. (2015). Belief in a zero-sum game as a social axiom: A 37-nation study. Journal of Cross-Cultural Psychology, 46(4), 525–548. https://doi.org/10.1177/0022022115572226
   Web of Science ®Google Scholar
• Sampietro, A. (2020). Use and interpretation of emoji in electronic-mediated communication: A survey. Visual Communication Quarterly, 27(1), 27–39. https://doi.org/10.1080/15551393.2019.1707086
   Web of Science ®Google Scholar
• Schwartz, S. H. (1992). Universals in the content and structure of values: Theoretical advances and empirical tests in 20 countries. In M. P. Zanna (Ed.), Advances in experimental social psychology (Vol. 25, pp. 1–65). Academic Press.
   Google Scholar
• Stieger, S., & Reips, U. D. (2010). What are participants doing while filling in an online questionnaire: A paradata collection tool and an empirical study. Computers in Human Behavior, 26(6), 1488–1495. https://doi.org/10.1016/j.chb.2010.05.013
   Web of Science ®Google Scholar
• Sudman, S., Bradburn, N. M., & Schwarz, N. (1996). Thinking about answers: The application of cognitive processes to survey methodology. Jossey-Bass.
   Google Scholar
• Sweller, J. (1988). Cognitive load during problem solving: Effects on learning. Cognitive Science, 12(2), 257–285. https://doi.org/10.1207/s15516709cog1202_4
   Web of Science ®Google Scholar
• Sweller, J., Ayres, P., & Kalyuga, S. (2011). Measuring cognitive load. In Cognitive load theory (pp. 71–85). Springer.
   Google Scholar
• Szeto, M. D., Barber, C., Ranpariya, V. K., Anderson, J., Hatch, J., Ward, J., Aguilera MN, Hassan S, Hamp A, Coolman T, & Dellavalle, R. P. (2022). Emojis and emoticons in health care and dermatology communication: Narrative review. Jmir Dermatology, 5(3), e33851. https://doi.org/10.2196/33851
   PubMedGoogle Scholar
• Tomyn, A. J., & Cummins, R. A. (2011). The subjective wellbeing of high-school students: Validating the personal wellbeing Index—school children. Social Indicators Research, 101(3), 405–418. https://doi.org/10.1007/s11205-010-9668-6
   Web of Science ®Google Scholar
• Tourangeau, R. (1984). Cognitive aspects of survey methodology: Building a bridge between disciplines. https://doi.org/10.17226/930
   Google Scholar
• Yan, T., Ryan, L., Becker, S. E., & Smith, J. (2015). Assessing quality of answers to a global subjective well-being question through response times. In Survey research methods (Vol. 9, pp. 101–109). NIH Public Access.
   Google Scholar