Using memes and emoji-scales in a web survey: experimental assessment of consequences for multimodal cognitive effort and data quality

ABSTRACT

Decreasing web survey participation rates lead to higher importance of ways to increase respondents’ motivation to participate and gamification is seen as one of possible ways to do so. However, there is no clear understanding on how it affects respondents’ cognitive load. This study investigates whether gamification reduces cognitive load among 18–25-year-old survey respondents. A laboratory experiment was conducted with 128 randomly assigned students to either a non-gamified or gamified web survey design. Gamification was implemented using emojis for ordinal scales and motivating memes between sets of questions. Cognitive load was measured through paradata, subjective evaluation, and neurophysiological indicator (pupil diameter dynamics). Results show higher cognitive effort for the gamification group based on pupil diameter dynamics, but subjective evaluation, mouse movements, and completion times show increased cognitive effort for the non-gamified group. Attention was higher for the gamification group, and data quality did not differ. Therefore, gamification may increase cognitive load, but it also increases motivation and engagement, leading to similar data quality compared to non-gamified web surveys.

KEYWORDS:

• Gamification
• cognitive effort
• pupil diameter
• paradata
• pupillometry

Acknowledgments

The authors are grateful to Dr. Inna Deviatko, Dr. Aigul Klimova, and Dr. Tatiana Semenova for valuable feedback and advice during the research process and article preparation, as well as for their help with data collection.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/13645579.2024.2359474

Notes

1. Script for data preprocessing and analysis is available at: https://github.com/rus-slm/Gamification-research/blob/main/diploma%20work.ipynb

Additional information

Funding

The research was supported by RSF under Grant (project number 2228-00968, project title: “Eye-tracking data and pupillometry in multimodal measurement of the respondents’ cognitive load”).

Notes on contributors

Ruslan Suleymanov

Ruslan Suleymanov is a Data Analyst in one of the largest Russian IT-companies. He earned a BA degree in Sociology and is now getting an MA degree in Business Informatics at HSE University. Sociological background is an advantage in product analytics to get user’s motivation and product experience. His research interests include users’ interaction with products and interfaces, and paradata and eye-tracking as ways to explore it.

Daniil Lebedev

Daniil Lebedev holds a PhD in Sociology from HSE University, where he also completed his BA and MA degrees in Sociology. He has served as a Lecturer in the Department of Sociological Research Methods at HSE University. His research interests focus on survey methodology, paradata, satisficing theory, and respondent behaviour during web survey completion.