Experienced survey participants, biased responses? – The example of cognitive reflection

ABSTRACT

The Cognitive Reflection Test (CRT) is used in behavioural studies to describe individuals as intuitive or reflective. In recent years, behavioural scientists have increasingly recruited participants on online platforms. Besides convenience, apt platform recruitment is believed to increase validity and facilitate more meaningful inferences towards the population of interest compared to studies based on students. But particularly with regard to CRT, there is an often-underestimated drawback: ‘experienced clickers’ might systematically distort findings.

KEYWORDS:

• Cognitive reflection test
• professional clickworkers
• measurement error
• extension replication

JEL CLASSIFICATION:

• C91
• D01

Acknowledgments

• The overall project has been approved by the institutional review board of the Martin Luther University Halle-Wittenberg.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

¹ For more information on the association between stated exposure to CRT and experience with surveys, see Appendix 3.

Additional information

Funding

The financial support of the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation – 388911356) is gratefully acknowledged.