Recruiting from the shallow end of the pool: Differences in cognitive and compliance measures for subject pool participants based on enrollment time across an academic term

ABSTRACT

Cognitive psychologists often recruit through university-organized subject pools. Such pools are effective for data collection, providing access to a convenient sample. However, there are potential concerns, including the commonly held belief that the students who volunteer for participation at the end of the academic term might provide less reliable data. Such intuitions are not unfounded, as previous research has found demographic, personality, and motivational differences between students who participate in subject pool studies at the start versus the end of an academic term. However, empirical data on cognitive performance are equivocal, with some studies finding evidence for cognitive differences based on time-of-term participation and some studies failing to. The current study administered a visual search task in a subject pool at the start and end of multiple academic terms. The end-of-term participants were less accurate than the start-of-term participants, with a larger performance difference on difficult targets. Furthermore, the end-of-term participants were also less reliable – more likely to arrive late or fail to show up altogether. These results suggest that extra care needs to be taken when recruiting from subject pools, and suggestions are provided for how the visual cognition community can best make use of this important research tool.

KEYWORDS:

• Subject pool recruitment
• cognitive psychology
• research practices
• academic term effects

Acknowledgements

The authors thank Laura Schubel and the GWU Visual Cognition Lab for their help with data collection. They also thank Michelle Kramer, Samoni Nag, and Dwight Kravitz for helpful feedback.

Disclosure statement

No potential conflict of interest was reported by the authors.

Data availability statement

The data and materials are available upon reasonable request to the authors.

Additional information

Funding

This work was supported by a postdoctoral fellowship to S. H Adamo from the Ford Fellowship Foundation (SHA) and a grant to S. R. Mitroff from the Army Research Office (grant W911NF-16-1-0274).