Are any samples representative or unbiased? reply to Pickering and Blaszczynski

ABSTRACT

Pickering and Blaszczynski’s paper (2021) claims that the problem gambling rate is inflated in paid online convenience and crowdsourced samples. However, there is a methodological flaw in their findings: they combined problem gambling rates from samples that are specific by design (e.g. at-least monthly sports bettors), and compared them to a problem gambling prevalence estimate from the general population. Pickering and Blaszczynski conflate three constructs: representativeness, bias and data quality. Data quality can be optimized through protections and checks, but these do not necessarily make samples more representative, or less biased. Many of the biases present in paid online convenience samples (e.g. self-selection biases) also apply to the gold standard of random digit dial telephone surveys, which is manifestly evident in very low response rates. These biases are also present in industry-recruited and venue-recruited samples, as well as samples of university students and treatment-seeking clients. Paid online convenience samples also have clear benefits. For example, it is possible to obtain large samples of very specific subgroups. Online surveys may reduce bias associated with self-reporting potentially stigmatizing conditions, like problem gambling. It is important not to discount research simply because it uses a paid online convenience or crowdsourced sample.

KEYWORDS:

• Gambling research
• online samples
• paid samples
• convenience samples
• recruitment

View correction statement:

Correction

Funding sources

No funding sources were declared by the authors in relation to this manuscript.

Constraints on publishing

No constraints on publishing were declared by the authors in relation to this manuscript.

Competing interests

Alex M T Russell has received funding from Victorian Responsible Gambling Foundation; New South Wales Office of Responsible Gambling; Queensland Justice and Attorney-General; Gambling Research Australia; New Zealand Ministry of Health; Australian Communications and Media Authority; National Association for Gambling Studies and the Alberta Gambling Research Institute. He has received industry funding for an evaluation of problem gambling amongst casino employees from Echo/Star Entertainment Group from 2014-2016. He has had travel expenses paid to present research by the Victorian Responsible Gambling Foundation and PsychMed. He is also affiliated with the University of Sydney. He declares no conflicts of interest in relation to this manuscript.

Matthew Browne has received research funds from the Victorian Responsible Gambling Foundation, Queensland Government Department of Health, Australian Department of Social Services, New Zealand Ministry of Health, Department of Families, Housing, Community Services and Indigenous Affairs, Department of Innovation, Industry, Science and Research, Australian Department of Foreign Affairs and Trade, Japanese Ministry of Economy, Trade and Industry. He declares no conflicts of interest in relation to this manuscript.

Nerilee Hing has received research funds from the Australian Research Council, Victorian Responsible Gambling Foundation, Gambling Research Australia, Australian Government Department of Social Services, Alberta Gambling Research Institute, the Australian Gambling Research Centre, New South Wales Office of Responsible Gambling, Queensland Justice and Attorney-General, Victorian Department of Justice, South Australian Independent Gambling Authority, New Zealand Ministry of Health, Australian Communications and Media Authority, Australia’s National Research Organisation for Women’s Safety, National Association for Gambling Studies, First Nations Foundation, Ogilvy, ORC International, and First Person Consulting. Prior to 2016, she also received funds for unpublished consulting work to improve responsible gambling measures by Echo Entertainment and Sportsbet and an honorarium from Singapore Pools for membership of its International Responsible Gambling Advisory Committee. She has had travel expenses paid to present research by the Victorian Responsible Gambling Foundation, National Association for Gambling Studies, NSW Office of Responsible Gambling, University of Alberta, Responsible Gambling Council Ontario, and the Canadian Consortium for Gambling Research. She declares that she has no conflicts of interest in relation to this manuscript.

Matthew Rockloff has received research funds from Gambling Research Australia, Victorian Responsible Gambling Foundation, Queensland Treasury, Victorian Treasury, NSW Responsible Gambling Fund, NSW Office of Liquor & Gaming, Tasmanian Department of Treasury and Finance, New Zealand Ministry of Health, Department of Families, Housing, Community Services and Indigenous Affairs, Alberta Gambling Research Institute and the First Nations Foundation. He declares no conflicts of interest in relation to this manuscript.

Philip Newall was a special advisor to the House of Lords Select Committee Enquiry on the Social and Economic Impact of the Gambling Industry. In the last three years Philip Newall has received research funding from Clean Up Gambling, and has contributed to research projects funded by GambleAware, Gambling Research Australia, NSW Responsible Gambling Fund, and the Victorian Responsible Gambling Foundation. In 2019 Philip Newall received travel and accommodation funding from the Spanish Federation of Rehabilitated Gamblers, and in 2020 received an open access fee grant from Gambling Research Exchange Ontario. Philip declares no conflicts of interest in relation to this manuscript.

Preregistration statement

No preregistration was declared by the authors in relation to this manuscript.

Data availability statement

No data set was declared by the authors in relation to this manuscript.

Notes

1. Exact comparisons between data from online panels and representative samples can be difficult, because studies that use online samples often have additional inclusion criteria, such as only accepting participants within a certain age range, or from a certain state, or only those who take part on a certain activity.

Additional information

Notes on contributors

Alex M. T. Russell

Alex M. T. Russell has worked in gambling research since 2011, and has been at CQUniversity since 2016. Dr Russell’s research interests focus on gambling behavior, including: social influences; sports betting; and using innovative research methodologies and statistics. In 2019 he was one of the ABC Top 5 Scientists.

Matthew Browne

Matthew Browne completed a PhD in psychophysiology methodology in 2002, publishing several novel methods for the analysis of EEG recordings. He has held continuing posts in major international research organizations including CSIRO. His main interests lie in the application of statistical and machine learnings methodologies across several disciplines.

Nerilee Hing

Nerilee Hing has a PhD (Gambling Studies) from University of Western Sydney. She was Founding Director of the Centre for Gambling Education and Research at Southern Cross University (2003-2016) before commencing as Research Professor (Gambling Studies) at CQUniversity. Professor Hing’s research interests include: online gambling; gambling advertising; and gambling stigma

Matthew Rockloff

Matthew Rockloff received a PhD (Psychology) from Florida Atlantic University in 1999. He was honored as a Jack Walker Scholar and an Aurel B. Newell Fellow (twice). He was named in the Top 15 Unijobs Lecturer of the Year Awards thrice (2011-2013) and received the 2017 Ig Nobel award (Economics).

Philip Newall

Philip Newall completed a PhD in Economics at the University of Stirling in 2016. Philip completed postdoctoral research fellowships at Technical University Munich and the University of Warwick before joining CQUniversity in October 2019. Dr Newall’s research applies findings from behavioral science to gambling.