The need for open science practices and well-conducted replications in the field of gambling studies

A field of study is only as strong as the research it produces. Although catchy and surprising results may generate acclaim, the research has actual value to the extent that the findings are true – they must be replicable (Popper, 1959; Schmidt, 2009). In the field of gambling studies, the scientific merit of published research has added importance given its ability to influence the regulation of gambling as well as efforts to prevent and treat disordered gambling. Unfortunately, there is reason to believe that many fields in the social sciences have a weak rate of replicability (Camerer et al., 2016, 2018; Open Science Collaboration, 2015), a revelation that has led to the so-called ‘replication crisis’ or ‘credibility revolution’. As a result of this crisis, attention has been directed at why many published findings fail to replicate. Central to these discussions has been: 1) low statistical power – the probability that a study will reject the null hypothesis (H0) when a specific alternative hypothesis (H1) is true – in published empirical research, and 2) the lack of open science practices (e.g. being explicit about how p-value(s) were obtained, posting data and materials to an online public repository). Low statistical power and the lack of open science practices decrease the replicability and credibility of reported findings. The net result has been decreased confidence in the results of published research and reforms aimed at improving new reports’ credibility.

Regrettably, the field of gambling studies has lagged behind other fields in recognizing this replication crisis and taking appropriate action. To our knowledge, as of 2019, not a single session of any academic gambling-oriented conference has been dedicated to the replication crisis, and not a single paper (until this issue of International Gambling Studies) has been published on the relevance of the replication crisis for the field of gambling studies. A positive sign is International Gambling Studies’ move toward encouraging contributors to fully disclose their research practices (i.e. open science). We laud the Journal for this change. We also laud International Gambling Studies for opening its pages to replication-based reports. This is a much needed change – a change that will help all stakeholders (e.g. researchers, policymakers, industry-related professionals) assess the replicability of past work in gambling studies. That said, we caution stakeholders that not all replications are created equal. It is important that all interested parties come to understand what constitutes a convincing replication, and why there is a need for a credibility revolution.

In the current commentary, we discuss why, regardless of journal mandates, it is imperative that contributors and reviewers consider a priori statistical power analyses, effect sizes, and ensure open disclosures about research practices. Sound research practices are essential to the vitality, credibility, and replicability of any field of research. However, a deep dive into all the aforementioned issues is beyond the scope of the current commentary; our commentary has two goals. First, we hope to initiate a conversation that focuses researchers’ attention on the replicability of previously published research in the field of gambling studies. Second, we want to shine a light on how (as a field) we can do better to ensure the replicability rate of future research allows all stakeholders to have a high degree of confidence in gambling-focused research.

Trouble in the henhouse: is the field of gambling studies heading toward a replication crisis?

It has long been known that statistical power is important to conducting informative research, and that the sample size employed should have at least 80% power to detect effects if they exist (see Cohen, 1965). Nonetheless, many studies in the psychological literature, which includes much of the field of gambling studies, is statistically underpowered (Bakker, van Dijk, & Wicherts, 2012; Cohen, 1990; Maxwell, 2004). In fact, the power of a typical two-group between-participant design has been estimated to be lower than 50% (Bakker et al., 2012; Stanley, Carter, & Doucouliagos, 2018)! Yet, the vast majority of published studies (90%) report p-values below the typical threshold for statistical significance (i.e. α = .05; Fanelli, 2010; Sterling, Rosenbaum, & Weinkam, 1995). Taken together, this means that many of the research papers published in social science journals (including this very journal) may be due to pure chance (or other shenanigans); not unlike the outcomes of electronic gaming machines. How could this be?

One culprit is publication bias (Rosenthal, 1979). Editors and reviewers have a strong tendency to accept for publication research that confirms reported hypotheses. That is, they tend to like a ‘clean story’ in which the tests of the hypotheses are statistically significant (ps < .05). The result is that large effect sizes with p-values that hover just under .05 are overrepresented in the published literature (Simmons, Nelson, & Simonsohn, 2011). A second culprit, owing perhaps to the first culprit, is that researchers tend to put studies that are not statistically significant and the corresponding materials (experimental conditions or variables) in a (now digital) file-drawer; even more perniciously, many analyses get left in the proverbial file drawer such that only a misleading subset of conditions, measures, and covariates are reported (John, Loewenstein, & Prelec, 2012; Simmons et al., 2011). Such selectivity can result in an inflated number of false positives appearing on the published record. A third culprit is that researchers have been trained explicitly (by supervisors and/or by reading Bem, 1987) or implicitly (via the review process) to HARK (Hypothesizing After the Results are Known). The result is that research reports are not necessarily written in line with the researcher’s original hypothesis. HARK-ing, akin to the other culprits in the replication crisis, has led to intellectually dishonest research reports. These culprits also raise the rate of false positives that appear in the published literature, thus undermining confidence in that literature (Kerr, 1998).

We want to be clear. We are not implying that researchers are consciously acting in bad faith (although, unfortunately, some do). It is easy for researchers to convince themselves that a secondary hypothesis or exploratory analysis was, in fact, what they had originally intended to study (see hindsight bias; Fischoff, 1975), or that something unexpected has been ‘discovered’ without considering that such exploration invalidates the interpretation of p-values. Additionally, norms develop in a field of study (e.g. HARK-ing) that can skew the research practices of researchers who frequently publish in that field. A collective effort is needed to improve research practices. However, a special burden is placed on the gatekeepers (editors and reviewers) to spur change.

Improving research practices (and the replication rate)

To improve the field of gambling studies, researchers must come to embrace open science practices (either on their own or via journal mandate). Open science begins and ends with full and accurate reporting of all aspects of the reported research (Cumming, 2014). To this end, current best practice is for researchers to, in advance of running the study, determine the hypothesis or hypotheses that will be tested and then pre-register their idea(s) as well as the plans for analysis on an open access research repository (e.g. Open Science Framework or AsPredicted; for pre-registration resources see https://cos.io/prereg/). This reduces HARK-ing by decreasing researchers’ degrees of freedom. Exploration is still possible, but pre-registration more clearly distinguishes exploration from confirmatory tests. A link to the pre-registration document should be included in the text of the article.

Included in the pre-registration document (as well as the text of the paper) should be the reporting of a priori power analyses. Despite the centrality of power in null-hypothesis significance testing (NHST; Gigerenzer, 2004), formal power analyses are rarely reported in the gambling literature. As previously stated, research in the social sciences (and likely the field of gambling studies) has been underpowered. The problem with underpowered (low N) studies is that the observed p-value is very unstable when small samples are drawn from the population (i.e. sampling variability; Cumming, 2014; Kline, 2004). The solution is not simply a matter of running large N studies (which has been made easier in the wake of some gambling operators’ willingness to allow researchers access to player-account data as well as crowdsourced data collection; e.g. Amazon’s Mechanical Turk). This is because the probability of observing a statistically significant effect increases as the sample size increases (Cohen, 1994; Meehl, 1978). With very large samples it is often more informative when a p-value is not significant than when it is significant.

Given the relation between the p-value and sample size, some researchers suggest the p-value should be either abandoned entirely (Ioannidis, 2005) or only interpreted alongside an effect size estimation (Cumming, 2012). Regardless of the direction the field of gambling studies takes, researchers should determine what constitutes a meaningful effect size before they conduct a study. For example, is a Cohen’s d of 0.01 sufficient to claim that a responsible gambling tool has a meaningful effect on limit adherence? What about d = 0.10? (see Funder & Ozer, 2019 for a discussion about what constitutes a reasonable effect size in psychological research). Once this determination is made, statistical programs like G*Power (Faul, Erdfelder, Lang, & Buchner, 2007) or Monte Carlo simulations can be used to establish the required N to detect that effect size. At present, the field of gambling studies is not engaging with such issues.

Researchers also need to be open about the method they employed. Simmons, Nelson, and Simonsohn (2014) suggested that all published papers should explicitly note how sample size was determined and all data exclusions (if any; i.e. how many participants were excluded from the analysis and why). Additionally, researchers should note all manipulations and measures in the study, even if they were not included in the reported analyses. This will allow readers of a given research report to get a complete understanding of what participants experienced. Such explicit clarifications signal quality and rigour, which is an advantage over papers that omit such clarifications. Moreover, researchers are learning that open science practices facilitate replication – an issue to which we now turn.

A replication by any other name?

Given the weak replication rate in other fields of study, it is high time for the field of gambling studies to engage in some replication efforts. For those new to the credibility revolution (or for those who have not thought about replication deeply), it is important to underscore that a completely exact replication of an original study is impossible (see Rosenthal, 1991; Tsang & Kwan, 1999). Instead, the best that a researcher can hope for is a replication study that only differs from the original study in terms of the participants who take part (see Schmidt, 2009). However, such replications are difficult (if not impossible) to conduct due to contextual differences that exist between the time, culture, and space in which the original study was embedded and that of the replication study. For example, there have been substantial changes in the gambling landscape in Ontario – Ontario Lottery and Gaming (OLG) has sold all casinos in the province to private industry. For this reason, among others, the release of a responsible gambling tool developed by OLG and tested by Wohl, Davis, and Hollingshead (2017) is uncertain. The tool in question asks players to estimate how much money they spent playing EGM’s at casinos and slot machine venues in Ontario and then provides real-time feedback about their actual expenditures. Wohl et al. (2017) observed that the feedback resulted in a downregulation of gambling expenditures during a three-month follow-up period among those who underestimated their losses. Given the aforementioned changes to the gambling landscape in Ontario, a close replication of Wohl et al. (2017) may be a difficult undertaking. Given this reality, it begs the question: What makes a good and convincing replication?

According to Brandt et al.'s (2014) ‘replication recipe’, for researchers to conduct a convincing close replication they must first explicitly determine the effect of interest. If the original study was a 2 × 2 between-participants design, the researcher must decide whether the effect of interest is one of the main-effects or the interaction (or perhaps a simple slope). This should be pre-registered. The researcher must then follow the method of the original study as much as possible (e.g. participant recruitment, materials, analyses). In the best-case scenario, the researcher discusses the replication attempt with the researcher(s) who conducted the original study and receives the materials necessary to conduct a close replication. Of course, this is not always possible due to, among other things, lack of cooperation from the original researchers, loss of the original materials, or the researcher(s) who conducted the original study are no longer in academia or deceased. Close replications become more difficult to conduct as the number of deviations mount because deviations introduce researcher degrees of freedom into the replication (i.e. judgement calls; see Simons, Shoda, & Lindsay, 2017).

Where close replications are not possible, researchers could attempt a conceptual replication. For example, if a replication of Wohl et al. (2017) was attempted, researchers could test the idea that many players are extraordinarily poor estimators of their gambling expenditures (time and money), and that making underestimates of expenditure salient to players can reduce their subsequent expenditures. Regardless of whether the replication is ‘close’ or ‘conceptual’, any deviation from the original study should be noted in text, and the results of the replication interpreted in light of those differences. The original researcher can help minimize the degrees of freedom of those who wish to conduct a replication by openly archiving materials and being explicit about the conditions under which they expect their results to generalize or not (Simons et al., 2017).

Next, the replication must have high statistical power. Simonsohn (2015) suggested that replication studies should have 2.5 times the original number of participants. This ensures that there is good power to detect the original effect (if true), and – importantly – also large enough to demonstrate that the original effect was implausible if null results are found in the replication study (notwithstanding alternative explanations around method or sample differences). This guideline is useful for ‘lab sized’ studies, but less applicable in large-N studies where the critical issue is less about potentially low statistical power and more about whether the effect size in question is large enough to be of theoretical and practical interest. Ultimately, central to a replication is that the researcher uses a sample size that allows for the detection of the effect size of interest; the 2.5 times guideline for smaller-N studies assumes that the effect size of interest is one that the original study had a reasonable chance of detecting. In very large-N studies, a more theoretical or practical approach to effect size and power must be taken. That is, statistically ‘significant’ effects may be trivially small and not practically significant (and are then also more plausibly due to idiosyncratic causes or confounds). Very large-N (original) studies have the benefit of estimating effect sizes very accurately. As such, replication studies are on firmer ground using the original effect size to power a replication effort.

A replication study may even use a smaller sample if replicating a large-N study's effect because there is firm justification based on a large original effect size. Alternatively, replication of a large-N study's original effect may use a smaller sample size if the replication is still well powered to detect the smallest effect size of theoretical or practical importance. For example, if a 50,000 participant study found a large correlation between trait impulsivity and gambling losses, it would not take a similarly large-N study to persuasively replicate this finding. If that same large effect were found in a small-N study, the 2.5 times sample size rule applies better.

In the spirit of open science, the researcher who conducts a replication (akin to those who conducted the original study) should make all details of the replication open access (i.e. freely and publicly available). Although we also encourage researchers of original data to do that, open access takes on a special importance in replications, especially where there is a failure to replicate. All researchers should be able to evaluate the ‘closeness’ of the replication, and how convincing the outcome is, for themselves.

The reaction of the research community: a cautionary conclusion

From a positivist perspective, with enough sound empirical research, the ‘truth’ can be approximated. Getting closer to ‘truth’ starts with sound, original open science. Researchers in the social sciences (and the Editors of International Gambling Studies) are coming to understand that the path to truth is also paved with replications. There will be bumps along the way. Theory and research findings that are now sacred cows may need to be revised or discarded. It is important that when that happens the research community refrains from ad hominem attacks on the researcher(s) who conducted the original research and/or the replicators. It is worth repeating that no replication is direct or exact. Science is a cumulative endeavour. As the field of gambling studies opens the door to replications, those who work in the field should keep an open mind as replications come online. The research conducted in this field aims to understand and prevent disordered gambling and help those who have a gambling disorder recover. This should be the focus. It will be heartening to see research replicate. However, some of the findings in the field of gambling studies will fail to replicate. When this occurs introspection should be done to understand why the original study failed to replicate, whether the effect is real (or not), and how to improve our science. Real players (present and past) are affected by the work the field of gambling studies produces.

We hope this commentary initiates a needed conversation about the state of the literature in the field of gambling studies, why replication and transparency are important, and how the results of attempts to replicate are framed. Perhaps more importantly, however, we hope this commentary serves as a call to researchers in the field of gambling studies to adopt open science practices. We suggest researchers read the Open Science Collaboration (2017) chapter about workflow as a pragmatic place to learn about how to open their science. A move to accept replication studies is hollow without a parallel move toward open science.

Conflict of interest

Competing interest

Dr. Michael J. A. Wohl. Funding: Dr. Wohl has received gambling-related research funding from provincial granting agencies in Canada. He has also received direct and/or indirect research funds from the gambling industry in Canada, United States of America, United Kingdom, Australia and Sweden. Additionally, he has served as a consultant for the gambling industry in Canada, United States of American, New Zealand, and Australia. Funds were also awarded by federal granting agencies in Canada and Australia for research unrelated to gambling. A detailed list can be found on his curriculum vitae (http://carleton.ca/bettermentlabs/wp-content/uploads/CV.pdf). No funds were provided to Wohl for the preparation of the current paper. Constraints on publishing: There are no constrains on publishing. Competing interests: There are no competing interest to declare.

Dr. Nassim Tabri. Funding: Dr. Tabri has received gambling-related research funding from Gambling Research Exchange Ontario (GREO)—a provincial granting agency in Canada. He has also received indirect research funds from the gambling industry in Canada, United States, and the United Kingdom. Additionally, he has served as a consultant for the gambling industry in Canada and the United States. Funds were also awarded by federal granting agencies in Canada for research unrelated to gambling. No funds were provided to Tabri for the preparation of the current paper. Constraints on publishing: There are no constrains on publishing. Competing interests: There are no competing interest to declare.

Dr. John M. Zelenski. Funding: Dr. Zelenski has received no funding for addiction-related research, nor has he consulted on these issues with industry. Constraints on publishing: There are no constrains on publishing. Competing interests: There are no competing interest to declare.

All authors declare that there are no potential conflicts of interest with respect to the research, authorship, and/or publication of this article. Correspondence concerning this article should be addressed to Michael J. A. Wohl, Department of Psychology, Carleton University, 1125 Colonel By Drive, Ottawa, Ontario, Canada, K1S 5B6. Tel: [phone] x 2908, E-mail: [email protected]