Editor’s note: replication crisis in the social sciences

Science seeks to pursue knowledge and objective truths through the conduct of carefully designed research methodologies and experimental paradigms. In disseminating the results of studies, the peer-review process adds or at least ought to add weight to the degree to which research publications can be viewed as advancing our understanding of any given field of inquiry. The peer-review process is designed to provide a degree of confidence that the research addresses an important question that builds upon and extends the extant literature, the methodology and analyses are appropriate and robust, and the results and conclusions do not exceed the data and consider alternative viewpoints. Pseudoscience, junk science, poor research practices, unaddressed biases and deliberate fraud act to challenge the extent to which the public and academics can maintain confidence in accepting findings and conclusions reached and represent a serious wastage of taxpayer public funding for research.

In recent times, some academics have expressed concerns of the flawed and questionable nature of some research practices and how institutional pressures have contributed to the push for academics to ‘publish or perish’ and gain grants to improve chances of promotion and to enhance the overall reputational status of universities (see Chambers, 2017 for an overview). In this volume, we publish two articles that draw attention to some of the practices and failures that are endemic within the social sciences and broader fields that represent a threat to reputational damage, and the need for self-reflection and adoption of improved standards of research conduct. Both Michael Wohl, Nassim Tabri, and John Zelenski and Debi LaPlante draw attention to the need for replication studies given their paucity, and the use of open-science and data repositories to allow for existing ‘truths’ and theories to be truly tested and confirmed. Both Wohl and his colleagues and LaPlante acknowledge that replication studies represent a potential threat in undermining existing theories and the reputational standing of authors, but these issues are more than subsumed by replication studies increasing the value and confidence of gambling related research.

Metrics used to evaluate the impact of research include publication in top-tiered high impact journals. As Chambers (2017) notes, most journals seek to publish articles that report novel, original and/or innovative findings that ultimately are used to attract citations and increase journal impact factors. Consequently, journals lack the motivation to publish replication studies given that these are unlikely to be cited to the same extent as the original publication. However, replication studies are crucial to the advancement of science where seminal findings are either challenged or supported. The former is important in avoiding the uncritical acceptance of findings that might represent Type 1 errors, while the latter adds weight to the reliability and validity of such findings.

An additional impediment is the failure of journals to consider publishing null findings resulting in the ‘file draw’ effect. This has serious implications in failing to inform other academics of blind alleys thereby avoiding wasted resources through others engaging in redundant research. In reflecting on the role and responsibilities of journal editors, the International Gambling Studies editorial team have elected to actively address these issues by expanding its policies to promote and improve its dissemination of scientific findings.

International Gambling Studies is changing its policies and submission declarations to encourage authors to adopt more stringent research processes in pre-registering their research and establishing a priori hypotheses and sample sizes to limit hypothesising after the results are known, and post hoc increasing recruitment to obtain sample sizes the ‘just reach significant differences’. Concurrently, the journal will allocate space to replication studies where the necessary rationale and justification for the need for replication is cogently argued. The implication of this policy is that authors will need to include the full disclosure and description of the methodology, measures, recruitment and management of sample size, excluded data and outliers sufficient to allow for independent researchers to complete a direct replication. This detail may be included in manuscripts, as supplementary online materials, or in links to pre-registration or open science sites and we will be asking reviewers and editors to consider these materials when making their recommendations. To encourage best practice, IGS will start offering Open Science Badges, a Centre for Open Science Initiative, which will be displayed on published articles. Badges will be awarded if the paper has been 1) pre-registered, 2) includes open data and 3) includes open materials. Full details will be available on the Instructions for Authors page. In addition to the many broad data repositories, we draw attention to the Transparency Project run by the Division of Addiction. This provides a data repository specifically for the gambling field.

We expect that papers that contain links to where data can be accessed and provide full methodological and analytical details will have higher citation rates as these will be used in subsequent replication studies. Nonetheless, the editorial team acknowledges that many authors have commenced research programs without pre-registration. The absence of pre-registration will not be a barrier to manuscripts being accepted for publication We hope that over time, we will see an increasing number of manuscripts being submitted that include pre-registration of specific research details and a link to where the data used in the research can be freely accessed by other research teams. We will encourage authors to cite recent research in addition to ‘seminal’ studies and to question older references given the pace at which the gambling field is changing. Confirmation of pre-registration and access to data will enhance the scientific standing of a manuscript, and the objective of the editorial team is to gradually transition to the stage where pre-registration will become an accepted standard practice.

Disclosure statement

No potential conflict of interest was reported by the authors.