Open data through Registered Reports can accelerate cumulative knowledge

The scientific ‘credibility revolution’ has, in many fields, ushered in fast-paced improvements to the way that research is conducted (Vazire 2018). Sparked by concerns regarding replication and reproducibility, open research practices including preprints, preregistration, Registered Reports, open materials, code, and data aim to change the research landscape by improving the robustness and credibility of findings (Pennington 2023). Peer Community In Registered Reports (PCI RR) is a new publishing platform that integrates all of these open science practices: researchers submit a Stage 1 Registered Report through a preprint server, and after undergoing peer-review and receiving in principle acceptance (IPA), this Stage 1 protocol is then preregistered. At Stage 2, researchers append their results and discussion to the approved protocol, along with open materials, code, and data¹ and, upon acceptance, this final preprint is then ‘recommended’ to the research community (see Eder and Frings 2021). The aim of this modified review process is to mitigate biased research practices and publication processes and, in this respect, Registered Reports appear to be working (Chambers and Tzavella 2022).

One benefit for authors submitting through the PCI RR publishing route is that they can chose to publish their work in any ‘PCI friendly’ journal without the need for additional peer review. Addiction Research & Theory is one such journal offering this publishing route, committing to accept Stage 2 manuscripts that have received a positive final recommendation through PCI RR that meet the journal’s scope and formatting requirements (see Pennington and Heim 2022).

As Handling Editor, I am pleased to announce that ART has published its first Registered Report through this route. Karhulahti, Vahlo et al. (2022) assessed how ontologically diverse screening instruments for gaming-related health problems differ in identifying associated problem groups. In addition to championing the authors adherence to open science practices, the goal of this editorial is to document the value of open data that is promoted by the Registered Report publishing model. I believe strongly that it is important to document the early history of open science practices and researcher’s experiences as they navigate them, particularly to overcome some of the perceived barriers associated with them and to further encourage uptake (see Norris et al. 2022). Below I first highlight the research findings by Karhulahti and colleagues and the acceleration of recommended research directions that stemmed from this team’s adoption of open code and data, before outlining more generally the positive changes we are observing as a result of the scientific credibility revolution.

In their Registered Report, Karhulahti et al. administered four central screening instruments (GAS7, IGDT10, GDT, and THL1) in gaming disorder measurement to a large, nationally representative sample of Finnish participants and showed that these instruments revealed different prevalence rates and considerable heterogeneity in group overlap. Based on these findings, they suggest that due to their foundational ontological diversity these instruments might measure different problems (or other constructs) to varying degrees. Their article concludes with recommendations for researchers to (a) define their construct of interest (e.g. whether they are measuring gaming disorder or gaming-related problems) and (b) seek evidence for good construct validity to ensure accurate measurement.

By sharing their code, data, and materials on the Open Science Framework repository, an independent team of researchers were able to follow one of Karhulahti et al.’s proposed future directions for this research: ‘to chart further ontological differences and similarities between constructs and/or instruments’ using an item-based network model. Billieux and Fournier (2022a) conducted this exploratory model using all of the items from the four gaming disorder assessment tools in the original study to assess potential communalities among these items. This network analysis indicated very high density of connections among all items with the authors suggesting that ‘these instruments are not reliably distinct and that their content strongly overlaps, therefore measuring substantially homogeneous constructs after all’ (pp. 1). Despite the different findings between the two teams, the authors agreed that the screening of gaming disorder requires improvement and harmonization with regards to its measurement. Moreover, Billieux and Fournier highlighted the benefits of open science practices in driving cumulative science forward.

Karhulahti, Adamkovič et al. (2022) then reanalyzed their data, again using network analysis, and wrote a reply to Billieux and Fournier. As the original dataset al.so included measures from non-gaming constructs, Karhulahti et al. decided to further test whether network overlap might also occur with other constructs – namely anxiety, depression, and bullying – that are ontologically distinct from gaming disorder. Given that these constructs do not share conceptual origins, Karhulahti et al. theorized that there should (following Billieux and Fournier’s argument) be little overlap between the items. However, their results suggested that there was indeed notable overlap between these constructs. In a parallel analysis, they also investigated whether a single-factor or four-factor structure was supported by this model, with the findings revealing that the optimal solution has ‘even more factors than the number of theoretical constructs included in the model’. Through sharing their data and engaging in collegial discourse, both teams of researchers then wrote a final reply which delved deeper into statistical parameters and exploratory decisions which may further explain these findings and deposited this on the Open Science Framework (see Billieux and Fournier 2022b).

Shortly after this collaborative journey, Karhulahti’s open data was again used by Amendola (2023) who applied the harmful dysfunction analysis (HDA) perspective to explore the prevalence of gaming disorder. HDA is considered a rigorous and thoughtful attempt to address serious conceptual problems in clinical psychology and psychopathology that delineates between dysfunction and harm (see McNally 2001; Wakefield’s 1992). Using this approach, Amendola showed that prevalence rates from the original sample slightly increased due to endorsing the minimum criteria of dysfunction and harm, but that the four instruments convergence was mostly unaffected. This reply concluded that HDA may provide a promising approach to identify individuals with gaming disorder most in need of support.

The research by Karhulahti and colleagues and these subsequent engagements with the ideas put forward therefore highlight how open data, through the Registered Report format, can help drive the generation of new knowledge. From a personal perspective, what happened after the publication of this article has been one of the most positive experiences in my editorial role to date; observing independent teams reanalyze open data and collaborate with original authors in a truly transparent and collegiate way has opened a window into the myriad of benefits that open science practices can bring.

Researchers commonly report that ‘being scooped’ and not receiving appropriate credit are barriers to data sharing (Gomes et al. 2022; Houtkoop et al. 2018), but this case in hand provides evidence that these concerns are not well-founded (see also Bishop 2015; Kathawalla et al. 2021). Instead, the collaboration between researchers can maximize use of limited resources (e.g. funding for data collection), allow pressing research questions to be addressed in an efficient and timely manner (as well as from different perspectives), and stimulate future research directions. The editorial team at ART looks forward to receiving more Registered Reports via the PCI RR platform and to supporting the increasing uptake of open science practices in our field with a view to helping elevate addiction science to a highly credible and reproducible research domain.

Ethical approval

The research in this paper does not require ethics board approval.

Disclosure statement

Charlotte Pennington is a Handling Editor at Addiction Research & Theory with a particular brief to support ART’s commitment to open science by overseeing Registered Reports. She is also a Recommender of PCI RRs (see https://rr.peercommunityin.org/about/recommenders).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes

1 To the maximum extent permissible by relevant legal or ethical restrictions.