Reducing uncertainty in research: introducing registered reports

1. EJPT’s mission statement

On its website, the European Journal of Psychotraumatology (EJPT) states ‘The Journal shares ESTSS’s mission to advance and disseminate scientific knowledge about traumatic stress’ (European Journal of Psychotraumatology, 2018; www.tandfonline.com/ejpt). So far, this reads like the mission statement of almost every society’s academic journal. However, the EJPT differs from many other journals in how it approaches this goal. Most notably, the EJPT embraces a Gold Open Access policy since its foundation in 2010 (Olff, 2010, 2016).With the announcement to allow the submission of registered reports, the EJPT has taken a leading role in the field.

2. Accessibility

In today’s world, we expect clinicians and policy makers to take their decisions based on scientific evidence. Although in the internet age, information and knowledge appear to be at hand whenever needed, access to scientific knowledge is often not. The fact that almost half of all the scientific articles published in 2011 are behind a paywall (Van Noorden, 2013) illustrates this problem. For scientists it is crucial to have access to these articles and therefore their institutions pay significant subscription fees for journals in which these articles are being published. However, most people outside the scientific community, including clinicians and policy makers, especially in low- and middle-income countries, do not have free access to the knowledge they need. Instead, they have to buy single articles, which can be very costly. All in all, this prevents many in need of scientific knowledge from accessing it.

One solution for this problem is publishing scientific articles open access (OA). This means that an article is made freely available, usually via the internet. There are two models of open access publication. First, there are Gold Open Access journals, which publish all their articles open access. The cost for publishing the articles has to be paid by the authors or their funders. Second, there are traditional subscription journals who offer their authors the option to pay an extra fee to make their article freely accessible. This second model is referred to as Hybrid or Green Open Access. In accordance with its mission statement, the EJPT falls in the former category, allowing free access to all its content.

Over the last years, more and more researchers decided to publish their articles OA. With regard to psychotraumatology, the proportion of articles published as OA in 2017 varied greatly between countries (with more than 35% for authors from South Korea to less than 15% for authors from Israel; Olff, 2018). From a macro perspective, the overall proportion of Gold OA publications in psychotraumatology increased from only a few percent at the beginning of the millennium to 17.7% of all articles published in 2017 (Olff, 2018). In the near future, the proportion of OA publications will likely continue to increase, maybe even faster than before. The underlying reason of this trend is that more and more funders release OA policies, requesting that research funded by them be published OA. Only recently a coalition of several national European funding agencies and the European Research Council announced their Plan S which restricts the target journals of research funded by them to Gold OA journals starting 1 January 2020 (cOALition S, 2018). This change in policy will likely result in dramatic changes in which journals psychotraumatology research can be published open access, given that, so far, the EJPT is the only Gold OA journal in the field (Olff, 2018).

3. Reliability

3.1. Replication crisis

Over the last years, the replication crisis unfolded in psychology. The finding that many experimental studies cannot be replicated did also affect other scientific disciplines, for example, cancer biology (Kaiser, 2018). So far, we are not aware of any studies investigating the field of psychotraumatology in this regard. Nevertheless, given that research in psychotraumatology is conducted by researchers from different fields already affected by the replication crisis (e.g. psychology, neuroscience and social science), the odds are high that psychotraumatology is affected too.

3.2. Wrong incentives

At the core of the replication crisis lies the insight that the way we currently practise science does not necessarily advance scientific knowledge. Instead of generating robust evidence, we spend huge sums of money to produce often underpowered, but overinterpreted, studies. We systematically favour significant over null results (Fanelli, 2012). Rarely is this bias towards positive findings the results of fraud like in the famous case of Diederik Stapel who fabricated complete datasets (Noort Committe, Drenth Committee, & Levelt Committee, 2012). Often scholars try to simply maximize their chances of advancing their career by following incentives that are deeply rooted in the academic system. For example, the number of publications or a scientist’s impact factor are important measures that many grant or university committees rely on when reviewing applications for grants or professorships. This makes publication in high impact journals a necessity for early career researchers. Traditional subscription journals have to maintain their brand by publishing a constant stream of spectacular and potentially ground-breaking findings. In summary, individual scientists are incentivized to publish as many spectacular studies with significant results as possible. By trying to do so, many researchers use questionable research practices (QRP). Most of us are aware of some QRP and have most likely conducted them ourselves. For example p-hacking, the process of reanalysing a dataset until a significant result has been found (Simmons, Nelson, & Simonsohn, 2011), or HARKing, hypothesizing after the results are already known (Kerr, 1998; for more QRPs see Spellman, Gilbert, & Corker, 2017). These QRPs have a positive effect on getting ‘publishable’ results. However, if most of the researchers in a field use QRPs to get significant results and only publish these, the field as a whole suffers the consequences by an accumulation of positive findings in the literature (Fanelli, 2010). This undermines the robustness of their studies as a whole, which has been named ‘the file drawer problem’ (because negative results do not get published but vanish into the file drawer; Rosenthal, 1979).

3.3. Changing incentives

Since the beginning of the replication crisis many suggestions were made as to how to fix the crisis (for an overview see Spellman et al., 2017). There was even a society founded in 2016 – the Society for the Improvement of Psychological Science – with the aim to improve psychological sciences. However, nearly all proposals aim at resetting the incentives for individual researchers by aligning them with the basic goal of scientific research: advancing knowledge. Against the background of the complex causes of this crisis, there will not be one single solution. For instance, FAIR (Findable, Accessible, Interoperable, Reusable) data principles which are being more and more embraced by researchers will speed up scientific progress. In this editorial we will bring forward how registered reports will advance reliable research.

4. Registered reports

4.1. The idea

The idea behind registered reports is to change the criteria for the acceptance of a publication (Nosek, Ebersole, DeHaven, & Mellor, 2018). In accordance with the aim to advance knowledge, publication should be granted based on the relevance of the question investigated and the appropriateness of the chosen methods and study design. The reasoning here is that all answers to important questions, if obtained appropriately, are important, sometimes the negative even more than the positive. Thus, publication should not depend on the results. Registered reports are a formalized way to achieve this.

4.2. The implementation

Registered reports are a two phase publishing model (Figure 1). First, authors submit a manuscript outlining the relevance of the question aimed to be investigated, their proposed methods, a power analysis with a sample size estimation, a detailed data analysis plan, and preferably pilot data. Furthermore, the plan is registered in a data base open to the public (sometimes with an embargo), the same way it is already mandatory for clinical trials. Then, the manuscript will undergo stage one peer review. If the manuscript fulfils the above mentioned criteria, the journal offers in-principle acceptance. This means that as long as the authors follow the protocol they submitted, the manuscript will be published when the study is completed, independent of the results. In the next step, the study is conducted and the manuscript completed by adding the results and their interpretation as a discussion, but leaving the introduction and methods unchanged. Deviations from the protocol have to be outlined and reasonably explained. Then the full manuscript undergoes stage two review. At this stage, the reviewers assess if the authors followed their protocol, also by reaching sufficient power, and if the conclusion stated in the discussion are in accordance with the results. If this is the case, the manuscript is published, independently of the results. For information on how to prepare Registered Reports submissions visit the EJPT instructions for authors page (https://www.tandfonline.com/action/authorSubmission?journalCode=zept20&page=instructions).

Figure 1. Registered report workflow diagramme (retrieved from https://cos.io/rr/).

4.3. Two frequently asked questions

The two concerns most often raised concerning registered reports are first, that with preregistration, exploratory analyses are not possible and that second, registered reports can only be used for original data. Both of these assumptions are not accurate. First, registering and following a protocol does not prohibit secondary explorative analyses. There will always be important questions raised while conducting the study. However, authors are required to state on what basis the explorative analyses were undertaken and cannot change the preregistered primary outcome of the study to an ‘interesting’ not registered secondary one. Second, registered reports are suitable for every kind of hypothesis testing. This includes studies using already obtained data (for which special templates can be found here: https://osf.io/bpuw3/) and/or using qualitative measures. More frequently raised concerns and arguments are detailed online at https://cos.io/rr/.

4.4. Personal advantages

4.4.1. Predictable processes

Registered reports have numerous advantages, also for researchers who decide to publish their projects as registered reports. First, it increases the predictability of the process of conducting research. The stage one peer-review takes place prior to the data collection and only focuses on the relevance of the proposed question and adequate study design, including sufficient methodology and power. Although the planning of a study may thus be more time consuming initially, changing essential parts of the methodology after data collection, surely is even more resource intense. Furthermore, once in principal acceptance is offered, the study can be conducted with the security that its results will be published. With regard to one’s own career, this increases the predictability of the outcome of a given scientific project. In addition, for example when applying for a new position during the conductance of a project, in principal accepted papers can be listed on the curriculum vitae, indicating a certain future outcome. Second, registered reports allow for fast publication, once the data is collected and the final manuscript written, because stage two review is limited to results and discussion and therefore faster than a traditional review process. Furthermore, in principal acceptance is granted by the journal who offered it, meaning that one does not have to ‘find a home’ for an article by repeatedly submitting it until it is rejected for the ‘non-novelty of the idea’, the results or the chosen methodology. This also increases the speed of the final publication. Third, as the results published as registered reports are more likely to be methodological adequate, developing further studies based on previously registered reports will also increase the likelihood of investigating a ‘true’ effect.

4.4.2. Students

High process predictability and guaranteed publications might not be very important for established senior researchers. However, they are crucial for students and early-career researchers whose career might end when a project gets prolonged or only produces ‘unpublishable’ results. Registered reports are especially valuable, also educationally, for students. With the two stage process, students have to focus on identifying a question relevant for their field and developing an adequate methodological and analytical plan first. The first round of review also exposes them to external reviewers helping to shape the proposed question and methods. Only if both are sufficient, the student can continue with the experiments. Once finished their studies, students are left with a methodological skill set, a basic idea of how a research project has to be planned and executed and at least one publication. Overall, this increases the certainty of a successful completion of the studies and also increases a student’s position when applying for a job in or outside of academia. Especially with regard to the above average prevalence of mental health problems in PhD students (Levecque, Anseel, De Beuckelaer, Van der Heyden, & Gisle, 2017), increasing security for PhD students is highly important.

4.5. Why does the EJPT care?

As stated at the beginning of the article, the EJPT shares the ESTSS’s mission to advance scientific knowledge about traumatic stress and disseminated it in a way that clinicians, policy makers and everyone interested can access it freely. Given these aims, research published in the EJPT has real life consequences for many people, including patients. Therefore, we have the ethical obligation to ensure that our research is reliable. Otherwise we might harm those who we want to serve. With this in mind, registered reports will help us as researchers, our scientific field and the EJPT as a journal to increase the reliability of the research that is being published.