We need to change our attitude, and journals can help: Reflections in response to Spiller & Olff (2018)

ABSTRACT

Adopting Registered Reports is an important step for the European Journal of Psychotraumatology to promote open science practices in the field of psychotrauma research. However, adopting these practices requires us as individual researchers to change our perspective fundamentally. We need to put fears of being scooped aside, adopt a permissive stance towards making mistakes and accept that null-results should be part of the scientific record. This is difficult because the culture in academia is competitive. Incentives are on publishing novel and positive results in high impact journals. A change in journal policies, such that openness and transparency are reinforced, can facilitate an attitude change in individual researchers.

HIGHLIGHTS

• Adopting open science practices requires researchers to change their perspective on openness and transparency.• Lasting changes in individual researchers depend on a change in academic culture such that high quality studies rather than positive results are rewarded.• Journal policies, such as accepting Registered Reports and enforcing open data and materials, are important for facilitating both individual and cultural change.

La adopción de Reportes Registrados es un paso importante para la Revista Europea de Psicotraumatología para promover las prácticas de la ciencia abierta en el campo de la investigación en psicotrauma. Sin embargo, la adopción de esta práctica requiere de nosotros como investigadores individuales de cambiar fundamentalmente nuestra perspectiva. Necesitamos poner a un lado nuestros temores de ser apartados, adoptar una postura permisiva para cometer errores y aceptar que los resultados nulos deben formar parte del registro científico. Esto es difícil porque la cultura en la academia es competitiva. Los incentivos están en la publicación de resultados novedosos y positivos en revistas con impacto alto. Un cambio en las políticas de las revistas, tales como la apertura y transparencia se refuerzan, pueden facilitar una actitud de cambio en los investigadores individuales.

采用注册报告(Registered Reports)是《欧洲创伤心理学期刊》推动心理创伤研究领域开放科学实践的重要一步。然而，采用这些实践需要我们作为个体研究人员从根本上改变我们的观点。我们需要把担心放在一边，对犯错误采取宽容的立场，并接受无效结果应成为科学记录的一部分。这很困难，因为学术界的竞争文化。激励措施是在高影响力期刊上发表新颖而正向的成果。期刊政策的变化，如开放性和透明度得到加强，可以促进个体研究人员的态度变化。

KEYWORDS:

• Registered reports
• open science
• replication crisis

PALABRAS CLAVES:

• Reportes Registrados
• ciencia abierta
• Crisis de Replicación

关键词:

• 注册报告
• 开放科学
• 复制危机

Recently, the European Journal of Psychotraumatology (EJPT) introduced Registered Reports (RRs) (Spiller & Olff, 2018). This is an important step towards mitigating the problems that have become apparent in psychology in recent years. One issue is that the literature contains an abundance of positive results, whereas null-results are scarce (i.e. publication bias). Another problem is that the replicability of empirical findings is not as straightforward as it should be. This may be due to researchers’ (inadvertent) use of Questionable Research Practices (QRPs; e.g. Wicherts et al., 2016). QRPs are strategies that increase the probability of false positive results, for example stopping the data collection prematurely because of a statistically significant result. We have no reason to believe that the field of psychotraumatology will be immune to these problems.

RRs provide a solution that has received increasing attention. Authors submit the background and methods of their study for review before data-collection. Only after in principle acceptance of the manuscript, the study is carried out. A second review procedure follows for the complete manuscript. If the reviewers are satisfied that the plan is carried out as intended and that the results are interpreted properly, the manuscript is published regardless of the results. Thus, the acceptance of a manuscript depends on the specificity and quality of the analysis plan rather than on the statistical significance or the newsworthiness of the outcomes. As such, RRs will safeguard against QRPs and publication bias, thus enhancing replicability.

1. Changing individual perspective

Generally there has been a call for openness and transparency in psychological research, and the number of journals offering RRs is growing (see https://cos.io/rr/). Submitting RRs means adopting research practices that clearly differ from the way research was conducted and published before. For us as individual researchers, RRs mean planning research and analyses with maximized methodological and statistical rigor. This will prevent (inadvertent) QRPs, but may seem challenging to some. For example, we may have to recruit larger sample sizes or adopt analytical techniques that we are unfamiliar with (e.g. Bayesian statistics). One solution is to collaborate in multi-site teams and with applied statisticians (Dahy, 2019).

Importantly, adopting open research practices means that we should change our attitude towards transparency fundamentally. Traditionally, we have been protective of our (new) projects for fear of being scooped. We regard it as devastating if someone else publishes about our ideas before we do (see for example Powell, 2006). Openness can also include storing our data publicly. We might be reluctant to do so for fear of exposing mistakes or inadvertent QRPs that yielded false positives.

2. Changing research culture

For a large part such fears are fueled by the competitive environment we work in. The current culture in science is characterized by rewarding the frequent publication of positive and innovative results. High impact publications lie at the root of successful fundraising, prestigious academic positions and even entire careers. In this culture, trading a focus on individual success for transparency and sharing will be extremely difficult.

The scientific system, that is, the whole of researchers, editors, funders, and relevant institutions, has a limited capacity for publishing and even processing (i.e. reading, using, building on) all research findings. Attention for the most relevant publications in this system enhances the reputation of individual researchers and this, in turn, translates into material advantages, such as prestigious academic positions and grants. Careers are shaped by this mechanism, and researchers who lag behind drop out. Rewarding quantity and impact in such a system aggravates competitiveness. Fear is fostered by the unpredictability of statistical significance, and as null results are less likely to yield publications, they can threaten job positions. In this culture, inadvertently resorting to QRPs that generate positive and publishable results is conceivable. Therefore, a prerequisite for good scientific practice is providing attention, reputation and material goods for a priori high quality study designs instead of significant findings. This will make QRPs inherently useless.

3. Journals can enable change

Journals can help because they are a major force in reinforcing research practices. Initiatives such as adopting RRs at EJPT can promote individual change in several ways. To begin with, incentivizing transparency will facilitate alternative perceived norms. Individual researchers in psychotraumatology are now explicitly supported by the EJPT to conduct open science. Normalizing transparency in research practices enhances the likelihood of an attitude change even in researchers who might be skeptical currently. In addition, the in principle acceptance of plans before data are collected will increase the number of publications with null-findings. This way, journals may help to increase our tolerance of negative results. All too often we feel that our study has failed when the results are not in line with our hypotheses. However, we need to realise that if we are dealing with probabilities, null-results are to be expected. RRs will help us to regard null-findings as an essential part of the scientific record. Moreover, the implementation of a thorough review process for RRs likely increases the quality of psychotraumatology research. However, transparency will go hand in hand with an enhanced probability of detecting mistakes. Adopting a permissive attitude towards mistakes is daunting, but unavoidable to progress towards ‘an efficient, progressive and ultimately self-correcting scientific system’ (citing Hardwicke et al., 2018).

We hope that the new trend will bring along further developments. As Spiller and Olff (2018) suggest, there are many solutions for the replication crisis of which RRs are one. Journal policies could extend to enforce the sharing of materials and data in publicly accessible repositories for regular submissions as well.

4. Resonance in researchers

We are looking forward to follow up how the registered reports initiative of EJPT will be received by psychotraumatology researchers. We applaud all pioneers who will be submitting a RR. For those who are interested in giving it a try, publications that provide a checklist for the preregistration of a study plan (Wicherts et al., 2016) might be helpful. We are eagerly awaiting to see the number of RRs growing, and we are confident that specific and precise study plans in the RRs will enhance the overall quality of research in psychotraumatology.

Disclosure statement

No potential conflict of interest was reported by the authors.