Pre-registration of audiology research studies: are actions following good intentions?

Dear Editor

It is now widely accepted that much of the body of evidence in a given field of research suffers from a lack of replication (Ioannidis 2016). Areas of research can be led down unfruitful avenues by studies purporting to report novel, interesting, findings but are in fact based on spurious, non-replicable data. This leads to a waste of scientific resource. One contribution to this replication crisis is questionable research practices, or researcher degrees of freedom. It is where, often through non-nefarious intentions, the decisions taken by a researcher post-data-collection result bias the results and inferences. The pre-registration of empirical studies seeks to reduce researcher bias and add transparency to scientific endeavour; ultimately leading to more reliable research (Munro and Prendergast 2019). The aim of this letter is to provide a snapshot of descriptive statistics regarding the extent to which the auditory research community are pre-registering their research studies. Data indicate that audiology performs at least as well (or perhaps equally poorly) as related fields of ophthalmology and neuroscience. The number of pre-registrations in audiology is trending upwards year-on-year, though as this is from a very low starting point there remains much work to be done. Recommendations are suggested.

Method

We used two complimentary approaches to identifying pre-registered studies, with the assumption that they should converge on a detailed snapshot of the current state-of-play.

1. First, journals were identified from the fields of audiology, ophthalmology and neuroscience and the number of pre-registered published studies was calculated.

2. Second, known pre-registration databases were searched for key words related to the fields of audiology, ophthalmology and neuroscience.

The reason for choosing two complimentary approaches is because it is feasible that a field of study could have markedly increased the number of pre-registrations, but many of these may end up not making it to publication, possibly due to the confirmation bias which exists in the published literature and the propensity to prioritise significant findings over non-significant ones (de Vries et al. 2018). Furthermore, a researcher who decides to move to pre-registering their studies will have a time-lag in which they are writing up work which pre-dates their change in ethos and this change would be seen in repositories before journals. The search window was January 2018 to July 2022 and items included those published and those available ahead of print.

All design choices were made a priori and the study was pre-registered on the Open Science Framework (https://osf.io/g5cws/). One deviation from the registered protocol is that we did specify we would go back 10 years. However, on conducting the data search, it soon became apparent there were unlikely to be many pre-registrations from more than 5 years ago.

Journal selection

To identify journals related to the field of hearing science, the following terms were used as keywords on the Web of Science platform; “audiology and speech-language pathology” and “otorhinolaryngology”. The 10 journals with the highest current impact factor (as given by the Journal Citation Reports Database for 2021, the most recent available at the time) were used to identify the relevant journals. Within these journals, the text of all articles was searched electronically for the terms “registered protocol”, “pre-registration”, “registered” and “protocol”. Any papers containing these search terms were then checked to ensure the study had been pre-registered. The total number of papers published in the defined window in these journals was recorded.

For the fields of ophthalmology, the search terms for identifying journals were “vision” and “ophthalmology”. For the field of neuroscience, the search terms were “neuroscience” and “psychology”.

Searching pre-registration repositories

The Open Science Framework, Clinical Trials, PROSPERO and ISRCTN were searched. For the fields of audiology, ophthalmology and neuroscience, the following search terms were used, respectively; “audiology or hearing”, “vision or ophthalmology”, “neuroscience or psychology”. With the “or” acting as a Boolean operator in each instance. In order to gain an estimate of the total number of papers published in these areas, the same search terms were entered into PubMed in order to gain an estimate of the total papers published in this time window containing these search terms.

Results

Journal search

The 10 audiology and speech-pathology journals identified had a total of 6385 publications (including those published ahead of print) in the time window of interest; 74 (1.15%) of which were pre-registered. The comparable pre-registration rates for the fields of ophthalmology and neuroscience were 1.46% (210 out of 14,321) and 0.65% (75 out of 11,383), respectively. Figure 1 shows the pre-registration rate for each of the three disciplines broken down by year. Inspection of Figure 1 suggest that all three research fields are slowly improving in the number of published articles, which were pre-registered, with the uptake most rapid for ophthalmology. Analyses indicated that an audiology published article was ∼75% more likely to be pre-registered than a published article in neurosciences (odds ratio = 1.77; 95% confidence interval 1.2–2.4). An ophthalmology research article was more than twice as likely to be pre-registered than a neuroscience article (odds ratio = 2.21; 95% confidence interval 1.7–2.9). There was no reliable difference in likelihood between audiology and ophthalmology (odds ratio = 12.5; 95% confidence interval = 0.96–1.63).

Figure 1. The percentage of published papers which are pre-registered in the 10 identified journals for each field, expressed as a function of time.

Database search

Pre-registration rates of 1.5% (623 out of 41,287) for audiology, 0.61% for neuroscience (3,879 out of 633,170) and 0.66% (1,015 out of 152,826) for ophthalmology were found using this approach. However, this search included results from clinical trials and also review articles pre-registered on PROSPERO. It is established practice that review protocols should be registered and certainly any later-phase clinical trial should be. There was heterogeneity in the ratios of clinical trial and PROSPERO registrations compared to those on the Open Science Framework across the three disciplines, with 42% of audiology pre-registrations (260 out of 623) being identified as a clinical trial or review, compared to 15% for neuroscience (580 out of 3,879) and 52% for ophthalmology (527 out of 1015). This difference across the fields was unforeseen and so the following descriptive statistics are a deviation from the planned analysis and represent exploratory results.

If the Open Science Framework is assumed to represent an estimate of pre-registrations which are basic, translational or applied scientific empirical studies, rather than clinical trials or review articles, then the rates of pre-registration are 0.6% (249 out of 41,287) for audiology and 0.48% (3007 out of 633,170) and 0.13% (196 out of 152,826) for neuroscience and ophthalmology, respectively. Figure 2 shows the year-on-year trends for pre-registrations identified by searching repositories, both including and excluding clinical trials and review articles.

Figure 2. The percentage of pre-registrations in each field as estimated by searching different repositories. The upper panel includes Clinical Trials Database, ISRCTN, PROSPERO and the OSF. The lower panel includes only OSF pre-registrations. The lower panel is likely the most accurate for comparing across disciplines, as there were clear differences in the distributions of clinical trials and reviews across the three disciplines, which will be a factor in the upper panel data.

Discussion

The rate of pre-registrations in the field of audiology is at least equal to the fields of ophthamology and neuroscience. For all three fields, the year-on-year trend shows an uptick in the rate of study pre-registrations, although in ophthalmology the issue is clouded somewhat by the large number of clinical trials and review articles which make up the majority of pre-registrations. Though different journals could be selected, and different approaches used for searching these journals, the data presented provide a snapshot of current practices.

In this letter, we have chosen to keep the specific journals anonymous. It is perhaps time that journals more explicitly acknowledge metrics such as the rate of study pre-registrations in their pages and publish these alongside the more heralded impact factor. Furthermore, any future mining of statistics such as these could remove the veil of anonymity that we have used here. The hope is that these upward trends continue towards a meaningful percentage of studies being pre-registered, and those journals that eschew this trend should perhaps be named?

As a rough snapshot of the current appetite for pre-registering studies, it is encouraging to see that audiology is on a par with the two other disciplines described here. It is also encouraging that, in both Figure 1 and the lower panel of Figure 2, the rate of pre-registration is typically increasing across all three disciplines. The downside is that the absolute numbers involved are small.

One final important comment to make, which can sometimes be overlooked when discussing the advantages of open science practices, is that pre-registering a study and having transparent research practices does not guarantee a good quality study. There can be fundamental flaws which were either in the pre-registration or overlooked. A pre-registered study should not lead to less scrutiny at the review stage. But a good quality empirical research study should go hand-in-hand with a good quality pre-registration. There is a way round this, the Registered Report route, which some journals offer. This turns peer-review into a two-stage process; the pre-registration is peer-reviewed and once accepted, the journal provides an ‘in principle’ acceptance, which means the paper is published so long as the pre-registration is followed. Registered reports have been around since 2012, but currently it is estimated that although over 300 journals offer this route now, only 591 stage-two reports have been published (Chambers and Tzavella 2022). Therefore, until this two-stage peer-review becomes the norm, there is a need to continue to discuss the value of pre-registration and the role both researchers and journals have in ensuring this becomes accepted, standard practice, rather than the exception.

Recommendations

1. Journals to provide pre-registration statistics on their webpages, which will keep the issue of replicability and researcher bias front and centre.

2. Journals could adopt a “badge” system, like Ear & Hearing, which makes it easy to identify papers which have ascribed to open science practices.

3. This study investigated the number of pre-registrations only. A future consideration will be how accurately studies conform to the pre-registration and how explicitly any deviations are noted (see Claesen et al., 2021) for an example from the Psychological Sciences). Reviewers should also be encouraged to look at how well the reported study concords with the pre-registration.

4. Authors should be encouraged to explicitly note any deviations from the protocol, or include a statement confirming that there were no deviations. This will make the review process more efficient and also provide it an extra layer of scrutiny.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The authors G.P. and K.J.M. are supported by the NIHR Manchester Biomedical Research Centre.