Enhancing quality in preclinical data: Of hot science and cool quality

There is a broad range of decisions that are made based on scientific research – from the development of novel therapies to environmental health and safety policies. Occasionally, the need to make such decisions becomes acute and captures the public attention to the extent that it starts to exert extreme pressure on scientists with the risk of biasing the research outcomes. These d\ays it is hard to find anyone who is not aware of or was not involved in discussions related to Covid-19 or the safety of modern 5 G cell phone technologies.

As a witness to the time pressure to find an effective treatment strategy, a huge number of publications related to Covid-19 have appeared since the beginning of the pandemic in early 2020. As of 16 June 2020, over 22,700 articles about Covid-19 can be found on PubMed and more than 2,200 Covid-19-related studies are registered on clinicaltrials.gov.

In his landmark 2005 paper entitled “Why Most Published Research Findings Are False” [1], John Ioannidis has presented an analytic framework that yielded a prediction that, at the first glance, sounds paradoxical: “The hotter a scientific field (with more scientific teams involved), the less likely the research findings are to be true As an explanation for this statement he wrote: “With many teams working on the same field and with massive experimental data being produced, timing is of the essence in beating competition. Thus, each team may prioritize on pursuing and disseminating its most impressive ‘positive’ results

Indeed, the heated discussion around the use of hydroxychloroquine, an anti-malaria drug, for the treatment of Covid-19, is happening largely because there was no time to carefully review the evidence and run confirmatory studies before the topic was picked up by policy makers. Interestingly, it turns out that claims not only of therapeutic utility but also of safety risks of hydroxychloroquine [2] are not supported by robust evidence.

Once there is a pandemic like Covid-19 and an acute need to find a treatment, it is no longer possible to avoid or reduce the pressure. However, what is still possible, though, is to prevent adverse effects of such pressure and to make sure that the urgently needed research data are of quality that can support.

Drug discovery and development is constantly under pressure to deliver innovative therapies, and both industry and academic have long recognized the need for establishing a systematic approach to ensure quality of data used in (e.g. when the evidence is robust enough to advance a novel molecule into clinical trials). Such a system has recently been developed by the EQIPD consortium (Enhancing Quality in Preclinical Data; eqipd.org) in a close collaboration with a large group of associated collaborators, and stakeholders representing research institutions, publishers, funders, learned societies, and professional societies, from nearly 100 organizations in Europe and North America.

The EQIPD Quality System was designed to support scientists in preventing and coping with the potential risks of bias while being lean – without having any negative impact on the freedom of scientific exploration [3]. Equipped with such a quality system, the research community will be in a better position to tell when and when not available data can serve as evidence for supporting.

For example, does the exposure to radio frequency radiation (RFR) from the use of current and emerging cell phone technology cause harmful health effects? This is a simple but very relevant question for all mobile phone users that read the news about the new technology standard 5 G being deployed and a hot public debate around the safety of this technology. One may expect to find an answer to this question based on high-quality evidence that enabled market access for this very important technology. However, in stark contrast to the area of drug development where risk mitigation strategies are enforced by governmental agencies regulating market access, current safety regulations in the field of environmental influences on human health are essentially engineering standards that do not take into account potential impact on human physiology other than short-term heating risks. This is quite concerning as there were hundreds of studies conducted to-date with the intention to study various biological effects of RFR but they have delivered inconclusive, often contradictory, results [4].

With multiple variables affecting outcomes of such studies (most importantly, exact RFR exposure level), lack of rigorous standardization may be seen as a solution to this problem of conflicting results. Indeed, does help to reduce variability within and between studies conducted by a given laboratory. However, standardization often comes at the cost of experimental tractability and biological generalizability of scientific observations [5]. And, particularly important for confidence in , standardization does not help with the internal validity of the research data.

A recent analysis [4] has suggested that, out of 81 reports on investigations of biological effects of whole-body RFR exposure, only one included information on the most basic set of four internal validity criteria such as randomization, blinding, sample size and inclusion/exclusion criteria [6]. Lack of this information in a report does not necessarily invalidate the presented dataset but prevents the use of the results as decision-enabling evidence.

Thus, one of the main objectives of quality systems such as the one developed by EQIPD is to support best practices in study design, conduct, and reporting [4]. When maximal rigor is applied, research efforts are certainly not wasted and can support .

Benjamin Franklin once said: “The bitterness of poor quality remains long after the sweetness of low price is forgotten.” To continue this thought, the coolness of good quality prevents hot science from evaporating (Figure 1). In practical terms, while we do not always appreciate that our data and publications may be used to justify a decision, enabling such use is nevertheless our duty. The quality system introduced by EQIPD makes such duty easier to fulfill.

Figure 1. Hypotheses may or may not be true and it is the rigorous testing of hypotheses that drives science forward. Excessive pressure burns resources often leading away from solid evidence. In contrast, good research practice helps to avoid waste and supports rational decision-making by confirming or rejecting the hypotheses

Additional information

Funding

This work has received funding from the Innovative Medicines Initiative 2 Joint Undertaking under grant agreement No 777364. This Joint Undertaking receives support from the European Union’s Horizon 2020 research and innovation programme and EFPIA.