‘Negative’ Clinical Trials in Rare Diseases and Beyond: Reclassification and Potential Solutions

Abstract

Currently, all clinical trials, including those in rare diseases, that do not demonstrate a statistically significant benefit (i.e., p < 0.05) on the primary end point are classified as ‘negative’. This classification does not take into account a myriad of factors, including whether the trial was sufficiently powered, or had other statistically significant end points, or was terminated before completion, or was even initiated. In this paper, I propose reclassification of these trials into five categories: true negative, underpowered, inadequate, terminated or uninitiated, and justifiable. These categories reflect the trial characteristics more accurately and will be more useful to all stakeholders, especially the patients who participated in the trial and their healthcare providers.

Lay abstract

Currently, categorizing clinical trials as ‘negative’ is based solely on the main outcome of the trial without taking into account whether there are other outcomes that could be positive for patients or other reasons for this result. This issue is particularly important in rare diseases where there are so few treatments available. Inaccurately labeling a trial as ‘negative’ or requiring patients and caregivers to delve into the technicalities to understand the labeling adds to their burden. In this paper, I propose reclassifying ‘negative’ trials into five new categories (true negative, underpowered, inadequate, terminated or uninitiated, and justifiable), each of which more accurately describes the outcomes. This should help patients better understand and navigate the realm of clinical trials.

Keywords::

• rare diseases
• reclassification
• solutions
• trials
• ‘Negative’

What is a ‘negative’ trial? The term itself is very poorly defined and has an obvious undesirable connotation. It raises the specter of failure and suggests that the treatment being evaluated is not effective for that disease. It does not take into consideration the various caveats that may accompany the trial. Indeed, it is an all-encompassing term that is absolute. When so applied, or rather misapplied, it does a disservice to trial participants, other patients with that disease, their caregivers, their healthcare providers, the trial investigators and the sponsors. In short, a ‘negative’ trial impacts everyone who is engaged in finding new and better treatments for that particular disease. So, who benefits from using this terminology? Essentially, only the academic purists who characterize the results in preconceived binary pigeonholes with little to no consideration for the context of the trial or the impact on patients and caregivers.

This is particularly problematic in rare diseases [1,2], which are defined as <200,000 cases nationwide in the USA [3] or four to five cases per 10,000 people in the European Union and the rest of the world [4,5], with approximately 50% affecting children [6]. Thus, in the rare disease arena there are very few patients by definition. The problem is exacerbated by the fact that patients with rare diseases are widespread all over the globe [2]. Most physicians are trained to recognize approximately 400 common ailments [7] and most likely will not recognize a rare disease when presented with a case. Some, even specialists, may treat only one or two patients with rare diseases over their whole careers [8]. These shortcomings are additional hurdles that contribute to the generally poor understanding of the pathophysiology and natural history of rare diseases [9], making the challenges to diagnosing rare diseases more severe. Furthermore, when one overlays the strict inclusion and exclusion criteria required in traditional clinical trial designs, the potential pool of patients available for a trial shrinks drastically [1,10]. So, are trials in rare diseases really ‘negative’ or do the facts warrant new lexis? I believe that it is important to reexamine the use of ‘negative’ as it pertains to clinical trials in rare diseases, provide appropriate context, and perhaps use other terminology when ‘negative’ does not accurately communicate the outcomes of the trial.

In this brief article, I will describe the term ‘negative’ as it is currently applied to clinical trials and offer a suggestion for reclassification of the trials that are currently lumped together under this broad banner. I will discuss some solutions to overcome the issues for each category and provide examples where possible. Finally, I will touch upon the utility of publishing these trials under the new categories.

Current application of the term ‘negative trials’

Currently, there is no generally accepted definition of ‘negative’ trials. This term is used to include all trials which did not meet the primary end point, regardless of the context or caveats. In other words, all trials in which the primary end point did not achieve a p-value of < 0.05. Thus, trials that were terminated for lack of sufficient or any enrollment, were poorly designed or insufficiently powered, etc. are all lumped together under this banner along with trials that truly are negative. This is clearly an overly simplistic assessment of clinical trials and classification into ‘positive’ and ‘negative’ categories [11–13]. It may also potentially be a misapplication of the statistical analyses performed [11,14].

Suggested recategorization of ‘negative’ trials

To address the issue of the overly broad use of the term ‘negative’ trials, I propose five categories described in detail below and summarized in the Table 1. In each case, I have provided potential solutions to the problems so that trials in each category can be reclassified as ‘completed’ with a clinically relevant outcome.

Table 1. Summary of proposed reclassification of ‘negative’ trials.

Category		Trial characteristics
1	Negative	Well-designed, well-conducted trial with adequate enrollment, but no benefits of treatment in any end point
2	Underpowered	Well-conducted trial, but underpowered to determine whether or not there was any benefit of treatment under investigation
3	Inadequate	Poorly designed trial that cannot determine whether or not there was any benefit of treatment under investigation
4	Terminated or uninitiated	Trials that were terminated or not initiated for any reason
5	Justifiable	Well-designed, well-conducted trial with adequate enrollment, without treatment benefit in primary efficacy end point, but with benefit of treatment in at least one other end point

Category 1: true negative trials

These are trials that are well designed with control arm(s) and outcomes that are appropriate for the study population. The intervention under study, the dose (if applicable) and treatment regimen are appropriate for the disease and the patient population. These trials have enrolled sufficient numbers of patients in each arm to provide adequate power even after attrition for various reasons to determine whether or not there is any treatment effect. In short, these trials were well designed and conducted in accordance with good clinical practices. In other words, there were no confounding issues or caveats with respect to trial design, conduct, or statistical analysis. However, despite such efforts, these trials yielded no beneficial effect of the treatment in any end point measured. This situation is unusual, although not absent, among rare diseases, where patient recruitment and retention for a traditional clinical trial is a huge challenge [2,9]. Nevertheless, when such is the result, these trials should continue to be termed ‘negative’.

Category 2: underpowered trials

Trials that fall in this category are those in which the end point is relevant to the study population and it is being measured appropriately but the trials have not enrolled enough patients to answer whether or not the treatment under evaluation had a clinically meaningful effect on the outcome being measured. When a trial is underpowered (i.e., has enrolled too few patients), the resulting outcomes can mask a significant treatment effect even when it is real – a type 2 error, statistically speaking. Such trials are at best inconclusive or uninterpretable, and not necessarily ‘negative’. These trials should be reclassified as ‘underpowered’ to describe them more accurately.

Unfortunately, a substantial number of trials, regardless of the prevalence of the disease being treated, fall into this category [15–18]. Even when there is no dearth of patients, conduct of such trials can be considered unethical [10]. When there are extremely few patients, available, as in rare diseases, conduct of such trials can be unconscionable, barring an exception listed below. Indeed, in my humble opinion, regulatory agencies should not permit such trials. An exception can be made for trials in rare diseases where there are very few patients and even fewer that meet the strict inclusion and exclusion criteria for a traditionally designed trial [1,10]. The caveats for this exception are the trial inclusion/exclusion criteria are similar enough to other trials for this disease, and the sponsor agrees to make the trial data available for combination with other trials for a meta-analysis. When trialists, sponsors, and patients collaborate, previously inconclusive trials could be combined to provide sufficient power to the meta-analysis for determining the utility of the treatment in question – an outcome that would be beneficial to all stakeholders, especially the patients who participated in the individual trials.

These conditions need to be considered from the outset at the trial design phase and adhered to subsequently to be respectful of participating patients. While not easily done, it is not impossible [19]. However, it is not unusual for trials in rare diseases, for example, Niemann-Pick disease, to be very disparate, making it difficult to perform a meaningful meta-analysis [20]. One option is to use adaptive trial design [21–23]. In this case, interim analyses allow for a change in the number of patients needed without affecting the validity of the analysis on the efficacy or safety of the treatment being evaluated. A major advantage of this design is that it allows for adding more patients during the ongoing trial to prevent underpowering and ensuring its success.

Unfortunately, many underpowered trials in rare diseases either are not completed or not reported even when completed [24]. This is particularly harmful not only to the participating patients, but also to other patients with this disease. Furthermore, it erodes the confidence of patients in the clinical trial process, making it more difficult to recruit patients for subsequent clinical trials. Given how closely knit the rare disease community is, such information can be shared quickly and recruiting problems can spill over to trials in other diseases. It is imperative, therefore, especially in rare diseases, that trials are conducted with the highest ethical standards and respecting the critical contributions of the participating patients.

Category 3: inadequate design trials

This category includes trials that have a design defect, often identified after the trial is completed, that could potentially result in a lack of statistically significant outcome in the primary end point. About 50% of all rare diseases occur in children. Unlike in diseases occurring in adults, it is not ethical to conduct trials with the usual process of determining toxicity, pharmacokinetics, dosing, time to response, etc. in children. It is also not meaningful to conduct these trials in adults as the relevance of the results to diseased children is undeterminable. Consequently, many of these trials make a best estimate effort to design a trial that should be adequate based on all the available evidence, given the assumptions and caveats. Unfortunately, sometimes these assumptions and caveats do not hold up as anticipated after the trial is completed. Study design defects include, but are not limited to, the primary outcome not being appropriate or accurately defined for the study population, the study population not being appropriate or sufficiently well defined for the intervention being evaluated, and the treatment regimen not being appropriate (e.g., based on wrong dosage calculation, the trial length being too short). The hallmark of this category is that the design defect is such that no clinically meaningful data can be derived despite post hoc efforts to adjust for the defect. Reclassifying these trials as ‘inadequate’ would more accurately describe them.

In several rare diseases like pulmonary arterial hypertension and Duchenne muscular dystrophy, the traditional primary efficacy end point is the 6-min walk distance [25–27]. However, this makes little sense when the patient is already in a wheelchair [28]. Yet, this end point continues to be used as the primary outcome, practically ensuring that the trial will fail and be characterized as ‘negative’. However, other end points that are relevant to the patient population are available to evaluate the safety and efficacy of the treatment being tested [29]. A relatively simple change in trial design of making the relevant outcomes co-primary end points would suffice to remove these trials from the ‘negative’ category.

It is also difficult to predict how long a trial should run, especially when evaluating the impact of a drug on slowing disease progression in a rare disease with poor or absent natural history data. For example, treatment with adrabetadex for 1 year did not show any benefit over placebo in slowing disease progression in patients with Niemann-Pick C1 [30]. However, treatment for longer periods seemed to slow disease progression in some patients [31,32], suggesting that a longer trial may have yielded a different outcome. Indeed, this was the request of the US FDA [33]. In this case too, a relatively small change would suffice to potentially reclassify the trial.

While the suggested changes are administratively simple, it must be noted that recruitment and retention of patients for trials in rare diseases is still a major hurdle [2,9]. Thus, conducting a trial all over again with an improved design with no guarantee of definite outcomes is a daunting challenge for patients and sponsors alike.

Category 4: terminated & uninitiated trials

Trials in this category are classified as terminated prior to completion or not ever started, despite being registered, and should be classified as such and not under ‘negative’. Reasons for termination or noninitiation include, but are not limited to, unanticipated safety concerns about the treatment under study arising during the trial, lack of an efficacy signal, insufficient or very slow enrollment, loss of a trial center affecting the integrity of the trial (e.g., due to a natural disaster) and loss of sponsorship. In each case, the trial should be labeled ‘terminated’ or ‘not initiated’ with the reason provided in parentheses (e.g., terminated [safety]).

As stated earlier, a number of trials in rare diseases are terminated early, usually due to a lack of recruitment and/or retention of patients [24]. Many of these trials employ the traditional clinical trial design that requires, at a minimum, treatment and control arms with a sufficient number of patients in each to yield a statistically significant difference (p < 0.05 or better) between the two arms. As well intentioned as the trialists and sponsors may be, this design is not always feasible in rare diseases where there are very few patients eligible for the trial. Thus, it is imperative that alternate trial designs be considered so that the trial yields a clinically meaningful outcome with extremely limited numbers of patients. One option for rare diseases is crossover trials in which each patient serves as their own control and can help demonstrate the impact of treatment relative to the control [23,34,35]. The disadvantage of crossover trials is that because treatments are sequential for each individual patient, they may take longer than a traditional trial with parallel treatment arms. Another option is the use of enrichment trial design [21]. This design identifies patients who benefit most through a multi-stage approach. The initial inclusion and exclusion criteria are intentionally broad to allow as many patients as possible to be enrolled. Each stage serves as a screening process to narrow the criteria for inclusion to identify the best responders, such as using biomarkers. The disadvantage of this design is that it is unclear how effective the treatment would be to the general population of patients with that disease. A third option is the use of adaptive trial design, the benefits of which were discussed earlier [21–23].

Category 5: justifiable trials

The key characteristic of trials in this category is that despite the primary outcome not achieving statistical significance, there is at least one secondary or post-hoc outcome that is clinically beneficial to the patient population being studied. The ‘negativity’ of such trials is limited to the primary outcome. Thus, characterizing the whole trial as ‘negative’ is not scientifically or medically accurate nor is it fair to the patients with the disease, their caregivers, the healthcare professionals treating them, the physicians conducting the trial, or the sponsors. The occurrence of a clinically beneficial outcome in the trial demonstrates that the treatment has value. As such, these should be reclassified as ‘justifiable’ trials, which would be a more accurate description than ‘negative’.

In trials on rare diseases, where participants are at a premium, the use of a composite primary outcome is particularly attractive as it increases the number of events, thereby potentially increasing the opportunity to differentiate the treatment under study from the control. However, it must be noted that using a composite end point does not necessarily increase the statistical power [11]. In a recently published Phase III trial in patients with familial adenomatous polyposis, a rare disease for which currently there is no approved pharmacopreventive treatment, there was no statistical difference between the combination arm and the monotherapy control arms in the composite primary end point [36]. On closer examination, the study revealed that the composite end point included a subjective component that confounded the outcome. This re-evaluation also revealed that compared with either monotherapy, combination therapy significantly reduced the risk of disease progression among patients with at least a partially intact lower GI tract [37]. Thus, although the treatment may not have been beneficial for all patients with the disease, it was very beneficial for a substantial subpopulation. Unfortunately, the journal editors chose to disregard the beneficial analyses in favor of limiting the publication to what was prespecified in the trial protocol. While their decision was consistent with the official documentation regarding the trial, I do not understand why an outcome, albeit based on a post hoc analysis, that would be beneficial to many patients would not also be included, especially when this journal published an article cited herein about the various issues that must be considered regarding such trials [11]. This case exemplifies how a trial deemed ‘negative’ based solely on the primary efficacy end point not achieving statistical significance may result in a publication bias that does not disclose the benefit to patients having this rare disease.

Publishing ‘negative’ trials

Registration of clinical trials in a publicly accessible database has been recognized for a long time as an effective way to diminish bias in reporting results from these trials [38]. In 1997, The Food and Drug Administration Modernization Act (FDAMA) mandated registration of trials involving investigational new drugs for serious or life-threatening diseases on a public database [39]. This led to the establishment in 2000 of ClinicalTrials.gov operated by the National Libraries of Medicine at the National Institutes of Health [40]. In 2005, the International Committee of Medical Journal Editors (ICMJE) issued a policy statement that helped increase trial registrations [41]. However, neither the law nor the ICMJE policy statement mandated reporting the results of the registered trials. Title VIII of the Food and Drug Administration Amendments Act, enacted in 2007, broadened the scope of FDAMA to include registration of all trials, except those in Phase I, involving any interventions (drugs or devices) that fall under the aegis of the FDA [42]. Furthermore, this legislation also required reporting on data no later than 1 year after completion of the study [43–45]. Failure to do so, without appropriate permissions, could result in severe penalties, such as withholding of federal research funding and up to $10,000 per day [43]. In 2012, the European Commission finalized its rules on registering clinical trials and reporting on their outcomes followed by the recommendations by the World Medical Association [46,47]. Although every single one of these regulations, policy statements and recommendations address registering and reporting clinical trials, none of them provide any guidance on categorizing trials as ‘positive’ or ‘negative’. That labeling was formed over time through usage by multiple stakeholders involved in the conduct and reporting of clinical trials and those who review the submitted manuscripts for journals.

Historically, there exists a publication bias toward ‘positive’ trials regardless of the prevalence of the disease [13,48,49]. However, as others have stated, it is important to publish data from ‘negative’ trials for a variety of reasons [13,49–52]. It is even more critical to publish data from ‘negative’ trials conducted in rare diseases. In the absence of peer-reviewed publications on ‘negative’ trials in rare diseases, other researchers and sponsors may attempt to conduct trials of similar design for the same disease using the same class of drugs, which would inevitably result in the same ‘negative’ outcome. This would be an unconscionable waste of patients’ lives and an unnecessary waste of limited resources. However, publication of these trials with the proposed reclassification affords all stakeholders to make proactive informed decisions about trial design and participation, thereby improving efficiency and being respectful of patient lives.

Conclusion

Conducting clinical trials in rare diseases remains a daunting challenge. It is particularly problematic in diseases with little, if any, natural history data. This situation results in many trials on rare diseases being categorized as negative solely on their not achieving statistically significant benefit on the primary end point – an overly simplistic view that disregards the complexities of conducting trials in rare diseases. In many cases, this categorization is inaccurate and unwarranted. In some cases, such a characterization can withhold lifesaving treatments from patients. To address these issues, I have offered a reclassification of these trials that better describes their characteristics and, consequently, is more informative to all stakeholders. This reclassification may also provide guidance to patients, trialists, and sponsors on how to better interpret their data in a more nuanced way than the overly simplified binary ‘positive’ and ‘negative’ categories currently employed. I have also suggested ways for trials in each new category to offset the challenges of their new classification through proactive trial design and implementation and/or post-hoc analyses. Most importantly, I have suggested solutions, including contextual publication of these trials, that would be respectful of all stakeholders, especially the patients participating in these trials. I believe that this is an important step toward changing the landscape of rare disease trials toward a more positive outlook.

Future perspective

The proposed reclassification described herein (true negative, underpowered, inadequate, terminated or uninitiated and justifiable) will help all stakeholders, especially patients participating in clinical trials, to better understand the outcomes of the trial. This reclassification provides a more accurate perspective of the trial outcomes by placing them in context. This contextual reclassification provides a basis for all stakeholders to discuss the outcomes more objectively. Finally, this reclassification and appropriate publication may also help sponsors and trialists proactively address the limitations of their trials by providing context for submissions to regulatory authorities, allowing for informed decisions to be made.

Executive summary

Current application of the term ‘negative trials’

• Currently, all trials, including those in rare diseases, that do not demonstrate a statistically significant benefit are classified as ‘negative’.

Suggested recategorization of ‘negative’ trials

• This classification does not take into account a myriad of factors discussed in this article.

• I propose reclassification of these ‘negative’ trials into five categories: true negative trials, underpowered, inadequate, terminated or uninitiated, and justifiable.

• These categories reflect the trial characteristics more accurately and publication with these new categories will be more useful to all stakeholders, especially the patients who participated in the trial and their healthcare providers.

Financial & competing interest disclosure

The author is an employee of rareLife solutions. The author has no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

No writing assistance was utilized in the production of this manuscript.