Changing paradigms in detecting rare adverse drug reactions: from disproportionality analysis, old and new, to machine learning

PLAIN LANGUAGE SUMMARY

Your physician, pharmacist, nurse, or even you can voluntarily report suspected adverse events associated with drugs. The FDA Adverse Reporting System (FAERS) and the WHO Vigibase are large databases that store individual reports of adverse drug reactions (ADRs). While some ADRs are very common, others are seen rarely. Detecting rare and very rare ADRs is extremely difficult but very important for the safe use of drugs. Databases such as FAERS and WHO Vigibase contain a large amount of data and are commonly used for analysis applying a statistical method called disproportionately analysis. This type of analysis determines whether there is a higher-than-expected number of adverse reactions for a particular drug. In the future, machine learning will complement this process by applying algorithms to the data, constructing and refining rules of inference, and building predictive models of ADRs. This paradigm change in testing for ADRs is expected to provide a better understanding of the factors impacting drug safety.

KEYWORDS:

• Adverse event reporting
• disproportionality analysis
• electronic health records
• machine learning
• deep learning
• neural networks
• text analysis

1. Understanding the relative (un)safety of drug treatments

An important consideration in drug treatment is the potential for adverse drug reactions (ADRs) requiring clinicians to conduct a risk-benefit analysis based on individual patient characteristics and risk factors. Lazarou et al. [1] concluded that ADRs were between the fourth and sixth leading cause of death in the United States with an overall incidence of 6.7% for serious ADRs and 0.32% for fatal ADRs. Summarizing several studies (yet without dating them), Montastruc et al. [2] implicated ADRs in 5–10% of outpatient visits and in 5–10% of hospitalizations; as well as being observed in 25–30% of hospitalized patients either at admission or incident during the hospital stay secondary to in-hospital treatments. The French network of regional pharmacovigilance recently released the findings of IATROSTAT, a notably well-designed study. This prospective multicenter investigation in 141 short-stay specialist medical wards randomly selected from 69 public hospitals in metropolitan areas of France followed patients consecutively referred from ambulatory care and admitted for 14 consecutive days (that is, seriously ill patients) between April and July 2018 [3]. Importantly, all cases were adjudicated by an independent ADR review committee. Of the 3648 hospital admissions, 309 (8.5%, 95%CI = 7.6%-9.4%) of hospital admissions were ADR-related. Of these, 8.8% were for hemorrhagic events, 6.5% for hematological disorders, 6.3% for acute renal failure, 6.0% for fluid and electrolyte disorders, and 5.2% were falls-related. Antineoplastic agents were the most frequently involved drug class (15.1%), with targeted therapies accounting for 22.8% of this class total; followed by antithrombotics (11.6%) of which 43.6% were antiplatelet agents, 39.6% were vitamin K antagonists, and 22.5% were direct anticoagulants. Noteworthy also, incretinomimetics comprised 19.9% of ADRs related to antidiabetic agents, and (unsurprisingly) 76.7% of ADRs related to analgesics were for opioids. The mortality rate after one month of follow-up was 1.3% (95%CI = 0.5%-2.8%).

Knowledge of ADRs is derived through pre-marketing clinical trials and subsequent post-marketing surveillance. However, rare ADRs, occurring at an incidence of between 1/1,000 and 1/10,000 and very rare ADRs, an incidence of less than 1/10,000, may be exceedingly difficult to identify [4,5].

It is critical that when a patient presents with a rare symptom, a rare syndrome, or a rare disease, various hypotheses about the origins of these rare events can be explored. The decision processes involved in resolving a rare adverse drug reaction is complicated by the rarity itself. There may not be much in the literature; in the best case, some (deeply buried) case reports – perhaps with speculative biological plausibility but seldom a clear pathophysiological pathway.

The era of post-approval (industry-sponsored) prospective observational studies seems to be mostly over – except if market share and revenue are at stake. An alternative solution is to draw on pharmacovigilance databases, such as FAERS, the US FDA Adverse Event Reporting System [6], or Vigibase [7], the WHO global database of side effects of medicinal products. Both are voluntary reporting systems that rely on health professionals and consumers to report ADRs; if they are so inclined. While FAERS collects reports from individuals directly, Vigibase receives its data through the intermediary of WHO member countries.

In this issue of the Journal, Liu and colleagues [8] report on an FAERS analysis of dropped head syndrome (DHS) that identified 193 reported cases. Using disproportionality analysis, they found that this rare ADR was associated with several nervous system, antineoplastic, and immunomodulating agents. Their study elicits reflection on the value of voluntary reporting databases, past and current approaches to pharmacovigilance research on rare ADRs, and how applying advanced statistical algorithms to electronic health records data might advance pharmacovigilance research and yield better understandings of rare ADRs. The coming years will indeed see a change in paradigms for detecting and analyzing ADRs – from the rather common to the very rare – that will enrich pharmacovigilance analytics and our understanding of the clinical incidence and biological dynamics of ADRs.

2. Disproportionality analysis: from the old to the new(er)

For its time (the early 1980s), the methodology of disproportionality analysis was rather ingenious. To quote Faillie [9], its aim is to determine whether there is a ‘higher than expected number of adverse reaction reports with a specific drug’ and whether the reporting rate of an ADR is ‘disproportionate’ relative ‘to other reactions in the pharmacovigilance database.’ In other words: does the observed rate of an ‘ADR-of-interest’ exceed, and by how much, its expected rate. Expected rate refers to the rate when both the ADR and the drug-of-interest are not associated but have been observed concurrently (for instance: an adverse event observed in a clinical trial but not attributed to the drugs being evaluated? The frequentist statistics were commensurate with the time: construct a 2 × 2 table; calculate the expected cell values and the difference between observed and expected values (O/E estimate); and subject it to a χ² (chi-squared) test of independence (a case of the gamma distribution). In the process, estimate the proportional reporting ratio (PRR) and its confidence interval. Of note, the Uppsala Monitoring Center, which manages Vigibase on behalf of the WHO, has developed triage algorithms for the early discovery and identification of ADRs [10,11].

Liu et al. [8] adopted a later generation of Bayesian disproportionality methods in which estimates are optimized as additional information is considered. Essentially, does adjusting estimates for variables A, B, C, etc. help to better and more accurately estimate disproportionality? The authors employed the multi-item gamma Poisson shrinker (MPGS): the Poisson distribution is applied because of the rarity of DHS, extending the use of the gamma distribution and reducing the likelihood of false positive results by ‘shrinking’ the O/E estimates toward an average value.

3. Expert opinion: the promise of machine learning to understanding ADRs

Regardless of approach, disproportionality analysis still leaves several questions unanswered. How can we advance the field of drug safety analysis beyond voluntarily reported data? Can we go deeper than mere associations of a presumed antecedent (drug) and a presumed subsequent (ADR)? Are there data sources of great(er) diversity and deep(er) granularity that would enable going beyond the (gradients of) disproportionality that the currently prevailing methods permit? Can we link disproportionality results with the underlying pathogenesis of rare ADRs – to better understand the ADR and to better detect ADRs in clinical practice?

Applying statistical algorithms and rules to a database to generate statistical pharmacovigilance estimates can alert safety evaluators to probable safety issues, comprising real safety signals. Electronic health records (EHR) have increased significantly in the past decade with the progression of health information technology and the Meaningful Use directive [12] in the US.

EHRs can present a data channel for evaluating safety events. Yet, they are also limited by the data entered and the information included. Further, the absence of standardized documentation requirements may lead to incomplete charts that hinder ADR detection and monitoring and thus misrepresent the actual versus observed ADR rates [13], especially across various inpatient and outpatient settings, large and small. For instance, an ADR detection and analysis method that adjusts for demographic and clinical confounders can only be applied to systems collecting such information. Likewise, a database devoid of chemical structure and drug target data may impair efforts to bridge the molecular and clinical domains [14,15].

Many stakeholders in healthcare are interested in mining patterns from the massive amounts of data continuously being collected and stored in large transactional or relational data sets. In turn, this informs decision-making processes to preventing ADRs. Despite the high interest, the sheer complexity of the data and the limitations of standard statistical techniques are often outside the reach of conventional statistical methods. In contrast, machine learning algorithms and applications can handle large complex datasets by capitalizing on computers’ large processing and analysis capabilities to detect patterns and outcomes. These can then be presented to clinicians for expert input on how to align these with actual clinical reasoning and thus embed rules of inference in the analytics [16]. The performance of these algorithms improves as they are exposed to more data.

The public availability of chemical and biological knowledge bases such as the DrugBank [15] can bridge the gap between the molecular and the clinical in ADR research [14]. Knowledge on, for instance, protein binding sites, biological pathways of drug action and metabolism, linkages between chemical substructures and specific toxicities, and chemical structural similarities between drugs, can be used as leverage to better understand the molecular determinants of ADRs [15]. Additionally, predictive models using deep learning algorithms – a subset of machine learning in which multilayered neural networks learn from vast amounts of data with varying nonlinear and linear relationships in solving problems in pattern recognition, data analysis, and control [17] – can be deployed, thereby allowing a more proactive approach to identifying and understanding ADRs. This assumes that ADRs and their determinants are, discoverable, predictable, and consequential to certain molecular actors [15].

Healthcare data, particularly ADRs, are often coded in unstructured free text, which limits the use of conventional statistical techniques that require structured data. Natural language processing, a machine learning algorithm that cross-examines free text using computational techniques, can extract and produce information from unstructured data, including categorized incident reports and ADRs [4,14], informing us of what safety incidents are occurring, and especially why.

Machine learning and deep learning have significantly improved text classification performance, potentially augmenting ADR classification from text. Typically called supervised models, they are reliant on large amounts of data with human expert-derived labels, straining resources and making them quite expensive. Semi-supervised deep learning models use a small set of labeled data and a relatively larger collection of unlabeled data for training algorithms and offer an alternative to fully supervised models [4,15].

To end with a caution – though a positive one. Advanced analytics will indeed advance our ability to detect and understand rare and very are ADRs that ‘older’ methods cannot. This should not be considered the solution to the problem of detecting (very) rare ADRs, but the trigger to the next step: ascertaining and validating whether a newly detected clinical event is indeed an ADR. In this, we have to return to thinking that is almost 60 years old – transformational as it was in 1965 and fundamental as it has been since. In his 1965 Inaugural Presidential Address of the Section on Occupational Medicine of the Royal Society of Medicine in London, Sir Austin Bradford Hill laid out nine criteria for inferring causality from epidemiologic data (surprisingly, he was also one of the pioneers of the randomized clinical trial) [18]: strength (or, in today’s terms, effect size); consistency (now, reproducibility); specificity; temporality; biological gradient (now, dose-response relationship); plausibility; coherence; experiment; and analogy.

4. Conclusion

While there is more than adequate methodology and technology to associate drugs and (rare and very rare) ADRs, this only answers the ‘is there a drug-linked ADR?’ question. This question is likely to remain unanswered if the volume of data and the statistical power to detect ADRs are not available. The paradigm is changing and, in time, powerful and advanced techniques that combine learning from example cases with complex reasoning – attributes of machine and deep learning – will prevail. Yet, in the end, what should govern this work is the careful, critical, and (counter)factual adjudication of whether a causal association between a presumed stimulus (drug) and an observed event (ADR) can and should be inferred.

Declaration of interests

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.