Machine learning for clinical trials in the era of COVID-19

Figures & data

Table 1. Summary and guide to the more detailed discussion in this paper.

Clinical trial challenges	Typical trial methodology	COVID-motivated opportunities	Representative method	Section
Improving data quality	Highly controlled environment; extensive data collection and monitoring of patients throughout the trial.	The pandemic and associated measures are causing disruptions to data collection in ongoing trials. Existing ML methods can be used to impute missing data and/or produce estimates robust to missing data. ML methods can also be used to flexibly model and uncover biases introduced by changing conditions over the course of the pandemic.	Time-series imputation using M-RNN (Yoon et al, 2018c)	2
Managing halted trials	In normal adaptive designs, interim analyses of realized clinical outcomes or surrogate end-points, in blinded or unblinded fashion, can be used to adapt recruitment strategies (e.g. refining sample size or eligibility criteria).	Many ongoing (non-COVID-related) clinical trials face temporary suspension. Unplanned interim analyses may present the opportunity to adapt recruitment strategies, in blinded or unblinded fashion, to increase the likelihood that re-started trials succeed. Further, if a trial is fully suspended, ML methods can be used for discovery of (heterogenous) treatment effects and for assessment of uncertainty.	Uncertainty assessment using conformal prediction under covariate shift (Tibshirani et al, 2019)	2
Extracting and incorporating prior information	Bayesian clinical trial designs enable the incorporation of prior information to borrow strength from existing studies (Hobbs et al. 2011).	Much observational evidence is generated by experimental use of drugs, small clinical trials and incomplete/halted trials. ML for causal inference can use this evidence to extract information and build prior beliefs to be incorporated in new studies.	Causal inference from observational data using BART (Hill, 2011)	2, 3
Using ML for drug validation trials	Limited ML-based design methods such as estimating individualized treatment effects (Alaa et al. 2017) or adaptive drug combination studies (Lee et al. 2020a)	The current COVID pandemic provides optimal conditions for existing ML methods for response-adaptive randomisation: the time to clinical endpoint is relatively short, allowing frequent adaptation; a constant stream of patients is arriving and quick action is key.	Sequential patient recruitment and allocation using RCT-KG (Atan et al., 2019)	3, 4
Rethinking the classical phase design	Multi-phased clinical trial with each phase focusing on specific aspects; limited knowledge transfer between phases. High confidence but long process.	Break the static multi-phase paradigm and substitute a dynamic, adaptive trial-collection-trial loop with frequent evaluation and adjustment, leading to faster convergence.	Considering efficacy and toxicity jointly in early stage trials using SEEDA (Shen et al, 2020)	4

Figure 1. The observed data contains information about patient characteristics $x$, assigned treatments and observed (factual outcomes) outcomes. The observed outcomes for the control (blue) and treated (red) patients can be used to train machine learning methods to estimate the response surfaces $g_0\left(x\right)\text{ and }{g}_1\left(x\right)$ for each treatment option. Using these response functions we can estimate individualized treatment effects and thus identify patients who would benefit most and patients who would benefit least most from receiving the treatment. This would not be possible if we only estimated the average treatment effect.

Figure 2. Given an observational dataset with patient features $X_i$, assigned treatments $T_i$ and factual outcomes $Y_i$ jointly sampled from the distribution ${\mathbb{P}}_{\theta }$, validation is needed (e.g. Alaa and van der Schaar 2019) to select the causal inference method, out of the large number available (e.g. Causal Forests (Athey and Imbens 2016), NSGP (Alaa and van der Schaar 2019), and GANITE (Yoon et al. 2018)) that will achieve the best estimate of the individualized treatment effects.

Figure 3. Distribution of treatment effects for subgroups identified by R2P and four benchmark methods using simulated data. The vertical axis is the estimated treatment effect; the horizontal axis indexes the subgroups identified by each method. R2P, CCT and CT-A each identify 5 subgroups, CT-H identifies 4 subgroups and CT-L identifies 3. (See the text for the description of the four benchmark methods.) Each box represents the range between the 25th and 75th percentiles of the treatment effects of the test samples; each whisker represents the range between the 5th and 95th percentiles.

Figure 4. Static vs online-learning-based clinical trial design.