A Simulation Study on the Estimation of the Effect in the Hypothetical Scenario of No Use of Symptomatic Treatment in Trials for Disease-Modifying Agents for Alzheimer’s Disease

Figures & data

Fig. 1 A directed acyclic graph encoding the causal assumptions regarding the data generating mechanism under the null hypothesis without an effect of the investigational treatment on the outcome (scenario 1). Observed variables are represented as squares, unobserved as ellipses.

Fig. 2 A directed acyclic graph encoding the causal assumptions regarding the data generating mechanism under the alternative hypothesis with an effect of the treatment on the outcome (scenario 2). Observed variables are represented as squares, unobserved as ellipses.

Fig. 3 Illustration of simulated scores. Each line represents the progression of a simulated patient, measured at baseline and after 1 and 2 years. Blue dots are values measured in patients receiving placebo, orange dots values measured in patients receiving the investigational disease modifying treatment. The distribution for each group is described at each of the three timepoints as box-and-whiskers (in blue for patients receiving placebo, in orange for patients receiving the investigational disease-modifying treatment).

Table 1 Performance of estimators under the null hypothesis without unmeasured confounding, scenario 1.

Estimator	Bias (SE${}^{\ast })$	Coverage (SE${}^{\ast })$	sqrMSE (MSE (SE${}^{\ast }))$	CI length (SE${}^{\ast })$	Type I error rate (95% CI${}^{\ast })$
Reference estimator – linear model using $Y_2^{\ast }$	–0.001 (0.003)	95.3 (0.952)	0.269 (0.113 (0.002))	1.329 (0.001)	0.022 (0.019, 0.025)
Observed values${}^{+}$	–0.001 (0.003)	95.18 (0.951)	0.261 (0.106 (0.002))	1.287 (0.001)	0.022 (0.019, 0.025)
Observed values – adjusted${}^{+}$	0 (0.003)	95.14 (0.95)	0.247 (0.095 (0.001))	1.218 (0.001)	0.023 (0.02, 0.026)
Loh’s g-estimation${}^{+}$, model-based SE	–0.001 (0.003)	95.19 (0.951)	0.268 (0.113 (0.002))	1.325 (0.001)	0.022 (0.019, 0.025)
Loh’s g-estimation${}^{+}$, bootstrap based SE	–0.001 (0.003)	95.21 (0.951)	0.268 (0.113 (0.002))	1.322 (0.001)	0.022 (0.019, 0.025)
Linear sequential g-estimation${}^{+}$	–0.001 (0.003)	95.06 (0.949)	0.266 (0.111 (0.002))	1.308 (0.001)	0.023 (0.02, 0.026)
Predictive mean matching (PMM)	0 (0.003)	95.64 (0.955)	0.248 (0.097 (0.001))	1.271 (0.001)	0.02 (0.017, 0.023)
PMM worsening adjustment 0.5	0 (0.003)	95.65 (0.956)	0.256 (0.103 (0.001))	1.308 (0.001)	0.02 (0.017, 0.023)
PMM worsening adjustment 2	0 (0.004)	95.73 (0.956)	0.286 (0.129 (0.002))	1.453 (0.001)	0.02 (0.018, 0.023)
PMM worsening adjustment 3	–0.001 (0.004)	95.66 (0.956)	0.311 (0.152 (0.002))	1.573 (0.001)	0.02 (0.017, 0.023)
Inverse probability weighting	0 (0.005)	93.74 (0.935)	0.397 (0.253 (0.004))	1.882 (0.005)	0.032 (0.029, 0.035)
Doubly robust inverse probability weighting	–0.001 (0.004)	94.77 (0.946)	0.287 (0.129 (0.002))	1.39 (0.001)	0.026 (0.023, 0.029)
Mixed effects model	0 (0.004)	94.75 (0.946)	0.297 (0.139 (0.002))	1.462 (0.001)	0.028 (0.024, 0.031)

Performance parameter are rounded to three digits.

* Monte Carlo estimates of the standard errors.

+ Estimator uses post-IE values.

Table 2 Performance of estimators under Scenario 2, in presence of treatment effect and in absence of confounding between intercurrent event and decline.

Estimator	Bias (SE${}^{\ast })$	Coverage (SE${}^{\ast })$	sqrMSE (MSE (SE${}^{\ast }))$	CI length (SE${}^{\ast }{)}^{\ast }$	Empirical power (95% CI${}^{\ast })$
Reference estimator – linear model using $Y_2^{\ast }$	0.004 (0.003)	94.61 (0.945)	0.235 (0.088 (0.001))	1.167 (0.001)	0.896 (0.89, 0.902)
Observed values${}^{+}$	0.035 (0.003)	94.24 (0.941)	0.229 (0.084 (0.001))	1.13 (0.001)	0.895 (0.889, 0.901)
Observed values – adjusted${}^{+}$	0.122 (0.003)	92.73 (0.924)	0.24 (0.091 (0.001))	1.088 (0.001)	0.855 (0.848, 0.862)
Loh’s g-estimation${}^{+}$, model-based SE	0.007 (0.003)	94.45 (0.943)	0.235 (0.088 (0.001))	1.163 (0.001)	0.897 (0.891, 0.903)
Loh’s g-estimation${}^{+}$, bootstrap based SE	0.007 (0.003)	94.38 (0.942)	0.235 (0.088 (0.001))	1.162 (0.001)	0.893 (0.887, 0.899)
Linear sequential g-estimation${}^{+}$	0.014 (0.003)	94.27 (0.941)	0.233 (0.087 (0.001))	1.148 (0.001)	0.897 (0.891, 0.903)
Predictive mean matching (PMM)	0.152 (0.003)	90.41 (0.899)	0.264 (0.107 (0.001))	1.121 (0.001)	0.8 (0.793, 0.808)
PMM worsening adjustment 0.5	0.127 (0.003)	91.85 (0.915)	0.26 (0.105 (0.001))	1.154 (0.001)	0.805 (0.797, 0.813)
PMM worsening adjustment 2	0.051 (0.003)	94.12 (0.939)	0.266 (0.112 (0.002))	1.288 (0.001)	0.783 (0.775, 0.791)
PMM worsening adjustment 3	0 (0.004)	94.57 (0.944)	0.285 (0.129 (0.002))	1.402 (0.001)	0.756 (0.748, 0.765)
Inverse probability weighting	0.118 (0.004)	91.57 (0.912)	0.332 (0.175 (0.003))	1.532 (0.004)	0.593 (0.583, 0.603)
Doubly robust inverse probability weighting	0.028 (0.003)	94.06 (0.939)	0.252 (0.101 (0.001))	1.217 (0.001)	0.855 (0.848, 0.862)
Mixed effects model	0.107 (0.003)	93.08 (0.928)	0.275 (0.119 (0.002))	1.27 (0.001)	0.746 (0.738, 0.755)

Performance parameter are rounded to three digits.

* Monte Carlo estimates of the standard errors.

+ Estimator uses post-IE values.

Fig. 4 Distribution of biases for the trials under the null hypothesis (scenario 1). Methods that use post-IE data are plotted in blue and methods that discard post-IE data are plotted in red.

Fig. 5 Distribution of biases for the trials under the alternative hypothesis (scenario 2). Methods that use post-IE data are plotted in blue and methods that discard post-IE data are plotted in red.

Fig. 6 Dependency between the performance and the difference in starting symptomatic treatment between treatment groups in scenario 2. The difference between the candidate estimator and the reference estimator depends on the difference in starting symptomatic treatment between treatment groups for the Mixed effects Model (MeM) setting values after initiation of symptomatic treatment to missing, and using the observed values in a linear model (Observed). No such dependency is observed for Doubly Robust IPW (drIPW) and Loh’s g-estimation (Loh’s g-est).