![MathJax Logo](/templates/jsp/_style2/_tandf/pb2/images/math-jax.gif)
Abstract
To improve the precision of estimation and power of testing hypothesis for an unconditional treatment effect in randomized clinical trials with binary outcomes, researchers and regulatory agencies recommend using g-computation as a reliable method of covariate adjustment. However, the practical application of g-computation is hindered by the lack of an explicit robust variance formula that can be used for different unconditional treatment effects of interest. To fill this gap, we provide explicit and robust variance estimators for g-computation estimators and demonstrate through simulations that the variance estimators can be reliably applied in practice.
1. Introduction
In randomized clinical trials, adjusting for baseline covariates has been advocated as a way to improve the precision of estimating and power of testing treatment effects (Freedman, Citation2008; Lin, Citation2013; Tsiatis et al., Citation2008; Yang & Tsiatis, Citation2001; Ye et al., Citation2023, Citation2022). We focus on binary outcomes in this article. When a logistic model is used as a working model for baseline covariate adjustment, the g-computation (Freedman, Citation2008; Moore & van der Laan, Citation2009) provides asymptotically normal estimators of unconditional treatment effects such as the risk difference, relative risk and odds ratio, regardless of whether the logistic model is correct or not. In May 2021, the US Food and Drug Administration released a draft guidance (FDA, Citation2021) for the use of covariates in the analysis of randomized clinical trials, and recommended the g-computation as a ‘statistically reliable method of covariate adjustment for an unconditional treatment effect with binary outcomes '.
However, to the best of our knowledge, no explicit robust variance estimation formula for g-computation is currently available that can be used for inference on different unconditional treatment effects of interest. Moreover, some existing variance estimation formulas in the literature, such as the formula in Ge et al. (Citation2011) for risk difference and two treatment arms, are model-based and do not fit the model-robust inference paradigm. Additionally, the formula in Ge et al. (Citation2011) does not take into account a source of variability due to covariates and nonlinearity of logistic model, which can lead to confidence intervals with insufficient coverage probabilities.
The purpose of this article is to fill this gap by providing explicit and robust variance estimators for g-computation estimators. Our simulations demonstrate that the provided variance estimators can be reliably applied in practice.
2. Robust variance estimation
Consider a k-arm trial with n subjects. For each subject i, let be the k-dimensional treatment indicator vector that equals
if patient i receives treatment t for
, where
denotes the k-dimensional vector whose tth component is 1 and other components are 0,
be the binary potential outcome under treatment t, and
be the baseline covariate vector for adjustment. The observed outcome is
if and only if
. We consider simple randomization where
is completely random with known
,
and
. We assume that
, are independent and identically distributed with finite second order moments. To simplify the notation, we drop the subscript i when referring to a generic subject from the population. Write the unconditional response means as
and
, where the superscript ⊤ denotes the transpose of a vector throughout. The target parameter is a given contrast of the unconditional response mean vector
denoted as
, such as the risk difference
, risk ratio
, and odds ratio
between two treatment arms t and s.
Throughout this article, we consider the g-computation procedure that fits a working logistic model , where
, and
and
are unknown parameter vectors (FDA, Citation2021). The logistic model does not need to be correct and is only used as an intermediate step to obtain g-computation estimators. Let
and
be the maximum likelihood estimators of
and
, respectively, under the working logistic model. Then,
is the predicted probability of response under treatment t. The g-computation estimator of
is
with
, and of a given contrast
is
. Hence, the g-computation takes a summary-then-contrast approach (Citation(2019), R1).
Next, we derive the asymptotic distribution of the g-computation estimator and apply the delta method to obtain the asymptotic distribution of the g-computation estimator
. As the logistic regression uses a canonical link, the first-order conditions of the maximum likelihood estimation ensure that, for
,
where
is the indicator of
. Hence, the g-computation estimator is equal to
where
and
is the number of subjects assigned to treatment t. Since
's are assigned completely at random,
and
can converge to
and
with
rate, respectively, where
is a fixed point and
is a function not necessarily equal to
under model misspecification but satisfies
due to the above first-order conditions,
. Then, by Kennedy (Citation2016) and Chernozhukov et al. (Citation2017),
where
denotes the remaining term multiplied by
converges to 0 in probability. Therefore, an application of the central limit theorem shows that, regardless of whether the working model is correct or not,
where
denotes convergence in distribution,
is the k-dimensional vector of zeros, and
By the delta method, when
is differentiable at
with partial derivative vector
, we have
Some examples are:
Note that we apply normal approximation for the log transformed risk ratio and odds ratio because the log transformation typically can improve the performance of normal approximation (Haldane, Citation1956; Woolf, Citation1955).
For robust inference, we propose the following variance estimator for that is always consistent regardless of model misspecification:
(1)
(1)
where
is the sample variance of
for subjects with
,
is the sample covariance of
and
for subjects with
,
is the sample variance of
for all subjects,
is the sample covariance of
and
for subjects with
, and
is the sample covariance of
and
for all subjects. These robust variance estimators can be directly calculated using our R package RobinCar that is publicly available at https://github.com/tye27/RobinCar.
To end this section, we describe the variance estimator in Ge et al. (Citation2011) for the g-computation estimator of risk difference in a two-arm trial and discuss why it can be inconsistent and underestimate the true variance. In our notation, Ge et al. (Citation2011) wrote the g-computation estimator
as
, where
and
. Then they applied the Taylor expansion
where
is the probability limit of
, and proposed
as a variance estimator for
, where
is the model-based variance estimator for
from the standard maximum likelihood approach. This approach has two problems. First, it uses the model-based variance estimator
, which may be inconsistent to the true variance of
under model misspecification. Second, from
the variance estimator proposed by Ge et al. (Citation2011) only accounts for the variance of the first term
but misses the variability from
that is not 0 as the function
is nonlinear. This second problem can lead to a confidence interval with too low coverage probability, which can be seen from the simulation results in the following section.
3. Simulations
We conduct simulations to evaluate the finite-sample performance of our robust variance estimator in (Equation1(1)
(1) ). We consider two arms or three arms, simple randomization for treatment assignments with equal allocation (i.e.,
for two arms and
for three arms), a one-dimensional covariate
, and n = 200 or 500.
We consider the following three outcome data generating processes.
Case I:
.
Case II:
and
.
Case III:
.
In order to determine the true values of the unconditional response means, we simulate a large dataset of sample size for each case and obtain that
for Case I,
for Case II, and
for Case III. In each case, the g-computation estimator is based on fitting a working logistic model
, which is correctly specified under Case I and Case III, but is misspecified under Case II.
For Cases I–II, which have two arms, we focus on estimating and also include the variance estimator in Ge et al. (Citation2011). For Case III, which has three arms, we evaluate our robust variance estimators for three common unconditional treatment effects for binary outcomes. The results for Cases I–II are in Table and for Case III are in Table , which include (i) the true parameter value, (ii) Monte Carlo mean and standard deviation (SD) of g-computation point estimators, (iii) average of standard error (SE), and (iv) coverage probability (CP) of 95% confidence intervals. We use sample size
200 or 500, and 10000 simulation runs.
Table 1. Simulation mean and standard deviation (SD) of , average standard error (SE), and coverage probability (CP) of 95% asymptotic confidence interval for
under Cases I–II and simple randomization.
Table 2. Simulation mean and standard deviation (SD) of g-computation estimators, average standard error (SE), and coverage probability (CP) of 95% asymptotic confidence interval based on robust SE (Equation1(1)
(1) ) under Case III and simple randomization.
From Tables –, we see that the g-computation estimators have negligible biases compared to the standard deviations. Our robust standard error, which is the squared root of variance estimator in (Equation1(1)
(1) ), is always very close to the actual standard deviation, and the related confidence interval has nominal coverage across all settings. In contrast, the standard error in Ge et al. (Citation2011) underestimates the actual standard deviation under Case I when there is no model misspecification, as well as under Case II when there is model misspecification, and the related confidence intervals have too low coverage probabilities in both cases.
4. Summary and discussion
In this article, we provide an explicit robust variance estimator formula for g-computation estimators, which can be used for different unconditional treatment effects of interest and clinical trials with two or more arms. Our simulations demonstrate that the variance estimator can be reliably used in practice.
In this article, for the purpose of being specific, we focus on the logistic model that regresses the outcome on the treatment indicators and covariates, which is arguably the most widely used model for binary outcomes. However, our robust variance estimation formula in (Equation1(1)
(1) ) is not limited to this model and can be used with different specifications of the working model (e.g., fitting a separate logistic model for each treatment arm) or with other generalized linear models using a canonical link for non-binary outcomes (e.g., Poisson regression for count outcomes). Additionally, although our article considers simple randomization, our robust variance formula in (Equation1
(1)
(1) ) can also be used for a complete randomization scheme where the sample size in every group t is fixed to be
, because this randomization scheme leads to the same asymptotic distribution as the simple randomization (Ye et al., Citation2023). Simulation results under this randomization scheme are similar to those under simple randomization; see Tables 3-4 in the Appendix.
We implement an R package called RobinCar to conveniently compute the g-computation estimator and our robust variance estimators, which is publicly available at https://github.com/tye27/RobinCar.
Appendix
In Tables –, we include simulation results under a complete randomization scheme where the sample size in every group t is fixed to be .
Table 3. Simulation mean and standard deviation (SD) of , average standard error (SE) and coverage probability (CP) of 95% asymptotic confidence interval for
under Cases I–II and complete randomization that fixes
.
Table 4. Simulation mean and standard deviation (SD) of g-computation estimators, average standard error (SE), and coverage probability (CP) of 95% asymptotic confidence interval based on robust SE (Equation1(1)
(1) ) under Case III and complete randomization that fixes
.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
References
- FDA (2021). Adjusting for covariates in randomized clinical trials for drugs and biological products. Draft Guidance for Industry. Center for Drug Evaluation and Research and Center for Biologics Evaluation and Research, Food and Drug Administration (FDA), U.S. Department of Health and Human Services, May 2021.
- ICH E9 (R1) (2019). Addendum on estimands and sensitivity analysis in clinical trials to the guideline on statistical principles for clinical trials. International Council for Harmonisation (ICH).
- Chernozhukov, V., Chetverikov, D., Demirer, M., Duflo, E., Hansen, C., & Newey, W. (2017). Double/debiased/neyman machine learning of treatment effects. American Economic Review, 107(5), 261–265. https://doi.org/10.1257/aer.p20171038
- Freedman, D. A. (2008). Randomization does not justify logistic regression. Statistical Science, 23(2), 237–249. https://doi.org/10.1214/08-STS262
- Ge, M., Durham, L. K., Meyer, R. D., Xie, W., & Thomas, N. (2011). Covariate-adjusted difference in proportions from clinical trials using logistic regression and weighted risk differences. Drug Information Journal: DIJ/Drug Information Association, 45(4), 481–493. https://doi.org/10.1177/009286151104500409
- Haldane, S. (1956). The estimation and significant of the logarithm of a ratio of frequencies. Annals of Human Genetics, 20(4), 309–311. https://doi.org/10.1111/ahg.1956.20.issue-4
- Kennedy, E. H. (2016). Semiparametric theory and empirical processes in causal inference. In Statistical causal inferences and their applications in public health research (pp. 141–167).
- Lin, W. (2013). Agnostic notes on regression adjustments to experimental data: Reexamining freedman's critique. Annals of Applied Statistics, 7(1), 295–318. https://doi.org/10.1214/12-AOAS583
- Moore, K. L., & van der Laan, M. J. (2009). Covariate adjustment in randomized trials with binary outcomes: Targeted maximum likelihood estimation. Statistics in Medicine, 28(1), 39–64. https://doi.org/10.1002/sim.v28:1
- Tsiatis, A. A., Davidian, M., Zhang, M., & Lu, X. (2008). Covariate adjustment for two-sample treatment comparisons in randomized clinical trials: A principled yet flexible approach. Statistics in Medicine, 27(23), 4658–4677. https://doi.org/10.1002/sim.v27:23
- Woolf, B. (1955). On estimating the relation between blood group and disease. Annals of Human Genetics, 19(4), 251–253. https://doi.org/10.1111/ahg.1955.19.issue-4
- Yang, L., & Tsiatis, A. A. (2001). Efficiency study of estimators for a treatment effect in a pretest–posttest trial. The American Statistician, 55(4), 314–321. https://doi.org/10.1198/000313001753272466
- Ye, T., Shao, J., Yi, Y., & Zhao, Q. (2023). Toward better practice of covariate adjustment in analyzing randomized clinical trials. Journal of the American Statistical Association, 117, in press. https://doi.org/10.1080/01621459.2022.2049278
- Ye, T., Yi, Y., & Shao, J. (2022). Inference on average treatment effect under minimization and other covariate-adaptive randomization methods. Biometrika, 109(1), 33–47. https://doi.org/10.1093/biomet/asab015