Bias Correction for Replacement Samples in Longitudinal Research

Abstract

Missing data are commonly encountered problem in longitudinal research. One way researchers handle missing data is through the use of supplemental samples (i.e., the addition of new participants to the original sample after missing data appear at the second or later measurement occasion). Two types of supplemental sample approaches are commonly used: a refreshment approach (additional participants are randomly selected from the population of interest) and a replacement approach (additional participants are selected based on auxiliary variables that explain missingness in the original data). Past research demonstrates that using a replacement approach produces biased parameter estimates because the addition of the replacement sample results in an unrepresentative sample of the population. However, replacement samples have been used in previous studies and the estimation bias has not been corrected. Thus, for this study, we propose and evaluate four ways to correct the bias introduced by replacement samples: a parametric bootstrapping replacement sample correction, a non-parametric bootstrapping replacement sample correction, a primary inverse probability reweighting correction, and a likelihood-based inverse probability reweighting correction. We evaluate their performance using a simulation study and an empirical study.

Keywords:

• Supplemental sample
• replacement sample
• missing data
• longitudinal design
• bias correction

Notes

1 We verified that this procedure does not reduce bias using simulations. In the first simulation we fit a growth curve model using the auxiliary variable as an observed covariate in the fitted model, and in the second simulation we incorporated the auxiliary variable using FIML (Graham, 2003). Results showed that these methods are not effective at addressing the bias caused by replacement samples.

2 Note that we did not consider the coverage rate of the 95% confidence intervals for the slope variance estimate because the confidence intervals calculated as $\mathit{estimate}\pm 1.96\times SE$ were inaccurate. We investigated the standard error estimates. Since the average standard error estimates were almost identical to the empirical standard errors, we believe that the standard errors were correctly estimated. Although the parameter estimates have similar bias to the complete data, the problem was that the slope variance estimates do not follow a normal distribution so 1.96 cannot be used in the above formula. Similarly for the intercept variance and correlation parameter estimates, we did not report the coverage rates. We believe that bias and relative efficiency are adequate to evaluate the performance of the correction methods in estimating these parameters. If substantive researchers are interested in confidence intervals of these parameter estimates, confidence intervals need to be computed in better ways, such as obtaining bootstrap confidence intervals.

3 In a pilot study we examined the power for the correlation parameter by setting the correlation between the intercept and slope to .3 (N = 400, missing rate = 8%, correlation between the auxiliary variable and latent slope =.3). Results showed a power value of 1 in all situations.

4 ⁴We used the rsem package in R to conduct the analysis. We set φ to 0 meaning none of the cases were down-weighted and the method was identical to the NML method.

5 We conducted a simulation to compare the results for the two complete data conditions between R (using the two-stage procedure for SEM in the rsem package) and Mplus Version 6 (using the MLR estimator). R and Mplus produced extremely similar results for all evaluation criteria across the five parameters. This suggests that observed differences between correction methods are a result of the different methods rather than differences between software. Complete results from this comparison are available in the supplementary file.