How Do Propensity Score Methods Measure Up in the Presence of Measurement Error? A Monte Carlo Study

Abstract

Considering that the absence of measurement error in research is a rare phenomenon and its effects can be dramatic, we examine the impact of measurement error on propensity score (PS) analysis used to minimize selection bias in behavioral and social observational studies. A Monte Carlo study was conducted to explore the effects of measurement error on the treatment effect and balance estimates in PS analysis across seven different PS conditioning methods. In general, the results indicate that even low levels of measurement error in the covariates lead to substantial bias in estimates of treatment effects and concomitant reduction in confidence interval coverage across all methods of conditioning on the PS.

KEYWORDS:

• measurement error
• Monte Carlo simulation
• propensity score methods

Notes

The online supplements (i.e., tables and figures of simulation outcome variables by design factors) are available at the author controlled website.

“s” denotes supplemental materials, available online. See Tables 3s, 4s, 5s for the mean balance by covariate type and reliability, by covariate type and the correlation among the covariates, and by trimming and sample size, respectively.

As previously reported for balance results, PS ANCOVA yielded exceptionally large bias when sample size was small and the number of covariates was large. When noted, these extreme conditions were excluded in the results of PS ANCOVA.