Testing Biased Randomization Assumptions and Quantifying Imperfect Matching and Residual Confounding in Matched Observational Studies

ABSTRACT

One central goal of design of observational studies is to embed nonexperimental data into an approximate randomized controlled trial using statistical matching. Despite empirical researchers’ best intention and effort to create high-quality matched samples, residual imbalance due to observed covariates not being well matched often persists. Although statistical tests have been developed to test the randomization assumption and its implications, few provide a means to quantify the level of residual confounding due to observed covariates not being well matched in matched samples. In this article, we develop two generic classes of exact statistical tests for a biased randomization assumption. One important by-product of our testing framework is a quantity called residual sensitivity value (RSV), which provides a means to quantify the level of residual confounding due to imperfect matching of observed covariates in a matched sample. We advocate taking into account RSV in the downstream primary analysis. The proposed methodology is illustrated by re-examining a famous observational study concerning the effect of right heart catheterization (RHC) in the initial care of critically ill patients. Code implementing the method can be found in the supplementary materials.

KEYWORDS:

• Biased randomization assumption
• Classification
• Clustering
• Imperfect matching
• Residual confounding
• Statistical matching

Supplementary Materials

Appendix contains additional literature review, extension of the proposed methodology to matching-with-multiple-controls, proofs, details on clustering algorithms, and practical strategies for improving a matched comparison. Code and data can be found in the code_and_data.zip file.

Acknowledgments

We would like to acknowledge the editor, associate editor, and three anonymous reviewers for their careful reviews and constructive comments which largely improved the article.

Conflict of Interest

The authors report there are no competing interests to declare.

Additional information

Funding

Research reported in this publication was partially supported by the National Institutes of Health (award RF1AG063481).