Testing for covariate balance using quantile regression and resampling methods

Abstract

Consistency of propensity score matching estimators hinges on the propensity score's ability to balance the distributions of covariates in the pools of treated and non-treated units. Conventional balance tests merely check for differences in covariates’ means, but cannot account for differences in higher moments. For this reason, this paper proposes balance tests which test for differences in the entire distributions of continuous covariates based on quantile regression (to derive Kolmogorov–Smirnov and Cramer–von-Mises–Smirnov-type test statistics) and resampling methods (for inference). Simulations suggest that these methods are very powerful and capture imbalances related to higher moments when conventional balance tests fail to do so.

Keywords:

• balancing property
• balance test
• propensity score matching

JEL classification :

• C12
• C15
• C21

Notes

Note that these tests may also be applied to count data if they are artificially smoothed as outlined in Machado and Silva 30.

In contrast, Imbens 20 and Lechner 26 discuss effect evaluation for multiple treatments. The discussion in this paper could be easily extended to their framework.

Testing for equality of conditional distributions is discussed in Li et al. 28, although for discrete conditioning variables, whereas we need to condition on a continuous p(X).

We test for equality in mean propensity scores among treated and non-treated units within a stratum at the 10% level of significance.

Shaikh et al. 39 show that with being the pdf of p(X) conditional on D=d, is a testable implication of a correctly specified propensity score and propose a specification test based on kernel density estimation.

It is, however, less efficient than estimation based on the true propensity score model.

Note that we do not know the exact p-value of the Koenker and Machado 25 test statistic because Andrews 2 only provides us with the critical values up to the 10% level, but not across the entire distribution.

The caliper is set to 0.1 standard deviations of the propensity score and 59 observations (6.5%) are dropped due to a lack of common support.

Additional information

Notes on contributors

Martin Huber

I have benefited from comments by Eva Deuchert, Bernd Fitzenberger, Michael Lechner, Enno Mammen, Blaise Melly, Conny Wunsch, conference/seminar participants in Linz (Annual Meeting of the Austrian Economic Association), London (cemmap conference on quantile regression), Geneva (Annual Meeting of the Swiss Society of Economics, Statistics), and, Freiburg i.B. (seminar in empirical economics), and, two anonymous referees.