Optimal Multilevel Matching in Clustered Observational Studies: A Case Study of the Effectiveness of Private Schools Under a Large-Scale Voucher System

ABSTRACT

A distinctive feature of a clustered observational study is its multilevel or nested data structure arising from the assignment of treatment, in a nonrandom manner, to groups or clusters of units or individuals. Examples are ubiquitous in the health and social sciences including patients in hospitals, employees in firms, and students in schools. What is the optimal matching strategy in a clustered observational study? At first thought, one might start by matching clusters of individuals and then, within matched clusters, continue by matching individuals. But as we discuss in this article, the optimal strategy is the opposite: in typical applications, where the intracluster correlation is not one, it is best to first match individuals and, once all possible combinations of matched individuals are known, then match clusters. In this article, we use dynamic and integer programming to implement this strategy and extend optimal matching methods to hierarchical and multilevel settings. Among other matched designs, our strategy can approximate a paired clustered randomized study by finding the largest sample of matched pairs of treated and control individuals within matched pairs of treated and control clusters that is balanced according to specifications given by the investigator. This strategy directly balances covariates both at the cluster and individual levels and does not require estimating the propensity score, although the propensity score can be balanced as an additional covariate. We illustrate our results with a case study of the comparative effectiveness of public versus private voucher schools in Chile, a question of intense policy debate in the country at the present.

KEYWORDS:

• Causal inference
• Group randomization
• Hierarchical/multilevel data
• Observational study
• Optimal matching

Acknowledgments

For comments and suggestions, the authors thank three anonymous reviewers, an associate editor, and Joseph Ibrahim. The authors also thank Magdalena Bennett, Jake Bowers, Nicolás Grau, Cinar Kilcioglu, Winston Lin, Sam Pimentel, and Paul Rosenbaum, and seminar participants at Johns Hopkins University and the University of Pennsylvania.

Funding

This work was supported by a grant from the Alfred P. Sloan Foundation.

Notes

1 For student in secondary schools, the SIMCE only collects test scores on language and math. At the primary school level, we have test scores for language, mathematics, social sciences, and natural sciences.

2 To be precise, we required the absolute differences in means in these covariates to differ at most by 0.1 standard deviations. Please see Table 3.

3 We match the students within treated school 1 and control schools {1, …, 5}, treated school 2 and control schools {1, …, 5}, and so on.

4 To be precise, we required the absolute differences in means in these covariates to differ at most by 0.1 standard deviations. The matched sample that we found did not only satisfy these mean balance constraints but achieved absolute differences in means smaller than 0.05 standard deviations. Please see Table 1.

5 This approximation scheme solves a linear program relaxation of the integer programming problem in cardinality matching and then rounds the solution by solving a linear program again or using a more specialized network algorithm. This approximation may violate to some extent some of the balancing constraints but it runs quickly (in polynomial time), and in many applications the violations to the balancing constraints are not substantial.

6 Another possibility is to use weights that are proportional to the total number of students in a matched cluster pair: w_k ∝n _k1 + n _k2 or w_k = (n _k1 + n _k2)/∑ ^K _{ℓ = 1}(n _ℓ1 + n _ℓ2). An analysis, with these weights did not alter the results.

7 Hansen, Rosenbaum, and Small (2014) noted that sensitivity to hidden bias may vary with the choice of weights w_k . To understand whether different weights lead to different sensitivities to hidden confounders, we can conduct a different sensitivity analysis for each set of weights and correct these tests using a multiple testing correction (Rosenbaum 2012b). We then report a single corrected p-value for a value of Γ.