Causal inferences from many experiments

ABSTRACT

The underlying statistical concept that animates empirical strategies for extracting causal inferences from observational data is that observational data may be adjusted to resemble data that might have originated from a randomized experiment. This idea has driven the literature on matching methods. We explore an un-mined idea for making causal inferences with observational data – that any given observational study may contain a large number of indistinguishably balanced matched designs. We demonstrate how the absence of a unique best solution presents an opportunity for greater information retrieval in causal inference analysis based on the principle that many solutions teach us more about a given scientific hypothesis than a single study and improves our discernment with observational studies. The implementation can be achieved by integrating the statistical theories and models within a computational optimization framework that embodies the statistical foundations and reasoning.

KEYWORDS:

• Causal inference
• optimization
• subset selection
• computational methods
• operations research

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. An exception is Zubizarreta's design, which is more flexible and optimizes directly on particular balance measures. All of the designs are the same, however, in that the implementations of these approaches provide one solution to whichever version of the matching problem is posed.

2. We also note that there are situations that are more amenable to causal inference analyses than others. These do not change with our method. For instance, matching methods and our method are more ideal when the control population is much larger than the treatment group size.

3. Other matching technologies match treatment and control units (not subsets) first then assess the success of the matching later by the level of balance achieved. Without knowing how all matching methods perform, it is difficult to assess if balance is good or ‘good enough’ because the baseline or optimal level of balance in a particular data set is unknown. In BOSS, the goal is optimal balance, not ‘good balance’. The optimal level of balance is the baseline or standard for assessing any particular balance level.

4. Note that we are not using randomized experimental data in our analysis, but using only the treatment group from the LaLonde data. We use the LaLonde data because it has been widely used in the literature. This provides a comparison for our model vis-à-vis other models as well as a benchmark estimate from the original data.

5. We presume that there is no spillover between individuals (i.e. we make the stable unit treatment value assumption (SUTVA).

6. We can see from the figure that as balance improves, one standard deviation around the estimate of the treatment effect includes the experimental benchmark, $\hat{\tau }=\mathrm{\$}1794$. To be sure, we do not know the true value of the treatment effect in this instance. We refer to the experimental benchmark here simply to offer some guidance, without certitude, and with sufficient wariness.

7. The objective function, meant to measure covariate balance is flexible in the BOSS technology. A researcher can define balance in any way. The BOSS routine will seek to optimize whatever balance measure is given to it. Our particular formulation for the LaLonde data is specified in Equation (1(1) $b=\sum _{j=1}^{8}K{S}_j+|t_j|+\mid 1-\frac{{\sigma }_{tj}}{{\sigma }_{cj}}\mid .$(1) ).

8. It is possible that we have not identified the best subsets. Certainly, the optimization procedure can still be and is still being refined.