Inference for treatment effects of job training on wages: using bounds to compute Fisher’s exact p-value

ABSTRACT

In the context of a training program’s randomized evaluation, where estimating wage effects is of interest, we propose employing bounds that control for sample selection as a model-based statistic to conduct randomization-based inference à la Fisher. Inference is based on a sharp null hypothesis of no treatment effect for anyone. In contrast to conventional inference, Fisher p-values are nonparametric and do not employ large sample approximations.

KEYWORDS:

• Nonparametric bounds
• randomization inference
• sample selection
• training effects

JEL CLASSIFICATION:

• C10
• C12
• J30

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ For more on this distinction see Rubin (2005) and Imbens and Rubin (2015).

² To analyze wage effects controlling for employment selection, conventional methods rely on strong and impractical assumptions (Lee 2009).

³ Strata are: $\left\{S_i(0),S_i(1)\right\}$: $\left\{1,1\right\}$, $\left\{0,0\right\}$, $\left\{0,1\right\}$, and $\left\{1,0\right\}$. In the first two, treatment does not affect potential employment, since units are always and never employed regardless of treatment, respectively. Potential employment is affected positively by treatment for units in the third stratum (i.e. employed only if treated), while the opposite is true for the last stratum (i.e. employed only if not treated).

⁴ Assumption 2 is plausible in our application since we find positive average impacts on employment, and the outcome is measured 4 years after randomization, minimizing potential locked-in effects. See Blanco et al., (2013, BFF hereafter) for details.

⁵ In the context of JC, BFF find indirect support for Assumption 3 after comparing average pre-treatment covariates (correlated with wages) between the $EE$ and $NE$ strata.

⁶ Constant additive treatment effect $Y_i(1)=Y_i(0)+C$, for a pre-specified $C$, is another alternative. Below, we employ log wages, suggesting a sharp null of $Y_i(1)/Y_i(0)=C$, for $C=1$.

⁷ In practice, obtaining an exact distribution when $n$ is large is not feasible, since there are $\left(\begin{array}{c}n\\ n_1\end{array}\right)$ assignment vectors, with $n_1$ treated individuals. In ourapplication, we employ 500,000 random draws from the set of assignment vectors to estimate the distribution. Imbens and Rubin (2015) recommend at least 250,000 draws.

⁸ Fisher (1935) incorrectly proposed adjusting for a different outcome, or concomitant variable, that may be on the causal pathway of the treatment affecting the outcome of interest by what amounts to regressing the outcome on the treatment and the concomitant variable (Rubin 2005). Principal stratification, however, is a valid way of estimating causal effects in the presence of a concomitant variable. Note that we are focused on the $ATE$ for $EE$ individuals whose potential employment (i.e. the concomitant variable) is unaffected by treatment and can, therefore, be treated as any other $X$.

⁹ Fisher p-values can be obtained in other applications employing similar assumptions and the stratification framework to construct valid model-based statistics, e.g. causal effects in medical trials with truncation-by-death (Imai 2008).

¹⁰ This stratum accounts for about 66% of whites.

¹¹ We focus on discussing the lower bound since, in this context, it determines the sign of the effect. All upper bounds are positive with basically zero exact p-values. Upper bounds could be of interest to rule out ‘extreme’ large effects, in which case the sharp null could be modified by selecting a corresponding value for $C$ (see footnote 6).

¹² Due to a lack of comparability in null hypotheses, Monte Carlo evidence on the relative performance of inference methods is not presented (see footnote 1).

¹³ In addition, the lower bound under assumptions 1–2 imply that there is a perfect negative correlation between employment and wages, which is unlikely given standard models of labour supply. The lower bound under assumptions 1–3 rules out the latter issue.