The Effectiveness of a Parents as Teachers Home Visitation Program on School Readiness: An Application of Complier Average Causal Effect Analysis

ABSTRACT

Purpose

The purpose of this study is to evaluate an encouragement trial of a Parents as Teachers (PAT) home visitation intervention on the school readiness of preschool children using an innovative analysis to address issues of selective enrollment.

Method

Families were given the opportunity to enroll in a PAT program through a randomized lottery. The PAT program is assessed using standardized measures of school readiness before and after the two-year program. A comparison of three different analyses is used to evaluate the program – Average Treatment Effect (ATE) analysis, Intent-to-Treat (ITT) analysis, and Complier Average Causal Effect (CACE) analysis. CACE is an innovative analysis developed specifically to diagnose bias arising from selective enrollment in the context of an encouragement trial.

Result

All three analyses (ATE, ITT, and CACE) provide statistically significant evidence of an effective PAT program. However, the effect sizes for the CACE analysis are over twice as large as the other two analyses. The Cohen’s D for CACE is .934 compared to .424 for ATE and .381 for ITT.

Conclusion

This study provides evidence of an effective PAT program. The comparison of ATE, ITT, and CACE analyses reveals the potential for meaningful under-reporting of the program’s impact if selective enrollment is ignored. CACE analysis demonstrates how selective enrollment can bias evaluations of home visitation interventions in general.

Keywords:

• achievement gap
• child development
• disadvantage
• encouragement trail
• immigration
• school readiness

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1. In medical research, it is called “contamination adjusted intention to treat” analysis (Imbens & Rubin, 1997; Sussman & Hayward, 2010), in economics the “Local Average Treatment Effect” (Imbens & Rubin, 1997), and in public health and epidemiology, the method is called the “Complier Average Causal Effect” (CACE see, e.g., Connell, 2009; Ludwig et al., 2011). CACE analysis the result of an interdisciplinary effort over decades that has recently achieved broad recognition (e.g., Angrist et al. (1996) are 2021 Nobel recipients).

2. A limitation with instrumental variable estimators, like CACE, is a tendency to produce inefficient standard errors, especially when working with small samples. Pooling the control groups increases the sample size and statistical power. In supplemental analyses, if we exclude the alternative controls, the standard errors are larger, but this inefficiency is countered by even larger treatment effects than those reported here.

3. Hispanic ethnicity and families with girls are also more likely to enroll in the program. We control for gender and ethnicity in our regression models to account for these differences.

4. There are two additional recodes of the baseline characteristics for this analysis. First, we use the natural log of household income, which is customary in regression modeling. Second, we collapsed non-Hispanic white/Asian/other category into the reference group given the small sample sizes for those racial categories. Additionally, one outlier is removed from the analysis because they reported an unusual change in their test score. This household experienced a number of complications that was making it difficult schedule visits and difficult to conduct lessons when visits were held.

Additional information

Funding

The author(s) reported that there is no funding associated with the work featured in this article.