The Implications of Teacher Selection and the Teacher Effect in Individually Randomized Group Treatment Trials

Abstract

In some experimental evaluations of classroom-level interventions it is not practically feasible to randomly assign teachers to experimental conditions. Given such restrictions, researchers may randomly assign students to experimental conditions and consider the teacher to be a part of the intervention. However, in an individually randomized evaluation of a classroom-level intervention, unless teachers are randomized to experimental conditions, it will not be clear whether differences in outcomes between program and control group students are a result of the core components of the intervention or the teachers (i.e., teacher effects). This article clarifies the correct interpretation of “program impacts” when this study design is used. In addition, using the magnitude of estimated teacher effects from past research, this article demonstrates that, if teachers are not randomly assigned to experimental conditions, it is difficult to establish whether the program works or whether the types of teachers selected to teach in program classrooms are simply more or less effective than their control group counterparts. The significant implications of the correct causal interpretation are discussed, and the limitations of this research design are explored.

Keywords:

• Experimental design
• causal inference
• teacher effects
• Individually Randomized Group Treatment (IRGT)

ACKNOWLEDGMENTS

This article was supported by funding from the Carnegie Foundation. I thank Thomas Brock, Howard Bloom, Lashawn Richburg-Hayes, Marie-Andre Somers, Fred Doolittle, Colleen Sommo, Reshma Patel, Stephen Raudenbush, Barbara Foorman, and three anonymous reviewers for their extremely helpful comments. The ideas presented and positions taken in the article are solely the responsibility of the author, however, and do not necessarily reflect views of the funders.

Notes

Note that some examples of the regression discontinuity design can be thought of as analogous to the IRGT trial (e.g., see Calcagno & Long, 2008) and are thus subject to the same concerns raised in this article.

Recent progress on this topic has been made in the field of psychotherapy (for examples, see Krause & Lutz, 2009; Lambert & Baldwin, 2009; Stiles, 2009). An additional notable exception is an article by Raudenbush (2008), where he describes this phenomenon using a potential outcomes framework.

In addition to this assumption, it also must be assumed that program and control teachers are equally effective at implementing the intervention being tested.

The work of Nye et al. (2004) assessed the effectiveness of teachers from kindergarten through third grade, using data from the 1980s, in a single state. Although generalizing from this work alone might not be prudent, their review of the literature suggests that many past nonexperimental studies find evidence of a teacher effect of similar magnitude. In addition, more recent research points in the same direction, suggesting that it is safe to assume that teachers do in fact matter (Clotfelter, Ladd, & Vigdor, 2007; Kane & Staiger, 2008; Kukla-Acevedo, 2009; Rockoff, 2003; Rockoff, Jacob, Kane, & Staiger, 2008).

Here, to “explain completely” does mean “to describe in words.” For example, knowing that program teachers were “volunteers” does not suffice. Rather, a precise measure of all teachers’ (program and control) propensities to volunteer is needed, as well as an understanding of the relationship between teachers’ propensities to volunteer and the outcome measures of interest. However, it is precisely because of our lack of confidence in the ability to model such selection processes that experiments are conducted in the first place.

Figure 2 can be altered to reflect different assumptions about the magnitude of the teacher effect. If the magnitude of the teacher effect is smaller than assumed, then the data points will have a slope of lesser magnitude. If the magnitude of the teacher effect is larger than assumed, then the data points will have a slope of greater magnitude.

The statistical significance threshold reported by MDRC is used for this analysis— using a two-tailed test with α = .10 (Scrivener et al., 2008). The robustness of causal inference threshold is equivalent to the Minimum Detectable Effect with α = .10 and β = .50.

These core components might be loosely defined as providing all services to families through center-based child care and education, parent education, and a minimum of two home visits per year to each family (Love et al., 2002, p. xxiv).