Assessing the impact of school-based mentoring: Common problems and solutions associated with evaluating nonprescriptive youth development programs

ABSTRACT

Like many youth development programs, most youth mentoring programs do not have prescribed practices that target specific outcomes. Because the construct of mentoring represents a broad range of potential activities, researchers face a conundrum when making generalizable causal inferences about the effects of this and similar services. On the one hand, researchers cannot make valid experimental inferences if they do not describe what they manipulate. On the other hand, experiments that include prescribed protocols do not generalize to most mentoring programs. In most cases, researchers conducting school-based mentoring program evaluations err on the side of not sufficiently specifying treatment constructs, which limits the field’s ability to make practically or theoretically useful inferences about this service. We discuss this reality in light of the fundamental logic of the experimental design and suggest several possible solutions to this conundrum. Our goal is to empower researchers to adequately specify treatments while still preserving the treatment construct validity of this and similar interventions.

Notes

¹An appealing aspect of the prescriptive approach espoused in the prevention science framework is that researchers may assess intervention fidelity, or the “extent to which an intervention’s core components have been delivered as prescribed…” (p. 375; Nelson, Cordray, Hulleman, Darrow, & Sommer, 2012; emphasis added), which is an apt proxy for establishing the suspected strength of the treatment construct as it was delivered. However, by this definition, to the extent that a treatment is not prescribed, assessing fidelity becomes less straightforward.

²For mentoring programs that are prescriptive or manualized (e.g., Brief Instrumental School-Based Mentoring; McQuillin & Lyons, 2016), researchers should consider recent work on contrasting prescribed treatments and identifying sources of variation in program effects (see, Weiss, Bloom, & Brock, 2014 and Nelson et al., 2012).