An introduction to implementation evaluation of school-based interventions

ABSTRACT

Data on implementation of school-based interventions adds highly valuable information to corresponding evaluation studies. Measuring implementation outcomes, such as fidelity or acceptability, provides information on how to improve current and future implementation processes. Moreover, analyzing intervention outcomes in combination with implementation outcomes sheds light onto the reasons for variabilities in an intervention’s effectiveness in different schools. The present paper provides a brief introduction to evaluating the implementation of school-based interventions. First, different types of implementation outcomes and approaches for selecting appropriate outcomes are introduced. Subsequently, measurement issues associated with implementation evaluation in school settings are discussed. Finally, requirements and advantages associated with linking implementation outcomes to intervention outcomes in data analyses are presented. The information is expected to be useful for researchers who are evaluating school-based interventions and especially for those who are new to the field of implementation evaluation.

KEYWORDS:

• Evaluation
• implementation
• fidelity
• education
• school-based interventions

Introduction

When interventions are implemented in real-world educational settings, it cannot be expected that implementation processes occur exactly as planned or that they are conducted identically at every school. Accordingly, evaluating the implementation of school-based interventions is highly relevant for at least two purposes: In process evaluations, information on implementation outcomes, such as the teachers’ acceptance of an intervention, can be used to continuously optimize current implementation processes and to learn for future implementation projects. In summative evaluations, analyzing intervention outcomes in relation to implementation outcomes helps determining whether an intervention’s (in-)effectiveness can be attributed to its contents and design or rather to its implementation (Lendrum & Humphrey, 2012). Thus, including data on implementation in evaluation studies brings valuable information to both researchers and stakeholders who are involved in the implementation process.

The present paper provides a brief introduction on how to evaluate the implementation of school-based interventions, focusing on the selection of implementation outcomes and their measurement as well as on connecting implementation and intervention outcomes in data analyses. At the end of each section, an example illustrates how researchers can apply the information in evaluation studies. The information in this introductory article is supposed to serve as a general overview on implementation evaluation, especially for researchers who conduct evaluation studies in schools and are not yet familiar with concepts from implementation science.

Specifying implementation outcomes

Implementation outcomes are defined as ‘the effects of deliberate and purposive actions to implement new treatments, practices and services’ (Proctor et al., 2011, p. 65). They serve as indicators for the success of implementation strategies and need to be distinguished from intervention outcomes, i.e., the indicators of an intervention’s effectiveness. For instance, a school-based violence prevention program might consist of ten sessions where school psychologists train teachers in how to react to bullying cases at school. While intervention outcomes might include the teachers’ knowledge, attitudes, and behavior in regard to bullying, implementation outcomes center on how the intervention was planned, delivered, and received.

A widely used taxonomy of implementation outcomes has been presented by Proctor and colleagues (2011), who describe the following eight constructs: Acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability (for definitions and examples see Table 1). Fidelity is the implementation outcome that has been measured most often in implementation evaluations (Durlak & DuPre, 2008; Shah et al., 2017). It can be defined as ‘the degree to which teachers and other program providers implement programs as intended by the program developers’ (Dusenbury, 2003, p. 240). Hence, only by evaluating fidelity we know whether a program has been carried out in practice as it was planned.

Table 1. Implementation outcomes based on Proctor et al. (2011) with examples from the school context.

Outcome	Definition	Example
Acceptability	The stakeholders’ satisfaction with different parts of an intervention	Students’ and teachers’ approval of an intervention’s content and structure
Adoption	The use or uptake of an intervention by an organization	A school’s decision to start implementing a new intervention
Appropriateness	The perceived compatibility of an intervention with a particular setting or stakeholder	Teachers’ perceived relevance of an intervention for tackling a particular problem at their school
Cost	An intervention’s implementation costs that can vary depending on implementation strategies and different settings	The costs of implementing an intervention in schools with varying infrastructure
Feasibility	The extent to which an intervention is applicable during the everyday operations of an organization	Teachers’ resources (e.g., time) to participate in an intervention
Fidelity	The degree to which facilitators deliver an intervention as it has been initially designed	The degree to which school psychologists facilitate an intervention as described in an intervention manual
Penetration	The reach of an intervention within a service setting	The rate of schools that are implementing an intervention within a school district
Sustainability	The extent to which an intervention is institutionalized within an organization	The continuation of an intervention in schools beyond its piloting phase

Moreover, measuring fidelity not only serves the monitoring of implementation activities, since the mere measurement of fidelity can also increase intervention effects (Klug, Schultes, & Spiel, 2018). Dusenbury (2003) describe the following dimensions of fidelity: Adherence to the theoretical guidelines of the intervention, dose, referring to the number of training sessions delivered or received by participants, quality of delivery by facilitators or teachers, participant responsiveness, which refers to the participants’ involvement and engagement in the intervention and program differentiation or the presence of an intervention’s core components. The core components of an intervention are those elements that have to be delivered in order to achieve desired outcomes according to an intervention’s underlying theory of change (Blase & Fixsen, 2013).

In terms of selecting the most relevant implementation outcomes for a particular implementation evaluation, numerous implementation theories, frameworks and models can guide the selection process (for an overview see, e.g., Moullin et al., 2015; Nilsen, 2015). As a foundation for specifying implementation outcomes, it is helpful to identify challenges and facilitators for the specific implementation project. The Consolidated Framework for Implementation Research (CFIR, Damschroder et al., 2009), for example, describes determinants that can impact the implementation of interventions at different levels. For school-based interventions, these include knowledge and beliefs about the intervention among school staff, the learning climate at particular schools, external incentives to conduct the intervention, the intervention’s adaptability to different school contexts and the engagement of stakeholders in the implementation process.

Guided by the CFIR framework, researchers can identify the most important determinants for their implementation project, at best in accordance with project stakeholders (Koh et al., 2020). Ideally, this process occurs in the planning phase of the implementation project and leads to a selection of appropriate implementation strategies, such as involving students and school staff in the implementation effort and collecting data on the need for the new intervention from various stakeholders (Cook et al., 2019). Based on determinants and strategies, evaluators can specify implementation outcomes for a particular evaluation study.

An example for specifying implementation outcomes

In an example that illustrates how implementation outcomes can be specified, a group of educational psychologists plans to evaluate the implementation of the gender competence program Reflect in 20 secondary schools (for more information on the program, see Kollmayer et al., 2020). The intervention consists of a teacher training, followed by a five-week supervision phase where teachers are supported in systematically integrating the contents into their classroom teaching. Before the intervention is implemented in schools, both the evaluation team and the research team who plans the implementation process meet with a group of representatives of teachers, principals, and students to discuss possible challenges and facilitators. Discussion results from the meeting reveal that many teachers are not aware of the upcoming implementation of Reflect or have little knowledge about the goals and contents of the intervention. Moreover, principals are worried that the intervention might not fit within their academic schedule and school infrastructure. The student representatives express a high interest in the intervention and offer to engage their fellow students in advertising the intervention.

In accordance with the named challenges and facilitators, the implementation team updates their implementation strategies. New strategies include an information campaign about the intervention on social media that is developed by students and discussions with principals from all participating schools about local conditions that should be considered in the implementation plan. The evaluation team decides to assess acceptability, appropriateness, and feasibility of the intervention in different schools as implementation outcomes. Furthermore, the study includes assessments of fidelity concerning the teachers’ implementation of the intervention in their classrooms.

Measuring implementation outcomes

Measuring implementation outcomes considerably strengthens intervention studies, since the measurement of implementation alone can increase intervention outcomes (e.g., Klug et al., 2018). However, operationalizing implementation outcomes can be challenging and, just as for intervention outcomes, a good measurement is key to well-interpretable evaluation results. However, psychometric properties of used instruments are often not reported for implementation evaluation studies and most instruments are only used once (Lewis & Dorsey, 2020).

For a number of implementation outcomes, there are validated instruments that can be applied in evaluation studies (e.g., Aarons et al., 2016; Ehrhart et al., 2016; Weiner et al., 2017). Still, there is a lack of validated instruments that were developed for implementation evaluations in school settings (see Clinton-McHarg et al., 2016; Lewis et al., 2015). Exceptions are, for example, the SUBSIST measure for the sustainability of behavior support interventions at schools (Kittelman et al., 2019) and the Tiered Fidelity Inventory (TFI) for school-wide positive behavioral interventions and supports (McIntosh et al., 2017).

Translating implementation outcome measures to other languages than English is another challenge for implementation evaluation. A recent systematic review on implementation outcome instruments in German language, for example, showed that there are hardly any validated instruments that can be applied in German speaking school settings (Kien et al., 2018). Accordingly, well-established instruments need to be translated and tested for their psychometric properties before their large-scale application in other languages (see, e.g., Kien, Griebler, Schultes, Thaler, & Stamm, 2021).

In terms of measuring fidelity, it might be necessary to develop tailored measures for new interventions (e.g., activity logs, observation checklists, or facilitator and participant surveys). However, since definitions of implementation outcomes differ between implementation frameworks (Martinez et al., 2014), it is important for evaluators to indicate the underlying definitions of their measures. Tailored fidelity measures can be based on the core components of an intervention (see, e.g., Schultes et al., 2015). In addition to defining core intervention components, it is essential to illustrate in logic models how these components are expected to contribute to a program’s effectiveness. Core components can describe didactic principles that should be followed when delivering the intervention (Schultes et al., 2019). A measure of fidelity could capture whether and how all of these core components were implemented in different intervention groups.

Since several stakeholders (e.g., teachers, principals, students, school psychologists) are involved in implementing school-based interventions, evaluation studies should consider implementation outcomes from multiple perspectives (Mowbray et al., 2003). Implementation data from multiple informants have been shown to be related differently to intervention outcomes (Schultes et al., 2015). Accordingly, triangulation in implementation outcome assessments enable evaluators to analyze inconsistencies across data sources to get a complementary picture of the implementation process. Similarly, by accumulating the strengths of both qualitative and quantitative methods of data collection and analysis, mixed-methods approaches allow for more comprehensive evaluations of implementation outcomes (Palinkas et al., 2011).

An example for measuring implementation outcomes

The team evaluating the implementation of the Reflect program in our example has decided to measure acceptability, appropriateness, feasibility, and fidelity as implementation outcomes. They decide to apply the Acceptability of Intervention Measure (AIM), Intervention Appropriateness Measure (IAM), and Feasibility of Intervention Measure (FIM) that have shown satisfactory psychometric properties in previous studies (Weiner et al., 2017) at a pre- and post-test. The team measures acceptability from the students’, teachers’, and principals’ perspectives. Measures of appropriateness and feasibility are only administered to teachers and principals. In terms of a formative evaluation, both the evaluation team and the implementation team discuss data from the pre-test with a group of stakeholder representatives from participating schools.

For the measurement of fidelity, the evaluation team asks teachers to document their teaching during the five-week-supervision phase in prepared checklists. In addition, independent observers attend two classroom sessions of each participating teacher and complete a fidelity observation checklist. At a post-test, fidelity in the teachers’ delivery of core components is also measured from the students’ perspective (see also Schultes et al., 2015).

Linking implementation outcomes to intervention outcomes

Evaluating implementation outcomes provides important information on how to strengthen the intervention by improving its implementation (Bywater, 2012). Moreover, differences in implementation across schools have been shown to be related to a variability in intervention outcomes (Lendrum & Humphrey, 2012). Thus, it is essential to analyze implementation outcomes together with intervention outcomes to adequately rate an intervention’s effectiveness. Otherwise, it is not clear whether researchers should attribute dissatisfying evaluation results to a poorly designed intervention or to implementation failures. In hybrid effectiveness-implementation study designs, both data on an intervention’s effectiveness and implementation are collected (Landes et al., 2019). In addition to assess multiple perspectives on implementation outcomes (see above), it is crucial to evaluate multiple implementation outcomes, since these can have different relations to intervention outcomes (Schultes et al., 2014).

It has become more and more common to report fidelity in relation to intervention outcomes of school-based interventions (Rojas-Andrade & Bahamondes, 2019). On the one hand, this makes it possible to determine whether intervention groups with higher levels of fidelity or with a fidelity rate exceeding a certain benchmark show better outcomes. This indicates whether the intervention is more effective in tackling the particular problem when it is implemented as designed. On the other hand, connecting fidelity to intervention outcomes helps refining the core components of an intervention. For that purpose, fidelity measures of each core component need to be related to intervention outcomes (Abry et al., 2015). The results might lead to a new prioritization in core components of the intervention, as some components might not be as important for reaching the desired outcomes as initially expected. Hence, it can be assumed that these components can be more easily adapted without a loss of program effectiveness, while other components should be implemented with fidelity.

School-based implementation evaluation data are usually collected from multiple groups at different levels. For instance, intervention outcomes can be assessed from students and teachers that are interrelated at the class level. Implementation outcomes can be collected from the facilitators of the intervention (e.g., school psychologists) and its recipients (e.g., teachers), all of whom belonging to particular groups (e.g., schools) that are nested in higher level groups (e.g., school districts or federal states). This hierarchical data structure, which is often comprised of group-level implementation outcomes and individual-level intervention outcomes, needs to be considered in data analyses, for example, by applying multilevel modelling (see, e.g., Schultes et al., 2015).

An example for linking implementation outcomes to intervention outcomes

In our example, the evaluation team collects fidelity data from students who have participated in the Reflect program. In addition, they measure fidelity in classroom observations and in checklists that have been completed regularly by teachers. At a pre- and post-test, they collect intervention outcomes from students, namely the students’ knowledge on gender issues and gender typicality in their occupational aspirations. The data comprise a hierarchical structure with students being nested in groups that are taught by teachers. Accordingly, the evaluation team conducts a multilevel analysis with the students’ intervention and implementation outcomes at the individual level and fidelity data that have been collected from teachers and classroom observers at the group level.

The results show that fidelity scores measured from the students’ perspective predict changes in their occupational aspirations, while the fidelity scores measured with the teachers’ checklists predict an increase in students’ knowledge. Classroom observation ratings of fidelity are related to both outcomes in students. An in-depth analysis of all fidelity data sources shows that the teachers’ adherence to the intervention manual was especially important to the students’ gain in knowledge, while the students’ participant responsiveness contributed most to their change in occupational aspirations (see also Schultes et al., 2014). From measuring implementation outcomes with multiple methods and from different perspectives, the research team has learned how to implement the intervention more successfully in the future. They recommend implementation strategies that support teachers’ program delivery for an increase in students’ knowledge and strategies that engage students for a change in their occupational aspirations.

Conclusion

Results from implementation evaluations are an invaluable resource for the improvement of school-based interventions. They not only inform evaluators and program planners on how implementation processes can be optimized, but also on implementation outcomes that are related to intervention outcomes. Accordingly, implementation evaluations make it possible to refine and prioritize the core components of school-based interventions. As a result, we know much more about how interventions can be effective and how to support schools in the best possible way.

In order to obtain high quality data on implementation outcomes, it is essential to develop and validate corresponding measures for educational settings. This includes an adaption of measures that have been used and tested in health settings as well as translating and validating implementation outcome measures in other languages than English. Moreover, it is advised to measure multiple implementation outcomes from different perspectives and to account for hierarchical data structures in data analyses.

Aside from all its benefits, implementation evaluation of school-based interventions is a complex endeavour that brings along certain challenges. Similar to other evaluation studies, implementation evaluation is not always considered in the planning phase of implementation projects. Thus, implementation frameworks are often only used retrospectively to interpret data that have already been collected (Cassar et al., 2019). Moreover, implementation evaluation needs additional resources from both researchers and schools that have to be considered early on in project plans and research funding. Also, working with elaborate study designs including data triangulation poses additional burden related to data collection on implementing sites. Accordingly, in order to benefit from the quality that considering implementation outcomes brings to evaluation studies, researchers need to convince funders of the advantages and discuss corresponding benefits and challenges with stakeholders.

Acknowledgments

The author thanks Takuya Yanagida and the anonymous reviewers for their helpful feedback on a previous version of this article.

Disclosure statement

No potential conflict of interest was reported by the author(s).