Program evaluation models and related theories: AMEE Guide No. 67

Figures & data

Table

Practice points

Educational programs are fundamentally about change; program evaluation should be designed to determine whether change has occurred. Change can be intended or unintended; program evaluation should examine for both. Program evaluation studies have been strongly influenced by reductionist theory, which attempts to isolate individual program components to determine associations with outcomes. Educational programs are complex, with multiple interactions among participants and the environment, such that system theory or complexity theory may be better suited to informing program evaluation. The association between program elements and outcomes may be non-linear—small changes in program elements may lead to large changes in outcomes and vice-versa. Always keep an open mind—if you believe you can predict the outcome of an educational program, you may be limiting yourself to an incomplete view of your program. Choose a program evaluation model that allows you to examine for change in your program and one that embraces the complexity of the educational process.

Table 1.  Comparison of evaluation models

CIPP studies	Context studies	Input studies	Process studies	Product studies
Logic model		Input element→	Activities element→	Output element→	Outcomes element
Kirkpatrick's 4-level model					Learner-related outcomes
Experimental/quasi- experimental models					Linear relationship of intended program outcomes to program elements

Figure 1. Logic model components.

Table 2.  Evaluation questions common to CIPP evaluation studies

Context	Input	Process	Product
• What is necessary or useful: in other words, what are the educational needs? • What are the impediments to meeting necessary or useful needs? • What pertinent expertise, services, or other assets are available? • What relevant opportunities (e.g. funding opportunities, administrative support) exist?	• What are the potential approaches to meeting the identified educational need? • How feasible is each of the identified approaches, given the specific educational context of the need? • How cost-effective is each identified approach, given the specific educational context of the need?	• How was the program actually implemented, compared to the plan? • How is/was the program implementation documented? • Are/were program activities on schedule? If not, why? • Is/was the program running on budget? If it is/was over or under the planned budget, why? • Is/was the program running efficiently? If not, why? • Can/did participants accept and carry out their roles? • What implementation problems have been/were encountered? • How well are/were the implementation problems addressed? • What do/did participants and observers think about the quality of the process?	• What positive outcomes of the program can be identified? • What negative outcomes of the program can be identified? • Were the intended outcomes of the program realized? • Were there unintended outcomes, either positive or negative? • What are the short-term implications of program outcomes? • What are the longer-term implications of program outcomes? • What impacts of the program are observed? • How effective was the program? • How sustainable is the program? • How sustainable are the intended and positive program outcomes? • How easily can the program elements be adopted by other educators with similar needs?