Evaluative studies in (medical) education

Introduction

Within society in general, education has rapidly gained importance just as the notion of accountability. Also within the medical domain the evaluation of education is emphasized more and more today. This editorial gives an overview of possible study designs which can be helpful for those who are engaged in the evaluation of medical education. Although a number of factors can be distinguished that determine the outcome of an educational program, the question how to organize the evaluation of a curriculum is in the first place determined by the aim of (the) evaluation. Therefore in this contribution we will focus on the aim of the evaluation and use a stepwise model in order to discuss the design and realization of evaluative studies.

Programming of the curriculum or the pre-phase evaluation

This kind of study focuses on evaluation of the activities, means, and preconditions needed in order to execute a curriculum. In such an evaluation the main focus is on the planning of the activities, the quality of the teaching materials, the interest shown by the target group, financial and organizational support, etc. and on the question of whether all conditions have been met in order to make the program successful. A separate part of the pre-phase evaluation is the cost–benefit analysis, which means the analysis of whether the efforts put in (means, manpower etc.) weigh up to the effects which are to be expected. An important consideration in this should be whether the same effect could be reached with less effort or more effect with the same effort. The pre-phase evaluation normally lines up with the development of the program and can thus result in improvements in the program before it is actually carried out. The realization of one or more pilot programs is often a valuable part of the pre-phase evaluation.

Formative or process evaluation

The actual realization of the program is assessed on the basis of the realization planned beforehand. Problems in the realization which can lead to stagnation, decrease of effectiveness, overrunning of the budget etc., can thus be pointed out and adjusted if necessary. The process evaluation can also help to assess the ultimate effect and can thus, on that basis, introduce readjustments.

Effect evaluation

This category agrees best with the general aim of evaluation which is the question to which extent the program meets the expectations as far as the effects and the efforts are concerned. The assessment of the effect can occur on different levels:

1. Do the activities reach the target group?

2. The opinion of the target group about the program

3. The changes in the knowledge and understanding of the target group

4. The changes in behavior of the target group

5. The effects of these changes for the (work of) the target group itself

6. The effects of the changes inside the organization

7. The social effects of the changes achieved, including innovation (that is the first attempt to carry out a new idea or process into practice).

The complexity of the evaluation will increase as one “goes into” these levels. On the one hand this is caused by the increasing abstraction of the evaluation criteria which are needed for assessment of the program, and on the other hand by the increasing problems in measurements and testing which will occur and will draw heavily on the reliability of the data and on the validity of the study as a whole. Many evaluative studies therefore only reach level C: meaning the measurements of changes in knowledge and attitude. Only by extensive research over a longer period of time remaining, can changes in behavior and their effects be analyzed.

In effect evaluation two criteria are often used. In the first place one analyses whether the effect reached corresponds with the goal of the curriculum: the determination of effectiveness. In many cases this will only partially be the case. Particularly then the second criterion is very important, namely, whether the effects gained equal the efforts put in the curriculum: determination of efficiency. Just as in the pre-phase evaluation in this cost–benefit analysis one should consider if more effect could have been reached by less effort.

The monitoring of the program

Even though hardly mentioned in the literature on evaluative research, monitoring is a very common form of evaluation. In monitoring one (periodically) tests whether a curriculum which is executed more or less as routinely still meets the criteria in planning, realization, effects, costs etc. When deviations from the fixed criteria are established, one needs to reconsider the aims and the way in which the program intends to reach those aims.

Conclusion

Because evaluative studies in (medical) education are especially important to the international society of psychosomatic obstetricians and gynecologists, as advocated recently by Bitzer (2007), we hope that this contribution will be a ‘helpful invitation’ to publish on this important issue in this journal.