Reinhard Stockmann, Cheltenham: Edward Elgar, 2011, 392 pp., UK£95.00 (hardback), ISBN 978-1849800426
This practitioner evaluation handbook concludes the trilogy of primary literature on evaluation, published in the ‘Social Scientific evaluation research’ series. This latest book has been devised to help students, with basic knowledge in the field of evaluation technics, and practising evaluators who want to better understand what makes a ‘good’ evaluation and how it should be planned and implemented. It is also written for those who require evaluations and need to understand how to plan and assess a professional independent evaluation process. Hence, the book is rightly guided by the belief that there are several possible answers to the question ‘What do we need evaluation for?’, and this according to the different evaluation objectives and to the different stakeholder perspectives, which are clearly explained throughout the volume. The 10 chapters draw upon materials and documentation produced by several training courses on evaluation given at the Center for Evaluation, at Saarland University, Germany – where the authors belong.
Most likely some of this training material is used in the introduction to provide a useful and pragmatic example of one of the biggest disasters in history, the sinking of Titanic, to show (1) why the evaluation is conducted, (2) for whom the evaluation is conducted, and (3) who benefits from the information obtained through the evaluation process. Basic evaluation knowledge – definitions, objectives, aims and approaches – is provided in the second chapter, with a synoptical scheme in which, according to Fitzpatrick's proposal, five evaluation approaches are presented. The chapter also presents an impact-oriented evaluation approach developed by the editor. This original approach was embedded into organization and innovation theory. Unfortunately, in this section, the editor provides the reader with very brief and non-exhaustive overviews of the various approaches presented in the current evaluation literature, giving rise to some doubts on the practical usefulness of the given information.
Chapter 3, on the contrary, is largely dedicated to the description of the main business management tools, such as Controlling, Auditing, Benchmarking and Balance Scorecard. These tools have significant differences from social evaluation approaches, as correctly pointed out at the end of the chapter, and are extensively illustrated in the new public management domain, so they do not need to be further discussed in the evaluative literature.
All of the practical and organizational aspects of planning and implementing an evaluation are clearly described in Chapter 4. By using several examples and case studies, this chapter provides a useful guideline to develop an evaluation conception – highlighting the main individual steps in planning an evaluation – and how to plan a tender for conducting an external professional evaluation. In order to further define these steps and to explain the overreaching logic of how evaluation research has to be conducted, chapter 5 provides information about evaluation design. The design logic is rooted in research, and it concerns not only the operational sequence step but mainly the ability to demonstrate that an action (cause) leads to an effect. The author divides into two main categories the design approaches to be considered upon planning an evaluation. The first is rooted in the positivist paradigm of exploring social reality, and concerns experimental and quasi experimental designs with specific variants. In the second, the author places the alternative approaches to ‘classical’ experimental design. In the latter, two examples are presented, both sharing a fundamental criticism of the linear cause-and-effect chain. The first is related to ‘qualitative social research’, with a more inclusive framework, like the ‘fourth-generation evaluation’ proposed by Guba and Lincoln in the 1980s. In the second example, the author chooses the system dynamics approach, originally developed by Forrester, to understand the behaviour of complex systems over time. This approach was further developed and applied to policy and programme evaluation in order to define a structural model linking policy interventions to results. While interesting and correctly structured around the attribution problem, this section appears not entirely exhaustive for the following reasons. First, when identifying alternative approaches to experimental design, significant evaluation designs are missing (i.e. the theory-based design with its specific variants such as the realistic evaluation proposed by Ray Pawson). Second, an overall picture to illustrate the main differences – in terms of utility value for evaluation – among design methods in identifying causes and offering explanations is missing.
The end of the chapter reminds us, correctly, that in real-world evaluation, there are many constraints – clients, budget, skills, time, etc. – that are not the same as for research and can strongly affect the final design. This is because the evaluation process is embedded in a social context, where stakes, interests, perspectives, etc., are often in conflicts. ‘The Social Context of Evaluation’ is thus the title of Chapter 6, which provides a valuable overview of the main variables that affect an evaluation process, properly underlining the main differences between evaluation research and traditional research contexts.
The following three chapters address some of the main questions associated with the methodology of social research, that are essential in any evaluation process. Starting with questions concerning the practice of measurement in the social sciences – introducing scale, indicators and indices – the chapters give a practical overview of the main data collection procedures and the common associated problems. They also introduce data management and methods of analysis, with a specific emphasis on statistical procedures that are of particular interest in evaluation context. In the last chapter practical tips to structure a good evaluation reporting are given. The main aims, needs and cautions in evaluation reporting are usefully described and are aimed at both evaluation clients and contractors.
In conclusion, this is a remarkable practical handbook covering a wide range issues, succeeding in presenting on overview of the basic questions related to planning and implementing an evaluation process. However, two issues that could be improved in a subsequent edition need to be highlighted. The first is that some crucial evaluation approaches and designs, heavily debated today, are not properly presented. This affects the volume in various sections, given the importance of this issue. The second is related to a certain discrepancy in the way various contents are tackled throughout the book. Some contents tend to be over-specialized while others are over-simplified, given the target book audience. This is, however, a common risk for books targeted at a wide audience, as in this case. For this reason, readers might wish to dip into it from time to time, but few of them, perhaps, will read from start to end.
© 2013, Francesco Mazzeo Rinaldi