Strengths and weaknesses of European Union policy evaluation methods: Ex-post evaluation of objective 2, 1994–99

Abstract

Baslé M. (2006) Strengths and weaknesses of European Union policy evaluation methods: ex-post evaluation of Objective 2, 1994–99, Regional Studies 40, 225–235. The paper considers ex-post evaluation of Objective 2 at the regional level, focusing on the experience of France over 1994–99. These evaluations were ambitious, and when looking at the reports it identifies both strengths and weaknesses. The paper considers issues that are problematic in the case of the Structural Funds – such as adjustment lags, causality, measurement and ‘cross-checking’ of results using national data – which are explored both through a case study of an ex-post evaluation for the Brittany region, and through examination of the mid-term evaluations for France in 2003. While the paper does find some improvements in the quality of evaluation in the recent mid-term reports, it also identifies continuing problems. These include a weakness in the ‘logical diagram of impact’, tracing the chain of causality from actions to impacts, so that the paper argues for a more-detailed model of intervention.

Objective 2 Ex-post evaluation Regional development France

Baslé M. (2006) Strengths and weaknesses of European Union policy evaluation methods: ex-post evaluation of Objective 2, 1994–99, Regional Studies 40, 225–235. The paper considers ex-post evaluation of Objective 2 at the regional level, focusing on the experience of France over 1994–99. These evaluations were ambitious, and when looking at the reports it identifies both strengths and weaknesses. The paper considers issues that are problematic in the case of the Structural Funds – such as adjustment lags, causality, measurement and ‘cross-checking’ of results using national data – which are explored both through a case study of an ex-post evaluation for the Brittany region, and through examination of the mid-term evaluations for France in 2003. While the paper does find some improvements in the quality of evaluation in the recent mid-term reports, it also identifies continuing problems. These include a weakness in the ‘logical diagram of impact’, tracing the chain of causality from actions to impacts, so that the paper argues for a more-detailed model of intervention.

Objectif 2 Evaluation ex-post Aménagement du territoire France

Baslé M. (2006) Strengths and weaknesses of European Union policy evaluation methods: ex-post evaluation of Objective 2, 1994–99, Regional Studies 40, 225–235. The paper considers ex-post evaluation of Objective 2 at the regional level, focusing on the experience of France over 1994–99. These evaluations were ambitious, and when looking at the reports it identifies both strengths and weaknesses. The paper considers issues that are problematic in the case of the Structural Funds – such as adjustment lags, causality, measurement and ‘cross-checking’ of results using national data – which are explored both through a case study of an ex-post evaluation for the Brittany region, and through examination of the mid-term evaluations for France in 2003. While the paper does find some improvements in the quality of evaluation in the recent mid-term reports, it also identifies continuing problems. These include a weakness in the ‘logical diagram of impact’, tracing the chain of causality from actions to impacts, so that the paper argues for a more-detailed model of intervention.

Ziel 2 Ex-post-Evaluierung Regionalpolitik Frankreich

Baslé M. (2006) Ventajas e inconvenientes de los métodos de evaluación de la política de la UE: evaluación ex-post del Objetivo 2, 1994–99, Regional Studies 40, 225–235. En este artículo analizo la evaluación ex-post del Objetivo 2 a nivel regional, centrándome en la experiencia francesa durante el periodo 1994–99. Estas evaluaciones fueron ambiciosa y observando los informes, se identifican sus fortalezas y debilidades. En este documento se estudian los problemas que afectan a los Fondos Estructurales – tales comoretrasos en los ajustes, causalidad, medición y comprobación de resultados usando datos nacionales – y que se analizan poniendocomo ejemplo un caso práctico de una evaluación expost para la región de Bretaña y examinando las evaluaciones intermedias para Francia en 2003. Aunque se observan algunas mejoras de la calidad de evaluación en los últimos informes intermedios, también seidentifican problemas que ocurren de un modo continuo. Este sería el caso de las flaquezas en el ‘diagrama logico de impacto’. Analizando la cadena de causalidad desde las actuaciones a los impactos, se arguye que es necesario un modelo de intervención más detallado.

Objetivo 2 Evaluación ex-post Desarrollo regional Francia

Acknowledgements

The author is indebted to the referees and journal editors for valuable assistance in finalizing the latest version of the paper. He is also grateful to the Centre for Strategy and Evaluation Services for providing the Objective 2 evaluation report prepared for the European Commission. All remaining errors and imprecision remain the responsibility of the author.

Notes

1. The merits and drawbacks of each technique was examined (e.g. problems of comparators, issues of performance for programme management and the risk of response bias).

2. Using exchange rates at purchasing power parity for GDP in the former case.

3. Where were the European funds allocated and for what kinds of programme or project? There were also questions of absorption capacity for high (or higher) levels of funding, and whether the local effect could be estimated.

4. For example, are the priorities and measures standardized? How well do the European Commission, Member States and regions coordinate their functions? And how well is good practice disseminated?

5. The latter is measured in purchasing power parity.

6. This is the third of the threefold typology of regions given above. They received 11% of funds and generated 13% of the jobs.

7. As opposed to the additionality of the Funds (i.e. which should not substitute for national funding).

8. At the start of 2006, DATAR was reorganized and renamed ‘Délégation Interministérielle à l'Aménagement et à la Compétitivité des Territoires’ (DIACT).

9. This seems reasonable, but there was no analysis of impact of widely disseminated ‘soft’ projects.

10. The questionnaire was sent to all 1963 beneficiaries, yielding 714 responses.

11. In detail, the chain consists of interventions, leading to activities and operations, followed by an implementation process, intermediate results and final outcomes in the dimension of goals (official objectives) and emergent effects (systemic impact).

12. For example, a satisfaction index of (say) 85% could indicate many things, e.g. satisfaction with the presence of aid, the real satisfaction of a need, the pleasure from a windfall profit, etc.

13. For details, see http://www.datar.gouv.fr/datar_site/datar_europesom.nsf

14. For consideration of these kinds of virtuous circle, see Suire (2005) and Boschma (2005), for example, on interactive and creative clusters.

15. More innovative regions have the capacity to transfer technology, provide coherent training for workers, networks externalities, local capture and provide a ‘reputational’ feedback mechanism.

16. As at November 2005.

17. A good example is the official French software for Structural Funds, PRESAGE, noted above.