Mechanisms Shaping an Evaluation System—A Case Study of Poland 1999–2010

Abstract

A criticism of evaluation practice in regional policy in Central and Eastern Europe (CEE) countries is that it stresses its function in terms of accountability and formal reporting on progress, rather than its role in policy or institutional learning. This article aims to test this critique through the use of ‘system thinking’ and focuses on the case of regional policy in Poland, where evaluation practice has developed dynamically since 2000. It contributes to the literature by developing an analytical framework that sets out the role of evaluation in regional policy learning and identifies key factors and mechanisms that determine the development of an effective evaluation system for regional policy. The analysis draws on all Polish evaluation studies conducted between 1999 and 2010, interviews with civil servants and experts involved in the implementation of cohesion policy programmes, as well as secondary data from earlier studies. The essay identifies key mechanisms and factors that determine the main function of evaluation as a learning tool that produces and utilises knowledge in the decision-making process. These main elements are: growth in funding of the policy; stability of institutions; motivators; and system architecture.

Notes

¹ The main methods used here are logic models and macro-economic modelling.

² This function has been supported mainly by the participatory evaluation approach (Cousins & Whitmore 1998; King 1998).

³ This purpose is addressed by empowerment evaluation (Fetterman & Wandersman 2007).

⁴ This purpose of evaluation has been a focus of the theory-driven approach (Astbury & Leeuw 2010; Donaldson 2007; Olejniczak 2011) and developmental evaluation (Patton 2010).

⁵ Parts of the data come from two ex post evaluation studies executed for the European Commission–DG Regio (Kozak et al.2009) and for the Polish Ministry of Regional Development–NEU (EGO s.c. 2010b). The first study included 34 interviews with two groups of interviewees: heads of agencies and their departments involved in the implementation of programmes co-funded by structural funds; and Polish experts in public administration and regional development. The second study included 88 interviews with three groups. The biggest group of interviewees included key civil servants responsible for the coordination and implementation of operational programmes. Two other groups were Polish experts in public administration and representatives of key NGOs involved in cohesion policy in Poland. Below we will refer to these sources as ‘Interviews 2009’ and ‘Interviews 2010’.

⁶ For example, regional policy is a system of its own—with a clear function of ‘supporting regional development’. However, it is also a sub-system within a bigger entity of public policies, which has a function of supporting overall national growth and wellbeing.

⁷ For example, events include elections, civil servants' rotation and delay in EU expenditure in a particular year; visible trends are the increasing rotation of civil servants every four years and the repeated problem of fund absorption. An example of an underlying, causal mechanism is when new elections trigger changes in civil service personnel, and those who are most experienced in EU management leave or are removed by new politicians. That in turn, after a delay, causes a problem with EU fund management and absorption.

⁸ To illustrate these connections in a graphical way causal-loop diagrams can be used (Lane 2008).

⁹ This in practice means choosing between competing paradigms of regional development and deciding on overall funds allocation. In the field of cohesion policy, it is obtained using three processes: diagnosis and planning (including designation of assisted areas); consultations and negotiations; and coordination.

¹⁰ In cohesion policy it covers three fields of activities: communicating and promoting the availability of funding for potential beneficiaries; running the application and selection process of projects; and allocating funding and ensuring financial transfers.

¹¹ Strategies and programmes are usually treated as theories that are verified by life. Reflection focuses on proving a causal relation between what was implemented and what actually has changed in reality for the target group, and explaining why and how things worked (Astbury & Leeuw 2010).

¹² There is no database of evaluations of single projects in Poland. Although they are numerous they have little policy relevance. They are usually performed as an ESF obligatory requirement and part of projects' implementation, with little attempt to trace effects and explain projects' theories.

¹³ Furthermore, the perception of the Polish ministry also evolves in this direction. Initially, the bulk of EU funding was labelled in Poland as regional policy; later it turned into cohesion policy and presently it is described as development policy. This change in wording mirrors changes in the philosophy regarding EU-funds as part of integrated development-oriented national activities.

¹⁴ At the beginning of each programming period annual financial allocations are made to each cohesion policy programme. Programmes are then required to spend funds by the end of the second, or in some cases third, year following the year in which they are allocated. These targets are called the n+2 and n+3 rules.

¹⁵ For example, in the financial package for the 2007–2013 period, funds can be contracted until 2013 and spent until 2015. In the 2004–2006 budgetary period, the Commission granted new member states a three year extension of funding (until 2009), instead of two years.

¹⁶ It is worth pointing out that it was only in 2009 that the first attempt was made to start a policy discussion on development needs that was not driven by the EU funds perspective (Boni 2009).

¹⁷ The review was done on the basis of data from media monitoring contracted by the Ministry of Regional Development. It covered all articles from the 2002 to 2009 period related to EU funds for 2004–2009 and published in four newspapers: Rzeczpospolita, Gazeta Wyborcza, Polityka and Wprost. Analysis of the articles' contents was carried out by Evaluation for Government Organizations (EGO s.c. 2010b).

¹⁸ Putting this in context, for the period 2004–2009 the system consisted of 65 institutions and 4,500 civil servants. The system was almost entirely developed within a period of 36 months. In 2007 it extended further because of the new programming period and the introduction of a decentralised implementation system for regional operational programmes.

¹⁹ In 2005, 70% of active units were experienced. In the following three years these were: 50%, 48% and 43%, respectively.

²⁰ Units were created in all 16 regional marshall offices. They focused on evaluation of the regional operational programmes in their regions.

²¹ These are notably: the NEU (Krajowa Jednostka Oceny), the European Social Fund Unit (both located in the Ministry of Regional Development) and the Evaluation Unit in the Polish Agency for Enterprise Development.

²² In 2010 only 22% of operating units had no experience in constructing or supervising evaluations.

²³ The most extreme examples from the pre-accession period consisted of over 100 evaluation questions raised in a single evaluation study. The standard for the first period were ToR with up to 20 research questions. Currently ToR prepared by the NEU usually include between five and seven key questions.

²⁴ For an extensive, empirical discussion on the importance of balance between these four elements see Ferry and Olejniczak (2008, pp. 63–66).

²⁵ Interviews 2010.

²⁶ The only exception is the Academy of Evaluation organised by EUROREG-University of Warsaw and the Ministry of Regional Development. This one year study programme is co-financed by the EU European Regional Development Fund and it is designed exclusively for civil servants.

²⁷ The clarity of language has been measured by the number of indicators of which a ‘Fog index’ is the most interesting one. It measures the length of phrases and their structure—the ratio between nouns and verbs. The shorter the phrases and the more verbs there are, the less foggy the message is and more understandable to the average reader. This technique is based on the ‘Plain English standard’.

²⁸ See footnote 5.

The author gratefully acknowledges valuable comments from the journal's reviewers as well as the support given at the stage of data collection by the staff of the National Evaluation Unit in the Polish Ministry of Regional Development and the Evaluation Section of the Polish Agency for Enterprise Development.