Organisational analytical capacity: Policy evaluation in Belgium^*

* The article presents (part of the) findings of PhD research conducted at KU Leuven, Public Governance Institute, Belgium.

Figures & data

Table 1 Evaluation activity in the Flemish cases, for the three dimensions.

Dimension of evaluation activity	Indicators	Evaluation activity in our cases
Incidence of evaluation practice	Exemplar of evaluation output (report)	– Cases that conducted minimum one evaluation: n = 18 – Cases without any evaluation intentions: n = 9 – Case with concrete plans to evaluate in the near future: n = 2
Regularity of evaluation practice	Number of evaluation reports on a yearly average	– Cases with maximum one evaluation per year, but also having years without evaluations: n = 11 – Case with four to ten evaluations each year: n = 7
Quality of evaluation practice	The extent of application of the following quality tools: (1) appointment of a steering committee or working group for the evaluations; (2) involvement of second readers for evaluation reports and interim versions; (3) involvement of stakeholders in the evaluation process; (4) conduct of meta-evaluations, which assess previous evaluation studies; (5) use of evaluation manuals. (6) other tools	– Cases with minimum score of 7: n = 5 – Case with lower score than 7: n = 13
	Scores per tool: 0 (never applied), 1 (occasionally applied) and 2 (systematically applied for all evaluations). The sum of the scores constitute a composite indicator.

Table 2 Necessary conditions for the incidence of evaluation.

	Conducting evaluations	Not conducing evaluations, or having plans to conduct evaluations	Not conducting evaluations, and not having plans to conduct evaluations	Conducting evaluations or having plans to conduct evaluations
Evaluation supply
Evaluation budget
Staff with evaluation expertise
Capable external evaluators
Evaluation unit
Availability of monitoring information
Skills to conduct evaluations in-house		0.63
Skills to outsource evaluations	✓ 0.69			✓ 0.77
Evaluation demand
Evaluation demand from sector minister and his/her advisers		0.67	0.60
Evaluation demand from organisational management				✓ 0.90
Evaluation demand from parliament
Evaluation demand from other organisations
Evaluation demand from civil society organisations
Legislative evaluation requirements		(0.48)	(0.39)
Organisational support for evaluations
Nature of the organisation
Organisational size
Organisational status
Salience
Measurability of outputs		✓ (0.42)	✓ (0.38)
Measurability of outcomes
Path of organisation
Pre-reform evaluation experience		0.79	0.64
Organisational stability
Ministerial stability

Table 3 Necessary conditions for evaluation regularity.

	Regularly conducting evaluation	Not regularly conducting evaluation
Evaluation supply
Evaluation budget
Staff with evaluation expertise
Capable external evaluators
Evaluation unit
Availability of monitoring information
Skills to conduct evaluations in-house
Evaluation demand
Evaluation demand from sector minister and his/her advisers
Evaluation demand from organisational management		✓ 0.69
Evaluation demand from parliament
Evaluation demand from other organisations
Evaluation demand from civil society organisations
Legislative evaluation requirements		0.83
Organisational support for evaluations
Nature of the organisation
Organisational size
Organisational status
Salience
Measurability of outputs		✓ 0.71
Measurability of outcomes
Path of organisation
Pre-reform evaluation experience	✓ (0.47)
Organisational stability
Ministerial stability

Table 4 Necessary conditions for the application of evaluation quality measures.

	Applying evaluation quality measures	Not applying evaluation quality measures
Evaluation supply
Evaluation budget
Staff with evaluation expertise
Capable external evaluators
Evaluation unit	✓ 0.56
Availability of monitoring information
Skills to conduct evaluations in-house
Evaluation demand
Evaluation demand from sector minister and his/her advisers	✓ (0.38)
Evaluation demand from organisational management		✓ (0 75)
Evaluation demand from parliament
Evaluation demand from other organisations
Evaluation demand from civil society organisations	✓ (0.36)
Legislative evaluation requirements
Organisational support for evaluations
Nature of the organisation
Organisational size
Organisational status
Salience
Measurability of outputs
Measurability of outcomes
Path of organisation
Pre-reform evaluation experience	✓ (0.33)
Organisational stability
Ministerial stability

Table

Conditions	Code 0 indicators	Code 1 indicators
Evaluation supply
(i) Evaluation budget (ii) Staff with evaluation expertise (iii) Capable external evaluators	Totally insufficient, rather insufficient	Rather sufficient, fully sufficient
(iv) Evaluation unit	No evaluation unit	Formal or de facto evaluation unit
(v) Availability of monitoring information (vi) Skills to conduct evaluations in-house (vii) Skills to outsource evaluations	Totally insufficient, rather insufficient	Rather sufficient, fully sufficient
Evaluation demand
Extent of evaluation demand from: (viii) Sector minister and advisers, (ix) Organisational management, (x) Parliament, (xi) Other organisations, (xii) Civil society organisations	No demand, hardly any demand	Sometimes demand, frequent demand
(xiii) Legislative evaluation requirements	No evaluation requirements in Flemish or European Union legislative documents	Evaluation requirements in Flemish or European Union legislative documents
(xiv) Extent of organisation-wide support for evaluations	Not at all, to limited extent	To major extent, to large extent
Nature of the organisation
(xv) Organisational size	Very low, low material weight ^a	At least average material weight
(xvi) Organisational status	Agency	Department
(xvii) Salience	Not at all, limited, rather limited	Rather much, much, very much
	Highest score of the assessments of extent of attention by media or parliament
(xviii) Perceived measurability of outputs, and (xix) Outcomes	Average score ≤3 and/or qualification: very difficult, difficult, rather difficult	Average score 3 and/or qualification very easy, rather easy, easy
	Average score of measurability on a scale of 1 (very difficult to measure) to 5 (very easy to measure) of the three most important outputs and outcomes of the organisation.
Path of organisation
(xx) Pre-reform evaluation experience	No/seldom evaluation practice prior to the NPM oriented reforms	Sometimes/frequent evaluation practice prior to the NPM oriented reforms
(xxi) Organisational stability	Organisations which underwent medium or large changes ^b	Organisations which underwent no or small changes
(xxiii) Ministerial stability	1 minister changes since reforms	No ministerial turnover

a This condition covers two indicators of equal importance: financial material weight and material weight with regard to personnel. The operationalisation is based on information provided by the Internal Audit of the Flemish Administration.

b This condition covers two indicators: changes as a result of the implementation of the NPM oriented reforms (weight: 60%), and changes independent from it (weight: 40%). The operationalisation is based on information provided by the Internal Audit of the Flemish Administration.

Table

Cases ^a	Dimensions of evaluation activity				Supply
	Conducting evaluations	Not conducting evaluations, and not having plans	Regularly conducting evaluations	Applying evaluation quality measures	Evaluation budget	Staff with evaluation expertise	Capable external evaluators	Evaluation unit	Monitoring information	Skills to conduct evaluations in-house	Skills to outsource evaluations
INTA2	1	0	0	0	1	1	0	0	1	1	1
DEPT7	1	0	0	0	1	0	1	0	0	1	1
INTALP3	1	0	0	0	1	1	0	1	1	1	1
INTA9	1	0	1	0	1	1	1	0	1	1	1
INTALP1	1	0	0	0	1	1	0	1	1	1	1
EXTA6	1	0	0	0	0	0	1	0	1	1	1
INTA5	1	0	0	0	0	0	0	0	0	1	1
DEPT3	1	0	1	1	1	1	0	1	1	1	1
EXTA2	1	0	0	0	1	0	1	0	1	1	1
EXTA3	1	0	1	0	1	1	1	0	1	0	1
DEPT11	1	0	1	1	0	1	1	1	1	1	1
INTALP2	1	0	0	1	0	1	0	1	1	0	1
EXTA1	1	0	0	0	1	0	1	1	0	0	1
DEPT1	1	0	0	0	0	0	1	0	0	0	1
DEPT5	1	0	1	0	1	1	1	1	1	0	1
DEPT2	1	0	1	1	0	0	0	1	0	1	1
DEPT8	1	0	1	1	0	0	0	1	0	0	1
DEPT6	1	0	0	0	0	0	1	0	0	1	1
INTA7	0	1	NA	NA	0	0	1	0	0	0	1
INTA6	0	1	NA	NA	1	0	1	0	0	0	1
INTA1	0	1	NA	NA	0	0	0	0	0	0	0
EXTA4	0	1	NA	NA	0	0	1	0	0	0	1
INTA3	0	1	NA	NA	1	0	1	0	0	0	0
INTA8	0	1	NA	NA	0	0	0	0	0	0	1
DEPT4	0	1	NA	NA	1	1	1	1	1	0	1
EXTA5	0	1	NA	NA	0	0	1	0	0	1	1
INTA4	0	1	NA	NA	1	0	1	0	1	0	0
DEPT10	0	0	NA	NA	0	1	1	1	0	0	1
DEPT9	0	0	NA	NA	0	0	0	0	0	0	1
Cases ^a	Demand							Nature of organisation					Path
	Demand from sector minister	Demand from management	Demand from Parliament	Demand from other organisations	Demand from Civil society	Legislative evaluation requirements	Support	Size	Status	Salience	Measurability of outputs	Measurability of outcomes	Pre-reform evaluation experience	Organisational stability	Ministerial stability
INTA2	0	1	0	0	0	0	1	1	0	1	1	0	1	0	1
DEPT7	1	1	1	0	1	0	0	1	1	1	1	1	1	0	1
INTALP3	0	1	0	1	1	1	1	1	0	1	1	0	1	0	0
INTA9	1	1	0	0	1	1	1	0	0	1	1	1	1	0	0
INTALP1	1	1	0	0	0	0	1	0	0	0	1	1	1	1	1
EXTA6	1	1	1	0	1	0	1	0	0	1	0	1	1	1	0
INTA5	1	1	1	1	1	0	1	1	0	0	1	0	1	0	1
DEPT3	1	1	0	1	1	1	0	0	1	0	1	1	1	0	0
EXTA2	1	1	1	1	1	0	0	1	0	1	1	0	1	0	1
EXTA3	1	1	0	1	0	0	1	1	0	1	0	0	1	0	1
DEPT11	1	1	0	1	1	1	1	0	1	1	1	0	1	0	1
INTALP2	1	1	0	0	1	0	1	1	0	1	1	0	1	1	0
EXTA1	1	1	0	0	0	0	0	1	0	0	1	1	0	0	0
DEPT1	0	1	1	0	1	0	0	1	1	0	1	0	0	0	1
DEPT5	0	0	0	0	1	0	0	1	1	1	0	0	1	0	0
DEPT2	1	1	0	1	1	1	1	1	1	1	0	0	1	1	0
DEPT8	1	0	1	0	1	1	0	1	1	1	1	0	1	1	1
DEPT6	0	1	1	1	1	0	1	1	1	0	1	0	0	0	0
INTA7	0	0	0	0	0	0	0	1	0	0	1	0	0	0	1
INTA6	0	0	1	0	0	0	1	1	0	1	1	0	0	0	0
INTA1	0	1	1	0	0	0	1	1	0	1	1	0	0	1	0
EXTA4	0	0	0	0	0	0	0	1	0	1	1	0	0	1	0
INTA3	0	0	0	0	1	0	0	0	0	0	1	0	0	0	1
INTA8	0	0	0	0	0	0	1	0	0	1	1	0	0	0	0
DEPT4	0	0	0	0	0	0	0	1	1	0	1	0	0	0	0
EXTA5	0	0	0	0	0	0	0	1	0	0	1	1	0	1	1
INTA4	0	1	1	1	1	0	1	1	0	1	1	1	0	0	0
DEPT10	0	1	0	1	1	0	0	1	1	1	0	0	0	0	0
DEPT9	1	1	0	0	0	0	0	1	1	1	1	0	0	0	1

a Case codes are composed by a letter combination and a number. DEPT stands for Department; EXTA for external agencies of public nature; INTA for internal agencies without legal personality; INTALP for internal agencies with legal personality. The numbers are chosen at random.