Harder governance built on soft foundations: experience from OECD peer reviews

ABSTRACT

The OECD international peer reviews represent a typical example of ‘harder soft governance’ at the international level. Lacking direct regulatory power, the Organisation exerts its influence via soft persuasion through peer pressure, entailing attention from the media, civil society, and other member country governments. Drawing on interviews and documentary material from the period 1996 to 2017, and participant observation from 1996 to 2005, this article examines the implementation of various measures of ‘hardening’ in three OECD peer reviews: the Environmental Performance Reviews and Economic Surveys carried out by the OECD proper, and the Energy Policy Reviews of the OECD International Energy Agency. The three reviews differ along the different dimensions of hardness, yet none of the reviews can be unambiguously classified as the hardest. The article highlights the multiple trade-offs between different types of hardening, as well as between hardness and softness. Hardening is not an end in itself and does not necessarily equate with greater impact. Greater attention should be paid to the directionality and multiple pathways of review influence, the roles of reviews in the service of socialisation, norm-creation and policy advocacy, and the essential function of soft elements as a foundation for hardening.

KEYWORDS:

• Energy policy
• peer pressure
• international organisations
• soft governance
• international peer review

Introduction

International peer reviews conducted by the Organisation for Economic Co-Operation and Development (OECD) and its Special Bodies, such as the International Energy Agency (IEA), are among the most advanced examples of ‘soft governance’ at the international level. Carried out on the member countries’ economic policies since the creation of the OECD in 1961 – and later in practically all areas of OECD activity – peer reviews constitute the very working method of the OECD, its ‘trademark’ (OECD, 2003; Sharman, 2005; Woodward, 2009), and a vital element of the organisation’s identity. The OECD peer reviews lack the traditional characteristics of hard governance: coercive power and sanctions for non-compliance. By contrast, they constitute one of the earliest examples of policy surveillance, combining monitoring, evaluation, and reporting (see Schoenefeld & Jordan, in press). ‘Peer pressure’ (e.g. Guilmette, 2007; OECD, 2003), would spur countries to voluntarily adopt recommended policy measures, fostering socialisation by individuals and countries keen to adhere to principles and values shared by the Western international community.

However, if ‘hardness’ and ‘softness’ are understood as two poles of a continuum, as argued by Knodt and Schoenefeld (2020), OECD peer pressure – and peer reviews as its concrete manifestation – relies on varying and context-dependent constellations of ‘softer’ and ‘harder’ elements of governance. Discussions on an appropriate balance between hard and soft are indeed frequent in the OECD technical bodies responsible for the reviews. Those calling for hardening point at the well-known weaknesses of soft governance: lacking coercive and sanctioning power, voluntary measures whose implementation depends on the will and engagement of the member countries often fail to bring about policy change (e.g. Schoenefeld & Jordan, in press; Ringel & Knodt, 2018). While earlier research has shown that OECD soft governance through peer reviews has had little direct impact on member countries’ policies (Armingeon, 2003; Lehtonen, 2005a, 2005b, 2007, 2009a, 2009b), the reviews can exert indirect influence, via multiple pathways such as gradual learning through socialisation (Marcussen, 2003), creation of ‘epistemic communities’ (Armingeon, 2003), shaping of problem formulation and policy agendas, dissemination of OECD ‘organisational discourse’ dominated until recently by mainstream, liberal economics (Dostal, 2004; Mahon & McBride, 2008, 2009), and by pushing states to conform to new norms and practices (Mahon & McBride, 2009). Peer pressure has proven most effective in ‘laggard’ states, because a country seldom wants to risk losing its credibility and being seen in an unfavourable light among its peers (Lehtonen, 2005a; Mahon & McBride, 2009, p. 88; Marcussen, 2003; Woodward, 2009, p. 68).

The article draws on illustrative examples from the OECD peer reviews, highlighting the choices and trade-offs involved in efforts at ‘hardening’. In EU energy, environmental and climate policies, no OECD-type peer reviews exist, yet the past years have seen a rapidly growing political support and demand for evaluation (Schoenefeld & Jordan, 2019), in particular, as part of the EU Energy Union surveillance mechanisms, analysed in other articles of this special issue, and partly spurred by monitoring and reporting obligations under the UN Framework Convention on Climate Change (Knodt & Schoenefeld, 2020; Schoenefeld & Knodt, 2020). This article contributes to the discussion by identifying types of harder elements in three OECD’s peer review programmes, and examining the various hardening-related review design choices and discussions thereof within the bodies responsible for the reviews. Drawing on the typology of ‘hardening’ outlined by Knodt and Schoenefeld (2020), I will highlight the trade-offs between the more conventional types of hardening (obligation; precision; justification; and blaming and shaming) and two more novel types of hardening: ‘bundling within a policy field’ and ‘enforcement by coupling with other policy fields’. The analysis underlines the limits of a dichotomous presentation of hard vs. soft governance, and points out key governance choices within the extensive grey area consisting of various combinations of ‘hard’ and ‘soft’ (Knodt et al., 2020; Trubek & Trubek, 2005). I will argue that the implicit nature of peer reviews as ‘advocacy evaluation’ (Arensman & van Wessel, 2018) crucially shapes the ‘soft vs. hard’ choices, and that hardening by an organisation such as the OECD rests upon the ‘soft’ foundations of trust, socialisation and community building.

Three OECD peer reviews provide the empirical material for the illustration: the Economic Surveys and the Environmental Performance Reviews (EPRs) – both carried out by the OECD – and the Energy Policy Reviews conducted by the International Energy Agency (IEA), a semi-autonomous body within the OECD ‘family’. The main insights come from the two OECD reviews, whereas examples from the IEA peer reviews illustrate topics specific to energy and climate policies. The empirical cases indeed serve for illustration, not for comparison. For the EPRs and Economic Surveys, the analysis draws mainly on documentary material concerning the OECD work and its reviews, including work on the role of the EPRs in policymaking, published elsewhere (Lehtonen, 2005a, 2005b, 2006, 2007, 2008, 2009a, 2009b). The research underpinning these publications included: participant observation (May 1996–May 2005) as a country delegate in the OECD Working Party for Environmental Performance (WPEP), responsible for the EPRs; an opinion survey amongst WPEP members in preparation for the second cycle or reviews (OECD, 1998); coordination of the Finnish EPR in 1996–1997; and participation as a country expert on the EPR teams reviewing Mexico (1997–1998) and Russia (1998–1999), and as an OECD consultant on the review of Sweden (2003–2004). Forty individuals involved in the ERPs and/or Economic Surveys were also interviewed: OECD secretariat experts, the WPEP delegates of Canada, France, Hungary, Japan, Mexico, the Netherlands, Portugal, and Slovakia; and other stakeholders in Finland, France, the Netherlands, and Portugal. For the energy policy reviews, ten interviews were conducted in 2010 with representatives of the secretariat as well as the reviewed or reviewing countries. The interviews are in the following used as appropriate to complement the documentary material.

The next section presents the OECD bodies involved and outlines the key features of the three peer reviews. Section three describes the types of hardening analysed in the article. Section four analyses the design choices and associated discussions concerning the review programmes in light of the selected types of hardening. Section five discusses the findings and concludes by summarising the key trade-offs and underlining the advocacy function of the reviews. Section six concludes by addressing the potential relevance of the findings for EU energy and climate policy.

The OECD bodies and peer reviews

The OECD was created in 1961, on the foundations of its predecessor, the Organisation for European Economic Cooperation (OEEC), set up in 1948 to execute the distribution of Marshall Aid (e.g. Guilmette, 2007; Mahon & McBride, 2008; Woodward, 2009). Today, with its 36 member countries, the OECD is a leading forum for officials from industrialised member and non-member countries to deliberate on topics in virtually all areas of public policy. Promotion of economic liberalism remains OECD’s core mission but, is, today combined with an increasing emphasis on global fairness, equity, and environmental protection (OECD, 2019).

The OECD is simultaneously a think-tank, databank, problem-solver, pathfinder, policy adviser, forum and meeting place – and sometimes as a deal broker (e.g. Julin, 2003; Marcussen & Trondal, 2011; Woodward, 2009). Despite its long-standing reputation as a ‘rich man’s club’, the OECD is far from monolithic (Mahon & McBride, 2009). It is generally seen as a relatively open and adaptive organisation, known for its impartial analysis, professionalism, and innovation (Gass, 2003; Marcussen & Trondal, 2011; Thygesen, 2008). Only a minor fraction of its committee work consists of preparing decisions for the OECD Council (Marcussen, 2003).

The IEA was founded in 1974, in the wake of the first oil crisis, to enable Western oil-consuming countries to manage similar future crises. From its early market-regulating and interventionist role in support of its founding mandate – energy security – the IEA moved in the 1990s closer to the OECD economic liberalism, yet endorsing state regulation especially to promote innovation and environmental objectives (Van de Graaf & Lesage, 2009). IEA has today 30 member countries.

Peer review as the OECD trademark

The OECD (2003, p. 9) defines international peer review as ‘the systematic examination and assessment of the performance of a State by other States, with the ultimate goal of helping the reviewed State improve its policy making, adopt best practices, and comply with established standards and principles’. Ideally, the review process would be characterised by dialogue and interactive investigation, in a non-adversarial setting, relying on mutual trust between the states involved as well as on shared confidence in the process, thus creating a system of mutual accountability (OECD 2003). Aldy (2014, p. 283) describes OECD peer review as a ‘facilitative process’, which allows a more ‘candid dialogue among participants’ than compliance mechanisms conducted under the ‘shadow of hierarchy’. Peer reviews do have also an ‘inquisitive’ character, designed to influence domestic public opinion, national administrations and policymakers through pressure from media, public scrutiny, dialogue and comparisons with peer countries (OECD, 2003, p. 9; Mahon & McBride, 2009). As the ultimate reviewers, delegates from other member countries can pose critical questions to their reviewed country peers (Carraro et al., 2018).

The reviews build on ownership and intensive scrutiny (Woodward, 2009, p. 69). Ownership is achieved via the involvement of the reviewed country government in the process, and is built over time via personal relationships and loyalties amongst country delegates involved (Woodward, 2009, p. 68; Marcussen & Trondal, 2011). At the same time, maintaining an image of independent review is crucial for the credibility of the process.

The three OECD reviews

The OECD Economic Surveys, conducted since 1961 by the Economic and Development Review Committee (EDRC), are published every 18–24 months for each OECD member country. The powerful OECD Economics Department coordinates the Economic Surveys.

The Environmental Performance Reviews (EPRs) were launched in 1992, under the auspices of the Working Party on Environmental Performance (WPEP) and coordinated by the Environment Directorate. The EEA (2005, p. 27) has described the EPRs as the most systematic international effort to report on and assess the integration of environmental and socio-economic policies. Largely for lack of secretariat resources, the reviews are conducted at long intervals of 8–9 years. The third cycle of reviews is to be finalised in 2021 (OECD, 2018).

Systematic IEA country reviews started in 1977, and are conducted at 4–5-year intervals, under the responsibility of the Standing Group on Long-Term Co-Operation (SLT). The IEA also reviews selected non-member countries, either on their energy policy as a whole or on a specific topic (e.g. the coal market, electricity sector reform, cleaner coal) (IEA, 2010).

All three review programmes also review selected key non-member countries.

The sequence of an OECD peer review

All three peer review processes follow a similar sequence of six phases (e.g. Guilmette, 2007; IEA, 2010; OECD, 2003; 2018; Woodward, 2009):

1. Setting up a review team, which prepares a desk study relying to a greater or lesser extent on material provided by the reviewed country government.

2. Review mission(s), the reviewers meeting various stakeholders in the reviewed country.

3. Drafting of the review report by the secretariat and the country experts, with varying degrees of consultation with the reviewed country.

4. Peer review meeting in Paris, other member countries’ delegates posing questions to the reviewed country delegation.

5. Publication of the review report

6. Follow-up of the implementation of the recommendations.

To support the analysis of performance, the reviews rely on the extensive OECD data and indicators.

Five stakeholder groups are central for the peer reviews (e.g. OECD, 2003; Lehtonen, 2005a, pp. 219–220):

• o the unit managing the review programme within the OECD/IEA secretariat

• o the member country government, especially its relevant ministry as the main ‘client’ of the reviews

• o the relevant OECD/IEA committee, which assembles for the peer review meetings to discuss and approve the individual review conclusions and recommendations, and to manage the review programme

• o the higher-level OECD/IEA policy committees that supervise the technical committees and take organisational and budget decisions

• o stakeholders in the reviewed country: other ministries, industry, NGOs, policy researchers and the public at large.

The three reviews also differ between each other in several respects, including the types of hardening of soft governance, discussed in the following.

Conceptual framework: types of ‘hardening’

Some of the categories of hardening identified by Knodt and Schoenefeld (2020) are not relevant for the OECD peer reviews. The reviews neither delegate political power to third party actors nor do they entail sanctioning. Private sector actors are involved in peer reviews, but decisions are made by the national governments, on the principle of consensus. By contrast, six other types of hardening are central in OECD discussions on peer reviews, namely obligation; justification; precision; blaming and shaming; bundling within a policy field; and enforcement by coupling with other policy fields.

Obligation through binding rules and commitments is not part of the OECD peer review logic. However, in the alternative meaning as the degree of ‘hardness’ in its formulation and wording, (Knodt & Schoenefeld, 2020), obligation is continuously present in the debates on the wording of review conclusions and recommendations. Justification, that is, the degree to which the states must justify their decisions concerning the recommendations, is frequently evoked in OECD discussions, as is precision in the definition of policy recommendations.

Although not officially part of the peer review logic, blaming and shaming is implicitly ever-present in OECD debates. Knodt and Schoenefeld (2020) define this type of hardening as the multiplication of arenas and platforms where ‘blaming and shaming’ could occur, via public exposure or peer scrutiny. Comparison and ranking of state practices would bring transgressor countries to compliance via the sheer ‘discomfort’ produced (Chayes et al., 1998; Mahon & McBride, 2009, p. 86). In this top-down logic of soft governance, ideas developed at the international level would gradually shape national and sub-national level policy, through blaming and shaming and via dissemination of best practice and new concepts (Trubek & Trubek, 2005, pp. 356–358).

‘Bundling within a policy field’ and ‘enforcement by coupling with other policy fields’ are close to Trubek and Trubek’s (2005) bottom-up perspective to soft governance. It stresses experimentation, deliberation and learning, which would help destabilise existing understandings, foster inter-sectoral and multistakeholder deliberation and confrontation of perspectives, transform policy networks, encourage policy experimentation, and spur actors to collectively redefine policies and objectives. Indeed, the OECD peer review logic relies on the belief that tangible policy impacts can be produced not only through coercion and regulation but above all through ‘ideational’ or ‘framing’ processes (López-Santana, 2006, p. 482). Hardening via ‘bundling’ means streamlining of different policy elements in a given policy area (e.g. climate, energy, or environment) to strengthen policy cohesion, synergies, as well as visibility and prominence of the policy field. In ‘coupling’, a soft policy field is linked with a ‘harder’ field that has sanction potential, e.g. through financial means. These types of hardening evoke two recurrent OECD peer review terms, namely policy integration and mainstreaming (e.g. Lehtonen, 2007, 2008, 2009b). In the following, I will combine these under the category of ‘bundling and coupling’.

Harder soft governance in the OECD peer reviews

This section describes those review features that most directly relate to the types of hardening outlined by Knodt and Schoenefeld (2020) and to a certain degree differentiate the three review programmes from each other.

Criteria of performance

The criteria for judging performance can be external or internal or a combination of the two. At one extreme, all countries would be judged against the same criteria – indicators derived e.g. from scientific evidence or international agreements, and the fulfilment of recommendations from the previous review. The opposite option is to examine the country’s achievements in reaching its own policy commitments. External criteria would provide greater hardness by minimising the influence of the reviewed country on criteria selection, hence enhancing objectivity, reducing political meddling in the process, allowing cross-country comparability, and facilitating ‘blaming and shaming’. It would also contribute to ‘precision’, by reducing ambiguity from country-specific and possibly vaguely defined criteria.

All three reviews make extensive use of indicators, mainly as background information. The EPRs are closest to the internal, ‘softer’, option, as performance is primarily measured against the country’s own objectives. For the energy reviews, the IEA ‘Shared Goals’ from 1993 provide a clear external backing. In the Economic Surveys, similar external criteria derive from the relatively homogeneous and coherent set of epistemological beliefs amongst the community of economic experts and officials involved (e.g. Lehtonen, 2005a, pp. 220–224), and from the fulfilment of recommendations of the previous review (Porter & Webb, 2008, p. 49). All reviews contain an economic element, but not quite in the same way. The Environmental Performance Reviews (EPRs) examine the cost-effectiveness of policies, without calling into question the country’s policy objectives, whereas the Economic Surveys go a step further towards ‘hardening’, by questioning the policy objectives when these are deemed to undermine efficiency (Lehtonen, 2007). In the preparations for the second cycle of EPRs, the high expectations of greater comparability (OECD, 1998, p. 4) were soon toned down. Delegates notably from non-European countries feared excessive Europeanisation of OECD environmental policy approach (Lehtonen, 2005a, p. 206), while the secretariat sought to prevent a backlash from countries reluctant to accept strict indicator-based comparisons.

Review topics

Discussions in the OECD have evoked choices between comprehensive and focused reviews. The former would examine the country’s performance across the entire policy area, whereas the second would allow the country to select specific topics for review. A comprehensive review would be ‘harder’ to the extent that it enables the monitoring of performance trends, facilitates blaming and shaming via cross-country comparison, and minimises the reviewed country’s influence on the review framework. It can foster ‘bundling and coupling’ by engaging in the process a broad range of actors from various subfields of policy.

All three reviews have traditionally combined coverage of the entire range of the given policy area, and deeper analysis of selected topics. More tailor-made reviews have been called for especially in committees responsible for the EPRs and IEA reviews (e.g. IEA, 2010; OECD, 1998). In the EPR discussions, the problem of heterogeneity of member countries was evoked: countries with mature environmental policy institutions might see little merit in a comprehensive analysis of well-established sectoral (water, air, nature protection) environmental policies, unlike countries with less advanced institutions and analytical capacity (Lehtonen, 2005a, p. 210). Blaming and shaming hence came into tension with bundling and coupling. Similarly, in the early 2000s, a group of Anglo-Saxon countries (notably the USA, UK, Canada and Germany)¹ advocated a ‘focused’ approach to the energy reviews. An interviewed energy-sector specialist interpreted this partly as the desire of energy-resource-rich countries to maintain sovereignty over their energy policies. In a compromise decision, countries were allowed to choose between a traditional comprehensive review and a more in-depth review focused on two themes (IEA, 2010).

Roles of different actors in drafting the review

Hardening depends also on the respective roles of the secretariat, reviewing country experts, the committee responsible for the review, and the reviewed country – especially in drafting the conclusions and recommendations. Weaker influence of the reviewed country would enhance blaming and shaming, by favouring objectivity and preventing the watering-down of review findings. However, closer engagement of the reviewed country would foster bundling and coupling.

The EPRs are drafted collectively by OECD Environment Directorate officials, with contributions from country experts, and comments from other OECD directorates. The preliminary conclusions are discussed only among the review team members at the end of the mission, and the draft report is sent to the member countries (WPEP) about a month before the review meeting. The reviewed country can suggest minor changes. The commenting rounds within the reviewed country’s administration can contribute to ‘bundling and coupling’. To an extent, the WPEP can impose recommendations without the full consent of the reviewed country. By contrast, the Economic Surveys and energy reviews are drafted by the OECD ‘country desk’ officers. The reviewed country officials and OECD secretariat communicate intensively on the conclusions and recommendations, to ensure that these match the reviewed country government’s needs and desires (e.g. Guilmette, 2007; Thygesen, 2008). In the IEA reviews, the tentative policy conclusions are presented to the government already at the end of the review mission, and the SLT group receives the report only a few days prior to the review meeting. The energy reviews are therefore clearly a joint product of the secretariat and the reviewed country’s energy administration.² The Economic Surveys most explicitly engage with top policymakers and labour market organisations in the country, during the second of the two review missions. Also the EPRs have recently incorporated a high-level discussion on the preliminary findings and recommendations between the OECD Secretariat and the reviewed country, two months prior to the review meeting (OECD, 2018).

No country experts participate in the Economic Survey mission. By contrast, in both the EPRs and energy reviews, this ‘peer element’ is an important part of the mission, with two to four country experts typically participating on the review team. In the EPRs, the meetings during the review mission gather government officials from various policy sectors to discuss a specific theme (water, air, economics, etc.), whereas the energy and economic review mission meetings are stakeholder-focused and cross-thematic. In this sense, the EPRs have the greatest potential to advance ‘bundling and coupling’.

The EPRs would seem to offer the greatest hardness in that they most clearly limit member country influence. In energy and economic reviews, the dominance of the secretariat in drafting can compromise the ‘peer character’.³ This could weaken both ‘blaming and shaming’ and ‘bundling and coupling’, if the reviews are considered as products of experts discredited as technocratic, distant, and ignorant of the political complexities involved (Carraro et al., 2018).⁴

Greater influence by the reviewed country can mean hardening, when the country officials seek harder recommendations on specific topics, in order to obtain support from a prestigious international organisation for their preferred policies (e.g. Lehtonen, 2005a, p. 208; Porter & Webb, 2008).

Language of recommendations

The degree of ‘obligation’ in the wording of conclusions and recommendations receives major attention in all OECD reviews. The EPR debates provide an illustration. To build confidence, the tone in the recommendations tends to be relatively ‘soft’, consensual, diplomatic, and general (Lehtonen, 2005b, p. 182). To improve the usefulness of the reviews, delegates from mostly northern European countries with advanced environmental policies (esp. Australia, Canada, Finland, and the Netherlands) called for more ‘hard-hitting’ and ‘punchy’ formulations. The suggestion was resisted mostly by southern European countries. In addition to the ‘laggards’ vs. ‘forerunners’ divide, differences may reflect distinct discussion cultures. The ‘Anglo-Saxons’ considered vague exhortations as useless, arguing these would fail to attract attention by the public, media, and policymakers (e.g. Lehtonen, 2005a, pp. 206–209), while those from the south argued that more direct language could be interpreted as offending.⁵ At stake was the continuity of the EPRs, and their legitimacy in the eyes of the main clients, the national governments. ‘Hardening’ through ‘obligation’ and ‘precision’ would therefore either facilitate or undermine blaming and shaming, depending on the specificities of the reviewed country.

Peer review meeting

As a venue for peer scrutiny, the peer review meeting can contribute to blaming and shaming. The EPRs are in this respect the hardest of the three, as a whole day is devoted to the review meeting, as opposed to half-a-day in the Economic Surveys and two hours in the energy reviews. Other features give the EPRs a softer tone. In particular, the meetings tend to be somewhat ceremonial, following a rather tightly predefined ‘script’ prepared by the secretariat, which only intervenes in the discussions to provide factual clarifications. In the Economic Surveys, the discussion is more free-flowing, with delegates free to pursue their questioning on the points they consider as most critical, and the secretariat participating actively. The IEA review meetings dedicate only one hour to actual discussion and exchange of views.

In EPRs and energy reviews, delegates to the review committee are usually sent from the country capitals, whereas the Economic Survey delegates come from the countries’ permanent OECD delegations, because of the high frequency of economic committee meetings. Both constellations foster socialisation typical of OECD work, each in its distinct manner: the EPRs and energy reviews attend to policy integration (and hence ‘bundling and coupling’), whereas the Economic Surveys contribute to the internal cohesion of a tightly knit community of economic experts (e.g. Guilmette, 2007; Lehtonen, 2007).

The EPR meetings are slightly formal partly because the reviewed country delegations are led by a high-level civil servant or a minister. High-level representation enhances the perceived status and visibility of the review, thereby facilitating blaming and shaming. However, it may undermine the frankness and technical quality of the debates, hence reducing precision, if the delegates hesitate to ask critical questions. Yet, sometimes ministers are more receptive to critical analysis than civil servants with limited leeway to deviate from official policy discourse (Lehtonen, 2005a, p. 185).

Depending on the composition of the reviewed country delegation, the meeting can foster policy integration in the reviewed country, by bringing together civil servants and stakeholders working in a given policy area. This is true in particular for countries with less mature institutions in the given policy area.⁶

Frequency of the reviews

The EPR delegates frequently regret the excessive length of a full review cycle, which undermines the continuity of exposure to criticism, and thereby weakens blaming and shaming. During the up to 10 years between reviews of a given country, many of the key policymakers move position, policies change, and the previous review gets forgotten. The Economic Surveys represent the other extreme, with a review cycle of only 18–24 months, whereas IEA reviews have an intermediate-length cycle of 4–5 years. Downsides of a shorter review cycle evoked in the debates include ‘evaluation fatigue’ (Taut & Brauns, 2003) in national administrations struggling with increasing reporting obligations, and the long time lag between policies and impacts, especially in the environment sector. The desire to conduct non-member country reviews has pushed for the extension of the review cycle, e.g. in the energy reviews (IEA, 2010).

Follow-up and monitoring of review recommendations

Monitoring and follow-up of compliance with the recommendations from the previous reviews represent hardening via ‘justification’. Mandatory reporting on the implementation of recommendations obliges the country to justify, in front of its peers, possible failure to comply.

In the EPRs, reporting on the implementation of review recommendations is voluntary, and countries are merely ‘encouraged’ encouraged to present a follow-up report 5 years after the review (OECD, 2018). Each Economic Survey contains a section reviewing the compliance with the recommendations from the previous review. The long review cycle reduces the viability of this option in the EPRs, as holding policymakers accountable becomes more difficult with the passage of time. Brief annual reviews are published in the intervening years between the energy reviews, to update on the country’s energy policy developments and implementation of the recommendations.

Financing the review

Secure long-term funding provides a review programme independence and better conditions for hardening than one in which the reviews are at the mercy of economic trends and changing member country priorities.

As the flagship of the OECD, the Economic Surveys enjoy great budgetary freedom, as their existence is hardly questioned. Although the EPRs and energy reviews receive dedicated funding from the OECD budget, both need to pay keen attention to the varied and changing needs and desires of the member countries, in a context of continuing budget limitations (Marcussen & Trondal, 2011). The conclusions and recommendations must be tailored to match the perspectives and policy approaches of the reviewed country governments (Radaelli, 2000) – especially the greatest budget contributors.

Trade-offs and synergies between types of hardening and between ‘hard’ and ‘soft’: why not simply harden the reviews?

Most of the empirical material for the analysis presented in this article dates from some time ago, and the OECD reviews have evolved since then. The Environmental Performance Review process has been slightly streamlined, with e.g. only two peer reviewing countries instead of the earlier three to four represented on the review team, and the report outline modified.⁷ The IEA has sought to ‘modernise the reviews by focusing on some of the key energy challenges’ (IEA, 2019). The urgency of climate and sustainability challenges may increase calls for greater ‘hardness’. However, the fundamental philosophy and approach of OECD work in general and the reviews, in particular, remain unchanged, as do the underlying tensions, trade-offs and ambiguities highlighted in the preceding analysis. These condition the possibilities, forms, and usefulness of hardening. Hardening comes at a cost and a ‘harder’ review is not necessarily better and more influential than a ‘softer’ one. Crucially, ‘hardening’ has multiple and partly mutually contradicting elements, and therefore does not lend itself to measurement along a one-dimensional scale. The Economic Surveys and energy reviews are softer than the EPRs in that they more explicitly seek to cater to the needs of the reviewed country government – through intensive interaction during the drafting of the review conclusions and recommendations. Yet, this interaction may enhance rather than undermine the impact and policy relevance of these reviews in the member countries. The Economic Surveys may be somewhat harder in the language of the recommendations and the conduct of the peer review meeting, and they certainly enjoy the greatest visibility and perceived prestige, as the flagship of the OECD. However, given the multiple possible impact pathways, it is not unambiguously clear which of the three review programmes has the greatest and most enduring impact on national policies. The uptake of recommendations by the reviewed country is only one, and often a minor type of influence, as compared with the longer-term and indirect effects via socialisation, mutual scrutiny, as well as creation, adoption, transfer and dissemination of shared norms and values (e.g. Lehtonen, 2005b; 2009a; Marcussen, 2003; Porter & Webb, 2008). Often the main influence operates through the review process itself and without major exposure to media and public scrutiny (Lehtonen, 2005a). This can involve, for instance, the creation, adoption and dissemination of shared norms across the OECD countries, or national officials using the support of their peers in other member countries in order to sway opinion in their own bureaucratic battles (Porter & Webb, 2008). If the OECD peer reviews have an influence, they do so not because of a ‘shadow of hierarchy’, but rather thanks to a ‘shadow of image’ – the fear of loss of face in front of a set of like-minded peers.

Trade-offs between types of hardening and softness

The OECD experience underlines the multiple trade-offs between different types of hardening but also between harder and softer elements of governance. For instance, using universally applicable criteria and indicators, covering the entire range of the policy area in question, minimising the influence of the reviewed country, and employing more precise and hard-hitting language would strengthen ‘blaming and shaming’, ‘justification’, ‘precision’, and ‘obligation’ but might, when paying insufficient attention to reviewed country needs and conditions, compromise policy relevance. The very notion of blaming and shaming is controversial in OECD work, as delegates are repeatedly reminded that the reviews are not ranking exercises, but measure achievements in relation to the countries’ own policy objectives. Table 1 provides a summary of the multiple trade-offs and their manifestations in the three OECD peer reviews, as presented in the previous section.

Table 1. Summary of the types of hardening in the OECD peer reviews, and the involved trade-offs and ambiguities.

	Review design choices		Elements of hardness and softness in OECD reviews			Types of hardening; ambiguities and trade-offs
	Hard	Soft	Environmental Performance Reviews	Energy Policy Reviews	Economic Surveys
Criteria of performance	Universally applicable, external criteria; and fulfilment of recommendations from previous review	Country’s own policy objectives	Soft: country’s own objectives; cost-effectiveness of policies Hard: indicators	Hard: Shared Goals of IEA; indicators	Hard: shared beliefs amongst the economist community; efficiency of policies; indicators	Blaming & shaming from comparability and objective criteria Precision from clearly defined criteria
Review topic(s)	Comprehensive review, covering the entire policy area	Focused review, (partly) on topics selected by the reviewed country	Hard: traditionally, comprehensive reviews Soft: constant demands for more focused reviews	Hard: traditionally, comprehensive reviews Soft: constant demands for more focused reviews	Hard: comprehensive reviews, with focus on links between macroeconomics and policy areas	Justification Blaming and shaming via cross-country comparison; comprehensiveness reduces the reviewed country’s influence Bundling & coupling through comprehensiveness can undermine blaming & shaming in advanced countries
Roles of different actors in drafting the review	Reviewed country has little influence on the review, esp. on drafting the conclusions and recommendations	Co-drafting by the secretariat and member countries	Hard: modest influence by the reviewed country; bundling and coupling via strong ‘peer element’	Soft: close interaction between the secretariat and the reviewed country government Hard: bundling and coupling via strong ‘peer element’	Soft: close interaction between the secretariat and the reviewed country government; weak peer element	Review team independence and dominant role of the secretariat: enhances precision and blaming & shaming, but may undermine bundling & coupling Precision and blaming & shaming may benefit from member country engagement (e.g. reviewed country demands for more precise, targeted and influential recommendations)
Language of recommendations	Precise and punchy, with detailed targets	Consensual, diplomatic, and general	Context-dependent	Context-dependent	Context-dependent	Precision, obligation, blaming & shaming Country-specific context (e.g. discussion culture) shapes the ability of blaming & shaming to operate
Peer review meeting	Long (full-day) meeting Spontaneous, free-flowing debate	Short meeting Discussion follows a strict predefined agenda	Hard: day-long meeting Soft: ceremonial meeting Bundling and coupling via cross-sectoral integration Sometimes ministerial participation	Soft: short meeting Bundling and coupling via cross-sectoral integration	Hard: secretariat active; free-flowing discussions Soft: short meeting Bundling: consolidation of the economist expert community	Blaming & shaming – peer meeting as a ‘semi-public’ event Precision and justification from central role of the secretariat Blaming & shaming via different pathways, depending on the background of the delegates Ministerial participation may enhance or undermine precision and blaming & shaming Bundling & coupling, in countries with a weak tradition of cross-sectoral work
	Affects hardness, depending on the context: Ministerial participation or not? Committee members from capitals or permanent OECD delegations?
Frequency of the reviews	Short review cycle, high frequency	Long review cycle	Soft: long review cycle	Harder: 4-year cycle	Hard: 18–24-month cycle	Blaming & shaming, at greater frequency
Follow-up and monitoring of review recommendations	Mandatory, according to clear guidance	Voluntary	Soft: voluntary follow-up reporting	Hard: interim annual reviews	Hard: follow-up in each review	Justification
Financing the review	Dedicated OECD budget funding	Funding dependent on short-term priorities of member countries	Soft: dependence on changing budgeting priorities	Soft: dependence on changing budgeting priorities	Hard: OECD flagship – secured funding	Provides the preconditions for hardness

An aspect that the hard vs. soft distinction does not fully capture is the vital importance of the review process as a source of influence. To be operational, the harder elements must build on ‘soft’ foundations, notably on trust-building, socialisation, community building, and shared identities amongst the member countries. On the other hand, ‘hardness’ is necessary to prevent delegates from turning their attention away from the OECD as a mere ‘talk shop’. Ideally, the peer reviews combine the learning and accountability functions of evaluation (e.g. Højlund, 2015; Lehtonen, 2005b; Schoenefeld & Jordan, 2019), foster learning through socialisation, imitation and coercion, as national civil servants meet each other in supportive surroundings, and thereby acquire skills and competences (Marcussen, 2003; Marcussen & Trondal, 2011). The country experts learn about other countries’ policies, policy evaluation, and the operation of the OECD and peer reviews, the reviewed country officials learn from being questioned by their peers, while the coordinator of the process in the reviewed country learns about the review ‘logic’ and her own country’s environmental policies, and gains organisational, cooperation, and negotiation skills. Crucially, the secretariat accumulates a formidable amount of skills and information, constituting an institutional memory that serves as a basis for norm-creation that the peer logic relies upon. The congenial elements of the review process – informal face-to-face encounters, site visits during the mission, cocktail receptions – are vital in building goodwill and sense of belonging at the margins of the formal and sometimes ‘inquisitive’ meetings.

Peer reviews as advocacy

Considerations relating to power are omnipresent in the peer reviews. For example, hardness is not applied quite equally across countries, with the smaller countries often albeit not always dealt with in a more direct manner than the larger ones (e.g. Guilmette, 2007, p. 116; Lehtonen, 2005a, p. 233; Porter & Webb, 2008, pp. 51–52). Moreover, whether the ideal of combining learning and accountability is achieved depends on a host of factors beyond the soft–hard distinction, including the ability of the OECD to subtly alter power relations within national governments, by identifying and supporting ‘policy entrepreneurs’ (Carraro et al., 2018, p. 9; de la Porte & Stiller, 2020) within the government. This points to the third, more implicit, function of the peer reviews, namely ‘advocacy’, in the service of specific ‘epistemic communities’ (Haas, 1992) of likeminded specialists such as civil servants, independent experts and NGOs, within the various member countries and in the OECD secretariat. This function is clear in the Environmental Performance Reviews, which seek to ‘mainstream’ environmental considerations in better-established policy fields, notably economics and finance, and strengthen the ‘environmentalist epistemic community’ in relation to its more powerful economic counterpart (Lehtonen, 2005a, 2005b, 2007; 2008).

Because the reviews and the associated policy field have their distinct status in OECD work and within national governments, advocacy operates differently in the three OECD reviews. The Economic Surveys help to consolidate and spread the agenda and discourse of liberal economics, already dominant within the member countries (e.g. Dostal, 2004; Lehtonen, 2007, 2008, 2009a; Mahon & McBride, 2009). The EPRs, as a relative newcomer in the peer review family, and representing a less mature policy field, must tread carefully, seeking a balance between the Organisation’s ‘mainstream’, and advocacy for an epistemic community still at the margins of the OECD. This explains some of the ‘ceremoniality’ and softness, but also the ‘hardness’ in the efforts to limit the influence of the reviewed country, which is essential for the image of independence and hence the credibility of the programme. The energy sector, in turn, lacks a clearly definable ‘epistemic community’, whose cause the IEA reviews would advocate. The IEA interviewees evoked cleavages within and between member countries concerning energy policy issues such as small vs. large, centralised vs. decentralised, fossils vs. renewables; economics vs. engineering; supply vs. demand; pro- vs. anti-nuclear; and priorities between climate, market, and security of supply concerns. Over time, the distinction between the liberal and interventionist countries in the IEA has faded,⁸ yet the reviews need to balance between the liberal economic doctrine of the OECD, member country sovereignty in a strategically central policy area, and energy security as the mainstay of the IEA. The considerable ideological and epistemic consensus within the ‘economist epistemic community’, in turn, gives the Economic Surveys a ‘harder’ backing.

These sector-specific differences underscore the fact that hardening has a direction. Even when hardening strengthens the impact of reviews, the key question is: impact through which pathway, in the service of which purposes, and over which timeframe? Hardening privileges those impact pathways that rely on accountability, top-down diffusion of policy ideas, and greater publicity, while possibly undermining the alternative pathways of bottom-up learning, through expert and civil servant interaction within arenas relatively free from external pressures. The possibilities and potential effectiveness of hardening, and the direction of its impacts are shaped by the involved epistemic communities, their objectives, values, ideology, internal cohesion, and place in the policy field in question.

Hardness built upon soft foundations: implications from OECD experience for the EU

The OECD’s potential strength lies in its ability to provide a forum for ‘frank debates on hot topics’ (e.g. Lehtonen, 2005a, p. 184). The absence of a ‘shadow of hierarchy’ appears precisely as a prerequisite for such frankness, but also for the secretariat experts’ ability to conduct the kind of independent, high-quality research that is a distinguishing characteristic of the OECD (Marcussen & Trondal, 2011, pp. 609–610). For the ‘inquisitional’ element of the reviews to work, the reviewed country government needs to feel ownership, and engage in the processes of socialisation. The OECD secretariat and the relevant peer review committees need to show sensitivity to the needs and desires of the reviewed country, while ensuring equal treatment of all member countries. Keeping the OECD’s heterogeneous membership happy means providing combinations of various types of ‘hardness’ and ‘softness’, tailored to country specificities. It also means balancing between types of hardness and between learning, accountability and advocacy as the key objectives of reviews.

For the EU, with its far greater coercive power, ensuring ownership and adhesion by all countries does not at the first glance appear as crucial as for the OECD. The EU is a highly political organisation – not an OECD-style global think tank or an ‘idea sweatshop’. Unlike the OECD, it can build on a common energy and climate policy as a mutually shared ultimate set of objectives against which to evaluate its member states. However, the EU and the OECD face similar challenges following from expanding membership: diminishing cultural homogeneity, reduced levels of trust and value sharing among reviewers (Pagani & Wellen, 2008, p. 273), and increasing difficulties at reaching consensus (Carraro et al., 2018; Ringel & Knodt, 2018). Multiple dividing lines between member countries persist, whereas in the current context of widespread disillusionment with – and even hostility towards – EU-level regulation, the EU cannot build upon a similar a priori goodwill that the OECD still enjoys, especially within national administrations.

In the energy sector, the EU’s situation resembles that of the OECD, with little direct coercive, regulatory power in a ‘controversial and normatively charged field’ (Carraro et al., 2018), member countries jealously safeguarding their sovereignty, with relatively low degree of value sharing and willingness to heed advice (Ringel & Knodt, 2018). In such a context, even the idea of peer reviews may appear as problematic. Yet, in its efforts at hardening its soft energy and climate governance the EU might do well paying attention to the impact of different types of hardening on processes of identity-building, socialisation, and joint norm-creation. Crucially, alternative types of hardening can either strengthen or undermine the likemindedness that soft governance relies upon. A practical albeit partial solution to the need to keep the increasingly heterogenous country membership satisfied – a precondition for effective soft governance – might be group-wise benchmarking of countries according to their specific needs (cf. Groenendijk, 2011). Lastly, harder soft governance is not only about country surveillance, but to a great extent also about policy advocacy and power battles between rival epistemic communities. Further research could explore the choices between alternative ways of hardening EU climate soft governance from the point of view of power relations amongst epistemic communities: which of the various epistemic communities in the energy and climate sector, each advocating their own climate policy solutions, will benefit from a given type of hardening? Hardening is multidimensional, and its impacts sometimes ambiguous and seldom politically neutral.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes on contributor

Markku Lehtonen holds a PhD in environmental economics from the University of Versailles Saint-Quentin-en-Yvelines (2005) and an MSc in environmental sciences from the University of Helsinki (1994). He works currently as an EU Marie Sklodowska-Curie Individual Fellow, at the Pompeu Fabra University in Barcelona, conducting research on controversies and appraisal concerning nuclear-sector megaprojects in Finland, France, Spain, the UK, and the OECD Nuclear Energy Agency. His recent research has concerned the role of expertise in environmental and energy policies, governance and controversies in energy policy (esp. nuclear energy, radioactive waste management, and biofuels), as well as deliberative democracy and public participation in environment-related policymaking. Markku is an associate researcher at Groupe de sociologie pragmatique et reflexive (GSPR), Ecole des Hautes Etudes en Sciences Sociales, Paris, and at Science Policy Research Unit (SPRU), University of Sussex, UK. He is also adjunct professor in environmental sociology at the University of Jyväskylä, Finland.

Additional information

Funding

This work was supported by the European Commission Marie Skłodowska-Curie Individual Fellowships (IF) grant number 794697-TENUMECA.

Notes

1 Interview with an IEA SLT delegate, 18 March 2010.

2 A delegate with experience from both energy reviews and EPRs described the peer meeting in the former as ‘grand theatre’, arguing that the conclusions and recommendations have in practice already been agreed prior to the meeting (interview, 11 March 2010).

3 A French IEA delegate underlined the importance of the peer character – the capacity country experts ‘to bring a bit of realism to the often rather theoretical views of the secretariat members’ (2 April 2010).

4 An IEA secretariat official observed that ‘purely academic recommendations were unlikely to find echo in the reviewed country’, and that an expert had to ‘make the government trust’ she was ‘on their side’ (12 March 2010).

5 A Portuguese WPEP delegate distinguished between the ‘Latin’ and Anglo-Saxon countries, the former being more accustomed to diplomatic, hidden messages, whereas the Anglo-Saxons would require more direct language (see also Gambetta 1998).

6 For example, the first EPRs of Chile and Mexico, which attracted great attention and minister-level participation.

7 The three parts of the report used to focus on (1) pollution control and water management, (2) environmental policy integration, and (3) international cooperation, but are now named as (1) ‘key environmental trends’, (2) ‘environmental governance and management’, and (3) ‘towards green growth’.

8 Interview with a high-level IEA official, 26 May 2010.