Holy Grail or inflated expectations? The success and failure of integrated policy strategies

ABSTRACT

Governments and international organizations increasingly pursue the development of integrated policy strategies to govern persistent societal problems that crosscut the boundaries of traditional jurisdictions. In spite of the rising popularity of such integrated strategies, little is known about their effects. Although it is generally assumed that integrated strategies result in better outcomes, the evidence base to support this claim is sparse. This is not to say that no attempts to study the relationship between integrated strategies and policy outcomes have been undertaken at all; this paper presents a research synthesis of the fragmented evidence base. Eligible studies are interpreted and discussed by using a heuristic that distinguishes between programmatic and political success and failure. Apart from synthesizing the impacts that integrated strategies have had, the paper reflects on associated explanatory conditions and methodological approaches that have been used. The review almost exclusively finds reports of failure and constraining conditions. At the same time, methodological approaches are found to be largely unconvincing and considerable research gaps remain. The paper, therefore, ends with a nuanced answer to the question of whether integrated strategies are worth pursuing and sets out various avenues for further research.

KEYWORDS:

• Policy integration
• integrated policy strategy
• policy outcome
• policy success
• policy failure
• implementation

1. Introduction

Governments and international organizations have taken an increasing interest into developing integrated policy strategies (IPSs) in the new millennium (Bogdanor 2005; United Nations 2015; Candel and Biesbroek 2016; Cejudo and Michel 2017). Recognition that many of today’s most pressing societal problems crosscut the boundaries of traditional jurisdictions, the effects of (too) successful New Public Management reforms, and the subsequent emergence of holistic governing paradigms, such as joined-up-governance and whole-of-government, have, among other causes, made pursuing policy integration into the quest for the philosopher’s stone (Rhodes 2000; Peters 2015). IPSs are a particularly ambitious form of policy integration. Different from more sector-initiated types of policy integration, such as ‘mainstreaming’ (e.g. Uittenbroek 2014) or ‘cooperation’ and ‘coordination’ (e.g. Geerlings and Stead 2003),¹ they are adopted to pursue a set of integrated goals by a polity as a whole (Metcalfe 1994; Candel and Biesbroek 2016). IPSs are here defined as explicit governmental attempts to address one or more crosscutting concerns – such as food insecurity, climate change, or homeland security – in a holistic manner and for that reason prescribe concerted policy-making efforts across sectors and, possibly, levels within a polity. Consequently, IPSs typically aim to align or even fundamentally redesign policy goals and instruments, and to strengthen and harness integrative capacities (Rayner and Howlett 2009b, 2009a). They are typically laid down in legislation, programmes and/or high-level agreements, which aim to set consequential governance processes in motion (Nordbeck and Steurer 2015). Following from these multi-level, -policy, and –goal dimensions, IPSs may be argued to lay down directions for meta-governing what have been termed ‘Type VIII’ or ‘complex multilevel policy’ mixes (Howlett and del Rio 2015).

In spite of the popularity of IPSs, surprisingly little is known about their effects on policy outcomes. Whereas the assumption that better integrated policy results in better outcomes is widespread among both scholars and policy-makers, it is not supported by a strong evidence base (Jordan and Lenschow 2010). This evidence gap at least partly results from considerable conceptual and methodological challenges in isolating the effects of policy in general, and of integrated policy in particular (Jordan and Lenschow 2010; Knill and Tosun 2012). This is not to say that nothing is known at all; this paper starts from the researcher’s observation that various scholars have attempted to assess the implementation and performance of integrated strategies across a range of governance systems and sectors (e.g. Drimie and Ruysenaar 2010; Vince 2015). At the same time, these attempts have remained fragmented and have not yet led to a more general scholarly debate about the impacts of IPSs. To allow for such a debate, this paper aims to strengthen the current state of knowledge by synthesizing and critically appraising the insights that previous studies have provided. The central question that the paper seeks to address is: what insights have previous studies provided into the impacts of integrated policy strategies on policy outcomes? The consequential synthesis may serve as a point of departure for future (comparative) studies of integrated strategies and associated outcomes.

The paper proceeds with establishing a framework that serves as basis for organizing the literature review. The framework conceptualizes IPSs’ impacts on policy outcomes in terms of ‘programmatic’ and ‘political’ success or failure (Bovens, ‘t Hart, and Peters 2001; McConnell 2010). In addition to reporting what existing studies have found regarding IPSs’ outcomes (i.e. their degree of success), the paper synthesizes associated explanatory conditions and discusses the methodological approaches that have been used. Categories of explanatory conditions were developed abductively, which is further explained under the methodological approach in Section 3. Section 4 presents the findings of the review. The paper ends with a discussion that critically appraises the current state of knowledge and provides an outlook for future research possibilities.

2. A policy success and failure heuristic to study outcomes

Policy outcomes are generally understood as the impacts that are produced by the implementation of policy outputs (the texts resulting from a decision-making process). A further distinction is made here by separating intermediate outcomes from eventual outcomes (cf. Knill and Tosun 2012, who make a slightly different distinction between outcomes and impacts). Intermediate outcomes are here understood as changes of institutions, policies, and political processes of or within a polity, that is, of the functioning of a political system itself. In this respect, the central question from the perspective of IPSs is whether and how a particular IPS’s objectives for changing the political system have been achieved (or of what sorts of unintended consequences have occurred). Ideally, such changes entail the alignment of efforts across a potentially broad range of governmental sectors and levels, involving changes in institutions (e.g. policy beliefs or policy-making structures and procedures), policies (goals, instruments, and policy frame), and political leadership (Peters 2015; Candel and Biesbroek 2016). The pursuit of one or more polity-overarching objectives is the defining characteristic of IPSs vis-à-vis other forms of policy integration in this respect.

Eventual outcomes are an integrated strategy’s impacts on society, that is, effects outside of the immediate political system. These are the types of effects that most policies are intended to generate ultimately, for example, reduced crime levels, strengthened adaptiveness to climate change, or strengthened food security. As hardly any of these outcomes are produced by the mere adoption of policy outputs, eventual outcomes are normally preceded by intermediate outcomes. Such an approach of outcomes closely resembles a top-down implementation perspective (Winter 2012). It should be noted, however, that whether and to what extent policies can produce their desired outcomes is contested (e.g. Mosse 2004). As a result, the assumed causal relationship between intermediate and eventual outcomes is an important empirical question in itself.

The effects of the adoption of an IPS on intermediate and eventual outcomes are here conceptualized in terms of policy success and failure. Research on the success and particularly the failure of public policies has a long tradition in the political sciences (e.g. Wolman 1981; Pressman and Wildavsky 1984; Bovens and ‘t Hart 1996). The fundamental questions that this stream of literature has aimed to address are: (1) how both failure and success of policies in achieving their goals can be conceptualized and empirically studied, and (2) what explains whether the outcomes of a policy are conceived as a success, failure, or something in between. It is beyond the scope of this paper to discuss the evolution of this whole body of literature (for excellent discussions, see: McConnell 2010; Howlett, Ramesh, and Wu 2015; Bovens and ‘t Hart 2016). Instead, the remainder of this section elaborates two distinct evaluation logics that are manifest in this stream of literature and that have been extensively elaborated in the work of Bovens, ‘t Hart and collaborators (Bovens and ‘t Hart 1996; Bovens, ‘t Hart, and Peters 2001; Bovens 2010; Bovens and ‘t Hart 2016). These logics can be used to study the effects of integrated strategies in different ways and are here used to cluster insights from the existing body of literature (see Section 3).

The two logics correspond with two fundamentally different ontologies regarding policies’ success and failure. A first approach to the success or failure of a policy contends that it is possible to objectively assess whether a policy’s objectives have been achieved. As a result, it is possible to draw relatively firm conclusions about whether a policy has been effective or efficient, and thus about whether it has been a success or a failure. Ideally, such an assessment would also include evidence about a policy’s unintended effects when weighing costs and benefits. This first approach to evaluating public policy is referred to as ‘programmatic evaluation’ (Bovens and ‘t Hart 1996) and can be summarized as pertaining ‘to the world of facts and social balance sheets: observable costs and benefits, original intentions and eventual outcomes. Its currency is “performance as measured”’ (Bovens and ‘t Hart 2016, 4). Scholars, governments and international organizations have developed many standardized ways of conducting such evaluations.

The second logic takes a social constructivist point of departure and approaches success and failure as being socially constructed in the public realm. Policies are argued to be subject to political contestations and are perceived and framed in different ways, depending on actors’ ideas and interests. Consequently, evaluation is not an objective or neutral endeavour, but a matter of interpretation, both by societal actors and the researcher. This second logic is referred to as ‘political evaluation’ and can be described as pertaining

to the world of impressions: lived experiences, stories, frames, counter-frames, heroes and villains. These are constructed in the way policies are being perceived and debated among their stakeholders, in the media and in the forums where policy-makers are held to account, such as citizen and institutional watchdogs, legislatures, courts. The currency of political evaluation is ‘reputation as conferred’. (Bovens and ‘t Hart 2016, 4)

Combining these two evaluation rationales with the distinction between intermediate and eventual outcomes allows making a classification of the different ways in which policy outcomes can be studied (Figure 1). Although this heuristic could in principle be applied to any type of policy, it is here principally developed to cluster existing insights into the effects of integrated strategies.

Figure 1. Four perspectives on evaluating policy outcomes.

Until recently, hardly any attention had been given to the success and failure of this emerging form of policy. This gap has only recently been put forward and partly addressed by Vince (2015). By combining insights from the literatures on integrated policy approaches and policy success and failure, she raises various expectations about why many grand integrated strategies eventually fail, most of which she confirmed in a case study of Australia’s Oceans Policy (AOP; see Section 4). Vince uses McConnell’s (2010) typology of process, programme, and political failure, which is different from the approach suggested by Bovens and ‘t Hart.² Vince argues that process failure, which she interprets as a policy’s implementation within the political system, may result from policy layering (i.e. the resilience of old policy frameworks), insufficient cross-sectoral coordination and capacities, and organizational politics. She expects programme failure to stem from poor policy design, unachievable goals or timelines, and, again, from policy layering. Political failure is argued to be related to positive incentives for symbolic policy-making and blame avoidance strategies. Many of these issues correspond with constraining factors found in studies of (environmental/climate) policy integration in general. For example, Peters (2015) mentions a long list of barriers stemming from risk avoidance, (new) public management, turf, partisan politics, beliefs, professionalism, and accountability, inter alia. The factors in this literature (e.g. also: Briassoulis 2004; Ross and Dovers 2008; Jordan and Lenschow 2010) are used to abductively categorize constraining but also enabling (e.g. Peters 2015) factors mentioned in the body of literature that is reviewed (see Section 3).

3. Methodological approach

The body of literature that this paper’s research synthesis (Section 4) is based on has been assembled and appraised by using systematic review methods. The advantage of systematic literature review methods over ‘ordinary’ literature discussions is that researcher bias can be limited and is made transparent (Petticrew and Roberts 2006; Gough, Oliver, and Thomas 2012). The specific approach applied in this paper is based on previous reviews by Biesbroek et al. (2013) and Candel (2014).

The data collection process is schematically depicted in Figure 2. First, an initial assessment of potentially relevant literature was performed in both Scopus and Web of Science to develop a query (see Appendix 1 for keywords). The final query was run on 1 April 2016 and included studies published up to that date. After excluding duplicates, this resulted in an initial body of literature. In the next step, titles, keywords, and abstracts were loaded into Endnote and judged on the basis of four inclusion criteria: (1) the publication is in English; (2) it is published as scientific article, book (chapter), or expert report (e.g. some OECD reports are covered by Scopus); (3) the publication focuses on explicit efforts of governments or international organizations to address one or more crosscutting problems through an IPS; and (4) the publication assesses or evaluates the IPS’s outcomes. Because it was not always self-evident from a publication’s abstract whether it met the third and especially the fourth criteria, the criteria were applied relatively loosely in this round, meaning that publications were included if they potentially touched upon the outcomes of an IPS. In the final selection round, the inclusion criteria were applied to the full papers, resulting in a final body of 18 eligible publications. This body of literature was complemented with one additional publication that was considered eligible but that did not come out of the search, because it referred to ‘integration’ instead of ‘policy integration’ (i.e. Drimie and Ruysenaar 2010). Table 1 provides an overview of all the publications included.

Figure 2. Data collection process.

Table 1. Overview of included studies.

Authors (year)	Crosscutting problem(s) that the IPSs addresses and locus	Integrated policy strategy/-ies
Alves et al. (2013)	Coastal zone management in Portugal	National strategy for Integrated Coastal Zone Management
Begg and Gray (2004)	Traffic levels and vehicle emissions in the United Kingdom	Integrated Transport White Paper
Black, Pérez-Escamilla, and Rao (2015)	Nutrition and child development in Colombia, Bolivia, and Mozambique	Four integrated nutrition – child development interventions
Casado-Asensio and Steurer (2016)	Climate change mitigation in Western Europe (EU-15)	NMSs
Drimie and Ruysenaar (2010)	Food security in South Africa	Integrated Food Security Strategy
Hertin and Berkhout (2003)	Environmental concerns in EU enterprise policy	No single strategy, but multi-annual EU commitments to EPI (embedded in various strategies/treaties: Amsterdam Treaty, Cardiff Process)
Jordan and Lenschow (2000)	Environmental concerns in Denmark, the Netherlands, Germany, and the United Kingdom	No single strategy, but multi-annual EU commitments to EPI (embedded in various strategies/treaties: Amsterdam Treaty, Cardiff Process)
Kivimaa and Mickwitz (2006)	Environmental concerns in Finnish technology policies	National integrated strategies of Science and Technology Policy Council
Klepp and Forster (1985)	Nutrition and public health in Norway	Nutrition and Food Policy
Mickwitz and Kivimaa (2007)	Environmental concerns in innovation policy and innovation concerns in environmental policy in Finland	Focuses on four strategies, but definition of strategy is rather broad (strategic reviews of Science and Technology Council, Tekes strategy document, Finnish Government Programme for Sustainable Development, Water Protection Targets for the Year 2005)
Nilsson, Eklund, and Tyskeng (2009)	Environmental concerns in waste management in Sweden	Not about single cohesive strategy but national waste policy that consists of series of integrative bills
Pollack and Hafner-Burton (2010)	Environmental and gender concerns in the European Commission	Commission strategies for EPI and gender mainstreaming
Russel and Jordan (2010)	Environmental concerns in the United Kingdom	No single EPI strategy but EPI system that comprises of blend of strategies and governance modes
Schout and Jordan (2005)	Environmental concerns in the EU	No single strategy, but multi-annual EU commitments to EPI (embedded in various strategies/treaties: Amsterdam Treaty, Cardiff Process)
Söderberg (2011)	Environmental concerns in Swedish bioenergy policy	Integrative Swedish bioenergy policy
Steurer and Clar (2015)	Climate change mitigation in Austrian building policy	EU, federal and provincial strategies for greening building sector (particularly federal Climate Strategy)
Vince (2015)	Oceans management in Australia	AOP
Vince et al. (2015)	Oceans management in Australia	AOP

Note: NMSs, National mitigation strategies; AOP, Australia's Oceans Policy.

The focus on explicit governmental strategies implies that publications in which it is the researcher rather than the study object who defines a particular integration challenge (e.g. as in most integrative assessment studies), studies that focus on integrative principles or requirements that are not embedded in some sort of strategy (e.g. as in many, though not all, studies on environmental and climate policy integration), or studies that deal with the potential for integrated policy-making or planning fall outside the scope of this review. In addition, the focus on strategies that address specific problems implies that studies of sustainable development strategies were not included if they did not also touch upon consequent or underlying strategies that target more concrete problems or concerns.

The final body of literature was read in detail and relevant segments of the publications were put in a data extraction table (Appendix 2). This table presents the raw data (i.e. relevant characteristics of and insights from the publications) along various categories: (1) the crosscutting problem(s) that is/are addressed by the strategy and the strategy’s locus; (2) the key concept(s), focus, and methodological approach for studying the outcomes of an IPS; (3) (insights into) programmatic and political evaluations of the success and/or failure of an IPS; and (4) any other relevant arguments in the paper or notes about the paper. In the analysis phase, programmatic and political evaluations of whether IPSs had been successful were clustered along the four categories in Figure 1. Additionally, insights about enabling and constraining factors were coded by using the qualitative data analysis program Atlas.ti. Codes were assigned in an abductive manner: similar factors were clustered and subsequently compared to the existing body of literature on policy integration (see Section 2) to assign labels. Thirdly, methodological approaches to studying outcomes were compared and clustered.

The approach chosen in this study has a number of limitations. First, the decision to only include English publications covered by Scopus and Web of Science means that potentially relevant studies published in other languages and outlets were not included. Second, although well-established terms, the query’s focus on ‘policy integration’ and ‘integrated policy’ implies that additional insights could be obtained from publications that cover similar sort of strategies without labelling them in terms of integration. As Table 1 shows, virtually all eligible publications are about sustainable development, the environment, or food and nutrition security; all of which are fields that have a relatively long tradition of thinking in terms of integrative concepts, both amongst scholars and policy-makers. In addition, the focus on IPSs excludes studies on the impacts of other types or degrees of policy integration (cf. Jordan and Halpin 2006; Candel and Biesbroek 2016). Lastly, the abovementioned example of the paper of Drimie and Ruysenaar shows that even papers with an explicit integrated policy focus could fall outside the query’s results. To mitigate this risk, included papers were checked for potentially relevant studies that were missed. Although this did not lead to any further additions, it cannot be ruled out that alternative papers about the outcomes of IPSs exist.

4. Synthesis: the success and failure of IPSs

This section synthesizes the evidence base about the success and failure of IPSs from the publications listed in Table 1 and Appendix 2. The first two sub-sections present the programmatic and political evaluations of IPSs’ outcomes, that is, to what extent they are considered a success or failure. The third sub-section reports the constraining and enabling factors that are found in the literature. The final sub-section elaborates the methodological approaches that were used. This section presents existing insights as emerging from the literature; a discussion follows in Section 5.

4.1. Insights from programmatic evaluations

The programmatic dimension of assessing success and failure proves much more common in the publications included than the political dimension. Whereas all studies performed some sort of programmatic evaluation (e.g. were initial objectives obtained?; with which measurable outcomes and impacts?), very few elaborated on the political dimension of evaluating integrated strategies (see Section 4.2).

The overall picture that emerges regarding the performance of the various integrated strategies is largely negative. Out of 18 studies, 10 evaluated the delivery of the specific integrated strategy/-ies they analysed as a failure (Hertin and Berkhout 2003; Begg and Gray 2004; Schout and Jordan 2005; Nilsson, Eklund, and Tyskeng 2009; Drimie and Ruysenaar 2010; Pollack and Hafner-Burton 2010; Steurer and Clar 2015; Vince 2015; Vince et al. 2015; Casado-Asensio and Steurer 2016), 2 studies concluded that the integrated strategies that were scrutinized were largely successful (Söderberg 2011; Black, Pérez-Escamilla, and Rao 2015), and 6 publications provided a mixed picture (Klepp and Forster 1985; Jordan and Lenschow 2000; Kivimaa and Mickwitz 2006; Mickwitz and Kivimaa 2007; Russel and Jordan 2010; Alves et al. 2013).

4.1.1. Intermediate outcomes

The overall negative assessment is largely due to the major implementation gaps that virtually all studies found. A most extreme example is provided by Begg and Gray (2004), who show that the UK’s 1998 Integrated Transport White Paper was never implemented at all, and thus never proceeded beyond a paper reality. Instead, the government shifted away from its ‘joined-up philosophy’ towards sectoral environmental and transport policies, whereby the latter was primarily focused on supply side investment rather than demand management. Other studies report implementation shortcomings across (1) sectors, (2) governance levels, and/or (3) with respect to policy instrument mixes. Regarding the first, a common gap identified is that the implementation of integrated strategies is highly sectorized at best, and limited to only one or a few government sectors out of a whole range of targeted sectors at worst (Drimie and Ruysenaar 2010; Vince 2015; Casado-Asensio and Steurer 2016). For example, studies of EPI strategies found that it is often only the environmental ministries or directorate-generals that engage in genuine implementation, whereas implementation beyond the environmental sector is generally weak (Jordan and Lenschow 2000; Hertin and Berkhout 2003; Pollack and Hafner-Burton 2010; Russel and Jordan 2010).

Similarly, various authors found imperfect implementation across governance levels (Schout and Jordan 2005; Alves et al. 2013; Steurer and Clar 2015; Vince 2015; Casado-Asensio and Steurer 2016). A good example is the study of Nilsson, Eklund, and Tyskeng (2009), who show that Swedish national waste policy, in which environmental concerns had been strongly integrated, hardly seemed to be integrated into local planning and decision-making practices, inter alia resulting in widespread construction of waste-incineration plants.

Thirdly, various studies report lagging implementation of policy instrument mixes or failures to adjust existing instruments (Nilsson, Eklund, and Tyskeng 2009; Russel and Jordan 2010). Regarding the latter, Mickwitz and Kivimaa (2007) for example show that although environmental concerns were fairly well integrated into Finnish technology policy strategies, integration was less complete at the level of associated instruments, such as concrete technology programmes or the financing of R&D projects.

Despite a prevalence of accounts of poor implementation, various more positive implementation results were reported as well. Most straightforward in this respect is that although integrated strategies were generally not fully implemented, various partial instruments, structures, and working practices were put in place nonetheless. For example, Jordan and Lenschow (2000) and Russel and Jordan (2010) report that various frontrunner EU countries have made considerable progress in adopting EPI instruments and structures, while Klepp and Forster (1985) show that various instruments targeted at changing Norwegian food production and consumption patterns were actually put into practice. In addition, Mickwitz and Kivimaa (2007) found that there was a relatively good integration of environmental and innovation concerns in Finland, at least at the macro-level. Even though expectations about the impacts of such partial implementation are modest, they are argued to have the potential for putting a crosscutting problem or concern more firmly on macropolitical and subsystem agendas, and as such leave a legacy for future policy-making processes (Klepp and Forster 1985; Vince et al. 2015).

4.1.2. Eventual outcomes

Because most of the intended intermediate outcomes of the integrated strategies covered were not achieved it is not surprising that most studies do not report on the impacts on eventual policy outcomes; most IPSs never got to a point where they could have been expected to generate such impacts. This also shows in observed shortcomings of governments to monitor and evaluate implementation and consequent outcomes (Kivimaa and Mickwitz 2006; Drimie and Ruysenaar 2010; Alves et al. 2013; Casado-Asensio and Steurer 2016). Hereby, it should be noted that not all integrated strategies aim at having a societal impact. For example, EU Environmental Policy Integration (EPI) commitments, as laid down in the Commission EPI program, the 1997 Amsterdam Treaty, and the Cardiff Process, are primarily targeted at changing policy-making efforts (Schout and Jordan 2005; Pollack and Hafner-Burton 2010). Studies that do reflect on eventual outcomes provide diffuse but largely negative insights, whereby most researchers make the side-note that their analyses do not well-isolate the effects of integrated strategies from other factors (Section 4.4).

That said, Russel and Jordan (2010) argue that the UK’s EPI system and associated strategies have not significantly improved the environment in their 15 years of being in existence. Casado-Asensio and Steurer (2016) and Steurer and Clar (2015) do find positive climate mitigation outcomes, but argue that these are probably not related to National Mitigation Strategies or Climate Policy Integration in Austrian housing policy respectively, as both types of integrated strategies were poorly implemented. Similarly, in discussing the effects of integrating environmental concerns into Finnish research policy, Kivimaa and Mickwitz (2006) argue that eventual innovations result from a variety of driving forces, of which policy measures are only one. Innovating parties (self-)reported some beneficial environmental outcomes as a result of the policy, but did so selectively. In their evaluation of the Norwegian Nutrition and Food Policy, Klepp and Forster (1985) find minor improvements in food consumption patterns (e.g. in terms of fat intake), but are rather critical about the scope of these improvements.

The only two studies that report mostly positive outcomes resulting from integrated strategies are a study of Black, Pérez-Escamilla, and Rao (2015) on integrated nutrition and child development interventions and a study of Söderberg (2011) on the Swedish bioenergy policy, which traditionally has a well-integrated environmental component. Black, Pérez-Escamilla, and Rao performed a review of existing studies of integrated nutrition and child development interventions. They found that most (quasi-)experiments of integrated strategies resulted in positive impacts on children’s cognitive, socioemotional, and/or motoric development. In one study that they reviewed the impacts on growth were limited, which they attribute to the poor implementation of the nutrition component of that particular intervention. Söderberg infers the success of the Swedish policy from the doubling of bioenergy production over the last two decades.

4.2. Insights from political evaluations

Strikingly, of the 18 studies included only Vince’s (2015) evaluation of AOP focused explicitly on the political rationale of assessing success and failure. The other studies almost entirely restricted themselves to a programmatic evaluation rationale, although various authors indirectly provided some insights into the political construction of IPSs’ success and failure. The latter is the most obvious in studies that take widespread disappointment and sense of failure with respect to a particular IPS as point of departure for their analysis, such as is the case with the studies of Pollack and Hafner-Burton (2010) and Schout and Jordan (2005) on EU EPI strategies and of Steurer and Clar (2015) on climate change mitigation in Austrian building policy. Conversely, in their analysis of the UK’s EPI strategies, Russel and Jordan (2010) start from the OECD’s praise for the British system, and later nuance the euphoria through their own – programmatic – analysis of what has been achieved. Another observation authors have made is that some strategies, like the Norwegian Nutrition and Food Policy (Klepp and Forster 1985) and the Swedish environmental-friendly bioenergy policy (Söderberg 2011) have survived government turnover(s). From this, these authors infer that these strategies could rely on relatively broad political support.

As elaborated in Section 2, Vince (2015) takes a more explicit interest in the political rationale of evaluating AOP. Vince shows that instead of struggling over different conceptions of the AOP’s performance, politicians hardly took an interest, positive or negative, in the strategy at all. Instead, they resorted to blame avoidance strategies by moving away from the holistic policy and its – nonetheless widely perceived – failure towards establishing sectoral policies, such as regional biodiversity plans. Because of a lack of public and political interest, attempts of NGOs, scientists and the Strategic Policy Institute to point at the AOP’s failure and to mobilize additional resources found little resonance and did not result in a revival of the strategy. Altogether, attempts to politicize AOP never really got off the ground due to a lack of issue-salience, and although the strategy’s initial development was easily agreed upon by the government and opposition, it died quietly eventually.

4.3. Factors that enable and constrain IPSs’ success

Virtually all studies included provide some reasons for why success and particularly failure have occurred in the cases they focus on. Most do so by pointing at a number of institutional or agency-centred factors that have either enabled or constrained the realization of an integrated strategy’s intended outcomes. Following from the almost exclusive focus on programmatic types of evaluation (Sections 4.1 and 4.2), most of the factors discussed in this sub-section have been used to explain the observed absence of outcomes that were intended. However, in principle many of them, such as turf wars and leadership, can also be used to account for political contestations and differences in social constructions of outcomes.

Because a majority of publications focused on failure, the number of constraining factors is higher. However, most of these factors can work both ways, that is, they may also have an enabling effect (see Section 5). Black, Pérez-Escamilla, and Rao (2015) make this explicit by arguing that sufficient staff capacity, political will, cultural embeddedness of interventions, and appropriate monitoring and evaluation are vital to successful implementation; while a shortage of these factors may result in failure.

4.3.1. Constraining factors

The list of factors that have been mentioned as constraining is long; I here focus on six main categories (Section 3). First, integrated strategies may have been poorly designed to begin with. This can be because of various reasons. For example, a strategy may not sufficiently capture the complexity that characterizes a crosscutting policy problem. Drimie and Ruysenaar (2010) show how this has been the case for South Africa’s Integrated Food Security Strategy, which emphasis on agricultural production and food availability obscures issues surrounding accessibility. Alternatively, failure can result from insufficient stakeholder input in the design phase, resulting in knowledge gaps or a lack of support in the implementation phase (Drimie and Ruysenaar 2010; Alves et al. 2013; Casado-Asensio and Steurer 2016). Furthermore, strategies may contain vague goals and instruments, resulting in unclarity for implementing actors (Kivimaa and Mickwitz 2006; Nilsson, Eklund, and Tyskeng 2009), or may layer these goals and strategies on top of existing sectoral policies, resulting in unclear or conflicting priorities (Vince 2015).

The last point relates to a second factor, which was emphasized in almost all studies: the persistence of sectoral priorities and a consequent lack of willingness or abilities to coordinate sectoral efforts. At some times, these coordination shortcomings ‘simply’ result from unclear roles and responsibilities, or a lack of ownership amongst policy sectors. The latter has for example been repeatedly shown for EPI strategies, which receive little priority outside of environmental departments (Jordan and Lenschow 2000; Russel and Jordan 2010; Casado-Asensio and Steurer 2016). At other times, they may follow from outright turf wars over jurisdictions and resources (Steurer and Clar 2015; Vince 2015).

Third, these same sort of coordination challenges occur across governance levels (Schout and Jordan 2005; Steurer and Clar 2015; Vince 2015; Casado-Asensio and Steurer 2016). This is most clear in federal systems and in the national implementation of EU strategies. Steurer and Clar (2015), for example, show that strategies to integrate climate change concerns in Austrian building policies largely failed because the länder felt the strategy was forced upon them by the federal state. Such problems may also stem from ‘grand’ strategies’ failure to set out lower level mechanisms (Russel and Jordan 2010).

A fourth factor is lack of political will and prioritization. Various authors note that political actors lost interest soon after the adoption of an integrated strategy; to avoid blame for not achieving ambitious holistic objectives or because there are greater (political) incentives to creating new policies and institutions than implementing those that have been previously agreed upon (Begg and Gray 2004; Vince 2015). In addition, lagging interest may be caused by a change of leadership, lack of issue-salience, political crises, or the prevalence of other priorities, inter alia (Begg and Gray 2004; Schout and Jordan 2005). A common example of the last point is the prioritization of economic interests over environmental concerns (Nilsson, Eklund, and Tyskeng 2009). As a consequence of insufficient political backing, implementing actors have little latitude vis-à-vis sectors that need to change existing policy efforts.

Fifth, the effects of integrated strategies are reported to be hampered by limited institutional capacities and resources. At some times, hardly any capacities to implement a strategy have been allocated to begin with (Schout and Jordan 2005; Drimie and Ruysenaar 2010; Russel and Jordan 2010; Vince 2015). At other times, institutional changes may result in a sudden loss of capacities. For example, splitting the UK Department for Environment, Transport and the Regions (DETR) into separate transport and environmental ministries resulted in a loss of integrative capacities for implementing the British Integrated Transport White Paper (Begg and Gray 2004). A sub-dimension of this factor is lack of or exclusively sectoral evaluation and monitoring capacities, constraining policy learning (Kivimaa and Mickwitz 2006; Drimie and Ruysenaar 2010). In terms of resources, studies have reported the absence of dedicated funds, which, for example, prevent the development of joint projects and programmes across sectors or levels (Drimie and Ruysenaar 2010; Vince 2015).

The sixth factor concerns choices of policy instruments. Even if instruments are deployed to pursue the goals of integrated strategies, these are often ineffective (Klepp and Forster 1985; Nilsson, Eklund, and Tyskeng 2009; Pollack and Hafner-Burton 2010; Vince 2015). This ineffectiveness may result from inconsistent mixes, applying single instruments to multiple goals, or a lack of hard measures. The use of an instrument for different purposes is well-illustrated by Klepp and Forster (1985), who attribute the disappointing effects of Norway’s Nutrition and Food Policy to the use of price subsidies for both health and agricultural productivity objectives, the latter of which came to dominate. Regarding the use of ‘soft’ versus ‘hard’ instruments, scholars have noted that exclusively relying on the former may be inadequate for getting sectors or governance levels to commit themselves to genuinely coordinating their policy efforts. While acknowledging the limitations of their approach, Pollack and Hafner-Burton (2010) present a convincing case by comparing the European Commission’s 4th Action Program on Equal Opportunities for Men and Women in the Commission, which is focused on the recruitment, retention and treatment of qualified female officials, with earlier programmes. Whereas the use of soft instruments in previous programmes only resulted in minor improvements, the incorporation of hard instruments in the 4th programme effectuated quantifiable improvement across most directorate-generals.

4.3.2. Enabling factors

Compared to the long list of constraining factors, relatively few explicit enabling factors have been observed and discussed. Moreover, most of these enabling factors are discussed rather briefly and are often framed as the ‘mere’ opposite poles of the abovementioned constraints. Apart from the already mentioned ‘hard’ policy instruments (Pollack and Hafner-Burton 2010), these include well-functioning inter-sectoral coordination, monitoring mechanisms, stakeholder engagement, and the use of sector-specific knowledge (all mentioned in: Söderberg 2011).

The two enabling factors that were emphasized the most are strong structures and procedures and political backing. Regarding the first, authors have argued that strong and internally well-coordinated government systems, such as that of the United Kingdom, offer relatively better possibilities for integrated strategies to be successful (Jordan and Lenschow 2000; Söderberg 2011). The same applies to the use of more robust procedures, such as the European Commission’s impact assessment procedure that has been accompanied by relatively strict, centralized and enforced guidelines (Pollack and Hafner-Burton 2010). At the same time, these structures and procedures still require firm political backing, which has been argued to be indispensable for integrated strategies to be translated into concerted action (Jordan and Lenschow 2000) or, when the latter proves too ambitious, to at least put a crosscutting concern on sectoral agendas (Vince et al. 2015).

An interesting suggestion is made by Mickwitz and Kivimaa (2007), who argue that a two-way mechanism of integration between sectors may be more successful than aiming to one-directionally address a crosscutting concern in multiple sectors. In their case, environmental concerns were integrated in Finnish technology and innovation strategies, while innovation concerns were integrated in environmental strategies. Such an approach could lead to strengthened ownership and commitment across sectors. At the same time, Mickwitz and Kivimaa find the same sort of implementation deficits in eventual instruments.

4.4. Methodological approaches used

Regarding the methodological approaches that have been used to study the outcomes of IPSs, a clear finding is that very few of the studies included are explicit about data collection and analysis processes; if any data has been collected to begin with. Nine publications have no methodological section or paragraph at all. Eight more studies mention the modes of data collection (and sometimes analysis), but do so rather summarily (Hertin and Berkhout 2003; Schout and Jordan 2005; Kivimaa and Mickwitz 2006; Nilsson, Eklund, and Tyskeng 2009; Pollack and Hafner-Burton 2010; Söderberg 2011; Steurer and Clar 2015; Casado-Asensio and Steurer 2016). Most of these studies mention the use of desk research of policy documents, interviews, and/or reviews of existing literature. However, explanations of underlying research designs and accounts of whether and how data was used and interpreted precisely are generally missing.

Authors who do concisely reflect on the methods used, often explicitly mention the restrictions of the methodological approaches they chose and are cautious in drawing conclusions (Klepp and Forster 1985; Kivimaa and Mickwitz 2006; Casado-Asensio and Steurer 2016). At the same time, the argument is made that these challenges should not result in excluding eventual outcomes from IPS evaluations, as changes within a polity (intermediate outcomes) are on themselves no guarantee for eventual policy success (Mickwitz and Kivimaa 2007; Pollack and Hafner-Burton 2010).

Only one study included has a more elaborate methodological reflection. Black, Pérez-Escamilla, and Rao (2015) performed a review of existing studies of integrated nutrition and child development interventions. It should be noted that they are only explicit about the methodological approaches of the studies they review, and not about the review methods they used themselves. The authors argue that answering questions about the impact of integrated interventions requires experimental or quasi-experimental study designs and discuss four robust studies that have adopted such a design. In a first study, undernourished pre-schoolers were randomly assigned to an integrated intervention in child care centres in the city of Cali, Colombia. The successful results of this study (see Section 4.3), resulted in the nation-wide Colombian ‘Hogares Comunitarios de Bienestar’ integrated programme, which was evaluated in a subsequent study by assessing how varied lengths of exposure to the programme impacted child development. This variable of programme exposure was also used in a study of a programme of integrated interventions in child care centres in Bolivia. In a fourth study, communities in Mozambique were randomly assigned to receive psychosocial stimulation for children and literacy workshops for parents. All these research designs allowed for drawing relatively firm conclusions about the effects of integrated intervention strategies.

In terms of the independent variables that are used to explain IPSs’ outcomes, studies vary considerably in the concepts that are used. However, most independent variables are related to institutional factors (see Section 4.3). Concepts that have been used include governance mechanisms (Alves et al. 2013), institutional arrangements (Drimie and Ruysenaar 2010), institutional strategies (Hertin and Berkhout 2003), institutional conditions (Söderberg 2011), institutional measures (Pollack and Hafner-Burton 2010), federalism (Steurer and Clar 2015), and administrative capacities (Schout and Jordan 2005).

5. Discussion

5.1. Success or failure? Drawing up the balance

This paper started off with the question of what insights existing studies have provided into the outcomes of IPSs. The overall picture that has emerged from the research synthesis in Section 4 is not particularly encouraging. Although it should be stressed that this line of research is still very much in its infancy, both in terms of the number of studies and the rigour of methodological approaches, most integrated strategies that have been studied never achieved their intended intermediate outcomes, let alone eventual outcomes on the ground. Few strategies failed to be implemented in their entirety, but the findings of most studies are closer to the failure than to the success end of the spectre (cf. McConnell 2010). The adoption of integrated strategies in itself shows that the political and administrative conditions of various polities are to at least some extent conducive to holistic governance of crosscutting problems. However, the implementation of these strategies appears to be seriously hampered by the same sort of challenges they are intended to overcome, such as lack of political engagement, inter-sectoral and multi-level coordination failures and turf wars, and shortages of capacities and resources. Most of these constraining factors are not unknown to public administration and policy scholars; as such, the analysis confirms the insights from previous publications on coordination and policy integration (e.g. Briassoulis 2005; Jordan and Schout 2006; Bouckaert, Peters, and Verhoest 2010; Peters 2015). Studies that did include eventual outcomes in their analyses report no or only marginal effects. A notable exception in this respect is the meta-study of Black, Pérez-Escamilla, and Rao (2015) on integrated nutrition and child development interventions, which found both micro- and macro-level strategies to have largely positive impacts on individual beneficiaries.

The findings of this study justify the question of whether pursuing IPSs is worth the effort at all. Are integrated strategies the Holy Grail for overcoming the problems that polities experience in governing crosscutting problems, or have expectations been inflated? There are two general answers to this question. A first line of reasoning entails that the constitutive parts of a polity, for example, sectors, levels, and agencies, will continue to compete with each other over jurisdictions, priorities, and resources (Jones and Strahan 1985; Nicholson-Crotty 2005). The adoption of an integrated strategy will not suddenly make these parts operate in a concerted manner (cf. Rayner and Howlett 2009a). Moreover, many integrated strategies target problems that have traditionally been relatively low on the agendas of decision-makers, such as environmental concerns. While the adoption of an integrated strategy may signify a periodic rise of issue-attention, sustained prioritization and resource allocation, which would be required for making large-scale integrated approaches work, are unlikely on the longer term (cf. Downs 1972). In short, the pursuit for optimal policy integration through integrated strategies may be unfeasible and inopportune. Some policy scholars have, therefore, made a plea in favour of less ambitious and glamorous forms of policy integration, for example, by organizing information exchanges across sectors and levels or by tackling the most obvious incoherencies (Jordan and Halpin 2006; Keast, Brown, and Mandell 2007). Although not included in this paper’s analysis, such more modest integration attempts may have fewer difficulties in overcoming the implementation barriers identified in Section 4.3.

An alternative response would be that it may be difficult to make IPSs work, but not impossible. The insights provided by the type of research reviewed in this paper can be instructive for realizing more successful policy delivery. Of course, this would still be a major task, but a combination of effective leadership, appropriate structures, and a coherent set of overarching ideas can make ambitious integration attempts work (Peters 2015). In their work on ‘boundary-spanning policy regimes’, Jochim and May (2010) for example discuss various historic US governance constellations, such as those on drug criminalization, pollution abatement, and disability rights, in which these factors resulted in achieving relatively high degrees of concerted action across sectors and governance levels. In addition, the relatively positive findings of the study of Black, Pérez-Escamilla, and Rao (2015) suggest that strategies may be more successful when they are well-circumscribed and have a clear dependent variable or target audience (e.g. undernourished children).

These two responses are not necessarily mutually incompatible. Instances of successfully implemented integrated strategies may indeed be rare, because of which these strategies should not be considered as a panacea for solving incoherencies and inconsistencies. At the same time, if a crosscutting problem manages to stay on high-level political agendas for a longer period or if pressure is sufficiently high, the deployment of an integrated strategy can still be an option. Additionally, although pursuing ‘lower’ forms of policy integration may certainly be more effective and opportune on the short term, integrated strategies can serve as important spots on the horizon for policy-makers (Candel 2016).

5.2. Avenues for strengthening research on IPSs

Whether a more convincing and definite answer to the question of whether IPSs may result in better outcomes can be given in the future, will for a large part rely on the further development of this line of research. The research synthesis presented in Section 4 showed various knowledge gaps that could serve as starting points for following studies; I name four.

First, as the current evidence base is very sparse, future studies of integrated strategies can further examine the insights that this paper has provided and consequently confirm, adjust, or complement them. Importantly, this needs to be done in a more systematic and rigorous manner. Section 4.4 showed that the methodological approaches of existing studies have not been particularly convincing; many studies did not reflect on methodological choices at all, while others were rather summary. Of course, this does not necessarily mean that these studies have not been performed rigorously and systematically, but this cannot be traced, let alone that they can be replicated. As a consequence, the insights that they provide, and therefore also the bases of this very review, should be treated with some reservation. This critique may not be unique to this stream of literature; still, improved methodological rigour and the use of innovative methods of data collection and analysis would be required to strengthen and expand the current evidence base. Various promising methodological approaches have been put forward in this respect, including experimental models and qualitative evaluation methods (for a good discussion of different approaches, see: Knill and Tosun 2012, 179–186). The study of Black, Pérez-Escamilla, and Rao (2015) provides some good examples of the former, using (integrated) programme exposure and the random selection of communities as methods for controlling for alternative variables. In addition, the state of knowledge about IPSs would benefit from more comparative research designs. Comparisons of how integrated strategies function in different contexts will result in a better understanding of the influence of differences in how governance functions are performed and by whom (cf. Peters and Pierre 2016).

Second, more attention ought to be given to how integrated outcomes are socially constructed, that is, the political evaluation rationale. Vince’s research suggests that achieving success in a political sense is particularly challenging in complex actor-constellations involving different sectors, levels, and arenas (Sections 2 and 4.2). At the same time, very little is known about such processes. As in politics perceived success is at least as important as measurable achievements and as the latter is no guarantee for the former (Bovens and ‘t Hart 2016), these evaluation dynamics would be an important research avenue; both to arrive at a fuller scholarly understanding, but also to make attempts at stronger and more durable integrative commitments (cf. Jochim and May 2010). It would hereby be important to also pay more attention to the target groups of integrated interventions. Many integrated strategies ultimately aim to achieve a better service delivery or more effective way of addressing societal problems, such as food insecurity or environmental issues. So far, little is known about whether and under what conditions beneficiaries, professionals, and other people in the ‘field’ indeed experience these pay-offs.

Third and building forth on the last points, little is known yet about factors that enable the implementation of IPSs. Although it may well be the case that failure is more widespread than success, the almost exclusive focus on failure has prevented acquiring insights into key conditions for integrated strategies to work. In addition, limited attention to success does not help in explaining why and how institutional and agency-centred factors affect IPSs’ outcomes, and whether they are necessarily factors of failure. Peters (2015, 43) puts this nicely by stating that ‘[i]f we have only failure stories then the very same factors we can identify as being present in policy failures may actually also appear in successes, meaning that we need to look for other causes for the observations’.

Fourth and related to Peters’ quote, Section 4.3 showed that previous studies have accounted for failure and (to a limited extent) success by pointing at relatively static and monolithic ‘factors’ or ‘barriers’. Many of these factors will not be unexpected to public administration and policy scholars, as they largely correspond with existing insights from the implementation literature. However, depending on their specific properties, interactions, and contextual conditions, many of these factors may contribute to both success and failure. A next step would therefore be to move beyond such lists of factors towards uncovering underlying mechanisms (cf. Hedström and Swedberg 1998; Biesbroek et al. 2014). Mechanism-based approaches explain a policy outcome through a configuration of mechanisms, which requires ‘understanding of the collection of actors (a1, … , ax), and their activities (b) under context conditions (C), as present during time period (t), so that the configuration of mechanisms (m1,  …  , mx) exhibits the plausible causal role (y) of the impasse (l)’ (Biesbroek et al. 2014, 110), or any other type of outcome. So far, the policy integration literature has only touched upon the relevance of such mechanism-based explanations and provided some generic examples (Biesbroek and Candel 2016), but a comprehensive research agenda remains to be unrolled.

Furthering studies on IPSs along these lines will result in both a more comprehensive and firmer scientific evidence base as well as a better informed point of departure for designing, deciding about, and implementing integrated approaches. It hardly needs saying, but may be good to repeat here, that eventually this knowledge and consequent interventions are hoped to have positive impacts on society. Only through continued conceptual and methodological innovation we can determine whether and under what conditions that is the case.

Notes on contributor

Jeroen Candel is an assistant professor at the Public Administration and Policy Group of Wageningen University, the Netherlands. His area of expertise is food (security) governance and policy, which he approaches through a Public Policy lens. He has a particular interest in conceptualizing and measuring policy integration. Dr. Candel has published in both high-ranking public policy and food journals. Beside his research and teaching, he engages closely with policymakers and stakeholders to advise about improving food governance arrangements and practices.

Acknowledgments

The author would like to thank Robbert Biesbroek and two anonymous reviewers for valuable comments on a previous draft of this article.

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1 Which can in their turn result or be embedded in an IPS.

2 In McConnell’s approach, process success or failure refers to the extent to which governments are successful in carrying out their basic governance processes, such as problem identification, policy formulation, and decision-making (cf. Peters and Pierre 2016); programme success is about whether a policy’s goals are achieved; and political success is whether decision-makers manage to reap political benefits from a policy. Contrary to the approach of Bovens and ‘t Hart, McConnell thus approaches programmatic and political success more in terms of the locus of success than as distinct evaluation rationales (which was pointed out by Bovens in a commentary, see: Bovens 2010, the arguments of which are the reason to adopt the distinction between programmatic and political rationales in this paper.

Appendix 1. Search terms

Because few studies of IPSs explicitly label these strategies as such, the scope of the query was extended to all studies that take an explicit ‘policy integration’ or ‘integrated policy’ perspective (i.e. they mention these terms in title, abstract or keywords). This approach resulted in retrieving a relatively higher number of publications that proved irrelevant for the purpose of this paper, but for the sake of comprehensiveness their titles, abstract and keywords were all carefully read and judged on the basis of the selection criteria.

In addition to ‘policy integration’ and ‘integrated policy’, the search term ‘boundary-spanning policy regime’ was used, as these regimes have been shown to be often undergirded by an integrated strategy (Jochim and May 2010; May, Jochim, and Sapotichne 2011). However, eventually this search term did not produce additional evaluatory studies.

The final queries, both performed on 1 April 2016, are:

Scopus:

TITLE-ABS-KEY (‘integrated policy’ OR ‘policy integration’ OR ‘boundary-spanning policy regime’)

Web of Science:

TOPIC: (‘integrated policy’ OR ‘policy integration’ OR ‘boundary-spanning policy regime*’)

Indexes = SCI-EXPANDED, SSCI, A&HCI, ESCI Timespan = All years

Appendix 2. Data extraction matrix

Author(s) and year	Crosscutting problem(s) or issue(s) that is (are) being discussed and locus	Key concept(s), focus, and methodological approach for studying/discussing (success/failure of) IPSs	Programmatic and political assessment of success/failure of IPSs	Other relevant arguments in the paper and notes about the paper
Alves et al. (2013)	Coastal Zone Management in Portugal	Integrated Coastal Zone Management (ICZM); Implementation; Governance mechanisms. No account of methodological approach. Discusses implementation assessments by European Commission and Portugese government	Programmatic: Implementation assessment by Portuguese government over period 2006–2010 showed that the implementation of a majority of ICZM measures had not yet been started or was ongoing. Previous assessment by European Commission showed strengths and weaknesses. Strengths: National strategy supported EU guidelines, legal support was reasonably consistent and including a long-term perspective, adaptive management gradually being integrated, necessity to use wide range of tools had been recognized. Weaknesses: Lack of attention to climate change and sea-level rise in monitoring, too general measures for protection vulnerable coastal areas, unclear stakeholder engagement, inadequate educational information, insufficient international cooperation, need to improve regional policies. Political: Not discussed	Focuses on outcomes in governance processes and practices, not ultimate outcomes on the ground. No reasons given for implementation shortcomings
Begg and Gray (2004)	Traffic levels and vehicle emissions in the UK	Integrated policy; Joined-up philosophy. No explicit reflection on methodological approach. Reviews evolution of government transport policy to examine the extent to which transport and environmental policies have diverged from the 1998 Integrated Transport White Paper (ITWP), which set out a fully integrated policy to tackle the twin pillars of congestion and pollution	Programmatic: Paper argues that the ITWP has never really been brought in practice. Instead, government has shifted away from ‘joined-up’ policy-making towards a narrower transport policy focused on supply side investment – with less emphasis on demand management – and a separate environmental policy centred on improving fuel efficiency and reducing emissions through legislation and agreements with motor manufacturers. Reason: Although congestion did emerge as a more pressing concern, this does not in itself explain the shift away from the joined-up approach set out in the ITWP, particularly the need to reduce car dependence. A combination of increasing public dissatisfaction, political shocks and institutional change also combined to effect a shift away from integration to political pragmatism. Ministry of DETR was split up in separate transport and environmental ministries. Political: Not explicitly reflected on, although argue that from the beginning there was a lack of political interest/salience. More interest in congestion than pollution (also because vehicle emissions were decreasing as a result of national and EU legislation)	Argue that more joined-up policy-making is required
Black, Pérez-Escamilla, and Rao (2015)	Nutrition and Child Development; No specific locus	Integrated interventions; Implementation. Review of scientific basis for the impact of integrated interventions on children’s linear growth and cognitive development after the first 1000 days and implementation strategies for integrated nutrition and child development programmes	Programmatic: Discuss four experimental and quasi-experimental studies of integrated interventions that all found a positive impact on children’s’ cognitive and/or socioemotional and or/motor development (two in Colombia, one in Bolivia, one in Mozambique). Integrated interventions consisted of exposing children to nutritional supplementation, psychoeducational stimulation, and/or training of parents and child care providers. Impacts on growth were mixed; in one study lack of impact was attributed to poor implementation of the nutrition component of the intervention. Find that, beyond above programmes reviewed, the number of evaluations has been limited. Find various opportunities and challenges in the implementation and evaluation of integrated programmes, including: staff workload, training, and supervision; cultural adaptation of early learning intervention material; monitoring and evaluation; sectorial funding and priorities, and political will. Political: Not discussed	Not clear whether all interventions were government-led. Implementation opportunities and challenges often generic, rather than associated with a specific programme
Casado-Asensio and Steurer (2016)	Climate change mitigation in Western Europe	NMSs. Desk research that draws from existing publications, documents issued by international organizations, national governments, and other public agencies	Programmatic: Paper compares NMSs in EU-15 and reports on their performance, whereby the paper has separate sections on processual implementation and impacts on greenhouse gas (GHG) emissions. Process: Negative Interest in NMSs beyond environmental ministries usually weak in implementation phase. Lack political salience, worsened by financial crisis. Consequently, inter-sectoral coordination is inadequate across Europe. Perform even worse in coordinating the various levels of government, also in federal states. Regarding the inclusion of non-state actors, broad consultation is rare, transparency is low, and participation in NMS processes does not translate into more coordinated, legitimate, or effective policies. Evaluation is often sectoral and does not result in learning. GHG emission reductions: Limited evidence, based on comparing emission trends with qualitative assessments of NMSs. Most countries outperformed targets, but argued that this is hardly due to NMSs, because: (i) in a number of countries few measures in NMSs were implemented, (ii) policies that have been implemented and contributed to reductions were often unrelated to NMSs, (iii) parts of reductions due to other, non-mitigation policies, (iv) parts of reductions due to developments outside of policy domain. Combined: Not a single strategy was effective in integrating climate policies across sectors and levels. Political: Not discussed	Authors note restrictions of desk research method used for assessing role NMSs in cutting GHG emissions.
Drimie and Ruysenaar (2010)	Food security in South Africa	Integrated strategy; Institutional analysis No explicit reflection on methodological approach. Study of institutionalization of Integrated Food Security Strategy (IFSS) and associated challenges are provided to demonstrate the importance of institutional arrangements to achieve food security	Programmatic: Argue that institutional arrangements and a disjuncture between the IFSS and the reality of food insecurity in SA present barriers to any meaningful implementation. The response strategy, as implemented, is inadequate to engage food security’s complexity. Conclude that IFSS has largely failed in its mandate as result of insufficient and inappropriate institutional arrangements to underpin the strategy. Secondly, complexity that defines FS in SA has been inadequately conceptualized and engaged within the IFSS, as an emphasis on agricultural production and food availability are not the core tenets of food security in the country. Issues around food accessibility clearly take precedence, but are entwined with issues of availability, utilization and stability. Number of institutional constraints that have limited success of IFSS identified: *The government department appointed to coordinate and facilitate the integrated strategy inside government has failed to do so in a comprehensive fashion, as its focus has been on a prosperous agricultural sector rather than ensuring ‘food security for all’. This has led to a ‘bias’ in the food security response in the country, which has inadequately engaged with the challenge. *The coordination of FS has been tasked to a directorate that does not have much administrative capacity. As such, the directorate has no mechanisms to drive the process or resource to ensure that other departments, let alone directorates in its own organization, work within the strategy. *There are no dedicated funds for government to spend on food security at all administrative levels. All budgets have been allocated by sector, preventing the emergence of joint projects and programmes, funded by one entity. *The absence of a Food Security Policy or legislative framework prohibits government from providing a clear line of authority, as well as means of enforcing non-collaboration and implementation of relevant programmes in a disjointed manner. *Stakeholder dialogue with civil society and indeed within government has been minimal. With access the cornerstone of food security and directly or indirectly reliant on income, Social Cluster interventions are quite possibly undermined by the lack of involvement from the Economic Cluster. Departmental structures aside, the IFSS has not sufficiently crystallized baseline information to give an indication of food insecurity in SA, despite significant investment into an information management system. Has not ensured a comprehensive monitoring and evaluating of the various food security programmes in progress and qualifying their impacts. Furthermore, it does not appear that there is sufficient information allowing for any alignment to stated policy objectives, let alone estimation of how far on track SA is to meeting those objectives	Facilitation of ‘joined up government’, although in existence in theory, requires a concerted effort and recognition of the issues within a wider array of governmental departments, and elsewhere, for it to become a reality
Hertin and Berkhout (2003)	Environmental concerns in EU enterprise policy	EPI; Ecological modernization; Institutional strategies Desk-based review of policy documents and series of interviews in Commission and EU member states to analyse integration of environment into EU industry and enterprise policy	Have developed own framework for analysis and evaluation of EPI (four core functions: sectoral agenda setting, horizontal communication, sectoral capacity building, policy learning), applied to implementation of EU EPI strategy in enterprise policy. Programmatic: Industry Council of EC invited Commission (DG Industry, later Enterprise) to develop EI strategy. Resulted in Commission staff working document, some progress made. Changed when Director-general was replaced. Eventually, very little substantial progress made. DG Enterprise and the Industry Council were unable or unwilling to present a comprehensive assessment of environmental issues in the sector, an action plan, or a reporting mechanism. The development of an operational industry/enterprise integration strategy has been postponed until the end of 2004. Application of framework: Appears that considerable agenda setting has taken place. The Cardiff Process has obliged policy-makers and officials to consider the link with industry/enterprise policy and generated a stronger political mandate for further environmental activities. Although Cardiff seems to have contributed to the development of some new environmental capacity in DG Enterprise, it was clearly insufficient to respond adequately to the mandate of the European Council. There is also no evidence that the sectoral EPI strategy has led to a substantial increase in horizontal communication. While a shift towards a more positive attitude towards the environment is noticeable in high-level policy documents, the Cardiff Process has not challenged the sectoral belief systems and institutional power structures in a more fundamental way. The relationships between the industry and the environment sector remain antagonistic in most areas. Overall, the impact of the initial three years of the Cardiff Process on EU industry and enterprise policy has been modest	No single strategy, but multi-annual EU commitments (embedded in various strategies/treaties) to EPI (Amsterdam Treaty, Cardiff Process). Conclude that specific weaknesses of sectoral integration strategies suggest that they will not be successful by themselves. EPI requires a heterogeneous set of conditions, such as political will or pressure, suitable procedures, resources and knowledge
Jordan and Lenschow (2000)	Environmental concerns in Denmark, Netherlands, Germany and UK	EPI; Greening; Implementation Draws on previous analysis of the implementation of EPI by the greenest member states and EU institutions (Lenschow in press) and 4 OECD reports to examine how well four frontrunner-states are responding to EPI commitments	Programmatic: Interestingly, though not surprisingly, member states have chosen different paths to achieve EPI that reflect own cultural and political make-up. Analysis reveals that even the most environmentally progressive member states are still a long way short of achieving what might be considered ‘full’ or strong EPI. UK: Try to green its traditionally strong and internally well-coordinated systems of central government. However, performance to date demonstrates unequivocally that EPI requires more than just strong structures and procedures. Also requires strong political backing from the highest levels of government in order to win reluctant departments. Germany: EPI initiatives only slowly being introduced. Main bottleneck is the lack of effective horizontal coordination at ministerial level. Lack systematic system of SEA. Germany’s frontrunner status perhaps more of a reflection on its past successes in regulating pollution than in addressing the more diffuse sustainability agenda. NL: Overwhelmingly interpret EPI as societal rather than governmental responsibility. Made considerable progress in integrating environmental concerns into sectoral policies at all levels of government. Economic appraisal techniques have been extensively used to determine policy strategies to reach the targets identified in negotiations with stakeholders. Denmark: Scores relatively highly on checklist. Scope for removing more environmentally damaging subsidies and reducing demand for transport still remains large, but proving that environmental protection can be reconciled with economic growth. Progressive force for ecological tax reform. To conclude, obvious danger in expecting too much to emerge from Cardiff Process if states do not take appropriate measures to implement EPI at home. Much remains to be done in this regard	No single strategy, but multi-annual EU commitments (embedded in various strategies/treaties) to EPI (Amsterdam Treaty, Cardiff Process)
Kivimaa and Mickwitz (2006)	Environmental concerns in Finnish technology policies	EPI; Implementation. Performed documents analysis and three interviews to evaluate the impact of the inclusion of environmental aspects in the strategies of the Science and Technology Policy Council on policy outputs and outcomes	Programmatic: Authors evaluate whether national integrated strategy led to the inclusion of environmental concerns by the National Technology Agency (Tekes) at three levels: (i) in technology programmes, (ii) in project funding decisions (policy outputs), and (iii) policy outcomes. For technology programmes, environmental objectives were included in most programmes studied. However, objectives were rarely all inclusive and focused mostly on energy efficiency or emissions. For policy outputs, environmental impacts are, in principle, among the criteria for selecting projects. In practice, however, the funding application forms do not itemize a section for environmental impacts. Environmental aspects are not considered as relevant in comparison to the technological or business potential. Ex post assessments of project outcomes do not mention the environmental impacts of the projects. Policy outcomes: Authors note that technological innovations result from a variety of driving forces, of which technology policy measures are only one. Use some ex post assessments of programme outcomes, made by participating parties, to report some alleged beneficial outcomes. End with stressing that some kind of ex ante and ex post environmental IA procedures would be needed to systematically evaluate EPI in outcomes. Conclude that environmental considerations have been included selectively, but have had no priority over other issues and emphasis diminished in recent years. Systematic EPI in policy outputs and outcomes is fairly weak. Mixed results partly stem from design of the integrated strategies themselves, as these did not include specifications about how environmental aims are to be followed up or reported. Political: not discussed	Note that it is often easier to evaluate integration at the strategy level than at the lower levels of the implementation chain, especially outcomes. Partly due to availability of documentation material
Klepp and Forster (1985)	Nutrition and public health in Norway	Integrated policy approach; Implementation; Production and consumption No explicit reflection on methodological approach. Use existing reports and data to review (i) implementation of the Norwegian Nutrition and Food Policy after its ratification in 1976, and (ii) changes in production and consumption since implementation of this policy	Programmatic: Implementation (process): Strategy had four aims: encourage healthy dietary habits, global aid, promote consumption of domestically produced food, and strengthen the rural economy. It was implemented through various strategies: consumer and producer price subsidies, marketing measures, specifications regarding the content and composition of foods, research, consumer information, and nutrition education in schools. Initial emphasis was on price subsidies; inconsistencies occurred because these subsidies were also used for economic and agricultural productivity goals. Emphasis shifted towards economically and politically low-coast education strategies. Consumption: Authors acknowledge that data have potential bias but nevertheless indicate changes in trends. Overall data available suggest changes in food consumption led to decrease in fat consumption from 41% of the total energy consumption per person in 1975 to 38% in 1983. Evaluation of Nutrition Campaign 1982–1983 showed that 72% of population was aware of the campaign, the book Everyday Food sold 100,000 copies and 70,000 of the follow-up. There has been no reported attempt to assess changes in actual eating habits as a result of this information effort other than to examine the national consumption data. It is virtually impossible to relate these changes in eating habits to disease outcomes such as incidence of cardiovascular diseases (which decreased, but may be due to other factors). Global aid: Percentage of GNP given as aid to developing countries increased, large proportion directly related to food and nutrition programmes. Food production and rural economy: Level of farm incomes and the area of fully cultivated land increased between 1975 and 1983. Degree of national self-sufficiency, however, remained fairly constant (largely due to increase of sugar imports). Conclude: Does this mean that the integrated approach is not workable? Certainly some of the evidence supports that conclusion. Change in food consumption is only modest, and subsidies are not consistent with the nutrition and food policy. On the other hand, there are various reasons for counting the policy at least a partial success: some changes in consumption and production nevertheless and increased foreign aid. Most significant is the potential for long-term, lasting change. Political: Not really discussed, although note that strategy has survived three governments and has support of all major parties in Parliament	Note that evaluating the impact of the policy is difficult, but ask why impact of implemented policy measures hasn’t been greater. Argue that trade-offs may have occurred in use of subsidies to promote agricultural sector (largely livestock). Failure to completely achieve the goals of the policy seems to be due to incomplete implementation. A potentially effective measure such as the food subsidy has not really been used to promote the policy goals and objectives
Mickwitz and Kivimaa (2007)	Environmental concerns in innovation policy and innovation concerns in environmental policy in Finland	(Two-way) Policy integration; Evaluation; Strategy; Instruments. No explicit reflection on methodological approach, but see Kivimaa and Mickwitz (2006). Examine how policy integration can be evaluated, which is demonstrated through study of integration environmental and innovation concerns (strategic reviews of Science and Technology Council, Tekes strategy document, Finnish Government Programme for Sustainable Development, Water Protection Targets for the Year 2005)	See also Kivimaa and Mickwitz (2006). Here, they also include analysis of integration innovation concerns in environmental policy. Use self-developed integration evaluation matrix. Programmatic: Results indicate that environmental concerns are fairly well integrated into technology policy strategies. At the policy instrument stage, that is, in the technology programmes and the R&D projects financed, integration was less complete. One could even speak about deliberate disintegration at the technology programme level, since in practice separate programmes have focused on environmental technologies, while environmental issues are integrated into other programmes only to the extent that the positive impacts are recognized in the application process. The aim to promote the development, commercialization and adoption of innovations through environmental policy is only partly integrated in the strategies studied. The idea that innovations should not be hindered has been largely internalized but, although promoting innovation is included in the Programme for SD, it is not incorporated into the Water Protection Programme or in the active permitting practices. Despite aiming to ensure that Best Available Technology (BAT) is spread, the ways in which environmental policies could be deliberately used to promote the development of BAT is not considered. Political: Not discussed	Focuses on four strategies, but definition of strategy is rather broad (as is distinction between strategies and instruments). Actual point of departure seems STC’s reviews, which stress integration of environment & innovation. Water policy is not integrated at all. Argue that only by including outputs and outcomes in the core of the evaluation the effects of procedural integration can be assessed. Argue that two-way mechanism of integration may be more successful
Nilsson, Eklund, and Tyskeng (2009)	Environmental concerns in waste management in Sweden	EPI; Implementation Performed series of case studies from waste-incineration plant decisions in five cities (interviews and documents analysis) to study how national waste policy (that has traditionally integrated environmental concerns) is implemented at the local level	Programmatic: Authors find an implementation gap between national waste policies and local decision-making. All cities that were studied embarked on construction of waste-incineration plants in the 2000s, while these have a low priority in the national waste hierarchy. The authors argue that perceived market opportunities (income from selling heat and power) prevailed in local decision-making. In addition, respondents indicated that they found national objectives and instruments unclear and often contradictory. Local decision-makers were rather unconcerned with the ‘soft’ national framework and were primarily concerned with the tax requirements and the market situation. In addition, although local waste planning is mandatory, it is found to be ineffective, and it can be questioned whether it has any role at all. Not all municipalities have updated plans. Third, whereas at national level NEQOs (system of sector responsibility for environmental performance) have been given an important role, these are hardly applied in local decision-making, resulting in a discrepancy between broader political intentions related to environmentally sustainable development and actual decision-making.	Not really about a single cohesive strategy, but about national waste policy that consists of series of bills (study is included because of the long tradition of integrating environmental concerns in these bills). Focused on processual impacts rather than eventual outcomes.
Pollack and Hafner-Burton (2010)	Environmental and gender concerns in the European Commission	Mainstreaming; EPI; Hard and soft incentives; Implementation. Study effects of Commission’s strategies for gender mainstreaming and EPI for organization’s processes and policy outputs of DGs and services. No detailed methodological account, but use congruence analysis (looking for predicted correlation between hard/soft incentives and policy outputs), process-tracing. Used Commission documents, existing studies, and own fieldwork over course of decade	Programmatic: Authors find that for both gender mainstreaming and EPI the Commission has primarily relied on soft incentives, which produced highly variable, selective, and (from a normative perspective) disappointing ‘take up’ of the respective mandate. For gender mainstreaming, the study finds that outside of a small number of pioneering DGs, efforts to socialize and persuade (soft incentives) officials to ‘take gender seriously’ have failed to significantly change behaviour. Counterfactual in argument is that hard incentives would lead to more consistent implementation. Authors acknowledge that they cannot test this claim directly but refer to the Commission’s Fourth Action Program on Equal Opportunities for Men and Women in the Commission, which focuses on the recruitment, retention and treatment of qualified women officials. Early programmes employed soft measures, resulting in only marginal improvements in most DGs. But the 4th Action Programme (2004), incorporated hard incentives, resulting in rapid, quantifiable improvement across virtually all DGs. The Commission’s 1993 EPI programme also primarily contained soft incentives. The result was a highly variable and generally disappointing take-up of the EPI mandate, which failed to produce substantial changes in policy outputs even among priority sectors like energy and transport. For the subsequent Cardiff Process, again incentives for sectoral policy actors to prioritize environmental concerns were negligible, and within a few years the initiative fell to disuse. The Union’s most recent EAP seeks to develop cross-sectoral thematic strategies and incorporates environmental considerations alongside others in a new impact assessment procedure. Also mostly soft ‘new governance’ mechanisms, limiting their impact on policy outcomes. Exception is IA procedure, which has been imposed through a series of increasingly strict, centralized, and enforced guidelines, with early signs of improved performance over time. Political: Not really studied, although authors implicitly argue that both programmes have been perceived as largely failing	Argue that measuring IO performance through outcomes is both methodologically demanding and risks conflating the actions of member states with the performance of particular IOs. Therefore, most studies focus on process. However, even elaborate process may in practice fail to produce ‘mainstreamed’ sectoral policies. Therefore consider policy outputs most reliable measures
Russel and Jordan (2010)	Environmental concerns in the UK	EPI; Implementation Assesses implementation and performance of the UK’s EPI strategy and system over time; no explicit methodological reflection	Programmatic: Start with observation that at least on paper UK EPI system is innovative and contains many more tools, mechanisms and processes than most other member states. However, works out differently in implementation; focus on five elements: (i) administrative and bureaucratic tools, (ii) green budgeting, (iii) strategies and/or strategy developing processes, (iv) policy appraisal system, and (v) strategic environmental assessment. Evidence presented suggests that these have not been uniformly effective; some elements have been spectacularly ineffective and some very weakly implemented. Regarding outcomes, although it has been in existence for 15 years, the UK’s EPI system appears to have not significantly improved the state of the UK’s environment. However, some recent developments that might improve the UK’s performance in the future. Conclude that implementation of EPI is quite sectorized, with its individual elements not working as well as envisaged. Political: Not really discussed, although start from the puzzling observation that the OECD applauded the advanced machinery of coordinative governance, while others doubted EPI performance in practice	Not really a single EPI strategy (although influential 2002 White Paper), but rather EPI system that comprises ‘blend of governance modes’, under Labour also including Sustainable Development institutions. Use Metcalfe scale to argue that UK’s use of grand SD strategies is arguably not satisfactorily supported by lower level mechanisms. Moreover, high-level committees do not appear to be proactively putting political pressure on the lower levels to raise conflicts to their attention. Consequently, cross-sectoral coordination simply does not get going. Also: little capacity and NGO engagement
Schout and Jordan (2005)	Environmental concerns in the EU	EPI; Governance; Implementation Extensive interview round to study EPI implementation (capacities) at EU level, and in UK, NL, and Germany	Programmatic: Interviews reveal that, thus far, no actor has put sufficient effort into building the required capacities in the form of EU and national networks. Instead of collaborating in order to ensure better coordination, everyone more or less expects someone else to implement EPI. EPI implementation has suffered from: (1) major differences in administrative capacities; (2) poor coordination in the Member States and the Commission; (3) weak network relations; and (4) ineffective common planning and priority setting. In addition, there is a danger of insufficient leadership since both national and Commission officials lack the incentives and the necessary overview to assume such a role. Political: Not really discussed, although point of departure is that many commentators consider EU EPI implementation as disappointing	No single strategy, but multi-annual EU commitments (embedded in various strategies/treaties) to EPI. Implementation failure is point of departure, rather than outcome of study, though various manifestations/reasons are provided. Argue that implementing networks may require considerably more central steering than EU White Paper appears to assume
Söderberg (2011)	Environmental concerns in Swedish bioenergy policy	Multi-sector EPI; Institutional conditions Performed textual analysis and interviews to study ‘what institutional factors seem relevant for multi-sector EPI and its outcomes’	Programmatic: Implementation of multi-sector EPI seems successful. Benefits from institutional arrangements which allow for: (i) open actor access to policy-making, (ii) use of environment-related knowledge, (iii) monitoring mechanisms, and (iv) both environment-sectoral and inter-sectoral coordination. Regarding policy outcomes, study argues that bioenergy ventures seem successful, as Swedish bioenergy production has nearly doubled in two decades. CAP/EU policies placed constraints at some point and later supported bioenergy production within agriculture. Conclude from this that EPI alone is not a guarantee for environmentally beneficial outcomes, at least not in multi-sectoral issues. Political: No explicit reflection, but notes that Swedish bioenergy EPI has survived government turnover	Paper is a bit problematic, both conceptually and methodologically. Central notion of ‘multi-sectoral EPI’ in bioenergy policy is rather unclear and seems a contradictio in terminis (as bioenergy is after all a policy (sub)sector). Hypotheses are presented, but not tested in a convincing way; are confirmed without sound supportive evidence base
Steurer and Clar (2015)	Climate change mitigation in Austrian building policies	Climate policy integration; CPI as governance and as Policy outputs and outcomes; Federalism Performed desk research and interviews to study the implementation of EU, federal and provincial policies aimed at greening the building sector and to determine whether Austria’s federal structure played an enabling or constraining role in this implementation process	Programmatic: While Austria’s annual GHG emissions rose by 5.9% between 1990 and 2011 (point of departure of paper is that Austria is performing poorly in mitigation), emissions of the building sector decreased by 25.5%, that is, 9% beyond the target of the Climate Strategy 2007. As more and bigger residences have increased GHG emissions of the sector, this reduction is even more impressive. However, the authors argue that these outcomes cannot be explained by CPI efforts, given the fact that only one of the federal agreements on building standards exceeded the status quo of new building and that most other federal and provincial mitigation efforts were either politically irrelevant (both climate strategies, the climate protection law and its amendment) or resulted in a federal zero-sum game (refurbishment cheque). Conclude that market forces rather than public policies were the main drivers behind the positive developments. Political: Not explicitly reflected on, though authors start from observation that Austria is perceived as underperforming regarding its climate change mitigation and show that there is widespread scepticism or lack of interest in CPI across levels of government	Argue that since the decentralized building sector is one of the few sectors in Austria that has reduced its GHG emissions, a quantitative study would most likely be misled to conclude that the Austrian federal setting facilitated mitigation. In contrast, qualitative analysis shows that it was a major obstacle for greening the building sector. Point at diagonal coordination challenges, turf wars and power struggles, and lack of ownership among provinces
Vince (2015)	Australia’s Oceans Policy	Integrated policy approaches; Policy failure (Programmatic, Process and Political) No explicit reflection on methodological approach; explores why large-scale integrated approaches are prone to failure by analysing the case of AOP	Elaborate discussion of aspects of policy failure, summarized in table. Programmatic: *Did not achieve intended goals: Original objective to achieve integration across sectors and jurisdictions was never achieved. *Did not achieve timelines: Whole process was slow due to: focus on only completing one RMP (Regional Marine Plan) in the most complex bioregion in Australia; NOO’s (National Oceans Office) newness and ambiguities of its role as an executive agency; lack of resources; other actors’ frustration with the process. *Has poor policy design (types of processed used – drift, conversion, layering, displacement and exhaustion): New policy and institutions were ‘layered’ over existing legislative arrangements that supported sector-based approaches to management. States were only involved through their membership in ANZECC. Process: *Encountered issues with multiple sectors: Sectors were frustrated with the process and each other. Conflict and distrust between sectors were reinforced due to existing legislative arrangements *Encountered issues with multiple jurisdictions/levels: The state and territory governments did not formally endorse the oceans policy. Their involvement in the South-East Marine Plan was limited to the administrative state ‘officer’ level. The policy only applied to Commonwealth waters and a high level of distrust was felt by the states about their limited role in the process. *Encountered issues with multiple actors – state, non-state or private. Identifiable winners and losers: Losers included the states through their limited participation in the development, implementation and decision-making process, and the NOO which was set up with a difficult task of integration but without the infrastructure and resources to achieve this goal. Lack of communication, policy coordination and cooperation between actors: NOO’s limited roles and power resulted in sectoral distrust and blockages. Other government departments reluctantly worked with NOO. The location of NOO also hindered communication and cooperation. *Did not demonstrate policy coordination, policy capacity or integrative capacity. Through new institutions: NOO lacked integrative capacity due to a lack of available resources, limited decision-making power and its location. It failed in its task to coordinate integration between sectors. NOO lost its integrative capacity when it lost its executive agency status and was absorbed into the environment department. NOAG’s (National Oceans Advisory Group) role was also limited to being an advisory body, and this limited its integrative capacity. Through existing institutions: ANZECC had a limited role in the oceans policy process only commenting on environmental issues. Existing government agencies such as the environment or fisheries departments had policy capacity to deliver outcomes but were reluctant to do this beyond their jurisdictional/sectoral scope. Establishing MBPs through the EPBC Act resulted in a shift from a holistic oceans policy to an environmental policy coordinated by the environment department. Political: *Policy process was used for political gain. Incentives for declaring new programmes: Harradine used political incentives to locate NOO in Hobart. Political incentives were also used to legislatively anchor policy through the EPBC Act. Focus was placed on the new programme rather the demise of the integrated oceans policy process. Actors exercised blame avoidance: Actors avoided blame for the failure of the oceans policy by focussing on implementing MBPs and NRSMPAs. *Actors used political gain to change or challenge The policy goals: Hill’s decision to make it a Commonwealth policy (without the states) in late 1998, resulted in full integration being impossible to achieve. The effectiveness of the policy instruments and the existing governance arrangements: The effectiveness of policy instruments and governance arrangements were questioned through the oceans policy’s review. The review enabled the subsequent ministers to slowly disassemble the ocean institutions and implementation processes (RMPs). *Political actor location and parliamentary interest: Executive leadership decisions led the policy to failure. Lack of any parliamentary attention dissipated interest in the oceans policy	Three main reasons failure: (i) policy and institutional design did not reflect full- or large-scale integration; (ii) the layering process of new policy and institutions on established silos that were designed to keep sectors separate was ineffective; and (iii) the institutions themselves did not have the integrative capacity required for successful implementation. Has been an experimental disaster
Vince et al. (2015)	Australia’s Ocean Policy	Integrated policy No explicit reflection on methodological approach. Paper examines what can be learned about integrated oceans governance through the Australian experience in developing and implementing AOP	See Vince (2015) for detailed account. Paper shares findings, although more positive conclusion: while the policy itself may not have made as much direct impact as was hoped, its influence and indirect impact on the policy setting and broadening of environmental objectives for sectoral management has been significant. It is clear that the policy has failed to deliver all that was hoped of it, but it did have a profound impact on the way the marine environment was regarded across multiple sectors	Findings of article largely coincide with Vince (2015)

Note: NMSs, National mitigation strategies; GHG, greenhouse gas.