Closing the policy gaps

Introduction

It is a fundamental principle of good public services that decisions are made on the basis of strong evidence and what we know works.

(UK Cabinet Office, 2013, p. i)

One thing you hope for, with politicians, is that they won't make the same mistakes over and over again.

(Goldacre, 2014, p. 174)

The last 20 years have seen increasing academic and public debate about the nature of science, the evidence it generates, and how this feeds into the development of government policies. These issues have been brought to the forefront of public and professional attention, for instance, by media coverage of climate change and the public scepticism that this has engendered (Butler & Pidgeon, 2009; Painter, 2011; Poortinga, Spence, Whitmarsh, Capstick, & Pidgeon, 2011). At the same time, governments have expressly sought to move towards ‘evidence-based’ policy-making (EBPM) grounded in ‘what works' – for example in the US through the establishment of President Barack Obama's evidence-based initiatives (Haskins & Baron, 2011) and in the UK by the government creating What Works Centres to support policy-making and implementation (Cabinet Office, 2013). This trend of focusing on ‘what works’ has itself come under repeated and critical scrutiny (Cartwright & Hardie, 2012; Davies, Nutley, & Smith, 2000; Palfrey, Thomas, & Phillips, 2012).

These developments form a background against which to view this timely special issue of Building Research & Information. It seeks to examine the concerns raised by them for those involved in the production, management and consumption of the built environment. Contributors were asked to explore a series of questions about how to close the gaps between the initial framing and formulation of policy objectives for the built environment and their eventual subsequent outcomes, particularly through monitoring and evaluation. They were invited to do so across a range of fields: urban regeneration and housing market renewal, design for education, the transition to low-energy building stocks, and recent innovations in procurement practice.

Response to the Call for Papers

The Call for Papers received an international response, mainly from within Europe, but submissions also came from North America, Southeast Asia and the Antipodes. The majority of these – as is the case of papers eventually included in the special issue – focused on some aspect of the transition to low-energy building stocks. Relatively few dealt, as suggested, with urban regeneration, housing, design for education or procurement practices, although there are papers in this special issue that are concerned with or touch on all these issues. Again, like most of the papers in this issue, the majority of the abstracts were concentrated on policy initiatives and regulatory changes rather than being used to address the impact of newly promoted technologies or of behaviour change campaigns. Attention was principally centred on national examples with only a few attempts made at international comparisons, as is also the case in this special issue.

The nine papers included in this special issue have common characteristics as well as exhibiting clear differences. All them share a concern with identifying lessons that can be learnt that could inform better policy-making and, hence, help to produce more successful policy interventions and initiatives. Most deal, at varying levels of detail, with the robustness of the assumptions underpinning policy-making and with how evidence could be collected to evaluate these. Most also contain overt (or at least decipherable) signposting to how policies and initiatives could be tracked and evaluated against their intended objectives and outcomes. Very few indicate that such checking of assumptions, or evaluation of objectives and outcomes, has been followed up and assessed in the light of actual experience, particularly through work commissioned by the agencies responsible for the policies or initiatives in question. In this sense, in the built environment at least, there remains a major disconnect – a lack of feedback or feed-forward loops for improving the relationship between the making of policy and subsequent understanding of its intended (or unintended) consequences. In part, this continues to occur, as the papers illustrate, because those who make policy (by framing the nature of the problems that need to be addressed and selecting the objectives that need to be achieved) are situated in national (government) organizations whilst those who are expected to implement them are most frequently locally based, often with their own pressing needs and constraints. Mechanisms for bridging these divides, the papers suggest, are poorly developed in relation to the built environment.

Evidence, policy-making, implementation and evaluation

Some authors interpreted the Call for Papers as assuming a linear process where presumably evidence is collected, policies are formulated, objectives are set, protocols for implementation are devised, and responsibility for implementing is then handed to third parties who may or may not be required (or choose) to evaluate whether intended outcomes have been achieved. A minority of the papers have explicitly questioned this linearity, seeking to broaden understanding of the processes involved, based on their empirical evidence, in order to:

• describe their much more iterative or recursive natures

• illustrate the interactive relationships between problem definition, evidence collection (and exclusion) when framing and reframing of policy options and priorities

• signal the ambiguities that may arise along the way – often giving rise later to unintended consequences

• draw attention to the competition between not just quantitative and qualitative forms of evidence, but to the multiplicity of rationalities and vested interests seeking to interpret these

• comment on the role of networks and communities of interest or practice in information brokering and transfer as well in foregrounding whose voices do or do not get heard and acted upon, and

• explain the need for policies to be based on both facts and values

In short, what the papers included in this special issue point to is the truth encapsulated by Goldacre (2014) in his book entitled I Think You'll Find It's a Bit More Complicated Than That. This complexity is explored below.

The nature of policy

Policy is variously defined as ‘a course or principle of action adopted or proposed by an organization or individual’ (Oxford Dictionaries) or ‘a set of ideas or a plan of what to do in particular situations that has been agreed officially by a group of people, a business organization, a government, or a political party’ (Cambridge Dictionaries). But the term usually comes qualified, as in public policy, policy analysis, policy instruments or policy delivery. As a result, policy activity can be divided into either an intention to act (policy framing/objective/decision) or the means to do so (policy instrument/toolbox/implementation). This special issue addresses both the type of policy decisions that are directly aimed at changing the built environment and the effectiveness of policy instruments that have been deployed by governments and other organizations. Although establishing the aims of public policy may be within the realm of politicians and engaged citizens, it is apparent that built environment professionals need to develop and maintain an informed knowledge base on why and how policy initiatives work. Such a knowledge base would both assist professionals to input into and work directly and effectively with those policy processes that define their ability to deliver successful projects and enable professional bodies to fulfil better their public interest obligations in relation to built environment policy.

The built environment represents a significant part of any nation's wealth, culturally as well as financially, and needs to be cared for, updated and expanded if that value is to be maintained and harnessed. As a result, built environment policies will always form a significant part of any government's plan of action across its many spheres of interest, including industrial, energy and cultural policy areas, but also in social, employment, environmental, health and education sectors. Built environment policy will inevitably be affected by each and every governmental department's spending plans and responsibilities.

The expectations of built environment policy are usually high. New homes, schools and hospitals are intended to deliver better living conditions; improved levels of education, as described by Ive, Murray and Marsh on outcomes from the UK's recent school building programme; and superior health outcomes, as discussed by Mills, Phiri, Erskine and Price on a national healthcare building design quality improvement strategy. New building technologies are expected to deliver cheaper, faster and higher quality facilities while at the same time delivering zero-carbon solutions and creating sustainable communities. As Janda and Topouzi outline in their paper on the expectations generated by the deployment of different narrative treatments, built environment policies can frequently be characterized as ‘hero stories’ and the sector endowed with the imagined capacity to ‘save society’.

Yet the track record for the successful delivery of the outcomes originally intended by the authors of built environment policies is generally patchy or else passed over in silence and remembered only by the victims. King and Crewe (2013), who are pre-eminent UK analysts of British governments, comment:

Typically, people hear about one blunder, then another, then another, without realizing that there are far too many of them to be accounted for by random one-off sets of circumstances and that they may instead have common origins.

(p. ix)

This is certainly true for many building programmes, but work is beginning on understanding the nature of both policy successes and failures and their ‘common origins’. Lessons need to be learnt from them so that more effective built environment policy can be developed in future.

Models of policy-making

Governments can rarely simply decide on a built environment policy without determining in detail how it will be delivered and considering how to overcome the obstacles in its way. This may commonly require extensive consultation and negotiation with stakeholder groups, delivery agencies and private sector partners. In recent decades an international literature has developed on policy formation focusing on the stages through which a typical policy progresses.

Typically referred to as ‘the policy cycle’, the stages are, for example (Strategy Policy Making Team Cabinet Office, 1999):

• understanding the problem – defining outcomes, resolving tensions, identifying stakeholders and deciding their role

• developing solutions – collecting evidence, appraising options, consultation, working with others, managing risks

• putting solutions into effect – communicating policy, supporting those who deliver, testing different options

• testing solutions and making it stick – evaluating success and adjusting action

Other models made be preferred. For instance, the ROAMEF (Rationale–Objectives–Appraisal–Monitoring (during implementation)–Evaluation–Feedback) cycle is favoured by the HM Treasury (2011) in the UK and is cited by Schweber, Lees and Torriti. However, as the Institute for Government stated in a 2011 report:

Virtually every interviewee dismissed policy cycles like ROAMEF as being divorced from reality. Most academics agree with this judgment, and in 1999 the Cabinet Office also explicitly rejected the use of policy cycles, on the basis that practitioners did not feel they accurately reflected the realities of policy making.

(Hallsworth, Parker, & Rutter, 2011, p. 5)

Rather, as the Institute indicated, policy-making does not take place in distinct stages; policies need to be designed, not just conceived, policy-making is often determined by events and the effects are often indirect, diffuse and take time to appear.

This is certainly one of the lessons of the papers in this special issue. Policy-making for the built environment is revealed as a messy business, led by external and internal events, which frequently gets sidetracked during delivery or abandoned altogether as new priorities take hold. Simmons describes the life and death of a policy initiative from the position of an insider. As involved outsiders, Cohen and Bordass relate the history of the UK's response to the European Commission's Energy Performance of Buildings Directive (EPBD) until the policy finally resulted in a fatal combination of ‘an excess of regulation and weak outcomes'. Several authors emphasize the need to recognize that policy-making is a live activity, ‘a continuous process’ (Moncaster and Simmons), needing constant adjustments and periodic refocusing to keep it on track.

While the ‘ideal’ policy is proactive and attempts to be ready, in advance, to shape events to suit a predetermined agenda, experience suggests that most policy-making is more normally reactive to both events and external influences. Several papers describe responses to European Union directives (Schweber et al.; Cohen and Bordass; and Gupta, Gregg, Passmore, and Stevens). Others highlight new priorities that overtake policy thinking even as it is being prepared or implemented (Simmons; Warwick; and Moncaster and Simmons), sometimes derailing it in the process. In the face of this, and the necessarily imperfect nature of policy-making, the innovative approach of outsourcing policy design and development to a neutral, transparent body, bringing government, the research community and industry together as described in Schweber et al.'s paper on the genesis and workings of the Zero Carbon Hub is a welcome movement towards more collaborative and consensual policy-making.

Policy innovation

The focus on new ways to develop and justify policy in recent years has both refined traditional practice and put greater emphasis on the role of evidence in policy-making. But, in practice, there is little that is fundamentally new beyond the ability of technology to accelerate and greatly increase data gathering and analytic capacity. As responses to climate change have revealed, such developments have also made public and media scrutiny a force to be reckoned with on an almost instant basis.

Janda and Topouzi emphasize the importance of framing in both defining and delivering successful outcomes, and Schweber et al. stress the need for framing the evidence in the policy process. But ‘framing’ can also be used as a means of looking at problems only in their best light and against their most favourable profile – in Schweber et al.'s phrase, to ‘limit the range of options that different actors perceive and entertain' in order to ‘provide the basis for alliances and coalitions'. The built environment sector is adept at selecting the best ‘frames’ and in consequence tends to place excessive faith in its abilities to achieve results and make claims that cannot reasonably be justified. It needs to learn to take a more rounded view and grapple with the higher levels of complexity and interconnectedness that most policy analysis requires. Changing from ‘hero’ mode to ‘learning’ mode, as recommended by Janda and Topouzi, may be essential for recognizing realistic limits to policy objectives and doing so could result in reducing the rate and extent of performance failure in the sector.

A major push in policy circles over the past few decades has been towards EBPM (Davies et al., 2000), but with many sceptical of its validity (LSE GV314 Group, 2014) enthusiasm may possibly have peaked. Such doubters include Janda and Topouzi: ‘is this linear, evidence-based approach wholly sufficient to enhance built environment performance, increase public understand and develop “proper” communication with policy-makers?'; Moncaster and Simmons: ‘policy, for all the efforts to present it as being “evidence-based” and to give it an ordering framework, is first of all the product of multiple rationalities and interests'; and Warwick: ‘however much governments call for policy-relevant research, and policy-relevant knowledge to support “evidence-based policy”, the gap between policy-making and practice remains'. Simmons, while not expressing quite the same misgivings, reveals that the whys and wherefores of many of the demands from government for evidence and its limitations may have relatively little to do with making the policy itself more effective.

While some authors cast doubts on EBPM, others continue the hard work of collecting and analysing data, revealing in the process how difficult it can be to source the necessary information on the outcomes of built environment policy. Gupta et al.'s paper on a programme deliberately designed to provide useful information, Retrofit for the Future (RfF), reports that only 45 RfF dwellings out of the 119 funded had available data on post-retrofit measured CO₂ emissions – and only 34 of these had comparable data available for pre-retrofit modelled CO₂ emissions. Ive et al.'s paper on the costs of rebuilding English secondary schools through the UK's Building Schools for the Future (BSF) programme, and the value achieved as a result, sets out a method for calculating such benefits, but the paper also clearly shows the difficulty in assembling even the most basic data to undertake this analysis. Clearly EBPM is still in urgent want of reliable data and has yet to be given the chance to prove its worth in practice.

There is currently a consensus that policy needs to be well designed if it is going to be effective and that this should preferably take place in open and transparent forums such as those discussed by Schweber et al. and Warwick. This has led, in turn, to the idea of policy co-design, whether involving stakeholder communities to draw on their experiential understanding of how things really are, or experts with greater oversight but sometimes less insight into results on the ground, or both. All such approaches have value, but they will greatly benefit from being combined with a well-informed risk-based assessment of the chances of policy success in the face of poor conception, underpowered implementation and, above all, politics (in the form of both contested ends and means).

Politics intervenes

There is no shortage of analysis of the difficulties of meeting long-term strategic goals in a short-termist tactical political arena (Clarke, 2014). This is particularly true for large-scale and slowly developing problems such as climate change and population growth on the demand side, but it also a problem for those parts of the supply side of the economy, such as the development and construction industries that take a long time to deliver and even longer to discover if what has been delivered works as intended. Some specific problems in these sectors include:

• Overcalling

  Exaggerating, whether deliberately or not, both the extent of the problem and even more frequently the ability of these industries to provide effective solutions. Warwick lists the policy objectives that underlay the former UK government's Ecotown initiative – proffered as a quick and easy route to increased housing affordability, economic growth, the utilization of alternative development locations, and the pioneering of sustainable design and technology, any one of which would be a major achievement. The professional and construction arms of both industries eagerly accepted the challenge, only for it to unravel several years later.¹

• Impatience

  Governments are always in a hurry, as Geoff Mulgan, former head of policy in the UK government notes, ‘The highly pressured timescales of government action preclude some kinds of testing and evaluation, and put a premium on quick judgements' (Mulgan, 2009, p. 133), and are ever ready to shift attention to the next initiative even before the previous one has begun to show results. In the rush forward there is a natural reluctance to look back, to undertake post-policy evaluation or to learn the lessons from earlier successes and failures.

• Changes of government, personnel and direction

  Policies are prone to change before they are even implemented and often long before their performance can be judged. Simmons outlines several of these shifts in his insider description of the policy process; likewise both Moncaster and Simmons in their paper on the development and outcomes of an environmentally sustainable schools programme in England describe the changes to policy objectives that occurred in the course of the BSF programme.

• Neglect and decay

  Despite the size of the construction sector in most economies, and its alleged ability to provide solutions to some large-scale problems (see Overcalling above), it rarely gains much consistent or detailed attention from policy-makers. Instead they often expect to be able to switch it on and off at short notice as economic exigencies require. In consequence long-term follow-through can be inadequate to ensure that policy formation and implementation employ the same goals and objectives, and policy delivery is often derailed or even abandoned (Janda and Topouzi, Simmons, and Warwick). Cohen and Bordass discuss how a policy instituted with the best of intentions gradually slips off the political agenda and how ‘any failings can be seen to have arisen more from limited policy integration and follow-through, than from intrinsic faults in the policy instrument'.

• Distraction and noise

  Occasionally policies are developed and announced with the aim of satisfying the demands of the news cycle and providing cover for other day-to-day or less scrutiny-friendly government activity. This not only applies to relatively small but high-profile built environment projects but also to apparently simple, headline solutions to complex problems such as the eco-towns Warwick describes.

• Temporal misalignment

  The time scales of the political cycle have little relationship to the much longer programmes associated with producing or retrofitting the built environment and short-term pressures frequently outweigh long-term benefit. This can lead to deferral of spending whether through instruments such as the Public Finance Initiative (PFI) or by neglect of repair, maintenance and resilience measures resulting in higher lifetime costs than necessary.

• Multiple jurisdictions

  Both policy-makers and policy actors are the servants of many masters including international, regional (e.g. European Union) and national politicians and overlapping legal jurisdictions. This inevitably leads to misleading signals, difficulties in interpretation and limits on the freedom to manoeuvre (Clarke, 2014, p. xviii).

• Events²

  Despite the best of intentions, the mistakes and misfortune that beset policies, whether through neglect, financial pressure or inexperience, are inevitable. They may have become more so as problems tend to become more complex and less tractable while administrative capacity is being reduced and expertise outsourced (Mills et al.).

Demands on built environment policy

The gap between policy intent and effective solutions remains difficult to close. This serves to increase the number of challenges that the built environment sector needs to address successfully. Many of the problems are in the ‘wicked’ or even ‘super wicked’ categories (Levin, Cashore, Bernstein, & Auld 2012; West Churchman 1967). Such problems include:

• Climate change

  The prime example of a well-recognized but still intractable policy issue for almost all sectors and certainly for the built environment (Thompson, Cooper, & Gething, 2015). Almost every paper in this special issue touches on the mitigation aspects of energy efficiency/reducing carbon emissions, even if some do so only tangentially. There is insufficient policy work on adaptation and resilience.

• Demographic change

  Although this is a problem far outside the sector's control, it is expected to provide solutions in the form of adequate housing, infrastructure and workplaces to accommodate rapidly changing populations. The policy issues that both Warwick and Mills et al. discuss are largely driven by such demographic pressures.

• Resource shortage and pollution

  While much of the energy debate is driven by concern over climate change, problems of energy supply, loss of biodiversity, pollution and security are often of more immediate concern.

• Finance and payback

  While there is considerable global capital available for investment in the property industry, it comes with an expectation of high profit margins. In practice, unless this condition is met, there is very little investment available for both social and physical infrastructure. Both Ive et al. and Mills et al. discuss the evidence underlying investment decisions for large-scale national investment programmes.

• International competition

  The globalization of finance and the international mobility of some elements of national economies have led to high levels of competition across the world, particularly between leading cities. Policy-makers have responded with measures including a range of architectural and other environmental policies (Bento, 2012) intended to enhance and improve the quality of their built environments.

Feedback loops

The need for feedback is as true for policy design as it is for the design of individual buildings, but with the potential for a great deal more impact. Poor or inappropriate policies for the built environment can create a negative legacy that lasts generations and is difficult to fix. In contrast, successful policy initiatives can have a multiplier effect across a wide range of individual projects and act as a long-lasting stimulus. Without the ability to learn what worked and why, policy-makers, civil society, and the construction and development industries are destined to repeat the mistakes of the past and will lack the capabilities to help develop and select the right mix of policies in future.

Given the numbers of players in the built environment sector, it would be appropriate to ask whose, if anyone's, responsibility it should it be to collect and provide feedback on policy outcomes – a problem that has similarly bedevilled the discussion on post-occupancy evaluation (POE) for decades (Cooper, 2001). Almost all parties have an interest, but most have failed to engage. Where they have, there has been a tendency to keep their learning and knowledge private and confidential. The requirement is for disinterested, rigorous monitoring, and open and honest evaluation. This may entail the involvement of third-party organizations supported, if not controlled, by the many parties to the production and use of the built environment, using a common language and well-defined and understood metrics.

The art of prediction

The policy gap can be understood as a matter of less-than-perfect prediction, as opposed to (eminently predictable) difficulties with interpretation, implementation and performance. If predictive processes and tools were capable of accurately modelling policy initiatives, then outcomes could be optimized at the development stage and followed through to a satisfactory conclusion. This clearly is not, and never will be, the case. The papers in this special issue vividly illustrate the many obstacles that can disrupt policy delivery. This is not to say that policy prediction cannot be significantly improved, in particular with the intelligent use of feedback on actual policy outcomes.

Feedback is an imperfect predictor of future performance, but unlike decisions on single buildings, policy initiatives produce multiple outcomes from a wide range of projects. When aggregated and analysed the data should be able to produce significant findings and results reliable enough to make sound judgments, even though the ability to access the detail of any particular scheme may become lost in the process.

Prediction cannot rely on simple analysis of what has worked in the past. Times and circumstances change and yesterday's answer is unlikely to be good enough for today's and, even more importantly, tomorrow's problems. Nonetheless, it is possible to build on a record of predictive success to produce probability-based assessments and gradually get it ‘less and less and less wrong’ (Silver, 2012, p. 232).³ Bayes’ theorem (McGrayne, 2011) provides the opportunity to use knowledge of past events to produce a probability rating for a hypothesis or a policy proposition. Over a series of events and with the benefit of feedback prediction can incrementally become more accurate.

This does not change the truth that prediction will always remain an art rather than a science, as acknowledged in the probability-based outputs of a Bayesian calculation. As resources and time run out for acting on some of the wicked and possibly existential problems confronting the built environment, it may become essential to judge better how likely policy interventions are to work in practice.

Improving the art of prediction is not simply about EBPM or evidence-based design. It is also about recognizing limitations and becoming more honest about the chances of achieving objectives. It requires data and feedback from previous policies and intelligently performed pilots and trials to improve policy design before initiatives are fully implemented. It does not expect a linear relationship between action and outcome.

The policy gap is incapable of closure solely by changes to policy design and delivery. This is because policy-makers can only work to the best available predictive performance and at present predictions often leap unhesitatingly and blindly across the void to promise remarkable and unevidenced results. Improving the quality of prediction is key to enabling learning and to developing policy that matches the expectations placed on the built environment to provide working, affordable and deliverable solutions.

Propositions

A more open and transparent policy process involving a wide range of participants has already been discussed (see also Schweber et al.). It is an essential component of more considered policy development and may, in any case, be inevitable in a world under the constant spotlight of both traditional and social media attention. With it needs to go a new culture of policy responsibility by a wider range of participants, which will require:

• clarity over policy objectives, instruments and timescales

• a willingness both to produce and to cope with more honest policy prediction

• increased policy literacy and understanding of policy instruments by professionals and their representative organizations

• a culture and a collectively agreed means of reporting back on policy and project outcomes by all parties from government to companies and individual professionals (see Gupta et al.)

• a regular reporting system on policy effectiveness

• an understanding that policy is a live activity requiring continuous monitoring and course correction as a precondition of implementation

• measures to make policy more effective, including:

  • consciously learning lessons from the past

  • building in resilience capable of accommodating change to both political circumstances and the external environment

  • understanding the probability and extent of success

  • the generation of open and inclusive ideas

  • transparent and accessible policy appraisal

  • policy trialling and testing as part of the design process

  • policy championing throughout the delivery stages

  • collection and evaluation of data on both process and outcomes for feeding back into future policy-making and assessment

It is very easy to oversell expectations of policy outcomes and any move towards a greater realism will undoubtedly deflate many of the high claims and ambitions of those on both the supply and demand sides of built environment production. But, in the face of powerful demands for built environment solutions to some of the enduring issues listed above, realism is a very necessary attribute. Evidence on the impact of previous building programmes needs to be urgently assembled and employed in order to help policy-makers develop both initiatives and instruments that are genuinely capable of delivery. This will require the involvement of researchers, practitioners, building managers and others in the development of a robust body of accessible knowledge and an evidence-enabled predictive capacity. Better and more effective policy is capable of achieving a great deal more than any number of good individual projects, but to truly realize this potential, a move is required to create a more policy literate and committed built environment sector capable of well-informed design at both project and policy levels.

Acknowledgement

The Guest Editors are grateful to the Edge, the multidisciplinary built environment think tank, for the opportunity to discuss many of the themes raised in this special issue, and for providing invaluable input.

Disclosure statement

No potential conflict of interest was reported by the Guest Editors.

Notes

1 See, for example, the story in Building (2015).

2 A reference to a supposed response ‘Events, dear boy, events’, by the British Prime Minister Harold Macmillan (1957–63) to a journalist who asked him what was most likely to blow governments off-course.

3 The chapter heading is based on a line in the poem ‘The Road to Wisdom' by Danish mathematician Piet Hein: ‘to err and err and err again, but less and less and less’.