Abstract
Policy predictions fail for the very many different kinds of case-by-case local factors described in the Building Research & Information (2015) special issue (vol. 43/4) entitled ‘Closing the Policy Gaps: From Formulation to Outcomes'. Work in philosophy of science shows that beyond the case by case, general systematic problems loom that make the gap between theory and practice hard to close. What is needed in response, it is argued here, are ways to cope with the gap and to build an expectation about it into planning predictions, into planning decisions, into the methods of implementing and monitoring, as well as into fallback and failsafe plans. Tracking implementation and outcomes is not only useful for post hoc evaluation but also a powerful tool for getting the intended outcomes in the first place and making the necessary adjustments.
Much of the rhetoric in favour of evidence-based policy and practice seems to subscribe to the idea: if it comes from science, it is going to be right. In the Building Research & Information special issue ‘Closing the Policy Gaps: From Formulation to Outcomes’ (), repeated failures of this idea can be observed, illustrating the gap between theory and practice. Since the special issue concentrates on the specific policy area of the built environment, this might be thought of as a problem that besets this area especially violently. But this kind of gap is the rule, not the exception: it is just what should be expected for social policy. The papers present a variety of concrete ways, different for different cases, in which final outcomes do not fit theoretical expectations. Work in philosophy of science points up a number of reasons why this is almost bound to occur. Predictions fail for the very many different kinds of case-by-case local factors described in the papers collected here; but beyond the case by case there are general systematic problems looming.
Table 1 Authors and titles of articles in the special issue ‘Closing the Policy Gaps: From Formulation to Outcomes’, Building Research & Information (2015), vol. 43(4); guest editors: Simon Foxell and Ian Cooper
It is important to recognize the systematic sources of trouble, otherwise there is an across-the-board likelihood of overbidding the claims that are made. Scientific evidence can without doubt improve the reliability of policy predictions. But too frequently the scientific base is assumed to offer the promise of far more certainty than it can deliver. There is danger here. If an unjustified degree of optimism exists about how effective policies can be, then there is a likelihood of getting policy deliberation wrong in the delicate balance between considerations of effectiveness, on the one hand, and considerations of cost and legitimacy, on the other. The policy community often expects results that cannot be achieved and fails to anticipate side effects that can be very harmful. Both these lead to wasted money and effort and to heartbreak and dashed hopes. The wrong projects get chosen. For the ones that are chosen, there is failure to establish the monitoring and feedback systems that provide early warning that things are going wrong. This would allow for things to be set right or for the project to be abandoned before too much money and effort has been expended. It is unrealistic to expect to close the gap , but it is feasible to narrow it. What is needed are ways to cope with a gap, to build an expectation about it into our predictions, into our decisions, into our methods of implementing and monitoring and into our fallback and failsafe plans.
The across-the-board problems arise from the relation between the demands of exact science and the nature of the blousy world around us that the science is supposed to help manage. In general, though scientific principles can be hugely successful in helping to deal with the world, they do not describe what actually happens. They describe what happens in models and very often the model does not sufficiently resemble the world it is supposed to model. As Janda and Topouzi remark, ‘Idealizations about people and their energy behaviours are familiar, but they are not necessarily accurate or true’ (p. 528). There are several perfectly general reasons for this that need to be borne in mind. First, even if the principles applied in the model are flawless, the outcomes that hold in the model hold in the world only ceteris paribus, i.e. the outcomes hold only so long as the model takes into account all the dominant causes of those outcomes that are at work in the world and none that are not. This is a tall order, especially in areas of social policy where social and political causes interact with technological ones. Researchers and policy-makers usually have a very loose grip on what all these social causes are and how they operate, let alone on how the mix of causes will interact together, as observed throughout this special issue.
The striking experimental and technological successes of some of the best basic science are a witness, rather than a counterexample, to the ceteris paribus nature of these principles. Testing a scientific theory depends upon devising (or finding naturally occurring) settings in which all the factors relevant to the outcome are describable by the theory, as well as knowing how to describe these factors. Otherwise, how would it be possible to know what the theory predicts should happen in order to compare that with what actually does happen? This is important in order to ascertain if the predictions of the theory are borne out. These are generally highly artificial settings, or highly managed ones, so-called ‘closed systems’, that are ‘overbuilt’ or otherwise protected against the intrusion of causes that distort the outcomes or make them unpredictable. Most people have an implicit understanding of this. For example, they know that dropping mobile phones onto a tile floor is detrimental, or not to use galvanized pipes for plumbing near the sea due to the potential for corrosion. This lack of control looms large when it comes to predicting the effects of planning policy. As Trygve Haavelmo, a Nobel Prize winner for his work in founding econometrics, explained to me in conversation about predictability:
Physics has it easy. No-one asks a physicist to predict the course of an avalanche but we economists are regularly expected to predict the course of the economy.
Second, the social and political influences that affect planning outcomes are often at the tail end of a long chain of events stretching backwards in time, or the outcomes are at the end of long chains of complicated social, political and technological causes stretching into the future. As Mills, Phiri, Erskine and Price report about one of the studies they describe:
Perhaps unexpected was the complexity of problems in organizational and collaborative terms. (p. 509)
But if John Stuart Mill is to be believed, this complexity should not be unexpected. It is just the shifting and unpredictable arrangement of the causes that enter these chains – the causes of the causes of the causes – that led Mill to argue that political economy must be a deductive not an inductive science. Even if we have observed some relatively stable relations between one set of factors and another in the past, those relations cannot be relied on to hold in future because the background of all the many causes omitted from consideration is not likely to stay fixed. As Ive, Murray and Marsh note, following Keynes' (1936) quote about the yield of an investment: ‘Our knowledge of the factors which will govern the [ … outcome] some years hence is usually very slight and often negligible' (p. 471).
A third reason has to do with the kinds of concepts that good science demands. These must not be vague or ambiguous, they must be well defined, precise and unequivocal. But for policy planning it must also be possible to use them to make predictions. That means that they must be linked together by principles that allow for inferences to be made from one to another. That too is a tall order. It is usually met by an extended back and forth process of mutual adjustment. Max Weber noted this process in describing the differences between the natural sciences and the study of society. He too, like Haavelmo, thought that physics has it easy. It can keep adjusting its concepts until it finds ones that fit into neat principles. But this approach cannot be applied when studying society. When it comes to society there are fixed topics of enquiry (the things society and its members want to know about which nowadays includes poverty, social exclusion, responses to some nudge policy, etc.) and there is good evidence that these are not governed by the kind of tight reliable principles found in science.
In the face of all the difficulties for predicting the outcomes of policies for the built environment recorded here, Foxell and Cooper (following McGrayneFootnote1) place some hope in Bayes’ theorem to provide ‘the opportunity to use knowledge of past events to produce a probability rating for a hypothesis or a policy proposition' (p. 404). I am far less sanguine about this than they, for just the same general reasons about science, models and real life mentioned above. Bayes’ theorem allows for use of real-life data and can treat real-life outcomes, but it can only do so via the mediation of an abstract model, though familiarity and practice with statistical inference may make the modelling steps in between almost invisible. The model requires a well-defined event space satisfying various formal conditions plus a precise measure over them that satisfies the probability axioms. To be of any practical use, the connections used for prediction among the events in the event space (e.g. which are modelled as probabilistically dependent? which independent? conditional or unconditional independence? conditional on what?) must reflect real relations among the happenings in the world that the events in the event space are meant to represent. So all the same problems arise about how the concepts that describe the events in a model map onto things that are actually happening in the world. This is exacerbated by the fact that the probabilistic relations in the model are even further removed from the world than ordinary scientific models since the probabilistic relations generally are warranted by some theoretical model-dependent assumptions about the relations (e.g. causal relations) among the abstractly defined events in the event space; then those theoretical assumptions need to be fitted on to the (at best rough) regularities found in the real world outside the laboratory and other specially controlled environments.
Thus I agree with the doubts that Janda and Topouzi explicitly expressed about faith that:
building scientists can gather (all) the evidence, understand it and properly communicate it to policy-makers, who will then develop effective policies based on this new understanding.
(p. 516)
There are good reasons coming not only from the experiences of failure recorded in this special issue but also from the very nature of the scientific enterprise itself for their concerns. As they suggest, it is very doubtful that:
accurate information alone will lead to better future planning in the complex system of the built environment. (p. 516)
What can lead to better future planning for the built environment? Better tracking is part of the answer. One of the questions in the original call for papers for this special issue is ‘How might the implementation of a policy or initiative be tracked and evaluated against its intended objectives?’. In their editorial, Foxell and Cooper remark:
Most [papers] also contain overt (or at least decipherable) signposting to how policies and initiatives could be tracked and evaluated against their intended objectives and outcomes. (p. 399)
Tracking is often done primarily for the sake of evaluation, with which it is linked in this remark: knowing that the right stages between policy and outcome occurred provides good evidence that the policy was indeed responsible for producing that outcome. But, as Eileen Munro argues in the field of child protection, tracking is not only useful for post-hoc evaluation, it is a powerful tool for getting the intended outcomes in the first place. Tracking can provide alerts to problems that a policy encounters as events unfold. It provides the opportunity to make corrections before it is too late or to abandon the endeavour early when there seems no way to set it operating as it should.
Second, better evaluation itself can be of help, especially when tracking is involved to uncover not only whether the policy worked but also how it worked. Understanding why is the key to better planning in future, to figuring out what policies will work where and when and what needs to be in place, or be put in place, to make their success more likely. Without this understanding, mere evaluation by itself – learning that a policy worked in this place or in this handful of places – can be very misleading. It can lead to ‘induction by simple enumeration', which is a very poor form of inference indeed. For example: ‘swan 1 is white, swan 2 is white … so all swans are white’ or ‘all the coins in my pocket are copper, so all coins are copper'.
More effective interdisciplinary cooperation in the planning and implementation of policy is a third important tool for getting better outcomes. As already noted, plans fail for a mix of natural and social causes, some naturally occurring, some artificially, even sometimes induced by the process itself in a looping effect. The hard science matters but equally does the knowledge of the social processes, like the political agenda of various players highlighted in Warwick's and in Moncaster and Simmons’ contributions, the structure of regulation and in particular its enforcement mechanisms, as made clear in Cohen and Bordass's paper, the kinds of subjective forces that undermine objectivity described in Simmons’ and in Schweber, Lees and Torriti's contributions and the effects of complex interactions among different agencies and the need for detailed understanding of local situations described in Mills, Phiri, Erskine and Price's article. The list continues. It is long and varied. What will make a difference will not be the same from one context to another, which makes it difficult to offer advice in advance about whose expertise to include, when and how. The important lesson is to be alert to the need to include interdisciplinary expertise and local knowledge from the very start of the policy planning process in almost every case.
Fourth, better outcomes will be achieved overall if policy-makers, strategists and the public have more realistic expectations about what can be achieved and what are the chances of success. The discussion above about the theoretical problems with expecting a high degree of certainty coupled with the huge array of concrete problems displayed in the papers in this special issue argue that, despite good use of good science in policy construction and deliberation, smooth sailing will not be the rule. Instead, it should be expected that problems will arise throughout the entire process, from the start of implementation to the final outcomes. Given the widespread predictable failure of models to be as robust as we would like, it is essential both to hedge our bets and to allow for, indeed to plan for, the need to correct, to redesign or to abort our projects. As several authors emphasize, it is important to keep firmly in mind that (in the words of Foxell and Cooper):
policy-making is a live activity, “a continuous process” [ … ] needing constant adjustments and periodic refocusing to keep it on track. (p. 401)
Disclosure statement
No potential conflict of interest was reported by the author.
Notes
1 McGrayne (Citation2011) makes a case for Bayes' theorem, but its use remains controversial and contested.
Reference
- McGrayne, S. B. (2011). The theory that would not die: How Bayes’ rule cracked the enigma code, hunted down Russian submarines & emerged triumphant from two centuries of controversy. New Haven, CT, and London: Yale University Press.