Is it really about the evidence? argument, persuasion, and the power of ideas in cultural policy

ABSTRACT

In the move towards a supposedly “evidence-based” cultural policy, “evidence” is rarely the main driver of decision-making. If “evidence” is not the actual basis of policymaking, then what is its real role? Why is there so much “bad” or unverifiable evidence of impact in cultural policy documents? The article suggests focusing on recent developments in policy theory for more accurate and sophisticated approaches on the connection between evidence and policymaking, and the role that ideas and values have in shaping policy. A closer engagement with theories of policy formation demonstrates policymaking has a fundamentally discursive character: it is based on ideas, processes of argumentation and persuasion, so can never be an ideologically neutral exercise. The article concludes that cultural policy studies can benefit from a more systematic engagement with policy theory.

KEYWORDS:

• Evidence-based policy
• cultural policy discourse
• ideational policy analysis
• policy rhetoric
• economic impact
• instrumental cultural policy

Introduction

The last 20 years of cultural policy in England have seen the increasing enlistment of the arts and culture to the service of social and economic governmental agendas, and the parallel growth in the expectation that cultural policies – predicated on such pragmatic rationales – should be based on solid evidence of their effectiveness. The capacity of the sector to produce proof of socio-economic impact is seen to lend legitimacy to the explicitly instrumental case for cultural expenditure. Interestingly – and counter-intuitively in an allegedly evidence-based regime – it is widely documented that explicitly instrumental policies are increasingly popular, in Britain and beyond, despite mounting evidence that the underlying data and performance measurements are often flawed, and methodologies for their collection either dubious or under-developed (Crossick & Kaszynska, 2016; Woronkowicz et al., 2019). Indeed, concerns over the quality, reliability and comparability of cultural sector data and other policy-relevant research consistently feature in cultural policy studies (Bilton & Soltero, 2020; Schuster, 1996, 2002; Selwood, 2002, 2010).

In 2009, building on an essay by the American moral philosopher Harry G. Frankfurt (2005), I argued that a lack of concern for “how things really are” is a prominent feature of public discourses around cultural policymaking and justifications for public subsidy of the arts in Britain (Belfiore, 2009). Despite the move towards a supposedly “evidence-based” policy, “evidence” rarely underpins decision-making, especially in policies surrounding the perceived social impacts of the arts. Rather than enhancing accountability and transparency in policy making, the pressures placed on English publicly funded cultural institutions to produce evaluations of their performance, and to subject themselves to the constant attempt to measure their efficiency and impact, has been pivotal in the escalation of very dubious “evidence” (Belfiore, 2009). Recent research focusing on the case study of the NEA, the US arts funding agency, reveals this is not limited to England, but a much broader issue to do with the purpose and use of impact and performance management data (Woronkowicz et al., 2019).

Ten years on, I am struck by the persisting relevance and salience of Harry G. Frankfurt’s On Bullshit (2005): the essay is even more significant and poignant now in a “post-truth” politics (McIntyre, 2018). However, an important question remains about the function that the rhetorical use of statistics, and “bad evidence” fulfil in the policy process. This article addresses this question and offers a fresh articulation of the cultural policy-making process, thus moving the cultural policy studies research agenda forward in fruitful ways. This is achieved by considering the insights that the cultural policy researcher can gain from work produced within the field of public policy theory, which, despite its obvious relevance, has been overlooked in the development of academic cultural policy studies.

The article’s argument is in two parts: the first, comprising of the next two sections, offers a map of prevalent arguments put forward in policy debates to justify financial interventions from the government to support the arts and creative sectors. This shows how policymakers and practitioners rely on notions of evidence-based policymaking, and what they believe to be impressive evidence, to project a sense of rationality and order on the policy process. The second half of the article draws on recent policy scholarship to argue that the sense of rationality and order is an illusion, and that the policymaking process is in fact messy, irreducibly political, and more influenced by ideas than evidence. Economic impact is offered as an example of such influential, institutionalized ideas with a significant power to shape policy discourse. It might appear to offer a way out of the justification problem, but the prevalence of economic impact arguments and “evidence” ultimately results in a misleading, excessively narrow, and potentially damaging discourse on the value of arts and culture, which will not lead to effective cultural or industrial policy. These two sections lead to the conclusion, the article’s main argument, that establishing and maintaining a dialogue between cultural policy studies and ongoing debates, scholarship and insights from policy theory should be a central ambition of a future research agenda on cultural and creative industries policy. As Paul Cairney (2016, p. 119) reasonably puts it, “if you want to inject more science into policymaking, you need to know the science of policymaking” (emphasis in the original). While the main goal of the article is to articulate how the realities of policymaking can inject insights into cultural policy research, a more realistic and accurate picture of the role that evidence plays in processes of policy formulation and change would also benefit policy actors and contribute to better policies.

Economic impact as evidence for policymaking

My analysis focuses on English policies as I am a scholar based in England. The arguments presented herein, however, aspire to a broader relevance. While it is important to acknowledge that data collection for the purposes of supporting policy decisions and interventions in the cultural sector has grown significantly over the past decades, and encompasses a range of spheres – from arts and health, to culture-led urban regeneration, the breadth and depth of patterns of cultural participation, etc. – for reasons of space I focus on attempts to evidence and measure economic impact within discursive formations around public funding and its justification. Economic impact analysis is, as Bille and Schulze (2006, p. 1057) explain, “a well-established and frequently-used methodology in economics, in which the impact of one activity on the rest of the economy is calculated using traditional multiplier models”.

By way of clarification, the examples of “bad evidence” discussed in the next section are not “bad” because they are necessarily false, or claiming an economic impact that does not exist. What is problematic, rather, is the lack of transparency, verifiability, and reproducibility of the impact calculations of which the “evidence” is the result. Indeed, this contradicts both the logic of impact measurement as a key determinant of public investment, as well as the principles of transparency and accountability which are so central to the rhetoric of evidence-based policymaking. Therefore, it is worth spelling out that critiquing the way economic impact is understood, measured, and rhetorically deployed in policy discourse does not equate to suggesting the irrelevance or unimportance of economic value and impact for the arts and creative sectors. Quite the contrary. Precisely because economic impact is important to both arts and creative industries policy and is a legitimate component of debates around funding and policymaking, economic impact calculations need to be carried out properly and rigorously, and they should be reported on and presented in ways that lend themselves to public scrutiny and verification. It is worth noting that the most vocal criticism and expressions of frustration at the miscalculations and the abuses of economic impacts in cultural policy has often come from cultural economists frustrated that their criticisms have gone unheeded by the policy community (Bille et al., 2016; Bille & Schulze, 2006; Klamer, 2008; Seaman, 2003; Van Puffelen, 1996). For example, prominent Danish economist Trine Bille and her co-authors (2016, p. 239), reconstructing the history of the popularity of the economic impact study in Europe at the end of the 1980s, observed that “the good news [of impressive returns on investment] was extensively used in political argumentation. The problem was that the analyses often were made on an incorrect basis, and the economic effects were often exaggerated”. Dutch economist Frank Van Puffelen, in 1996, went as far as advocating for the outright cessation of economic impact studies due to the abuses that were made of them in policy discourse. Similarly, it has been mostly economists, not only naïve arts “luvvies” and “culturalists” (Klamer, 2008), who have lamented the narrowness of the policy focus on economic value. Bille et al. (2016, p. 240) remind us that “a focus on the potential economic impact of culture can create a misguided cultural policy”, and Michael Rushton (2020) has claimed in no uncertain terms: “Economists don’t like ‘Economic Impact’ studies – they know that the conception of them is wrong, and they lead to bad reasoning” (emphasis in the original).

The rhetorical power of economic impact

The English experience of the past 15 years indicates that claims of social and economic impact and their evaluation remain central to strategies of legitimation and justification within the cultural sector (see for instance: Bazalgette, 2017; CEBR, 2019, 2020; CIF, 2015; DCMS, 2016). As Gray and Wingfield (2011) suggest, the reason for this lies in DCMS’ lowly political status and significance, which mean DCMS finds itself effectively a hostage to instrumental concerns that it has limited control over.

In particular, recent rhetoric eulogizing the extrinsic benefits of the arts and culture has had a distinctive economic flavour. The nature of the evidence takes a number of forms; for example: the contribution of the arts, culture and “creativity” to increased trade and competitiveness for the UK, as suggested in the British Council report, Culture Means Business (2013) and the 2016 Culture White Paper (DCMS, 2016); the value of British heritage for tourism (HLF, 2016); the “macroeconomic impacts of the arts and culture industry” (CEBR, 2013); the contribution of the arts and culture to the national economy, both directly and indirectly, through spillover effects felt in the wider economy (CEBR, 2019, 2020); the ways in which public funding of the cultural sector directly results in economic growth (CIF, 2015) and impacts the “wider creative economy” (Metro-Dynamics, 2020, commissioned by ACE), and even the contribution of Arts Council funded buildings to the liveliness of England’s high streets (ACE, 2020). Whatever specific form it might take, the purported contribution that the creative sectors make to the British economy has been a staple ingredient in their ongoing lobbying of government for increased investment and beneficial policy interventions, such as more favourable tax regimes.

The sustained campaign, by DCMS and other cultural and creative sector lobbying organizations (such as the Creative Industries Federation), to have the creative economy included in discussions and plans for the Industrial Strategy, which was one of Theresa May’s key policy commitments upon her instalment as Prime Minister in 2016, is a prime example of the flourishing and intensifying of the creative economy rhetoric. As part of this concerted lobbying activity, the DCMS commissioned Sir Peter Bazalgette – Chair of broadcaster ITV and, between 2013 and 2017, Chair of Arts Council England – to head an Independent Review to set out a series of recommendations to government “for the continued growth of UK’s Creative Industries” (DCMS, 2017). The report was published in September 2017, and its goal was to set out the key areas in which the government and the sector should work together for the purpose of developing “a Sector Deal for the Creative Industries” (Bazalgette, 2017, p. 5).

The report makes strong claims for the contribution of the creative economy to economic recovery and to employment, and even bolder projections of its possible future contributions, should the government offer investment and support to sustain current levels of growth. There is nothing new in the argument presented in the Bazalgette Review, but what makes it significant is that – unlike its numerous predecessors – this review resulted in substantial investment in creative industries R&D via the newly set up Industrial Strategy Challenge Fund. After the publication of the Bazalgette Review, DCMS (2017) launched the £80 million Creative Industries Cluster Programme, administered by the Arts and Humanities Research Council (AHRC), the main body for the distribution of public funding for arts and humanities research.¹ This represents a direct investment of £39 million – in addition to HE and creative sectors match funding. Rhetorically speaking, the report was successful in making a persuasive argument in support of increased investment in a particular range of activities that the report presented as key to the success of the cultural and creative sector.

Making its case for enhanced government investment, the Bazelgette review (Bazalgette, 2017, p. 11) gives prominence to a set of economic data originally presented by DCMS as their Sector Economic Estimates for 2017:

The Creative Industries are a success story, playing a key role in the UK’s economic recovery. They contributed £87.4bn in GVA in 2015, 5.3% of the UK economy (comparable to the Construction or Information sectors) and between 2010 and 2015 grew by 34% – faster than any other sector. They have also outperformed other sectors in terms of employment growth: between 2011 and 2016, employment in the sector increased by 25.4% (circa 400,000 jobs) compared to 7.6% average across the wider UK. The sector is also a net exporter of services (£11.3bn surplus in 2015).

The report states: “[b]ased on current trends, the Creative Industries could deliver close to £130bn GVA by 2025 and approximately one million new jobs could be created by 2030” (Bazalgette, 2017, p. 11). On what basis Bazalgette and his team reached the conclusion that the creative sector’s GVA contribution might increase by about 40% over 10 years is unclear, for all that the report says about these calculations is merely a footnote stating that these are “linear forecasts” based on DCMS’ Economic Estimates for 2016, and job growth data from the Office of National Statistics’ labour data for 2016'. No indication is given of the origin of this estimate and on the methodology used for this calculation. Online research reveals no other publicly available source for this assessment. In other words, the reliability of this data needs to be accepted on trust, its only claim to validity being its inclusion in a prominent and official report: there are no means to check the estimation, or to replicate the calculation. Nonetheless, upon publication of the Bazelgette Review, the then Culture Secretary Karen Bradley commented that “[t]he UK’s Creative Industries are an economic powerhouse and the government is committed to removing the barriers to its growth” (in DCMS, 2017).

Declared economic benefits and impacts of the arts and creative industries – especially when they are produced by public agencies and obviously aimed at supporting and justifying policy – are often contested, though with no discernible effect on their popularity (Belfiore, 2009; Bille et al., 2016; Crossick & Kaszynska, 2016). In the context of an official commitment to evidence-based decision making, it is questionable that a report whose primary aim is to offer evidence for the development of new policy initiatives entailing the investment of significant public resources should base a key aspect of its case on data and evidence that the public are expected to take at face value. The accuracy of the calculations of such economic benefits is not the target of critique here; nor is the aim of the discussion to suggest the claimed benefits are necessarily false or untrue. The arts are an industry and have economic impact on the rest of the economy as Bille and Schulze (2006, p. 1064) state. Nonetheless, economic impact studies “do not, however, indicate that the multiplier effect is greater in the arts sector than in other sectors of the economy”. Therefore, without a meaningful engagement with the concept of opportunity costs (public investment in the creative sector would entail no funding for another sector as funds are limited), these calculations of impacts and benefits do not in themselves offer a rationale for investment or increased funding, even though the reports they are presented in claim they do.

The Bazalgette report, thus, seems a good example of a rhetorical approach to lobbying and to “making the case” for funding. The focus here is on producing a compelling picture of a thriving sector that can – with careful investment – enhance national productivity. Since the investment did indeed follow the publication of the report, we can assume that the argument was deemed to be persuasive by the relevant key decision makers within government. Yet, simultaneously, challenging the argument in the report is arduous because, whilst the source of the original DCMS data for the economic projections of future economic impact are offered, no indication was given of how they were deployed in the forecasting exercises: the argument then cannot be tested, verified and/or disputed, and no genuine public debate or scrutiny can take place.

The Bazelgette Review is not exceptional. Unverifiable “evidence” in support of claims of economic impact abound in English cultural policy discourse. The allure of the rhetorically impressive is strong. A good example is represented by an intriguing recent development which suggests independent efforts to evaluate the economic impact of different components of the cultural and creative sectors, in different geographical contexts and at different points in time, all seemed to identify the same finding: that for every pound sterling of public investment, the taxpayer could expect a “return” of £10. The appeal of this “evidence” is clear: who could argue against the uncontestable wisdom of investing in something that produces a tenfold return of investment? Arguments in defense of public subsidy of the arts and culture cannot get any more compelling. One of the earliest examples of this argumentative strategy is a 2013 report by the Local Government Association (LGA) entitled Driving Growth Through Local Government Investment in the Arts (2013). On page 16, the report focuses on the example of York Museums Trust, maintaining it “represents a ‘return on investment’ of around £10 of impact for every £1 invested by City of York Council”. No indication is given of the source for this estimate, who produced it, based on what data or what methodology. No Council report or similar is available on public record that refers to, or gives any information on, the study the data originates from. No explanation is offered for what “return of investment” might mean in this context: does the £10 represent gross impact? Net impact²? The expectation is that the reader will accept the information offered on trust and be wowed by the “return” on investment, not that the evidence should be scrutinized for accuracy.

In January 2019 Culture Liverpool, the City Council department responsible for events and the city’s cultural infrastructure, produced a report that was presented to the City Council’s Culture and Tourism Select Committee. Its aim was to “update the Culture and Tourism Select Committee on a review of the work and achievements of Culture Liverpool in 2018”, particularly in relation to the activities that were programmed and delivered on the tenth anniversary of Liverpool’s stint as European Capital of Culture in 2008. The report offers a range of impressive data on the numerous achievements and impacts – cultural, social, and economic – of the activities and organizations supported by Culture Liverpool:

It is estimated that for every £1 of revenue support that LCC [Liverpool City Council] contributes to the cultural sector at least another £10 is brought into the city and that at least £45 million of economic impact is generated. (Culture Liverpool, 2019, p. 22; emphasis added)

As previously, no information is offered on the source of this calculation or the methodology, nor can we tell whether the £10 “brought into the city” represent gross or net impact, yet this was unquestioningly reported and amplified through the local press, and a number of city-wide marketing and business websites.³ Presumably, members of the Select Committee would have been able to request further information, but this is not available on public record. The public might be able to find out more through Freedom of Information requests, unless the information was deemed financially sensitive, but that the onus to investigate should fall on the reader is not compatible with a commitment to transparency and accountability which is, supposedly, the foundation behind the push for evidence-based policymaking.

How, then, are we to judge data presented as valid evidence for policy making when that “evidence” eschews validation and scrutiny? In the rest of this article, I draw on recent developments in theories of policymaking that, I argue, allow us to recast this question within a different understanding of the relationship between evidence and policy: one that focuses less on research as a source of “evidence” and more on the role of ideas, values and beliefs in shaping the processes of policy formation and change.

The problem with the “rationality project” in public policy

Understanding how cultural policy is made, why there is a gap between policy discourse and policy reality, and why unrealistic claims, dubious statistics and unverifiable data should have a prominent role in public debates over the social, health and economic impacts of the arts, public spending on culture, and cultural participation (to mention just the most prominent themes), requires we reject the linear understanding of policy formation that dominates contemporary cultural policy rhetoric. This linear model is equally enshrined in institutional measures to incentivise, assess and reward the impact of research beyond the academy (see, for instance, the Research Excellence Framework, REF, in the UK) (Smith et al., 2020). Instead, the article embraces what Kathryn Smith (2013) calls the “ideational approach” to understanding the effect of research on policy, and argues in favour of a discursive understanding of the policy process based on argumentation and persuasion, and the centrality of “institutionalised ideas”.

The field of public policy and policy analysis was traditionally predicated on the equation of rational analysis with good policy. Deborah Stone (2002, p. 7) in her influential book Policy Paradox refers to it as “the rationality project”, and suggests its origin lies in the “mission of rescuing public policy from the irrationalities and indignities of politics, hoping to make policy instead with rational, analytical and scientific methods”. Thus, exquisitely political and highly controversial issues have been reformulated in purely technical terms. Tania Murray Li (2007, p. 7) refers to this as “rendering technical”, whereby “[q]uestions that are rendered technical are simultaneously rendered nonpolitical”.

The notion that identifying “what works”, rather than political ideologies, should be the driver of policy is a prime example of this conscious attempt to turn policy and decision making into an ideology-free process. Arguably, “instrumental cultural policy” is part of this broader political and administrative development: once matters of cultural value are reframed as questions of impact, and political discussions as to how public subsidy for the arts and culture might be justified are transformed in debates over the best method to provide “research evidence” to support policy claims, the “case for the arts” becomes a technical issue. Thus, it is transformed into a purely methodological problem: that of the quest for the ultimate impact evaluation toolkit, applicable to all art forms, all audiences and in all geographical and social contexts (Belfiore, 2004; Belfiore & Bennett, 2010). Yet, as policy studies show, this is not how things work: “the idea of [evidence-based policy making] sums up a large collection of longstanding myths, hopes and political slogans about the use of information in central government”; it is, “a vague, aspirational term, rather than a good description of the policy process” (Cairney, 2016, p. i). Smith (2013) argues that the analysis of policy statements, documents and decisions, reveals decision makers do not use evidence to achieve goals they set for themselves, even when such evidence exists and is available. This is due to several reasons: policymakers lack knowledge to assess the reliability and full implications of the evidence; or the available evidence might be limited, or problematic; they might be facing political limitations to the range of options which are ideologically feasible at the time, and on occasion might be aiming for vague and unclear policy goals (Cairney, 2016). Cairney (2016, p. 5) uses the term “bounded rationality” to refer to this. This is a different reality from the ideal (and idealized) scenario of evidence-based policymaking, and it is also one that is much less appealing (see Smith, 2013, p. 71). Therefore, we should “conceive of the interplay between research and policy as involving a continual exchange and translation of ideas”, rather than a linear process of evidence injected into the policy arena with a view of directly influencing decisions (Smith, 2013, pp. 74–75).

Indeed, the problem with the rationalist approach – and one of the complications of the reliance (at least at the level of policy rhetoric) on evidence as a primary basis for policy – is that de facto it obscures and pulls outside of the scope of mainstream policy analysis central aspects of the policy making process that relate to people, power and politics (Parsons, 2002). As a result, the rational model of policy making promotes an artificially depoliticized view of how policies come to be (Weiss, 1977).

Moreover, policy-relevant research is itself highly political, and often motivated by opportunistic considerations (Radin, 1995). As Scullion and Garcia (2005, p. 120) noted, “what the cultural sector really wants from research is the killer evidence that will release dizzying amounts of money into the sector. Its expectation of research can be unrealistic”. This does not account for the fact that significant policy change takes place over long periods of time, slowly and gradually, making it highly unlikely that any single injection of new evidence – “killer” or otherwise – could ever result in a sudden flood of new money into the cultural sector, even at times of fiscal possibility (Cairney, 2016). The problem, as Cairney (2016, p. 54) explains, is that policy analyses are produced without references to theories of the policy process, further compounded by “a general lack of awareness of policy scholarship” among practitioners, resulting in them drawing on “older concepts such as the policy cycle and stages approach […] largely rejected by policy scholars”.

This does not mean that research is irrelevant to the policy process, or that policy-sensitive research is doomed to fail, however, how research evidence feeds into the development of policies for the cultural and creative sectors remains under-researched and not fully understood. The theoretical reframing proposed here represents one step towards addressing this gap in knowledge and understanding. This entails accepting that the connection between evidence, the research that produces it, and policy formation and change is diffuse and harder to capture than the “linear” rhetoric of impact suggests (Cairney, 2016; Smith, 2013).

Towards a rhetorical model: cultural policymaking, argumentation, and the power of ideas

To explain “a rhetorical model of cultural policy”, I draw on recent policy studies literature on the role that ideas and evidence really play in policy formation (Cairney, 2012). At the heart of this body of work lies the fundamental contention that “[p]olicy analysis and planning are practical processes of argumentation” (Fischer & Forester, 1993, p. 2), and that language plays a performative role in policymaking. As Giandomenico Majone (1989, p. 8) argues, “facts and values are so intertwined in policy-making that factual arguments unaided by persuasion seldom play a significant role in public debate”. As Cairney and Oliver (2017, p. 2) observe, “[p]olicy studies suggest that actors are influential when they ‘frame’ their evidence in simple, manipulative and/or emotional terms to generate policymaker attention”. So, we must accept that the label “evidence-based policy” is twofold: on the one hand it is “an ideal state to which we should aspire (an empirical proposition)”, and on the other “an ideal-type to compare with what is possible in the real world (a theoretical proposition)” (Cairney, 2020, p. 4; emphasis in the original). What is it not (and was never meant to be, in the mind of the policy theorists that have defined the concept) is a descriptive label for the reality of policymaking (Cairney, 2016).

This section builds on recent policy studies' work on an “ideational” understanding of policymaking, and how policy change happens. I argue that dialogue with policy theory helps us understand the function that “bad evidence” has in cultural policymaking. As noted previously (Belfiore, 2009, 2012), the arguments and the “evidence” deployed in policy debates are adopted for their perceived rhetorical efficacy. What remains unclear is why certain ideas are more rhetorically powerful and more persuasive than others. Let us take instrumentalism as a focus – and the kind of economic instrumentalism deployed in the policy documents previously discussed: what makes the idea of economic impact (with the attendant attempts to measure it, state it, and over-state it in order to build “the case” for the arts around it) so rhetorically appealing, resilient and, consequently, dominant in policy discourse?

The remaining section, thus, focuses on the idea of impact (and particularly, economic impact) and its role in shaping and directing policymaking in the cultural sector, over and above any evidence that said impact might exist (or not). As Smith (2013, p. 73) notes, “literature focusing on agenda setting emphasises the importance of understanding how particular policy problems are constructed as this not only informs the potential solutions but determines who is involved in the policy process”. Ideas, then, work as policy frames that can have an important impact on policy process and outcomes (Smith 2013, p. 73). This framing is deployed strategically in public discussions, in advocacy reports and lobbying tactics to push for desirable outcomes, relying on the ability of policy actors to embed their preferred ideas and issues at the centre of decisionmakers’ attention. As Weiss (1989, p. 118) suggests, these processes are deployed as “weapons of advocacy and consensus” holding significant sway over the policy process. Not all ideas have the potential to realistically develop into fully fledged and viable policy frames though: in a world of competing ideas and attention seeking, the preoccupations of policymakers are impacted by the institutional structures “in which policy makers live” (Weiss, 1989, p. 110).

Then, consider instrumentalism as one such “policy frame”, which rose to prominence within English cultural policy discourse in the 1980s and remains dominant today. This is despite criticisms and attempts by many to develop alternative visions and policy frames leading to a different approach to justify arts funding and policy objectives, such as access and participation – the debate around “cultural value” being a notable example (Belfiore, 2012, 2020; Crossick & Kaszynska, 2016; Scott et al., 2018). The idea of impact (both social and economic) is fundamental to the articulation of the instrumental policy frame. Cairney (2016, p. 33) maintains that “[f]raming is one part evidence and one part emotional appeal, and our focus is on the use, rather than the properties, of evidence” (emphasis in the original). Hence, if we want policymakers to pay attention to research evidence, it is necessary for it to be framed to fit in with the policymaker’s pre-existing priorities and beliefs. It is precisely this which accounts for the prominence of the economic framing of the question of impact: “in countries such as the UK, a story about a policy solution is generally more powerful if framed in terms of its demonstrable value for money” (Cairney, 2016, p. 61).

Kathryn Smith (2013) reveals how the availability of evidence can be useful for policymakers to push and sell ideas to other policy actors, the media, professionals in the sector, etc., and help new ideas emerge. What evidence very rarely does is effect direct, immediate, and unmediated policy change. While “ideas themselves have a degree of agency” (Smith, 2013, p. 150), they take a long time to exert influence over policy change. Nevertheless, the adoption of this focus in policy analysis leads to the conclusion that it “makes more sense to study the policy influence of ideas than evidence” – the corollary to this being that “researchers are often viewed by policymakers as one of several sources of ideas, not necessarily of evidence” (Smith, 2013, p. 108). When looking at the ideas that drive policymaking and how they are framed in policy discourse, however, we need to keep context in mind, for as Weiss (1989, p. 110) argues, at any one time, the “handful of currently prominent themes is drawn from the inventory of themes and symbols acceptable in the mainstream political culture”. Based on this premise, I will now offer an interpretation of the rise to prominence of the idea of economic impact as a policy rationale in England (and, to a degree, Britain) with the attendant issue of the pressures to demonstrate such impact, and the vexed issue of “evidence”.

Arguments of an instrumental nature (both in social and economic terms) were present in state interventions on arts and culture, in England, even before a national cultural policy infrastructure was properly developed (Minihan, 1977). In the 1980s, when Western economies were still reeling from the fallout from the 1970s oil crisis, the idea of economic impact firmly embedded itself in justificatory policy rhetoric. This is evident in the first national strategy document published by Arts Council Britain in 1984, entitled The Glory of the Garden, following the Council’s first real term reduction in funding in 1982. In the preface, the then chairman, Sir William Rees Mogg admits:

On the whole I prefer to argue for funding the Arts Council on the contribution of the arts to British civilization. But it is obvious that the attraction of London as an artistic metropolis is essential to Great Britain’s appeal to tourists. The loss of London’s arts would cost the balance of trade as much as the loss of a major industry. The economic argument is quite clear cut and it applies most strongly to London, though it also applies to the regional arts. (In Wilson, 2019, p. 187)

This point is reinforced in John Myerscough’s report The Economic Importance of the Arts in Britain (1988, p. 2), whose introduction acknowledges the context of decreasing central government funding as key to the report’s advocacy intentions:

Arguments based on [the arts’] intrinsic merits and educational value were losing their potency and freshness, and the economic dimension seemed to provide fresh justification for public spending on the arts.

Unsurprisingly, the report originated as an initiative by the Calouste Gulbenkian Foundation, later joined by the Office of Arts and Libraries,⁴ the Museums and Galleries Commission,⁵ Arts Council of Great Britain, and the Craft Council (Myerscough, 1988). That is, all of the principal national cultural policymakers and funding bodies were involved in the production of one of a series of research reports aimed at recasting public subsidy of the arts and culture as a viable “investment”, noting also “[t]he rising tide of European policy interest in the economic dimension of the arts” (p. 2). It is telling that the main argument concerning arts subsidy was that “the arts are a cost-effective means of cutting the unemployment count” (p. 7), an argument which, indeed, is echoed in the austerity-era Bazalgette Review, as shown earlier.

The prominence of the idea of economic (and now also social) impact remained unchanged even in the more prosperous decades that followed. As Robert Hewison (2014, p. 24), a committed chronicler of cultural policy under New Labour, notes: “Creative Britain needed a creative economy in order to ensure the continuous innovation on which growth depended”. Consequently, publicly subsidized arts and culture “would change from being the needy supplicant for costly subsidy to the grateful recipient of positive investment, and would drive the creative economy” (Hewison, 2014, p. 25). This is supported by the statement of intent expressed by New Labour’s most influential Culture Secretary, Chris Smith (1998), in his volume Creative Britain, where he lists the party’s cultural policy priorities as: access, excellence, education and economic value.

This brief historical excursus shows how instrumentality has become a powerful cultural policy frame in England, and one which has made it difficult for policy actors and researchers to push for less narrowly instrumental policies. The consensus over the scarcity, poor quality and methodological flaws of a significant proportion of the “evidence” of impact has done little to dislodge the idea of impact from its central place in policy discourse (Crossick & Kaszynska, 2016). Similarly, that forty years-worth of impact “evidence” gathered by various policy and cultural sector actors should not have yet warranted success in the quest for a convincing “case for the arts” seems not to have affected confidence in this approach, nor stemmed the flow of policy reports claiming to be “evidencing” economic impact, and “demonstrating” the economic value of the arts. If the idea of impact (be it economic or social) had “worked” in making a case, we would not be seeing a continued influx into policy discourse of new impact evaluations making bolder claims than those that preceded it, yet blissfully mindless of the careful and detailed criticisms made of methodologies and the improper uses of impact evaluation studies in policy argumentation. If impact measurement held the keys to funding justification, considering its long history, the “case” should have been made by now. Yet, the job of demonstrating the benefits of investment in the arts and creative industries is never done: ACE’s website has an entire section entitled “Make the case for arts and culture” giving its clients instructions on how to measure impact, accompanied by the monitor: “it’s vital that we keep demonstrating why publicly-funded art and culture is important and what benefits it can bring to individuals, communities, and society as a whole”.⁶ Successive Culture Secretaries have all appealed to the sector to help them gather evidence of impact as a way to unlock funding. The appeal of economic (and social) impact studies endures, and their number has multiplied since their surge in popularity in the 1980s, but none (nor indeed all of them cumulatively) seem to have “settled” the question of whether the arts and creative industries are worthy sectors of public investments. Furthermore, the blind faith many in the arts and cultural sector place in evidence of positive economic impacts ignores the fact that, in the public sector, funding decisions need to be cyclically made and re-made, and spending priorities re-articulated (for instance, as a result spending reviews), so budgets will always have to be allocated and require negotiations between competing demands for resources, and financial metrics – no matter how problematic – are an entrenched part of this process.

So why the enduring popularity of economic impact rhetoric?. Kathryn Smith (2013, p. 71) suggests that “[a]n analytical framework prioritising the role of ideas usefully replaces a focus on actual material interests with a focus on exploring what actors believe to be in their interests (and why)”. Let us rely on such a framework to make sense of the resilience of the idea of impact in cultural policy, even in the face of insufficient evidentiary support.

The discussion presented thus far points to the conclusion that the idea of economic impact in cultural policy aligns with Smith’s (2013, p. 113) definition of institutionalized ideas. In line with the analytical framework of “institutionalism”, Smith (ibid.) sees ideas “as both constitutive of institutions and as entities that are shaped by institutions”, noting that, once they have been institutionalized, ideas take on powerful self-perpetuating qualities. The example of the rhetorical dominance of economic value and impact in cultural policy discourse, thus, shows us the interconnection of two powerful institutionalized ideas: firstly, that economic growth is – and should remain – the overarching policy goal for government’s policy and investment (whereby it makes sense that the arts and cultural sector should be expected to play their part in reaching it), and secondly, that the arts and culture should try to make a convincing case for their contribution to this as a means to legitimise their demands for (and their receipt of) public funds.

If economic value and its measurement have become institutionalized ideas within the institutions of cultural policy making in England, what are the mechanisms through which these ideas operate as frames that resist challenge? The answer lies in the principles of value for money, cost–benefit analyses and other economic valuation methods which are expected to drive public spending (O’Brien, 2010). Policy discourse (not limited to the cultural sector) is dominated by “econocrats”, who operate based on “the belief that there exist fundamental economic tests or yardsticks according to which policy decisions can and should be made” (Self, 1975, p. 5). Whilst the economic perspective is legitimate, the predominance of the economic frame in shaping the debate around cultural value is detrimental to the quality of public debates around the role of the subsidized arts and culture in society. There is no doubt that the sector itself finds it hard to talk about the value of what it does without the recourse to economic arguments (Belfiore, 2012; Crossick & Kaszynska, 2016). The econonocratic approach to value, and the resulting understanding of the inevitable trade-offs that decision-making in public policy entails in purely monetary terms, has resulted in an impoverished debate. It is precisely this “modern form of ‘economic imperialism’ in the field of the intellect” (Rothbard, 1989, 45) that is at the root of the proliferation of “bad evidence” of economic impact. It is this econocratic dominance that we need resist if we are to develop, collectively and collaboratively, a refreshed and re-energised critical, yet policy-relevant, research agenda around cultural policy rationales. As Nick Couldry (2010, p. 5) explains, “when we now try to think beyond the horizon of neoliberalism, it is at the end of an extended history of neoliberalism’s normalization, the embedding of neoliberalism as rationality in everyday social organization and imagination”. It is this powerful and all-embracing frame that led to the institutionalization of economic impact as a dominating idea in cultural policy, and “evidence” alone will not be sufficient to challenge its centrality to cultural policymaking.

Rethinking “evidence”

I have often put the word “evidence” between inverted commas because, as an earlier section of the article shows, the shift to a rhetorical model of policy entails a radically different notion of “evidence” from the generally received one in mainstream cultural policy discourse. As Majone (1989, p. 10) explains, “[e]vidence is not synonymous with data or information. It is information selected from the available stock and introduced at a specific point in the argument in order to persuade a particular audience of the truth or falsity of a statement”; consequently, “criteria for assessing evidence are different from those used for assessing facts”. Here, “evidence” is indeed whatever argument can convince a reasonable audience and produce consensus over a policy measure (Greenhalgh & Russell, 2006, p. 39). Cairney (2016, p. 22) warns us that we should not even expect agreement, among different actors, around what evidence is: while academics tend to equate evidence with “research”, policymakers and civil servants have a more expansive understanding that equates it to data (including raw data), analysis, gray literature, but also “advice from experts, lessons from other governments, public opinion […] and, in some cases, anecdotal evidence of success”.

An important difference remains, though, between good and bad evidence, between rigorously and sloppily collected data, verifiable and unverifiable calculations, and solid or intellectually dishonest analyses. But, in both cases, the evidence and its analysis fulfil a rhetorical function as a tool of persuasion, and distinctions between “good” and “bad” evidence might not always straightforward. Majone (Ibid., 39) is explicit in stating that “when the analyst uses persuasion, he is always acting, at least in part, as advocate rather than disinterested adviser”. It is now widely accepted that the policy analyst is far from being a detached and objective observer of what happens in the policy field (Radin, 1995, 2006). Yet, as documented by Smith (2013), for many academic researchers the advocacy and politically engaged nature of attempting to produce research “evidence” to influence policy can be especially tricky, or even incompatible with the autonomy that is deemed to be the foundation of robust scholarship (Smith, 2013). When commenting on the policymakers interviewed for their research, Haynes et al. (2011, p. 591) note how “most of them would be sceptical about the extent to which researchers represented disinterested technical advisers rather than political accomplices, particularly in policy areas with high levels of contestation and high stakes” (emphasis added).

Conclusion: ideas, persuasion and cultural policy

This article started by exploring the significant presence of unverified (and unverifiable) and disproportionate claims for the economic impacts of the arts, and the questionable “evidence” on which they are often based in public discourses on arts funding and policy in England. I argued that we must interrogate the very notion of “evidence-based policy” and to explore the real function of data, information, and evidence in the policy process, and then demonstrated the benefits of engagement with contemporary developments in policy theory. Drawing on Smith’s (2013) work on ideational policy analysis, I proposed an argumentation approach to the policy process, which highlights the rhetorical aspects of cultural policy, and the power of ideas and frames as a potentially fruitful interpretative model.

Viewing policies as rooted in rhetoric and argumentation makes for a broader (and more realistic) notion of what constitutes “evidence” in the policy process, and what roles it actually plays in fostering policy change. By highlighting the performative role of language, the rhetorical model draws attention to the socially constructed and discursive nature of policy evidence, and the fact that “the world of policy making is not one of transferable and enduring scientific truths, nor is it exclusively (or even predominantly) concerned with ‘what works’” (Greenhalgh & Russell, 2006, p. 35). This notion of evidence as an element of persuasive argumentation offers the discrete advantage of demanding the serious consideration of the central role of ideas, beliefs and values in policymaking – a crucial aspect which the official discourse of evidence-based policy has tended to obscure. Bringing this “ideas perspective” into policy sensitive research might result in the improvement of the evidence being produced.

Embracing the argumentation thesis has important implications for policy research because policy analysis is as much a discursive and rhetoric-based practice as policy making itself. If – as Greenhalgh and Russell (2006, p. 37) propose – policy is to be seen as a social drama – then the policy researchers (whether based in a government department, a think tank or academia) are its actors, just as politicians, arts administrators and policymakers. As actors, we can envisage for policy researchers a distinctive role in the policy process: they all have a part to play in this drama; they all have a voice and a perspective to contribute to debates around policy. Yet, as actors, researchers are also implicated in the politics of the policy process: claims for objectivity and neutrality need to be questioned.⁷

An understanding of policy rooted in argumentation facilitates the reformulation of the highly problematic “research vs. advocacy” dilemma (Belfiore, 2016) in a way that can reflect the difference between the two activities whilst avoiding a stark and antagonistic contraposition, which would not necessarily offer an accurate picture of the actual practice of cultural policy research. Acknowledging that – in the reality of the policy process – there are situations or actors for which the difference between the two is either less marked or less important than for others (such as academics – who may base their professional credibility on precisely such distinctions – see Smith, 2013 and Smith et al., 2020) would open possibilities for a more accurate study of the relationship between research, advocacy and policymaking. A rhetorical approach allows us to envisage the goal of a cultural policy research field that aims at what I would call a “balance of arguments”. With this term, I refer to a polyphonic debate around policy, where many different perspectives and methodological approaches are represented. What I am proposing here is an adaptation of the notion of “multiple advocacy” developed by Alexander L. George (1972, p. 751), which refers to “a process of debate and persuasion designed to expose the policymaker systematically to competing arguments made by the advocates themselves” (Majone, 1989, p. 40).

If we see all actors in the policy drama as advocates of a certain perspective which reflects the nature of their role (e.g. the local politician, the civil servant, the policy advisor, the academic researcher, the political pundit, the think tank analyst, the artist, the general public, etc), we can begin to imagine a pluralistic model of policy debate. The pluralism in question would not be limited to the participants involved in the process of policy argumentation and persuasion. The “balance of arguments” would accommodate a diversity of disciplinary and methodological approaches by embracing all the disciplines and fields of research related to the study and understanding of cultural policy. This allows for disciplines, such as the arts and humanities, which have been sidelined within current policy debates by the predominance of statistics and economics (and their preferred quantitative methods), the chance of an active involvement in policy-sensitive research. This requires that cultural policy studies engages systematically and meaningfully with contemporary developments in policy studies and theory – something which might seem obvious, yet has not become embedded in the field.

As Carol Weiss’ states, policymaking is a political process (Weiss, 1977, p. 533): evidence from research feeds into processes of policy development and change, and will only be a part of the policymaking story. A heightened awareness of the irreducibly political nature and power of ideas in policy making – irrespective of the claims made in policy reports – would provide an important insight for cultural policy studies.

Acknowledgements

I would like to acknowledge the constructive feedback of the anonymous reviewers, and Dr Dafina Paca for her assistance with copyediting.

Notes

1 Although administered by the AHRC, all the documentation relating to the Cluster schemes made it clear that, in order to be eligible, projects needed to have an industry innovation focus, and that proposals with a predominant research component would not be funded.

2 Net impact refers to the additional economic impact attributable to the York Museums Trust’s expenditure, and activity over and above the spending that would have happened in the area even without the Trust’s activities. Gross impact refers to the overall economic impact without any adjustment.

3 E.g.: https://lbndaily.co.uk/liverpools-year-culture-generates-108m-city-economy/; https://www.marketingliverpool.co.uk/2019/01/22/culture-delivers-massive-boost-for-city/; http://regeneratingliverpool.com/news/culture-delivers-massive-boost-for-city/; https://www.cultureliverpool.co.uk/news/culture-delivers-massive-boost-for-city/ (all accessed on 6/03/2020).

4 The Office of Arts and Libraries was established in 1979 when responsibility for arts and libraries was transferred to the Chancellor of the Duchy of Lancaster (see https://discovery.nationalarchives.gov.uk/details/r/C814).

5 The Commission ceased to exist in April 2000. Many of its functions were passed to the Museums, Libraries and Archives Council and, eventually, to Arts Council England (see https://discovery.nationalarchives.gov.uk/details/r/C98).

6 https://www.artscouncil.org.uk/toolkits/make-case-art-and-culture [accessed 2 June 2021].

7 Whilst objectivity and neutrality are beyond reach, “disinterestedness” – in the sense of a resistance to explicitly frame a piece of research in light of the requirements and interests of policy advocacy – is not just possible, but necessary (Belfiore, 2009).