Beyond the “Toolkit Approach”: Arts Impact Evaluation Research and the Realities of Cultural Policy‐Making

Abstract

This article presents a reflection on the possibility and potential advantages of the development of a humanities‐based approach to assessing the impact of the arts, which attempts to move away from a paradigm of evaluation based on a one‐size‐fits‐all model usually reliant on empirical methodologies borrowed from the social sciences. A “toolkit approach” to arts impact assessment, as the article argues, demands excessive simplifications, and its popularity is linked to its perceived advocacy potential rather than to any demonstrable contribution it may make to a genuine understanding of the nature and potential effects of artistic engagement. The article also explores the relationship between research, advocacy and the actual realities of policy‐making with a view to proposing a critical research agenda for impact evaluation based on Carol Weiss’s notion of the “enlightenment” function of policy‐oriented research. In particular, the article attempts to highlight the contribution that cultural policy scholars working within the humanities could make to this area of policy research.

Introduction

Debates around the social impacts of the arts and the development of methodologies for their measurement and evaluation have played a prominent role in cultural policy discourse and research over the past 20 years. In much of the West, and particularly in Anglo‐Saxon countries, the recourse to the alleged powers of the arts to bring about deep personal change in pre‐identified groups “at risk” of social exclusion or apparently affected by a “poverty of aspirations” (Jowell 2004) has become central to official justifications of public spending in the cultural sector. Clive Gray (2002) identifies this as the result of a strategy of “policy attachment”, whereby the arts, which constitute a policy area commanding small budgets and little political clout, have progressively attached themselves to economic and social agendas, thus benefiting from the larger budgets and greater political influence of those areas of public policy.

In the context of the ostensible commitment of Western governments towards evidence‐based policy‐making, it is perhaps unsurprising that one of the principal results of the policy developments briefly outlined above should have been the blooming of “impact studies”. These purport to be able to measure and assess the extent to which the subsidised arts have a socio‐economic impact, thus contributing to governmental economic and social policy (or failing to do so). This has encouraged in politicians, civil servants, arts funders, cultural administrators and large strands of cultural policy analysis, the development of a “toolkit mentality”, and the quest for a straightforward method of impact evaluation, easily replicable in different geographical contexts and equably applicable to different art forms and diverse audiences.

The methodologies that have been so far applied to the evaluation of the socio‐economic impacts of the arts have been subjected to extensive criticism both by arts professionals — who have also been lamenting the excessive instrumentalization of the arts that, they argue, has accompanied these developments (see, for example, Brighton 1999, 2006; Hytner 2003; Tusa 2000, 2002, 2007) — and academics, who have exposed the flaws, ideological bias and advocacy purposes lurking in many of them (Belfiore 2002; Hansen 1995; Merli 2002; van Puffelen 1996; Selwood 2002). Nevertheless, impact evaluation, and performance evaluation more generally, are still popular, not just in the arts and culture, but in the public sector as a whole (Flynn & Hodkinson 2001). Evaluation and performance measurement are now seen to be crucial in informing budgetary allocations in the United Kingdom and in many other modern Western welfare states (Davies et al. 2006; Radin 2006).

This article aims to present an alternative approach to the arts impact assessment question which avoids the excessive simplifications of the toolkit approach, and acknowledges and engages with the complexity that any rigorous attempt to measure the effects of people’s aesthetic experiences will inevitably throw up. In order to do so, the article builds on and develops the insights we have gained from our previous work around the challenges, both theoretical and methodological, posed by the attempt to develop a rigorous approach to articulate and evaluate the alleged social impacts resulting from engagement with the arts, and presents a critical discussion of the role of evidence in cultural policy‐making. This inevitably requires an exploration of the tensions between genuine research into the potential impacts of the arts and advocacy in the cultural field, and of the relationship between both activities and the actual process of making decisions and shaping cultural policies. As a matter of fact, we know fairly little about how policy decisions are really made, but what we do know — and this is confirmed by research carried out in the field of public policy studies (see, for example, Page 2006; Weiss 1995) — would seem to suggest that evidence is but one of the “ingredients” from which policies are created, and might, in fact, not even be one of the main ingredients, as the rhetoric of evidence‐based policy making would lead one to expect. Indeed, the exploration of the powerful, yet often unacknowledged, role of values and deeply‐held beliefs in the “transformative power of the arts” in policy development and implementation is an important aspect of the analysis presented here. The article finally concludes with some suggestions for a future research agenda in arts impact evaluation.

The Context: On the Problematic Nature of Contemporary Arts Impact Research

As Susan Galloway (2009) observes in her recent assessment of the state of current arts impact evaluation, the focus of research has so far centred on technical rather than epistemological issues. She further points out that “[a]rguably, the main issue for advancing our understanding of the effects of arts interventions is ontological; it is not research methods but the most effective ‘orientation’ or ‘logic of enquiry’”; consequently, the crucial question that still needs answering is “what types of research approach are best suited to investigating the social effects of the arts?” (Galloway 2009, p. 126; original emphasis). Galloway attempts to deal with this question by suggesting that the so‐far neglected approach of theory‐based evaluation as developed within the social sciences might be a useful avenue for researchers to pursue. This article represents a similar quest for an alternative theoretical approach to arts impact assessment, but one that aims to explore whether this could be found from within the humanities. The article suggests that despite the marginal role that humanistic perspectives have so far had in shaping the arts impact debates, there might be advantages to a humanities‐based approach to this area of enquiry, especially where the need to better grasp the role of ideas and beliefs in the policy‐making process is concerned. The article therefore represents an invitation to the scholarly community to further explore and test the hypothesis propounded by this article, so that the potential contribution of the humanities to public debates around the role and impact of the arts in today’s society can be fully explored and its viability over other more traditional and established approaches properly assessed. Our suggestion that the humanities might have a positive contribution to make to this area of scholarship, however, means that the understanding of the “humanities” which underscores the article needs further qualification. Traditionally, the label was used to indicate a clearly identifiable group of disciplines: history, the Classics, English, the fine arts, divinity and philosophy (Plumb 1964). More recent definitions tend to add to these fields of study “their modern (or postmodern) offspring: cultural studies; religious studies; visual studies; postcolonial studies, and feminist studies” (Edgar & Pattison 2006, p. 93). Inevitably, this definition is somewhat simplistic, for certain social sciences — such as, for example, cultural anthropology, critical sociology and social psychology — may be seen to overlap, to an extent, with the humanities both in subject matter and methodologies. However, it is generally agreed that the humanities refer, as the name suggests, to fields of enquiry that are primarily concerned with the exploration of the human condition and of the products of human existence: language, beliefs, writings, artefacts, and social and cultural institutions. The sciences, on the other hand, have at their core the examination of human beings as physical, biological or chemical entities, and their results “are validated against what are taken to be pre‐existent and objective realities” (Edgar & Pattison 2006, p. 93): DNA works the way it does irrespective of human understanding (or lack of understanding) of the mechanisms governing it. By humanities‐based approaches to arts impact research, we therefore refer to a scholarly endeavour located within the broader bundle of disciplines listed above, and aiming to address questions of values and explore the deep‐seated beliefs about the arts and culture that have shaped both cultural policy practice and research over time.

The article’s call for a humanities‐based approach to the arts impact debate is predicated on the observation that public and academic discourses around the powers of the arts to transform individuals and society seem dominated by a number of unquestioned assumptions. These are conveniently summarized by this passage from a speech delivered, in 2003, by the then British Minister for the Arts Estelle Morris:

I know that Arts and Culture make a contribution to health, to education, to crime reduction, to strong communities, to the economy and to the nation’s well‐being but I don’t always know how to evaluate it or describe it. We have to find a language and a way of describing its worth. It’s the only way we’ll secure the greater support we need. (Morris 2003)

The citation above is in no way exceptional, but rather representative of much of the official rhetoric about the social impacts of the arts and the centrality of their measurement to matters of public funding; nor is this rhetoric limited to Britain alone, for much of the same arguments can be found worldwide (Belfiore & Bennett 2008, pp. 1–12). Some of the presumptions emerging from Morris’s statement can be articulated as follows: first, that the “arts” and “culture” constitute clearly identifiable entities; secondly, that these entities do indeed have specific, recognizable impacts; thirdly, that these impacts can be expected to be positive; fourthly, that these impacts can (and should) be evaluated and described, and it is only a question of finding the appropriate method and language to achieve this. Historically, the toolkit approach that has resulted from this quest has tended to privilege quantitative approaches borrowed from the disciplines of economics and auditing, so that the humanities have found themselves squeezed out from this methodological search. ¹ Furthermore, as the citation above unequivocally confirms, discussions of the impacts of the arts and their measurement have become entangled in debates around funding, so that the two are rarely considered independently from one another. As a result, advocacy considerations have often encouraged an uncritical research agenda in this area, a problem the second half of the article will focus on in greater detail.

In order to move the arts impact debate ahead, it is thus important to distance oneself from a position of advocacy and to inquire instead into where these commonly‐held and largely unquestioned beliefs in the social benefits of the arts actually come from, a task for which a humanities approach might be particularly suitable. For instance, our previous historical work in this area, which looked at claims made — from the times of Plato to the present day — for the power of the arts to bring about both individual and societal transformation, shows that policies based on the perceived benefits of the arts are ultimately based not on evidence, but rather on millennial, deep‐seated beliefs about the role and functions of the arts in society which, through incorporation in the educational system and important cultural institutions, have become accepted and are therefore rarely questioned or scrutinized in contemporary Western societies (Belfiore & Bennett 2008). Yet, despite its influence, contemporary public debates around policies for the arts seem to have become disconnected from this rich and varied tradition of thinking and writing about the effects, both positive and negative, of the arts (Belfiore & Bennett 2007a).

Another problematic feature of the impact discourse is the extent to which notions of “impact” and claims for the “transformative powers” of the arts have become — in debates around public arts funding — a shorthand for a much broader and complex question, namely: what are the value and function of the arts in contemporary society? Insofar as it attempts to bypass the contested nature of any discussions around cultural values, the arts impact evaluation discourse is destined to fail in providing a convincing answer to this question. Inevitably, matters of value (and especially matters of public value) are more complicated and politically sensitive than any toolkit or one‐size‐fits‐all approach could ever hope to deal with. ²

A further complication in the attempt to capture and study the effects of the arts lies in the persisting uncertainty and vagueness of the literature around the specific mechanisms (cognitive, emotional, psychological, etc.) through which the arts are perceived to alter people’s behaviour, sensitivity and their understanding of themselves and the world around them. Yet, we would argue that before “impact” and its measurement can be discussed in any meaningful way, we need a better understanding of the interaction between people and the arts. A review of the research carried out in this area (both in scientific and humanistic fields of enquiry), indeed, leads to the conclusion that, despite great improvements in our understanding of aesthetic responses, the mechanisms by which people might be deeply affected by the arts are still largely unclear, and we simply cannot expect to predict how individuals might react to each art form or specific artworks (Belfiore & Bennett 2007b). It logically follows that broad generalizations about people’s experiences of the arts are never likely to be convincing, and the notion that policy‐makers can plan projects and cultural activities around predefined desirable impacts seems equally dubious.

A further theoretical problem that emerges from any discussion of the effects of the arts is represented by the difficulty in coming up with a robust definition of what is actually signified by the term “the arts”. The arts, as well as individual art forms, are not ontologically stable entities, but are socially and culturally constructed notions, which change over time and around whose nature or definition there is very little consensus, even among “experts”. John Carey (2005, p. 29) has recently argued that “[a] work of art is anything that anyone has ever considered a work of art”, thus suggesting that a universally valid and objective definition of “art” may be beyond reach. Nevertheless, it is obvious that if one cannot say with any degree of clarity what something is, it becomes very difficult to say (and measure) what it does and how it does it (Belfiore & Bennett 2009).

The complexity inherent in any serious discussion of the notion that the arts may have significant social impacts should by now be clear: with the present levels of knowledge around aesthetic reception, it is not possible to make any meaningful broad generalization about how people respond to the arts, and if or how they might be affected by the experience. Even less plausible is the possibility of actually “measuring” any of these aspects. Significantly, such problematic issues have not been raised within academia alone, but are also acknowledged by practitioners, researchers and policy‐makers concerned with the practical shortcomings of current ways of attempting to measure empirically the social impact of the arts, and evaluate the effectiveness of policy measures aiming to promote social inclusion through engagement with the arts (Belfiore 2006; Cowling 2004; Reeves 2002).

If, then, the toolkit approach to arts impact assessment is inherently flawed, what future lies ahead for this strand of research, if any? Is the attempt to understand, assess and quantify the effects of artistic engagement a pointless exercise? There are, of course, two possible answers to this question, and it is here that the tension between “pure” research and research for advocacy purposes becomes clearer and sharper.

If the aim of impact research is to tackle the challenging questions raised by the complexity of people’s aesthetic experiences, then the impacts of the arts and the nature and effects of people’s response to the arts are a worthwhile area of enquiry and one in which research is badly needed to address the gap in knowledge pointed out above. However, if the aim of the exercise is to make a compelling case for the funding of the arts (and, historically, this type has accounted for a great proportion of the research carried out in this area), then arts impact assessment is simply not worth the time, effort and resources it requires. For, if the “evidence base” provided for the socio‐economic impacts claimed is found to be questionable, then it is not going to provide an effective platform for the pursuit of advocacy. Obviously, this dilemma throws up important questions about the direction of future arts impact research, about the need to separate rigorous impact evaluation research from advocacy, and more generally about the place that evidence and the production of evidence have in contemporary cultural policy‐making. The exploration of these questions is at the heart of the rest of this article.

What Role for Research in Cultural Policy‐Making?

As we have argued elsewhere (Bennett 2007), a serious critique of contemporary arts impact assessment and practice such as the one presented above brings to the surface very clearly not only the tensions between research, evaluation and advocacy in the arts, but also the sense of a “missing element” in much of the public debate around the social and economic impact of the arts. Indeed, despite the resilient popularity of impact studies, there has so far been perplexingly little effort to establish to what extent “evidence”, when available, has had a significant effect (or, in fact, any at all) on the actual process of decision‐making and policy formation.

In the United Kingdom, for instance, the government’s Policy Action Team 10 (PAT 10), responsible for arts and sport, produced an influential report in 1999 on the alleged contribution of the sector to New Labour’s social inclusion and neighbourhood renewal agenda. In the foreword to the report, Chris Smith, at the time Secretary of State for Culture, stated very confidently:

This report shows that art and sport can not only make a valuable contribution to delivering key outcomes of lower long‐term unemployment, less crime, better health and better qualifications, but can also help to deliver the individual pride, community spirit and capacity for responsibility that enable communities to run regeneration programmes themselves. (Department for Culture, Media and Sport 1999a, p. 2)

This report is generally credited with having given real momentum to the notion of social impact as one of the key rationales for public investment on the arts in Britain and the resulting search for a reliable method of evaluating it (Belfiore 2002). But, paradoxically, another report, also commissioned by the PAT 10, was published in the same year, which concluded that: “it remains a fact that relative to the volume of arts activity taking place in the country’s poorest neighbourhoods, the evidence of the contribution it makes to neighbourhood renewal is paltry” (Department for Culture, Media and Sport 1999b, p. 6). It seems significant, in the context of an official commitment to evidence‐based policy, that, despite the formal admission of the lack of solid evidence of the effectiveness of the arts in contributing to social cohesion and neighbourhood regeneration, the rhetoric of impact should have continued to define British debates around arts funding and policy right up to the present day. It is clear that evidence was not the sole, nor in fact the prime, driver behind policy in this case. This raises a number of questions, which this section of the article will explore, namely: to what extent is “evidence” the real basis on which policies are formed, even in the context of evidence‐based cultural policy? And: what is the relationship between research and the policy sphere, and is this relationship inevitably mediated by the requirements of advocacy?

Research and Policy: A Tenuous Link?

In spite of the increasing popularity and acceptance of evidence‐based policy (hereafter, EBP) within both policy theorization and practice, the pervasive perception that policy development remains largely unaffected by research is well documented in the literature (Davies et al. 1999; Kogan 1999; Leviton & Hughes 1981; Weiss 1977). As a matter of fact, it has been suggested that “[w]e still know relatively little about the dynamics of the policy process and how research evidence impacts on this process” (Nutley & Webb 2000, p. 29) and that, effectively, when it comes to public decision‐making, “[m]uch activity remains an act of faith” (Davies et al. 1999, p. 3). Bulmer (1987, p. 7) suggests that, as far as the British case is concerned, the perceived lack of influence of academic research in the policy sphere might be connected to the British system of civil service and the country’s political culture, which tends to give more weight to the kind of knowledge that derives from “accumulated experience”, and according to which universities are (at least traditionally) the place dedicated to the formation of future elites rather than to the creation of “useful knowledge”. Yet, criticism of the alleged lack of utilization of important research findings can be found internationally, and in the attempt to better understand this phenomenon, a dedicated body of research began to be developed in the mid‐1970s which explored precisely the use or non‐use of research in policy‐making (Weiss 1995, p. 140).

Weiss (1995), in reviewing this strand of research, admits that the studies considered could not identify a clear connection between individual pieces of research and specific policy developments. Yet she comes to the following conclusion:

The studies also found that decision makers believed that they were influenced by policy‐oriented research. Often they could not cite the name of any particular study and many of them could not even remember reading a research report. But in circuitous ways research findings came into circulation and ideas from research percolated into the policy arena. People talked about them at meetings and conferences; lunchtime conversation centered around them. And people had the sense that they had heard generalizations from research and that these ideas had influenced their thinking. (Weiss 1995, p. 141)

This belief in the connections between research and policy, even in the face of difficulties in tracing specific examples of impact and precise trajectories of influence, can perhaps explain the persisting emphasis, in the official governmental rhetoric, on research as a source of the knowledge that is required in the policy‐making process. For reasons of space, the examples of such rhetoric are selected from the British case, yet similar trends, as observed earlier, are operating within most Western liberal states (Radin 2006).

In order to explain the potential contribution of research to the policy sphere, we will need to explore, if briefly, the nature of an evidence‐based approach to the policy process. The notion of EBP is built around the explicit commitment, in New Labour’s 1997 electoral manifesto, to finding out what measures are effective in resolving identified problems, and in letting the motto “What works is what counts” guide both policy and spending decisions (Wells 2007, p. 22). In 2000, in an important speech to the Economic and Social Research Council ³ , David Blunkett, then Secretary of State for Education and Employment, reiterated the party’s view of evidence as a policy driver:

It should be self‐evident that decisions on Government policy ought to be informed by sound evidence … The Government has given a clear commitment that we will be guided not by dogma but by an open‐minded approach to understanding what works and why. (Blunkett 2000, p. 12)

Blunkett’s statement is useful in that it points our attention towards two important elements of EBP: the focus on “what works” and the centrality of “policy evaluation” in the policy process. The emphasis on results entails the need to establish and evaluate the effectiveness of any new measure implemented to resolve a perceived problem. The interest around arts impact evaluation is, therefore, a clear example of the professed need to verify that policies implemented on the premise of the arts’ alleged power to bring about social cohesion and change (as seen in the PAT 10 report cited above) have been effective in delivering against those policy objectives. Because of the interests vested in the results of such evaluations, their “political misuse” (Datta 2006, p. 420) is an important problematic in EBP. Yet, for the supporters of EBP, the rigorous evaluation of public policies and programmes is an essential component of a democratic and transparent government (Chelimsky 2006, p. 33).

Blunkett’s quote also spells out a third central tenet of the EBP philosophy, that is, the notion that it constitutes an ideology‐free procedure to guide decision‐making in the public sphere. As Blunkett’s words show, the government itself seems to have subscribed to a highly depoliticized idea of its own working, by explicitly rejecting an ideologically informed view of politics in favour of a modernization agenda which emphasizes a managerialist and technocratic understanding of the state, whereby decisions are based on the rational use of research evidence and scientific knowledge as opposed to questions of ideological doctrine and propriety (Wells 2007, p. 23).

Taken at face value, the notion that policy‐making ought to depend on the rational use of rigorously acquired knowledge of social problems, their causes and possible solutions seems hardly objectionable. However, the official emphasis on “what works” is misleading in that it seems to presume, and it is predicated upon, the possibility of policy‐making as a politically neutral exercise. In this respect, however, New Labour is but following in a long tradition of attempts to detach the field of public administration from the controversial and contested sphere of politics. As early as 1887, in his The Study of Administration, Thomas Woodrow Wilson — who later became the twenty‐eighth president of the United States — laid out what would become an influential articulation of the distinction between “politics” and “administration” when he maintained: “The field of administration is a field of business. It is removed from the hurry and the strife of politics”; for Wilson, “administration lies outside of the proper sphere of politics. Administrative questions are not political questions” (quoted in Behn 2001, p. 43). Arguably, in Britain, New Labour’s Third Way politics has pushed one step further these long‐standing efforts to remove, at least at the rhetorical level, the task of public administration and government from ideology and political calculation (Finlayson 2003).

Yet, as much policy research over the past two decades has clearly shown, the sphere of public administration is hardly value‐neutral or ideology‐free, and the stress on research, knowledge and evidence as the drivers of policy merely obfuscates the inherently political nature of the policy‐making process (Radin 1995). This emerges clearly from Carol Weiss’s definition of policy‐making:

The policymaking process is a political process, with the basic aim of reconciling interests in order to negotiate a consensus, not of implementing logic and truth. The value issues in policymaking cannot be settled by referring to research findings. (Weiss 1977, p. 533)

Furthermore, it is important to point out that, if it is impossible to dismiss the inevitably political nature of the policy process, it is equally impossible to presume the political neutrality of policy analysis. As Radin (1995, p. 92) argues, policy analysts “cannot insulate themselves from the dynamics of politics, interests groups and deadlines”. To complicate matters further, analysts are sometimes unable (or unwilling) to accept and admit that their own values and beliefs might influence how they approach policy issues in their professional practice (Radin 1995, p. 100). ⁴

Thus, as Parsons (2002, p. 54) notes, the theorization and practice of EBP results in an artificial depoliticization of the policy process, for it obscures, from the outset, the role of what are, in effect, central aspects of policy‐making: people, power and politics. ⁵ Nevertheless, as Weiss explains:

The prevailing concept of research utilization stresses the application of specific research conclusions to specific decisional choices. A problem exists; information or understanding is needed to generate a solution to the problem or to select among alternative solutions; research provides the missing knowledge; the decision makers then reach a solution. (Weiss 1977, p. 533)

How can we explain the persistence, at least at the policy‐rhetorical level, of a linear model of policy‐making, which conceives of a close and direct relationship between the identification of a problem, research into its nature, causes and possible solutions and policy formulation? This is a crucial question if we are ever to gain a better grasp of how policy‐making functions in reality, and the following section offers an attempt to answer it.

Varieties of “Reason”: Evidence‐Based Policy as “Instrumental Rationality”

Sanderson (2002, p. 1) has suggested that “[t]he increasing emphasis on the need for evidence‐based policy indicates the continuing influence of the ‘modernist’ faith in progress informed by reason”. As an approach based on a view of policy‐making rooted in rational decision‐making, EBP is thus deeply embedded in the European — and more broadly, Western — forma mentis as it has come to be shaped by crucial historical moments in intellectual history such as the scientific revolution of the seventeenth century, the Enlightenment, and the subsequent technological and scientific developments.

What we are therefore suggesting is that the persistence of a linear model of the relationship between knowledge creation through research and policy development is tightly linked to a certain idea of rationality which is the legacy of the Enlightenment. Not even postmodernist critiques of the faith in reason and the project of modernity seem to have completely dispelled the widespread optimism about the centrality of scientific research as a route to progress. In fact, contemporary society has witnessed the proliferation of what Di Maggio (2001, pp. 249–250) refers to as “techniques aimed at maximizing instrumental rationality”, by which he means “the systematic attempt to understand and act on one’s understanding of systems of cause and effect”. However, rationality has not always been understood in such terms, and in tracing the history of the development of this “instrumental rationality” one can also begin to make sense of the marginalization of the humanities in policy‐sensitive areas of research today. In order to explore these themes, this portion of the article will draw on the work of British philosopher Stephen Toulmin.

The developments which ultimately resulted in the prevalence of an instrumental rationality took place gradually, over a period of at least three centuries, but culminated in the 1600s. Galileo, Bacon and Descartes represent the principal figures in this intellectual revolution which signalled the shift from the Middle Ages to modernity: this moment was marked by the development of a “theory‐centred” approach to philosophy and an emphasis on universality which would go on to dominate modern philosophical thought until the questioning of the Enlightenment legacy and of traditional accounts of the development of modernity at the hands of postmodern theory in the mid‐twentieth century. Toulmin (1990, p. 14) qualifies this as both a scientific revolution — in that it resulted in remarkable advancement in the scientific areas of physics and astronomy, and in the theorization of the centrality of experimentation in the scientific method — and a philosophical one — in that it produced a new understanding of human knowledge and how it can be acquired.

Thus, Descartes, in his Meditations (1641), endeavoured to find solid foundations for crucial areas of human knowledge that are “clear, distinct and certain” (quoted in Toulmin 1990, p. 72), thus opening up the way for the philosophical rationalism that would eventually become the hallmark of modernity and its project of human progress. A central objective of Cartesian philosophy, then, was to establish and promote “new, mathematical kinds of ‘rational’ certainty and proof” (Toulmin 1990, p. 75). The corollary of these developments is important for the enquiry in hand, for it is clear that, from the 1600s onwards, the predominance of a rational approach to knowledge formation modelled on the scientific method brought about a momentous change in attitude centred on “the devaluation of the oral, the particular, the local, the timely and the concrete” in favour of abstract, universal and timeless theories (Toulmin 1990, p. 75):

In a world governed by these intellectual goals, rhetoric was of course subordinate to logic: the validity and truth of “rational” arguments is independent of who presents them, to whom, or in what context — such rhetorical questions can contribute nothing to the impartial establishment of human knowledge. For the first time since Aristotle logical analysis was separated from, and elevated above, the study of rhetoric, discourse and argumentation. (Toulmin 1990, p. 75; emphasis in original)

In a later study, Toulmin (2001) develops this argument further and shows how the mid‐seventeenth century was the time when that tension between different methods of enquiry — that of the natural or exact sciences on the one hand, and that of the humanities on the other (which C. P. Snow would later characterize as “the two cultures”) — first began to develop:

Certain methods of enquiry and subjects were seen as philosophically serious or “rational” in a way that others were not. As a result, authority came to attach particularly to scientific and technical inquiries that put those methods to use … Beside the rationality of astronomy and geometry, the reasonableness of narratives came to be seen as a soft‐centred notion, lacking a sold basis in philosophical theory, let alone substantive scientific support. Issues of formal consistency and deductive proof thus came to have a special prestige, and achieved a kind of certainty that other kinds of opinions could never claim. (Toulmin 2001, p. 15; emphasis in original)

The post‐1600s distinction between serious and non‐serious methods of philosophical analysis that the passage above identifies is predicated on the distinction between two different notions of “reason”, embodied by the disciplines of logic and rhetoric:

The analysis of theoretical arguments in terms of abstract concepts, and the insistence on explanation in terms of universal laws — with formal, general, timeless, context‐free, and value‐neutral arguments — is nowadays the business of Logic; the study of factual narratives, about particular objects or situations, in the form of substantive, timely, local, situation‐dependent, and ethically‐loaded argumentation, is at its best a matter for Rhetoric. Academic philosophers and serious‐minded theorists in any field are concerned only with the first. (Toulmin 2001, pp. 24–25)

The corollary of this argument is clear: the humanities and the exact sciences represent different ideas of philosophy and different notions of reason. The former are sensitive to the subjective nature of human experience, sceptical towards generalizations and posit that absolute certainty might be an unattainable goal; the latter aspire to order reality according to a set of general and universal rules and theories, with the aspiration to certainty — or at least formal certainty (Toulmin 2001, p. 32).

This distinction, which from the outset had a strong normative dimension to it, seems to have been internalized by the humanities themselves together with the underlining charge of inferiority and the resulting crisis of confidence. This could arguably be seen as the root cause for the “rhetoric of doom and gloom” that seems to run through so much of the old and current literature on the “state of the humanities”. In the mid‐1960s, English historian J. H. Plumb (1964) felt the need to edit a series of essays under the title Crisis in the Humanities. Since then, anxiety over the perceived loss of credibility and negative image of the humanities has been mounting ⁶ , and numerous scholars have recently been reflecting on the “dangers” facing the humanities (Menand 2005), their perceived “uselessness” (Bérubé 2003), and have been pondering over “the fate of the humanities” (Hohendahl 2005) and the “humanities in ruins” (Szeman 2003); they have even wondered whether we should simply bid “farewell to the humanities” on account of the “collapse” of their very raison d’être (Wang 2005). Importantly, however, American literature professor Michael Bérubé, long‐standing champion of the humanities and of the values of a liberal education, has suggested that doubts about the practical utility of the arts and humanities (on which the seemingly negative perception of the humanities is based) might in fact be, to a large extent, “a self‐inflicted indignity” (Bérubé 2003, p. 25). Such “indignity”, for Bérubé, originates from a misguided conviction — rooted in the cultural shift discussed by Toulmin — that the natural sciences are always necessarily more “useful” (and, by proxy, more valuable) than the humanities, and the attendant feeling that the latter are inevitably doomed to fail if they are to compete with these other disciplines on pragmatic and utilitarian grounds. ⁷ Whatever one makes of such gloomy perceptions of the image of the humanities, the fact remains that the presumed lack of “usefulness” of the humanities (and the resulting defensive or even combative hostility of some humanists to research that is directed by pragmatic and utilitarian interests), combined with the often arcane and obscure jargon of its disciplines and the widespread feeling that they might have little relevance beyond the academy ⁸ , might at least in part explain the tendency of humanities scholars to focus on “critical” rather than “administrative” research, which “derives its funding and orientation from governmental or private interests” (Sterne 2002, p. 60). This, in turn, might help to illuminate the relative marginalization of humanistic perspectives from policy‐sensitive research or other areas of enquiry where the value of research tends to be measured in terms of its socio‐economic impact.

But how do the social sciences and hence the study of public policy fit into this dual classification of philosophical approaches? In light of the perceived success of the natural sciences in predicting and controlling natural reality, the social sciences and economics — which strived to achieve the same in the realm of the human sciences — chose to follow the mathematical method propounded by Galileo. Yet, Toulmin (2001, p. 66) suggests that even in the fields of the social sciences and economics, the traditional reliance on a form of enquiry modelled on the natural sciences has meant the fallacious acceptance of “data” as purely “factual” and therefore neutral; this, in turn, will inevitably lead to “misunderstandings and errors of practical judgement”.

Therefore, the tendency of EBP to rely on quantitative methods borrowed from the social sciences and economics (at the expense of humanistic perspectives) that was discussed earlier is but a symptom of the persisting higher prestige of logic over rhetoric. Inevitably, this had important repercussions on the development of the research methods of policy analysis. In contrast to the multidisciplinary methodological approach initiated by some of its founders, such as Harold Lasswell, the discipline of policy analysis, which blossomed in the 1960s and 1970s, has evolved in more restricting ways and along technocratic lines derived from the neo‐positivist/empiricist methods that predominated at the time:

This has generated an emphasis on rigorous quantitative analysis, the objective separation of facts and values, and the search for generalizable findings whose validity would be independent of the particular social context from which they were drawn: that is, a policy science that would be able to develop generalizable rules applicable to a range of problems and contexts. (Fischer 2003, p. 4)

Essential to this method has been an intention to sidestep ideological and value conflicts generally associated with policy issues by translating political and social problems into “technically defined ends to be pursued through administrative means” (Fischer 2003, p. 4). In the case of cultural policy, and arts impact assessment more specifically, this has meant the elaboration of the question of what role and functions the arts may have in contemporary society in an explicitly technical fashion rather than as a question of cultural values. The difficulties of impact assessment have therefore been translated into a purely methodological problem that can be solved through the search for a suitable and workable “impact evaluation toolkit”, and the development of improved and more effective performance indicators for the cultural sector (Madden 2005). However, if we accept that policy‐making is an inherently political exercise which can never aspire to be value‐free, and that the reliance on “evidence” acquired through positivistic methods cannot circumvent the importance of variables such as ideas and values in the policy process, we need to assess the extent to which advocacy might be an inevitable aspect of research utilization in the policy sphere. The following section looks at this problem in more detail.

Research, Evidence and Advocacy in the Policy Process

Policy theorists agree that a consequence of the long dominance of the “rational” model of policy‐making has been the creation of the expectation that the policy process is going to be organized, systematic and easily directed towards its designed goals (Hill 1997, p. 9). However, as we have seen, the policy‐making process in reality is more complicated than the model presumes, and growing awareness of this has resulted in a strand of research inspired by the recognition that “policy processes are complex, influenced by a variety of external factors which are hard to control and in some respects haphazard” (Hill 1997, p. 2). Recent alternative policy models have thus emphasized the irrational aspects of the policy process. Policy scholars have developed metaphors such as “the black box of decision‐making”, where different inputs such as “demands” and “supports” ⁹ are fed in and undergo a “conversion process” that produces “outputs”, namely decisions and policies (Hill 1997, p. 20), and the “policy primeval soup”, where all sorts of different policy solutions float together waiting to be fished back up by “policy entrepreneurs” when the right problems come along and require fixing (Ahearne 2006, p. 3). Other policy scholars have brought to light the discursive practices that are at the root of the policy process, and have focused their enquiry on the place of ideas, values and power relations in policy‐making (Fischer 2003).

What all these different scholarly developments have in common is that they seem to confirm the contention that “[i]deology, expediency and public preferences compete with scientific evidence for the ears of the Ministers” (Davies et al. 1999, p. 4). In this scenario, to suggest that the production of evidence and the selection of what evidence to refer to in the process of policy design could ever be a politically neutral exercise is at best naive, and at worst misleading. Interestingly, in his aforementioned speech, David Blunkett (2000, p. 13) did quite candidly acknowledge that “politicians have a tendency to believe research when it reinforces their own view”; he also admitted that, in his own experience as a minister, he might himself have let his “prejudices over‐ride the legitimate empirically‐based evidence”.

If instrumental considerations and political expediency cannot be ignored in the attempt to understand the policy process, it becomes understandable how the suggestion could be made that what has really taken place is not, in fact, a shift towards “evidence‐based policy‐making”, but rather a phenomenon of “policy‐based evidence‐making” (Belfiore & Bennett 2007a). In the latter case, evidence is produced to support and legitimize policies that would have been implemented anyway because of political will, irrespective of evidence (the PAT 10 reports and the notion of the arts as a means to promote social inclusion and cohesion being a case in point), or to support advocacy and lobbying activities by organizations with a specific agenda to push forward. The growth of research commissioned by governments, foundations, lobby groups and, more importantly, think‐tanks has also had an important role in the trend towards “policy‐based evidence‐making”. Research in this case is created by institutions that have clear research as well as policy agendas: they operate very differently from other research environments, such as, for instance, universities, which work according to well‐established academic norms and where usually the research that is carried out tends to be dictated mostly by the researcher’s own intellectual interests (Radin 1995, p. 40). As Weiss (1995, p. 149) puts it: “The rise of advocacy groups and think‐tanks on the right and on the left that use research simply as argumentation in support of their positions opens the whole endeavour [of policy‐oriented research] to question”.

It is, indeed, challenging (though not necessarily impossible) to guarantee the freedom required to ask the types of complex, exploratory and genuinely open‐ended questions required for knowledge production in the context of policy‐oriented commissioned research. As Sanderson (2003, p. 342) points out: “the policy client normally has a strong interest in seeing findings that can be used to improve policy design or implementation within timescales dictated by the political process”. However, it would be naive to suggest that universities are the natural environment for disinterested and rigorous policy‐sensitive research to flourish unhindered. Universities do not operate in splendid isolation from the forces and influences of politics and the market.

In his aforementioned speech, ex‐minister Blunkett expressed his commitment to ensure that policy influence should have a prominent place in the “Research Assessment Exercise” (RAE), a procedure the government adopts to assess the quality of the research carried out in British universities; importantly, universities’ results in this exercise affect the levels of public funding they will receive. Recent documents by Research Councils UK (2006, 2007) echo Blunkett’s sentiment and clearly announce the intention of including impact on policy and the projected socio‐economic impact of the planned research activities as one of the criteria used to decide on the allocation of research funding to universities. Leviton and Hughes (1981) have demonstrated that advocacy considerations also have an important bearing on whether research findings are utilized in the policy sphere or not. This means that, in a climate where policy influence is considered a relevant criterion for the allocation of research funds, the type of research that is more likely to provide the “evidence” that politicians and decision‐makers with vested interests need might be supported over more controversial or politically undesirable research agendas, regardless of matters of intellectual merit and methodological rigour. ¹⁰

The temptation to articulate research questions in advocacy‐friendly terms can be witnessed in the arts impact debate, where research has often focused on asking how the (presumed) positive social impacts of the arts might be measured, rather than asking whether the arts have social impacts, if these impacts can be expected to be positive and, more generally, whether people’s responses to the arts are amenable to measurement and generalization. However, does the research agenda in policy‐sensitive areas have to be enslaved to the requirements of advocacy? Or is it possible to come up with a disinterested research agenda in arts impact assessment than can aspire to “influence” the policy sphere without compromising the integrity and rigour of the enquiry? The concluding section of this article argues that this is, indeed, possible to envisage. Carol Weiss’s (1977) notion of the “enlightenment” function of policy‐oriented research can provide us with a model of the relationship between research and policy that represents a middle ground between the perceived dichotomy of either the “influence” or “irrelevance” (Blunkett 2000) of research in policy design.

Conclusion: The “Enlightenment” Function of Research and a Way Forward for Arts Impact Assessment

The argument put forward by Weiss is that the way in which research evidence comes to shape policy might be subtler and more complex than allowed by a causal model of utilization which sees policy design as a direct result of certain relevant pieces of research:

Evidence suggests that government officials use research less to arrive at solutions than to orient themselves to problems. They use research to help them think about issues and define the problematics of a situation, to gain new ideas and new perspectives. They use research to help formulate problems and to set the agenda for future policy actions. And much of this use is not deliberate, direct, and targeted, but a result of long‐term percolation of social science concepts, theories, and findings into the climate of informed opinion. (Weiss 1977, p. 534; emphasis in original)

This understanding of the relationship between research and policy goes some way towards making sense of the fact that policy‐makers are often incapable of making connections between their own decisions and research, yet remain convinced that research was an important source of ideas and knowledge for them. This is because, through contact with research, “[b]its of information seep into [the policy‐maker’s] mind, uncatalogued, without citation” (Weiss 1977, p. 534), but even when it cannot be retrieved or consciously referred to, research gives decision‐makers “a background of ideas, concepts and information that increases their understanding of the policy terrain” (Weiss 1995, p. 146).

Research, therefore, affects policy not so much through immediate and direct impact on the design of public policies, but rather mainly through what Weiss (1977, p. 535) calls its “enlightenment” function: “the major effect of research on policy may be the gradual sedimentation of insights, theories, concepts, and ways of looking at the world”. This kind of diffuse and undirected input of insight from research into the policy sphere can gradually result in profound shifts in thinking and perceptions around social problems and their solutions, ultimately determining new and significant policy developments. Importantly, Weiss’s (1995, p. 143) research on research utilization shows that policy‐makers are well disposed to this “enlightenment” view of research: “usefulness” for them does not necessarily mean that the findings are going to be immediately implemented, and “studies that [help] people think in different and innovative ways” hold a significant appeal for them. Interestingly, confirmation of this comes from none other than ex‐minister David Blunkett (2000, p. 21), whose speech declares that: “We need researchers who can challenge fundamental assumptions and orthodoxies and this may well have big policy effects much further down the road”.

We would, indeed, argue that our aforementioned historical study of the powerful and long‐standing beliefs in the “transformative powers of the arts” (Belfiore & Bennett 2008) might have a similar “enlightenment” function for the cultural policy‐maker. Whilst it was not designed to fit snugly around current policy debates and advocacy‐led research priorities, it might provide the kind of background ideas, concepts and analysis that could move arts impact research forward in interesting directions, and might ultimately feed back into policy debates. If we take up Weiss’s (1995, p. 141) invitation to broaden our understanding of research utilization so as to accommodate the more general notion of “conceptual use”, we can begin to conceive of a relationship between the spheres of academic research and public policy that does not necessarily have to be mediated or shaped by an advocacy agenda.

In clarifying the nature of the arts impact research agenda we are proposing, it is useful to refer to the distinction put forward by public policy theorist Giandomenico Majone (1988, p. 157) between two possible types of policy analysis. The first one is concerned mainly with the problem of allocating limited financial resources among competing ends; cost–benefit analysis and evaluations are central to this type of research, which is underscored by an instrumental notion of rationality. The second type of policy enquiry, on the other hand, attempts to embrace the complexity of the policy process, the essence of which is identified in processes of argumentation. This second type of analysis is rooted in an older (Toulmin would say pre‐Enlightenment) notion of rationality conceived as “a process of finding acceptable reasons, discovering warrants for one’s beliefs or actions” (Majone 1988, p. 157). Majone (1988, p. 158) further suggests that the main task for this form of policy analysis “is not to determine theoretically correct solutions, but to raise issues, probe assumptions, stimulate debate, and especially to educate citizens to distinguish good and bad reasons”.

We would suggest that Majone is offering us a useful pointer for a fruitful arts impact research agenda which is not confined to the demands of an instrumental rationality: a critical approach that aims at an open enquiry of the problems, both theoretical and methodological, which are inherent in the project of understanding the response of individuals to the arts and trying to investigate empirically the extent and nature of the effects of the aesthetic experience. In this kind of scenario, the humanities would certainly have a role to play in the production of knowledge and insights that may, eventually, feed into the public sphere, and “enlighten” both public opinion and decision‐making around the role of the arts in contemporary society and their place in government policy. This would go some way, we think, towards the reinstatement of that complementariness of logic and rhetoric, of the exact sciences and the humanities, which characterized the human pursuit of knowledge before the intellectual and scientific revolution of the 1600s, thus restoring that “Balance of Reason” which Toulmin (2001, p. 29) calls for.

Acknowledgements

The authors would like to acknowledge the financial support of the Arts and Humanities Research Council, UK, and of Arts Council England through their Fellowship in Arts Impact Assessment scheme.

Notes

1. Caust (2003) suggests that this trend goes beyond just impact assessment, and laments that the entire realm of arts policy has been ‘captured’ by economists and marketers. Similarly, Rothbard (1989, p. 45) suggests that the hermeneutic invasion, on the part of economics, of other spheres of enquiry amounts to ‘a modern form of “economic imperialism” in the realm of the intellect’.

2. Arts Council England’s recent large‐scale ‘public value campaign’ seems to be a first indication of a new willingness in the sector to acknowledge and tackle questions of value — and public perceptions of value — in cultural policy (see http://www.artscouncil.org.uk/about-us/research/public-value-programme).

3. This is the main body charged with distributing public funding for academic research in the economic and social sciences in the United Kingdom.

4. In our own case, however, our own perception of the way in which humanistic disciplines had been sidestepped by quantitative social research methodologies and econometrics, and our contention that it may be worth exploring if humanities‐based approaches could make a valuable contribution to the field of cultural policy studies, was always explicitly acknowledged in the framing of this article. Whether this article provides convincing arguments in favour of a humanistic approach to arts impact evaluation research remains for the reader to decide.

5. For example, a serious gap has been pointed out in our understanding of the role of policy advisors and civil servants and their ideas in shaping policy formation (Page 2006; Page & Jenkins 2005; Parsons 2002).

6. Menand (2005, p. 11) maintains that ‘[i]t is possible to feel that one of the things ailing the humanities today is the amount of time humanists spend talking about what ails the humanities’.

7. Yet, as Bérubé (2003, p. 26) goes on to argue: ‘surely the more speculative sciences, from astrophysics to evolutionary theory, do not have quite the same claim on practical utility; surely some endeavours in pure mathematics or cosmology contribute no more than does the study of medieval tapestry to the economic or physical well‐being of the general citizenry.’

8. The image problem of the humanities, their perceived lack of confidence and the charges of ‘uselessness’ and irrelevance that are often moved against them are discussed in Belfiore (2009).

9. Hill (1997, p. 20) defines ‘demands’ as involving ‘actions by individuals and groups seeking authoritative allocations from the authorities’ and ‘supports’ as ‘actions such as voting, obedience to the law, and the payment of taxes’.

10. The humanities have not been unaffected by such trends, as attested to by the recent adoption of an ‘impact strategy’ on the part of the Arts and Humanities Research Council — the main public funder of arts and humanities academic research in Britain — aimed at maximizing the socio‐economic impact of funded research (see http://www.ahrc.ac.uk/About/Policy/Documents/impact%20strategy.pdf).