Theory-based evaluation and the social impact of the arts

Abstract

The well-documented challenges in researching the social impacts of the arts are closely related to key issues in contemporary social research and evaluation, most particularly the problem of causal attribution. The article contends that some of the most common criticisms of the evidence base for the social impact of the arts relate to the successionist model of change which underpins positivist social science research and evaluation. Illustrating this with reference to research on the arts and quality of life, the article considers the alternative generative understanding of causation that underpins theory-based evaluation (TBE) approaches, favoured recently in the UK as part of the “What Works?” agenda. While these approaches fit well with knowledge about the determinants of arts impact, the article considers whether in fact TBE approaches offer an effective strategy for understanding how and why arts engagement can result in social change. The limitations and possibilities of TBE are considered with reference to four recent UK studies of the impact of the arts on individuals which make use of them.

Keywords:

• arts
• social impact
• theory-based evaluation
• realistic evaluation
• Theory of Change
• quality of life

Acknowledgements

The author would like to acknowledge the support of the Arts and Humanities Research Council and to thank the anonymous referees and colleagues who provided helpful suggestions.

Notes

For example Reeves (2002) summarizes the range of reasons for the deficiencies in social impact evaluations identified by Blake Stevenson Ltd (2000), Jermyn (2001), Matarasso (1996), Moriarty (1997) and Shaw (1999).

The reasons ascribed to this are remarkably familiar to the arts. They include the lack of co-ordination and incentives for academic research in this area (Millward et al., 2003, p. 31), a point referred to in relation to the arts (Belfiore, 2002; Oakley, 2004; Selwood, 2006).

In the arts, for example, this has been found to be the case for studies looking at the relationship with individual quality of life (Galloway, 2006) and academic attainment (Winner & Cooper, 2000). The latter involved a rigorous meta-analysis of studies which found a picture of conflicting findings, particularly between studies carried out in different countries which the authors felt might be accounted for by national cultural differences influencing the status of the arts within school education and the context, and therefore motivations, for pupils to choose to study the arts (p. 63).

For example, White and Rentschler's literature review identified 12 key articles, four of which were empirical studies and eight of which were conceptual (2006, p. 4).

The few explicit criticisms of successionism made in the context of the arts include Matarasso, who discussed the relative strengths and weaknesses of causal link and causal mechanism analysis in his 1996 Working Paper for Comedia (1996, pp. 19–20) and, more recently, Miles and Clarke in the context of the arts in criminal justice (2006). Implicitly, the main reason for Belfiore and Bennett's negative assessment of empirical research relates to the successionist model of causation deployed, with its “view of perception as a merely passive process (in which the percipient merely responds to the visual, musical or linguistic stimuli)” (2007, p. 238).

In evaluation these are typically concerned with three issues: did the intervention fail to produce the predicted outcomes, or is it the types of delivery of the intervention which failed, or did defects in the evaluation methods used fail to identify or measure the outcomes? (Rychetnik, Frommer, Hawe, & Shiell, 2002).

The definition used was developed and agreed in partnership with the local Community Arts Council.

These were overall life satisfaction, an aggregate of satisfaction with each of a number of specific quality of life domains, happiness and subjective well-being.

In fact these are some of a range of factors outlined by Matarasso as likely to influence the outcomes of an arts encounter. He also mentions the location and social situation of the art work and the resources available; the aims, agenda and relationships of stakeholders; the personalities, skills and experience of the arts practitioners involved; and the practicalities of timescale etc. (1996, p. 21).

In which a musical stimulus acts on a (passive) listener to produce and emotional, cognitive or behavioural response.

They propose three groupings of determinants of impact: factors intrinsic to individuals, factors intrinsic to the artwork or arts activity, and factors intrinsic to the social context in which the arts encounter takes place (Belfiore & Bennett, 2007).

The latter aims to make explicit the connections between interventions and outcomes, and so provide a plausible explanation of how and why the desired change is expected to come about. Weiss argues that, in practice, ToC has been little used to address causal mechanisms, tending to be concerned with theories of programme implementation, in other words, the links between the intervention activities and outcomes (Weiss, 1997). For the realistic evaluation perspective on types of theory in ToC see Pawson and Tilley (1997, p. 26).

Although the studies are categorized here as examples of TBE, it should be noted that the authors themselves refer sometimes to their methods as realist and, subsequently, the terms are sometimes used interchangeably here.

The Scottish Arts Council's National Lottery Arts in Social Inclusion scheme, which is targeted at Social Inclusion Partnerships (SIPs).

The study was commissioned by Arts Council England, the Department for Culture, Media and Sport and the Offender's Learning and Skills Unit at the Department for Education and Skills and forms part of a research plan initiated by the Research into Arts and Criminal Justice Think Tank (REACTT).

The study is significant first, because it represents the first large scale arts and mental health evaluation to be conducted in the UK, and second, because it applies validated measures of mental health outcomes which have tended not to be used in arts and mental health evaluations (Anglia Ruskin/UCLan, 2005, p. 4).

According to the latter view arts practice is free or separate from context. For an elucidation of the intellectual origins of this division, see Belfiore and Bennett (2006, p. 20).

These theories were selected because, in the author's view, they appeared to have the most scope for applicability to the selected projects within the constraints of the study (Miles & Clarke, 2006, p. 5.1).

For a description of Appreciate Inquiry, a philosophy of affirmative change management, see Kilroy et al. (2007a, p. 20) or Hammond and Royal (2001).

Information provided to the author in a personal interview.

Examples of realist approaches to this question within the social sciences include Ragin (1987).

It is worth noting that, in contrast to Pawson's position, other practitioners of TBE view it as compatible with traditional experimental approaches (see Cook, 2000).

In this group of studies evidence of “hard” outcomes was desired based on assumptions made, usually about vulnerable groups, for example that paid employment, less medication or more “integration” with mainstream society is necessarily beneficial.