Measuring the Social Return on Investment of community sport and leisure facilities

ABSTRACT

Rationale/Purpose: There is a growing demand from managers and policy makers for evidence on the wider impacts of sport and physical activity. This is driven by the need to demonstrate accountability for public expenditure and effectiveness in relation to public policy. The research presented in this paper addresses a gap in knowledge relating to the social impact of local sport and leisure facilities.

Design/methodology/approach: A Social Return on Investment (SROI) framework was used to measure the impact of sport and physical activity across 12 community sport and leisure facilities in Sheffield. A range of methods were used to measure general participation by regular visitors and a targeted therapeutic exercise programme for specific participants.

Findings: The research found the social value of outcomes related to general and targeted participation were £21.67 m and £0.26 m, respectively, and that for every £1 spent a SROI of between £1.20 and £3.42 was generated.

Practical implications: The research enables managers to identify the value of facilities beyond the financial indicators commonly used in performance management.

Contribution: It contributes to knowledge on valuing the non-market benefits of sport. The research provides a methodological example of using SROI to measure the value of local sport and leisure facilities.

KEYWORDS:

• Social impact
• social value
• SROI
• sports participation
• exercise referral scheme

Introduction

In the twenty-first century, there has been a growing demand from policy makers in the sport and leisure sector for evidence in support of the wider social impacts of sport and physical activity (Davies, 2010; King, 2009). This is partly driven by the need for greater public accountability at a time of reducing public expenditure, which helps to defend continuing government subsidies at both national and local authority levels. There is a body of evidence which suggests that sport and physical activity generates net positive benefits for individuals and society (Taylor et al., 2015). The strongest and most robust evidence is in health, where there is consensus that sport and physical activity creates preventative (primary) and therapeutic (secondary) physical and mental health benefits (Warburton et al., 2006, 2007). However, there is also lower grade but significant evidence to suggest that participation enhances individual subjective wellbeing, including life satisfaction and happiness (Downward et al., 2017; Downward & Rascuite, 2011; Fujiwara et al., 2014a); improves pro-social behaviour and reduces crime and anti-social behaviour, particularly for young men (Nevill & Poortvliet, 2011; Nichols, 2007; Vinluan, 2005); promotes bonding social capital and collective action, particularly volunteering (Downward et al., 2013; Tonts, 2005); and has a positive effect on educational outcomes, including psychological and cognitive benefits (Centers for Disease Control and Prevention, 2010; Newman et al., 2010).

In response to the need for evidence to demonstrate social impact, some policy makers have sponsored research on the social impacts of sport and physical activity at the national level (Davies et al., 2019; Rebel Mulier Institute, 2019), and the local level (Baker Tilly, 2013; ICF GHK, 2013; RSM, 2016). This research has focused on assigning a monetary value to various social impacts or “non-market” effects. However, the little research that has been undertaken for community sport and leisure facilities in the UK has been partial, concentrating on selected social impacts and not considering the full costs of operating the facilities. Our study provides a more holistic calculation of the Social Return on Investment (SROI) in sport and physical activity at the local level, focussing on 12 facilities (nine sport and leisure centres; three golf courses) in Sheffield, England. These facilities are owned by Sheffield City Council (SCC), but the management responsibility lies with Sheffield City Trust (SCT), a charitable trust formed in 1988, and its non-profit operating company Sheffield International Venues (SIV).

The first part of the paper briefly describes the important policy context for the research. Relevant academic literature is then reviewed in relation to the social impact of sport, the SROI framework and SROI studies in sport. An explanation of the specific methods employed for this research is then given, and the results of the study are presented. The discussion considers the relative and absolute importance of general sport and physical activity participation by regular visitors to the researched facilities, compared with targeted therapeutic programmes of sport and physical activity for specific participants. Moreover, it considers the management implications and limitations of the research. The paper concludes with an appraisal of the value in using SROI to make the case for investment in an improvement of sport and leisure services.

Policy context

A major policy justification for subsidising sport and physical activity is that many of the claimed social benefits are not paid for through a conventional market system; rather they are outcomes which derive automatically from mass participation by individuals. Theory suggests that the social benefits would be under-provided without government stimulation of sports participation (Gratton & Taylor, 1985). The potential social benefits of sports participation include health care savings from improved mental and physical health, increases in skills and productivity from improved standards in education, reduced criminal justice costs from lower crime and anti-social behaviour, and increased social inclusion (Taylor et al., 2015). Economic theory terms such social benefits as “external benefits” because they are external to normal free market payments by individuals. External benefits are non-rival in that one person benefiting does not prevent others from benefiting; and non-excludable, meaning that it is not possible to exclude any individuals from benefiting. An example is that if sport improves mental and physical health, it prevents at least some participants from suffering from health problems (individual benefits for participants), but also frees health resources to deal with non-preventable problems (external benefits through which anyone can benefit).

In recent years, government sport policy in the UK has tended to focus on the wider social benefits of sport (Davies, 2010; Department for Digital, Culture, Media and Sport [DCMS], 2002; King, 2009). This provides a clear rationale for government subsidisation of sport, helping to stimulate external benefits to society. Since 1997, there has been a general shift in sport policy from a position of funding “sport for sport’s sake” to investment in sport for “societal good” (Davies, 2010; King, 2014). Sport policy under successive Labour governments in the period 1997–2010 clearly emphasised the role of sport in delivering social policy outcomes related to social exclusion, including health, crime and education objectives (DCMS, 2002; King, 2014). Although there was a notable change in direction in 2008 to focus on performance in the run up to the 2012 Olympic Games (DCMS, 2008), national sport policy maintained a strong focus on the objective of increasing participation to improve the health of the nation. The most recent government strategy for sport in England, Sporting Future (HM Government, 2015), clearly repositions sport and physical activity first and foremost as a force for social good. It identifies how sport and physical activity can have a meaningful and measurable impact on people’s lives in relation to five outcomes: physical well-being, mental well-being, individual development, social and community development and economic development. These outcomes are at the heart of Sport England's strategy, Towards an Active Nation (Sport England, 2016). Sport England is the agency tasked with development of grassroots sport in England. Local government sports policy has tended to mirror national government policy. Local authorities have provided subsidies for sports facilities throughout the last century, and particularly since the 1970s. The main objective has been to increase sports participation for a variety of reasons, including the promotion of external benefits (Coalter, 2007; Gratton & Taylor, 1985; King, 2014).

The objectives of SCT include providing facilities for recreation for the benefit of the residents of Sheffield; promoting the physical and mental health of Sheffield residents; and the promotion of social welfare in the city. Since SCT was formed, it has received significant subsidies from SCC to run most of SCC's sports facilities. However, as a result of the national government's austerity policies, and particularly the impact of 40% cuts in grants from national government to local authorities, the subsidy from SCC to SCT has reduced significantly, from over £6 million in 2011/2012 to just over £1.5 million in 2017/2018 (financial information from SCT accounts). The financial pressure led to an understandable emphasis on financial performance by SCC and SCT. Nevertheless, trustees did not lose sight of their social objectives and in 2015 commissioned an SROI study for 12 of its community sport and leisure facilities – the research presented in this paper. They were aware that in attempting to defend the subsidies from SCC, they needed to provide evidence of the social return from their facilities. They were also aware that there was little evidence relating to their non-financial performance and none relating to social impacts such as health and wellbeing.

The lack of evidence relating to social objectives in public sector subsidised sport in the UK has drawn much criticism over the last thirty years. In 1989 a seminal report by a government watchdog, the Audit Commission, called for better evidence of why local authorities subsidised sport (Audit Commission, 1989). An independent review commissioned by the government called for a better evidence base on which to build further public investment in sport (Carter, 2005). Other studies have criticised the evidence base for subsidised public sector sport (e.g. Coalter, 2007).

King (2014) argues that the case for public sport and recreational services in many local authorities is perceived to be weak due to difficulties in measuring the value of its services. The lack of relevant evidence is a critical weakness in an era of increasing scrutiny and accountability in public sector funding. It is particularly damaging in light of the Public Services (Social Value) Act, 2012. This requires all public sector procurement to have regard for social, economic and environmental benefits from contracted services, in order to increase value for money from these services. Because SCT is responsible for running 12 of SCC's sports facilities until 2024, it would be entirely appropriate to expect SCC to consider the performance of SCT with respect to social objectives and outcomes. Therefore, SCT's initiative in commissioning an SROI study of the researched facilities is very relevant to recent government thinking in public services. The purpose of the research presented is to measure and value the social impacts of sport and physical activity which takes place across the 12 sport and leisure facilities operated by SCT.

Review of literature

Social impact of sport

Alongside the growing recognition and use of sport as a policy tool for achieving wider social benefits, there has been a resurgence in academic and evaluative research on the social impacts of sport. There is a long history of such research in this area, with studies on the social benefits of outdoor recreation in the USA emerging in the 1960s, and on leisure and quality of life in the UK from the 1970s onwards. This includes evidence relating to both individual outcomes, such as improved personal physical and mental health, and community outcomes such as improved community cohesion and trust. Nevertheless, until recently there has been little consensus on whether there is sufficient robust evidence to support the case for investment in sport (Coalter, 2007; Davies et al., 2019; Taylor et al., 2015).

There are large-scale literatures on various social outcomes including health, crime, education, social capital and subjective wellbeing, which debate and discuss both the range of meanings that lie beneath these concepts as well as the impact of sport and physical activity on each of these domains. It is beyond the scope of this paper to review these bodies of evidence in detail. Furthermore, they are synthesised by others elsewhere, most recently by Taylor et al. (2015), who published a systematic review of literature on the social impacts of sport, based on 240 references selected from a search strategy which yielded 16,807 hits. However, it is important to critically summarise current knowledge on the social impact of sport to provide a context and rationale for the research presented, and because this evidence is used in the research to identify the population level change in social outcomes which occurs as a result of engagement in sport and physical activity in the researched facilities.

Taylor et al. (2015) argue that sport may have “turned a corner” from being criticised as under-researched to one of noteworthy evidence in several areas. The most scientific and robust evidence relates to health, where there is consensus that participation in sport and physical activity generates population-wide primary (preventative) and secondary (therapeutic) physical and mental health benefits. This includes the prevention and treatment of chronic diseases including, but not limited to, cardiovascular disease, strokes, diabetes, obesity, some cancers, various neurological conditions and clinical depression (Allender et al., 2007; Kumar et al., 2018; Warburton et al., 2006, 2007). The dose response relationship is complex and varies by disease (Davies et al., 2019). Nevertheless, there is consensus that regular moderate intensity exercise is recommended for benefits to be realised (Department of Health and Social Care, 2019; U.S. Department of Health and Human Services, 2018). The evidence suggests that there are also negative effects such as sports injuries, although the research is more variable and generally based on cross-sectional data (Allison, 1999; Oughton & Tacon, 2007; Walsh, 2011). This contrasts with most health-related evidence, which is synthesised from high-quality studies including meta-analyses and systematic reviews, randomised control trials (RCT), cohort studies and case controls. The research design of these studies enables the causal relationship between sports participation and health to be clearly established.

There is also significant evidence that engagement in sport and physical activity through participation and volunteering can have a positive effect on the subjective wellbeing of individuals. Research suggests that active people are generally more satisfied and happier than those that are physically inactive (Downward & Rascuite, 2011; Fujiwara et al., 2014a; Kumar et al., 2018; Rasciute & Downward, 2010). While this evidence tends to be based on analysis of large secondary cross-sectional datasets, regression analysis is commonly used with an instrumental variable approach to establish causal evidence of a relationship between sport and subjective well-being (Fujiwara et al., 2014a; Ruseski et al., 2014; Wicker & Downward, 2019). In the absence of experimental data, this is the best approach and generally more rigorous than many previous studies, which make no attempt to control for confounding factors (Fujiwara et al., 2014b).

In the other areas of social impact, including education, crime and social capital, there is sufficient evidence, albeit of a lower quality, to suggest that sport and physical activity has net positive effects. The literatures in these areas tend to be more heterogeneous, focusing on different contexts and subgroups, and employing a wide range of research methods. Taken together, the evidence in each of these areas may not be as robust as in health. However, when judged against the hierarchy of evidence (Sackett et al., 1996), there are some examples of higher quality studies including systematic reviews, RCTs and cohort studies (Taylor et al., 2015).

In the education literature, there is a “weight” of evidence (a large number of studies reporting similar findings), which suggests that sport can have a positive effect on young people. The literature suggests a positive relationship between sports participation and education intermediate outcomes (such as behaviour and attendance) and final outcomes (such as attainment and progression) (Marsh & Kleitman, 2003; Martin, 2010; Taylor et al., 2015). It identifies three pathways by which sports participation translates to educational benefits – psychological/affective benefits, cognitive benefits and social capital effects (Centers for Disease Control and Prevention, 2010). There are a few contrasting studies that identify negative aspects for specific groups (e.g. Eitle & Eitle, 2002), although by and large, the net effects of sport and physical activity on education outcomes are considered to be positive (Davies et al., 2019).

In crime literature, similarly, there is a weight of evidence pointing to the positive effects of sport in terms of promoting prosocial and reducing antisocial behaviour, in particular for young males (Carreres-Ponsoda et al., 2012; Howie et al., 2010; Nichols, 2007). Nevertheless, some negative effects are reported such as illegal alcohol consumption by young people and in certain sports, violent behaviour (Endresen & Olweus, 2005; Terry-McElrath & O'Malley, 2011). There is a strong focus on programmes designed to prevent future offending by young people generally, together with other mitigating factors. Relatively fewer studies examine sports programmes specifically designed for delinquents and/or at-risk youth (Meek, 2018). At the heart of both types of programme is a model of social learning, through which antisocial and prosocial behaviours are moderated by risk factors, protective factors and interactions with others (Witt & Caldwell, 2010).

In the case of social capital, which includes networks and relationships together with trust and reciprocity (Putnam, 1993, 2000), there is evidence that participation and volunteering can enhance bonding capital (between groups sharing similar characteristics and values) and bridging capital (reaching across groups with different characteristics and values) (Downward et al., 2013; Kumar et al., 2018). However, there is a minority of contrasting studies that also identify negative effects of sports participation such as reinforcing social exclusion (Tonts, 2005).

Social Return on Investment framework

Collectively, there is clear evidence of a relationship between sport and physical activity and multiple social outcomes. However, there is considerably less evidence on quantifying the monetary value of sport and active recreation at the population and sub-population level. This makes it difficult for stakeholders, including the managers of sport and recreation facilities, to demonstrate social and financial value, and make the case for investment (King, 2014). Keane et al. (2019) provide a comprehensive review and critical analysis of previous and current methods for assessing value creation in sport and note that a plethora of conflicting methods have been used. They identify SROI as one of several methods which has been adapted and applied to both population and sub-population level domains and conclude that the social impacts of sport and active recreation are highly suited to being measured and valued using this approach.

SROI is a framework used for understanding, measuring and valuing the net social impacts of an activity, organisation or intervention (Nicholls et al., 2012). It was developed from Cost–Benefit Analysis (CBA), together with sustainability accounting and financial accounting (Social Value UK, 2014). SROI offers a framework for social impact valuation which is guided by several clear principles – identified in the methodology section of this paper below. It is built on a theory of change in the form of a logic model, which is developed in consultation with stakeholders. Inputs, outputs and outcomes are measured using a range of methods, which depend on the data being utilised and the judgement of the researchers as to what methods most accurately identify the quantities being estimated. Valuation techniques for outcomes include preference and well-being methods from CBA and financial metrics used in accounting (Fujiwara, 2014). SROI expresses the value of social outcomes in relation to the cost of achieving them.

SROI is a transparent and conservative approach to social impact measurement, which only includes material outcomes, namely those that if excluded would affect the decisions of stakeholders. Stakeholders’ involvement is a fundamental principle of the SROI approach, enabling the measurement of outcomes that matter to the people affected by the intervention. Davies et al. (2019) note how SROI can be applied in a wide range of contexts at any level of rigour. It has an external quality assurance process, which verifies the measurement process at higher levels of rigour if required. Furthermore, SROI uses money as a common metric, which enables multiple inputs and outcomes to be included and a single monetary ratio to be created that is easy to understand. However, the SROI approach is not without it challenges. Banke-Thomas et al. (2015) comment on the difficulty of attaching monetary values to non-market goods and establishing the counterfactual. Moreover, Fujiwara (2014) notes the lack of guidance in valuing long-term outcomes. However, these concerns are not unique to SROI and are often shared by other approaches which seek to value non-market goods. A fuller discussion of the strengths and weaknesses of SROI can be found elsewhere (e.g. Banke-Thomas et al., 2015; Davies et al., 2019; Fujiwara, 2014; Keane et al., 2019; Vardakoulias, 2013).

SROI measurement in sport

SROI is used globally across a range of policy areas, especially by public agencies and third sector organisations to measure social impacts, understand where value is created and justify investment (Fujiwara, 2014; Millar & Hall, 2013). In the UK, it is also increasingly used by local government and the private sector. However, SROI is a relatively new method for social impact measurement in the sport and physical activity sector, growing in prominence since 2010.

At the population level, SROI has recently been used to measure the value of the sports sector in England (Davies et al., 2019), Wales (Sport Wales, 2019), and Greater London (London Sport, 2020). All studies adopt the method developed for the seminal study in England. They demonstrate that sport creates value to society across multiple social outcomes, with the highest value for subjective wellbeing, accounting for more than two-thirds of the overall social value generated in most studies. The SROI studies demonstrate that sport generates a positive and substantial return on investment to society. Sport Wales (2019) also found that the social value of sport was nearly three times greater than economic value, as measured by Gross Value Added, highlighting the importance of capturing social impact when making the case for sport. While these studies represent the most comprehensive estimates of the wider social impacts of sport to date, to maintain a high level of academic rigour they only include those outcomes sufficiently evidenced at the population level. Moreover, they only measure general participation rather than targeted interventions of sport and physical activity. Therefore, they are only likely to represent a conservative estimate of the value of sport (Davies et al., 2019). Following the development of this method in England, other European countries have similarly used SROI to measure the national value of sport (Rebel Mulier Institute, 2019).

SROI is used more widely at the local (sub-population) level to measure the value of sport and physical activity interventions (e.g. Black Country Consortium, 2016; Butler & Leathem, 2014; Charlton, 2014; Hopkinson, 2016; ICF GHK, 2013; Lombardo et al., 2019; Ozgun, 2016). Several Charitable Trusts with responsibility for sport and leisure facilities have commissioned studies to evaluate their activities (e.g. Baker Tilly, 2010, 2012, 2013, 2014; RSM, 2016). However, these studies have focused on selected projects and programmes, rather than general participation across the facilities, resulting in a partial rather than a holistic valuation of the organisations’ activities. Furthermore, they do not estimate full costs of these activities, therefore they are social value studies, not SROI studies. Even so, these studies provide value to the organisations commissioning them. In an increasingly challenging financial and political context where there are ongoing reductions in local government finance, King (2014) highlights the importance of sport and recreation services being able to demonstrate either a business case or social impact and value to avoid rapid decline in the future.

Methodology

The social impact of the 12 researched facilities was measured using an SROI framework. The application of an SROI framework requires judgement throughout the research process. In order to guide key decisions and assumptions and ensure that the researchers remained unbiased our study adhered to the seven principles of SROI, which are: involve stakeholders; understand what changes (from stakeholders and relevant literature); value things that matter (to stakeholders); only include what is material (in terms of scale and impact for stakeholders); do not overclaim (e.g. when faced with a variety of estimates, select a more conservative one); be transparent (about limitations and assumptions); and verify the result (Nicholls et al., 2012). To minimise bias, stakeholders were involved in consultation throughout the analysis, in order that the value and the way that it is measured was informed by those affected by, or who affect, sport and physical activity.

Carrying out an SROI involves six stages, which are presented in Figure 1. These stages provide the structure for documenting the measurement process and presenting the results in the next section. The research presented in this paper is evaluative, measuring the SROI of sport and physical activity that took place in the researched facilities during 2014/2015.

Figure 1. Stages of SROI.

Research design

There were two parts to the research; both utilised an SROI methodology and involved the six stages outlined in Figure 1.

Part 1 measured the SROI arising from general participation in sport and physical activity across the 12 researched facilities in Sheffield, listed in Table 1. Part 2 measured the SROI of the Physical Activity Referral Scheme (PARS), a targeted intervention for people with various medical conditions, operating in six facilities (also listed in Table 1). The PARS was launched in 2005 and over 15,000 people have gone through the scheme to date. Participants of the scheme are referred by their GP (General Practitioner) or health care professional. They undergo a 12-week personal exercise programme under the guidance of an exercise referral instructor, which comprises a gym-based exercise programme and five personal consultations throughout the duration of the programme. Measurement of both general participation and participation through the targeted intervention were considered important for measuring the overall impact of community sport and physical activity in the researched facilities.

Table 1. Facilities included in the SROI.

Facility	Type of facility	PARS operating at venue
Ponds Forge	Mixed use sports centre	✓
Hillsborough Leisure Centre	Mixed use sports centre	✓
Graves Tennis & Leisure Centre	Mixed use sports centre	✓
Concord Sports Centre	Mixed use sports centre	✓
Westfield Sports Centre	Mixed use sports centre
Springs Leisure Centre	Mixed use sports centre	✓
EIS Sheffield	Dry use sports centre
Heeley pool	Swimming pool and gym	✓
iceSheffield	Ice rink
Birley Wood Golf Course	Golf course
Beauchief Golf Course	Golf course
Tinsley Park Golf Course	Golf course

For Part 1, we adapted the national SROI model for participation in sport and physical activity in England using local data for Sheffield (Davies et al., 2019). The national SROI model utilises substantial evidence on the social impact of sport and physical activity to quantify the relationship between sport and various outcomes, including health, crime, education and subjective wellbeing (Taylor et al., 2015).

For Part 2, we designed a bespoke SROI of PARS using the conventional bottom-up approach adopted in SROI studies (Nicholls et al., 2012). This part of the research was designed to tell the story of SCT’s targeted actions including the rehabilitation benefits of sport and physical activity for those suffering from ill health. The impact of targeted interventions is not captured in the work of Davies et al. (2019). However, it was considered important for a complete understanding of the health and wellbeing objectives of SCT, hence included in this research.

Data collection

After establishing the scope of the research, the first stage of an SROI is to identify the stakeholders to be included. In the context of this research, stakeholders are defined as people or organisations that influence or experience change because of participation in sport and physical activity in the researched facilities. It was not possible to include all stakeholders, so based on the principles of SROI only those that experienced material change were included. Stakeholders included in this research were SCT, SIV, SCC and participants.

Data from stakeholders were collected using a range of mixed methods, including a postal survey, interviews and focus groups. Secondary data from a range of organisations were also utilised to derive values for inputs, outputs and outcomes, including financial proxies. The data collected for each stage of the SROI are summarised in Table 2. For Part 1 of the research, fewer stakeholder interviews were considered necessary because the outcomes for participants were identified using the scientific evidence which underpins the national model (Taylor et al., 2015). In addition to the data collection outlined in Table 2, a follow-up interview with the Health and Wellbeing Manager at SIV was also carried out in 2019 to investigate the management actions taken as a result of the SROI research, and the responses to the results by SIV and other stakeholders.

Table 2. Summary of data collection for the SROI.

Stages of SROI	Data collection
	Part 1	Part 2
Stakeholders	Interviews (n = 4) with representatives of SCC, SCT & SIV	Interviews (n = 6) with representatives of SIV and PARS staff Focus groups (n = 3) with PARS participants SROI PARS (postal) survey of participants(n = 152)
Inputs	Interviews (n = 4) with representatives of SCC, SCT & SIV SIV Management Accounts National data on consumer spending sport and physical activity National Benchmarking Service (NBS) data SIV facility participation data	Interviews (n = 4) with representatives of SCC, SCT & SIV SIV Management accounts
Outputs	National Benchmarking Service (NBS) data SIV facility participation data	SIV PARS data
Outcomes	National Benchmarking Service (NBS) data SIV facility participation data NHS England data Literature findings as reported in Taylor et al. (2015), Davies et al. (2019) Various secondary data sources for health care costs (Cancer Research; British Heart Foundation; Dementia UK; Diabetes UK)	SROI PARS (postal) survey (n = 152) NHS data for health care costs

Part 1

For Part 1 of the research, the inputs (that stakeholders contribute to stimulate outputs and outcomes), were extracted from the management company, SIV’s accounts and the interviews. Consumer spending inputs were derived using national data on sport-related consumer spend (Sport England, 2013) multiplied by the number of unique participants using the SIV facilities. Unique participants were derived using data from National Benchmarking Service (NBS) customer surveys at six of the researched facilities¹ and participation (throughput) data from SIV’s attendance monitoring system. The SIV throughput data was available in the form of total annual visits to these facilities, therefore an adjustment was made to convert attendance into unique participants using the NBS data relating to frequency of usage, collected previously at a sample of the researched facilities. Similarly, output data (number of participants) was derived using both SIV participation data and the NBS survey data.

SROI is an outcomes-based measurement tool, as measuring outcomes is the only way to be sure that changes for stakeholders are taking place. For the general participation SROI, the outcomes were those identified in the national model, with the exception of volunteering, as the researched facilities have no direct volunteers (these are associated with clubs). There were 10 social outcomes included in Part 1: six health-related outcomes, two education-related outcomes, one related to crime and antisocial behaviour, and another to subjective wellbeing. Further details of the underpinning evidence and assumptions used to support inclusion of these outcomes, and the valuation methods, can be found in Davies et al. (2019).

Outcomes data for Part 1 were generated from a mix of the output data on participant numbers; estimates of the extent of different social benefits from the literature evidence (Davies et al., 2019); and the value of these benefits from various public and third sector sources – mainly concerning the prevalence rates of relevant social problems and the costs of treating them. For health and crime benefits the financial proxies are represented by the saved treatment costs brought about by lower prevalence through sports participation. For enhanced education achievements, enhanced human capital and subjective wellbeing from sports participation, the proxies are provided by specific academic references (Fujiwara et al., 2014a; Griffiths et al., 2017; Hayward et al., 2014).

Part 2

For Part 2 of the research, input data were derived from the SIV (PARS) management accounts and the interviews. Output data (number of participants registered) was obtained from the SIV PARS data.

Outcomes were initially identified through interviews with SIV and PARS staff and focus groups with PARS participants. The qualitative findings were used to inform the SROI PARS survey, which was designed to measure and value outcomes for a larger sample of participants. The SROI PARS survey was implemented by post and included questions on change experienced by participants, attribution and valuation. The SROI PARS survey sample was 152, which was 16% of those questioned (974²). The majority of the sample was composed of individuals who had completed the scheme (110 respondents).

Personal benefits to individuals (e.g. improved physical health, mental health, confidence, happiness and relationships with others), were identified and measured using a five-point self-reported Likert scale. They were valued using a willingness-to-pay (WTP) approach. WTP is an established valuation approach in economics used for estimating the amount that individuals are prepared to pay to secure a good or avoid something undesirable (such as poor health) (Orlowski & Wicker, 2019). Benefits to community (health care savings) were identified and measured using the method developed by Fujiwara et al. (2014b) for measuring non-market effects. Financial proxies are represented by the saved costs of reduced medical service usage.

Assumptions

As with all SROI studies, a series of assumptions underpin the research presented. By using the national model for Part 1 of the research, it was assumed that sport and physical activity generates similar social outcomes in Sheffield to those experienced by the national population in England. The assumptions underpinning the national model and therefore the general participation SROI model (Part 1) are discussed elsewhere and are not repeated here (Davies et al., 2019).

It is conventional within SROI studies to measure impact over several years. The application of the framework to a sector such as sport and physical activity requires a different approach. The inputs which generate sport and physical activity participation, from which social impacts are derived, are largely continuing operating costs for the researched facilities and services which are repeated each year, i.e. a continual input process, rather than a one-off investment which yields returns over a period of future years. We, therefore, considered an annual estimate the most appropriate method for calculating the value of social impacts from sport undertaken in researched facilities: i.e. comparison of a year's input with the value of the social impact generated by a year's participation in sport. This is essentially a simplifying assumption which renders SROI techniques of discounting and estimating drop-off unnecessary.

Results

This section presents the findings from the general participation SROI followed by the PARS SROI.

Part 1: general participation SROI

The Impact Map is central to an SROI. It is the story of how an activity, intervention or policy makes a difference. The logic of the model is that stakeholders provide resources (inputs) to deliver sport and physical activity (outputs), which result in outcomes for stakeholders. Figure 2 gives an overview of the Impact Map for general participation in the researched facilities. The evidence in support of the links and causality between inputs, outputs and outcomes is in the extensive literature reviewed by Taylor et al., 2015, plus subsequent literature updating (Davies et al., 2019).

Figure 2. Impact Map: Part 1 – general participation.

Table 3 summarises the inputs required to facilitate participation at the researched facilities. The inputs were relatively straightforward to identify, although care was needed to ensure there was no double counting. Total inputs were £18.01m, of which the majority (£15.12m) was generated from consumer spending on activity charges, equipment, sports clothing, footwear, travel and other costs. The remaining inputs (£2.89m) were from a subsidy provided by SCC.

Table 3. Summary of inputs.

Stakeholder	Inputs	Value (£)
SCC	SCC funding	2,671,648
	SCC rates paid	215,695
Sub total		2,887,343
Sports Participants	Consumer spending at SCT/SIV on activity charges	12,487,807
	Consumer spending on equipment costs	1,017,642
	Consumer spending on sport clothing and footwear	1,287,732
	Consumer spending on travel and other costs	329,286
Sub total		15,122,467
Total		£18,009,810

Outputs are a quantitative summary of an activity. In this research, the primary output was participation in sport and physical activity at the researched facilities. The indicator used was “participating in sport at least once per week”, which was consistent with the threshold used in the national SROI model (Davies et al., 2019). Using the approach outlined in the methodology, the total number of unique participants meeting the participation threshold of at least once per week was estimated at 16,106.

Table 4 summarises the value of the social outcomes related to participation in sport and physical activity at the researched facilities. The “quantity” column in Table 4 relates to the amount of change associated with each type of outcome (e.g. the number of cases of ill-health or criminal incidents prevented) and the “value” column shows the financial proxies used to measure the “impact”, the corresponding outcomes in monetary terms. The largest impact was associated with subjective wellbeing, totalling £15.85m. This is approximately 73% of the estimated social impact of general participation in the researched facilities. The “external” impacts through which the rest of society is estimated to benefit as a result of sport make up the other 27% of social impacts – i.e. related to improved health, reduced crime and anti-social behaviour and improved education outcomes. The impact of health outcomes, which collectively were the second largest contributor, was £4.70m. As shown in Table 4, the largest health impacts were created by reduced incidences of dementia and self-reported good health – the former largely because of a high cost saving per person (£32,887), the latter because of the higher numbers affected (13,688 sports participants). In comparison, the values attributed to the education-related outcomes (£1.09m) and crime (£0.03m) are considerably smaller than the other two outcome areas. In part, this is because of the very conservative assumptions made about the impact of sport and physical activity on these outcomes, due to the varied evidence in these areas.

Table 4. Social outcomes valuation summary: Part 1.

Outcomes	Quantity	Value (£)	Impact (£)
Health
Reduced risk of CHD/stroke in active men and women by 30%	251	3635	912,959
Reduced risk of breast cancer in active women by 20%	3	47,908	154,862
Reduced risk of developing colon cancer by 24%	1.5	47,908	71,941
10% reduction in incidence of type 2 diabetes	61	3545	215,225
Reduced incidence of dementia by 30%	60	32,887	1,979,329
Sports participants are 14.1% more likely to report good health than non-participants^a	13,688	100	1,364,266
Subtotal			4,698,583
Crime & antisocial behaviour
Sports participation leads to a 1% reduction in criminal incidents for males aged 10–24 years	18	1590	28,649
Education
Sports participation leads to a 1% increase in educational attainments (aged 11–18)	0.6	1221	687
Graduates who participate in sport at university have a higher starting salary than their non-sporting counterparts (enhanced value of human capital)	186	5824	1,086,058
Subtotal			£1,086,745
Subjective wellbeing
Sports participation (in the last year) is found to be associated with higher subjective wellbeing	14,065	1127	15,851,255
Total value of all outcomes			£21,665,233

^aExcludes participants already accounted for in other health outcomes.

The SROI value was calculated by dividing the total value of the benefits/outcomes by the total costs/inputs. Total inputs were £18.01 million, while the total value of outcomes was £21.67 million. This gives a SROI of 1.20 – i.e. for every £1 invested in sport and physical activity in 2014/2015 in the researched facilities, £1.20 worth of social impact was generated. Given the importance of the SCC subsidy to the finances of the facilities, a community SROI was also calculated, i.e. the external benefits from health, education and crime divided by the SCC subsidy. This community SROI was 2.01 – for every £1 of subsidy, £2.01 worth of external, community benefits were generated.

One of the key components of SROI is to test the sensitivity of the SROI ratio to variations in data used (e.g. outcome measures, financial proxies). Using more conservative assumptions for the health, education and crime outcomes while keeping the subjective wellbeing outcome constant, gave an SROI ratio of 1.13. Using more ambitious values for health, crime and education increased the SROI ratio to 1.29. This indicates that the SROI is not overly sensitive to variations in the key assumptions for the health, crime and education outcomes under consideration. On the other hand, adopting higher values for improved wellbeing found in the literature (Marsh et al., 2010) increased the ratio to 18.28. This confirmed the validity of the decision to use the more conservative values derived from the wellbeing valuation approach (Fujiwara et al., 2014a).

Part 2: PARS SROI

As with the general participation SROI, an Impact Map of PARS was created to identify what changed for stakeholders. The Impact Map for PARS is shown in Figure 3. The primary beneficiaries of PARS were the participants. The links between inputs, outputs and outcomes for PARS is evidenced by the qualitative responses of participants in focus groups and the SROI PARS survey results (see below), and also supported by the responses of PARS staff in interviews.

Figure 3. Impact Map: Part 2 – PARS.

The total cost inputs for the PARS in 2014/2015 were £76,024. This was largely staff costs for exercise referral instructors and administrators. It includes some central costs but excludes facility costs.

The primary output was participation in the PARS. In total there were 1224 participants registered across six of the researched facilities and two GP practices in Sheffield in 2014/2015. 574 fully completed the programme and 650 were non-completers, giving an adherence rate of 47%.

Individual outcomes and values

Figure 4 summarises the main outcome changes experienced by participants, reported in the SROI PARS survey. In terms of net changes for the group, (i.e. those reporting better outcomes minus those reporting worse), the largest change was for physical health. There was a net increase of 45% in the percentage of people reporting an improvement (i.e. 59% minus 14%), followed by confidence (39%), happiness (36%), mental health (33%) and relationships with others (25%).

Figure 4. Outcome changes experienced by PARS participants.

The findings were supported by qualitative comments made by participants in interviews, focus groups and the PARS survey. A selection of the comments from the survey are included in Figure 5. They provide qualitative endorsement of the outcomes in the Impact Map of Figure 3. Such qualitative comments do not, however, contribute to the quantitative estimates of outcomes and values.

Figure 5. Individual benefits expressed by PARS participants.

Many participants that completed PARS commented on the social aspects of the programme. The following quote reflects the strong social connectedness expressed by participants:

I formed lasting friendships through the scheme. It generates a great sense amongst patients of togetherness. Also, there is a collective wish to help each other and improve fitness … 

Several participants also reported how their participation in PARS had impacted on others. One PARS respondent explained:

By creating a supportive and safe framework for me to find out what I was capable of, it made those around me worry less about my rehabilitation.

As discussed in the methodology, individual outcomes were valued using WTP. PARS participants were asked to value the overall benefit of the PARS as it was considered too difficult for participants to value each outcome separately. The average WTP per person per month was £27.47 per month (£329.64 per year). Participants who completed the 12-week PARS were willing to pay slightly more (£29.76 v £22.50 for non-completers).

In calculating the overall value of individual outcomes, participants were asked to identify the extent to which the outcomes they identified were due to their participation in the scheme, rather than other factors which may have contributed to the outcomes which changed for them. From this attribution, rates were calculated, which are presented alongside other pertinent data in Table 5. The overall net annual value attributable for individual outcomes was £139,807.

Table 5. Value of individual outcomes: Part 2.

	Completers	Non-completers	Overall
Total number of people on scheme	574	650	1224
Any improvement(s) post scheme	83%	34%	57%
Beneficiaries	478	222	700
WTP per month^a	£29.76	£22.50	£27.47
Annualised value of benefits	£170,875	£59,927	£230,802
Attribution rate	62%	55%	61%
Net annual value attributable to scheme	£106,744	£33,063	£139,807

^aRounded to nearest 1p.

Health care cost savings

In addition to the individual benefits identified above, the PARS creates benefits to the community in the form of healthcare cost savings. According to research carried out on behalf of the DCMS (Fujiwara et al., 2014b), people who rate themselves as four or five (out of five) for self-reported health (where 5 = “excellent” and 1 = “very poor”) are 35% less likely to visit GPs regularly (six or more times per year). Moreover, they are less likely to incur medical costs elsewhere such as medication and in-patient treatment.

From the SROI PARS survey, just under half of participants reported better physical and/or mental health from participating in the programme, and 42% of these rated their health as four or five (out of five). Therefore, based on the evidence from Fujiwara et al. (2014b) study about the reduced likelihood of this cohort visiting the GP regularly, out of 1,224 PARS participants, 255 were estimated to have created healthcare cost savings (1,224 × 49.7% x 42% = 255). The financial proxy for healthcare cost saving was calculated by multiplying the average NHS cost per person by the percentage reduction in medical service usage reported by people in good health (i.e. £2069 × 35% = £724 per year).

Similarly, as with calculating the overall value of individual outcomes, participants were asked to identify the extent to which the outcomes they identified were due to their participation in the scheme. As shown in Table 6, the overall net annual value attributable for health care cost savings was £120,523.

Table 6. Value of healthcare cost savings: Part 2.

	Completers	Non-completers	Overall
Total number of people on scheme	574	650	1224
Better physical and/or mental health	77%	26%	50%
Total number of people reporting better health	441	171	612
People with better health scoring 80+ for health state (out of 100)	47%	29%	42%
Health beneficiaries	206	49	255
Annual NHS cost savings per person associated with self-reported good health	£724.15	£724.15	£724.15
Indicative cost savings per year	£149,074	£35,391	£184,465
Attribution rate^a	67%	60%	65%
Net cost savings attributable to scheme	£99,289	£21,234	£120,523

^aWeighted average attribution across two health outcomes.

Table 7 summarises the total value of the social outcomes attributed to PARS. The combined value is £260,330 and the average value per participant is £213. While the value per person completing PARS was more than four times the value per non-completer, as shown, there was still benefit gained by those that started but did not complete PARS.

Table 7. Social outcomes valuation summary: Part 2.

	Completers	Non-completers	Overall
Number of participants	574	650	1224
Individual benefits	£106,744	£33,063	£139,807
Healthcare cost savings	£99,289	£21,234	£120,523
Total value	£206,033	£54,298	£260,330
Value per participant	£359	£84	£213

Dividing the value of the outcomes by the inputs of PARS (£76,024) gives an SROI ratio of 3.42:1. For every £1 invested in the SIV PARS, £3.42 worth of social benefit was generated.

Discussion

In comparing the two parts of the SROI, there is a contrast between the comparative scales of the social benefits created and the comparative SROI. In terms of scale, general participation in sport and physical activity across the researched facilities creates £21,665,233 worth of social benefit, whilst participation in PARS creates £260,330. The latter is just over one per cent of the scale of benefit of the former. However, in terms of SROI, the general participation achieves an SROI of 1.2, whilst participation in PARS achieves an SROI of 3.42, so the return on investment of PARS is nearly three times that of general participation at the researched facilities. These findings demonstrate that most of the social value in the researched facilities is generated from mass participation. They suggest that other organisations that have commissioned studies to evaluate their activities, which have focused on selected projects and programmes (Baker Tilly, 2010, 2012, 2013, 2014; RSM, 2016), are likely to have underestimated the value of their activities by some margin. The findings also suggest that although targeted interventions are important to certain sub-groups of the populations and can produce a higher return on investment than general participation, inclusion of interventions such as GP referral schemes into national SROI studies are unlikely to make a material difference to overall value (Davies et al., 2019).

The research revealed that PARS generates more conspicuously life-changing benefits for many of its participants than mass participation, and it targets people with health problems who might not of their own volition engage in such a programme of physical activity. The SROI identifies quantitatively relevant non-market benefits which are always in danger of being underprovided, because the market only caters for private benefits (Gratton & Taylor, 1985). What it does not demonstrate, however, is the qualitative difference that a service such as PARS makes, both in terms of the social need for the provision of such opportunities to people with health problems and in terms of the life-changing effects of the programme. This highlights the importance of qualitatively documenting stakeholders’ stories of change, as part of the measurement process, as in Figure 5 (Nicholls et al., 2012).

SROI guidance explicitly advises against making comparisons between the social return estimates of different studies, largely because of the different methods and data sources employed within the SROI framework (Nicholls et al., 2012). Comparisons are particularly invalid between this study's findings and those of facility studies identified in the literature review, because the latter only valued specific activities and did not calculate an SROI. Nevertheless, the results from Part 1 of this study can be compared with the results of the national study (Davies et al., 2019), since the former are adapted from the latter. The Part 1 SROI in this study is estimated at 1.20, compared with the equivalent national finding of 1.91. The difference is possibly linked to the higher costs of participating inside facilities (rather than more general participation which includes informal, outside activity); and also the lack of social value from volunteers in the current study, whereas for the national study this value is significant.

Management implications and responses

The research presented in this paper has various management implications for the SCT group. It is widely recognised that a key strength of SROI is that it can be used for organisational learning, by enabling staff to improve their services (Arvidson et al., 2013; King, 2014; Millar & Hall, 2013; Nicholls et al., 2012). The Health and Wellbeing (HWB) Partnerships Manager at SIV confirmed that to date, the most productive use of the SROI findings has been internal within SIV. The SROI findings have helped to legitimise the charitable identity and objectives to SIV staff at all levels, which has influenced positively their attitude to users relevant to social outcomes, e.g. from disadvantaged groups.

The SROI study has also enabled SCT trustees and SIV management to identify the key success factors for the SROI at the researched facilities. In the case of the participation-wide results, key factors are the participation numbers and the frequency of participation (SROI relies on the number of regular participants). Happily, these factors fit with an increasingly sales-driven culture at SIV, particularly for memberships, and efforts to improve retention of members. The drive for more members also fits with SCC's ambition to become the most active city in the UK.

Regarding PARS, the critical success factors are the number of referrals and the completion rates of PARS participants. According to the HWB Partnerships Manager, the Clinical Commissioning Group (CCG) has reacted positively to the SROI results and is showing confidence that PARS is a beneficial programme for GPs to refer patients to. The SROI for PARS would increase substantially if the proportion of those registering who complete the programme was increased from 47%, particularly for females because their completion rate is significantly less than that of males. SIV management have used the SROI evidence to redesign the induction process to PARS – in particular to re-orientate the first session away from detailed and lengthy questioning about health and activity, which was off-putting to new, anxious referrals, and towards encouraging a more social, interactive beginning which concentrates on participants’ needs. Another success factor for PARS is the quality of the instructors and it is revealing that some of the case studies in the research featured instructors who started as participants in PARS. This is a commendable feature of the scheme but the HWB Partnerships Manager identified a low turnover of such instructors because they are very pleased to be progressing in this manner, so there are limited future instructor vacancies on the programme.

Externally, the SROI provides the SCT group with evidence to strengthen the financial and social case for investment with stakeholders. The failure to achieve a break-even budget for the SCT group has led to criticism within SCC and in the local press. However, none of this criticism has acknowledged the compensatory social return from local authority subsidies. While various stakeholders in the City such as the Director of Public Health, the CCG and the National Centre for Sport and Exercise Medicine have reacted to the SROI findings positively, they have no power over SCT finances. SCC have drawn up their own Social Outcomes Framework, with associated key performance indicators, that does not currently draw on the SROI evidence. This represents a missed opportunity to develop a common measurement framework for the sector in the City, which should be addressed. SCT should continue to publicise the findings externally to support the case for investment and seek to engage the local authority more fully at an early stage in any future studies.

Limitations

As with all SROI studies, there are limitations with the research presented in this paper. The outcomes measured in Part 1 (general participation) are limited to those where a clear link was strongly and empirically evidenced. Consequently, some potentially relevant outcomes (positive and negative) may have been excluded from the estimation process, such as depression and sports injuries. This is caused mainly by a lack of evidence for the quantitative effects of sport and physical activity on such social outcomes, discussed earlier in the paper (Taylor et al., 2015). It is not a weakness that is likely to be addressed at the local level and needs to be resolved by academic research on the social impact of sport. It is also a limitation shared by the population-wide national sport SROI studies (Davies et al., 2019).

At a local level, other limitations relate to available data for use in the SROI calculations. The NBS survey results used for frequency of visit data – important for estimating the numbers of regular participants – were one or two years older than the reference date for the SROI, and they only related to half the selected facilities. In the absence of user surveys at facilities, membership data would enable frequency of visit to be identified, but that leaves a minority of users in the facilities who are not members, instead preferring pay and play. Obtaining frequency of visit information from such users would be difficult without user surveys therefore this must be prioritised in future studies, either through existing or bespoke surveys of participants.

A limitation of the SIV PARS data collected was the lack of follow-up data on physical activity and biometrics collected by SIV for PARS at either 6 or 12 months. This meant that it was not possible to establish the extent to which the activity levels and consequent health changes for PARS participants were sustained. While this data would provide a more objective measure of health status for the SROI than self-reported, it is unlikely to be undertaken by SIV because of operational, time and resource pressures. A more likely means of researching the long-term effects of PARS is a partnership with Sheffield Hallam University, which already has usage and membership partnerships with SCT. Such resource issues may also prove to be a barrier for other facility managers and operators wishing to use the SROI method (King, 2014).

Finally, the PARS SROI only measures the value of social outcomes to participants. However, the stakeholder interviews, focus groups and PARS survey suggest a wider group of beneficiaries including friends and family. The research methodology did not extend to the friends and families of PARS participants, for resource reasons, but the testimony of the participants themselves on these “external” beneficiaries suggests that the estimated SROI underestimates social benefits. This represents a limitation of the study and an area for improvement in the future.

Conclusions

For some, there is an ideological incongruence in assigning a monetary value to non-market goods (Arvidson et al., 2013; Millar & Hall, 2013). Many objections derive from the viewpoint that individual and social values are more important for human wellbeing than economic values, and as such are simply non-tradeable (Orlowski & Wicker, 2019). However, social impacts are not valueless, and indeed are often the principle justifying public subsidies. The point of utilising a SROI framework is to make the value of social impacts more explicit, which will hopefully lead to a more reasoned consideration of them in policy and management decision-making. SROI is about value not money. Money is simply a common readily available unit and as such is a useful and widely accepted way of conveying value (Nicholls et al., 2012; Orlowski & Wicker, 2019).

The research presented in this paper has used an SROI framework to assign a monetary value to the social impacts generated through participation in sport and physical activity at 12 facilities in Sheffield, in a holistic manner not achieved in other facility-based studies. In doing so, it has enabled the SCT group to identify the value of its facilities beyond the financial indicators commonly used in performance management. Externally, this strengthens the business case for retaining or even expanding services in a competitive environment. Internally, this helps improve strategic and operational planning to improve effectiveness and efficiencies (King, 2014). The SROI results provide a sound reason for continued investment by funding stakeholders, in the facilities generally and PARS specifically. Particularly strong SROI estimates are evident for public sector funding of the facilities (SROI = 2.01) and for PARS (SROI = 3.42). However, in regard to local authority funding by SCC of SCT, the financial pressure from reduced SCC budgets has proven to be a much greater influence on their funding of SCT than evidence of social returns to public subsidies. Nevertheless, such evidence might become more relevant if and when local authority budgets allow.

This paper provides a transparent methodological example of how SROI can be used by facility operators and managers at the local level, to measure the social value of their activities. However, the application of SROI in this context is not without challenge. Various practical barriers exist to applying the method more widely in the sector (King, 2014; Millar & Hall, 2013). As demonstrated, SROI studies require considerable evidence and data, which can be resource intensive to collect. Furthermore, because of the complexity of the method, the technical expertise required is often not available within organisations. Therefore, SROI studies often need to be commissioned, which can be costly and prohibitive. Nevertheless, for those organisations such as SCT and SIV, that can overcome these barriers, SROI is a useful method for enabling managers to “prove” the value of their activities and services, and “improve” the social value they generate in the future. Both of these purposes are particularly important to organisations with health and wellbeing objectives at their heart.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by Sheffield City Trust.

Notes

1 Data were collected at Concord Sports Centre (2013); English Institute of Sport (2014); Ponds Forge International Sports Centre (2013); Westfield Sports Centre (2014); iceSheffield (2014); Hillsborough Leisure Centre (2014).

2 The contact details for 250 participants were not available.