The Research Excellence Framework (REF): Assessing the Impact of Social Work Research on Society

Abstract

This paper reviews one aspect, impact, of the forthcoming assessment of research in UK universities, the Research Excellence Framework (REF), and examines its meaning and potential for enhanced partnerships between social work practice and academia in the context of the current economic crisis. Examples of case studies being developed to show how research has societal impact are described and some of the complexities of what, on the surface appears to echo social work's desire to make a positive difference to the lives of people in society, are drawn out. The importance of the REF for the integration of social work practice and academia has been rehearsed many times. This paper argues that making an impact is everybody's concern and practitioners and those who use social work services and their carers have a role to play in its creation and identification.

Keywords:

• research
• impact
• practice/academy relations

Social work is described in the QAA Subject Benchmark statement (QAA 2008) as an ‘applied academic subject’. As it is an academic discipline with a substantial number of part-time and shared practice-university appointments (Mills et al. 2006), we can easily see that the ways in which social work research is assessed, is of interest to all involved in the discipline, be it practitioners, managers, policy-makers, service users or academics.

A considerable amount of UK research is publicly funded and this fuels the Government's drive to determine the impact on society of that research. This interest in producing social benefit is not unique to the UK, for example, US researchers seeking funding from the National Science Foundation must justify their work in terms of ‘broader impacts’, which may include promoting education and training, broadening participation, or delivering more general social benefits (Sarewitz 2011). Nor is it an unreasonable demand; indeed, a focus on socially beneficial outcomes of research has struck a cord with many in social work and social care who have questioned the academy-practice articulation (Social Work Task Force 2009). However, some UK observers consider it ties in with ‘the current coalition government's demand that researchers “do more for less”’ (Gilbert 2010, 357).

The forthcoming UK evaluation of research activity includes an attempt to assess the wider impact of academic research. Professor James Goodwin, head of research at the charity Age UK, recently stressed the importance of ‘converting research into a message that will influence people's thinking’ (BU 2011). This article provides information about this initiative and what it means for social work as well as outlining how we, as a contributing University, are grappling with the requirements in order to develop a framework that will demonstrate impact outside of formal academia, which is an important facet of assessment of research impact in this scheme.

Background

Higher Education incorporates the notion that undertaking research, as well as the outcomes arising from it, are of use to the wider society (Higher Education Funding Council for England, www.hefce.ac.uk). Of course, social work and social policy academics generally agree with this presumption, one which cements the partnership links with practice currently so high on the Government and Social Work Reform Board agendas (Social Work Reform Board 2010). Up until now, a degree of control over the funding for these activities was exercised through the Research Assessment Exercise (www.rae.ac.uk) which needed to be completed by all UK HEIs in receipt of state funding for research. However, to capture more of a sense of the impact of research, a new system called the Research Excellence Framework (REF) has been introduced. This new assessment will aim to assess the quality of academic research in all disciplines, including social work which is assessed within a sub-panel alongside social policy (www.hefce.ac.uk) and integral will be identification of the ‘impact’ that academic research has had on society and the economy (HEFCE 2011a). Impact, as may be imagined, has a particular meaning in terms of the REF.

This framework, the REF, requires UK universities to put forward a number of case studies for each subject area (in REF jargon ‘Unit of Assessment’) submitted. The number of impact case studies depends on the number of full-time staff whose research publications are included for assessment, for instance, 1 to 14.99 full-time equivalent staff members require two case studies, and 15 to 24.99 staff entered into the REF need three and so on. The case studies detail research produced by academic staff at that particular university within an assessment period (1 Jan 2008 to 31 July 2013), which identify the social, economic or cultural impacts that have taken place beyond academia and which are underpinned by good quality research. This criterion excludes research that simply improves teaching within the institution or that which adds to the sum total of knowledge about a particular phenomenon, forcing academics to think more widely how their work has added to social benefit. Fortunately, there is a sub-clause that indicates that impacts on higher education, students and teaching may be included if they extend ‘significantly’ beyond that particular university. For social work, this may, of course, impact on the commitment to continuing professional development activities relating to post-qualifying social work education, which are designed to improve and enhance practice, and, therefore, the outcomes for people who use social work services.

Case studies must also include information about how the academic department submitting the research has supported and enabled these outcomes during the above mentioned assessment period (HEFCE 2011a). The case studies, and the research environment, will be assessed by a national committee of experts in the relevant field which includes academics and importantly users of research from practice. The attempt to ensure relevance and value-for-money for publicly funded research is clear, and maybe welcomed where it contributes to increased well-being, better services and so forth. It must be remembered, however, that it is ideologically driven, associating research with a tangible outcome perhaps rather than simply increased knowledge and understanding. It may seem, on the surface, important and a victory for common sense (whoever determines this), but it promotes one specific view of research only.

The case studies form only part of a larger submission by a university to one of the national REF sub-panels. Thus social work research will, in most universities entering the REF, be submitted to Panel 22 ‘Social Work and Social Policy’ (HEFCE 2010). Each university makes the decision on where to submit its research on the basis of its own academic disciplinary strengths, however, and the expectation as to where it would achieve the highest overall assessment and, a core reason for submitting to the exercise, most funding. For example, universities with a strong history of Sociology research may decide to put their social work researchers in the same submission and submit to Panel 23, whilst a neighbouring university with a strong Social Work or Social Policy research group is most likely to put their social work research in Panel 22. This tactic, in itself, has implications for impact assessment, and highlights further the need for social work research to be built and undertaken in partnership with social care agencies seeking effective and efficient methodologies of care, practitioners, as end-users of research, and service users as those who will receive the ‘fruits’ of the research.

Scoring for each submission will comprise three parts: 65% will be based on the top four academic papers produced by each member of staff involved in the submission, 15% on the research environment (i.e. the university's support for researchers, seminar programmes, training opportunities, PhD students and so forth) and 20% on the non-academic impact of selected pieces of research (i.e. case studies) conducted in the research unit.

This all sounded very reasonable until we tried to apply the framework with its current guidance to write about the impact related to the aspects of social work research undertaken at Bournemouth University (BU). Not all social work research concerns outcomes in practice but includes scholarship about what the provision of social work, its philosophy and direct care means to those undertaking it or receiving it as well as whether the practices and organisation studied can affect and potentially change people's behaviour in the wider society. While this raises doubts about whether it is possible to capture fully the impact of social work research at all (and social work itself for that matter), clearly some pathway needs to be found to do this. As this seems to represent a fundamental shift in funding and appreciating research, both social work academics and practitioners need to engage in a debate as to where this leads. That research should be appraised as to the impact it has had on improving social, economic, health, environmental aspects of life and quality of life is unlikely to be questioned. However, should such outcome-focused research be the only type supported, the sector is likely to lose those deeper understandings and meanings that have developed and now permeate practice. Indeed, it may be that such theoretical research has engendered practice cultures that have positively changed lives.

Developing a ‘provable’ story, or putting together causal links to demonstrate that research has led to such an impact is an extremely challenging task. It may also privilege an instrumental view of research and universities linking with a focused neo-liberal positivism that currently infuses performance culture and target driven social work. Imagining such a pessimistic outcome, may in itself lead to the generation of new research; however, it certainly demands understanding of the REF and a need for social work practitioners and managers, policy-makers, people who use services and academics to engage in dialogue concerning the future(s) of social work research.

In social work this is taking place at a time of change in which practice-academy partnerships are once again promoted, and the value of research is being moulded within this context. Alongside these changes, there is considerable retrenchment in public services (Farnsworth and Irving 2011) and an increasing need, given UK demography (Parker Forthcoming). Calls for partnership for better practice may conflict with, seemingly, more immediate practice needs, which may disrupt the positive potential for partnerships between practice and the research academy. This and similar issues have to be addressed by all UK universities preparing their submissions for the 2014 REF. In social work submissions, this may involve collaboration with practice partners in research about how best to present the case studies and how to find the non-academic evidence of impact, including impact that extends significantly beyond academia if relating to students, teaching and so forth. Further, questions arise as to what to do if one can find only scattered bits and pieces or even evidence of negative impact. What could a university do about it to improve the impact before 2013, and indeed beforehand as the REF panels require universities to inform them of the impact case studies to be submitted in each area so that appropriate panel members to judge that impact can be located? This paper addresses some of these key issues and offers some possible ways of generating the ‘evidence’ of non-academic impact of research conducted in the field of social work from the perspective of one university. We start with the fundamental question: ‘What does the REF consider to be impact?’

Defining Impact

The REF defines impact as ‘(a)ny identifiable benefit to or positive influence on the economy, society, public policy or services, culture, the environment and/or quality of life’. Impact includes, but is not limited to, an effect on, change or benefit to: the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding; of an audience, beneficiary, community, constituency, organisation or individuals; and in any geographic location whether locally, regionally, nationally or internationally (HEFCE 2011b, 40).

A case study could incorporate more than one type of impact generated from a single piece of research, for example research into a new social work intervention in child protection may have quality of life, social and economic impacts. As noted above, impact excludes influences within the academic sphere or the advancement of scientific knowledge alone. Hence impact on teaching social work students; anticipated impact (i.e. no evidence yet); or dissemination to the public (without evidence of impact or benefit, as defined above, from that activity) cannot be claimed as impact for the REF. This definition excludes dissemination activities and academic impact, so, for example, an excellent study of social work theory which is widely quoted but does not seem yet or directly to affect practice or the well-being of those using social work services would not count as having impact. Impact from education can only be included if it affects people significantly beyond the university undertaking the research. The initial timeframe referred to earlier over which the impact is to be demonstrated is January 2008–July 2013, based on (high) quality research undertaken during the period January 1993 to December 2013. The way in which impact is defined in the REF aligns closely with current calls for mutually beneficial partnerships between universities and social work agencies, and offers potential to identify new and better ways of working for practitioners, promoting those areas of research that may be likely to generate such impacts. However, this is prospective and highlights perhaps the need to develop partnerships for research designed to improve practice.

Box 1How Aspects of Impact Might Apply to Social Work Research Outcomes

•	Impact on different types of beneficiaries such as: individual social workers or users of social work services, social services agencies and voluntary or private organisations, communities (however defined), policy-makers, politicians, law makers, regional development groups, and other bodies including the natural environment.
•	Impact on, processes, behaviours, beliefs, policies, procedures, practices including the avoidance of harm as well as creating positive benefits.
•	Impact can be historical, already occurred but still active such as a change in the way mental health assessments are conducted arising from approved social work prior to the development of Approved Mental Health Professionals (AMHP).
•	Impact arising from the unit's portfolio of research as a whole not only (a) the impact of one or each researcher or (b) all research from that unit. It is here, perhaps, that theoretical and conceptual research may influence the later application of outcome research, and is something that needs to be considered as a rounded approach to future research within this changed environment.
•	Expert review which will include academic and non-academic representatives to assess the impact.

Searching for Impact

Box 1 suggests how aspects of impact might be applied to social work research outcomes; however, we suggest that the tricky bit is trying to establish the wider societal impact but this might be achieved by using one of the following avenues:

•	Conduct a new piece of primary evaluative research specifically to establish the effect of the implementation of the original research.
•	Ask researchers involved in or funders of the original research to identify their knowledge of its impact and return to those who participated in the research or have been affected by it.
•	Search the web for academic publications and reports produced by staff as an indication that these outputs have been picked up by those not directly involved in the research community, and search the ‘grey’ and practice literature for examples of use in the public, non-academic, domains.

Future partnerships between agencies and universities will allow impact areas and evaluation to be written into research. We have outlined current approaches below with an example of each.

Conduct Additional Research

On completion of a piece of primary research, one could design a before-and-after study of the implementation of research findings in practice. This is a possibility for considering impact for the current REF but also as part of a strategy for future research. This is relevant to some of the BU self-efficacy research evaluating post-qualifying social work. It is, however, time-consuming and expensive to undertake and not altogether conclusive. A research assistant who trawled the dissemination of the original research for impact, found several little pearls. Creating the causal chain, however, needed further work, highlighting that research has been completed and disseminated for others to use but this use has not been evaluated. Again, this indicates a need for a practice-academy partnership approach to future research that may assist its implementation and fine-tuning.

Conducting before-and-after studies would appear to be a good example of trying to measure research impact in practice. However, there are some downsides to this approach as suggested above. First, an evaluation study is a major research project in itself and the researchers would need to be resourced (funded) to investigate any impact of the original study if the evidence is to be clearly unearthed. The time and money needed to do the additional research would make it very hard for universities to measure impact of their research in this way, more so, perhaps, given current funding constraints and concerns that raised fees in 2012 will have a negative financial impact on them. Secondly, it might be possible to tease out the impact of an medical intervention which lends itself to research using a Randomised Controlled Trial (RCT) or a quasi-experimental approach, but how are we to study the impact of a piece of social research which is less clear cut, more community-based or which is theory-based as so many are within social work research? Thirdly, following the principles of evidence-based policy and practice, practice-actions/behaviours, policy or views should not be changed on the basis of a single study alone, but on the whole body of research evidence available at the time. SCIE research reviews are important in drawing this material together (www.scie.org.uk).

Asking the Researchers

We asked ourselves ‘What evidence can we produce to show possible impact of our work on the wider world?’ This was not an easy process, partly because researchers are probably more aware of the impact of their work on other researchers and students rather than the general public or practitioners, partly because researchers have been socialised for decades to focus on academic outputs such as research-based papers in high-quality scientific journals, or presenting papers at scientific conferences, and partly because evidence of impact — however it may be defined but especially according to the HEFCE definition — is generally rather vague.

Searching for Evidence

The third approach is using the search techniques developed for systematic reviewing (Haig and Dozier 2003; van Teijlingen et al. 2006; Macdonald 2003) to find non-academic information, policy papers and other grey literature that quotes specific examples of our social work research at BU. We found searching the electronic databases useful in locating snippets of information. For example, searching Google Scholar we ignored all scientific papers that quoted BU research and only focused on the non-academic papers and reports. However, what we also found was a lesser focus on social work research than other disciplines, given the lack of attention to bibliometric review of social work outputs in the past. This may hamper the pinpointing of impact currently.

Having considered sources of and techniques for finding ‘impact’ when we considered our research and scholarship activities over the required timescale, what we could see was an emerging focus on hearing the voice of previously unheard stakeholders; and, perhaps more problematic, for ‘impact’, a focus on pedagogy and its outcomes. There were other themes too and understanding how these could be brought together as social work case studies requires a fresh vision as the following examples demonstrate:

•

The results of a programme of research beginning with a collaborative study funding by the Big Lottery of the perceptions and experiences of being older gay and lesbian adults in a rural area (Fenge et al. 2009; Fenge 2010) led to the reporting of experiences, its communication in workshops to social care agencies working with older adults and shared academic outputs between researcher and participants (Fenge, Fannin, and Hicks 2011), methodological innovations (Fenge, Fannin, and Hicks 2011) and further opportunities for research and dissemination including a short film (Fenge and Jones 2011). It would seem that impact should be fairly easy to portray in this case. However, a research assistant employed to review the work and search for evidence of impact found most of the impact to be conceptual, theoretical and to remain within the academy. It is unlikely that this is the case, of course, as participation in the research and its dissemination affected the lives of those taking part, and anecdotal evidence suggests this is indeed the case. But it is not possible to verify this impact, partly because of data protection issues relating to the workshops, partly the attrition of earlier participants and partly the ‘messiness’ of demonstrating clear causal links between the research and changed practices. It may be that returning to participants, after appropriate ethical and legal concerns have been dealt with, may provide more evidence. The latter is likely to be fairly anecdotal, and would be expensive to undertake. This leads us to confirm that finding impact of our work is neither straightforward nor cheap.

•

The development and application of self-efficacy scales for the self-assessment of learning and identification of student learning needs, practice educators, and for enhancing the research-mindedness of practitioners (Parker 2006, 2007; Quinney and Parker 2010) combined with research concerning Post-Qualifying Social Work education focusing on the production and nurturing of better practitioners for improved human outcomes (Brown et al. 2009; Bates et al. 2010; Warren et al. 2010). The research was overtly centred on practice outcomes, the needs of social service agencies and social workers in respect of their continuing professional education. Whilst we were concerned that impact to develop the skills of new and experienced practitioners in practice may not be legitimate within the REF we were pleased to note the inclusion of research that extended significantly beyond the reach of the university, especially given that our BU work has clearly informed the policy practice debate.

Finding the evidence to support the impact has not been easy, despite (a) the efforts of our research assistant; and (b) the reach of BU's work. However, we believe the research programme undertaken to enhance social work practice through continuing professional education of practitioners has had an effect on, changed practice and benefited the economy, society, culture, public policy and services, health, environment or quality of life beyond academia. Impact on students within the HEI but who are practising with the general public outside of the HEI ensures that the benefits are accrued externally, making this significant in terms of value for money and a fusion of the Higher Education/public interface. The research has also influenced policy, practice and the direction of education and support for newly qualified social workers (NQSWs) with our work used in Reform Board and Select Committee reviews. Gaps in previous research have been identified as evidenced within the Children's Workforce Development Council NQSW research programme and our research has contributed to addressing some of these, whilst developing further the support and focus needed (Keen et al. 2009). The reach has been international in terms of self-efficacy (Holden, personal communication), national in informing SCIE and informing practice in local authorities in England and university programmes in England and Scotland. Follow up studies relating to post-qualifying education in the workplace and with beneficiaries of services are planned to provide further evidence of impact. For instance, currently, over 40 local authorities out of 152 have accessed our programme for NQSWs. The research work is used in the Social Work Reform Board response to inform Assessed Year in Employment standards through the Career Working Group.

Searching for the evidence is demanding and demonstrating objectively (causation of impact?) the impact is not a simple task. The research concerns human relations and interactions and testimonial support is but a subjective measure and we are unsure of the worth of such ‘evidence’. Evidently, our research has been widely disseminated at practitioner and academic conferences and in social work journals. The future planned benefit of this research to society, economy, culture, environment, quality of life, is to improve/develop social work practice to protect the most vulnerable in society.

We consider that, even at this early stage, there is positive impact from our research to a wider community although we are currently unable to measure accurately the exact effect. We are all learners in describing research impact, despite, perhaps on the surface at least, this being one of the drivers of research in this area. However, it is arguable that where further reports or forms of guidance arise from these activities, then to some extent, the impact of this on future practice has to be implied.

Conclusion and Implications for Practice

There is much more discussion needed to demonstrate the impact of social work research. This despite the commitment to social and quality of life improvements that motivates so much of that research across the sector. However, we feel this has made a small inroad into these areas, bearing in mind that assessment will be by experts in the field – academic and practice. In the context of professional and academic development, the application of the REF should provide an important measure of effect in terms of the allocation of resources but it is important to remember that it will have its limitations and that it makes demands not only on the academy but also on practitioners and the ways in which research is seen and used. As we have noted, this initiative suggests a sea change that questions whether research that does not have this type of impact can be justified in terms of ethics and use of resources. One positive outcome from this new aspect of research evaluation is that researchers will be supported by their institutions when trying to make their work relevant to society. It will be important to ensure that the partnership approach becomes embedded and that this includes practitioners' research foci. This is not only for REF purposes, which represents an HEI assessment, but it will help ensure that universities retain the capacity to educate critical-thinking, engage practitioners, and help address some of the challenges arising from the current period of public spending restraint and welfare retrenchment.

However, should the conceptual, theoretical and critical ‘thorn-in-the-side’ research be marginalised or disappear it is likely to make the research landscape less vibrant, anodyne and potentially much less useful: balance is required. Another question not yet explored, is whether this venture might also result in better communications between universities themselves and, given the Reform Board emphasis on practice-academy partnerships, practice agencies, or whether it might have the opposite effect. We are sure there are many people in practice working with universities grappling with the same issues and we would very much like to have discussions with you.

Some of the implications these changes bring will benefit from partnerships in the development, execution and dissemination of social work research and constructing a strategy for research impact is one way in which people can work together to ensure outputs benefit those funding, delivering or using services. It is important to recognise that the beneficiaries may not always want the same outcome as researchers! We have also, in this new world, to ensure we do not lose potentially rich sources of meaning and understanding in our fluid and rapidly changing society. These understandings may or may not result in impact on the long term but they add colour and vibrancy to our approach to social work and potential impacts may be lost if this research is curtailed. In any research impact strategy, ways of making room for this balanced approach will be needed, and in a co-developed social work plan this remains the case.

Much of BU's research, as demonstrated in our case studies, actively involves participants (service users and/or practitioners), often as co-researchers. Where this is the case potential impact can be planned into the research itself, and as social work academics we can learn from the increasing requirement of funding bodies to delineate impacts at the proposal stage.

Researchers and university management need to grapple the issue of the resources required to demonstrate impact. Without considering the time-costs of systematic review, follow-up and replication research or the employment of a research assistant ways of demonstrating impact will be lost and any focus on it will be but tokenistic, paying lip-service to an initiative that has many positive features to it. It is important for all stakeholders to be aware of the way research impact is going to be assessed, and the likely changes that will be seen in research itself. It is hoped this paper goes some small way to inform a wider audience of these developments.