Development and evaluation of interventions in social work practice research

ABSTRACT

Methodological pluralism is required in social work practice research to enable researchers to answer diverse practice-based questions. This is particularly the case for developing and evaluating interventions for use by social workers in multiple contexts. This paper illustrates the multiple methods required to develop and evaluate social interventions, using the example of Connecting People. The intervention model was developed from an ethnography of social work practice and piloted in a quasi-experimental study. Focus groups and semi-structured interviews were used alongside a further quasi-experimental study to examine its implementation in one particular practice setting. A randomised controlled trial is currently underway to evaluate the effectiveness of the model in another country. Practice researchers need to develop expertise in multiple methods to respond flexibly to the demands of intervention development and evaluation in social work practice research.

ABSTRACT

社会工作实践研究需要多元主义方法论, 使研究者能够回应不同实践为本的问题, 特别有助于开发和评估让社会工作者在多种情景下得以运用的干预措施。本研究以社会介入模型“Connecting People”为例, 说明开发与评估社会干预措施所需要的多种方法。此干预模型发展于社会工作实践的民族志, 并以一项准实验研究作为试点。在进一步的准实验研究中同时使用了焦点小组, 半结构式访谈方法, 以考察其在特定实践环境中的实施情况。目前, 本研究正通过一项随机对照试验来评估该模型在其他国家的有效性。实践研究人员需要发展多种方法的专业知识, 以灵活应对社会工作实践研究中干预开发和评估的需求。

KEYWORDS:

• Methodological pluralism
• practice research
• social intervention
• evaluation
• implementation

Introduction

Social work practice research takes many forms. It draws upon many different methods to answer practice-based questions. Practice researchers are required to be methodological pluralists, able to use many different methods to answer questions arising from social work practice (Webber 2020). Reflecting the complexity and diversity of the contexts in which social workers are practising and the multiple issues they are addressing daily, researchers need to be flexible and creative, yet always rigorous, in the design of their studies. They need to develop forms of knowledge that can inform practice, and this is not always a linear process. It is not always possible to answer practice-based questions in a single study using a single method. This is particularly the case for developing and evaluating interventions for use in social work practice. A sequential and sometimes iterative approach is required, requiring multiple studies across different contexts to develop and test an appropriate intervention for widespread use.

This paper illustrates the multiple methods required to develop and evaluate social interventions, using the example of Connecting People. This is an intervention model developed for social workers and others to use to support people to enhance their social networks (Webber et al. 2015, 2016). It addresses the following practice questions: What skills are required to support people in developing their social networks? What processes are involved? What barriers need to be overcome? What is the role of the agency? What is the most effective approach to take which can be used in many different contexts? Although this example is drawn from research conducted in England, the process (and indeed the resulting intervention model) is replicable in China as well as internationally.

Intervention development and evaluation are not linear processes. It requires different methods at different parts of the process, and earlier stages in the process may need to be revisited before progressing further. Also, different contexts and different interventions may require different approaches to be taken. The example provided here, therefore, should be accepted as just that.

The approach I took is fully articulated in Webber (2014) and summarised in of that paper (p. 175). The process starts with an understanding of the nature of the problem and draws upon multiple sources of evidence to inform the intervention development process. This includes local knowledge, practice knowledge and explanatory knowledge alongside more formal data on the problem’s incidence and prevalence. This information informs the development of the intervention and, as described below, is best conducted as a collaborative approach with practitioners and service users. The intervention model can then be tested – for feasibility and then effectiveness. Various pre-experimental and experimental methods can be used, but this does not need to be limited to randomised controlled trials, as these are not always feasible or desirable. Implementation studies can then be conducted to explore how the intervention model can best be implemented in different contexts.

Need for intervention

The process of intervention development begins with a thorough understanding of the need for the intervention. For example, this can be a common social problem or something pertaining to a particular group of people. There needs to be a strong enough rationale for the intervention; if there is no problem to solve, then it is arguably unethical to intervene.

An understanding of the problem is often gained through social epidemiology, usually cross-sectional or cohort studies. These studies describe the nature and extent of the problem, in different groups or places, at a single point in time or longitudinally over a period of time. Connecting People, for example, was developed in response to studies which identified inequalities in access to social capital (resources accessible via social networks (Lin 2001)). People with mental health problems have less access to social capital (Webber and Huxley 2007; Lin 2000), as a result of having smaller social networks (Cullen et al. 2017; Xu 2019); experiences of discrimination (Webber et al. 2014); internalised stigma (Drapalski et al. 2013); and insecure attachment styles (Webber, Huxley, and Harris 2011), for example.

Connecting People supports people in developing new social connections and addresses social isolation and loneliness, both of which have significant public health consequences (Leigh-Hunt et al. 2017; Rico-Uribe et al. 2018). Social isolation (being alone or having few social contacts) is a significant health risk and associated with both poor mental health (Giacco et al. 2021; Luo et al. 2021) and early mortality (Holt-Lunstad and Steptoe 2022). Loneliness is the subjectively experienced gap between desired and actual social relationships and is a distressing experience. It too has a significant negative impact on mental health (Erzen and Çikrikci 2018; Beutel et al. 2017; Lim et al. 2018; Wang et al. 2018).

Connectedness has been identified as key to recovery from mental health problems (Leamy et al. 2011), though support with social relationships is often overlooked in mental health services (Ma et al. 2020). Social interventions for people with psychosis have increased network size, which helps to address social isolation (Anderson, Laxhman, and Priebe 2015), for example, and interventions for loneliness show some promise, (Ma et al. 2020; Lloyd-Evans et al. 2020). However, few interventions support social participation (Webber and Fendt-Newlin 2017; Howarth et al. 2016) and stronger evidence is required.

Intervention development

Connecting People was developed using ethnography to explore good practice and some of the complexities involved in supporting people with their relationships and social networks. The Medical Research Council’s (Craig et al. 2008) guidance on the development and evaluation of complex interventions offered limited insights into the early stages of the process, particularly intervention development and early testing. However, more recent guidance (Skivington et al. 2021) has addressed this. For social interventions, arguably inherently complex, it is particularly important to consider the context in which they are to be applied, existing good practices, and the relationship between the worker and the service user.

Ethnographic research is typically conducted by a researcher observing practice in one or a limited number of settings to understand interactions, behaviours and contexts in some depth. However, to explore good practice across diverse contexts, we used a combinative ethnography methodology (Baszanger and Dodier 1997) to observe practice in National Health Service (NHS) community mental health teams, housing support agencies, social enterprises and voluntary sector organisations (Webber et al. 2015). This involved a researcher observing direct practice, interviewing workers and service users about their experiences, and exploring how workers supported service users to make new connections with other people and develop their social relationships. This study focused on young people recovering from an episode of psychosis and included older people with other mental health problems. The study was large, with 73 workers and 51 service users participating across two phases of fieldwork. This enabled us to capture a broad range of experiences and develop the intervention model iteratively with participants.

The analysis revealed workers’ skills, attitudes and roles; processes involved in connecting people; the role of the agency and barriers to network development. Some examples of the themes emerging from the fieldwork were:

• Reducing power differentials works: people can better connect when differences in status are removed. Undertaking shared activities with a common purpose and shared responsibility help to achieve this.

• Non-stigmatised locations bring people together: places are sometimes detrimental to wellbeing or are associated with negative experiences, making them less hospitable for people to connect with others. Places such as cafes accessible to the whole community are less likely to be associated with the stigma of mental health problems than mental health services, for example.

• Local knowledge is essential: supporting people to engage with their community requires good local knowledge, not only of local services, groups and resources, but informal networks. Workers who spend time getting to know the community of the people they work with are likely to be more effective at supporting people to connect with others.

• Connections occur through shared activities: identifying a person’s interests and then connecting them with others who share that interest is more likely to lead to new social connections. Common interests in common promote shared activities and conversations and provide a reason to meet new people.

• Focus on an individual’s goals: support needs to be person-centred and focused on what the individual wants to achieve. Goal-setting needs to be undertaken together so that a plan, and the steps needed to be taken to put the plan into effect, can be mutually agreed upon.

• Informal contacts provide access to resources: the focus of the intervention activity should be on building connections with people who could become friends. Social networks develop through an individual’s acquaintances, becoming friends rather than knowing people within organisations.

• Social capital is not a panacea: people’s networks cannot provide all the resources that an individual may need. Relationships can sometimes be difficult for people, and we should not assume in our practice that social connections are straightforward for everyone.

These findings were shared with practitioners (n = 18) and service users (n = 16) in a series of focus groups at the end of each fieldwork phase, through which we iteratively developed an intervention model. This was refined using a Delphi consultation method (Linstone and Turoff 1975) with twelve international social care and social capital experts, including practitioners and service users. The result was the Connecting People intervention model, which articulated the processes involved in supporting people to connect with others (Webber et al. 2016).

The Connecting People model is dynamic and represents a co-production between practitioners and service users. It is not a traditional or linear model of inputs being provided from a practitioner to a service user but an interactive process in which both the practitioner and service user agree on shared aims and work together to achieve them. The worker comes with a “can do” attitude, good local knowledge and the ability to work with flexibility and confidence to support the service users to achieve their goals. Similarly, the service user enters the process with an openness to try something new and to step out of their comfort zone, with the appropriate support in place. The barriers the practitioner and service user may need to overcome are explicit within the model to ensure that plans are made to tackle them. In addition, the agency in which the intervention occurs is acknowledged as part of the model, as its location within its community is crucial to the success of Connecting People. Its ability to engage with local informal networks, its modelling of good practice within the organisation, and its ability to provide a conducive environment for fostering new social connections make it a vital component of the intervention model.

The complexity of the Connecting People model was subsequently simplified into eight steps to make the process a little more transparent for practitioners and service users:

1. Getting started – having initial discussions with the service user to find out how they feel about trying new things and meeting new people

2. Existing connections – mapping the existing connections that the service user has with other people and services

3. Making plans – identifying aspirations and goals of the service user with regards to their social connections, and make a plan to support the person to achieve these

4. Stepping out – supporting the person to try something new, including some gradual exposure work or attending the first meeting with them

5. Taking stock – reviewing progress and identifying what is working and what is not working

6. Working around barriers – making plans to work around or overcome barriers which have become apparent in earlier steps in the process

7. Organisational culture – helping to shape the organisational culture of the host agency to make it more conducive to Connecting People working effectively

8. Reviewing the process – taking a step back and looking at the whole process from the service users’ and practitioners’ perspectives, potentially using fidelity measures to identify areas for further practice development

Intervention piloting and evaluation

The next stage is to pilot the intervention model, which often requires a pre-experimental or quasi-experimental design. A full experimental design – a randomised controlled trial – requires some evidence that the intervention is likely to improve outcomes and not harm people. It would be unethical to randomise people to receive or not receive an intervention without some pilot evidence. Therefore, the intervention pilot looks for indicative evidence of effectiveness.

There are a number of design options which have been usefully summarised by Simpson (2020):

• One-group post-test – validated measures are administered at the end of the intervention; there is no control group

• One-group pre-test/post-test – validated measures are administered before and after the intervention; there is no control group

• One-group pre-test/post-test and follow-up – validated measures are administered before and after the intervention, and at a point again in the future; there is no control group

• Post-test only design with non-equivalent groups – validated measures are administered after the intervention in a group receiving the intervention and a group not receiving it

• Untreated control group design with dependent pre-test and post-test samples – validated measures are administered before and after the intervention in a group receiving the intervention and a group not receiving the intervention

• Untreated control group design with dependent pre-test and post-test samples using switching replications – validated measures are administered before and after the intervention in a group receiving the intervention and a group not receiving the intervention; the control group receives the intervention after the intervention group has received it and completed the follow-up measures.

There are many variations on these designs, but in common, they use standardised measures with proven reliability and validity to assess the outcomes of the intervention. Outcomes are identified in the modelling process or may drive the development of the intervention itself. For example, studies evaluating Connecting People, which was developed to address inequalities in access to social capital, have used the Resource Generator-UK (RG-UK) (Webber and Huxley 2007) as the primary outcome measure. This was used because it assesses the resourcefulness of an individual’s social network and has been validated for use within the UK general population, including people with mental health problems.

Connecting People was piloted using a one-group pre-test/post-test design. Fourteen agencies were identified who appeared willing and able to implement Connecting People within their routine work. They worked with adults with a mental health problems or a learning disability, though our scoping work found that no adaptation was required to the model for it to work with different groups of people. We provided two-day training to staff in the participating agencies, nine of which were community mental health teams, four were voluntary sector agencies, and one was a local authority day service. The training focused on supporting the agency to implement the model in their practice.

The pilot study recruited 155 new referrals to these agencies, and we interviewed them before they experienced the Connecting People intervention and nine months later. In addition to the RG-UK, we measured their social inclusion (using the Social and Community Opportunities Profile (Huxley et al. 2012)) and mental wellbeing (using the Warwick-Edinburgh Mental Well-Being Scale (Tennant et al. 2007)) as secondary outcomes. After nine months, we obtained follow-up data on 117 (75.5%) participants and analysed their data according to their exposure to Connecting People. We measured the agencies’ ability to implement Connecting People using a fidelity scale and found that 30 participants had experienced “high fidelity” Connecting People. When comparing their outcomes with the 87 participants who received low or moderate fidelity Connecting People, we found that their access to social capital and perceived social inclusion improved. The mental well-being of all participants improved during the study period. The economic evaluation found a non-significant trend towards the cost-effectiveness of Connecting People when implemented with high fidelity. The full findings are reported in Webber et al. (2019).

The positive findings in the pilot study led to further work in other countries. Connecting People was adapted for use in very different contexts in Sierra Leone and Nepal, for example. In addition, it has been applied in services for people with mental health problems and experience in the criminal justice system in Atlantic County, New Jersey, the United States. The effectiveness of this is being evaluated in a randomised controlled trial, where participants are randomly allocated to Connecting People in addition to routine care (intervention group) or routine care (control group). The outcomes being measured include access to social capital, health and well-being, and criminal justice outcomes.

Intervention implementation

The Connecting People pilot study found that all but one of the teams in the “high fidelity” group were in the voluntary sector. These agencies were typically more embedded within their communities and more able to use Connecting People in their daily work. However, this study found that an NHS community mental health team for older people was able to implement Connecting People with high fidelity (Webber et al. 2019). As practice research is concerned with enhancing practice and learning how contexts shape practice, we were keen to investigate what it would take to implement Connecting People in other community mental health teams. These teams provide multi-disciplinary support to people with severe and enduring mental health problems, focusing predominantly on medical or psychological interventions. Social interventions supplement these approaches and can address social problems which people may face.

The study exploring the process and outcomes of Connecting People implementation in NHS mental health services used a quasi-experimental pre-post design with a control group (Webber et al. 2021). The study was conducted in five mental health NHS Trusts, where one community mental health team was selected to implement Connecting People, and a second was recruited as a control team. We co-produced an implementation pack comprising an implementation manual, training manual, practice guidance and service user guide with service users and practitioners to enable teams to implement the model in their routine work (Moran et al. 2020). These materials were designed to support training within teams and to be used by practitioners to support the use of Connecting People in their routine practice.

The study recruited 151 participants who were interviewed twice over six months. The implementation teams started their participant recruitment after Connecting People implementation was initiated. Access to social capital was the primary outcome, measured using the Resource Generator-UK (Webber and Huxley 2007). The secondary outcomes were mental wellbeing (Tennant et al. 2007), goal attainment (Turner-Stokes 2009); health-related quality of life (The EuroQol Group 1990) and experience of recovery (Law et al. 2014). The findings of the study were that there were no differences in the primary and secondary outcomes between the implementation and control group, although a high follow-up rate of 84% (n = 127) was achieved (Webber et al. 2021). This was explained by the poor implementation of Connecting People as evidenced by the fidelity scores remaining similar between the teams in the implementation and control groups.

To explore the barriers or facilitators of implementation, we used focus groups of practitioners and qualitative questions in the service user follow-up interviews. Each team also completed the Implementation Climate Scale (Ehrhart, Aarons, and Farahnak 2014) at baseline to assess their readiness to implement evidence-based interventions. We found that the control teams rated themselves higher on this scale than the implementation teams; this indicated that the control teams (which did not implement Connecting People) were more conducive environments than the implementation teams to undertake this work. The data we collected from the implementation teams supported this finding. In particular, Connecting People was only partially implemented in three teams and not implemented at all in two teams. The training was organised in only one team, who developed a two-hour session for their team members. Supervision was provided by a manager and a senior social worker in two teams, though the implementation materials were largely only “skimmed through” and not used routinely.

The qualitative process evaluation found that the priorities for practitioners in the implementation teams were attending to mental health crises. This took up most of their time, and, along with high caseloads, they found that they had no time to search for activities or immerse themselves in their service users’ communities. In addition, there was a lack of organisational buy-in to the model, which felt like additional work for the intervention teams. Some people also appeared ambivalent about whether they wanted to get well, as they feared losing welfare benefits and the support of practitioners from the community mental health team. When people recovered from a mental health crisis, they were discharged from the community mental health team quite quickly due to pressure on caseloads, which did not provide an opportunity for practitioners to support them with their social connections, as this work was not appropriate during a mental health crisis.

The focus groups and interviews identified some factors that assisted implementation. For example, practitioners who understood the rationale and importance of Connecting People were able to champion it, particularly where organisations supported its implementation. Practitioners argued that manageable caseloads; ongoing training, supervision and consultancy; and strong leadership for Connecting People were essential for its implementation. In addition, practitioners need to be given the time to engage with their service users’ communities to gain up-to-date knowledge of activities, groups or resources in their local area; and, of course, there need to be community resources for people to engage with.

The Connecting People studies have identified how practitioners can effectively support people to make new social connections and enhance their social networks. They have provided guidance on the practice involved and the organisational context required to implement the model with high fidelity. This knowledge has informed the development of a new intervention model – Community-Enhanced Social Prescribing – for use in primary care settings (Morris et al. 2022). A feasibility study is currently underway to test the model’s feasibility in practice and establish indicative outcomes in preparation for an evaluation of its effectiveness.

Conclusion

Practice research involves using multiple research methods to address the various questions arising from social work practice. A form of methodological pluralism requires expertise in a range of research designs and methods to produce knowledge to inform practice. This is particularly the case for developing and evaluating social interventions, which requires a mixture of qualitative and quantitative methods at different stages of the process, each answering different types of questions. As the example of Connecting People has illustrated, there is no definitive approach or set of steps to take in this process. While randomised controlled trials are often held up as the “gold standard” method to evaluate interventions’ effectiveness, there are alternative designs, some of which are favoured in particular situations. The experience of developing and evaluating Connecting People has highlighted the importance of gaining expertise in various methods. This is worthwhile for practice researchers to consider as they set out on the intervention development and evaluation process.

Acknowledgments

The development and evaluation of Connecting People were funded by the National Institute for Health and Care Research (NIHR) School for Social Care Research (SSCR). The views expressed are those of the author and not necessarily those of the NIHR SSCR, the NIHR or the Department of Health and Social Care. The author would like to sincerely thank all the participants of the studies discussed in this paper and the researchers who have collaborated with him.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the NIHR School for Social Care Research [grant refs T976/T09-032/KCLMW; T976/T11-017/KCLMW; C088/CM/UYYB-P114]..