Investigating outcomes of social care programmes using creative study designs for service excellence

ABSTRACT

In this paper, the author has reviewed four pieces of practice research that aim to investigate the outcome of social care programmes. Special attention is given to the process where the concerns of social work practitioners and stakeholders, including academic supervisors, agency administrators, funders and policy makers involved in the dialogues. In addition to the requirement of rigour in study design, priority is given to the significance of the contribution to practice knowledge through adequately interpreting and disseminating research findings through dialogues. Based on the mapping of individual practitioner−researcher, exemplars of randomised controlled trial and clinical data mining in social care programme evaluation are selected to illustrate how they respond to the research questions in a specific practice setting.

作者在本文中回顾了四项以探讨社会照顾项目成效为目的的实践研究, 尤为关注社会工作者与利益相关者——包括学术督导、机构主管、资助者及政策制定者之间的对话过程。在满足研究方法的严格要求外, 重点在于探讨怎样通过对话将研究结果充分地解释及传播, 从而对实践知识提升作出贡献。基于一个实务工作者-研究者的检索, 本文选取了评估社会照顾项目成效的随机对照实验和临床资料勘探作为范例, 以说明这些研究如何在特定的实践环境中回应研究问题。

KEYWORDS:

• Social care programmes
• outcome
• randomised controlled trials
• clinical data mining
• doctor of social work dissertations

Before joining the universities in Hong Kong as a faculty member, I was a clinical social worker offering individual counselling and therapeutic group programmes for 17 years. After studying some clinical approaches and being involved in implementing and evaluating intervention outcomes, I gradually learned to apply different practice research methods that could be embedded in the social care programmes. Being inspired by my supervisors and mentors, Professor Cecilia Chan, Professor Ng Siu Man and Professor Irwin Epstein, I realised how research findings can inform and advance social care practice.

In the past ten years, I have changed from a practitioner-researcher to a researcher-practitioner. Most of the study design I have been applying is a randomised controlled trial (RCT) in evaluating intervention outcomes. At the same time, I continued to share the valuable findings I got from non-RCT design in earlier years. I have supervised several doctoral social work students’ projects using different practice research methods. In this paper, I would like to review these interesting projects and discuss how I map different study designs with their study objectives. Although there has been a golden rule about RCTs placed at the top of the pyramid in evidence-based practice, we should never underestimate each study’s unique contribution to advancing social care and service excellence.

Advantage of Randomised Controlled Trial and its application in social care

Evidence-based practice (EBP) refers to “the integration of best research evidence with clinical expertise and patient values” (Thyer 2004), and the adoption of the EBP framework is considered essential in the field of social work, both professionally and ethically (McNeece and Thyer 2004). Although priority has been given to randomised controlled trials in EBP, the practitioner often encounters research questions relating to real-world practice settings. In practice research, a matching of study design and research objectives should be the central question about how a study should be conducted. In this article, I shall briefly discuss how RCT and clinical data mining (CDM) are chosen in four research projects. All of them have provided significant practice knowledge in their areas of interest.

Case study 1: a feasibility and outcome study of a brief mindfulness-based programme for parents of children with developmental disabilities

There has been a growing interest in applying mindfulness-based programme in service users facing stress in parenting and caregiving. In an early initiative, agency managers and practitioners have approached me for a collaboration to develop a brief mindfulness-based programme for parents of preschool children with developmental disabilities. Although a benchmark mindfulness-based stress reduction programme has been well documented, and there has been substantial evidence about its benefits in reducing the stress of parents and caregivers (Cachia, Anderson, and Moore 2016; Hou et al. 2014; Hwang and Kearney 2013). Such programme lasts for 20 to 27 hours, and their intensity poses a challenge to training, fidelity and supervision that prevents a wider generalisability, dissemination and implementation (Rycroft-Malone et al. 2014). Besides, the content and format of a brief programme may be more appealing for parents who have been burdened.

Therefore, we followed the procedure of intervention research proposed by Fraser et al. (2009). Based on mindfulness and its demonstrated evidence in stress reduction, we developed programme materials and conducted a pilot test. The programme lasted for six weeks, and each session lasted 1.5 hours. To provide quality feedback for participants, each programme accommodated 15 to 20 parents. After giving the effort to programme refinement, we conducted an RCT to evaluate the programme’s effectiveness using a multi-site trial (Lo et al. 2017). The NGO recruited 180 parents from their preschool childcare centres, and participants were randomly assigned to the experimental group (the brief mindfulness-based programme) and control group (standard care), but parents could enrol for a one-day mindfulness workshop after the completion of the study. Outcome measures (parental stress, depression, child behaviour, mindful parenting and marital satisfaction) were collected for intervention and control groups before and after the programme.

Results showed that after the programme, parents had significant improvements in parental stress and depression, and a subscale from parental stress further suggested an improvement in parent-child dysfunctional interaction. Based on the pre-test scores, parents with severe stress and depression reported more significant improvements with moderate effect sizes of 0.62 for stress and 0.57 for depression (Lo et al. 2017).

Based on the encouraging results of the study, I received two competitive research grants and conducted two follow-up studies by integrating the brief mindfulness parent programme with a child mindfulness programme for children with Attention Deficit-Hyperactivity Disorder symptomology (Lo et al. 2020) and economically disadvantaged families (Lo et al. 2019). In these family intervention studies, a wait-list randomised controlled design was applied. Children and their parents were randomly allocated into mindfulness intervention and the wait-list control group.Participants from the wait-list group received the same intervention after the intervention group had completed the programme. Both studies reported positive child and parent outcomes (Lo et al. 2019, 2020). Conducting RCTs for an innovative programme or an adapted version for a new target population becomes a straightforward pathway for generating evidence and knowledge about applying for a brief mindfulness programme for parents and families.

Case study 2 – a Zentangle group intervention for older adults with depression

My mindfulness and Doctor of Social Work student Henri Chan has been interested in developing his expertise in mindfulness approaches for years. After completing professional training and becoming one of the earliest certified Zentangle practitioners in Hong Kong, he has delivered many Zentangle workshops for older adults in many service centres. Zentangle is an innovative method that integrates mindfulness withdrawal to promote calmness and awareness, emphasising the “relaxed focus” aspect (Roberts and Thomas 2012). Although Zentangle has been widely implemented in some countries, both in the general and clinical populations, empirical studies of its outcomes are limited. Henri has been very passionate about promoting Zentangle in social care and was keen to contribute his Doctor of Social Work Study to investigate Zentangle intervention’s benefits for Chinese older adults. He chose a wait-list RCT design, and his study has become an important study in supporting the evidence of the Zentangle programme, especially for depression (Chan 2021).

With the support of collaborating elderly service centres, Henri recruited participants in elderly centres in the community. He recruited a sample aged 60 or above and scored five or above on the Patient Health Questionnaire-9 (PHQ-9), having no formal diagnosis of other mental health illnesses other than the major depressive disorder. Henri developed a six-session intervention protocol based on the original Zentangle method and refined the programme after a few pilot studies and collecting comments from participants and experts. It consisted of six weekly sessions, with eight to ten participants in a group and 1.5 hours per week session.

Similarly, Henri used well-validated standard measurement tools (depression, self-compassion, positive and negative affect) to evaluate the outcomes by comparing the conditions between the intervention group and the wait-list control group. He further collected data at three-time points, including the baseline (T0), post-intervention (T1) and a six-week follow-up (T2). After the study period, the same programme was provided for participants assigned to the wait-list control group. The method yielded a between-group comparison condition, and the between-group parameters such as age, sex, living arrangement, history of psychiatric diagnoses and whether they were on psychiatric medication were not statistically significant with p > .05 (Chan 2021; Chan and Lo Under review).

Henri finally reported that older adults from the intervention group, when compared with the control group, have shown more significant changes in depressive symptoms, self-compassion, positive and negative affect. Using repeated measure ANOVA, he reported significant Time x Group effects for depression, self-compassion, and positive affect and negative affect with a moderate to large effect size. Six-week follow-up indicated that such improvements were sustained. Henri’s study provided preliminary support that the Zentangle programme is an effective alternative treatment approach for older adults with “mild-to-moderate” depression that can effectively reduce depressive symptoms and cultivate self-compassion.

Both case studies applying RCT designs demonstrated their advantages in promoting social care and service excellence. RCT has been considered the gold standard of evidence when determining the effectiveness of practice interventions. Compared with quasi-experimental designs, RCT can provide stronger evidence that such positive outcomes could only be attributed to the intervention programme and no other contextual factors that have been controlled. In response to the ethical concern that many frontline practitioners cannot accept that some participants had been excluded from the intervention, a wait-list RCT would be a compromise that has been commonly used in social care. Practitioners and participants would feel relatively acceptable to receiving the same programme with a time lag instead of being excluded from the potentially beneficial programme.

There has been a growing interest and competence in the execution of RCTs in the social work profession (Solomon, Cavanaugh, and Draine 2009). Social work practitioners, like psychologists and other health care professionals, can claim their role as experts regarding service interventions. Researchers can contribute to evaluating programme effectiveness and accumulating practice knowledge that may have significant implications for policy-making.

Clinical Data Mining as a valuable research method in social care

Clinical data mining (CDM) is a practice-based retrospective research strategy, and practitioner-researchers who adopt CDM would systematically retrieve, codify, analyse and interpret available data to evaluate the practice, programme and policy implications of their findings (Epstein 2010). CDM can refine and enhance practice wisdom, describe and evaluate social work practice, promote “evidence-informed” practice, identify best practices and promote social work “reflectiveness” through the analysis of available data.

The steps in the CDM process include searching for and collecting all prospect data sources, such as case records, medical records and computerised information; assessing core samples for available variables, such as demographics and presenting problems, interventions and outcomes; identifying key practice questions that are answerable with the data; consulting research literature for prior studies; creating qualitative and/or quantitative retrieval tools; making sampling and design decisions; promoting reliability and validity; collecting and planning the analysis; analysing data; interpreting and utilising findings; and disseminating findings.

As discussed, EBP based on RCT involves the random assignment of participants into experimental and control groups. However, the design of RCT has a few limitations in feasibility and generalisability. RCT is usually based on specific research objectives with strict inclusion and exclusion criteria, and its external validity is often questioned (Rothwell 2006). CDM enables a researcher to conduct a study that can answer questions unique to own practice settings, and it does not require a sophisticated research design. Despite the fewer restrictions in research design, CDM provides meaningful results in generating practice knowledge and promoting advancement in evidence-based practice (Epstein 2010). Two exemplars of CDM are provided to illustrate the unique feature and its significance in practising knowledge building.

Case study 3 – clinical data mining on outcomes in cognitive behaviour group therapy

Before I studied PhD, I was involved in a community mental health service which offered a cognitive behaviour group therapy (CBGT) for several years. It was based on a standardised protocol, and in my NGO, the same programme was implemented for three decades (Hong Kong Family Welfare Society 1989). Inspired by the CDM method, I made use of the existing clinical records and realised that it could be a meaningful way to explore the strengths and limitations of the programme (Lo et al. 2011).

This study was a retrospective study based on the data collected from participants joining CBGT between the years 2000 to 2008. After getting permission from my agency, I identified 672 participants’ profiles for this study. In each profile, outcome measures included the Psychiatric Symptom Rating Scale (PSRS; Hong Kong Family Welfare Society 1989), which was based on a social work’s scoring after intake interview, a standardised self-reported scale, the 12-item version of General Health Questionnaire (Goldberg 1972), attendance records and demographic data. The PSRS covered 24 items and seven domains, including physical health, work adjustment, use of leisure and interpersonal adjustment, social perception, social interaction pattern, the balance between emotional and rational life, and psychosexual adjustment.

Results of this CDM study suggested that CBGT conducted by social workers was effective and was consistent with most outcome studies based on cognitive behaviour therapy in terms of effect sizes (a Cohen’s d of 1.08), which indicated a large effect size of CBGT. However, 38.0% of the participants still reported experiencing significant levels of impairment after the programme, suggesting the programme still has room for improvement.

Analyses of poor outcome predictors shed light on the limitations of CBGT. With a large dataset and the comprehensive assessment of PSRS, we uncovered interesting practice knowledge from analyses of dropout and poor outcome predictors that were seldom investigated by RCTs but generated knowledge that was interested in many practitioners. Overall, the programme has a low dropout rate of 6.1%, and we identified two predictors of dropout, namely low energy and use of counselling services. Low energy participants were found to have a 2.5 times higher dropout rate than higher energy participants (8.6% vs 3.5%). Besides, participants who were currently in use of counselling services less than half of the dropout rate when compared with those who did not receive counselling services (3.7% vs 7.7%).

Based on the pre-treatment symptom severity of CBGT completers estimated by PSRS scores and the changes in GHQ scores, this study identified some predictors of poor outcome of CBGT, including relationship difficulties (strained family relationships, lack of empathy, intolerance of being alone), cognitive symptoms (rigid thinking and attitudes, ruminations and worries, low self-esteem), physical distress (physical complaint, sleep disturbance and loss of appetite) and suicidal ideation.

Using such a rich dataset in CBGT, I found those who did not respond well in the programme, which was never included in the research questions and hypotheses of RCT, but still contribute important knowledge about what areas of the programme should be improved for better practice (Epstein 2010, 2022).

Case study 4 – a CDM study on effects of treatment programme of a girl’s home for adolescents with emotional and behavioural challenges

The CDM method impressed my Doctor of Social Work student Eliza who is the centre-in-charge of a girl’s home in Hong Kong. She had been thinking about how to conduct a study for evaluating the therapeutic programme of the girl’s home. After struggling with the study method due to the impossibility of conducting an RCT and the restrictions of the residential service setting in data collection, she realised that CDM is probably her best solution. Using available quantitative data based on staff observational data collected after admission and reviewed every half-yearly, she finally completed her DSW dissertation with a CDM study (Poon 2021).

The study sample was collected from the girls admitted to the residential home between April 2007 and December 2017. All girls were required to complete an assessment package within one month after admission and re-assessed every six months. In the dataset, a total of 457 girls completed the data at the first time point (T1), 371 girls were re-assessed six-month follow-up (T2), and 116 girls’ record was included in the 24-month follow-up (T5). According to their first assessment results, girls were assigned to join different therapeutic groups, including a social skill training group, anger and emotional management group, addiction prevention group, cognitive training group, problem-solving group and positive psychology group.

Using pair sample t-tests, the study found significant reductions of overall behavioural symptoms and depressive symptoms from T1 to T2 and significant reductions in depressive symptoms and non-compliance from T1 to T5. Moreover, there is a significant increase in life skills development, independent skills and psychological well-being from T1 to T2, and such positive changes pf independent skills and psychological well-being sustained from T1 to T5 (Poon 2021; Poon, Lo, and Epstein Under review).

Overall, the significant improvement in girls’ overall behavioural symptoms, depressive symptoms, and life skills development is encouraging. Previous literature suggested. We may not be able to answer all questions about why the girls performed better in some outcome measures but not others. However, this study has filled the emptiness about the effects of a residential home for girls and similar programmes outside North America and Europe. We also wanted to explore the reasons for the absence of changes in improper sexual behaviour. A number of factors have been identified, including the girl’s developmental stage and their readiness to reveal their experiences relating to sexuality to staff. We further discuss the suitability of individual items in this problem scale, such as revealing gay orientation according to social work practice ethics. Areas have been identified to inform service improvement for the home after the CDM study, such as integrating self-reported measures into staff’s scoring of outcome behaviour, including girl’s history of trauma into intake assessment and design of individualised, multimodal treatment (Poon 2021; Poon, Lo, and Epstein Under review).

In this paper, we have reviewed four pieces of practice research. In addition to sharing the study findings, what I would also like to explain is the process where the concerns of social work practitioners and stakeholders, including academic supervisors, agency administrators, funders and policy-makers involved in the dialogues. The choice in methodology for practice research is not only based on rigorous academic standards but also its contribution to practice knowledge through adequately interpreting and disseminating research findings through dialogues (Lo et al. 2018). Recently, the keynote speaker of this international conference Irwin Epstein gives a new metaphor to practice research and the relationship between research and the practice of a love affair (Epstein 2022). Such metaphor beautifully illustrates the politics in the transaction of two sides that fills with interdependence, mutual utility, doubts and criticisms within and outside the relationship. It also reminds us that in practice research, no single perfect method can fit all. Different study designs are selected to answer specific research questions based on the mapping of individual practitioner-researcher. There are always unanswered questions and limitations in each piece of study and will leave the fellow and another piece of study to continue the discovery. The fellow and another piece of study to continue the discovery.

Disclosure statement

No potential conflict of interest was reported by the author(s).