Examining the Complexities of Conducting Randomized Controlled Trials in Child Welfare Settings

ABSTRACT

Background

Randomized controlled trials (RCTs) are considered the gold standard for assessing the effectiveness of interventions. At the same time, it is challenging to evaluate interventions using experimental designs in child welfare settings.

Objective

This study explores the facilitators and challenges faced when carrying out a pilot RCT of a home-visitation intervention, Family Partner, that was implemented within Norwegian child welfare services.

Method

This study draws on 29 qualitative in-depth interviews and four focus group interviews that were carried out with child welfare workers, managers and stakeholders.

Results

Results revealed one facilitator and four challenges with carrying out RCTs in child welfare settings: The main facilitator was the staff’s belief in the intervention, while the challenges were 1) ethical concerns in participant recruitment and information sharing, 2) resource management challenges in randomization, 3) emotional strains in response to randomization, and 4) contamination between the intervention and control participants.

Conclusion

While we present possible solutions to the challenges, our results also highlight the importance of careful planning, preparation, piloting, and using mixed method research approaches when conducting RCTs within child welfare services.

Introduction

Randomized controlled trials (RCTs) are regarded as the gold standard in assessing intervention effectiveness (Thyer & Pignotti, 2011). By assigning participants to different groups through a random process, RCTs enable researchers to rigorously investigate the causal relationships between the intervention under study and its outcomes (Hariton & Locascio, 2018). RCTs are the cornerstone of evidence-based practice, which can be defined as “the integration of best research evidence with clinical expertise and patient values” (Sackett, 2000). Improving the effectiveness of interventions for children and families have received international attention and in Europe there has been an emphasis on the implementation of effective evidence-based practices, including intervention studies with robust research designs, such as randomized controlled trials (European Union, 2024; WHO, 2014).

While some municipalities in Norway implement internationally recognized interventions to improve the effectiveness of child welfare services, such as Multisystemic Therapy, Parent Management Training Oregon, Functional Family Therapy, Marte Meo, and International Child Development Program, few of them have been evaluated in a Norwegian child welfare context (Bragdø-Ellenes & Torjesen, 2020; Christiansen et al., 2015; Haugevik & Neumann, 2020; Ljones et al., 2019; Otterlei et al., 2021). One explanation for lacking evaluation may be that the interventions within child welfare are often complex and consist of multiple sources and interacting mechanisms, heterogeneous target groups, and local adaptations, which also make it difficult to assess “what works” (Craig et al., 2008; Malmberg-Heimonen et al., 2018b).

Previous research has demonstrated that carrying out RCTs in child welfare contexts is challenging (Dixon et al., 2014; Jaramillo et al., 2023; McLaughlin, 2012; Mezey et al., 2015; Oakley et al., 2006). According to Glasgow et al. (2005), the issue of randomization in complex settings is particularly challenging, such as in underserved populations and low resource settings. Accordingly, the study by Dixon et al. (2014) evaluated an intervention for children with challenging behavior. The authors faced several challenges, including both opposition and support among social workers, recruitment issues impacting the statistical power of the RCT, participants switching between study arms causing contamination, and significant differences between participants in treatment and control arms at baseline. Based on the results, the authors recommended incentives for participating social workers, contractual agreements, adequate funding, acknowledgment of ethical dilemmas and for experimental conditions to be met. Other studies have demonstrated the importance of establishing contracts, policies, and procedures to mitigate potential confusion upon start of study (Jaramillo et al., 2023; Mezey et al., 2015).

Another challenge in conducting RCTs may emerge when practitioners involved in delivering the intervention also play active roles in recruiting, screening, and randomizing participants. Child welfare workers, for example, might experience ethical dilemmas with randomization (Oakley et al., 2003). In one of the few RCT studies carried out in the Norwegian child welfare services, Kirkøen et al. (2023) found that child welfare workers experienced emotional reactions when families facing numerous challenges were assigned to the control group. These reactions were especially strong in cases where workers noticed a lack of progress among these families, leading them to form a strong belief that the treatment being tested was more beneficial than the standard care provided to the control group. This highlights the ethical and emotional dilemmas faced by practitioners involved in trials, particularly when those tasked with delivering the intervention also play a part in screening for eligibility or the randomization process. The necessity for intervention providers to believe in its efficacy complicates their role in allocating participants to the control group (Oakley et al., 2003).

Yet another issue faced when carrying out RCTs concerns random allocation (Oakley et al., 2003). This challenge can also have a practical dimension. Child welfare workers are accustomed to offering interventions and describing them in an appealing way to motivate parents to participate. In randomized studies, it is desired for families to be blinded. It is not intended for families to know anything about the intervention before they have agreed to the study and possibly are assigned to the specific treatment condition. It is therefore important to prevent the promotion of the intervention being tested prior to participation.

Considering the multitude of challenges associated with RCTs in complex settings, a diversity of methodological approaches is recommended. For example, The UK Medical Research Council framework for evaluating complex interventions recommends that evaluations in health and social sciences should use both qualitative and quantitative data, as well as theoretical perspectives that emphasize implementation, context, and system adaptation. Such approaches may be particularly important in multi-site RCT studies, where the same intervention can be implemented and received differently at different locations (Oakley et al., 2006; Skivington et al., 2021). When carrying out RCTs in child welfare settings, previous research underscore that certain aspects should be in place, namely strong collaboration and communication with the agencies, a well-developed program model, a structured referral process for eligible participants and data to measure baseline characteristics (Pergamit & Hanson, 2022)

Given the unique, and context-dependent features of child welfare services in different countries (Berrick et al., 2017), challenges identified, and the lack of intervention studies using RCTs, there is a need to explore facilitators and challenges with carrying out RCTs within child welfare.

Aim of the study

RCTs are considered the gold standard for evaluating the effectiveness of interventions (Thyer & Pignotti, 2011). Governing authorities are therefore increasingly advocating rigorous evaluations of the care received by families and children in child welfare services (European Union, 2024; Ministry of Children and Equality, 2016–2017; WHO, 2014). At the same time, assessing the effectiveness of complex interventions is a well-known challenge (Glasgow et al., 2005; Malmberg-Heimonen et al., 2018a; Rogers, 1999). Drawing on qualitative interviews with child welfare workers, managers and stakeholders, this study describes practical and methodological experiences of carrying out a pilot RCT of a home-visitation program at three child welfare offices across Norway. The following research questions were explored: (1) how do child welfare workers experience taking part in an RCT study? (2) what facilitators and challenges are identified when carrying out the study?

The family partner intervention

Family Partner is an intensive home-visitation intervention that aims to reduce child maltreatment in families with complex support needs by strengthening parenting skills, self-efficacy, trust in the welfare services and children’s well-being. Central to the intervention is the family partner, who is employed in child protection and ensures that the families receive specially adapted, and supportive help. The eligibility criteria were families where at least one child is between the ages of 0–12, where children live under potentially harmful care conditions because the parent(s) are struggling in several areas of life (for example mental difficulties and physical health, substance abuse, violence and coping with emotions, high levels of conflict, breakups and economy). Further, the parents have problems with emotional and practical parenting skills, for example, parenting methods, emotional connection and understanding of the child, routines and boundaries, but they are motivated to receive help.

Families meeting the eligibility criteria were recruited by case workers. Once eligible and willing, participants received information about the study. If they agreed to participation, participants received a written consent form through an online survey platform. Once participants consented to participation, they were directed to the baseline questionnaire, after which they were randomly allocated to one of two groups: the intervention group or the control group. If allocated to the intervention group, participants received further information about the intervention by family partners. Those in the control group received ordinary services. Ordinary services differed across the child welfare offices, typically including counseling and parental guidance with the family’s caseworker. Counseling and parental guidance are the most common support measures in the Norwegian child welfare services. It typically includes counseling on different aspects of the parent-child dyad and improving the parents’ relational caregiving competence, however, the content is varied and little studied (Ljones et al., 2019). The frequency of the meetings was lower than in the Family Partner intervention and did typically not encompass practical assistance. For further information about the intervention and recruitment, please see the study protocol (Pedersen et al., 2023)

The Family Partner intervention draws upon three foundational theoretical frameworks: Bandura’s concept of self-efficacy (Bandura, 1986), emphasizing the influence of one’s confidence in their own abilities on success likelihood; Bronfenbrenner’s ecological perspective on family as a pivotal setting for human growth (Bronfenbrenner, 2005); and Ablon’s model advocating the effectiveness of a cooperative problem-solving strategy (Greene et al., 2003).

The intervention consists of: (a) parental training; (b) home visitations; (c) practical assistance; (d) a measurement feedback system including monthly scoring schemes; (e) an emphasis on a therapeutic relationship with parents to create trust; and (f) coordination of services. Families, in collaboration with family partners, complete monthly scoring schemes, utilized to identify the primary areas of focus for the intervention, including housing, school/kindergarten, parenting practices, mental health, homework, family activities, social connections, and the role of the family partner. The estimated duration of the intervention is 9 months, but it can be extended by 3 months if considered beneficial to the families. The intervention period is divided into a 3-month start-up phase, a 3–6-month working phase and a 3-month finalizing phase. Throughout the intervention the total engagement varies from 50 to 100 sessions and family partners commit to provide an average of five weekly service hours to each family, with a higher frequency of service hours during the initial phase and a gradual decrease toward the final phase. In addition to home visits, conversations can also take place via text messages and phone calls.

The primary objective of the pilot trial was acceptability of Family Partner, by studying retention to determine feasibility of a large-scale RCT. The secondary objective was to study outcomes on self-efficacy, sense of control and mental health. Before the study began, two family partners were recruited for each child welfare service. Each family partner has the primary responsibility for five families but collaborates on the efforts in all the families. Some of these family partners were recruited internally, meaning that they changed positions within the service, while others were recruited externally. The requirements for the family partners were suitable professional training (social work or child protection) and experience from frontline services, work with families and personal suitability. Their professional background varied, with some having longer intervention experience and more experience with casework. Family partners received a two-day course before starting work. Additionally, Family partners were provided with guidance on demand about specific cases.

Materials and methods

The qualitative study

The qualitative data used is a part of a study utilizing a mixed methods approach, namely a pilot-randomized study evaluating Family Partner, a home-visitation intervention implemented in the Norwegian child welfare services. To study the implementation of the intervention, qualitative interviews were conducted on several occasions during the trial. The qualitative interviews were particularly important to capture experiences from a broad set of those delivering the intervention, but also managers and stakeholders’ experiences.

The pilot trial was designed as an individually randomized study carried out in three Norwegian municipal child welfare offices between years 2021 and 2024. The municipalities were of varying size, with 18,000, 27,000 and 43,000 inhabitants, with small, medium-sized and large child welfare offices. Every child welfare office employed two full-time family partners, each supported by a coordinator who was responsible for dedicating half of their time to assisting the family partners and facilitating communication within the office. In both small and large offices, the family partners were integrated into the teams that provided families with parental guidance and additional support, with the coordinator serving as the team leader. Conversely, in the medium-sized office, the family partners functioned as a separate entity, not aligned with any team and were overseen by a coordinator who did not partake in office leadership.

Data in this study consists of 29 qualitative in-depth interviews with case workers, managers and family partners in child welfare services, other stakeholders, as well focus-group interviews with family partners. To capture a broad range of experiences and understandings of the Family partner intervention, managers purposively selected case workers based on their familiarity and involvement with the project and their experience with recruitment.

Interviewees were informed about the research project prior to participation, gave their consent to participate, and were informed about their right to withdraw from the study. The in-depth individual and focus group interviews that lasted between 50–75 minutes were carried out by the authors of this study. The focus-group interviews were carried out at four different time points, evenly spread throughout the trial. We decided to conduct both focus group interviews and in-depth interviews with family partners to capture generalized experiences across the teams, while also providingfamily partners the opportunity to disclose information on a one-to-one basis. While the focus group interviews primarily dealt with experiences related to delivering the program, the individual interviews specifically focused on experiences with recruitment, randomization, and screening. All interviews were taped, transcribed by a research assistant, and later translated to English by the first author.

Data analyses

The qualitative data were analyzed through an inductive and semantic thematic analysis (Braun & Clarke, 2012). An inductive approach in thematic analysis is considered a “bottom-up” approach, where the analysis follows patterns in the data, meaning that the analysis does not adhere to a predetermined framework. The data in this study were analyzed in six stages and coded. The results were continuously discussed within the research group. The steps in the analysis of in-depth interviews included: (1) becoming familiar with the data, (2) generating initial codes, (3) searching for themes, (4) reviewing themes, (5) defining and naming themes, and (6) producing findings.

In the initial phase, we read the transcriptions to become acquainted with its contents. Based on this and insights from the interviews, we generated initial codes, highlighting references to challenges and facilitators; specifically related to codes such as randomization, funding, time management, ethics etc. These codes were then consolidated into themes, and the themes were reviewed in relation to the introduced codes and the dataset. Notably, the process of analyzing the data was not linear, meaning that we reverted to previous themes and discussed them. Finally, after reviewing and discussing the themes, we dived our results into four overarching themes.

Results

Based on the interviews we identified one facilitator and four challenges related to conducting RCTs in child welfare settings. A facilitator for implementation was the strong belief in the intervention among staff, while challenges included ethical concerns regarding recruitment and information sharing; resource management challenges, emotional strain among child welfare workers and challenges with families randomized to the control group receiving interventions similar to the Family Partner.

The beliefs in the intervention

Our interviews revealed a high level of acceptance for the intervention and beliefs in its effectiveness for the families, which served as a motivational factor among those recruiting and those delivering the Family Partner intervention. Flexibility, trust, and coordination between services were identified as favorable key elements of the intervention. Moreover, the possibility to practice “traditional social work” by allowing individual approaches that are typically not possible in ordinary services. Some family partners had previously been caseworkers in child welfare, but the beliefs seemed to be similar among those who had and those without previous casework experience. Overall, family partners believed that the flexibility of the intervention allowed them to focus their resources on the family rather than on reporting and standardized routines. One Family Partner expressed the following:

…I think, actually, the whole child welfare field could benefit from thinking outside the box, if I can put it that way. Because it’s become very boxed in and rigid. There’s a lot of stress. And you don’t have time to lift your gaze and think either visionary or differently. There seem to be a lot of rules and routines and procedures for how things should be done when working as a contact person or family counselor. (Family partner)

Case workers in the child welfare system were responsible for recruiting participants and served as the families’ contact person when meeting external service providers such as social services or schools. Case workers perceived family partners as a positive addition to the existing services. Reduced reliance on external services was highlighted as a relief in their daily work. Case workers’ tasks often involve obtaining assistance from external agencies, which can result in them having less knowledge about what is happening with families and less opportunity for gradual reduction of intervention. In the context of standard care practices, case workers are also involved in conducting home visitations and administering parent training. However, they often do not possess the requisite time and capacity to offer these services with the same level of thoroughness and regularity as family partners do. Furthermore, owing to time constraints there is a tendency to outsource these services. With Family Partner being delivered in-house within the services, knowledge about the families’ challenges and needs benefits both the service and the family. One case worker said the following:

I am happy to take cases where there is a family partner because then you get a little more time. And at the same time, you know they [the families] are getting close follow-up. So, that’s very good. There are many cases where you need close contact. And if it’s only once in a while, you can’t follow up well enough, and you won’t gain their trust either. What’s also great about Family Partner is that it lasts over time. I’ve seen in some cases that it takes a long time for many before they dare to actually open up properly about what the challenges are. (case worker)

The findings show that case workers and stakeholders are positive toward the Family Partner intervention, specifically highlighting its flexibility, trust-building potential, and the ability to improve the quality of services delivered to families within child welfare services.

Ethical concerns in participant recruitment and information sharing

Screening and participant recruitment to the pilot trial were conducted by the case workers. Whenever a child welfare service received a new case involving a child under the age of 12, they would assess each case to determine whether it met the study’s inclusion criteria. After the screening, potential participants were approached by their case worker to ask whether they wanted to take part in a study in which they could be randomly assigned to one of two groups: one receiving standard follow-up or another receiving a home-visitation intervention. Delegating the recruitment process to practitioners arguably offers several advantages, such as the clinical expertise applied during recruitment and their training in handling potential adverse experiences associated with becoming a new child welfare client. However, our findings also reveal that entrusting case workers with screening and recruitment responsibilities presents several challenges.

Each child welfare office is organized within different divisions and types of follow-up services. To prepare case workers for their involvement in the study, the researchers provided informational materials and conducted site visits to deliver presentations on the principles of RCT design as well as the reasons and methods for recruiting participants into the intervention. However, interviews with case workers responsible for recruitment highlighted a range of challenges. A prevalent theme was the uncertainty regarding information. While case workers may be deeply concerned about their clients’ situations, clients are rarely obligated to engage with child welfare services’ follow-up procedures. Therefore, their willingness to participate is a crucial factor for case workers. Typically, this involves informing clients about the type of intervention they may receive, its scope, and its goals. In an RCT design, this presented a challenge because case workers were unaware of which group their clients might be allocated to. Hence, when recruiting families, they could not “sell” the Family Partner intervention, rather they would have to motivate the families to participate in a research project where they might get a home-visitation intervention or ordinary services. Our data shows that some case workers experience this situation as demanding and difficult to navigate in with the families involved. One case worker put it like this:

And then, in child welfare, we are trained to think about interventions quickly, right? What can help this family? We start that early in the assessment process. When we gather information and assess, we always have in mind what potential interventions could be, and we discuss openly with families about it so they can understand our work better and what we can offer. We try to make it less intimidating and be transparent throughout, being clear that we haven’t concluded anything yet, but we are considering various options. So, when we can’t present the family partner right away, we often talk about many other types of interventions. Then, when we think about Family Partner, we might have discussed other interventions and partially presented them, so it becomes a bit confusing, like, ‘Weren’t we in agreement about this? What do we think now?’ So, it can get a bit complicated. If you understand what I mean, it becomes a bit… It would be much simpler if we could introduce the concept of the family partner right from the beginning. (case worker)

Similarly, another case worker expressed ethical concerns regarding the design of the study. They mention that not providing the client with an appropriate amount of information regarding the study resulted in difficulties with recruitment:

… one of the challenges was when we had to promote research without necessarily mentioning the Family Partner aspect. It can feel a bit awkward when suddenly we say, ‘We’re not going to know anything about what you answer, and it’s completely outside of our control,’ and then they respond, and we come back and say, ‘Yes, because you answered this way, you have been selected,’ right? That can create a bit of discomfort, not necessarily dissatisfaction, but it feels ethically uneasy and not being clear about it. However, I think we’ve resolved that issue with the information we provide now, which clearly states that as part of the research, participants might be selected to try a new type of intervention, and it’s entirely voluntary. (case worker)

The ethical concerns regarding randomization were not necessarily about assigning families to an untested intervention. Rather, they may have been centered on families not knowing which intervention they would receive, and the potential disappointment of getting “standard practice” instead of the family partner intervention. While delegating the recruitment process to case workers offers advantages like their clinical expertise, it also presents challenges. Case workers faced difficulties related to the study’s design, particularly in situations where they could not immediately introduce the specific intervention being studied, causing confusion among potential participants. Notably, an element that might have caused additional confusion is the shared name of the intervention and the research project, that is Family Partner. The shared name may have led to additional confusion as both groups receive information from those delivering Family Partner and researchers studying the intervention.

Nine months into the pilot trial, we adjusted the recruitment protocol to amend some of the case workers ethical concerns and increase the number of recruitments. Instead of introducing Family Partner, case workers asked the family to participate in a research project about their experiences with the child welfare services. After they had consented to participate and had been randomized to intervention or control, the case worker could talk about which follow-up they would receive. If the family were in the intervention group, they would be offered Family Partner, and the case worker could talk about what the follow-up would entail. With this protocol, the case worker did not have to introduce any uncertainty to the families about what kind of follow-up they would receive.

Resource management challenges in randomization

In the pilot trial, each family partner could be responsible for a portfolio of up to five families. These families could vary in terms of “difficulty,” meaning that some demanded more time and resources than others. Family partners relied on case workers to refer families to the intervention. Once assigned to the intervention group, Family Partners would initiate contact with the families.

For the randomization procedure, we employed a 1:1 ratio. However, this design presented specific challenges for child welfare offices. In this office, the first 7 participants were randomized to the control group, leaving them with two family partners and no families to assist. Managers from this office expressed that this situation posed economic challenges. It is a well-known issue that case workers often face time and cost constraints; therefore, having two workers with empty portfolios over time was considered unfeasible. Family partners were consequently assigned to other cases, which resulted in two main issues. The first issue was the potential for contamination, and the second was their limited capacity. Once they were engaged with other clients, they expressed that they had fewer resources available to accommodate new families when they were subsequently randomized to the control group. This is further illustrated by one of the managers from this service:

Yes, the experience has been that it has been difficult to deal with this randomization. Because we need to be able to find the right intervention when the time comes, so to speak, right? I can’t choose or wait for us to be randomized or wait for us to have capacity… So, when the investigation or follow-up comes and discusses cases with me that need interventions, we start with a preliminary clarification: ‘Is it an extensive intervention? Could Family Partner be a fit, or could it not?’ So, you already start thinking: ‘Well, they have capacity, so we can apply. They don’t have capacity, so we can’t.’ And it’s not certain they’ll get it. So, this has been a troublesome process for us. We didn’t get anyone for a long time, right? Initially, we got two, and then it’s been almost three quarters of a year before we got two more. And I haven’t been able to keep those two for so long without giving them other tasks. So now I see that they’re already full, so some new cases that we could have sent for randomization, I won’t take that chance because I don’t have the possibility to give them Family Partner if they ended up in the intervention group. (manager)

As indicated in the statement above, the uneven distribution of participants in the two arms of the study strained available resources, specifically in the largest child welfare office. Consequently, family partners were allocated to different families, filling up their portfolios. This, in turn, created challenges in recruiting participants due to capacity issues for Family Partners. It also meant that family partners followed up participants in the control group. They were instructed by their manager not to use the Family Partner manual, but this created a risk of contamination. The manager had to choose the best for their organization, rather than follow the research protocol.

Emotional strains in response to randomization

Our results suggest that randomizing families in child welfare services have several practical implications. Our qualitative material revealed that they were both practically challenging for case workers and emotionally strained them over time. For instance, some case workers found it straining to spend resources on recruiting participants to the study, only to have them randomized to the control group. This resulted in not only extra work because they had to find new services for those randomized to the control group, but also led to disappointment over time. Many case workers and family partners hold strong convictions about the positive potential of Family Partner. When families are allocated to the control group, this was a challenge for several of the practitioners involved in the recruitment process. One case worker expressed it like this:

… I think it has been a bit difficult in the sense that I have had several cases where I really felt that they needed a family partner. And then [they were placed in the] control group. And that does affect motivation to recruit, I have to be honest about that. Also, it’s sometimes the case that the families where I think there is the greatest need for a family partner, well, there is something, there can often be a lack of understanding, a lack of initiative from parents, right? They might not be the ones most eager to participate in research and to answer questionnaires and such. It’s not necessarily the easiest group to engage with in that way. (case worker)

Another case worker expressed similar difficulties regarding the randomization procedure:

I think it has been very useful the way we’ve done it now, not just conducting research but also implementing some interventions. We are contributing to the development of something that could be useful for other child welfare services in the long run… However, when it comes to research, it’s challenging. Randomization is necessary to assess the effectiveness, but for individual caseworkers working on it, it can affect their motivation. I think we have a caseworker who didn’t have two of their cases selected, and that might have had an impact on their motivation. (case worker)

The difficulties regarding randomization were shared by a manager as well. Due to several cases in a row being randomized to the control group, the manager was left with less work compared to their other colleagues, which they felt was difficult.

No, some did get it, right. The atmosphere initially was good and positive. We wanted this and found it exciting. Then it had a bit of a dip because, well, no one is getting this offer, so it became quite challenging. People lost their enthusiasm a bit, you know, thinking, “Well, no cases are coming in. We need something to do. These are [quiet] days. We need more to fill our time.” It was the same for me in that regard. I saw my colleagues’ caseloads filling up, and they had two cases while others were overloaded. So, I had to distribute the cases. We started to wonder; can we manage this? It’s becoming difficult. (manager)

There were, however, variations among the case workers. For one, employing RCT designs is seen as an important aspect of uncovering new knowledge, something that might outweigh the potential downsides.

I think that’s how it has to be done. I haven’t had the thought that it shouldn’t be this way. I believe randomization is necessary. I think the child welfare service has struggled with it more. They had the wrong approach initially when promoting the family partner, and then not getting it led to disappointment. They figured it out around Christmas and realized, “Yes, no, we’re selling research here. We’ll see if there’s a family partner afterward.” That made it easier for them as well. (caseworker)

In summary, these findings revealed several challenges related to randomization, specifically tension between research methodology and practitioners’ beliefs in the intervention’s effectiveness, and the feeling of withholding those interventions for families in need, thus underscoring the need to consider types of randomization procedures in child welfare settings. This also illustrates that even though the staff’s beliefs in the intervention generally is a strong facilitator for implementation success, challenges that can arise when practitioners with a strong belief in the intervention are tasked with both recruiting participants and deliver the outcome of randomization to the client, especially in situations where the outcome is perceived as disappointing.

Contamination between intervention and control group participants

Through workshops and qualitative interviews, we observed contamination of the Family Partner intervention in one site, the large child welfare office, where the influence of the Family Partner intervention extends from the intervention group to the control group. As described earlier, several families ended up in the control group during the initial recruitment phase. This led to family partners being put on hold, because the manager found it difficult to prioritize the research at the expense of the rest of the organization’s needs. Family partners were therefore used to follow up on some families in the control group. In these situations, it was not the Family Partner methodology that was delivered to the families, but rather similar intensive in-home support. A manager in the child welfare office stated:

…We have an almost identical intervention that we offer outside of Family Partner, which we call IMT. It’s about intensive and targeted environmental therapy. They’re almost identical, except maybe there’s a more detailed methodology behind the intervention we own. So in that sense, there’s not much difference…But IMT, is it also delivered by family partners now, or how is it done?…Yes, it is. And it’s a bit unfortunate in a way because it becomes so similar…But there’s also something about timing. It was the people who had the capacity. It was their expertise that we needed, so they were chosen. Because I had to use the people I have and the expertise they possess. (manager)

Similarly, one of the family partners delivering the intervention stated this:

Yes, I don’t have a complete overview of who ended up in the control group. [name] would have that information. But we are indeed working on both aspects. There was a long time when we weren’t doing that, and as a result, we didn’t have cases. So, the rest of the team was overworked, and we didn’t have anything to do. That’s when [name] assigned us to IMT cases as well. (Family partner)

When further asked about the difference between intensive environmental therapy (IMT) and Family partner, the family partner responds: When asked about the difference between IMT and Family partner, one caseworker responds:

I think it’s unfortunate, actually. And for me, it’s somewhat… We have made some changes to the phases in the family partner program, but I feel like we’re doing pretty much the same thing. Trying to differentiate between this and that, but the template isn’t very different either because your template is heavily based on the IMT. And IMT, in turn, is based on that project, and both IMT and family partner have learned from things in that project. So, I feel like I’m getting a bit confused, you know? (caseworker)

In another child welfare office, similar concerns were shared: ”Yes, and the problem is that we say something like, “We see that you’re struggling, and we want to offer you this” We did that in a case last summer when the criteria were slightly different. But then it turns out they don’t get it; they end up in the control group. So, we’re kind of shooting ourselves in the foot because we have to come up with a similar service. Because we can’t say that we’re very concerned and believe you need to be part of this, but you weren’t selected, so you don’t get help. So, we have to offer something, not necessarily the family partner, but something similar in a way” (case worker)

In the smallest of the municipalities, they were convinced of the positive effects of the Family Partner program and attributed these effects to the combination of parental guidance and environment-centered therapy. By the end of the intervention period, they had initiated a hiring process to employ more caseworkers capable of delivering this blend of services to families. It remains uncertain whether the new caseworkers would also support families in the control group, but it is a possibility.

As seen in these findings, there were instances where families initially assigned to the control group ended up receiving services that closely resembled the Family Partner intervention. This happened due to the ethical obligation of child welfare services to provide assistance, and a possible lack of understanding from the office managers of the logic of a randomized controlled trial. While different programs were delivered to the treatment and control group, both interventions held common elements, and were sometimes delivered by family artners to both the control and intervention group.

Discussion

This study aimed to describe the experiences, facilitators and challenges faced when carrying out a pilot trial of a home-visitation program at three child welfare offices across Norway. Drawing on qualitative data from a process evaluation, our results uncovered four overarching challenges: 1) ethical concerns in participant recruitment and information sharing, 2) resource management challenges in randomization, 3) emotional strains in response to randomization, and 4) contamination between the intervention and control participants. We also identified a facilitator, that was the staff’s strong beliefs in the intervention.

We found that social workers and stakeholders were initially positive and informed regarding the idea of randomization. Still, during the recruitment phase, the case workers and stakeholders identified challenges related to the randomization procedure. These challenges ranged from ethical concerns surrounding what and how much information the caseworker could share with the potential participant prior to participation in the research project, lack of motivation due to families being allocated to the control group and challenges related to resource allocation. These results align with previous research carried out in similar contexts (Dixon et al., 2014; Jaramillo et al., 2023; McLaughlin, 2012; Mezey et al., 2015; Oakley et al., 2003; Oliveira et al., 2022). For instance, the study by Oakley et al. (2006) highlighted that child welfare workers may face ethical dilemmas with randomization, feeling uneasy when individuals or families – with whom they are directly engaged and believe could benefit from a new intervention – are randomized to the control group. In one of the few Norwegian RCT studies in child welfare services, Kirkøen et al. (2023) found that child welfare workers experienced profound emotional reactions when families facing numerous challenges were assigned to the control group.

Delegating the screening process to case workers, which we did in the pilot trial, may lead to emotional reactions stemming from the challenge of balancing the dual roles of “researcher” and “helper.” This dynamic could potentially hinder the pace of recruitment and challenge the exchange of information between recruiters and researchers. Similarly, this discomfort may also reflect the case workers belief in the intervention, and the families’ potential disappointment of being randomized to the control group. While ethical concerns regarding the utilization of RCT designs are not unexpected (Mezey et al., 2015), they may also reflect a level of skepticism toward the concept of evidence-based practices in the field of social work and child protection (Finne, 2019; Finne & Malmberg-Heimonen, 2023).

Addressing these ethical concerns and challenges is possible through several strategies. First, refining the recruitment procedures and reducing the responsibilities of practitioners in the recruitment and screening processes can alleviate some of the ethical tensions. The scientific research team should consider screening and recruiting participants. After randomization, the child welfare workers can offer treatment in accordance with the allocation. Moreover, it is imperative to explain the rationale behind random allocation clearly to all stakeholders involved. Spending ample time in discussions with stakeholders can foster their support, emphasizing the importance of making participants in the control group feel valued and well-informed about their crucial role in the research. Such approaches aim not only to mitigate the ethical and practical challenges identified but also to leverage the initial positive reception toward randomization by social workers and stakeholders, enhancing the overall efficacy and ethical conduct of RCT designs in child welfare services (Oakley et al., 2003).

One notable finding of this study relates to the type of randomization procedure. This pilot trial was organized as a multisite study, meaning that intervention was delivered and studied across three different sites. Simple randomization relies on chance, and when dealing with small groups, imbalances between the groups may occur. In other words, simple randomization techniques may lead to unequal group sizes, especially with a small sample size, even if the procedure is carried out correctly (Shibasaki & Martins, 2018). This study demonstrates that imbalance between groups, specifically many participants being allocated to the control group in the startup phase, affected both the case workers motivation, and issues regarding resource management. One solution to consider in similar instances is block randomization to increase the probability of equal number of participants in each group (Efird, 2011). There are, however, challenges associated with this approach as well, such as researchers or subjects being able to predict the allocation outcome (Berger, 2016). A solution to handle the risk of too many participants being allocated to the control group at the beginning of the trial is to ensure a large number of cases are available prior to the trial launch. This ensures that those delivering the intervention are fully engaged with an adequate caseload from the outset, thereby circumventing any initial period of low activity. For researchers opting to forego randomization while still aiming for causal inference, the adoption of longitudinal quasi-experimental designs presents a feasible strategy (Rogers, 1999).

Moreover, our findings indicated the presence of contamination, where the influence of the Family Partner intervention extended partially from the intervention group to the control group. Although distinct interventions were administered, both shared similar, in fact identical characteristics as described by one participant, and were in one site, delivered by the same individuals for both groups. Considering that the child welfare services are obliged to provide the necessary follow up to all families in need of care, the control group might get equal or even more comprehensive services than the intervention group. In a future full-scale RCT, one solution is to predefine the measures given to the control group. This is to mitigate the risk of contamination and consequently, the risk of underestimating the treatment effect of Family Partner.

Furthermore, the enthusiasm for the intervention can serve as a double-edged sword. While we experienced that the belief in the intervention was a crucial factor for recruitment, it may also pose a challenge when participants are allocated to the control group, as they might perceive the services as substandard. For instance, one caseworker noted the need to ‘create a similar service” for a family allocated to the control group. These findings underscore the importance of not underestimating the necessity of field preparation prior to starting a study.

To address this concern, one potential solution is to employ cluster-randomized designs, which involve randomizing groups of individuals rather than individual participants. In our context, this approach offers the advantage of allowing the intervention group to recruit participants for the intervention without being concerned by the randomization process. One approach in a cluster-randomized design may be to rely on pre-established allocation lists instead of randomization. In a Norwegian study of a comprehensive follow-up of low-income families the researchers were compelled to randomly assign families from lists of families that had been predefined as eligible (Malmberg-Heimonen et al., 2017). This randomization process was carried out without encountering obstacles or resistance from the office staff. Cluster randomization of child welfare offices removes both the randomization process from the case workers, as well as it reduces the chance of contamination. However, it’s important to note that cluster randomization is also not without its challenges. For instance, case workers may exhibit reduced motivation to recruit if their site is assigned to the control group, potentially challenging the internal validity of the study (Dron et al., 2021).

In summary, our study underscores the inherent challenges of conducting RCTs within child welfare services. Furthermore it underscores the necessity of carrying out pilot-studies to examine unchartered territory. These challenges contribute to the limited availability of evidence-based approaches in this field, as noted in previous research (Bragdø-Ellenes & Torjesen, 2020; Haugevik & Neumann, 2020; Ljones et al., 2019; Otterlei et al., 2021). To align our practices with the Norwegian child welfare act and provide evidence-based services, it is important to explore research design that are appropriate to assess causal relationships. This study has shed light on some of these challenges and proposed potential strategies to address them. However, to facilitate the delivery of evidence-based practice, more research is needed to better understand the complexities of carrying out RCTs in child welfare settings.

Study limitations

This study has some limitations. First, the qualitative material represents a small number of caseworkers. Including a larger number of caseworkers would allow us to further assess the acceptability and attitudes toward the Family Partner intervention, recruitment, and study design. On the other hand, by including a broad range of experiences with the implementation family partner, we were able to study challenges faced from multiple perspectives. Although all interviewed caseworkers were familiar with the Family Partner intervention, one limitation of this study is that we did not track the consistency in caseload management among family partner client cases, thus complicating the assessment of potential consequences.

Conclusion and implications

The results of this study shed light on the challenges encountered when carrying out RCTs in child welfare settings. The findings highlight that one of the main obstacles faced by child welfare workers was determining what and how much information should be shared prior to participation in the research project, along with ethical and practical considerations after randomization. Reflecting on the challenges we have identified during the pilot of this randomized trial in child welfare services, it is vital for future studies to acknowledge these obstacles. By strategically designing studies to reduce such barriers and developing comprehensive plans to address foreseeable difficulties, researchers can more effectively tackle the complexities involved in conducting RCTs within child welfare context.

Availability of data and materials

The data analyzed during the current study are available from the corresponding author on reasonable request.

Ethics approval and consent to participate

The Norwegian Agency for Shared Services in Education and Research approved this study (804402). All methods were performed in accordance with the relevant guidelines and regulations. Informed consent was obtained from all subjects and/or their legal guardian(s) before the start of this study.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The research council of Norway funded this project [320080].