Implementing a Residential Dialectical Behavior Therapy Informed Treatment Model to Improve Adolescent Mental Health: Feasibility, Fidelity, and Acceptability

ABSTRACT

Dialectical Behavior Therapy (DBT) is an outpatient treatment that addresses severe emotional and relational difficulties and is successfully applied across diagnoses, populations, and settings. In this feasibility study, we examined a residential DBT-informed treatment model in Norway, the Care and Development Model (CDM), aimed to improve adolescent mental health outcomes. Study participants included 42 adolescents (22 males and 20 females) with a mean age of 15.13 years (SD = 1.41, range 13–17 years), as well as the adolescents’ caregivers and residential staff. Data collection lasted from May 2018 until May 2021 and combined standardized self-report measures of program feasibility and acceptability, including treatment satisfaction and alliance, as well as customized program fidelity measures combining on-site observation and interviews with adolescents and staff, protocol data on treatment characteristics, and self-report checklists for treatment adherence. Results indicate that residential staff consider the implementation of CDM appropriate, but also distinct challenges were mentioned regarding the population and setting. Moderate to high satisfaction and alliance levels were reported by both adolescents and therapists. The CDM program shows promise and may be feasible and acceptable, although considerable effort to implement it is required. Implications for practice and research are discussed.

KEYWORDS:

• evidence-informed intervention
• feasibility
• acceptability
• dialectical behavior therapy
• adolescent

Introduction

Numerous studies have shown that a substantial number of adolescents in out-of-home care have experienced maltreatment, trauma exposure and severe psychosocial strains (Briggs et al., 2012; Jozefiak et al., 2016), contributing to the development of mental health difficulties that require targeted intervention (e.g., Bronsard et al., 2016; Li et al., 2019). This group is also overrepresented in statistics for poorer life-outcomes, including low educational attainment, high unemployment, substance abuse problems, criminal activities, and increased mortality rates (Gypen et al., 2017; Kristofersen & Sverdrup, 2013). In recent years, interest in developing and disseminating effective treatment interventions that target transdiagnostic mental health difficulties and behavioral problems has gained speed (James, 2017; Klodnick et al., 2020). One such intervention with an extensive evidence-base is Dialectical Behavior Therapy (DBT; Linehan, 1993), a modular skills and principle-based treatment that addresses various behavioral targets commonly found in the residential adolescent population, such as high impulsivity, comorbidity and severe emotion regulation difficulties (Duppong Hurley et al., 2017; Frensch & Cameron, 2002).

Development of the Care and Development Model (CDM)

The Care and Development Model (CDM; Omsorgs- og Endringsmodellen in Norwegian) is a comprehensive, milieu-based residential program intended to improve long-term mental health outcomes for adolescents under the protection of Norwegian governmental Child Welfare Services (CWS). The program aims to enhance general residential care for adolescents placed in state institutions by applying fundamental skills training to improve emotion regulation, relational competence, and various prevalent mental health difficulties (Jozefiak et al., 2016). The program-model integrates principles and practices derived from standard DBT (Linehan, 1993) and the DBT adaptation for adolescents (DBT-A; Rathus & Miller, 2014). Notably, DBT has a demonstrated pedigree as a transdiagnostic treatment, including adaptations for adolescents in outpatient (e.g., Mehlum et al., 2014) and residential care settings (e.g., Klodnick et al., 2020; McCredie et al., 2017).

While it is feasible to utilize foundational principles of standard DBT as well as age-appropriate teaching modules from DBT-A, in our study, practice adjustments for the residential setting were necessary. For that reason, the CDM has integrated features from the Integrated Treatment Model (ITM; Schmidt & Salsbury, 2009), a residential DBT-adaptation including specific treatment targets often recognized in residential populations such as assaultive or program destroying behaviors (e.g., preventing skills-group training sessions by setting off the fire alarm). The ITM was originally designed for, and implemented in, a forensic setting in the Washington State Juvenile Rehabilitation Administration program in the United States (Schmidt & Salisbury, 2009), but remains to be tested in a randomized study. Consequently, the CDM had to be tailored to the Norwegian cultural and judiciary setting (e.g., no internet restrictions, not allowed to prevent adolescents from leaving the institution). See, Table 1 for a detailed view of treatment functions, methods and modes of DBT-A in comparison to CDM.

Table 1. Functions, methods and modes of DBT-A versus CDM.

Function	Methods/Strategies	Modes in DBT-A	Modes in CDM	Delivery in CDM
Enhance client motivation and use of skillful behavior	Commitment and exposure strategies, behavioral assessment, chain analysis, contingency management, cognitive modification.	Individual sessions	Individual sessions	Weekly individual sessions with individual counselor. Length of 45–60 minutes.
Enhance skills acquisition and capabilities	Modeling, behavioral rehearsal, psychoeducation, coaching, feedback, homework.	Skills group training with caregivers	Skills group training for adolescents	120 minutes weekly group skills training incorporating a five-module format over a six-week rotation.
Generalization to environment	Phone consultation, homework, in vivo and crisis interventions, relationship repair.	Phone consultation	Milieu	Trained staff reinforce skills. Token economies support skillful behavior.
Enhance therapist capabilities and motivation.	Weekly consultation, treatment manuals, supervision, continuous training and acquiring knowledge.	Consultation team	Consultation team	Two weekly consultation team meetings (120 minutes total). Weekly supervision for on-site coordinators.
Structure the environment	Case management, family interventions.	Case management	Case management	Case management by staff.

Based on Salsman & Linehan (2006), Rathus and Miller (2014), Schmidt & Russo (2019)

However, successfully transferring treatment models from research settings to new practice environments is not a straightforward matter. While the defined structure and clarity of DBT fits well within residential confines, a recognized standard approach does not exist (Fox, 2019). Debates have circled around whether evidence-based interventions should be implemented with maximum fidelity, or whether evidence-informed models should even be encouraged (Durlak & DuPre, 2008); furthermore, interventions often require substantial adaptations to fit specific settings (McHugh & Barlow, 2012). Therefore, before examining the effects of DBT-informed interventions, we need to know if the intervention is suitable and accepted by staff providers and adolescent residents. There is very limited comparative research on DBT-informed residential treatment for adolescents (e.g., Klodnick et al., 2020; McCredie et al., 2017), and to our knowledge, this is the first study of a cross-site comprehensive intervention reporting on a combination of these constructs. Therefore, this study extends the growing field of implementation research for evidence-informed program models in real-world settings.

Aspects of Feasibility, Fidelity, and Acceptability

An important feature of implementing an evidence-informed intervention in a new setting includes exploring whether the intervention is feasible for further implementation (Bowen et al., 2009). Providers may struggle to deliver DBT at a consistently high degree of technical skill and proficiency (Bedics & McKinley, 2020) especially in cross-site implementations (Borrelli, 2011). Therefore, intervention element fidelity should be assessed, including the extent to which various model elements are implemented (Perepletchikova & Kazdin, 2005) as well as dimensions of therapist adherence (i.e. the delivery of the intervention’s core elements) and competence (i.e. whether elements are performed with sophisticated clinical and teaching skills; Forgatch et al., 2005).

Residential adolescents often form poorer working alliances (e.g., Korchmaros & Stevens, 2014), indicative of a lower degree of agreement, collaboration, respect, and trust within the therapeutic alliance (Horvath et al., 2011). Alliance is modestly associated with adolescent outcomes (Shirk & Karver, 2003) and treatment satisfaction (Hawley & Weisz, 2005; Karver et al., 2018). Relatedly, a working alliance between adolescents and providers is a prerequisite for ensuring adolescent engagement and commitment to time-consuming treatment tasks. However, mental health difficulties and variations in treatment motivation may present significant challenges (e.g., obtaining commitment, providing intended dosage). This is especially demanding when adolescents are under mandatory placements. Furthermore, adolescent perception of provider fidelity is predictive of adolescent outcomes, where fidelity is highly correlated with adolescents alliance ratings (Duppong Hurley et al., 2017). Nevertheless, these constructs are less explored in DBT-informed residential treatment, and we would argue that examinations of residential programs need to include descriptions of how the intervention is delivered and accepted by adolescents, caregivers, and providers.

Aims of the Present Study

Implementation factors must be examined before determining the CDM’s effectiveness. In this study, we aimed to describe whether the model is a good fit for the residential youth population and setting. Therefore, we evaluated the CDM’s a) feasibility (through providers’ ratings of appropriateness), b) fidelity (through protocol fidelity, treatment adherence and provider competence) and c) acceptability (through providers’ acceptability, participants’ treatment satisfaction, and adolescent’s and provider’s therapeutic alliance).

Method

Implementation Setting

The Office for Children, Youth, and Family Affairs (Bufetat) is a state agency under the Ministry of Children and Families and is responsible for providing residential services within the Norwegian child welfare system (https://www.bufdir.no/en). Residential care is provided in government CWS facilities when out-of-home care is necessary and foster home placement is not an option. All placements are made under the provisions of the Child Welfare Act, with or without the adolescent’s own consent. As formal detention and incarceration of youth is used sparingly, adolescents under the age of 16 with high-risk criminal behavior are placed in CWS institutions. The present study started in 2018 with six regular youth care institutions and one institution specialized in serious behavioral problems located in Northern Norway. In 2019, one regular care institution was discontinued. The primary focus of all institutions is to improve adolescent mental health through high quality evidence-informed care. Antecedent to implementing the CDM, the institutions had incorporated practices from milieu-based Cognitive Behavior Therapy (e.g., individual sessions and milieu-based group activities). Although institutions were similar in terms of target groups, treatment structures and shared regional management, some had a more established CDM program at the study start. All institutions had units of four to 10 residents between 12 and 18 years old, and the average length of stay was approximately nine months. Consistent with regular practice, all institutions were open care-institutions (Bengtsson & Böcker Jacobsen, 2009) where adolescents attended regular schools and participated in leisure activities outside of the institutions.

Staff Training. The CDM foundational training included a comprehensive 10-day, 70-hour program that encompassed all modes and modules of the intervention. The training program was run by coauthor Dr. Henry Schmidt, a DBT trainer and member of the DBT- Linehan Board of Certification, who also co-developed the residential ITM intervention for the Juvenile Rehabilitation Administration in the U.S. The first CDM foundational training occurred in 2015, with additional trainings in 2018 and 2020, and supplementary two-day, 15-hour booster sessions occurred twice per year. Both trainings and boosters were delivered in-person and included didactic and interactive elements intended to increase provider familiarity and fidelity to the model (e.g., role-plays addressing treatment principles, reporting own target behaviors on diary card). Delivery was monitored through ongoing treatment provider coaching and feedback during weekly consultation team meetings. On-site coordinators led as-needed institution specific training, and staff received weekly supervision that focused on principle review and practice (Ivanoff & Marotta, 2018). In October 2020, 89.8% of staff had received the CDM foundational training, and 91.5% had participated in at least one CDM booster (M = 4.32 boosters).

Adolescents Skills Training, Monitoring Target Behavior, and Individual Sessions

Adolescents were continually enrolled into the program upon arrival. Following a “pre-treatment phase,” which focused on orientation and attaining agreement to participate, adolescents received weekly individual and skills group training sessions as well as continuous milieu treatment throughout their stay. The DBT target hierarchy was modified to include assaultive, running away, and program destroying behaviors along with suicidal/self-injurious behaviors. Adolescents were expected to track their target behaviors through completing individual and group skills diary cards prior to sessions (Linehan, 1993). Skills group training sessions were led by two residential staff members, which consisted of previous learned skills material and homework review, followed by a presentation of new skills with didactic instructions, group discussion, and illustrative activities (e.g., role-plays). The CDM curriculum utilized Norwegian translations of the DBT-A handouts (Rathus & Miller, 2014), and included interpersonal effectiveness, emotional regulation, distress tolerance, mindfulness, and Walking the Middle Path modules. Since the average length of stay was longer than the 24-week DBT-A protocol that has been previously tested in inpatient settings (e.g., McCauley et al., 2018), the skills-group curriculum was repeated twice yearly, allowing for considerable reinforcement of core concepts. Residential staff were responsible for in-the-moment skills coaching and strengthening in the milieu. All adolescents were assigned a team of three providers who worked across shifts, ensuring the availability of a team provider throughout each day. One or more of these providers delivered the individual sessions. Family sessions were not implemented during the study period. However, the institutions included caregivers in family visits or provided skills-training materials and progress updates through electronic mail or telephone contact when the adolescent had contact with either their biological or foster families.

Study Participants

Adolescent participants. Participant recruitment started in May 2018 and ended in May 2020. Adolescents were excluded from participation if they had an acute placement, poor Norwegian comprehension, previously received extensive DBT-A, received a cognitive impairment diagnosis (i.e. severe intellectual disability or autism), or showed severe symptoms of psychotic disorders. In total, 60 adolescents were invited to participate: four declined, two moved out before pre-phase, 10 were excluded from participation (four had received DBT-A, four had poor Norwegian comprehension, one had a cognitive impairment diagnosis, and one remaining adolescent had an acute placement), and two adolescents withdrew from the study after post-phase. In all, 42 adolescents (22 males and 20 females) with a mean age of 15.13 years (SD = 1.41, range 13–17 years) participated. Regarding mental health, 64.3% reported mental health and/or drug-related problems at admission and 45.2% were in concurrent contact with the Children and Young People’s Psychiatric Outpatient Services (BUP). Adolescents reported a high prevalence of suicidal thoughts (57.1%) and acts of self-harm (66.7%) within the past three months, as well as previous suicide attempts (40.5%). When outcome analyses began, post-data from 38 discharged adolescents were available for inclusion. Adolescents spent a total of 2,228.42 weeks at the institutions, giving a mean of length of stay of 53.06 weeks (SD = 30.42) with a range between 5.57–123.29 weeks.

Caregiver participants. One of the adolescent’s parents or caregivers was invited to participate in cases where the adolescent was to return to their home environment and/or receive regular parental visits. Caregivers did not actively participate in skills group training sessions. Thirteen adolescents did not have a caregiver participant for various reasons (i.e. no contact, deceased parent). In three instances, caregiver participants were the adolescent’s caseworker from the municipal CWS who had close contact with the adolescent. In total, 26 unique caregivers participated in the study. Excluding caseworkers, 23 caregivers provided information regarding the family’s demographics: 15 (65.2%) were female and 8 (34.8%) were male; 14 (60.8%) were biological mothers, five (21.74%) were biological fathers, one (4.35%) was a grandfather, and three caregivers (13.00%) did not specify their relation to the adolescent.

Staff participants. To ensure that staff participants had frequent contact with the adolescents, staff were eligible for participation if they worked more than 50% during day or evening shifts. In May 2018, 93 staff members were recruited from seven institutions: 64 regular milieu-staff, 13 on-site coordinators, and 16 residential leaders. Due to the closure of one institution, 79 staff members remained at the last staff measure in 2020: 55 regular milieu-staff, 11 on-site coordinators, and 13 residential leaders. Staff had a mean age of 42.3 years (SD = 9.6), 23 were male (35.9%) and 41 were female (64.1%), and they had various educational levels and backgrounds (e.g., child welfare, social work, education, and nursing). Staff were employed on average for 80.8 months (SD = 67.2, range 1–264 months). Forty providers (68.97%) reported that they delivered individual sessions and 39 (67.24%) reported that they facilitated skills group training sessions.

Procedure and Ethical Statement

Data were obtained from a larger project that evaluated both implementation and preliminary outcomes of the CDM. The project was approved by the Norwegian Center for Research Data (NSD) in April 2018. Data were collected via mixed methods, including protocol-data, self-report questionnaires, observations, and interviews, at various points throughout the project period from May 2018 to May 2021 (see, Table 2). Data collection included seven institutions between May 2018 to October 2019, and six remaining institutions between November 2019 to May 2021. All adolescents and caregivers were presented with study information before intake, and written consent forms were obtained. Participants 16 years of age or older and caregivers signed their own consent form, and for younger adolescents, caregivers or CWS caseworkers signed consent forms. Participants received automated text message reminders for questionnaires. An online, digital self-reporting and management program (http://checkware.com) was utilized to collect data concerning staff feasibility measures, acceptability outcomes for all study participants, and certain fidelity outcomes (e.g., checklists on individual sessions adherence, diary card completion, presence of milieu program elements). Checkware provided automatic text-message reminders, and participants utilized personalized login credentials to provide their information. Other data concerning program fidelity outcomes (e.g., consultation team and supervision conductance) were reported by leaders through institutional protocol forms using Microsoft Office Excel, which were updated weekly and submitted to the research team. All data were handled anonymously and securely according to research standards. When adolescents completed the full pre and post batteries, they were compensated with a 100 Norwegian Kroner (NOK) gift card (approx. $12) at each time point. Staff and caregiver participants were not compensated.

Table 2. Time-points for data collection.

Measure	Type	Informant	Time-point
Feasibility of Intervention Measure	Self-report	Residential Staff	Oct 2020
Intervention Appropriateness Measure	Self-report	Residential Staff	Oct 2020
Acceptability of Intervention Measure	Self-report	Residential Staff	Oct 2020
DBT Program Fidelity Scale	Observer rating	Adolescents, Residential Staff and Leaders	April 2019
CDM Program Fidelity Logs	Fidelity logs	Residential Leaders	May 2018 – June 2020
Adherence and Competence Checklist	Self-report	Residential Staff	Dec 2018 – May 2021
Milieu Mode Elements Checklist	Self-report	Adolescents and Residential Staff	Nov 2018 – April 2020

Measures

Feasibility Measures

Feasibility and Appropriateness of the CDM. The Feasibility of Intervention Measure (FIM) and The Intervention Appropriateness Measure (IAM; Weiner et al., 2017) are brief, pragmatic measures tailored to specific implementation strategies. Each scale has 4-items rated by providers on a scale from 1 (completely disagree) to 5 (completely agree). Items are summed for a total score (range between 5 to 20), where higher scores indicate greater feasibility and appropriateness. The measures have demonstrated strong internal reliability, test-retest reliability, and sensitivity to change (Weiner et al., 2017). Internal consistency in this study was strong (Cronbach’s α = .84 for FIM and Cronbach’s α = .92 for IAM).

Fidelity Measures

Measures of CDM Program Fidelity. The CDM program involves a set structure for expected frequency and duration of modes, thus examining its deliverance gives an indication of program sustainability. Therefore, detailed logs of treatment characteristics concerning conductance of consultation teams, supervision, skills group training sessions, and individual sessions were provided to examine protocol fidelity.

Client Compliance. Measures of adolescent compliance included (a) skills training session attendance, (b) individual sessions received, (c) percentage of time adolescents brought a completed diary card to individual sessions, and (d) provider’s report of adolescent skills attainment, motivation, commitment, and progress.

Therapist Adherence and Competence. To gauge the preliminary assessment of adherence, the research team developed a short checklist (yes or no response) following individual session benchmarks (see Table S1 in Supplemental materials). Two versions of the checklist were developed to reflect whether the diary card was filled out by the youth in advance. In addition, the checklist included provider self-reported competence and perceived adolescent motivation and commitment. These items were scored on a scale from 1 (strongly disagree) to 6 (strongly agree).

Measures of the Milieu Mode. In order to measure the presence of milieu specific CDM elements, a checklist was developed by the research team (see Table S2 in Supplemental materials). The seven-item measure (yes or no response) encompassed program elements (e.g., behavioral correction, skills use) for adolescents, milieu-staff, and on-site coordinators.

Program Fidelity Scale. We revised the DBT-Linehan Board of Certification DBT Program Fidelity scale (available at: https://dbt-lbc.org/index.php?page=101141) to examine CDM fidelity. Its wording was adjusted to CDM vernacular, and included 17 items concerning program, consultation team, adolescent assessment, milieu treatment, crisis management, and adaptation elements (see Table S3 in Supplemental materials). The rating of each item included reports from leaders, providers, and adolescents, as well as rater-observations at the institutions. At the time of the rating, the five institutions with adolescent participants who had consented to the study were rated on-site. Both the lead author and an independent DBT-trainer not connected to the study group rated two institutions concurrently, but independently.

Acceptability Measures

Intervention Acceptability. The Acceptability of Intervention Measure (AIM; Weiner et al., 2017) is a brief measure tailored to specific implementation strategies. The 4-item AIM was rated by staff and leaders on a 5-point Likert scale from 1 (completely disagree) to 5 (completely agree). Items are summed for a total score (range between 5 to 20), where higher scores indicate greater acceptability. Internal consistency in this study was Cronbach’s α = .95.

Adolescent Treatment Satisfaction. The Youth Satisfaction Questionnaire (YSQ; Stüntzner-Gibson et al., 1995) was completed to assess adolescents satisfaction following the intervention. We utilized the first section of this measure, a total of 5-items adapted from adult satisfaction scales (CSQ; Attkisson & Zwick, 1982; Larsen et al., 1979), as it applies more adolescent-appropriate language with simplified response alternatives. YSQ items are scored using a 3-point scale: 1 (Yes), 2 (Somewhat), to 3 (No). Internal consistency in this study was Cronbach’s α = .85.

Caregiver Treatment Satisfaction. Caregivers completed the Client Satisfaction Questionnaire (CSQ-8; Larsen et al., 1979) at adolescents’ discharge to assess their satisfaction with the intervention. The CSQ-8 items are scored on a scale from 1 to 4 (total scores range 8–32) with unique alternatives corresponding with each question. Internal consistency in this study was Cronbach’s α = .90.

Therapist Treatment Satisfaction. Each provider completed six items from the Therapist Satisfaction Inventory Index (TSI; Chorpita et al., 2015) during the adolescent’s discharge. The six items utilized from the original 16-item TSI therapist-report measure (Addis & Krasnow, 2000) included three responsiveness and three effectiveness subscale items (Cronbach’s α = .82 for responsiveness and Cronbach’s α = .81 for effectiveness; Chorpita et al., 2015). Internal consistency in this study was Cronbach’s α = .83 for responsiveness and .69 for effectiveness.

Therapeutic Alliance. Alliance was also assessed during adolescent discharge via the Therapeutic Alliance Scale for Children and Therapist revised (TASC-r; Shirk & Karver, 2011; Shirk & Saiz, 1992). The TASC-r is a 12-item measure rated on a 4-point scale ranging from 1 (not at all) to 4 (very much) with parallel versions for adolescents and providers. Internal consistency in this study was Cronbach’s α = .90 for providers and .87 for adolescents).

Data Analysis

We used descriptive statistics to summarize all protocol-data concerning conductance and attendance of supervisions, consultation teams, and individual and skills group training sessions. For the DBT Program Fidelity measure, score averages for all scales (program, consultation team, and milieu treatment), and item scores for the remaining three items (tracking of outcomes, crisis management protocol and adaptations) were performed. To check for potential differences between the two raters, independent t-tests for average scores were conducted. Bivariate correlations were also reported to examine the strength of the relation between such variables as participants’ age, satisfaction and adherence reports, while t-tests were performed to examine whether mean level of satisfaction and adherence reports differed according to participants’ gender. We used inferential statistics (e.g., repeated measures ANOVA) to examine the presence of milieu program elements across timepoints. Post hoc analyses with Bonferroni correction, adjustment for multiple comparisons, were reported.

Results

Feasibility Outcomes

Feasibility and Appropriateness of the CDM

Staff and residential leaders rated the CDM as highly feasible for the residential setting: 15.67 (SD = 2.38) for feasibility, and 16.06 (SD = 2.73) for appropriateness (maximum range between 5 to 20).

Fidelity Outcomes

Measures of Fidelity to the CDM Program

A total of 635 consultation team meetings were conducted with a mean duration of 98.16 minutes (SD = 29.07). On average, between 2.40–2.95 monthly consultation team meetings took place. In addition, a total of 313 supervision sessions were held, with up to three institutions participating within the same session. Thirty supervision sessions (8.75%) were planned, but not held (e.g., staff attending other trainings, low staff census during scheduled time). During the same time-period, 584 skills group training sessions were conducted across all institutions, giving an average of 3.35–3.72 sessions per month across all institutions. Seventy-six (11.52%) skills group-training sessions were planned, but not held, for the same reasons as supervisions. See Figure S1 in Supplemental materials for details on skills group training session duration.

The maximum number of possible delivered individual sessions was 1,980 (M = 48.29, SD = 30.53). The number of registered sessions was 410 (M= 10, SD = 14.25), indicating that 20.71% of possible sessions were given. See Figure S1 in Supplemental materials for details.

Client Compliance, Motivation and Progress

Regarding skills group training sessions, at least 75% attendance was achieved in more than 50% of sessions (see Figure S3 in Supplementary Material). As for individual sessions, individual sessions, one adolescent was discharged before registration of sessions commenced, and 15 adolescents (M_age = 14.67; nine males, six females) did not receive sessions. These adolescents had a combined length of stay of 720.27 weeks (M= 48.01 weeks, SD = 29.89). Upon closer examination of the remaining 26 adolescents (63.41%), they participated in an average of 15.77 sessions (SD = 15.18), comparative to 29.26% of possible sessions.

Additionally, adolescent compliance was examined through individual diary card completion. Out of the 402 individual sessions reported, a completed diary card was presented in 270 sessions (67.3%). The adolescents did not bring a completed diary card to 132 sessions (32.7%). See, Figure 1 for adolescent skill attainment, motivation, commitment, and progress rated by providers at individual sessions.

Figure 1. Provider ratings of adolescent skills attainment, motivation, commitment and progress.

Therapist Adherence and Competence

Following the 270 individual sessions with a completed diary card, providers completed checklist A, giving a mean score of 4.29 (SD = 1.35, range 1–6). Following the 132 sessions without a completed diary card, individual counselors completed checklist B, giving a mean score of 3.81 (SD = 1.98, range 1–8). Results indicate higher therapist adherence when a diary card is completed prior to the session. Providers rated their competence following 400 individual sessions, with the majority reporting that they strongly agree or agree (93%) with the statement that they are competent in their deliverance of CDM (M = 5.08, SD = .55).

Measures of the Milieu Mode

Five ratings were performed reporting the presence of CDM program elements in the milieu mode. In November 2018, adolescents reported a medium presence (M = 3.20, SD = 1.93), while milieu-staff reported a high presence (M= 5.40, SD = 1.17), and on-site coordinators reported a medium presence of program elements in the milieu (M= 4.17, SD = 0.94). In April 2020, adolescents reported a lower presence (M= 2.92, SD = 2.40), milieu-staff reported a similar presence (M= 5.37, SD = 1.29) and on-site coordinators reported a higher presence of program elements in the milieu (M= 5.60, SD = 0.70). Repeated measures ANOVA over five measurement times revealed that mean milieu scores differed significantly between time points for on-site coordinators (F (4, 16) = 3.19, p = .005), but not milieu-staff (F (4, 88) = 2.21, p = .94). Post-hoc tests using the Bonferroni correction revealed that the milieu mode score for on-site coordinators had significantly increased between the first (M= 4.40, SD = 5.10) and last measurement (M= 6.00, SD = .32) (p = .028), with no significant difference in between measures.

Program Fidelity Scale Measure Outcomes

Results from the revised DBT Program Fidelity scale show a total mean score of 66.8 (ranging between 50–78; max score 85) across all institutions and raters. Fidelity ratings by the two different raters only differed for two items, indicating an 94% level of agreement. See, Table 3 for details.

Table 3. Congruence with DBT program fidelity across institutions.

	Items	Item mean and range (max 5)	Overall scale means and range
Program Elements	4	4.05 (3.00–4.75)	16.20 (12–19)
Consultation team	6	3.58 (2.67–4.17)	21.50 (16–25)
Milieu-treatment	4	3.83 (2.75–4.75)	15.30 (11–19)
Adolescent assessment	1	4.8 (4–5)	-
Crisis management	1	5 (5–5)	-
Adaptation Elements	1	4 (2–5)	-
Overall Scale	17	3.93(2.94–4.59)	66.80 (50–78)

Acceptability Outcomes

Provider Acceptability of the Intervention

Staff and residential leaders (n = 79; 81%) reported on the acceptability measure. The mean score for acceptability was 17.86 (SD = 2.42), indicating that the providers viewed CDM as highly acceptable for the residential setting (maximum range between 5 and 20).

Satisfaction with the Intervention

Adolescent satisfaction level (n = 28) was high, with a mean of 7.00 (SD = 1.85) out of 9 (ranging between 3–9). Therapist satisfaction (n = 36) was also high, with a mean satisfaction level of 24.61 (SD = 2.68) out of 30 (ranging between 18 and 30). Caregiver satisfaction (n = 11) was high, with a mean level of 25.27 (SD = 4.96), out of 32 (ranging between 18 and 31). There was no significant correlation among the participants’ satisfaction levels, as well as in conjunction with the length of stay. However, younger adolescent participants reported a higher level of satisfaction (r= −.533, p= .003). T-test analyses revealed that the mean level of satisfaction also did not differ according to adolescent participant gender (p> .05 for all comparisons).

Therapeutic Alliance

Adolescents (n = 25) reported a moderate level of alliance on the TASC-r measure, with a mean of 34.00 (SD = 8.44) out of 48 (ranging between 14 and 44). Therapist alliance scores (n = 36) were similarly high, with a mean of 34.14 (SD = 5.99) out of 48 (ranging between 21 and 45). Therapeutic alliance scores were not associated with adolescents’ age, gender, nor length of stay. However, therapist-rated alliance was significantly and positively correlated with therapist (r= .442, p= .007) and adolescent (r= .395, p= .037) treatment satisfaction.

Discussion

This paper was a part of an evaluation of the CDM implementation, a cross-site DBT-informed treatment model aimed at improving outcomes for adolescents in residential care. Implementing a comprehensive, multi-modular treatment model to Norwegian cultural and judiciary residential setting is undeniably very resource intensive, and evaluation research presents invaluable insight into refining future approaches to improve adolescents’ outcomes. Overall, our results identify several strengths concerning the CDMs implementation, as well as challenges met while executing a study in adolescent residential care.

Regarding feasibility aspects, a very high percentage of providers attended CDM foundational training (89.8%) and supplemental boosters (91.5%), implying a high-perceived familiarity and competence with the model. A significant amount of training, consultation team meetings, and expert guidance was provided, substantially more than could be identified in similar studies (e.g., Klodnick et al., 2020; McCredie et al., 2017). Although providers found the CDM both feasible and appropriate for issues often faced in residential care, the overall findings reflect that the CDM is demanding to implement, both in theoretical and practical terms.

Adolescent skills group training sessions were held on average three to four times a month, a third of which lasted near the expected duration between 90–120 minutes. In contrast to Klodnick et al.’s, (2020) findings, skills group engagement was much higher than individual sessions in our study. Comparably, the generally low number of delivered individual sessions stands in contrast with the relatively high ratings of adolescent compliance indicated by both individual diary card completion and providers’ ratings of adolescent motivation (see Figure 1). Notably, some adolescents may have received pre-treatment sessions during their stay which were not properly registered as individual sessions. It is unknown whether the low number of delivered individual sessions are an indication of the specific institutions’ structural inability to adhere to the program, because adolescents declined to participate, or a combination of the two.

Individual session therapist ratings indicated a high-perceived competence, though therapists reported higher adherence when adolescents brought completed diary cards. One possible explanation may be that providers are more confident in their deliverance when the structure of the sessions are seemingly more predictable, rather than when the adolescents fail to present a diary card and the therapist has to skillfully explore possible hindrances to the adolescent’s motivation or familiarity with treatment expectations. Notably, as we do not examine changes in therapist adherence over time, possible improvements in deliverance for specific providers or institutions are not reflected in our analyses. Sessions with less compliant adolescents may have been interrupted more often by egregious behavior, thus requiring DBT-compliant therapist actions not included in the simplified checklist (e.g., ending the session prematurely).

Regarding the milieu-mode, results indicate that the presence of program elements were recognized across respondents and timepoints. One possible explanation for why only on-site coordinator scores significantly increased between the first and last measurement might be that they have extended program knowledge, and thus, more readily recognize elements in the milieu. Alternatively, the shifts in available adolescents and staff at each timepoint may simply reflect the observation of the participants at that exact time, thus, not providing an indication of development over time. Regardless, the periodic reporting across participants gives us insight that program elements are present, and noticed, in the milieu.

As for the program fidelity scale ratings, the considerable variation across institutions denotes that while some are running well-integrated programs in terms of modes and elements, others have more difficulty. This may be a consequence of the demands of a system-wide rollout causing unevenness between institutions (Ivanoff & Marotta, 2018). These results, taken together, may be a reflection of the paradoxical dilemma of the adaptable multi-modality and flexible use of DBT principles, and the training and practical deliverance complexity of all modes. This relates to a common debate regarding whether interventions should be implemented with complete fidelity or whether further adaptation should be encouraged (Durlak & DuPre, 2008) since the nature of this setting and population may be a barrier to implementation fidelity.

Lastly, regarding the acceptability domains, results indicate that adolescents, caregivers, and therapists experienced a high degree of satisfaction with the CDM, with comparable satisfaction levels found for adolescent treatment programs (Chorpita et al., 2015; Weisz et al., 2017). Furthermore, adolescents and therapists had similar medium to high ratings of therapeutic alliance, indicating that they experienced meaningful alliances despite complex challenges. Although these ratings do not reflect the entire adolescent group, this data is valuable, as it gives an indication of the acceptability for the adolescents who were engaged in treatment. In addition, including caregivers gives us unique insight into their satisfaction with the program. Since we could find no comparable alliance or satisfaction outcomes in residential DBT studies, these are the first, and maybe long awaited, attempts at assessing these aspects.

Limitations

Although these findings are promising, this study has several important limitations and findings should be interpreted with caution. Similar to McCredie et al. (2017), this study had a fairly large adolescent sample for a residential setting, but its heterogeneity may nevertheless limit statistical power. Since selection of participants based on DBT-defined criteria was not possible, and adolescents had varied lengths of stay, the sample may contain adolescents who were more responsive, or simply, more exposed to the CDM than others. This may be reflected in the number of adolescents who both attended the treatment modes and completed the outcome measures. Although the research team tracked attendance for individual sessions and overall attendance for skills group training sessions, the received dosage for each adolescent is unknown. Furthermore, as the fidelity outcomes denote, there may be significant variance between institutions and providers. This may be connected to the fact that a very heterogenous group of providers in terms of educational backgrounds may affect the reception and deliverance of standard CDM training. As such, some excelled at providing elements such as establishing alliances, adhering to principles in individual sessions, and utilizing contingencies in the milieu, while others had potential for improvement. In the available data, there were too few subjects and client-therapist pairs to run analyses comparing institutions or provider effects. Additionally, although we do not know the specific impact of the closing of one institution and additional staff turnover, it is likely these presented substantial challenges as new staff had to be oriented to the comprehensive program. The one-time assessment of program fidelity may not accurately reflect current implementation status as the program was under continual development throughout the study. In addition, this study mainly utilized self-report data. This limitation is especially relevant for individual sessions where the lack of trained observer rating for adherence measures makes it difficult to determine the true level of delivery competence in this mode. To minimize reporting bias in future studies, one should consider expert observational methods to assess deliverance and fidelity of milieu and skills group training modes. Furthermore, adolescent incompletion of research-related activities indicates that some adolescents were less inclined to participate in treatment modes and evaluation. Therefore, since the acceptability data only reflects the adolescents and caregivers who completed the measures, we know very little about those who did not respond. For example, only three adolescents who did not attend individual sessions completed the satisfaction questionnaire and the reflections of non-compliant adolescents are unknown. Lastly, the possible implications of the COVID-19 pandemic on both the program (e.g., participants motivation, deliverance, and attendance in treatment modes) as well as the research project (e.g., fewer collaboration meetings between providers and project management) should be noted.

Implications for Practice and Research

Despite the limitations noted, the current study has several strengths. Given the challenges often faced in carrying out residential care research, this study provides a unique insight into how a residential DBT-informed treatment model was received by an entire organization, and most importantly, by the adolescents. This is a significant strength considering how difficult implementation of a comprehensive, multi-modal treatment is, especially within the residential context.

Additionally, this study identified possible points of improvement for future residential DBT-informed treatment. Firstly, since the CDM is a comprehensive program, there is a constant influx of new adolescents, but there is no alternative treatment program for non-compliant adolescents. This conflicts with standard DBT, which relies on continual commitment to treatment, thus potentially affecting adolescent compliance. Moving forward, comparable programs should define lack of adolescent treatment commitment, and how to approach this in terms of what is expected of the adolescents, in line with treatment principles.

Secondly, there is no way to delineate whether adolescents who need the intervention the most actually receive more, and not less. This is a valid concern moving forward, as making sure clients are following the program is a vital principle in DBT (e.g., four-session rule for discontinuing non-adherent clients). The attendance rates for groups may indicate that adolescents value this mode and perhaps that staff are better at delivering it. Contrarily, the low rates of individual sessions indicate the need for improvement, perhaps both in engaging the adolescents as well as providing more proficient individual sessions. Future research should look closer at the characteristics of adolescents who declined to participate in the CDM modes to determine what hindered their participation and to evaluate what non-compliant adolescents need moving forward. Although a strength of a DBT-informed program is that it provides an explicit theoretical underpinning, it also raises a concern whether all adolescents should receive the same intervention. Future implementations need to include CDM specific assessments and protocols that follow DBT principles for non-compliant adolescents to discern whether these adolescents are eligible for the CDM, or whether supplemental protocols or programs are needed.

Furthermore, a challenge to realizing individually tailored therapeutic interventions and milieu therapy in residential care is empowering staff to proficiently provide modes regardless of the variation in mental health expertise (Jozefiak et al., 2016). One possible strategy to ensure a clearer definition of tasks and strengthening treatment competence is assigning staff members to specific treatment functions in reference to skill-level or task preference (Fox, 2019), which may contribute to improved sustainability of both individual providers and the program itself.

Concerning individual session adherence, training may need to focus further on the flexible application of treatment principles during sessions where adolescent behavior does not follow the expected course. In DBT, flexibly applying principles throughout sessions requires not just familiarity with the theoretical basis, but also practical skills, which must be taught, shaped, and reinforced. In addition to consultation team meetings and supervisions, recordings of individual sessions made available to experts for adherence ratings and principle-focused supervision would provide an opportunity to strengthen the deliverance of this mode.

As for the implementation challenges, although DBTs modular skills-based structure is a suitable model for innovative approaches, balancing the evidence-informed model to meet the complex needs of the clinical population and the constraints imposed by the residential institutions is critical to delivery. New setting applications involve deciding which elements are essential, and this decision must not be taken lightly; providers and authorities need to ensure that management policies and guidelines do not prevent the inclusion of critical elements of evidence-based practices (e.g., commitment, noncompliance protocol). Policy makers and providers who consider implementing a DBT-informed treatment model must scrutinize the quality of relevant empirical evaluation studies (Ivanoff & Marotta, 2018) before deciding whether this approach is a good fit for both their organization and clients.

In addition, this study has both produced and adapted several useful tools for evaluating the implementation of DBT elements in residential treatment settings, including a simplified checklist for individual sessions, a cross-informant measure for milieu-treatment features, and a revised fidelity scale for evaluating residential DBT-informed interventions. Since the Program Fidelity scale gives an indication of how well necessary program elements are implemented, it may be considered as a program standard measure going forward. This tool proved helpful to the central authorities in developing quality improvement plans in response to low scoring measure items for both the overall program, as well as the individual institutions.

As for future research, it is vital to identify implementation barriers (i.e. staff turnover, non-compliant adolescents) that may directly affect the deliverance of treatment elements (James, 2017). This is especially relevant for prerequisite for an effectiveness evaluation such as achieving program fidelity. If the program elements are lacking, a reduced effect may erroneously lead researchers or providers to conclude that the model itself is the problem, and hence, labeled as ineffective or inappropriate for this population (Ivanoff & Marotta, 2018).

Lastly, this study does not explore the causal chain between receiving such an intensive treatment model and changes in adolescents’ mental health status and behavioral targets. As a natural extension of this, there is a need for further randomized controlled studies examining the CDM’s effectiveness.

Conclusions

This study presents an evaluation of the feasibility, fidelity, and acceptability of a cross-site DBT-informed treatment model aimed at improving mental health outcomes for adolescents in residential care. Findings provide promising evidence that this comprehensive approach can be appropriate for, and accepted by, a residential care setting and population, reflected in ratings from adolescents, caregivers, and institution staff. This study identifies challenges as well as provides guidance for future development and dissemination of evidence-informed interventions in residential settings. More research on implementation factors and adolescent outcomes following residential DBT-informed treatment is needed.

Implications And Contribution

• Although implementation of CDM is demanding, adolescents, caregivers, and staff deemed the residential DBT-informed treatment model as appropriate and acceptable.

• Implications include specialized staff training, approaches to evaluating provider adherence, and developing DBT-consistent protocols for adolescent noncompliance.

• Publicly available tools for evaluating DBT elements in residential treatment were produced and adapted.

Acknowledgments

The authors would like to acknowledge the generous support from organizational leaders, and the effort of the many providers, caregivers and adolescents in Bufetat region North Norway. Lastly, a thank you to DBT-Trainer Marthe Stornes for her co-assessment of program fidelity.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Data Availability Statement

Due to the sensitive nature of this research, participants of this study did not agree for publicly share their data, therefor, supporting data is not available.

Supplementary Material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/0886571X.2022.2090481.

Additional information

Funding

This study is part of a larger study funded by The Research Council of Norway (Project No. 262789.)