External and Internal Validity Considerations in Youth Effectiveness Trials: Lessons Learned from the COMET Study

ABSTRACT

Objective

Effectiveness trials aim to increase the generalizability and public health impact of interventions. However, challenges associated with this design present threats to external and internal validity. This paper illustrates these challenges using data from a two-site randomized effectiveness trial, the Community Study of Outcome Monitoring for Emotional Disorders in Teens (COMET) and presents recommendations for future research.

Method

COMET was a randomized effectiveness trial conducted in 19 community mental health clinics in two states comparing three interventions: treatment as usual (TAU), TAU with measurement-based care (TAU+), and the Unified Protocol forTransdiagnostic Treatment of Emotional Disorders in Adolescents with MBC (UPA). Participants included 176 clinicians (mean age = 35.5; 85.8% cisgender female; 53.0% racially and/or ethnically minorized) and 196 adolescents (mean age = 14.7; 65.3% cisgender female; 69.4% racially and/or ethnically minorized). Analyses outlined participant flow from recruitment to study completion, described participant characteristics, and examined site differences.

Results

Analysis of participant flow suggested that recruitment and retention of clinicians and adolescents was challenging, raising questions about whether participants were representative of participating clinics. Both the clinician and adolescent samples were racially and ethnically diverse and adolescents were low income and clinically complex. Significant site differences were observed in clinician and adolescent characteristics.

Conclusions

While this study was successful in recruiting a diverse and historically under-represented sample, difficulties in recruitment and retention raise questions about external validity and site differences present challenges to internal validity of study findings. Suggestions for future effectiveness studies, drawing from implementation science approaches, are discussed.

Over the past 20 years of child mental health intervention research, there has been an emphasis on improving the uptake of evidence-based practices (EBPs) in clinical practice settings (Glasgow et al., 2003; Hariton & Locascio, 2018; Williams & Beidas, 2019). Initial testing of interventions focused on efficacy trials which test an intervention under “ideal” conditions (e.g., delivering treatment at university research clinics to that maximize treatment adherence for a target population), with the goal of ensuring the internal validity of the intervention (i.e., does the intervention work as intended?; Porzsolt et al., 2015). However, an overreliance on developing and testing interventions under ideal conditions may result in interventions less likely to hold up under the realities of routine clinical practice settings, such as high-need patient populations and time and resource constraints (Kennedy-Martin et al., 2015; Weisz et al., 1992). As such, the “translational research subway” has expanded to include a range of designs suitable to taking an intervention through the development phase in efficacy trials to effectiveness trials testing the intervention in more “real world” settings, to implementation science studies that focus on assessing intervention effectiveness and identifying strategies to ensure greater uptake (Lane-Fall et al., 2019). The purpose of this paper is to consider the role of effectiveness trials in the development and testing of interventions, with an eye toward what utility these trials do and do not have for advancing the public health impact of these interventions.

Given the emphasis on increasing the public health impact of its funded research, the National Institute of Mental Health (NIMH) has strongly emphasized effectiveness research as a funding priority and a key goal for intervention development (National Institute of Mental Health, n.d.). Compared to efficacy trials, which are designed to minimize confounding factors and maximize internal validity, effectiveness trials are designed with an eye toward external validity. As such, effectiveness trials take place in routine practice settings, such as community clinics or schools, typically enroll clinicians working in these settings and clients seeking services and often utilize fewer exclusion criteria than efficacy trials (e.g., allowing comorbidities; Singal et al., 2014). Given that the impact of an intervention is influenced by numerous characteristics of the intervention, inner setting (e.g., agency resources), outer settings (e.g., financing), and individuals involved in implementation (e.g., clinician and client characteristics (e.g., clinician and client characteristics; Damschroder et al., 2009), the goal of these strategies is to test drive interventions in circumstances representative of the contexts in which they will eventually be applied in routine care. By doing so, the hope is that these trials will increase the applicability of interventions to clinical practice settings and increase their utility for populations historically excluded from the intervention evaluation process (e.g., Medicaid funded clients, minoritized populations). Interventions tested in effectiveness trials are ideally compared against intervention as usual, with the goal of understanding whether the experimental intervention improves upon routine care outcomes. Results from effectiveness trials may therefore be more relevant than efficacy trials for clinicians and policy makers who use research evidence to make care decisions (Singal et al., 2014) and can also inform the development and testing of implementation strategies to support intervention uptake and sustainment (Lane-Fall et al., 2019).

A substantial literature describes the challenges of balancing internal and external validity in intervention research (e.g., Fredericks et al., 2019; Miklowitz & Clarkin, 1999). While effectiveness trials may increase generalizability, the reduced focus on internal validity and purposefully more heterogeneous samples often makes findings challenging to interpret, particularly in the context of multi-site trials conducted across multiple clinics within multiple mental health contexts. It is well-documented that mental health interventions evidence a “voltage drop” when tested in effectiveness trials, but whether this is due to characteristics of the interventions or of clients, clinicians, or settings is unknown (Weisz et al., 2013). In addition, the assumption that findings from effectiveness trials generalize to broader routine clinical practice is generally untested. It is possible that clinics, clinicians, and clients willing to take part in effectiveness trials are not representative of the larger populations to which findings are intended to apply (Weisz et al., 2013). For example, clinicians who are willing to take part in these studies may be more open to evidence-based practices than their peers. Clients who enroll in these trials may be less stressed and more able to take on extra activities than other clients, have a greater need to receive research payments, have more prior education making them open to the research process, etc. As such, there is a need to understand: (1) who enrolls in effectiveness trials, (2) who remains in effectiveness trials, and (3) the implications for interpreting data from effectiveness trials, particularly within multi-site trials designed to increase generalizability across settings.

This paper describes participant characteristics, outlines participant flow from recruitment to study completion, and examines site and condition differences in a two-site randomized effectiveness trial, the Community Study of Outcome Monitoring for Emotional Disorders in Teens (COMET; R01s MH106536 & MH106657). This study was funded under the first round of review of a NIMH clinical trials call for collaborative effectiveness trials (National Institute of Mental Health, 2014). The goal of the paper is to highlight the opportunities and challenges associated with this study design particularly under the conditions of the NIMH Collaborative R01 mechanism and provides recommendations for future studies. This paper also highlights opportunities for researchers who utilize effectiveness designs to consider steps to incorporate methods from implementation science.

Method

Design

The COMET trial was a two-site randomized controlled trial conducted in 19 community mental health clinics (six in South Florida and thirteen in Connecticut). The trial focused on testing two interventions: the Unified Protocol for Transdiagnostic Treatment of Emotional Disorders in Adolescents (UP-A; Ehrenreich-May et al., 2017a) and measurement-based care (MBC) using the Youth Outcomes Questionnaire (YOQ; Burlingame et al., 2005), an MBC system that tracks youth symptoms and therapy alliance. The study compared three conditions: (1) treatment-as-usual (TAU); (2) TAU plus MBC (TAU+); and (3) UP-A plus MBC (UP-A). The Institutional Review Boards at University of Miami and University of Connecticut School of Medicine approved all study procedures, and the study was preregistered at ClinicalTrials.gov (NCT02567266).

A full description of the study design has been published in a protocol paper (Jensen-Doss et al., 2018). Briefly, clinicians were recruited through clinic administrators, consented to participate in the study, and completed baseline measures before being randomly assigned to one of the three study conditions. Randomization took place within clinics to ensure that all three conditions were available in each clinic. The only exception to randomization was that if clinicians left the study, replacement clinicians were placed in the condition from which the clinician had departed or randomized only between conditions with openings if multiple clinicians were being replaced at one time. All clinicians received two hours of training in study procedures. Clinicians randomized to the TAU+ and UP-A conditions also participated in a four-hour didactic workshop in the YOQ system and MBC, and clinicians randomized to the UP-A condition completed a 12-hour workshop on the UP-A. Ongoing weekly consultation was provided to clinicians in the TAU+ (30 minutes) and UP-A (60 minutes) conditions. All clinicians treated study clients as part of their regular caseloads. Clinicians audio recorded sessions, completed a session report every session, completed measures at 8- and 16-weeks for each study client, and were asked to repeat some of the baseline measures after they completed their first study client. Depending on the clinic, clinicians were either reimbursed for training and consultation time through their clinic or directly by the study at the rate of $30 per hour. All clinicians received $60 per client for completing study measures.

Although specific recruitment procedures varied by clinic, families of potential adolescent study participants typically received information about COMET at the initial phone contact, or after completing the clinic’s intake assessment. If they were interested in participating in the study, caregivers completed a phone screen conducted by project staff to determine initial study eligibility and, if eligible, were scheduled for an in-person baseline assessment. At the baseline assessment, adolescents and their caregiver assented/consented to participation and completed an assessment conducted by an independent evaluator to further assess eligibility and obtain baseline measures. Eligible adolescents were then randomized to one of the three study conditions and assigned to a clinician in that condition at their clinic. Randomization was blocked by clinic and was stratified by whether or not the adolescent had a depressive disorder (major depressive disorder, persistent depressive disorder, unspecified depressive disorder, or adjustment disorder with depressed mood), whether or not the adolescent was taking psychotropic medications, and whether they spoke English or Spanish. After the baseline assessment, adolescents and their caregiver participated in follow-up assessments at 8 weeks (mid-treatment), 16 weeks (post-treatment; intended to correspond with the average length of the UP-A), and 28 weeks after treatment initiation (3-month follow-up).

Participants

The study proposed to recruit 18 clinician participants (3 per clinic across 6 clinics), who were employees or trainees working at the participating clinics. Clinician study eligibility included: 1) being at least a part-time employee or completing an approved practicum or internship of at least one year at a participating clinic, and 2) being able to speak and read English.

The study planned to recruit 222 youth. Youth study eligibility included: 1) being between the ages of 12 and 18 with clinically significant symptoms of anxiety or depression, as defined by a Clinical Severity Rating (CSR) of 4 or higher on any DSM-5 anxiety, obsessive-compulsive or depressive disorder, including adjustment disorders, as determined by the Anxiety Disorders Interview Schedule for the DSM-5, Child Version, Child and Parent Report Forms (ADIS-5-C/P; Silverman et al., 2023), 2) deemed eligible for outpatient services by a study clinic, 3) living with a legal guardian at least 50% time who was willing to attend treatment sessions, and 4) having a caregiver who was able to complete all study procedures in English or Spanish. Youth exclusion criteria included: 1) receiving concurrent psychosocial interventions, 2) suicidal behavior that warranted a higher level of care than routine outpatient treatment, and 3) other indicators that would make the UP-A contra-indicated (e.g., significant substance abuse; IQ < 80).

Measures

Given the focus of this paper is on participant characteristics and participant flow through the study, only baseline measures and study records were used.

Clinician Demographic and Professional Characteristics

A Clinician Background Form was developed for this study. Clinicians reported on their age, gender (male; female; transgender [male to female]); transgender (female to male) and other]; ethnicity (Hispanic/Latinx versus not), and race (White; Black or African American; Asian; American Indian/Alaska Native; Native Hawaiian or Other Pacific Islander; Other/Multiple; subsequently analyzed as White, Black, and Other/Multiple due to small sizes of other racial groups). Clinicians also provided information on professional characteristics, including highest degree obtained, professional specialty (social work; school psychology, clinical psychology, counseling, pastoral counseling, psychiatry, education, and other; analyzed as clinical psychology, social work, counseling, and other, due to small sample sizes in other categories), years of professional experience, whether they were professionally licensed (yes/no), the number of hours per week they worked at their clinic, caseload size, and the number of hours per week they received clinical supervision. Clinicians also completed session-by-session forms indicating whether they provided treatment in a clinic office or in the community (e.g., home, school). Because treatment setting could vary across clients, being treated by the same clinician was treated as an adolescent-level variable.

Adolescent Demographics

A Family Background and Medical History Form was developed for this study. Youth reported on their age, gender, and sexual orientation (heterosexual; gay/lesbian; bisexual; other; don’t know/would rather not say). Caregivers reported on youth ethnicity (Hispanic/Latinx versus not), race (White; Black or African American; Asian; American Indian/Alaska Native; Native Hawaiian or Other Pacific Islander; Other/Multiple; subsequently analyzed as White, Black, and Other/Multiple due to small sizes of other racial groups) and family income. Family income was analyzed as a 3-category variable that estimates a family’s federal poverty level (FPL; U.S. Census Bureau, 2021a, 2021b). Each family indicated an income category (e.g., $0–10,999, $11,000–20,999, etc.) and the number of people in their home. The midpoint of the selected income range and the reported number of people in home were used to estimate a family’s FPL status. Families were classified as low income (at or below federal poverty level) middle income (greater than the FPL but less than the median family income in the U.S. of $86,000; U.S. Census Bureau, 2021a, 2021b); and high income (equal to or above the median family income in the U.S.).

Independent Evaluator Rated Diagnoses and Symptom Severity

Independent evaluators (IEs) interviewed the adolescents and caregivers using the ADIS-5-C/P (Silverman et al., 2023), a semi-structured interview assessing youth anxiety and related disorders. IE’s assigned primary and comorbid diagnoses as well as clinical severity ratings (CSRs) for each diagnosis. Based on the interview, they also rated each adolescent’s symptom severity with the Clinical Global Impression-Severity (CGI-S), a 7-point IE rating of psychopathology severity (Guy, 1976), ranging from 1 (no illness) to 7 (extremely severe), with 4 being the cutoff for a clinical diagnosis. IEs also rated the adolescent’s functioning using the Children’s Global Assessment Scale (CGAS), a 100-point rating scale used to measure global functional impairment (Shaffer et al., 1983).

Independent evaluator reliability procedures included: 1. Independently rating four ADIS-5-C/P gold standard criterion tapes. One of the four tapes could have been a live, collaborative match with a previously certified IE; 2. Being observed by a previously certified IE administering the ADIS-5-C/P and having the administration recorded; and 3. Receiving approval from COMET Principal Investigators based on the recording. To prevent rater drift, IEs received ongoing supervision at each site by the study PIs. To assess inter-rater reliability, all assessments were video recorded and 20% of assessments were randomly selected for masked review by a certified IE per year. Reliability for independent evaluator ratings was good (CGI-S ICC = .80; CGAS ICC = .89).

Adolescent and Caregiver Rated Symptoms

Adolescents and their caregivers self-reported on symptoms using three rating scales with parallel youth and caregiver reports. Anxiety symptoms were measured via the Screen for Child Anxiety Related Disorders (SCARED; Birmaher et al., 1997) Total Score and depression symptoms via the Mood and Feelings Questionnaire (MFQ; Angold et al., 1995; Messer et al., 1995). The Strengths and Difficulties Questionnaire (Goodman et al., 1998) Total Score was used to assess overall symptom severity. Baseline alphas in the COMET sample for all three scores were high across all reporters (α’s = .79–.95).

Data Analysis Plan

The goals of this paper are largely descriptive, so data were analyzed mostly using descriptive statistics. Multilevel modeling (MLM) was used to account for the nesting of adolescents and clinicians within clinics, and to examine differences in sample characteristics by study site and treatment condition.

Results

Clinician Results

Study Recruitment and Participation

Figure 1 presents the CONSORT diagram for clinician participants. 174 clinicians (98 in Connecticut and 78 in South Florida) were consented and randomized to one of the study conditions; this was 966.7% of the original target enrollment of 18 clinicians. Forty-nine clinicians were randomized to TAU, 63 to TAU+, and 62 to UP-A. Across conditions, 10.9% (n = 19) of clinicians withdrew from the study prior to training. Rates of dropout prior to training did not differ significantly across sites or study conditions (p = .90).

Figure 1. COMET clinician participant consort diagram.

Note: CONSORT diagram detailing the flow of clinician participants through the study. TAU = Treatment as usual; TAU+ = Treatment as usual plus measurement-based care; UP-A = Unified Protocol for Transdiagnostic Treatment of Emotional Disorders in Adolescents plus measurement-based care.

One-hundred and fifty-five clinicians were trained in the study procedures across the three study conditions. Of these, 42.5% (n = 67) never treated any study clients; this rate did not differ across sites or study conditions (p = .072). As detailed in Figure 1, most of these clinicians left the study prior to being assigned a case. Eighteen of these clinicians were still enrolled at the end of the study but did not have a case assigned to them before the study ended. The most common reason that these clinicians left the study was leaving their position at the participating clinic (see Figure 1). Eighty-eight clinicians treated at least one study case and these clinicians treated 2.2 clients on average (SD = 1.8). Only half (50.6%, n = 45) of clinicians who treated any clients treated more than one client; this did not differ across study sites (p = .20). However, clinicians in the TAU+ condition were less likely to treat more than one client compared to clinicians in the UP-A (B = −1.16, SE = .57, p = .04) and TAU (B = −1.19, SE = .57, p = .04) conditions.

Collapsing across clinicians who did and did not treat study clients, only 61 (34.7%) of the clinicians who were originally enrolled in the study were still enrolled when the study ended. The majority of these (n = 48, 78.7%) left the study because they left their position at the participating clinic. Other reasons for withdrawing (clinicians could indicate more than one) included having too many competing demands on their time or no room on their caseload (n = 20), no longer being interested in taking part in research (n = 5), personal reasons (n = 8), the clinician being promoted to a supervisory role where they would no longer treat clients (n = 7), and the clinician having other changes to their role that precluded participation (e.g., no longer seeing adolescents; n = 3). Nineteen clinicians left the study because their clinic withdrew, and another 8 were withdrawn by study staff (e.g., for not responding to attempts to schedule training).

Clinician Characteristics

Table 1 describes the demographic and professional characteristics in the whole clinician sample and by study site. Overall, clinicians were 35.5 years old (SD = 10.1), 85.8% cisgender female, 32.4% Hispanic/Latinx, and 71.6% White. When ethnicity and racial data were combined, 53.0% of clinicians identified as racially and/or ethnically minoritized. Sites differed on the percent of Hispanic/Latinx clinicians, with the South Florida site (51.3%) having significantly more Hispanic/Latinx clinicians than Connecticut (CT) (19.4%; B = 1.63, SE = 0.51, p = .005). Nearly all (90.3%) clinicians were master’s level, and from a range of professional specialty backgrounds, including clinical psychology (21.0%), social work (25.6%), counseling (40.3%) and other disciplines (e.g., marriage and family therapy, art therapy, 13.1%). Professional specialty was compared across sites using categorical MLM, where site predicted the likelihood of appearing in one category versus another. South Florida clinicians were more likely to fall in the counseling group (53.8%) relative to the other group (7.7%) compared to the CT clinicians (29.6% counseling, 17.3% other, B = 1.88, SE = .66, p = .01); no other site differences were significant.

Table 1. Clinician demographic and professional characteristics.

	Whole Sample N = 176	Connecticut Site (CT) N = 98	South Florida Site (SFL) N = 78	Site differences
Age M (SD)	35.5 (1.1)	36.4 (10.5)	35.5 (10.1)	NS
Gender				NS
Cisgender Male	13.6% (24)	14.3% (14)	12.8% (10)
Cisgender Female	85.8% (151)	85.7% (84)	85.9% (67)
Transgender- Female to Male	.6% (1)	0.0% (0)	1.3% (1)
Ethnicity
Hispanic/Latinx	32.4% (57)	19.4% (19)	51.3% (40)	CT < SFL, p = .005
Not Hispanic/Latinx	67.0% (118)	79.6% (19)	48.7% (38)
Not reported	.6% (1)	1.0% (1)	0.0% (0)
Race				NS
Black	18.8% (33)	13.3% (13)	25.6% (20)
White	71.6% (126)	73.5% (72)	69.2% (54)
Other/multiple	4.0% (7)	5.1% (5)	2.6% (2)
Not reported	5.7% (10)	8.2% (8)	2.6% (2)
Highest level of education				NS
Bachelor’s Degree	5.7% (10)	7.1% (7)	3.8% (3)
Master’s Degree	9.3% (159)	89.8% (88)	91.0% (71)
Doctorate	4.0% (7)	3.1% (3)	5.1% (4)
Professional specialty^1				SFL counseling versus other > CT counseling versus other, p = .01
Clinical Psychology	21.0% (37)	19.4% (19)	23.1% (18)
Social Work	25.6% (45)	33.7% (33)	15.4% (12)
Counseling	4.3% (71)	29.6% (29)	53.8% (42)
Other	13.1% (23)	17.3% (17)	7.7% (6)
Years Professional Experience M (SD)	4.1 (5.6)	4.1 (5.9)	4.1 (5.6)	NS
Licensed	33.0% (58)	46.9% (46)	15.4% (12)	CT > SFL, p = .001
Hours worked at clinic per week M (SD)	32.5 (11.4)	31.7 (12.4)	33.4 (10.1)	NS
Caseload M (SD)	21.9 (14.4)	22.1 (18.2)	21.7 (7.8)	NS
Hours per week of supervision M (SD)	1.8 (4.6)	1.4 (1.0)	2.3 (6.8)	NS

Note: 1. Analyzed using categorical MLM, where site predicted the likelihood of appearing in one category versus another; NS = No significant site differences; CT = Connecticut Site; SFL = South Florida Site; M = Mean; SD = Standard Deviation.

Clinicians had 4.1 years of professional experience (SD = 5.6) on average; this did not differ across sites. In the whole sample, 33.0% of clinicians were licensed; CT clinicians were more likely to be licensed (46.9%) than South Florida clinicians (15.4%; B = −1.62, SE = 0.43; p = .001). Clinicians worked an average of 32.5 (SD = 11.4) hours per week, received 1.8 (SD = 4.6) hours of weekly clinical supervision, and had a caseload of 21.9 clients (SD = 14.4); these results did not differ by site.

Comparisons of clinician characteristics across conditions indicated that randomization was largely successful. The only difference across conditions was on professional specialty, with TAU therapists more likely to fall in the counseling group (51.0%) relative to the “other” group (8.2%) compared to the TAU+ clinicians (30.2% counseling, 14.3% other, B = 1.26, SE = .53, p = .02).

Adolescent Results

Study Recruitment and Participation

Figure 2 presents the CONSORT diagram for adolescent participants. Participating clinics referred 1442 youth to the study for possible inclusion. The large referral number was in part due to one clinic that referred all adolescents who had treatment intakes scheduled without any further screening for emotional disorders. Of these, 588 (40.7%) were contacted and assessed for initial eligibility via phone screen over 3.5 years. Over half (57.3%) of these adolescents were excluded due to either not meeting inclusion criteria (59.9% of adolescents excluded at the screening stage) or declining participation (34.4%). Two hundred fifty-one adolescents completed baseline assessments. Fifty-five of these youth were excluded from the study at this stage, largely due to either not meeting inclusion criteria (41.8%) or declining to participate after the baseline (32.7%). The most common reasons that youth did not meet study criteria included suicidality that required a higher level of care and co-occurring psychiatric conditions contraindicating study treatment (e.g., a sole primary diagnosis of externalizing disorder). An additional 14 youth were not included in the study due to administrative clinic issues (e.g., adolescent was mistakenly assigned to a non-study clinician) or other logistical reasons. This resulted in a final sample of 196 youth randomized to participate in the study. This was 88.2% of the original study target of 222 adolescents.

Figure 2. COMET youth participant consort diagram.

Note: CONSORT diagram detailing the flow of adolescent participants through the study. TAU = Treatment as usual; TAU+ = Treatment as usual plus measurement-based care; UP-A = Unified Protocol for Transdiagnostic Treatment of Emotional Disorders in Adolescents plus measurement-based care.

As detailed in Figure 2, across conditions, rates of completing follow-up research assessments were 83.7% at 8 weeks, 75.0% at 16 weeks, and 72.4% at 32 weeks. Assessment completion rates did not differ significantly across sites or study condition (all p’s > .05).

Adolescent Characteristics

Table 2 describes the demographic characteristics of adolescent participants in the whole sample and by site. The sample was 14.7 years old on average (SD = 1.7). 65.3% of adolescents identified as cisgender female, 33.2% as cisgender male, 1.0% as transgender (female to male), and 0.5% as “other” gender. About half of adolescents indicated their sexual orientation was heterosexual (55.6%), 2.6% gay or lesbian, 10.2% bisexual, 6.6% other, and 24.0% indicated they did not know or would rather not say. Sites did not differ on participant age, gender, or sexual orientation. There were site differences on rates of Hispanic/Latinx ethnicity, with higher rates in South Florida (53.4%) than in CT (28.0%; B = 1.76, SD = 0.59, p = .008). Over half of the participants identified as White (60.7%), 23.5% as Black, 12.2% as other or multiple races, and 3.6% did not report their racial identity; this did not differ significantly by site. Overall, 69.4% of participants reported being from an ethnically and/or racially minoritized group.

Table 2. Adolescent demographic characteristics.

	Whole Sample N = 196	Connecticut Site (CT) N = 93	South Florida Site (SFL) N = 103	Site differences
Age M (SD)	14.7 (1.7)	15.1 (1.7)	14.3 (1.6)	NS
Gender				NS
Cisgender Male	33.2% (65)	33.3% (31)	33.0% (34)
Cisgender Female	65.3% (128)	64.5% (60)	66.0% (68)
Transgender- Female to Male	1.0% (2)	1.1% (1)	1.0% (1)
Other	.5% (1)	1.1% (1)	0% (0)
Sexual Orientation^1				NS
Heterosexual	55.6% (109)	53.8% (50)	57.3% (59)
Gay or Lesbian	2.6% (5)	1.1% (1)	3.9% (4)
Bisexual	1.2% (20)	12.9% (12)	7.8% (8)
Other	6.6% (13)	8.6% (8)	4.9% (5)
Doesn’t know/would rather not say/missing	24.0% (49)	23.7% (22)	26.3% (27)
Ethnicity
Hispanic/Latinx	41.3% (81)	28.0% (26)	53.4% (55)	SFL > CT, p = .008
Not Hispanic/Latinx	58.7% (115)	72.0% (67)	46.6% (48)
Race^1				NS
Black	23.5% (46)	11.8% (11)	34.0% (35)
White	6.7% (119)	68.8% (64)	53.4% (55)
Other/multiple	12.2% (24)	16.2% (15)	8.8% (9)
Not reported	3.6% (7)	3.2% (3)	3.9% (4)
Income^1				SFL middle-income versus high-income > CT middle-income versus high-income, p = .037
Low Income, ≤FPL	31.6% (62)	18.3% (17)	43.7% (45)
Middle Income, >FPL and <$86,000	38.8% (76)	38.7% (36)	38.8% (40)
High Income, >$86,000	23.5% (46)	38.7% (36)	9.7% (10)
Not reported	6.1% (12)	4.3% (4)	7.8% (8)
Treatment setting
Office-based	61.7% (121)	94.6% (88)	32.0% (33)	NS

Note: 1. Analyzed using categorical MLM, where site predicted the likelihood of appearing in one category versus another; NS = No significant site differences; CT = Connecticut Site; SFL = South Florida Site; M = Mean; SD = Standard Deviation; FLP = Federal Poverty Level.

Approximately a third (31.6%) of participants fell in the low-income group, 23.5% in the middle-income group, and 23.5% of participants in the high-income group; 6.1% of families did not provide income information. There were significant income differences by site. Categorical HLM indicated that the South Florida participants were more likely to fall in the middle income (42.1%) versus high income (10.5%) categories relative to the CT participants (40.4% middle income vs 40.4% high income, B = 1.11, SE = .49, p = .037). About two thirds of families received services provided in clinic offices (61.7%) rather than in community settings like home and school, but there were large differences by site. In CT, 94.6% of treatment sessions took place in clinic offices and only 31.0% of South Florida treatment sessions occurred in clinic offices, but this did not reach statistical significance in the MLM models, perhaps due to reduced power associated with the very small number of community-based settings in CT.

The distribution of youth demographic variables across study sites indicated that randomization was generally successful, with two exceptions. Categorical MLM indicated that the UP-A participants were more likely to fall in the low-income (47.6%) versus high-income (17.5%) group relative to the TAU participants (21.9% low-income vs 28.1% high-income, B = 1.23, SE = .58, p = .037) and more likely to fall in the middle-income (34.9%) versus high-income group relative to TAU participants (50.0% middle-income; B = 1.16, SE = .46, p = .012). UP-A participants were also more likely to fall in the White racial category (72.3%) versus Other racial category (4.6%) relative to the TAU (60.9% White vs 17.2% Other; B = 1.51, SE = .61, p = .030) and TAU+ conditions (55.0% White vs 17.2% Other; B = 1.61, SE = .70, p = .023).

Table 3 details the baseline clinical characteristics of the youth sample. The average CGI-S score at baseline was 5.6 (SD = .8), falling between the categories of “markedly” and “severely” ill. The average CGAS score was 45.6 (SD = 5.7), falling in the range described as “Moderate degree of interference in functioning in most social areas or severe impairment of functioning in one area.” The average CSR rating for the participants’ primary diagnosis was 5.6 (SD = 0.7), a score of 4 represents a clinical diagnosis with moderate impairment, while scores of 5 and 6 represent severe and markedly disabling impairment. On average, youth had 2.5 (SD = 1.2) diagnoses at baseline. There were no site differences in these IE rated measures at baseline, except for the number of diagnoses, which was higher at the South Florida site (M = 2.7, SD = 1.1) than the CT site (M = 2.2, SD = 1.0; B = 0.51, SE = 0.13, p < .001).

Table 3. Adolescent clinical characteristics.

	Whole Sample N = 196	Connecticut Site (CT) N = 93	South Florida Site (SFL) N = 103	Site differences
CGI-S M (SD)	5.6 (.8)	5.6 (.7)	5.7 (.7)	NS
CGAS M (SD)	45.6 (5.7)	45.8 (5.9)	45.4 (5.5)	NS
Primary CSR M (SD)	5.6 (.7)	5.6 (0.8)	5.7 (0.7)	NS
Number of Diagnoses M (SD)	2.5 (1.2)	2.2 (1.0)	2.7 (1.1)	SFL > CT, p < .001
Youth SDQ Total M (SD)	14.0 (6.6)	14.9 (6.3)	13.3 (6.7)	NS
Youth SCARED Total M (SD)	3.8 (15.6)	33.1 (15.0)	28.7 (15.9)	NS
Youth MFQ Total M (SD)	21.2 (15.6)	22.7 (15.7)	20.0 (15.3)	NS
Caregiver SDQ Total M (SD)	14.8 (6.5)	14.6 (6.4)	14.9 (6.6)	NS
Caregiver SCARED Total M (SD)	27.5 (13.4)	29.3 (13.3)	26.0 (13.3)	NS
Caregiver MFQ Total M (SD)	19.9 (12.8)	20.6 (14.3)	19.3 (11.3)	NS

Note: NS = No significant site differences; CT = Connecticut Site; SFL = South Florida Site; M = Mean; SD = Standard Deviation; GCI-S = Clinical Global Impression-Severity; CGAS = Children’s Global Assessment Scale; CSR = Clinical Severity Rating; SDQ = Strengths and Difficulties Questionnaire; SCARED = Screen for Child Anxiety Related Disorders; MFQ = Mood and Feelings Questionnaire.

Table 4 shows the specific diagnoses assigned to adolescent participants at baseline, including rates of any diagnosis (i.e., primary or comorbid) and rates of the primary diagnoses. 75.5% of youth had primary diagnoses that were of anxiety or related disorders and 27.5% had a primary diagnosis of depression or related disorders. These percentages do not sum to 100% because some youth had co-primary diagnoses or adjustment disorder with mixed anxiety and depression, which was counted in both categories.

Table 4. Adolescent diagnoses.

	% (N) With Diagnosis^1	% (N) Diagnosis was Primary or Co-Primary
Any Anxiety or Related Disorder	95.4% (187)	75.5% (148)
Separation Anxiety Disorder	2.6% (5)	.5% (1)
Social Anxiety Disorder	54.9% (107)	23.5% (46)
Selective Mutism	0.5% (1)	0% (0)
Generalized Anxiety Disorder	76.5% (150)	42.9% (84)
Specific Phobia	8.7% (17)	2.0% (4)
Panic Disorder	2.0% (4)	0% (0)
Agoraphobia	1.5% (3)	.5% (1)
Unspecified Anxiety Disorder	1.5% (3)	.5% (1)
Obsessive-Compulsive Disorder	7.7% (15)	3.6% (7)
Post-Traumatic Stress Disorder	5.1% (10)	1.0% (2)
Any Depressive Disorder	55.1% (108)	27.5% (54)
Major Depressive Disorder	17.9% (35)	11.7% (23)
Persistent Depressive Disorder	30.6% (60)	11.3% (22)
Unspecified Depressive Disorder	7.1% (14)	2.0% (4)
Adjustment Disorder with Depressed Mood	1.0% (2)	1.0% (2)
Adjustment Disorder with Mixed Anxiety and Depressed Mood^2	4.0% (8)	2.0% (4)
Bipolar Disorder	0.5% (1)	0% (0)
Disruptive Mood Dysregulation Disorder	1.0% (2)	.5% (1)^3
Attention Deficit Hyperactivity Disorder	20.9% (41)	2.6% (5)^3
Conduct	1.0% (2)	0% (0)
Oppositional Defiant Disorder	4.1% (8)	.5% (1)^3

Note: 1. Percentages do not add up to 100% because adolescents could receive more than one diagnosis. 2. Counted toward the Any Anxiety or Related Disorder and the Any Depressive Disorder categories. 3. Although having a primary diagnosis of an emotional disorder was required for study inclusion, co-primary externalizing disorders were permitted.

Although baseline IEs ratings indicated youth were quite impaired, youth and caregiver self-report data indicated less symptoms severity (see Table 3). On the caregiver SDQ, the average Total Score fell in the borderline clinical range (M = 14.8, SD = 6.5). Clinical cutoff scores are not available for the youth SDQ, but the average Total Score was comparable to the caregiver report (M = 14.0, SD = 6.6). Average scores on the youth (M = 21.2, SD = 15.6) and caregiver (M = 19.9, SD = 12.8) MFQ also fell short of the recommended cutoffs for a depression diagnosis, although this is consistent with findings that that most youth were being treated for anxiety. Expectedly, average scores on the youth (M = 30.8, SD = 15.6) and caregiver (M = 27.5, SD = 13.4) SCARED were higher than the cutoffs for a likely anxiety disorder. There were no site differences on the rating scale measures. A randomization check found no condition differences on any of the IE rated or rating scale measures.

Discussion

The goal of this paper was to describe the participant characteristics and study flow of a two-site randomized effectiveness trial, with the goal of highlighting opportunities and challenges associated with effectiveness trials in community settings as well as to guide implementation science efforts. Our findings suggest that attaining the goals of recruiting and retaining a representative sample of clinicians and adolescents in order to conduct a “real world” test of interventions with high internal and external validity is quite difficult.

Previous studies have found that clinicians working in community clinics are largely master’s level, non-Hispanic/Latinx White, and female (Amaya-Jackson et al., 2018; Bartlett et al., 2016; Jensen-Doss et al., 2020). The clinicians who enrolled in this study similarly identified as mostly White and female and held master’s degrees. However, there were notable site differences in clinician ethnicity across both sites, with more than half of South Florida clinicians identifying as Hispanic/Latinx, consistent with U.S. Census Data. As such, over half of clinicians in the sample identified as coming from a racial and/or ethnic minoritized group, suggesting that our sample may be more diverse than clinicians nationally. We also were successful in recruiting clinicians from a range of professional specialties, increasing the applicability of study findings across a range of professional backgrounds. Analyses of randomization indicated that clinician characteristics were the same across study conditions, with the sole exception being professional specialty: there was a higher proportion of clinicians from a counseling background versus “other” background (e.g., marriage & family therapists, art therapists) and this difference was more pronounced in the TAU condition than in the TAU+ condition.

The adolescents who participated in our sample were quite diverse in terms of race and ethnicity, with around 40% coming from a minoritized racial group and around 40% reporting Hispanic/Latinx ethnic identity. Taken together, less than 1/3 of the sample identified as non-Hispanic/Latinx and White. As might be expected with a sample of adolescents with internalizing disorders, about 2/3 identified as cisgender female. Interestingly, only just over half identified as “straight,” with nearly a quarter of adolescents indicating they did not know or would rather not say or leaving the item blank. Adolescents primarily lived in low- to middle-income households. Clinically, IEs rated adolescents as highly impaired and very diagnostically complex, with comorbidity being the rule rather than the exception. Although primary diagnoses of depression and/or anxiety were allowed, nearly 3/4 of the sample had a primary anxiety disorder diagnosis, and only about 1/3 had a primary depression diagnosis. When sites were compared on demographic characteristics, some site differences did emerge, with the South Florida site again having a higher percentage of Hispanic/Latinx participants and a higher proportion of low-income adolescents than Connecticut. The South Florida adolescents also had a significantly higher average number of diagnoses per participant. Examination of whether study randomization was successful found only two adolescent demographic variables that differed across study conditions. UP-A participants were more likely to be in lower income categories than TAU participants and more likely to fall in the White racial category than participants in both the TAU and TAU+ conditions. There were no clinical differences across study conditions.

One driving question of the study was whether effectiveness trials like this one yield more representative samples than efficacy trials, increasing the generalizability of outcomes to community settings and their relevance to decisions about whether to proceed to implementing them in real-world practice settings. Compared to previous efficacy trials of the UP-A and cognitive-behavioral treatments for youth anxiety and depression (e.g., Ehrenreich-May et al., 2017b; Kendall et al., 2010; Treatment for Adolescents with Depression Study Team, 2005), this study did have a higher proportion of racially minoritized youth. Prior studies did not report participants’ gender beyond male/female and did not report sexual orientation at all, so it is not possible to compare our sample to them on those dimensions. Similarly, differences in how income was reported across samples makes it difficult to determine whether this trial was more successful than previous efficacy trials in recruiting a low-income sample. Clinically, our sample appears to be more diagnostically complex and severe than previous efficacy studies; baseline CGI-S scores in this sample were higher than in previous efficacy trials (Ehrenreich-May et al., 2017b; Kendall et al., 2010; Treatment for Adolescents with Depression Study Team, 2005). 78.6% of our participants had more than one diagnosis, higher than the approximately 50% comorbidity rate in the Kendall et al. (2010) and the Treatment for Adolescents with Depression Study Team (2005), but comparable to the comorbidity rate of 76.5% in the previous UP-A trial (Ehrenreich-May et al., 2017b). As such, it appears that we were successful recruiting a more diverse and clinically complex sample than those obtained in traditional efficacy trials. The degree to which the sample is representative of treatment-seeking youth with emotional disorders in the participating clinics is more difficult to determine. Our rates of racial/ethnic minoritized youth and youth living below the federal poverty level are comparable to or higher than census data for the communities participating in the study (U.S. Census Bureau, 2022a, 2022b, 2022c). However, we did not have access to data to compare our participants to other youth at participating clinics, so it is possible that our participants were not “typical” clients at these agencies.

One of the biggest challenges associated with this study was the recruitment and retention of both study clinicians and participating adolescents. On the clinician side, clinics were quite helpful in recruiting clinicians to take part in the study. However, retaining clinicians in the study proved to be quite difficult. To treat the 196 adolescents who enrolled in the study, we had to recruit 176 clinicians, which was nearly ten times the original proposed sample size of 18. Of these, only about a third of clinicians remained in the study until it ended. Not surprisingly given the high rates of turnover among mental health clinicians (e.g., Beidas et al., 2016; Ben-Dror, 1994), the most common reason for leaving the study was clinician turnover from their positions. As a result of this turnover and because of adolescent recruitment difficulties, only half of the clinicians who were consented into the study actually treated any study clients, and only about half of those clinicians treated more than one client. Given that study staff spent between 2 and 18 hours orienting and training each clinician who enrolled in the study, the return on investment for this training time was quite low. These issues were present across all study conditions, with the exception that the TAU+ clinicians were less likely to treat more than one study client than clinicians in the other two conditions. This is somewhat surprising given that the UPA clinicians had a much higher training and consultation burden than clinicians in the other two conditions. Anecdotally, some clinicians shared that they enrolled in the trial because they were interested in becoming trained in the UPA, so it may be that clinicians were less interested in the MBC training or less comfortable using MBC and therefore were less likely to treat multiple clients in the trial.

One challenge that was likely intertwined with the clinician retention issues was challenges recruiting adolescents to participate in the study. The original recruitment target for this trial was 222 adolescents. We were able to successfully recruit 196 adolescents, 88.2% of our original goal. To recruit this sample, we attempted to contact 1442 youth, screened 588 for possible study inclusion, and conducted 251 baseline assessments. Our final sample therefore represented 13.5% of youth who were referred to the study, 33.3% of the youth we were able to screen, and 78.1% of the youth whose families participated in the baseline assessments. Challenges that contributed to youth recruitment included difficulties working out appropriate, often idiosyncratic, referral procedures from clinics, difficulty making contact with families, and difficulty getting families to agree to take part in the study assessments. One challenge was also driven by our multi-site study design. At the time of the study, nearly all outpatient youth mental health services in Connecticut were being provided in clinic offices, whereas services in South Florida were increasingly being offered in homes and schools. To reduce cross-site variability, we attempted to exclusively enroll office-based clients to the study for the first year, but this proved to be a significant barrier to recruitment in South Florida. Lifting this restriction led to significantly higher rates of adolescent enrollment at that site. Once adolescents were recruited, we were able to retain more than 70% of them through the duration of the study, and retention of participants did not differ across study conditions.

In order to address difficulties recruiting adolescents, the study added additional clinics and clinicians to try to increase the reach of adolescent recruitment, going far beyond our original clinician recruitment targets, as noted above. Many clinicians had to wait long stretches between being trained in the study procedures and being assigned a client, leaving time for turnover and also for clinicians to become uncomfortable with the idea of trying out a new practice. As noted above, many never treated any clients and those who did rarely treated more than one. Taken together, these data highlight several threats to internal validity that may arise in multi-site effectiveness trials. While the inclusion of two sites may increase the generalizability of findings, the numerous differences in adolescents, clinicians, and treatment settings across the two study sites, and likely between clinics even within sites, raise questions about the degree to which the data can be considered a single sample and introduce variability in the data that may be difficult to interpret. While randomization was largely successful, the differences observed in clinician professional specialty, family income, and adolescent race introduce other threats to internal validity. Outcome analyses will need to include an examination of site differences in outcome and may need to include covariates to try to account for site and condition differences. The heterogeneity of participants also highlights the importance of examining moderators of intervention response (Glasgow et al., 2003). In addition, the limited number of adolescents treated by each clinician and the limited number of treating clinicians within each participating agency will make it difficult to understand clinician and agency effects in the sample.

Similarly, despite the goal of increased external validity, effectiveness trials like this one may continue to have threats to external validity. First, given how difficult it was to recruit adolescents, we do not know how representative these adolescents were of the general client population at the participating clinics. Although these participants appear to be representative of their communities on some basic demographic characteristics, data were not available to compare our sample to other clients at these clinics. It is possible that families who were willing to take part in research differed from other families seeking treatment on variables we were not able to measure, such as stress, time to participate in research, openness to science, etc. Although we made our inclusion criteria as broad as possible, some adolescents were excluded from the study because they required a higher level of care, so it is possible that our sample was less clinically severe than the general clinic populations. If youth are not representative of agency client populations, then effectiveness trials may not effectively be testing the robustness of interventions to client-level challenges that that could be encountered as researchers and agencies move toward implementing them.

Similarly, it is challenging to confirm how representative participating clinicians were of the clinicians employed at participating clinics. Anecdotally, clinics shared that they tried to target clinicians who they knew would be excited about the opportunity for additional training and/or dependable, which likely decreased representativeness. Consistent with this, although not the focus of this paper, a prior publication using this dataset found that our participating clinicians were significantly younger and less clinically experienced compared to national survey samples (Patel et al., 2022). Additionally, clinicians in our sample had more favorable attitudes toward evidence-based practices and measurement-based care than national averages, indicating that they were motivated to participate in research activities. Implementation theory suggests that working with clinicians with more positive attitudes facilitates implementation success (e.g., Damschroder et al., 2009), so the effects observed in effectiveness trials may not generalize when agencies attempt to use them with their broader workforce. Future studies should try to obtain data from clinics about their clinician characteristics to facilitate examination of sample representativeness.

The lessons learned from this trial and from the broader implementation science literature point to several recommendations for future effectiveness trials. First, our challenges recruiting adolescents point to different strategies needed to obtain a fully-powered, representative sample of adolescents. For example, it is possible that our research assessments were too burdensome relative to the compensation being offered, particularly for our baseline assessment. In particular, using the ADIS, which we selected to be consistent with prior efficacy trials of youth anxiety interventions and the UP-A (Ehrenreich-May et al., 2017b; Kendall et al., 2010), meant that baseline assessments were typically taking at least three hours. As the field increasingly moves toward transdiagnostic interventions (Marchette & Weisz, 2017), it is not clear that the level of diagnostic precision afforded by an assessment like the ADIS is really needed for inclusion and intervention evaluation purposes. Given that NIMH also requires a thorough assessment of treatment mechanisms under its experimental therapeutics approach, the field will need to rethink its approach to assessment to ensure thorough, but feasible assessment of treatment mechanisms and outcomes, particularly in effectiveness studies attempting to recruit under-resourced, historically under-represented families. It is also possible that families did not see the benefit of participating in research and were reluctant to take part. The emerging literature on the role of marketing evidence-based interventions directly to consumers (Becker, 2015) may provide increased knowledge of how best to increase the appeal of research participation and access to evidence-based practices.

On the clinician side, it is also possible that our study posed too much of a training and consultation burden, particularly for clinicians in the UP-A condition. While the training literature suggests that intensive training is likely needed to support clinician behavior change (Herschell et al., 2010), there is a critical need for research to identify “how low we can go” (Lyon et al., 2022) to balance training needs with the realities of clinical practice setting demands.

One design change that also may have solved some of the challenges encountered here is an implementation-effectiveness hybrid design (Curran et al., 2012) that studies effectiveness while also attending to implementation strategies. Designs focused on simultaneously attending to effectiveness and implementation have been highlighted as a key way to speed up the translation of science into public health impact (Beidas et al., 2023). Ideally, randomization would take place at the level of clinics or units within clinics so that implementation of MBC and UP-A could be supported through modified workflows, local supervision, and other organizational supports. In this study, where clinicians were randomized within agencies, clinicians assigned to the experimental conditions were often implementing new interventions in isolation, receiving support from study consultants but not from their regular supervisors or their colleagues. While this approached helped maximize internal validity by decreasing potential contamination and yielded higher power than randomization at the cluster level, it is likely increased clinician burden, decreased external validity, and lowered the potential for sustained intervention use after the trial ended. Effectiveness researchers can also benefit from applying implementation strategies into the design and conduct of the research activities, such as building time into the study to cultivate relationships with agencies, identifying and partnering with champions at each agency to support recruitment and data collection, and directly engaging clients and families into designing recruitment strategies (Powell et al., 2015).

In sum, while efficacious interventions exist for youth mental health problems, the “voltage drop” when interventions are tested in more complex and resource-limited settings hinders the public health impact of these interventions. Designs which can help with some of the limitations of efficacy trials, such as effectiveness trials, have their own limitations. The current study found difficulty in recruiting representative samples of community clinicians, retaining community clinicians in the study, recruiting youth and families into the study, and high heterogeneity and comorbidity in the youth sample. Several of these challenges may be addressed in future studies by applying more of an implementation science lens to effectiveness trials, including utilizing implementation-effectiveness hybrid design which simultaneously attends to implementation strategies and intervention effectiveness.

Disclosure Statement

Dr. Ginsburg receives funding from the National Institute of Mental Health (NIMH) and the US Department of Education, Institute for Education Sciences (IES), and serves as a consultant for the National Association of School Nurses. Dr. Ehrenreich-May receives funding from Patient-Centered Outcomes Research Institute (PCORI), the Children’s Trust, NIMH, IES, Henry Jackson Foundation, and the Batchelor Foundation. Dr. Ehrenreich-May makes a royalty from the sales of the Unified Protocols for Transdiagnostic Treatment of Emotional Disorders in Children and Adolescents, as well as a profit from some clinical training, consultation and implementation support services related to these published materials. Dr. Rosenfield received funding from NIMH, the National Cancer Institute, the National Institute on Drug Abuse, and PCORI. Dr. Jensen-Doss receives funding from NIMH. No other potential conflict of interest was reported by the author(s).

Additional information

Funding

This project was funded by the National Institute of Mental Health, awards MH106536 & MH106657.