External and Internal Validity Considerations in Youth Effectiveness Trials: Lessons Learned from the COMET Study

ABSTRACT

Objective

Effectiveness trials aim to increase the generalizability and public health impact of interventions. However, challenges associated with this design present threats to external and internal validity. This paper illustrates these challenges using data from a two-site randomized effectiveness trial, the Community Study of Outcome Monitoring for Emotional Disorders in Teens (COMET) and presents recommendations for future research.

Method

COMET was a randomized effectiveness trial conducted in 19 community mental health clinics in two states comparing three interventions: treatment as usual (TAU), TAU with measurement-based care (TAU+), and the Unified Protocol forTransdiagnostic Treatment of Emotional Disorders in Adolescents with MBC (UPA). Participants included 176 clinicians (mean age = 35.5; 85.8% cisgender female; 53.0% racially and/or ethnically minorized) and 196 adolescents (mean age = 14.7; 65.3% cisgender female; 69.4% racially and/or ethnically minorized). Analyses outlined participant flow from recruitment to study completion, described participant characteristics, and examined site differences.

Results

Analysis of participant flow suggested that recruitment and retention of clinicians and adolescents was challenging, raising questions about whether participants were representative of participating clinics. Both the clinician and adolescent samples were racially and ethnically diverse and adolescents were low income and clinically complex. Significant site differences were observed in clinician and adolescent characteristics.

Conclusions

While this study was successful in recruiting a diverse and historically under-represented sample, difficulties in recruitment and retention raise questions about external validity and site differences present challenges to internal validity of study findings. Suggestions for future effectiveness studies, drawing from implementation science approaches, are discussed.

Disclosure Statement

Dr. Ginsburg receives funding from the National Institute of Mental Health (NIMH) and the US Department of Education, Institute for Education Sciences (IES), and serves as a consultant for the National Association of School Nurses. Dr. Ehrenreich-May receives funding from Patient-Centered Outcomes Research Institute (PCORI), the Children’s Trust, NIMH, IES, Henry Jackson Foundation, and the Batchelor Foundation. Dr. Ehrenreich-May makes a royalty from the sales of the Unified Protocols for Transdiagnostic Treatment of Emotional Disorders in Children and Adolescents, as well as a profit from some clinical training, consultation and implementation support services related to these published materials. Dr. Rosenfield received funding from NIMH, the National Cancer Institute, the National Institute on Drug Abuse, and PCORI. Dr. Jensen-Doss receives funding from NIMH. No other potential conflict of interest was reported by the author(s).

Additional information

Funding

This project was funded by the National Institute of Mental Health, awards MH106536 & MH106657.