Effectiveness and Efficiency of Observationally Assessing Fidelity to a Family-Centered Child Intervention: A Quasi-Experimental Study

Abstract

Assessment of fidelity that is effective, efficient, and differentiates from usual practices is critical for effectively implementing evidence-based programs for families. This quasi-experiemntal study sought to determine whether observational ratings of fidelity to the Family Check-Up (FCU) could differentiate between levels of clinician training in the model, and from services as usual, and whether rating segments of sessions could be equivalent to rating complete sessions. Coders rated 75 videotaped sessions—complete and 20-min segments—for fidelity, using a valid and reliable rating system across three groups: (a) highly trained in FCU with universal, routine monitoring; (b) minimally trained in FCU with optional, variable monitoring; and (c) services as usual with no training in the FCU. We hypothesized that certain dimensions of fidelity would differ by training, whereas others would not. The results indicated that, as expected, one dimension of fidelity to the FCU, Conceptually accurate to the FCU, was reliably different between the groups (χ² = 44.63, p < .001). The differences observed were in the expected direction, showing higher scores for therapists with more training. The rating magnitude of session segments largely did not differ from those of complete session ratings; however, interrater reliabilities were low for the segments. Although observational ratings were shown to be sensitive to the degree of training in the FCU on a unique and theoretically critical dimension, observational coding of complete sessions is resource intensive and limits scalability. Additional work is needed to reduce the burden of assessing fidelity to family-centered programs.

ACKNOWLEDGMENTS

Justin Smith was supported by National Institute on Drug Abuse grant DA027828 (PI Hendricks Brown); National Institute of Mental Health research training grant MH20012 (PI Elizaebeth Stormshak); and by the Implementation Research Institute (IRI) at the George Warren Brown School of Social Work, Washington University in St. Louis through an award from the National Institute of Mental Health (5R25MH08091607) and the Department of Veterans Affairs, Health Services Research & Development Service, Quality Enhancement Research Initiative (PI Enola Proctor). Thomas Dishion, Daniel Shaw, and Melvin Wilson were supported by National Institute on Drug Abuse grant DA016110. Samantha Montag was supported by National Institute on Drug Abuse grant DA027828. Elizabeth Stormshak was supported by Centers for Disease Control and Prevention grant CE001389. The authors thank the members of the research team in Eugene, Phoenix, Pittsburgh, Portland, and Charlottesville, and the families who participated in the two trials. Specifically, we appreciate the contribution of Lisa Brauer for her work in developing the COACH fidelity coding system and coding of videotaped intervention sessions. We are also grateful to C. Hendricks Brown for statistical consultation and Enola Proctor for feedback on early drafts of this paper.

DISCLOSURE STATEMENT

No potential conflict of interest was reported by the authors.

Notes

¹ The two trials administered different caregiver reports of child problem behaviors. Study 1 administered the Child Behavior Checklist (CBCL; Atkins, Steyvers, Imel, & Smyth, 2014), a multiscale questionnaire used to assess behavioral problems in youth ages 1.5 to 18. Study 2 used the five-item Conduct Problems subscale of the Strengths and Difficulties Questionnaire (SDQ; Gallo et al., 2014). Five items from the CBCL were selected that matched those of the SDQ (i.e., fighting, lying, stealing, noncompliance, losing one’s temper). Internal consistencies were acceptable for the SDQ (α = .80) and the five items of the CBCL (α = .77).

² The p values are given to the fifth decimal point in text because of the Bonferroni correction.

³ Unstandardized results are presented due to Mplus only calculating unstandardized path estimates for indirect effects when using Bayesian estimation.

⁴ Complete results of the reanalysis are available by request from the first author.