Connecting technical assistance to implementation progress in child welfare capacity building efforts

ABSTRACT

Child welfare jurisdictions implement practices designed to improve child outcomes. Three federally funded Child Welfare Capacity Building Centers provide technical assistance (TA) to jurisdictions to support implementation of practice improvements. The aim of this study was to understand implementation progress among jurisdictions served by the Centers. Analyses indicate (1) amount of TA was positively associated with achievement of implementation milestones; (2) jurisdictions with higher levels of foundational capacity were more likely to achieve milestones; and (3) coaching was associated with faster milestone achievement. Methodological limitations and implications for future evaluations of TA in child welfare are discussed.

KEYWORDS:

• Child welfare
• technical assistance
• implementation
• organizational capacity
• Coaching

Disclosure statement

No potential conflict of interest was reported by the authors.

Data Availability Statement

The data that support the findings of this study are available from the corresponding author, HM, upon reasonable request.

Notes

1. Titles IV-B and IV-E of the Social Security Act enable federal funding for services to prevent child abuse and neglect. Title IV-E also provides federal financial assistance to state and tribal child welfare jurisdictions to cover costs associated with foster care maintenance and administration.

2. The Children’s Bureau administers state CIP grants to the highest court system in each state. The purpose is to assess and improve foster care and adoption laws and judicial processes. Current awardees include all 50 states, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands.

3. Centers make the determination of brief or intensive based on the estimated amount of time and support needed to achieve the outcomes identified for the project. In actual practice, however, some projects that were categorized as brief at the start of services ended up taking more than 9 months for various reasons (e.g., staff turnover, temporary halts in a project while an agency focused staff resources elsewhere, or natural disasters in the jurisdiction’s area). Similarly, an intensive project may achieve its aims sooner than expected.

4. The ICF Institutional Review Board found this project to be in compliance with the human subject protection requirements specified in HHS regulations, 45 CFR part 46 (approval number ICF IRB FWA00000845).

5. Direct TA refers to hours TA providers spent in live communication and partnership with jurisdiction staff; it does not include planning or preparation activities TA providers may do before or after working directly with a jurisdiction.

6. Some projects ended early because of interruptions from natural disasters in the jurisdictions’ service area, insufficient funding, or a decision by the jurisdiction to shift focus to other priorities.

7. Direct services are live communications, either in person or virtual, between Center TA providers and members of a jurisdiction, intended to build capacity; these services do not include hours that TA providers spent planning or preparing for service provision.

8. Note that because of the funding structure of the Collaborative, CBCS engaged in twice the number of projects and invested more hours and targeted more milestones than the other two Centers. Thus, no inferences beyond description should be made about these findings.

9. Centers selected and tracked achievement of implementation milestones only once projects were established. Centers frequently engaged jurisdictions in the earliest step of the CM Approach during the assessment process, prior to the start of a project. Thus, achievement of the three milestones in that step may not be reflected in these data.

10. Potential respondents were invited via e-mail to take the online survey and were provided a web link to it; nonrespondents received e-mail reminders to encourage participation.

11. CIP directors also had the option of receiving an electronic version of the survey administered by the study team immediately after the event.

12. The number of staff in tribal child welfare programs is typically smaller than that in states, and the study team aimed to limit burden on tribal nations by introducing data collection efficiencies wherever possible. Because the information included in the foundational capacity survey was captured in the CBCT assessment of tribes, the Center asked to use those data to complete the foundational capacity survey on behalf of the tribal staff. The study team granted that accommodation.

13. In many cases, project teams served by CBCS had multiple team members respond to the survey. (CIPs and tribes had only one respondent per team.) In cases where multiple respondents from the same team responded, an average team response was calculated for each survey item by taking the mean of the responses from each team member. We used these averages in the calculation of the foundational capacity score for those state projects.

14. The number of contact hours spent on achievement of specific milestones within any given step would be lower than the contact hours spent on achievement of all milestones combined (at the aggregate and for any given project), and so a smaller multiplier (×10) was used to clarify the results of these analyses, as compared with the multiplier (×100) used in the analysis of the effect of contact hours on achievement of all milestones combined.

15. Results available from the first author upon request.

16. We ran a series of models to test the effect of the proportion of TA hours that were in the form of coaching, consultation, or facilitation combined, and the effect of each TA strategy individually, on the dependent variables of milestone achievement and time (in months) to achievement. Results available upon request.

17. The number of observations in this model (n = 528) is lower than that in the milestone model above (n = 701) because not all jurisdictions provided foundational capacity survey data.

18. There were 88 unique jurisdictions served; some jurisdictions were served by more than one Center.

Additional information

Funding

This project was funded by the Children’s Bureau, Administration on Children, Youth, and Families, U.S. Department of Health and Human Services, under contract number HHSP233201400026C. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official views of the Children’s Bureau

Notes on contributors

Heidi Melz

Heidi Melz, Senior Research Associate at James Bell Associates, has 20 years of research and evaluation experience with policies and interventions that address child welfare issues and family structure, functioning, and wellbeing. Her recent work has focused on the evaluation of federally funded child welfare interventions, including the study of organizational capacity building through the evaluation of the Child Welfare Capacity Building Collaborative. Her program and policy expertise includes foster care, teen pregnancy prevention programs, and healthy marriage interventions. Dr. Melz received a Ph.D. in human development and family science from the University of Texas at Austin.

Tammy Richards

Tammy Richards, Senior Research Associate at James Bell Associates, has 19 years of experience in applied social science research, program evaluation, and evaluation technical assistance. In the past 10 years, her work has centered on child maltreatment, including research on the intersection of child protection and substance abuse, child abuse and neglect court case processes and outcomes, and implementation of initiatives to improve child welfare services. She has provided evaluation technical assistance to state and local agencies working to improve their child welfare practices. She specializes in literature reviews, development of conceptual frameworks, evaluation and instrument design, and mixed methods analysis. Previously, Richards was a research associate and analyst for the University of Southern Maine, Muskie School of Public Service, directing and conducting program evaluations of child and family services. Before her positions in research and evaluation, she provided direct services to adolescents, youth in foster care, adults with dual diagnoses, and domestic violence survivors. Ms. Richards received an M.Ed. in psychology from Antioch University New England.

Jing Sun

Jing Sun Senior Manager and Data Analyst at ICF, has 15 years of experience conducting research and evaluation in child welfare. For the evaluation of the Child Welfare Capacity Building Collaborative, Ms. Sun is the Quality Assurance and Analyst Lead for the service delivery and outcome data system used in this study. She brings to the analyses advanced training and experience in statistical analysis, including multilevel regressions, data visualizations, and Geographic Information System analysis of spatial data. Ms. Sun holds an M.A. degree in psychology from New York University.

Loren D. Masters

Loren Masters, Research Analyst at James Bell Associates, has 9 years of experience conducting public health and social science research. Her areas of content expertise include health disparities, child welfare, substance use and misuse, epidemiology, and prevention science. Her areas of statistical expertise include survey design, data management and integration with intensive longitudinal data, multilevel modeling, latent class analysis, time-varying effect modeling, and linear and logistic regression analysis. Ms. Masters received an M.P.H. from Penn State College of Medicine.

Anita P. Barbee

Anita P. Barbee, Professor and Distinguished University Scholar at the Kent School of Social Work and Family Science, Center for Family and Community Well-Being, University of Louisville, has been evaluating child welfare training, practice innovations, and workforce interventions through state contracts and federal grants for the past 30 years. She has been a part of teams evaluating the Children’s Bureau efforts to provide training, technical assistance and capacity building to states, tribes, and courts for the past 18 years. Dr. Barbee received her Ph.D. in social psychology from the University of Georgia in 1988 and her MSSW from the University of Louisville in 2001.