Implementation Science and Prevention in Action: Application in a Post-Permanency World

ABSTRACT

This article describes how implementation science and intervention research guided the process of selecting and implementing an evidence-informed intervention (Tuning in to Teens; TINT). TINT was provided as a selective prevention effort offered to families with youth aged 10 to 13 years old, with characteristics that suggest an elevated risk for post-permanency discontinuity. Usability testing findings: Contact was made with 54% of families, and 12% participated in the intervention. Multivariate results found no statistically significant differences between families who responded to outreach efforts and those who did not; families who participated in TINT and those who did not. Implications: Large public child welfare systems wanting to implement evidence-informed interventions can follow the steps detailed in this paper for selecting, adapting and implementing an intervention. Further, providers that seek to offer post adoption and guardianship services, a growing service need, may gain some insights into activities that promote service usage with this population.

KEYWORDS:

• Adoption
• implementation
• intervention research
• science selective prevention
• Tuning in to Teens

View correction statement:

Correction

This manuscript outlines the process of the National Quality Improvement Center for Adoption and Guardianship Support and Preservation (QIC-AG) approach to implementing and evaluating an intervention. The QIC-AG is a 5-year project funded by the Department of Health and Human Services, Administration for Children and Families, Children’s Bureau. The QIC-AG is testing interventions in eight sites across the United States. The conceptual lens applied to the QIC-AG amalgamates key components of implementation science (Fixsen, Blase, Naoom, & Wallace, 2009; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005), program evaluation (Testa, 2010), and intervention research (Fraser, Richman, Galinsky, & Day, 2009; Permanency Innovations Initiative Training and Technical Assistance Project (PII), 2016). Recent passage of the Family First Prevention Services Act in 2018 requires the use of “promising, supported, well-supported” (research-informed) services and practices. One place to look for research-informed interventions in child welfare is the California Evidence Based Clearinghouse for Child Welfare (CEBC). However, the CEBC lists few interventions that meet the criterion of “well supported by research” (Framework Group, 2014). The process of building evidence for what works in child welfare is needed; yet, the process toward adapting research-informed interventions for use with a child-welfare population is something that needs to be better understood. This paper addresses this gap by providing a case example of how a large state agency went about the process of developing a theory of change and problem statement. They then went on to select a research-informed intervention and adapted it for their specific target population of adoptive and guardianship families with youth between 10 and 13 years old at the onset of the intervention. This paper will summarize the approach of the predevelopment of an intervention and then focus in on how that approach was applied in one site (New Jersey), including a discussion of the strengths and challenges associated with merging the approach into a public child welfare system. This paper will conclude with some preliminary analysis of recruitment results from usability testing and the first round of the formative evaluation from the New Jersey project.

QIC-AG approach

The QIC-AG established a conceptual framework to guide the work of the project, the QIC-AG Permanency Continuum Framework (PCF; qic-ag.org). Acknowledging that preparation for adoption or guardianship needs to begin upon initial contact with the child welfare system, the PCF outlines stages pre- and post-finalization to ensure long-term stability and well-being. This manuscript focuses on the state of New Jersey’s efforts to implement a selective prevention intervention for families who have adopted or assumed guardianship of children formerly in foster care. Selected interventions target groups with an elevated risk for a specific issue; they are designed to address problems early on and allow limited resources to be targeted at a group that is most likely to benefit (Offord, 2000; Springer & Phillips, 2006).

Identify and explore

The first stage in the implementation of a research-based intervention is referred to as “identify and explore” or the “exploration” phase (Fixsen et al., 2005; Framework Workgroup, 2014; Fraser et al., 2009). This stage sets the tone for the implementation process and lays the foundation for a rigorous evaluation. During this phase, an intervention is selected that is hypothesized to address the needs of a deliberately identified target population. As described by the Framework Workgroup (2014), the purpose of this stage is to (a) identify and understand the problem, (b) develop a theory of change, (c) research solutions, and (d) select an intervention to apply and improve, replicate and adapt, compare and learn, or develop and test.

Identify and understand the problem

A population template (adapted from the Federal government’s IV-E Waiver sites and PII projects (PII, 2016; U.S. Department of Health & Human Services [USDHHS], 2013) was utilized to assist with defining the target population. In many research projects, existing data are used to guide this process; however, little data exist that follow children and families after adoption or guardianship has been finalized. In addition, post-permanency services are not mandated and their availability varies from state to state (see, for instance, Fuller, Bruhn, Cohen, Lis, Rolock, & Sheridan, 2006; Houston & Kramer, 2008; Vandivere, Malm, & Radel, 2009). Similarly, the tracking of long-term outcomes, after a state has finalized an adoption or guardianship, is not typically conducted, and the calculations are not straightforward (see, for instance, Rolock, 2015; Rolock, & White, 2016). To address this concern, the project utilized data from a broad range of sources, including discussions with stakeholders, key informants, and anecdotal data collected during this phase.

In New Jersey, this phase began with the engagement of a Project Management Team (PMT) and a Stakeholder Advisory Team (SAT) to work with the QIC-AG team. The QIC-AG team included two researchers associated with the QIC-AG evaluation team and two QIC-AG implementation staff who worked in partnership with the New Jersey Site Implementation Manager. The PMT included the additional staff from the New Jersey Department of Children and Families (DCF) representing different state functions. The SAT included professionals and consumers who can identify cross-cutting needs of children and families within the community. In addition, facilitated meetings were held with New Jersey families who had adopted or assumed guardianship and post-adoption counseling (PAC) providers. Together, these stakeholders and staff provided insight into the challenges faced by adoptive parents and guardians and the supports they thought would be most helpful. Families shared that they felt isolated, expressed the need for support from other parents, and reported that they were unaware of services available to them. While most parents reported that they had no immediate concerns, they were worried about what their families may experience as their children approached adolescents. Families reported that they lacked the information about what to expect during this developmental phase, and they were unaware of support and services available to them or their children. The PAC providers reported that there were resource gaps in what was available to offer families post-adoption. In addition, stakeholders and DCF staff reported that DCF was expanding post-adoption services to include families who had achieved permanence through kinship legal guardianship (KLG). There was a robust array of services including PAC provided by therapists who have received post-degree certification in adoption competence through a partnership between the DCF and Rutgers University.

In addition to these meetings, the evaluation team examined the state administrative data on post-permanency outcomes. As part of this project, the evaluation team examined the long-term outcomes for children adopted or who exited foster care through KLG in New Jersey to determine whether there was a developmental stage at greater risk and covariate predictors of risk. This analysis revealed that post-permanency discontinuity was most likely to occur for children in adolescence. Their likelihood of discontinuity in the adolescent years significantly increased for those who had lived in group care (a shelter, treatment home, or congregate care setting) during their time in foster care. The likelihood of discontinuity was also higher for children whose adoption or guardianship was finalized between the ages of 6 and 13 years of age.

The identify and explore phase required the team to put predetermined solutions aside while the gathering of information from a variety of stakeholders occurred. Feedback from the stakeholders caused DCF to rethink their assumptions about access to services. Prior to this process, there was an assumption that access to services would occur through a comprehensive resource clearinghouse of a variety of resources, which families were informed of in annual mailings. Upon hearing that families were unaware of how to access services and felt unsupported after adoption or guardianship, DCF implemented an immediate change in services. To help ensure that families knew services were available and how to access them, DCF expanded upon their existing “welcome letter” to newly adopted or KLG families to include the personal contact information of the family’s subsidy worker and instituted a follow-up phone call by that worker to the family. While this may affect the comparability of past outcomes, raising awareness of this issue resulted in DCF staff feeling compelled to provide an immediate response.

Through meetings with families, and the PMT and SAT workgroups, the following problem statement was developed: There are developmental tasks in adolescence that may be complicated by adoption or guardianship. Post-adoptive or KLG families may be unprepared to address these unique challenges; without training and supports, these families are at increased risk for post-permanency discontinuity.

Develop a theory of change

The next step was to develop a theory of change, a description of how and why a desired change is expected to occur. A well-articulated theory of change serves as a guide, or road map, for how the intervention is expected to work, why the change is expected to occur, and what steps are necessary to bring about the expected change. It is also used to identify the intervention that will be implemented. An essential step in developing a theory of change is developing a common understanding of the problem to be addressed, and how and why the problem should be addressed. In this project, the work of developing the Theory of Change was guided by a root cause analysis (isixsigma, n.d.) to arrive at this common understanding. This work resulted in the following theory of change for New Jersey:

There are developmental tasks in adolescence that may be complicated by adoption or guardianship. Post-adoptive or Kinship Legal Guardianship (KLG) families may be unprepared to address these unique challenges; without training and supports, these families are at increased risk for post-permanency discontinuity. If we increase the skills and knowledge associated with caring for youth as they enter adolescence, then there will be an increase in the capacity of the parents or guardians to address the issues within their families. If parents or guardians are able to meet the needs of the youth in their families then there will be increased post-permanency stability.

Research solutions

Once a theory of change is developed, an intervention must be identified that addresses the “if” statements in the theory of change. QIC-AG sites were asked to select interventions that had been previously tested, rather than developing new interventions. Ideas for interventions were obtained through an examination of exiting data bases that summarize the level of evidence for existing interventions. The NIRN Hexagon Tool was used to assess how well a particular intervention might meet identified needs; how it fits with current practices and community and agency values, and cultural considerations; the availability of training, level of development of the curriculum; the level of evidence from prior research that the outcomes that might be expected if the program or practices are implemented well, and, the expertise and availability of the intervention purveyor to assist with training and adaptations that might be necessary (Blase, Kiser, & Van Dyke, 2013).

Intervention selection

New Jersey Team reviewed written program details for more than 20 interventions, read the associated research, and gathered information available on line about the interventions. Using A Framework to Design, Test, Spread, and Sustain Effective Practice in Child Welfare (Framework Workgroup, 2014), they closely examined interventions that had been previously tested. No research-based programs could be identified that addressed the specific challenges likely to be experienced in adolescence with children who were adopted. Therefore, interventions were identified that addressed either post-permanency issues or adolescent development issues, with the understanding that the team would need to supplement the intervention and test the revised model against the original. After a broad search, the list of potential interventions was narrowed down to three possible interventions that were inputted into the Hexagon Tool (Blase et al., 2013). In addition, the New Jersey team interviewed purveyors for the top three interventions and met with staff who had implemented the interventions. This information was compiled into a report that was presented to the SAT and PMT. Tuning in to Teens (TINT) was unanimously selected as the intervention as it (a) addresses the developmental stage of adolescence, (b) is provided in a group setting providing an opportunity for social support, (c) focuses on the parent, (d) is intended to enhance skills rather address deep-end problems (selective intervention stage appropriate), and (e) could be easily adapted to include the adoption lens. It was determined that engaging parents or guardians of children aged 10–13 would be the appropriate developmental stage for this preventative service. The hypothesis was that parents or guardians with children in this age range would be motivated to participate as they considered the challenges associated with adolescence.

There was some concern that TINT, developed in Australia, might not be culturally congruent for the diverse families of New Jersey. Accessibility of the purveyor for training and supervision was also considered, as the purveyor is not based in North America. Finally, there were concerns that while TINT had proven successful with many families, including blended families, there was no specific research on its impact with adoptive families (Havighurst, Harley, Kehoe, & Pizarro, 2012; Havighurst, Kehoe, & Harley, 2015; Kehoe, Havighurst, & Harley, 2015, 2014). Despite these concerns, the New Jersey implementation team was confident that the purveyor’s flexibility and willingness to support adaptation and implementation made TINT the best choice.

Implementation planning

Once an intervention was identified and the research question crafted, the next step in the process was to give thoughtful and careful consideration to the “what, how, and who” of offering the intervention. The cornerstone of implementation planning was the completion of the Initial Design and Implementation Plan (IDIP). The IDIP served as a tool to thoughtfully and strategically plan for successful implementation of the initiative and to ensure that the initiative had intervention validity and implementation integrity. To accomplish this, the IDIP detailed the following: (a) development of a research question and plan, (b) what will be implemented, (c) how the system will be modified or readied to support the intervention, and (d) who is going to do the work. If done well, an IDIP has many benefits, including the promotion of a well-developed, logical approach to implementation and the description of strategies to address ongoing implementation issues (Fixsen et al., 2009, 2005; USDHHS, 2013). Planning activities provide the process for thinking through the intervention’s critical components, allowing for anticipation of possible barriers and the steps to address them, and developing a common understanding of how the identified program goal will be achieved. Each of the sections of the IDIP will be described below.

Development of a research question and plan

The development of a well-built research question and a plan for recruiting and tracking of study participants are critical primary steps to a successful evaluation. Using guidance from the medical field, and other federal and state initiatives (Richardson, Wilson, Nishikawa, & Hayward, 1995; Testa, 2010), the project used the PICO framework to arrive at their site-specific research question. This includes clear articulation of four key components: P – a well-defined target population, I – intervention to be evaluated, C – comparison group, and O – outcomes expected to be achieved. The New Jersey team developed the following primary research question:

Will children currently between the ages of 10 & 13 who are receiving an adoption or KLG subsidy, are not open for services with DCF, and meet one of the following criteria: at the time of finalization were between the ages of 6 and 13, or were in group care while in foster care (P) experience a reduction in post-permanency discontinuity, improved well-being, and improved behavioral health for children and youth (O) if they receive Tuning in to Teens (TINT) (I) compared to similar children who receive services as usual (C)?

Once the PICO elements are well defined, the project logic model naturally flows from this discussion. A logic model elaborates on the PICO framework through a graphical depiction of the relationship between program inputs, implementation supports, program outputs, short-term outcome, and long-term outcomes as well as factors such as external conditions and end values (Testa, 2010). A point of clarification on the age criterion is used for identifying the target population. A child’s age at adoption or guardianship (finalization) and a child’s current age were both used to identify the target population. A child’s current age needed to be between the ages of 10 to 13, and their age at finalization needed to be between the ages of 6 and 13.

What will be implemented?

An intervention cannot be well implemented if it is not well understood. The IDIP allows a site to explore the intricacies of the intervention by building upon the information gathered though the use of the Hexagon Tool (Blase et al., 2013) previously described, including the availability of materials needed to implement the intervention and think comprehensively about the fit of the philosophy, values, and principals of the intervention with existing initiatives and values of the site, as well as need for adaptations. An adaptation workgroup, made up of professionals working for or with the DCF Adoption Operations unit and QIC-AG implementation staff, was formed.

One concern raised about TINT was the need to adapt it to a US environment. TINT was developed at the University of Melbourne, Center for Mindfulness in Australia. The workgroup determined that TINT facilitators who are sensitive to cultural nuances in how the materials are perceived would be able to use the existing curriculum with little adaptation; there was no need to completely rewrite the curriculum in American Standard English. Finally, TINT had not been previously tested with families who had adopted or assumed guardianship, so the curriculum was reviewed to assess areas where it may need to be augmented to address issues related to families formed through adoption or guardianship. The adaptations workgroup reviewed each chapter of the curriculum and developed adoption and kinship overlays consisting of one or two pages of information that to address the unique needs of families formed through adoption or guardianship. Examples of issues that the workgroup identified were trauma (how this impacts the youth’s current behavior), identity formation (learning and discussing the youth’s birth history and birth family), feelings of abandonment and rejection (from the perspective of the youth and the parents or guardians), and a sense of belonging (especially as the youth seeks autonomy during adolescence). Additionally, the workgroup recommended vignettes and examples to reflect the experiences of adoptive parents and guardians and their children. These overlays were shared with the purveyor. To accommodate the time needed to address these adaptations, an extra week was added to the curriculum delivery.

How the system will be modified or readied to support the intervention

This examination involved examining the system’s existing capacity to support service delivery. The site developed a work plan that delineated the activities that need to occur before the first clients can be served. Through consultation with the PMT and SAT, the outreach procedures were developed. The PMT and SAT were particularly helpful in setting the tone and content of the outreach materials. Outreach materials included a flyer in the first mailing, followed by a letter that explained the project. Outreach to families was targeted by county, and addresses were used to determine the most proximate training sites for the majority of families. Outreach efforts began approximately 8 weeks prior to the start of TINT sessions. Approximately 2 weeks after the mailings, families were called by phone to ascertain their interest in the project. In order to reach the most people, calls were made at various times throughout the workday, evenings, and weekends. In considering system adjustments that were needed in New Jersey, the tracking and sharing of information related to the intervention required the most development and work. Finally, data use agreements were established to define the protocol for the secure transfer and sharing of sensitive information.

Who is going to do the work?

When implementing an intervention, it is important to delineate not only what needs to be done but also who specifically is going to do the work. TINT implementation requires facilitators who were (a) knowledgeable of the child welfare system, (b) trained in adoption competence, (c) experienced in working with families, and (d) preferably experienced in managing groups. Recruitment of candidates included post-adoption counselors and state agency staff. It was important that this group was ethnically and culturally diverse. After the 20 facilitators were selected, the purveyor, Dr. Sophie Havighurst, came to New Jersey from Australia to deliver the 2-day training. Given the intensity of the sessions, a co-facilitator model was used, pairing a state staff with an adoption clinician. One facilitator was designated as the lead facilitator and was responsible for reporting on session fidelity, addressing implementation needs, and coordinating the web-based supervision with the purveyor.

Further, the team considered carefully the various likely predictors of enrollment and barriers to participation that are outlined in the Conceptual Model of Parent Involvement (McCurdy & Daro, 2001). This model illustrates the many individual, provider, program, and neighborhood factors that may lead to an intent to enroll, enrollment, and eventual retention in services. While this is a detailed staged model that presumes rational decision-making, in brief it suggests that participants must weigh numerous factors when deciding whether to participate in a service, such the cost-benefit, readiness to change, social acceptability of using a service, and their actual ability to participate. Service providers can make it easier for potential participants to engage in a service in a number of ways, such as by making it accessible, timely, relevant, and effective. The QIC-AG project team considered these items when deciding the time of day and distance to travel, child care costs, provision of food, facility type (i.e., other types of services in the building), language for the recruiting material, phone engagement process, and others with the hopes of reaching the intended participants as effectively as possible.

Installation

During installation, the ground work was laid for completing usability testing and ensuring that the supports needed for implementation were in place. During usability testing, a formal implementation plan was developed to ensure that the intervention could be delivered as intended and that the processes were in place and working as intended. Once the site assessed the system, made necessary adjustments, they moved to the formative evaluation phase.

During usability testing phase, TINT was offered in three sites. Two facilitators and two observers were assigned for each site. Since TINT is an evidence-based model, the primary work of the usability phase was to refine the recruitment and engagement process. The team met and established a series of questions that they would like to address during the usability phase of the project; questions such items as does the tracking sheet capture the information we want to capture? Can the calls be made in accordance with the schedule? Does the phone call protocol allow us to reach the family by phone? Is geography a hindrance to registration/attendance? Usability testing allowed the project to ascertain what was working well, and what needed adjusting. As a result of usability testing, project staff made adjustments to the tracking sheet and were able to establish a call schedule that worked for both families and corresponded to staff availability. Project staff were able to map the distance between where families lived and the implementation sites. With this information, they identified clusters of families in certain areas of the state for targeted outreach with subsequent outreach efforts.

Prior to implementing the study, the institutional review boards at the universities of the first and second authors and the Research Review Committee at the New Jersey DCF reviewed and approved the research protocol, including the outreach and consent protocol, associated with this study. Key staff responsible for assisting with the intervention completed training on human subject protections through the second author’s university.

Preliminary intervention recruitment results

At the initial phase of intervention implementation, DCF staff collected data on their outreach efforts. For this manuscript, and prior to the end of the project, these data were analyzed by the evaluation team. The goal of this analysis is to provide a preliminary response to the following two questions: (a) what were the outcomes of recruitment efforts (i.e., number of families contacted, registered, and attended) and (b) were there any differences in the recruitment efforts (contact with the family, registration, and attendance) based on the pre-permanency factors or child demographics available in the administrative data? This section will summarize results from the recruitment process for the first two cohorts, the usability cohort (cohort 1), and the second cohort, which also serves as the first cohort of the formative evaluation.

Data

The samples selected for the usability cohort were families who had adopted or assumed guardianship in three New Jersey counties and had a child between the ages of 13 and 14 and met one of the following conditions: (a) the child was adopted or guardianship was transferred when the child was aged 6 or older or (b) the child was in a congregate care setting prior to aged 6. Children whose families were not currently receiving an adoption or guardianship subsidy, and those currently active with DCF, were excluded from the sample. For the usability cohort, the child’s current age was slightly higher than the target population, in an effort to preserve the target population for the formative evaluation phase. The samples selected for the second cohort (the start of the formative evaluation) were children currently between the ages of 10 and 13, all other selection criteria were the same.

During the first two cohorts, a total of 503 families were assigned to the study. Of those, one was deceased, and two could not be linked to the administrative data. These 3 cases were omitted from the analytical sample, bringing the sample to 500 families. Of the 500 families eligible for randomization, 383 (77%) were assigned to the intervention and 117 (23%) to the comparison condition (Table 1). Finally, five cases were missing the information on one variable (miles from home to the training site) and were excluded from the multivariate analysis. Data on recruitment efforts were linked to DCF administrative data. The administrative data were used to examine differences in recruitment outcomes for families selected to participate in the early cohorts of the project. These data are used to examine differences in recruitment outcomes (e.g., families who were successfully contacted vs. those who were no successfully contacted; differences in registration and attendance outcomes).

Table 1. Number of children assigned to the intervention.

	Intervention		Comparison		Total
Cohort 1	165	85%	29	15%	194
Cohort 2	218	71%	88	29%	306
Total	383	77%	117	23%	500

Dependent variables of interest

Two dependent variables were examined, contacted by staff and attended.

Contacted by project staff

Families for whom the project staff was able to make contact, through phone or email, were coded as 1, not able to contact were coded 0.

Attended

Attendance at TINT sessions was coded as 1 for any family who attended 4 or more sessions, otherwise they were coded as 0. The threshold of 4 is recommend by the purveyor.

Covariates of interest

Child demographic, program, and several pre-permanency characteristic variables were used.

Child age at the time of permanence

Prior research has found that children adopted as very young children experience a lower risk of instability after permanence (Palacios, Rolock, Selwyn, & Barbosa-Ducharne, 2018; Rolock, & White, 2016; Rolock, White, Ocasio, Zhang, MacKenzie, & Fong, 2018). Age was entered as a continuous variable.

Time since permanence

Extant research has found that instability after adoption or guardianship typically takes several years post-finalization. The time (in years) was entered as a continuous variable.

Number of placements while in foster care

Based on federal guidelines (Adoption and Foster Care Analysis and Reporting System), the following types of placement settings were excluded from the count of placements: temporary stays in hospitals, camps, respite care, runaway episodes, and institutional placements. However, any of the following were considered to be distinct placements: shelter care, foster care, kinship care, treatment foster care, group homes, residential treatment, and independent living placements. The number of placements in foster care was coded as a continuous variable.

Placed with a relative

Extant research has found that kin placements are often more stable than non-kin placements. Children who were adopted or in KLG with a relative were coded as 1, and non-relative adoptions were coded as 0.

Type of legal permanence

Adoption was coded as 1 and KLG as 0.

Placed in congregate care while in foster care

Congregate care was included as a way to account for potential difficulties that may arise after the formal foster care services and supports have ended. Children who were ever placed in congregate care (i.e., an institution or group home setting) while in foster care, and prior to adoption, were coded as 1; children who were never placed in congregate care were coded as 0.

Long-term foster care

A variable was created that counted the time a child spent in foster care (i.e., from the date of initial case opening to date of adoption finalization). Children who spent 3 or more years in foster care were coded as 1 (having experienced long-term foster care), and children who spent less than 3 years in care were coded as 0.

Child’s race

Child’s race was coded as 1 if African-American, otherwise coded as 0.

Child’s gender

Child’s gender was coded as 1 if female, otherwise as 0.

Number of outreach calls

The number of outreach calls was entered as a continuous variable.

Distance to training

The distance (in miles) between the family’s home address and the training location was entered as a continuous variable.

Analysis plan

The outcomes of the recruitment efforts will be examined and bivariate statistics will be used to analyze group differences. Two key recruitment outcomes will be explored: Contact with families and intervention session attendance. These two outcomes will be explored as dependent variables in bivariate and multivariate analysis. Key covariates include demographic characteristics and recruitment data collected as part of the project. In addition, the project data were linked to child welfare administrative data to examine pre-permanency characteristics. We employed logistic regression to examine the impact of the covariates on the two binary outcomes of interest (contact with families and session attendance).

Results

Recruitment for TINT occurred with the 383 children assigned to the intervention group. Of the 383 families who were sent a recruitment letter and received follow-up phone calls, staff were able to make contact with 54% (205) of the families; 21% of families registered for the intervention, there was an overall attendance rate of 14%, and 12% ultimately attended at least 4 sessions (Table 2). Approximately half of the sample (49%) could not be reached by phone and it is unknown whether they received the mailing, although any mail that was returned to sender resulted in a search for alternative address through a search service. These families may have been passively declining, as opposed to those families who declined services after talking with a staff member. For those who declined services after speaking with project staff, staff asked them for the reason they declined; 42% said that it did not fit their schedule; 38% stated that they were doing fine, or were not interested in the intervention; and the remaining 20% cited a variety of reasons, including their poor health, issues related to a special-needs child, or cited transportation-related issues.

Table 2. Recruitment efforts and results.

Assigned to the intervention group		Contacted by project staff		Registered		Attended any sessions		Attended four or more sessions
Cohort 1	165	84	51%	26	16%	27	16%	23	14%
Cohort 2	218	121	56%	53	24%	26	12%	22	10%
Total	383	205	54%	79	21%	53	14%	45	12%

Bivariate results (summarize in Table 3) found very little differences on most characteristics. The exceptions were the program variables. Staff were more likely to make contact with families who lived closer to the training site. In addition, those families that the staff were unable to make contact with received more calls than those who were successfully contacted.

Table 3. Bivariate results.

			Contacted (n = 205)				Attended (n = 45)
	All (n = 383)		Yes		No		Yes		No
	N	%	N	%	N	%	N	%	N	%
Child’s gender
Female	221	58	95	46	111	54**	26	13	180	87
Male	162	42	110	62	67	38	19	11	158	89
Child’s race or ethnicity
African-American	218	57	113	52	105	48	33	15	185	85
White	78	20	42	54	36	46	5	6	73	94
Hispanic	51	13	27	53	24	47	4	8	47	92
Other	36	9	23	64	13	36	3	8	33	92
Kinship relationship
Kin	179	47	92	51	87	49	14	8	165	92*
Non-kin	204	53	113	55	91	45	31	15	173	85
Type of legal permanence
Adoption	310	81	165	53	145	47	38	12	272	88
Guardianship	73	19	40	55	33	45	7	10	66	90
Time in foster care
Less than 3 years	162	42	84	52	78	48	19	12	143	88
Three or more years	221	58	121	55	100	45	26	12	195	88
Ever placed in congregate care prior to permanence
In congregate care	51	13	28	55	23	45	12	24	39	76**
Never in congregate care	332	87	177	53	155	47	33	10	299	90
Mean	M	SD	M	SD	M	SD	M	SD	M	SD
Age at permanence	7.47	2.83	7.42	2.90	7.53	2.74	6.93	3.08	7.54	2.79
Miles from training center^++	7.19	5.94	6.87	5.63	7.56	6.29	5.78	3.86	7.38	6.15
Number of outreach calls	2.33	1.19	1.86	1.08	2.87	1.07***	1.49	0.84	2.44	1.18***
Time since permanence	4.72	2.86	4.78	3.01	4.66	2.68	5.32	3.14	4.64	2.82
Number of moves while in foster care	3.23	2.14	3.13	2.06	3.36	2.22	3.22	2.08	3.24	2.15

⁺⁺N = 378

*p < 0.05; **p < 0.01; ***p < 0.001.

Results from the logistic regression models, summarized in Table 4, found very little differences in pre-permanency characteristics between those families for whom the staff were able to contact and those for whom they were not able to contact. Two variables rose to the level of statistical significance. Families with daughters (compared to sons) were more likely to be successfully contacted by staff (HR = 2.29; 95% CI [1.42–3.70]). In addition, more calls to families were associated with increased likelihood of successful contact (HR = 2.36; 95% CI [1.91–2.93]). In addition, little difference was observed in characteristics between those who attended TINT sessions and those who did not. The only variable that was statistically significantly associated with attendance was the number of calls made to families (HR = 2.33, 95% CI [1.63–3.34]).

Table 4. Logistic regression results.

	Predicting contact with project staff			Predicting attendance
Variable^a	HR	95% CI		HR	95% CI
Female (male)	2.29***	1.42	3.70	0.82	0.41	1.63
African-American (not African-American)	1.62	0.98	2.68	0.52	0.23	1.16
Age at permanence	1.01	0.82	1.23	1.12	0.83	1.52
Miles from training center	1.03	0.99	1.07	1.05	0.98	1.12
Number of outreach calls	2.36***	1.91	2.93	2.33***	1.63	3.34
Time since permanence	0.95	0.78	1.16	1.04	0.78	1.40
Number of moves while in foster care	1.11	0.98	1.25	1.04	0.88	1.24
Permanence with kin (non-kin)	1.23	0.74	2.06	2.09	0.95	4.60
Adoption (guardianship)	1.00	0.53	1.90	0.93	0.34	2.54
Length of time in foster care	0.63	0.36	1.09	0.97	0.43	2.17

^aReference categories for categorical variables shown in parentheses.

*p < 0.05; **p < 0.01; ***p < 0.001.

Conclusion and implications for intervention research

As part of a national 5-year study, this study reports on the experience of taking implementation science and rigorous intervention work and putting it into action in a “real-world” scenario. The project team adapted a research-supported intervention for use with a new target population. Implementation occurred in a large public child welfare system; it required practice as usual to be retooled to adapt to the requirements of implementation science and rigorous evaluation. Furthermore, prevention efforts were targeted at a population that the child welfare system had not approached in this fashion before.

From the child welfare system perspective, taking seriously the role of intervention research required the system to think through things differently than they would traditionally, such as when implementing a new project in response to a mandate or new funding. With the QIC-AG project, an interdisciplinary team led this effort at the local level. Practitioners, those with an interest in implementation work, key stakeholders, and university researchers collaborated to assess the available information, research solutions, and setup systems to collect information that could later be analyzed. This required flexibility from all sides; adapting intervention and implementation science to the needs of a large child welfare system, and thinking through alternatives collectively to select and plan for an intervention with the most promise. This manuscript describes the process used by the QIC-AG. This project used materials provided by the federal government that is publicly available to guide our work (see, for instance, A Framework to Design, Test, Spread, and Sustain Effective Practice in Child Welfare, (Framework Workgroup, 2014); the Permanency Innovations Initiative Training and Technical Assistance Project, (2016)). Agencies interested in similar projects can use these tools as a starting place.

The decision to offer TINT as a selective intervention (Heinrichs, Bertram, Kuschel, & Hahlweg, 2005; Offord, 2000; Springer & Phillips, 2006) to adoptive and guardianship families posed some unique challenges. First, targeting adoptive and guardianship families whose interactions with the child welfare system likely ended at the time of finalization, which could have been a decade earlier, made contact potentially difficult. Many public agencies do not maintain current contact information for families after an adoption or guardianship has been finalized. Second, this study targeted families who were not currently indicating distress, opting instead to provide a proactive intervention for a target population for whom the data suggested may be at increased risk for discontinuity due to the child’s experiences while in foster care and the youth’s current age. This selective intervention was designed to prepare parents to coach their teens through the normal developmental processes of adolescence that are likely to be complicated by the unique circumstances associated with involvement in the child welfare system and adoption or guardianship (see, for instance, Atkinson & Gonet, 2007; Child Welfare Information Gateway, 2012b; Egbert, 2015; Schwartz, Cody, Ayers-Lopez, McRoy, & Fong, 2014).

With these challenges noted, this project made contact with just over half (54%) of the families they attempted to reach. Despite efforts to reach families by mail, e-mail (when available), and phone, and the use of a locater service when existing contact information was exhausted, staff were unable to make contact with the remaining 46%. However, multivariate analysis did not find any systematic differences between those who the project was able to contact and those for whom contact was not achieved, after exhausting all available options. Of those who spoke with the project staff and said they were not interested in participating, 42% reported that they could not fit it into their schedule, and another 38% reported that they were not interested, or doing well and not in need of this intervention.

Recruitment study results to date found that approximately one-fifth (21%) of families registered to attend the intervention (TINT), 14% attended any sessions, and 12% attended at least four sessions, the minimum suggested by the purveyor to see an intervention effect. Not knowing in advance what proportion of families to expect to attend made planning for the TINT sessions difficult. Staff did not know how many participants to plan for and were discouraged by what they perceived as low uptake. Project staff instituted additional follow-up calls to remind potential participants of the TINT sessions before the second cohort. This call was deemed the “turkey sandwich call,” as registrants were asked to indicate what they would like for dinner in the hopes that this would create a social contract to follow-through on attendance or provide an opportunity for families to rethink their decision and inform staff that they would not be attending. Facilitators found that participants used this opportunity to withdraw and it reduced the no show rate on attendance.

Estimating the proportion of families who may participate in a prevention-focused intervention is difficult. Extant research in family-focused prevention interventions report low recruitment and attendance rates, and systematic reviews have found that most of these studies do not report enough information to accurately calculate population-based recruitment or attendance rates (Heinrichs, Bertram, Kuschel, & Hahlweg, 2005; Spoth & Redmond, 2000). A key long-term outcome of the QIC-AG project is a reduction in post-permanency discontinuity. In a recent longitudinal study examining post-adoption instability in New Jersey, 4% of children reentered state custody after an adoption finalization (Rolock, White, Ocasio, Zhang, MacKenzie, & Fong, 2018), and national estimates are between 1% and 10% (Child Welfare Information Gateway (2012a). The 14% TINT attendance rate is a higher percentage of families than the proportion who experienced discontinuity in the state. Results from a survey conducted in Illinois with adoptive parents and guardians found that 85% of families say they are doing well with the supports and services they currently had and did not need any additional supports or services (Fuller, et al., 2006). In addition, the National Survey of Adoptive Parents found that 51% of respondents reported that they do not receive parent-training services and do not want such a service. Another 38% reported receiving some sort of parent training, and 12% reported not receiving but would like to receive parent training (Malm, Vandivere, & McKlindon, 2011). Attendance rates for prevention efforts vary based on the intervention focus (e.g., child or parent-focused interventions), family belief systems, perceived program benefits, intervention demands (e.g., number of sessions, distance to travel) and are often moderated by family characteristics (e.g., the number of children in the house, socioeconomic status, educational level), among other things (see, for instance, Barrera et al., 2002; Charlebois et al., 2001; Heinrichs, Bertram, Kuschel, & Hahlweg, 2005; Spoth & Redmond, 2000). This is an ongoing project. The project team will continue to monitor participation and ensure participation is as easy for families as possible, with the goal of reaching the intended participants as effectively as possible. Results will be examined by subpopulations to understand which groups are most likely to benefit from the intervention. Among the advantages of prevention efforts is the idea that services are provided early, resulting in participants being able to better manage their issues. Prevention efforts that reach even a small proportion of families have the potential to have long-lasting effects. This study focuses on the process associated with the decision-making process and predevelopment stages of development of a research-informed intervention in child welfare. This is part of a comprehensive study of short and intermediate-term outcomes for families who involve longitudinal survey data, administrative data, and fidelity monitoring.

Acknowledgments

The authors would like to thank the New Jersey Department of Children and Families (DCF) for their research partnership, and the Office of Adoption Operations for their assistance with understanding the data provided.

Additional information

Funding

The QIC-AG is funded through a five-year cooperative agreement between the Children’s Bureau, Spaulding for Children, and its partners the University of North Carolina at Chapel Hill, the University of Texas at Austin and the University of Wisconsin-Milwaukee. Funded through the Department of Health and Human Services, Administration for Children and Families, Children's Bureau, Grant #90CO1122. The contents of this study does not necessarily reflect the views or policies of the funders, nor does mention of trade names, commercial products or organizations imply endorsement by the U.S. Department of Health and Human Services. This information is in the public domain. Readers are encouraged to copy and share it, but please credit the National Quality Improvement Center for Adoption and Guardianship Support and Preservation (QIC-AG).