Knowledge Translation Efforts in Child and Youth Mental Health: A Systematic Review

Abstract

The availability of knowledge translation strategies that have been empirically studied and proven useful is a critical prerequisite to narrowing the research-to-practice gap in child and youth mental health. Through this review the authors sought to determine the current state of scientific knowledge of the effectiveness of knowledge translation approaches in child and youth mental health by conducting a systematic review of the research evidence. The findings and quality of the 12 included studies are discussed. Future work of high methodological quality that explores a broader range of knowledge translation strategies and practitioners to which they are applied and that also attends to implementation process is recommended.

KEYWORDS:

• Knowledge translation
• children's mental health
• systemic review

BACKGROUND

The last decade has seen an ever-increasing interest in the implementation of evidence-based practices in child and youth mental health (CYMH) service delivery. These efforts have largely been driven by government policy makers and service leaders to improve client outcomes for which they are ultimately accountable. Ensuring the treatments and practices available are based on solid research evidence is viewed as the method to achieve positive outcomes for children and youth. In child and adolescent mental health services, the term “evidence based” is used to differentiate therapies—generally psychosocial—that have been studied with varying degrees of rigor from therapies used in usual care, that have not been evaluated (Hoagwood, Burns, Kiser, Ringeisen, & Schoenwald, 2001). Meta-analyses of evidence-based child psychotherapy intervention trials point to a consistent beneficial effect of treatment compared with no treatment (Kazdin, Bass, Ayers, & Rodgers, 1990; Weisz, Weiss, Han, Granger, & Morton, 1995). The knowledge base about effective treatments for certain mental health problems among children and youth is growing and improving (Haynes, 2010) and researchers and intervention developers are working to understand the ingredients of successful interventions (Sexton & Kelley, 2010). In addition to the evidence base, databases of evidence-based programs are available to providers and practitioners (e.g., Blueprints for Violence Prevention; SAMHSA's National Registry of Evidence-Based Programs and Practices), and practical assistance for ‘how to’ implement in real world settings is beginning to emerge from some research teams (e.g., National Implementation Research Network, Penn State EPIS Center; Evidence-Informed Practice Implementation Support Curriculum, Barwick et al., 2011). Most of the evidence-based treatments that have been established as effective in research settings are difficult to implement in real world contexts due to lack of practical implementation know-how and support, and consequently, many are not monitored to ensure their fidelity and effectiveness (Bickman & Hoagwood, 2010).

More recently, the CYMH sector has looked to the field of knowledge translation (KT) and practice-based implementation research (PBIR) in an attempt to gain knowledge of the most supported approaches for promoting practice change and implementation of evidence-based practices (EBPs) (Barwick et al., 2008; Boydell, Pong, Barwick, & Stasiulis, 2008; Barwick et al., 2005). This, in turn, has led to greater study of the implementation science process as it relates to mental health services for children and youth (Fixsen, Naoom, Blasé, Friedman, & Wallace, 2005).

Historically, the CYMH field has not provided clients with the most empirically supported treatments or practices (Huang, Hepburn, & Espiritu, 2003; Hoagwood & Olin, 2002). Indeed, much “practice as usual” in CYMH falls short of being identified conclusively as beneficial when the evidence supporting its use is subjected to scientific scrutiny (McLennan, Wathen, MacMillan, & Lavis, 2006; Henderson, MacKay, & Peterson–Badali, 2006; Weisz & Jensen, 1999; Jensen, Weersing, Hoagwood, & Goldman, 2005). The failure to consistently, expeditiously, or appropriately “translate” available scientific knowledge about behavioral healthcare so that it may be used by the relevant players in such care (e.g., service providers, systems of care, seekers or recipients of care, policy and decision makers) to improve outcomes collectively defines a serious problem within the domains of CYMH, school-based mental health, and healthcare in general.

Many KT studies have been conducted over the last two decades in other health-related disciplines such as medicine, HIV, and substance use, and there are several high quality reviews of reviews that synthesize what is known to be successful for a range of objectives, audiences, and outcomes (Bero et al., 1998; Grimshaw et al., 2001; Grimshaw et al., 2004; Grimshaw et al., 2006; Wensing, Wollersheim, & Grol, 2006; O'Connor et al., 2006; Estabrooks et al., 1999). The apparent absence of such a review within the context of CYMH, however, prompted us to conduct an initial scoping exercise. Our information scientist (JM) conducted searches in MEDLINE, PsycINFO, and the Cochrane EPOC Register in order to identify existing systematic reviews or overviews that had, at minimum, systematically and comprehensively searched for evidence concerning the effectiveness and efficiency of KT strategies within CYMH. No evidence syntheses with such a specific KT focus were found.

The aim of the authors in this review was to assess the current state of scientific knowledge regarding the effectiveness of KT strategies or interventions for CYMH. We use the CIHR definition of KT: a dynamic and iterative process that includes synthesis, dissemination, exchange, and ethically-sound application of knowledge to improve health, provide more effective health services and products, and strengthen the health care system (Canadian Institutes of Health Research, 2009). This process takes place within a complex system of interactions between researchers and knowledge users which may vary in intensity, complexity, and level of engagement depending on the nature of the research and the findings as well as the needs of the particular knowledge user.

METHODS

The current review was carried out within the context of a broader program of research on KT. We report here on the subset of studies found within that search that addressed KT in CYMH specifically. We describe and illustrate (Figure 1) the articles identified for our larger searches and only the evidence in CYMH is synthesized, as per the goal of the researchers in this study.

Search Strategy

A systematic search of the literature was performed to include all studies of any empirical research design that evaluated the effectiveness of KT interventions (e.g., reminders to change providers' behavior or increase adherence to guidelines, etc.) in CYMH. The comprehensive search strategy was designed to include the full range of relevant health and social science peer reviewed and grey literature.

Our search strategy included the following electronic databases: MEDLINE/PREMEDLINE (1966–March, 2009), Cochrane Central Register of Controlled Trials (CENTRAL), Cochrane Database of Systematic Reviews (CDSR) (Issue 1, 2009), CINAHL (1982–March, 2009), Embase (1980–2009, week 10), Cochrane EPOC Register (inception–March 2009) LILACS (index of Latin American and Caribbean health sciences literature; 1982–March 2009), ADOLEC (database of literature on adolescent health; inception–March 2009), PsycINFO (1980–March 2009), and Sociological Abstracts (1952–March, 2009).

FIGURE 1 PRISMA flow diagram—The data provided represent the flow of information through the phases of the systematic review.

The search in MEDLINE included the concepts of child psychology combined with the concepts of KT. The terms used in the concepts for child psychology included MeSH (Child Psychiatry, Adolescent Psychiatry, exp Mental Disorders, Mental Health, exp Mental Health Services, exp Community Mental Health Centers) with textword terms (child$ or adolescent or pediatric$ or paediatric$ or youth or teen$ or school age) and the age limit for child. The terms used in the concepts for knowledge translation included MeSH (Diffusion of Innovation, Evidence-Based Medicine) and textword terms ((implementation adj2 research) or (implementation adj2 practic$) or (evidence adj2 implementation) or [(evidence or knowledge or research) adj2 (implementat$ or translat$ or transfer$ or mobilization or exchange or utilization or diffusion or implement$ or disseminat$ or adoption or uptake) or (innovation adj2 (implementat$ or translat$ or transfer$ or mobilization or exchange or utilization or diffusion or implement$ or disseminat$ or adoption or uptake or uptake) or (translat$ adj2 (research or result$) or disseminat$)].%

Eligibility Criteria

Within CYMH we sought to include studies that evaluated the effectiveness of at least one KT intervention/strategy intended to improve any outcome(s) related to the organization, delivery, or receipt of CYMH care (e.g., practice change, guideline implementation). It was required that the KT intervention directly target any pertinent stakeholder(s) other than the actual care users (e.g., children or their caregivers); studies directly targeting care users were excluded. Inclusion criteria for the search defined studies involving children and youth from 0 to 24 years (24 years being the upper limit of the World Health Organization's definition of ‘youth’), English language studies of any research design and any year. Studies with main outcomes other than behavioral or organizational (i.e., changes in knowledge, attitudes) were excluded. It was decided that studies that employed simulated assessments to measure outcomes would be included, given their potential to contribute important information to the review.

Data Management

References from the search were imported into a Reference Manager™ (The Thomson Corporation, 2012) database and tagged to indicate the source database. A de-duplication was performed after each set of results was imported, followed by all references being uploaded into the Systematic Review System (SRS) software to manage and record screening information. The final set included 6,494 citations.

Selection Process

Each application of eligibility criteria involved a calibration exercise and a form was developed and tested especially for this review. Using SRS software from TrialStat™ (Ottawa, ON, Canada, two reviewers independently assessed the eligibility of each bibliographic record. Studies passing this first screening were retrieved and independently evaluated by two reviewers (see Table 1 for screening questions). All conflicts were resolved via a discussion between the two reviewers, or if necessary, third party intervention.

Data Abstraction

Data were abstracted independently by two reviewers (LB and NR) and input into qualitative tables developed for the review. Data included results, evaluation strategy (e.g., research design), key characteristics of the KT intervention (e.g., objective, components, media, timing), participant characteristics, treatment fidelity, implementation strategy, and risk of bias. The Cochrane Effective Practice and Organisation of Care (EPOC) Review Data Collection Checklist (2002) was used to assess the risk of bias in RCT/CBA/ITS study designs. It should be noted that EPOC review system does not review studies that have subjective primary outcome measures and only accepts objective measures. We included such studies as per our inclusion criteria and scored relevant items as Not Applicable (N/A) when the outcome measure was subjective. Risk of bias was assessed for all other quasi-experimental study designs using the relevant items from Downs & Black (1998) checklist. Study quality was determined by the number of criteria met on the respective assessment scales. When a study was described by more than one report, data were abstracted from all documents.

RESULTS

Results of the Search

A PRISMA flow diagram illustrates the number of records at each phase of the review (see Figure 1). Of 6,494 records entered into initial relevance screening, 5,845 were excluded. The remaining 649 records were successfully retrieved and were subjected to full-text screening if the focus was KT in CYMH. This assessment excluded 561 citations, leaving 88 citations for the data abstraction phase. An additional 55 reports were excluded because they were not in the realm of CYMH. A further 21 studies in CYMH were excluded for one of three reasons: (a) they did not assess the effectiveness or efficiency of a KT strategy or intervention (n = 11); (b) they assessed main outcomes other than behavioral or organizational (n = 7); or (c) the stakeholders were the actual care users (i.e., youth or their caregivers; n = 3). Overall, 12 relevant reports were identified, five from community-based CYMH (Barwick, Peters, & Boydell, 2009; Henggeler, Shedow, Cunningham, Donohue, & Ford, 2008; Liddle et al., 2006; Homer et al., 2004; Tucker, Derscheid, Odegarden, & Olson, 2008) and seven from school-based CYMH (Atkins et al., 2008; Lerman, Vorndran, Addison, & Kuhn, 2004; Moore et al., 2002; Rohrback, Graham, & Hansen, 1993; Scott et al., 2005; Wallace, Doney, Mintz–Resudek, & Tarbox, 2004; Webster–Stratton, Reid, & Hammond, 2001) (see Table 2: Key study characteristics).

TABLE 1 Review Screening Questions

Level	Question
	Does this bibliographic record (i.e., citation, key words, abstract) appear to describe at least one of the following:
1	A primary study, which investigates the effectiveness or efficiency of at least one KT intervention/strategy that targets directly any pertinent stakeholder(s) other than actual or potential (“public-at-large”) care users (or their carers, in the case of children and youth) in order to improve any outcome(s) related to the organization, delivery, or receipt of child and youth mental health (CYMH) care;A systematic evidence synthesis [e.g., systematic review (SR), health technology assessment (HTA), or systematic overview (SO) of systematic reviews or HTAs], which investigates the effectiveness or efficiency of at least one KT intervention/strategy that targets any pertinent stakeholder(s) other than actual or potential (“public-at-large”) care users (or their carers, in the case of children and youth) in order to improve any outcome(s) related to the organization, delivery, or receipt of non-CYMH care;A primary study published or disseminated after 1997, which investigates the effectiveness or efficiency of at least one KT intervention/strategy that targets any pertinent stakeholder(s) other than actual or potential (“public-at-large”) care users (or their carers, in the case of children and youth) in order to improve any outcome(s) related to the organization, delivery, or receipt of non-CYMH care?
2	A primary research study, which employs any research design to evaluate the effectiveness or efficiency of at least one KT intervention to [assure or] improve the organization or delivery of any type(s) of CYMH care for individuals under the age of 24 years (e.g., from primary prevention to long-term care for mental health difficulties or disorders, which include the addictions); OR a SR, HTA, or SO of existing SOs or SRs/HTAs of primary research evidence from studies having employed any research design to evaluate the effectiveness or efficiency of at least one KT intervention to [assure or] improve the organization or delivery of any type(s) of non-CYMH care (i.e., adult mental or non-mental healthcare, or pediatric non-mental healthcare); OR a primary study, whose KT-related effectiveness or efficiency results from at least one attempt [assure or] improve the quality of non-CYMH care were disseminated after 1997.
2	KT intervention(s) of any type (i.e., professional practice/behavior-related organizational, financial, or regulatory) or complexity (i.e., from single focus, passive dissemination-mobilization to multiple focus, active implementation-adoption-uptake interventions/strategies.
2	At least one effort was made to directly influence any outcome relating to the (elimination of behavior-, organization-, finances- or regulation-related barriers to the) implementation-adoption of effective or efficient delivery of care (e.g., adherence to or utilization of recommended, gold standard healthcare) or its similarly optimal organization, financing or regulation (e.g., changes in access, organizational structure or climate-readiness-resistance)a) The study may also or instead assess the impact on pertinent health outcomes.b) Excluded are main outcomes that reflect possible changes in knowledge, attitudes, stereotypes, intentions or levels of satisfaction; these may be abstracted only if they are evaluated as possible mediators in studies that employed review-relevant main (e.g., behavioral, organizational) outcomes.
2	At least one effort was made to directly influence any stakeholder group-organization other than actual or potential (“public-at-large”) care users (or their carers, in the case of children and youth) (e.g., service provider individuals, teams-group-units or organizations; policy-makers; healthcare systems).Consequently, excluded are interventions, which employ “patient decision aids,” public health promotion or education strategies such as mass media campaigns (unless directed at service providers or their organizations/systems), parent/guardian training or school-based education.
2	This report describes a primary study that investigated the effectiveness or efficiency of at least one KT intervention that was intended to [assure or] improve non-CYMH care.

Study Characteristics

Within community-based child youth mental health (CYMH), two studies came from the addictions field, two from CYMH in healthcare, and one from community-based CYMH more broadly. Study designs included one cluster randomized controlled trial (RCT), one RCT, one interrupted Time Series (ITS), one controlled before and after (CBA), and one study was quasi-experimental in nature. Within school-based CYMH, six studies stemmed from the field of school-based mental health more broadly and one from early childhood education, where teachers were not the main focus of the study. Study designs in school-based CYMH included two cluster RCTs, one RCT, and four quasi-experimental designs. Upon further investigation, four of the seven included studies in school-based CYMH used simulated assessments to measure outcomes. All studies but one were published in the United States and all but one were published between the years 2001 and 2009.

Description of the Studies

Table 2 describes important features of the 12 included studies. Practitioner/teacher training, in the form of workshops, meetings, or conferences, was the main KT strategy in all but four studies; one in community-based CYMH that evaluated the effectiveness of an intensive quality assurance intervention following workshop training; one in community-based CYMH that evaluated the effectiveness of communities of practice; one in school-based CYMH that investigated the use of trained facilitators; and another in school-based CYMH that investigated the influence of key opinion leaders. All of the strategies were interpersonal in nature and often included one or more additional components; for example, didactic presentation, group discussion, role-play, audio/video, handouts/printed material. Training approaches in both sectors varied in length from three hours to a series of learning events over several years. In community-based CYMH we see some examples of training extending over a longer period of time with the goal of fostering sustainability. Each of the included studies is described below.

TABLE 2 PICO Table for Included Studies

Setting	Study	Design	Participants	Intervention	Comparison	Outcome(s)	Quality rating*
CYMH	Barwick, 2009	Cluster RCT	CYMH practitioners	Communities of Practice	Practice as usual	Practice change; use of outcome measurement tool; use of implementation supports	+
CYMH	Henggeler, 2008	RCT	Therapists	Workshop + intensive quality assurance	Workshop only	Therapist use of contingency management cognitive-behavioral and monitoring techniques	N
CYMH	Homer, 2004	CBA	Physicians and nurse practitioners	Conference, toolkit	No intervention	Consistency with 10 specific recommendations from AAPG concerning ADHD	N
CYMH	Liddle, 2006	ITS	Adolescent drug treatment program staff: Social workers, mental health technicians, registered nurse, program director, medical director	Technology transfer approach including training and supervision	None	Number of weekly therapy sessions; extra-familial contact; in-session MDFT content; characteristics of program environment	−
CYMH	Tucker, 2008	Quasi-experimental two-group non-randomized pretest posttest design	Registered nurses	Workshop, distribution of educational materials	No workshop	Use of child behavior management strategies & skills	−
Education	Atkins, 2008	Cluster RCT	KOL teachers, MHPs, classroom teachers	Key opinion leaders + MHP consultation	MHP consultation only	Use of ADHD assessment & intervention strategies	−
Education	Lerman, 2004	Quasi experimental	Special education teachers, 1 teacher-in-training	Workshop, distribution of educational materials	None	Proportion of preference assessment, direct teaching, and incidental teaching skills performed correctly	+
Education	Moore, 2002	Quasi experimental	Elementary school teachers	Training	None	Percentage of correct teacher responses during simulated functional analysis	+
Education	Rohrback, 1993	Cluster RCT	Elementary school teachers, principals	Teacher training and principal intervention (2 × 2 design)	Intensive vs. Brief teacher training + principal intervention vs. No principal intervention	Quantity, integrity, & maintenance of substance abuse program implementation	−
Education	Scott, 2005	Quasi experimental	Certified school staff (description of staff is not offered)	Facilitator training	Experts	Frequency of selected strategies during simulated functional behavior assessment	+
Education	Wallace, 2004	Quasi experimental	Teachers and a school psychologist	Workshop	None	Percentage of correct responses during simulated functional analysis	+
Education	Webster–Stratton, 2001	RCT	Head Start teachers and teacher assistants	Teacher training	Regular Head Start program	Classroom management skills (criticism, praise, classroom atmosphere, harsh discipline, positive techniques)	N
Note. This table provides population, intervention, comparison, outcome, and quality data for each included study.*+: Study meets >50% of quality criteria.−: Study meets <50% of quality criteria.N: Study meets 50% of quality criteria.

Interventions in Community-Based CYMH

Three of five studies in community-based CYMH included training as the main KT strategy; one examined communities of practice, and another evaluated an intensive quality assurance intervention.

Homer et al. (2004) evaluated whether attending an educational conference on attention deficit hyperactivity disorder (ADHD), preceded by physician and nurse practitioners' self-assessment of their ADHD diagnosis and management practice, would result in care improvements. The conference was a face-to-face 1.5-day meeting; the comparison group was a convenience sample of professionals who did not attend the conference. Conference participants also received an ADHD toolkit to assist in evidence-based diagnosis and treatment of children with ADHD. The intervention (n = 18) and comparison groups (n = 15) differed only in that the latter included practices with a much larger proportion of visits with public insurance. The main outcomes were key measurable processes of the American Academy of Pediatrics diagnosis and treatment guidelines (e.g., source of information for making ADHD diagnosis, use of DSM-IV criteria in diagnosis, evaluation for coexisting conditions, offer of options for treatment, provision of ADHD management plan, set goals with family, plan for follow-up) obtained via chart abstraction by a subset of clinicians in both groups (191 charts pre-intervention; 153 charts post-intervention); the degree of inter-rater reliability and whether the assessment was blind was unclear. Results indicated gaps in quality of care pre-intervention with one statistically significant between group difference: intervention clinicians were less likely to evaluate for coexisting conditions than those in the comparison group (p = 0.02). For the most part, care improved in both groups over time, although it improved in more areas for the intervention group. Post-intervention between-group differences were significant for only two practices: evaluation for coexisting conditions (p = 0.02) and offering treatment options (p < 0.001).

Liddle et al. (2006) employed an ITS design to evaluate a multi-component, multi-level technology transfer intervention in which adolescent drug treatment program staff (n = 10) were trained to implement an evidence-based practice. It is important to note that for ITS studies, EPOC requires three data points both before and after the intervention for inclusion; this study had only three data points in total. EBP training took place over a six month period and was didactic in nature and role specific (i.e., social workers received 30 hours of training, mental health technicians and the nurse received 15 hours, and teachers received 11 hours). Participants worked through practice manuals, practice videotapes, and applied their learning to ongoing cases. In addition to training, social workers attended 10 hours of individual supervision and 15 hours of supervision with their clients and the trainer. Trainers met weekly with the program and medical directors to discuss training issues. Following training, five 1-hour implementation meetings were held with all staff to discuss implementation issues. A 14-month implementation phase followed training to facilitate practice change through regular use of the EBP model, regular supervision, and booster training. Weekly 1-hour whole staff implementation and clinical meetings were held throughout the implementation phase to troubleshoot issues and plan for the durability phase. Outcomes of interest included the average number of weekly therapy sessions, weekly contacts with the Department of Juvenile Justice and school personnel across study phases, in-session EBP-specific content (all measured through case records/notes and/or observation), and adolescent ratings of the program environment. It is unclear if there was blinded assessment of outcomes or completeness of the data set. A subset of cases were randomly selected to assess EBP fidelity via observation. Results indicated that therapists conducted more weekly sessions and extra-familial contacts in the implementation and durability phases than at baseline (p < 0.001 for all). Therapists increased their number of weekly individual sessions from implementation to durability (p = 0.20) and covered more EBP-specific themes in implementation and durability than in baseline (p = 0.014). Youth reported reliable differences in four of five program environment dimensions across study phases. There was evidence of sustainability for some outcomes.

In a pilot study, Tucker et al. (2008) evaluated the effects of a 3-day standardized parent/teacher training program aimed at improving interactions with children and adapted to meet the needs of registered child and adolescent psychiatric nurses (n = 27). Nurses in Cohort 1 (n = 12) were assigned to the intervention immediately and those in Cohort 2 (n = 15) were assigned to wait list control group and scheduled for training one year later; assignment was based on availability and logistics. Nurses learned the principles and skills of the program through highly active participation including watching videotape vignettes, actively engaging in role plays, and participating in ongoing dialogue with the trainer; multiple reading assignments were also given. Primary outcomes of interest for the purposes of this review included participants' self-reported ratings of child management strategies and observed child management skills. Results are provided for both cohorts at baseline and for Cohort 1 post-training. Losses to follow-up were not reported. At baseline, Cohort 1 participants reported greater frequency (p = 0.007) and usefulness (p = 0.047) of inappropriate strategies than did Cohort 2. From baseline to post-training, Cohort 1 increased in self-reported frequency of use of praise and incentives (p = 0.036) and frequency in working with parents (p = 0.01). No differences in observational data were observed between cohorts at baseline. For Cohort 1, three of five observation categories changed significantly: positive verbal statements increased (p = 0.016), negative verbal statements declined (p = 0.028), and use of commands during the play session declined (p = 0.013).

Barwick, Peters, and Boydell (2009) examined whether CYMH practitioners assigned to a community of practice (n = 18) would adopt a mandated standardized outcome measure more readily than those who engaged in practice as usual (PaU; n = 19). The method of randomization for this RCT was not clear. Communities of practice were structured around key principles for sustainability and success; they met six times over an 11-month period; and were hosted and facilitated by a trainer for the outcome measure. Outcomes relevant to this review included self-reported practice change and use of available implementation supports, as well as objective measurement of use of the outcome measure within each CYMH organization. Results indicated no significant between-group differences in self-reported practice change or use of implementation supports, although community of practice participants did engage in greater use of the outcome measurement tool in practice.

Following workshop training, Henggeler et al. (2008) tested the use of an intensive quality assurance (IQA) protocol to support practitioner fidelity to contingency management practice (n = 18) compared with a workshop-only (WSO) condition (n = 12). CYMH therapists were randomized to IQA or WSO conditions following the workshop (teams with the same supervisor were randomized together to avoid contamination), although the randomization procedure and whether the outcomes were blindly assessed was unclear. Following a five month baseline period, practitioners received training materials and attended an active 2-day workshop (including a detailed description of practice components, expert demonstration, small group exercises, role-play, and feedback and reinforcement) to learn contingency management practice. During the 4-month post-workshop period and 6-month sustainability period, IQA participants received ongoing training and supervision, while WSO participants could access consultation on request. The primary outcome was therapist implementation of practice procedures (cognitive-behavioral and monitoring techniques) assessed by caregiver and youth reports. Based on youth reports, IQA (p = 0.01) practitioners increased their implementation of cognitive-behavioral techniques from baseline through 4-months post-workshop, while the WSO group showed no change; there was evidence of sustainability. Caregiver reports suggested that IQA practitioners initially increased their use of cognitive behavioral techniques but returned to just above baseline levels by 4-months post-workshop and that WSO practitioners exhibited a slight decrease but returned close to baseline by 4-months post-workshop. There were no observed differences in practitioner use of monitoring techniques.

Interventions in School-Based CYMH

Five of seven studies in school-based CYMH included training as the main KT strategy; one study investigated the influence of key opinion leaders and another evaluated the use of trained facilitators.

Lerman et al. (2004) evaluated a workshop to train teachers (n = 4 public school teachers, n = 1 Masters of Education student) on an EBP for children with autism. The workshop took place 3.5 hours a day over five consecutive days. Information and skills were taught via lectures, handouts, and role-play sessions with feedback. The main outcome was the proportion of skill components (as determined from the literature) performed correctly by teachers in simulated assessment (initial workshop) or with a child (follow-up and generalization). Results indicated that teachers' proportion of correctly performed skills at baseline was no greater than 65%. All teachers subsequently mastered (met or exceeded the accuracy criterion) the skills through role play and implemented them with ≥80% accuracy.

Moore et al. (2002) evaluated the effectiveness of training teachers (n = 3) to use functional analysis (a behavioral intervention) methods. The primary outcome was the percentage of correct teacher responses which was assessed through videotape observation. In Phase 1 of the training, teachers were provided with written and verbal information pertaining to functional analysis, their knowledge was tested, and they implemented functional analysis conditions without feedback on their performance; this phase lasted one day. In Phase 2 of the training, teacher performance data from Phase 1 was reviewed, experimenters modeled the correct behavior during role-play with a student, and teachers practiced while receiving direct feedback. Subsequently, teachers received feedback on their functional analysis skills in their own classrooms. Results showed that the percentage of correct teacher responses in Phase 1 was generally at or below 60%; during Phase 2, means for all teachers exceeded 95%. The authors noted that all teachers implemented the procedures with integrity in their classrooms.

Rohrback et al. (1993) used a 2 × 2 randomized control design to evaluate the effectiveness of intensive teacher training and principal intervention on teachers' adoption of a substance abuse prevention curriculum. Participants were elementary school teachers (n = 60) and principals (n = 25) from four school districts and schools were randomly assigned to either an intensive (full-day workshop including presentation of theory, modeling of skills, and guided practice) or a brief (2-hour meeting in which curriculum lessons were described) teacher training condition. Within each district, schools were randomly assigned to either the principal-intervention (one-to-one meeting with each principal that included persuasive communication and an explicit request to encourage and monitor curriculum implementation in their schools) or no-principal-intervention condition. The method of randomization was not described. Relevant primary outcomes included the effect of teacher training and principal intervention on the quantity (percentage) of program delivery observed and the effect of teacher training on the integrity of implementation as assessed by blind observers in the classroom. Within one school year, 78% of all teachers had implemented one or more of the program lessons (21% implemented all lessons). There was no effect of teacher training on the quantity of initial implementation but the principal intervention was found to increase the quantity of initial implementation (p < 0.05). Intensive teacher training was not found to affect implementation integrity. Only 25% of teachers maintained implementation into the second year.

Wallace et al. (2004) used a workshop format to train two teachers and a school psychologist to implement functional analysis methods. Simulated analyses were conducted at baseline. Participants then attended a 3-hour workshop which consisted of a description of functional analysis, videotaped demonstrations, and role playing. Following the workshop, simulated assessments identical to those in the baseline condition were conducted. Verbal feedback was provided to participants immediately after the session if they failed to achieve 90% fidelity; simulated sessions were conducted again, after feedback. A single in-classroom functional analysis session was observed for one participant. The outcome of interest was the percentage of correct teacher responses during simulated assessment. Results indicated that during baseline, no participant scored above 50% correct responses. After the workshop, two of three participants scored between 96% and 100% in all conditions, while one participant failed to meet criterion for one of the three conditions; this was rectified after the participant was given feedback. The in-classroom session was conducted with 100% accuracy.

Webster–Stratton et al. (2001) investigated the impact of training on the classroom management skills of teachers and teachers assistants (n = 37). This was part of a larger study focusing on parents and children and, consequently, very little information pertaining to the training strategy and outcomes was provided. Thirty-six classes at 14 Head Start (school-readiness program) centers were randomly assigned to intervention and control groups in a 2:1 ratio. The intervention took the form of six monthly 1-day workshops (36 hours of training) that included watching videotapes of other classroom teachers and group discussion of teacher-student interactions. The main outcomes were teacher classroom management style and classroom atmosphere assessed by observation. Following the intervention, teachers in the experimental condition showed better classroom management than control teachers (p < 0.01).

Atkins et al. (2008) examined the relative and combined influence of peer-identified key opinion leader teachers (KOL; n = 12) and mental health providers (MHP; n = 21) on classroom teachers' self-reported uptake of classroom practices for children with ADHD. A subset of low-income urban schools were assigned to the KOL (KOL + MHP; n = 6) or comparison (MHP only; n = 4) conditions. Sixty-one teachers participated in the KOL group, while 54 teachers were in the comparison group. Key opinion leaders were nominated by 70% or more of their teacher peers and were found to have significantly more years of teaching experience (p < 0.02). Mental health professionals were recruited from local mental health agencies. Both KOLs and MHPs were trained in ADHD best practices via a web-based course. First, immediately following training, KOLs and MHPs began dissemination activities with classroom teachers. Second, at the beginning of the following school year, workshops were held with teachers in both conditions to give everyone exposure to ADHD best practice strategies (through distribution of manuals, KOL endorsement of strategies, and MHP promotion of strategies) and provide a stronger test of the KOL influence. Via interview at five time points over two years, teachers in each condition self-reported their use of the strategies in the preceding month and how often MHPs and KOLs (intervention group only) had provided direct support for their classroom interventions in the preceding month. All teachers provided data for at least one time point. Teachers in the KOL condition reported greater use of the strategies at the beginning of the study than did teachers in the comparison group (p < 0.01) and there was a trend toward this being sustained over time. KOL influence accounted for the differences in reported use (p < 0.001) between groups; the influence of MHP support was non-significant.

Scott et al. (2005) investigated the degree to which school-based teams that included trained facilitators (n = 5) were able to use the outcomes of functional behavior analysis (FBA) to develop intervention strategies in comparison with FBA experts (n = 3). Staff from four schools were provided six hours of training to act as facilitators for school-based intervention teams. Training was partly didactic but also included guided practice with feedback, small-group practice, and collaboration. At least one trained facilitator sat on each team, although no two teams were composed of the same people, as teams were composed of people most familiar with each student. A total of 31 behavior plans were generated for particular students over the course of the study. Experts in the comparison group were provided with a written description of each student's case and the hypothesis of function used by the team and asked to develop a plan. The main outcomes were the number and type of intervention strategies selected. Results demonstrated that experts selected more instructional strategies (p < 0.001), fewer negative consequences (p < 0.001), and far fewer exclusionary strategies (p < 0.001) than did teams.

Overview of Results/Study Quality

Overall study quality ranged from poor to fair for RCT, ITS, and CBA studies (see Table 3: Quality assessment by study design). Four of seven studies in this group met or exceeded 50% of the EPOC criteria for study quality; the remaining three studies met fewer than 50% of the EPOC criteria. These studies failed to meet EPOC guidelines and suffered from poor documentation of procedures and methods which resulted in a quality review that yielded many ‘unclear’ judgments. There were no differences in quality between community-based CYMH and school-based CYMH sectors for these types of studies. The quasi-experimental studies, however, fared better with respect to quality with four of five exceeding 50% of the quality criteria. The studies of higher quality, however, had extremely small sample sizes (most with fewer than five participants) and used simulated assessments to evaluate research outcomes. Additional weaknesses among the quasi-experimental studies included lack of blinding in outcome assessment, lack of clarity around adjusting for confounding variables; time intervals between intervention and outcome; time period for recruitment; and reporting loss to follow-up. As there was but one community based CYMH study in this group, we cannot comment on sector differences.

At least one attempt to objectively measure behavior change was made in all but one study in school-based CYMH that relied on teachers' self-reported behavior change. It is important to note several additional points relevant to study quality: (a) the ITS study had only three total time points (baseline, implementation, durability), whereas EPOC criteria require a minimum of three points both before and after intervention for inclusion; (b) none of the quasi-experimental studies were randomized or declared by the authors as blinded; (c) within school-based CYMH, four of the seven studies had five participants or fewer.

DISCUSSION

We systematically reviewed the empirical KT literature in the field of CYMH up to March 2009. The review of KT practices in CYMH yielded only 12 studies, most of which were focused on practitioner/teacher training for behavior change. The majority of studies were published in the United States between 2001 and 2009. This review is unique because it is the first review of KT studies in this field and included community-based and school-based mental health.

The community- and school-based sectors did not appear to differ with respect to methodological rigor; however, simulated assessments were common among the school-based CYMH studies. Both sectors included studies possessing methodological weaknesses, mainly as a function of poor reporting and small sample size in many of the school-based CYMH studies. Four of the five quasi-experimental studies were conducted within school-based CYMH; the quasi-experimental studies were of higher quality than the RCT/ITS/CBA studies, although this was likely due, in part, to the less stringent criteria of the Downs and Black (1998) quality measure. These higher quality studies were largely simulated assessments with extremely small sample sizes, limiting their external validity. No conclusion can be drawn regarding which sector is doing the best methodologically based on the limited number of studies (and their variability) included in this review. In the community-based CYMH studies, KT efforts targeted a range of professionals including registered nurses, physicians, substance abuse therapists, social workers, and child youth workers. In school-based CYMH KT studies, the focus was on teachers and some administrative staff (principals). As such, the type of practitioner studied in CYMH KT research has been varied to date, and this is important with respect to external validity, or the extent to which the results of individual studies are generalizable across professions. Methodological considerations for future research include examining the effectiveness KT strategies among varied professions, conducting assessments in real-world contexts with larger sample sizes, and ensuring sound reporting of methods and results.

TABLE 3 Quality Assessment by Study Design

Quality assessment of RCTs
RCT	Concealment of allocation?	Follow-up of professionals?	Blinded assessment of primary outcome?	Baseline measurement?	Reliable primary outcome measures?	Protection against contamination?
Henggeler, 2008	NC	NC	NC	✓	✓	✓
Atkins, 2008	NC	NC	n/a	X	n/a	✓
Webster–Stratton, 2001	✓	NC	✓	NC	X	✓
Barwick, 2009	NC	X	✓	✓	✓	✓
Rohrbach, 1993	NC	X	n/a	X	n/a	✓
✓: Done X : Not DoneNC: Not clear
Quality assessment of quasi-experimental studies
Quasi-Experimental	Was an attempt made to blind those measuring the intervention?	Do the analyses adjust for different lengths of follow-up of participants?	Were the statistical tests used to assess the main outcomes appropriate?	Was compliance with the intervention(s) reliable?	Were the main outcome measures used accurate (valid and reliable)?	Were the participants in different intervention groups recruited from the same population?	Were study participants in different intervention groups recruited over the same period of time?	Was there adequate adjustment for confounding in the analyses from which the main findings were drawn?	Were losses of participants to follow-up taken into account?
Tucker, 2008	X	X	✓	✓	✓	✓	X	NC	NC
Lerman, 2004	X	✓	✓	✓	✓	✓	✓	NC	✓
Scott, 2005	X	✓	✓	✓	✓	✓	✓	X	X
Moore, 2002	X	NC	✓	✓	✓	✓	NC	NC	✓
Wallace, 2004	X	NC	✓	✓	✓	X	✓	NC	✓
✓: Don X : Not DoneNC: Not clear
Quality assessment of ITS
ITS	Protection against secular changes?	Data were analyzed appropriately?	Reason for number of points pre- and post-intervention given?	Shape of the intervention effect was pre-specified?	Protection again detection bias?	Blinded assessment of primary outcome?	Completeness of data set?	Reliable primary outcome measures?
Liddle, 2006	NC	✓	X	X	✓	NC	NC	✓
✓: Done X : Not DoneNC: Not clear
Quality assessment of CBA
CBA	Baseline measurement?	Characteristics for studies using second site as control?	Blinded assessment of primary outcomes?	Protection against contamination?	Reliable primary outcome measures?	Follow-up of professionals?
Homer, C. J. 2004	✓	✓	NC	✓	NC	NC
✓: Done X : Not DoneNC: Not clearNote. The data provided represent the quality assessment of included studies grouped by study design (RCT, ITS, CBA, and quasi-experimental).

This review included only those studies that reported behavioral outcomes. Studies that looked only at change in attitudes or knowledge were excluded, as change in attitudes and knowledge do not necessarily lead to behavior change, which is the desired outcome when training people to learn a new skill or adopt a new practice. Furthermore, changes in attitude and knowledge are inherently more difficult to measure, as they must be inferred and are at best, self-reported constructs. Knowledge is a difficult construct to capture reliably and validly (Schrader & Lawless, 2004) and the multidimensionality of attitudes is also problematic (Seeman, 1993). Included studies employed various methodologies to measure behavioral outcomes and were considered more methodologically rigorous if they reported on observed behavior change (as was the case in all but one study reviewed) rather than self-reported behavior change. That said, we recognize that behavioral observation is extremely labor intensive (videotaping, coding, etc.) and costly, and therefore, not always feasible. Self-report measures are inherently problematic as they are often tainted by social desirability, attitudes, cognitive processes, mood, and personality (Spector, 1994; Donaldson & Grant–Vallone, 2002). We were not able to comment on the reliability of primary outcome measures where self-report was the sole method used or where it was used in combination with observational measures, as EPOC does not review such studies. Future work in this area should aim to capture behavior change objectively where feasible.

Where the KT strategy is practitioner training, as was the case in 10 of 12 included studies, emphasis needs to be paid to principles of adult education in recognition that short, quick doses of training do not typically translate to behavior change in practice (Joyce & Showers, 2002). Such brief doses were evident in school-based CYMH, whereas in community-based CYMH, training took place over a longer period of time and some attention was paid to sustainability. A noted strength was that studies that used practitioner training relied on relatively interactive training strategies such as role playing, video review, and group discussion. These strategies are supported by work from the field of andragogy (the methods or techniques related to teaching adults), which recommends that adults need to focus more on learning process and less on the learning content, and active strategies such as role playing, simulations, and self-evaluation are most useful (Kearsley, 1996). Additional research, including our own, has found that practitioners desire learning outcomes that can be put to immediate use, in concrete, practical, and self-benefiting terms (O'Connor, Bronner, & Delaney, 2002; Barwick, Bennet, Johnson & Chaban, in preparation; Barwcik, Johnson, & Bennett, in preparation). Training for practice change must begin to incorporate additional essential components of adult learning theories. The emphasis on training found in this review, however, underscores the paucity of studies pertaining to other KT strategies (e.g., reminders, opinion leaders/champions, mass media campaign, performance feedback) in this field, highlighting an additional direction for future investigation. Another key direction for future research is greater attention to the implementation processes that, together with KT strategies—such as practitioner training—are needed to achieve behavior change (Damschroder et al., 2009; Fixsen et al., 2005).

Published commentary, descriptive and qualitative work has addressed implementation issues in the CYMH literature for the last decade. However, the small number of published studies measuring behavioral outcomes, and the recency of their publication, suggest that within CYMH, KT and implementation science are still in their infancy. As we look to the future of scaling up and implementation research in CYMH, new KT research needs to take into account the limitations of early KT and diffusion research, as set forth by Greenhalgh and colleagues (Greenhalgh et al., 2005). Firstly, models of diffusion can suggest hypotheses about the reasons for adoption (or non-adoption or rejection) but they do not provide an explanation of either how or why people adopt particular innovations at particular rates, or whether efforts to influence adoption will be met with success. Most importantly, we believe, the types of KT strategies and diffusion approaches studied in the early research were somewhat fixed and static, where investigators selected an innovation, selected a method of transfer, applied it, and stood back with little consideration to the process of implementation required for practice change. We now know that innovations often require modification as they are scaled up or applied, and this process of modification merits study in its own right. We also know that practice change requires both strategy and implementation process, without which success is severely hampered (Fixsen et al., 2005).

We have a growing understanding of the complexities inherent in practice change and innovation scale up. Early research made little concession to the influence of context, including the historical and socio-cultural contexts that facilitated early diffusion research. We are now more aware that what is adopted and effectively used at Site A, for instance, may not be readily adopted and effective at Site B, and that organization and individual readiness for change and related contextual factors play an important role (Cummings, Hutchinson, Scott, Norton, & Estabrooks, 2010). A new wave of practice-based implementation research is already underway. In Ontario, Canada, we have studied the change process as Ontario's largest CYMH service organization shifts from practice as usual to evidence-based and evidence-informed practices (Barwick, Kimber, & Fearing, in press; Brown, Rounthwaite, & Barwick, in press).

LIMITATIONS

Certain limitations of this review are acknowledged. As a result of being part of a larger program of research, the 7,626 papers identified in the first instance of our electronic search represents more than what might have been found for this field alone. Also, our search strategy included only English language publications and we did not include substance abuse as a search term, although studies from the addictions field did emerge from our searches. Others may wish to take this direction for future reviews. Our ability to draw substantive conclusions from the results was hindered by the small number of included studies, their variability, and poor reporting, all of which hampered the validity of findings.

CONCLUSION

All 12 studies reported expected behavior change but it must be acknowledged that there are very few published empirical studies of KT efforts in the field of CYMH prior to March 2009. Study quality was largely poor as a result of insufficient or unclear reporting and small sample size. Future studies in KT need to focus on observed rather than self-reported behavioral change, where feasible, in real world contexts versus simulated contexts and the sustainability of behavioral changes over time. They should also study a broader range of KT strategies, applied to a range of practitioners, and where training is involved, employ methods supported by evidence-based adult education. Studies must also attempt to capture the complexity of practice change and implementation process issues in addition to implementation outcomes. We recognize that the work reviewed represents the beginning of our journey to understand how best to support practice change in CYMH. This is essential if we are to ensure that evidence-based services are on offer for children and youth. Subsequent reviews of this work will likely produce a larger number of studies in KT for CYMH. Research in this area should heed the quality deficits discussed herein and look to KT studies and strategies in other fields in order to move the field forward.

Notes

The authors declare that they have no competing interests.

HS was the primary author of the grant proposal to CIHR. MB assumed the primary investigator role half way through grant completion. JM conducted the database searches. MB, HS, KB, DB, JM, ML, and AW participated in level one and level two abstractions. Level 3 abstractions were conducted by LB and NR and overseen by MB and JM. ML developed the PRISMA table and drafted the Methods section with JM. MB prepared the draft of this manuscript and assumed responsibility for the final article. All authors read, made comments and edits, and provided approval of the final manuscript.

The authors wish to acknowledge our community partners: the Learning and Research Institutes, Hospital for Sick Children; the Ontario Ministry of Children and Youth Services; Children's Mental Health Ontario; and the Provincial Centre of Excellence for Child and Youth Mental Health at CHEO. We would like to acknowledge Andrew Lumb for his review work in the early stages of this grant and Nathalie Richard for her work on data abstraction. In addition, we would like to acknowledge the Canadian Institutes for Health Research (MOP-84198) who funded this review.