Introduction to the Special Issue on Child Welfare Literature Reviews: Literature Reviews, Systematic Reviews and Meta-Analyses—How Can Child Welfare Administrators, Practitioners and Researchers Determine What We Know?

Introduction

Just over a decade ago, the Journal of Public Child Welfare (JPCW) was established to fulfill the need for a forum for publication of research directly related to publicly funded child welfare. Founding Editors Drs. Katharine Briar Lawson, and Martin Feit recognized that although existing scholarly journals were publishing child welfare-related work, the emerging volume of research in this area necessitated a focused publication to serve as a resource to the field. Under the leadership of Editor Emeritus, Dr. Alberta Ellett, the journal has worked to serve in this capacity, publishing a wide range of research to inform child welfare policy and practice.

Over many years we have seen repeated calls for the use of research evidence in child welfare policy and practice (Lindsey & Shlonsky, 2008; Testa & Poertner, 2010), while simultaneously acknowledging that the evidence base in this field was early in development compared with other fields (Barth, 2008; Usher & Wildfire, 2003). Concurrently, the evidence-based practice movement has progressed, beginning in medicine (Sackett, Richardson, Rosenberg, & Haynes, 1997), and later promoted in social work (Gibbs, 2003; Gibbs & Gambrill, 2002). In 2008, Barth warned,

if efforts are not focused on better ways to achieve outcomes [in child welfare], policy makers and other individuals outside of the child welfare profession who do not have an accurate understanding of the issues will take on this task. It is imaginable that the manner in which they do so will not match child welfare’s work requirements and generate advances in the field of child welfare.” (2008, p. 152).

Given the aims and scope of the JPCW, the Editors felt it was time to devote a special issue to child welfare literature reviews to inform our readership of the status of the evidence base on topics related to child welfare practice. However, in consulting with the editorial board, it quickly became apparent that describing and synthesizing the existing evidence in child welfare involves far more than seeking reviews of the literature and compiling them in a special issue. If sound methods are required to determine the effectiveness of interventions in individual studies, these same standards need to be applied to the synthesis of studies reporting on the effectiveness of these same interventions.

An evidenced approach to literature reviews

Traditional narrative literature reviews are descriptions of author-selected studies that are reported to the reader in varying levels of detail. The importance of each included study is considered and weighted by the author with respect to how much it tells them about significant outcomes and related processes; the range of evidence is then summarized and the author’s opinion about the intervention is rendered. There is often a set of recommendations. The “literature” is replete with examples of this type of review, but it turns out that many of the same reasons that drive us to actually test interventions using valid sampling techniques, informative study designs, reliable and valid outcome measures, and appropriate statistical tests are also in operation when we attempt to synthesize evidence across a range of studies.

To illustrate, consider a primary study (i.e., not a review) that reports positive outcomes for an intervention that was given to 25 clients selected by the intervention developer. The study reports on progress, pre- and post-intervention, for the 10 clients who completed the program. The original sample could be described as a “convenience sample”—they were accessible and were selected for inclusion by someone who has a vested interest in showing positive results. Why were these clients included? How many were offered the intervention and turned it down? How representative of all clients with the problem is this group? In terms of effectiveness, there is no control or comparison condition, meaning that it is unclear what would have happened to a similar group of clients who did not receive the intervention. Furthermore, results are only provided for the 10 clients who completed. What happened to the other 15 clients? Might they have fared worse? The most we can say about such a study is that the findings are of interest and more rigorous testing of the intervention is required.

Next, consider a narrative literature review as a single research study that reports outcomes for a particular intervention, and each study that is included in the review is a participant who receives the intervention. How were the participants (in this case, the original studies included in the review) selected? Is the sampling inclusive of all study participants? Are each of the pre–post measures for each participant the same and measured at the same time? Do each of the participants receive the same intervention? Are all of the outcomes reported for all of the participants? Are the study methods of equal quality for each participant? Is the statistical significance calculated using statistics, or just “eyeballed”? How are the outcomes for each participant weighted in terms of their importance? Are certain participants more important than others in making an assessment of effectiveness? Is the level of effectiveness (effect size) calculated using a standardized measure, or is it just a guess by the author? The most we can say about such a narrative literature review is that the findings are of interest but that a more rigorous evidence synthesis is needed.

Fortunately, these more rigorous synthesis methods have been developed and are widely in use. Systematic reviews are transparent, replicable evidence syntheses that follow a detailed protocol that is developed prior to the evidence search. The protocol pre-specifies these points: a comprehensive search strategy to identify and assess all relevant studies (including gray literature) for a clearly articulated research question, clear inclusion and exclusion criteria, explicit screening and data extraction procedures that involve multiple raters, and detailed synthesis plans for meta-analysis (statistical combining of studies) that, when possible, is used to generate an average expected treatment effect across studies (Higgins & Green, 2011). In short, systematic review methods contend with many of the known biases found in narrative literature reviews, providing the child welfare field with a valid process for developing an evidence base in policy and practice. They represent the top of the evidence hierarchy for effectiveness studies (Puddy & Wilkins, 2011) and, because they are supposed to be updated regularly, systematic reviews are living documents that represent the evidence base over time.

How far have we come? Following in the footsteps of the Cochrane Collaboration (n.d.) which was established to promote evidence-informed health care through high quality systematic reviews and other research syntheses (http://www.cochrane.org), the Campbell Collaboration (n.d., http://wwwcampbellcollaboration.org) promotes evidence-based policy and practice through systematic reviews and other research synthesis methods in a growing number of domains including crime and justice, education, international development, knowledge translation and implementation, and social welfare. Both groups are part of a worldwide effort to systematically synthesize evidence in health and social services and, by extension, child welfare.

However, of the 60 systematic reviews posted in the social welfare category, as of this writing a number are directly relevant to public child welfare. Most involve out-of-home care, although many others related to the behavioral health treatment of children who could also be served by the child welfare system. That said, a study of the impact of Campbell reviews (Maynard & Dell, 2017) found that the 52 social welfare systematic reviews they examined were downloaded 136,356 times, were cited 3,184 times and surveys of authors indicated that a number of reviews influenced policy and practice. There are also a number of other groups and individual academics that are conducting systematic reviews, some of which can be of high quality (EPPI-Centre https://eppi.ioe.ac.uk/cms/); Centre for Reviews and Dissemination https://www.york.ac.uk/crd/).

Despite these gains, the use of systematic reviews in social science still has a long way to go. While being the top of the evidence hierarchy, they are not required in order to be registered on government and government-funded websites purporting to list “effective” or “evidence-based” programs and services. Even when they are based on a search, this approach is problematic for a number of reasons, including variations in what is required to achieve effective status on each list; selective outcome reporting in primary studies (i.e., only some outcomes are reported rather than all of the outcomes that were measured; Dwan et al., 2010; Norris et al., 2012); conflicts of interest and other biases that might be influencing findings (Gorman, 2016); an apparent inability to remove programs from the list even in the presence of evidence that an intervention is not effective (Gorman, 2015); and the major problem of replicability of findings for a large number of randomized controlled trials (Collaboration, 2015). Systematic reviews, while they still have the potential to carry bias (Uttley & Montgomery, 2017), are the most promising method for keeping known sources of bias to a minimum and, at the very least, are fully transparent, living documents that allow practitioners and policy-makers to easily access current best evidence when making critical decisions.

Our solution for the special issue

Given the overwhelming evidence that systematic synthesis gives us our best shot at developing a viable evidence-base in child welfare, why is it so slow in coming? The answers are likely to be manifold, but surely include a long social science tradition of conducting narrative reviews, vested interests defending positions taken long ago, a lack of high quality primary studies upon which to draw despite the fact that urgent decisions must be made, and perhaps an aversion to the somewhat mechanistic nature of synthesis methods. The biggest reasons, however, may be that high quality systematic reviews are extremely difficult to execute, they tend to take a long time, they generally require a team, and they can be costly. In short, just like any form of rigorous research, systematic reviews can be frustrating, lengthy processes filled with unexpected challenges and delays.

Therefore, for the special issue, we decided not to limit submissions to full, high quality systematic reviews but rather to require a transparent and replicable approach to the literature reviews that included as many systematic review procedures as possible—all in the service of minimizing the variable biases that can be introduced through narrative reviews. The Call for Papers required authors use systematic processes for their review, and encouraged the use of standardized approaches to systematic reviews such as the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement, including the PRISMA checklist and flow diagram (http://www.prisma-statement.org/) .

The PRISMA Statement consists of 27 essential items that should be present to ensure the thorough and transparent reporting of a systematic review or meta-analysis (Liberati et al., 2009; Moher, Liberati, Tetzlaff, & Altman, 2009). While emerging from the arena of health care research, the PRISMA guidelines are relevant to any systematic review (Moher et al., 2009). Even those reviews that are not directly investigating intervention research can still seek to meet PRISMA statement expectations related to how source material was located, selected for inclusion, evaluated for potential biases, and synthesized for the reader’s consumption.

While the PRISMA checklist is not intended to be used as a quality assessment instrument (Moher et al., 2009), it can be useful for appraising the degree of thoroughness with which authors have reported the results of their investigations. An adequately reported systematic review should provide the reader not only with an accurate and thorough summarization of the reviewed research evidence, but also enough information about the investigation itself to facilitate assessment of the review’s strengths and limitations. To this end, the PRISMA checklist was employed as one component of the special issue’s review process, which standardized the process and allowed us to determine the extent to which the reviews met a set of stringent criteria for avoiding bias. The 27-item checklist provided a consistent basis whereby submissions across a broad range of topic areas could be compared and contrasted, at least in terms of completeness and transparency—no small task in a field of research with as many subdomains as child welfare. Later in the review process, as the strongest candidates for publication began to emerge, the PRISMA checklist assisted in the provision of concrete editorial feedback to authors as they refined and revised their manuscripts, often across multiple iterations. Although this deliberate focus on the concepts encapsulated in the PRISMA checklist undoubtedly increased the editorial burden and authorial demands, we are confident that it has ultimately resulted in a special issue that will prove more valuable to the readers as they consider how the research evidence presented within can best inform child welfare practice, policy, and research.

Producing a special issue on this topic presented a number of challenges that may be less common for other special issues we have published focused on particular areas of child welfare practice, such as in-home services or intervention with families challenged with disabilities. This special issue could not be produced in the same time frame as others—competent and systematically conducted reviews take a long time. This presented a challenge in waiting to receive an adequate number of submissions to consider moving forward in the revision process to publication. Identifying appropriate reviewers for this issue was also a challenge. Fortunately several JPCW Editorial Board Members agreed to participate, and we are indebted to Drs. Anita Barbee, Rick Barth, John Orme, and James Whitaker for stepping up to participate in a rigorous review process. In addition to those willing to look at the rigor of the literature review process, it was important to identify subject matter experts for each paper as they all focused on different topics relevant to child welfare. While we received many submissions, we believed it necessary to uphold systematic review methods, as reflected in PRISMA, to the greatest extent possible and, therefore, a number of submissions could not be moved forward to publication. Those authors remaining were asked to undergo multiple revisions in the interest of bringing the reviews to the best possible position considering the limitations of the topic, existing literature, and timeframe for publication. In some ways, this grand compromise may have resulted in a series of rapid evidence assessments—a synthesis product that comes as close as possible to the standards for a systematic review given time and budget constraints.

The process was far more difficult for authors than, perhaps, they anticipated. One author team, after their manuscript was accepted commented that this was the most intensive and challenging review process they have undergone. Anecdotally, Campbell Collaboration authors often report a similar experience. We do not report this to extoll the virtues of our journal but to underscore that this work is difficult. However, if we wish to legitimately inform the field about the evidence for key questions of importance to child welfare through the use of reviews, we need to become far better and raise our standards.

Unlike substantive area-focused special issues, we cannot provide a historical account of the advancement of the field in the topical area or a summary of the extent literature. We will instead comment on the relative relevance of the various types of literature reviews in informing policy and practice decisions. Child welfare decision-makers no doubt find themselves at a loss to know how much evidence is enough to support the use of a specific practice, program, or service, or how to interpret the results of reviews, even those that are of high quality. However, we must endeavor to use the best methods available for synthesizing the research evidence we have. This will help us determine what we know and what we need to find out. Eventually, if we do this well enough, we will find ourselves the owners of a living, world library of solid evidence that can be used to better guide the important practice and policy decisions we will all face in the years to come.

Summary of articles in the special issue

Given the wide range of topics of relevance to the JPCW readership, and the varying status of the evidence base across them, this special issue includes articles ranging from services used by child welfare clients to outcomes of interest to youth and families served by the system to training for foster parents to a broader review on implementation fidelity. In “Predictors of Mental Health and Developmental Service Utilization Among Children Birth to Five in Child Welfare: A Systematic Review,” Keyser and Ahn (2017) assess the evidence base as it relates to factors predicting the use of services for the most vulnerable children in this population. Two articles focus on different topics of relevance to other youth and young adults served. Morton and Wells (2017) focus on families experiencing a narrow but highly concerning form of substance misuse, and the state of knowledge regarding outcomes of older youth raised in those families, in “Behavioral and Substance Use Outcomes for Older Youth Living with Parental Opioid Misuse: A Literature Review to Inform Child Welfare Practice and Policy.” Geiger and Beltran (2017) review an area of research with a short history but the potential to have a great impact through looking at factors which may influence the success of youth who have aged out of the foster care system in higher education, in “Readiness, Access, Preparation, and Support for Foster Care Alumni in Higher Education: A Review of the Literature.”

Two articles examine the evidence base around two out-of-home care outcomes. In “Family Reunification: A Systematic Review of Interventions Designed to Address Co-occurring Issues of Child Maltreatment and Substance Use,” Murphy, Harper, Griffiths, and Joffrion (2017) highlight four intervention approaches that have been used with this population. The article “Risk and Protective Factors of Foster Care Re-entry: An Examination of the Literature” examines the ecological domains of child, family, community and system factors associated with re-entry or its prevention, by authors Semanchin Jones and LaLiberte (2017).

The final two articles review the literature as it relates to implementation of child welfare interventions. Uretsky and Hoffman (2017) focus on the evidence base regarding manualized training programs focusing on providing caregivers with the skills to deal with the types of behaviors in youth that are often related to another critical child welfare outcome of interest—placement disruption—in “A Systematic Review and Meta-Analysis of the Evidence for Group-Based Foster Parent Training Programs in Reducing Externalizing Child Behaviors.” Finally, in “Contributors to Fidelity of Child Welfare Related Interventions: A Review,” Bartley, Bright, and DePanfilis (2017) examine specific organizational strategies appearing to yield greater fidelity across sites.