Changing the world one systematic review at a time: A new development methodology for making a difference

Abstract

This paper presents a methodology that is increasingly being adopted in international development, namely systematic review. It starts from the premise that rigorous and structured systematic reviews of research evidence have the potential to ‘change the world’ by providing accurate comprehensive summaries of knowledge for decision-makers, and goes on to outline the key processes involved in conducting these reviews. It outlines the methodology in terms of the key research stages of: establishing your review question, collecting data, analysis, reflecting on the strengths of your methods, and drawing conclusions. In doing so, it lays down a challenge to development researchers, to engage with the approach and improve the rigour of our literature reviews.

Keywords:

• methodology
• literature review
• systematic review
• research impact

1. Background

Is it only me, or do other academics, researchers and students ever wonder why they bother doing research? It is a slow and unwieldy process and it is often hard to see what impact we have, if any (Adato & Meinzen-Dick, 2002; Lavis et al., 2003:165). Despite this, we continue in the hope that we are addressing questions of importance to the people whose lives we research (Hovland, 2007). We hope we are avoiding bias, our findings are as objective as possible, and we are contributing to the wider ‘field’ in which we work by advancing theory and/or testing hypotheses (Boaz & Ashby, 2003:5). For many of us, our intention is to provide timely information for decision-makers so they can make ‘evidence-informed decisions’. Despite these aspirations, there is a risk we are producing research that is mediocre or addresses questions only we are interested in and which then sits on a shelf unread.

It might be different if there existed summaries of all of the available research evidence for any given question that were relevant, effective, systematic, comprehensive, up-to-date and accessible (Moynihan, 2004:3). If they did exist, researchers and research commissioners could refer to them before starting new research to ensure they were not unnecessarily duplicating existing work. Researchers could send in their research findings to contribute to the growing pool of knowledge. Policy-makers and practitioners could use these to inform their decisions. Policies would be more effective, with less money wasted on the ill-informed decisions. Maybe, the world would be changed for the better.

This was the vision of the first systematic reviewers. The Cochrane Collaboration, for example, was the first and is the largest international network of systematic reviewers, whose vision is ‘that healthcare decision-making throughout the world will be informed by high-quality, timely research evidence’ (Cochrane Collaboration, 2012).

1.1 What is a systematic review?

Systematic reviews are essentially large pieces of ‘secondary’ research. To conduct a review, you need to search out, collect, read and combine all of the available relevant research on a given topic in order to provide ‘the answer’ to a question of importance (Mulrow, 1994:597). That question might be an exploratory, theoretical one, in which case you would seek out conceptual research; or it might be one of the effectiveness of an intervention, in which case you would seek out impact evaluations. Systematic reviews typically take a team of two to four people around a year to complete. They require specialist methodological skills and topic expertise. The review process is laborious, and typically results in a long and detailed report for the purposes of completeness and transparency. Reviews often begin by identifying many thousands of potentially relevant references, which are then filtered down by relevance and quality to include only the good quality pertinent research (see for example Duvendack et al., 2011; Stewart et al., 2012b).

You might ask whether all this effort is worthwhile when more traditional literature reviews are so much simpler and primary research seems more ‘cutting edge’. But systematic reviews have been proposed as a means to avoid research sitting on a shelf unread; and instead contributing to understanding issues and finding solutions to the world's problems, and enabling decision-makers to access the best available research evidence to inform their decisions (Boaz et al., 2002:1; Sebba, 2006:2). The suggestion is that, if you want your research to make a difference, then you need to engage with systematic reviews, even if you do not conduct them yourself.

1.2 A brief history of systematic reviews

Systematic reviews are new to the development field, having been introduced through the work of the International Initiative for Impact Evaluation (3ie), funded by international donors such as the UK's Department for International Development. Since the first reviews in international development were commissioned in 2010/11, more than 200 have been published and many more are now being conducted.²

Having said this, systematic reviews are not entirely new. They have been conducted in healthcare for over 20 years. Indeed it is the norm in medicine for healthcare policies, clinical guidelines and individual doctors' practice to be based on the findings of the best available evidence as summarised in a systematic review (Cochrane Collaboration, 2012). Some 20 000 reviewers undertake and publish systematic reviews on healthcare issues that are used on a daily basis by practitioners and policy-makers alike.

To give an example of the power of systematic reviews in development it is worth returning to the topic micro-credit. Whilst these small loans have long been considered a silver bullet solution to poverty enabling poor people to start businesses and lift themselves out of poverty, systematic reviews of the evidence have now overturned this assumption, clearly demonstrating through a comprehensive and systematic assessment of the relevant research that there is no evidence for the positive impact of micro-credit on the lives of the poor (Stewart et al., 2010, 2012b; Duvendack et al., 2011).

Given the potential power of this methodology, it is surprising that it has only recently been adopted in international development. This article aims to introduce the approach to a wider community of development researchers and practitioners in a field where the approach is still widely unknown.

1.3 In what ways might the development researcher engage with systematic reviews?

As explained below, we can seek out reviews when designing our research to ensure we build on the wider body of research in our area and do not unnecessarily duplicate work. We can ensure our research is available for inclusion in reviews of the evidence, by publishing our research in full, including details of the context, methods and all findings (both negative and positive). We can adopt the principles of systematic reviewing in our work, and encourage the systematic review of available research to address questions and develop ideas. And we can, of course, conduct reviews ourselves.

It is possible to do ‘mini’ systematic reviews, often known as rapid evidence assessments, which might take between three and six months (UK Cabinet Office, 2008). These tend to be less ambitious in scope, with a narrower regional or topic focus. Pragmatic decisions are made about what can be achieved in a limited time, and steps are taken to ensure rigour is maintained despite the tight timeframe (Stewart et al., 2012a). These short reviews are more common when addressing policy imperatives, where timeliness is particularly important.

Another shortcut to systematically reviewed evidence is to draw on existing systematic reviews. Where several relevant reviews exist, a ‘review of reviews’ can be conducted, which applies systematic principles to available reviews. Also increasingly common is the reanalysis of data published within systematic reviews; for example, drawing on the same collection of relevant literature that has been systematically sought and screened for relevance to the topic.

2. What is involved in a systematic review?

As suggested by the name ‘systematic review’, in order to reach our final synthesised findings of the available evidence base we must first follow a logical process. This includes the stages employed in most research: establishing your research question and conceptual framework, collecting your data, analysing it, reflecting on the strengths and weaknesses of your methods, and drawing out your results and conclusions. Each stage and its application to development research are discussed below.

2.1 Establishing your question and conceptual framework

To ensure a review question is relevant and focused it is important to read around the topic and ask people what it is they want to know. Potential users of the review may include policy-makers, practitioners, research funders or those using the services you are researching, such as patients or particular members of the public.

It is also important that a review question is answerable. The broader the question, the more literature you are likely to find and the bigger the task of reviewing it. As with all research, if you are trying to find the answers to a lot of questions it will take you time. On the other hand, if your question is too narrow it may not be useful to anyone. In the area of microfinance, the first reviews sought to answer the question ‘does it work?’ (Stewart et al., 2010; Van Rooyen et al., 2013); however, later reviews have attempted to unpack more closely the circumstances in which different microfinance services work and for whom (Stewart et al., 2012b). Involving a range of stakeholders in the initial stages of a review can increase its usefulness (Rees & Oliver, 2012). There may also be other related reviews already published that draw on a similar literature and you may be able to draw on their search results. Completed systematic reviews, and those underway, are routinely published via a number of online libraries so they can be accessed easily (see Table 1).

Table 1: Key systematic review libraries

Systematic review library	Areas covered	Website
3ie database of systematic reviews	Development	www.3ieimpact.org/evidence/systematic-reviews/
Cochrane Library	Health	www.thecochranelibrary.com/
Campbell Library	Social policy, including development	www.campbellcollaboration.org/library.php
Environmental Evidence Library	Environmental research	www.environmentalevidence.org/Reviews.htm

Having established that your question is relevant, realistic and as yet unanswered, you then need to establish your conceptual framework to ensure a shared understanding of the key ideas in the review. This is particularly important if addressing a cross-disciplinary question. As well as ensuring shared understanding of concepts, it is also important to consider the logic behind what you are studying, its purpose, who it is for and what it is meant to achieve. In addressing these questions you will establish the conceptual framework for your review.

2.2 Data collection

There are three stages to data collection in a systematic review: identifying relevant research to include, critically appraising each included paper to establish its relevance and quality, and extracting the findings from the included studies.

2.2.1 Identifying relevant primary research to include

To decide which studies you are going to use to answer your review question, you must first establish relevance and quality criteria. These are specific standards that you apply to each identified paper to assess firstly whether it is relevant to your review and, if judged to be relevant, whether it is of sufficient quality for you to have confidence in its findings. The first set of criteria is known as inclusion and exclusion criteria, and the second as quality appraisal criteria (see Van Rooyen et al., 2013). Having established these criteria, you then want to search in such a way as to identify all of the potentially relevant research. As with all data collection, the more you search for literature, the greater the chance that you capture everything, but the more time you will have to spend screening out the irrelevant studies. Your ability to identify relevant research depends on systematic searching of mainly, although not limited to, online electronic databases. Developing and testing a systematic search strategy for your review is therefore essential and requires you to be familiar with searching online databases, or to have access to specialist help (Brunton et al., 2012:115–8).

2.2.2 Filtering your search results for relevance

In all likelihood, searching for literature using electronic databases results in hundreds if not thousands of hits. The first stage of filtering your results is therefore applied to the titles and abstracts, which are most easily available with your electronic search results. This allows you to remove any clearly irrelevant literature, reducing the number of full texts that you then have to access. Your inclusion and exclusion criteria can then be applied a second time to the full papers or reports and your list of included studies established. To reduce bias, more than one researcher is usually engaged in these filtering processes, with two researchers either screening all of the abstracts and full texts independently or comparing results or sub-sets to ensure inter-reviewer reliability before sharing out the task between them.

2.2.3 Critically appraising the quality of each relevant study

Having established which studies you will include in your review, you then need to assess their quality. This will allow you to decide which studies are not good enough quality and should be excluded from your review and/or whether to weight the findings of the relevant studies to give greater significance to some above others. Whilst precise criteria for judging quality will depend on the nature of the research you are reviewing, and to an extent the accepted standards within the discipline within which you are working, the following broad principles commonly underpin quality criteria:

• The appropriateness of the methods used for addressing the question (e.g. the use of only qualitative data for measuring impact or the use of only quantitative data for explaining impact would both be considered inappropriate).

• The extent to which the methods were applied appropriately (e.g. a study that compared the impact of an intervention amongst two groups of people who differed in age and gender from one another would be considered low quality).

• The extent to which the findings were in keeping with the methods employed (e.g. a study that concluded on the acceptability of an intervention to a population when no qualitative methods were employed would be considered low quality).

As with your relevance criteria, more than one researcher should apply your quality criteria to ensure consistent application and reduce the potential for bias (Waddington et al., 2012:365).

2.2.4 Extracting the findings from the included studies

The final stage of data collection for your systematic review is the extraction of findings from all those studies judged to be relevant and good quality. By specifying in advance which outcomes you are interested in and searching out findings from within your included studies for each of these outcomes, you are able to comment on the availability of, or lack of, evidence addressing each outcome. The extraction and recording of findings from studies is therefore done in a structured way, allowing findings related to particular outcomes to be collated.

2.3 Data analysis or ‘synthesis’

Depending on the question you are seeking to address in your review, and the type of data you have available to you, there are a number of ways to synthesise the findings of your studies. Broadly speaking there are two key approaches: statistical meta-analysis and narrative synthesis.

Statistical meta-analysis involves combining the numerical data from each study into one ‘pooled effect size’, almost as though all the participants in the individual studies are being re-entered into one larger study. Fortunately, in order to do this, you do not need individual-level data on each participant, but can standardise the available study-level statistics and then combine them to give a pooled result. Original datasets are therefore not required and the information within a published paper is usually sufficient. Details of how to conduct these analyses are freely available (Higgins & Green, 2011), and there are a number of software packages that can assist with the process (Bax et al., 2007). There are numerous examples available, and readers may want to read further in freely available online systematic reviews; such as Tripney and colleagues' meta-analysis of the evidence on technical and educational vocational training (Tripney et al., 2013), and Volmink and Garner's meta-analysis of directly observed therapy for treating tuberculosis (Volmink & Garner, 2007). It is not uncommon in development economics to conduct statistical meta-analysis; however, the selection of studies to be combined in this way is not always comprehensive or transparent. Meta-analyses (outside of systematic reviews) should not therefore be assumed to be ‘systematic’ in the same way as systematic reviews are.

Narrative synthesis, as the name suggests, enables you to bring together the findings of different studies using qualitative summaries (Barnett-Page & Thomas, 2009). Matrices are used to enable the results from individual studies to be compared and contrasted and patterns identified. This process is not unlike analysing qualitative data within a primary study. If your review is addressing questions about how or why something is happening, it will focus on qualitative data of participants' views and experiences, so narrative syntheses of individual studies' findings will always be necessary. If you are exploring the impact of an intervention, it is possible to combine qualitative summaries of findings together with quantitative data within synthesis matrices; for example, including a description of their findings and the direction of statistical effect (showing positive or negative impact of an intervention and/or the size of the impact assessed). As above, the reader may want to refer to published examples in reviews of microfinance (Stewart et al., 2010) and of agriculture (Hall et al., 2013).

Whether you have used statistical or narrative approaches to combine the findings of individual studies in your review, your synthesised results should be an accurate representation of ‘what we know’ on the specific question(s) addressed in your review. As such, they provide a valuable oversight of the evidence base in which we can have confidence (Waddington et al., 2012:359).

2.4 Reflecting on the strength of your methods

As with all research, it is important to reflect on the strengths and weaknesses of your research before drawing conclusions and making recommendations. In relation to systematic reviews, it is important to consider the following elements of your methodology: the extent to which your search was comprehensive, the likelihood of ‘missing data’ in terms of the publication bias of original studies, the accuracy of your data management, the potential for bias with your methods, the potential for bias within your included studies, and the variation between the findings of the different studies included in your review (Centre for Evidence-Based Conservation, 2010:39).

As outlined above, your aim is to identify all of the research relevant to your specific topic. Depending on how widely you cast your net, you may be systematically excluding certain literature. You need to reflect on your search results and the proportion of relevant ‘hits’. Did you search all possible sources using all relevant terms? If not, what might the implications be? Could you have missed a body of literature that is relevant? What about different languages? Is it possible that you have excluded some evidence because it is catalogued in other-language databases, or because papers were only available to you in other languages? Even if you have searched comprehensively, it is likely that there is bias within the published research base with academics and journals more likely to publish positive results. You can counter this bias to an extent by searching for unpublished reports on websites, in libraries and through contacting authors. You can also check lists of funded research and follow-up with research teams to request their findings. You should, nonetheless, reflect on the possible implications of publication bias on your review.

As with all research, it is easy to introduce errors when dealing with large volumes of data, or in this case research papers. Have you taken steps to ensure your data are accurate? What about other potential sources of bias? If you or your team have a particular view on what your findings should be, you need to reflect on this. Be honest about what surprised you in your findings, and clear about how your findings differ from those of others.

It is often, although not always, the case that the studies within your review will vary in their findings, displaying heterogeneity. It is possible to conduct a statistical test (I²) to assess the extent of heterogeneity in a review. There are many reasons why results might differ across studies, and you should explore these. Sub-groups of studies may show different results, for example those conducted in rural areas and those in urban areas, suggesting that there is a relationship between these contexts and the outcome of the programme that you are researching. Whilst it is not always possible to determine what is causing variety within the studies in a review, these issues should be explored in the review discussion.

2.5 Drawing conclusions and making recommendations

There are different practices with regards to making recommendations from systematic reviews. The Cochrane Collaboration avoids making recommendations for policy, preferring to leave the interpretation and application of the evidence to decision-makers (Schünemann et al., 2008:1). However, they do go to some lengths to produce accessible lay summaries of review findings. The EPPI-Centre, on the other hand, tends to draw out ‘implications’ for policy, practice and research (see for example Hawkes & Ugur, 2012). Whichever approach is taken, there are some simple principles that guide conclusions from systematic reviews. There is a clear need to distinguish between an absence of evidence (i.e. no/few studies that measure the outcome of interest) and evidence of absence (i.e. studies that show something is not the case or does not work). Simple statements such as ‘there is no evidence that x works’, all too often found in systematic reviews, can easily be misinterpreted. It is also all too easy for systematic reviewers to assume an authority in a topic and be overly critical of research in that area. Whilst it is true that systematic reviewers become very knowledgeable very quickly, and are practiced at critiquing others' work, there is a difference between research that is not suitable for a systematic review which is seeking to generalise findings and research that has no value. Much of the research that is excluded from systematic reviews for having a high ‘risk of bias’ will still be useful in specific contexts and for particular purposes.

3. Conclusion

Systematic reviews are valuable summaries of the research evidence base, and as such play a key role in evidence-informed decision-making. They provide a means by which development research can be combined to provide greater value than the sum of its parts, whilst avoiding bias and ensuring credibility. As well as providing relevant, effective, systematic, comprehensive, up-to-date and accessible summaries for decision-makers (Moynihan, 2004), they also have the potential to drive improvements in the rigour and reporting of development research more broadly (Cochrane Collaboration, 2012). In doing so, they help to address some of the key challenges in the field, including issues of rigour in design and implementation and the generalisability and utilisation of research findings (Van den Akker, 1999).

It is important to acknowledge that systematic reviews are only one form of knowledge that will inform a policy or practice decision, with financial decisions, political views, historical factors, experience and context all playing a part. In contrast to traditional literature reviews, however, they are comprehensive, providing a complete picture of the relevant and high-quality evidence available. Detailed reporting means they are transparent and replicable, enabling the reader to make informed decisions about their relevance and quality. Engagement with potential users of the review ensures their applicability to decision-making. In this sense, systematic reviews may not provide the answer: but they do nonetheless provide a new and important channel through which research can make a difference in international development.

Notes

²See 3ie's database of systematic reviews: http://www.3ieimpact.org/en/evidence/systematic-reviews/