Reflections on systematic reviews: moving golden standards?

A systematic review should be a robust and sensible summary of research. Here, I present some of the most common types of systematic reviews and offer some insights into the differences and similarities among them. The aim is to describe how varying approaches to systematic reviews have different requirements depending on input, analyses, and possible output. The overview is followed by a discussion of basic requirements that cross the different methodologies to increase rigour, transparency, and replicability.

The role of a systematic review is mainly to synthesize the knowledge base in a research field on a certain topic or research question. In some sense, it will guide us to know what has been examined and what works so far. To my knowledge, no single textbook describes how to perform different systematic reviews; thus, this commentary attempts to draw a picture of the options available and the benefits and drawbacks of different approaches.

As an interdisciplinary field, hospitality and tourism research is drawing on ideas and ideals from several fields depending on the starting point of the author(s). Some disciplines seem to have a preferred golden standard for conducting and reporting reviews whereas others seem less concerned about reporting standards. Tourism and hospitality journals do not have a long or strong tradition for publishing review papers, but the numbers are increasing. Given the lack of strong standards, published review papers are of different genres (i.e. conceptual, critical, theoretical) and of varying qualities, as is also the case for the Scandinavian Journal of Hospitality and Tourism (SJHT; see for example Black, Weiler, & Chen, 2019; Gjerald & Øgaard, 2008; Jensen, Lindberg, & Østergaard, 2015; Mura & Sharif, 2017; Müller & Hoogendoorn, 2013; Qian, Law, & Wei, 2018; Solvoll, Alsos, & Bulanova, 2015).

My first-hand experience with authoring review papers is that the quality requirements are increasing as we speak. This is partly due to the maturity of the genre, but also due to the increasing expertise among authors, editors, and peer reviewers. In this vein, I would add that SJHT is open to a variety of approaches to reviews as long as future manuscripts meet the overall criteria outlined at the end of this note.

Several approaches to systematically reviewing a research field, topic, or question exist, and what follows here is a short outline of the main features of common methods and approaches as well as references to examples of such papers covering topics in hospitality, tourism, sports, service, and leadership studies.

Narrative reviews and narrative synthesis (e.g. Mair, Ritchie, & Walters, 2016; Yang, Khoo-Lattimore, & Arcodia, 2017) are generally comprehensive summaries aimed at theorizing. Narrative reviews may cover a wide range of issues on a given topic, such as a historical perspective. A typical narrative review does not necessarily state or follow rules about the search for evidence or reveal how analytic decisions are made about the relevance of studies and the validity of the included studies.

Meta ethnographic reviews or meta syntheses (i.e. Hoon, 2013; Noblit & Hare, 1988) are another way of systematically reviewing and integrating findings from qualitative studies (e.g. Lazazzara, Tims, & de Gennaro, 2019). Qualitative syntheses require more than using a systematic approach to collecting, analysing, and interpreting results across studies. There is also a requirement for developing new knowledge through the development of an overarching interpretation evolving from the combined interpretation of the primary studies included in the synthesis, thereby going beyond the findings of individual studies in order to build theory.

Integrative reviews (e.g. Akerjordet, Furunes, & Haver, 2018; Baum, Kralj, Robinson, & Solnet, 2016; Haver, Akerjordet, & Furunes, 2013; Oliveira & Lumineau, 2019) allow for the inclusion of original papers with different methodologies in the same review paper (Whittemore & Knafl, 2005). This type of review is suitable when reviewing a field with relatively few published papers and with varying methodology, including qualitative, quantitative, and mixed methods. The strength of an integrative review is that all types of papers can be included; the weakness is that there are limited opportunities for statistical calculations across studies. There are several ways of enhancing the transparency in reporting the synthesis of qualitative research, for instance the ENTREQ framework by Tong, Flemming, McInnes, Oliver, and Craig (2012), which is a statement consisting of 21 items grouped into five main domains: introduction, methods and methodology, literature search and selection, appraisal, and synthesis of findings.

In some disciplines, quantitative systematic reviews are used to compile the best evidence or guide policymaking (e.g. Gomezelj, 2016; Weed, 2006). Systematic reviews are underpinned by positivistic epistemology and aim for comprehensive sampling, possibly also including unpublished studies. The strength of a systematic review is the possibility to use statistical techniques to combine results of eligible studies or to use scoring of the levels of evidence, depending on the methodology used (e.g. SIC). Often several raters are used to resolve any scoring differences among them. Compared to meta-analyses, systematic reviews do not require the use of the exact same scales. A weakness is that qualitative studies are excluded (i.e. Roy, Lassar, Ganguli, Nguyen, & Yu, 2015).

Thus far, meta-analysis (e.g. Svensson et al., 2018), following Cochrane’s Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011), is the golden standard in many disciplines, allowing for the re-analysis of data across published and unpublished studies. The basic principle behind meta-analyses is that there is a common truth behind all conceptually similar scientific studies, measured with a certain error within individual studies. A meta-analysis also has the capacity to contrast findings from different studies and find patterns among study results, sources of disagreement among those results, or other interesting relationships that may appear in the context of multiple studies. The strength of a meta-analysis is that information across studies is aggregated to get a higher statistical power and a more robust point estimate than from a measure derived from one individual study. One of the weaknesses is that only papers that have used the same measures and with high reporting standards can be included. For both systematic reviews and meta-analyses, there is a tradition for including unpublished studies, particularly when the number of published studies is low. Including unpublished studies reduces publication bias, which is the risk of only including published studies. This is a bias, as studies with significant results more often are published than studies with no significant findings.

General requirements

Having acquired the first-hand experience with review papers, I would strongly argue that there are several criteria, as discussed herein that should be applicable across review genres to strengthen contribution and quality. Review papers could have an open or a narrow focus, but all types need a specified, well-justified research question, problem statement, or hypothesis.

Across approaches, reviews should contribute to knowledge by being timely, systematic, focused, and critical. Being timely concerns several issues, including being up to date, while simultaneously not forgetting or automatically excluding older studies. In addition, there has to be a certain number of papers or analytic units across studies to be able to conduct a review, but also for a review to be interesting. Thus, timely also refers to how far a research field has come for a synthesis to be appropriate. Given that systematic reviews are a prerequisite for building new knowledge by means of new studies, it is relevant to ask which reviews are most likely to fulfil the needs of readers and researchers at a given time. Also, critiquing previous research and influential papers is essential for high-level knowledge construction in any research field (Zarezadeh, Benckendorff, & Gretzel, 2018).

Different research fields and journals have adopted different approaches and standards for conducting and reporting systematic reviews (Kim, Bai, Kim, & Chon, 2018). Across different review genres, I would strongly argue that an open, systematic, well-reasoned, and well-documented search strategy is core. Whereas some recent reviews have stated that selected hospitality and tourism journals were searched, I would maintain that such procedures create bias, unless the aim of the paper is to review all papers on a specific topic in a limited sample of journals. Rather, open searches across several information sources (i.e. databases, search engines, and manual searches of relevant academic journals) will increase the contribution of a review paper. Online databases have different qualities that the researcher should consider when performing the review. One of the main criteria for database selection when performing a systematic review should be replicability. Whereas Google Scholar has a broad outreach, replicability is challenging as two searches would not retrieve the same results and only display the first 1000 hits in ranked order. In cross-disciplinary databases, such as Web of Science, or disciplinary databases such as Hospitality and Tourism Complete and Business Source Complete, replicability is easier. Some databases, such as Web of Science and Cinahl, also provide search tools to find the best search terms. To ensure that searches are replicable, the exact search strategy and search terms should be reported in the review paper. There will always be a language bias, and researchers have to realize that by searching and including only papers written in English, they risk excluding several relevant contributions published in other languages.

After displaying a well-defined search strategy (databases, search terms, time limits) illustrated by a flow chart (see, for instance, PRISMA; Moher, Liberati, Tetzlaff, & Altman, 2009), the review paper should clearly state and justify the inclusion and exclusion criteria. Determining which papers should be included or not includes decisions on whether to include published peer-reviewed papers only or to include grey papers, editorials, theses, etc. Another decision concerns whether you are including conceptual papers and already published systematic reviews or are limiting the analysis to empirical papers only. Authors should build clear argumentation for the choices made in relation to their research questions.

A review should include a quality assessment of included papers. Different disciplines and different review genres have different requirements for quality assessment. For assessing qualitative papers CASP (Critical Appraisal Skill Programme, Singh, 2013) is a useful 10-point checklist. For meta-analysis, the Cochrane guidelines have strict requirements (Higgins & Green, 2011). As a general rule, more than one author is needed to reduce bias. Furthermore, study characteristics should preferably be displayed in a table. Yes, the tables might be long, but there are opportunities to provide this kind of information in appendices or for researchers to share files on, for example, Research Gate, Academia, and Google Scholar. The transparency of which papers are included is preferably done in a data extraction table, displaying characteristics of the research study details. Should studies be excluded because of low quality, low response rates, or limited samples? If so, authors have to argue why such papers are excluded; if not, authors should comment on how the included studies meet quality requirements.

All publications need to state the unique contribution of the review paper. Describing the research to date is not enough because a synthesis is more than the sum of included papers and should provide a meta perspective. When systematic reviews are published, then what? A solid systematic review should not only synthesize the current knowledge, but also look for potential bias in the included studies. Whereas systematic reviews and meta-analytic reviews can offer recommendations for future practice, the opportunity to address a future research agenda should not be missed. Moreover, authors’ ability to point toward the future and point out future research lines will distinguish an interesting review from a less interesting review paper. One of the challenges is thus to keep focused, but with a broad interest.

A general challenge within tourism and hospitality research seems to be building accumulated knowledge. This may partly be reflected in the design (i.e. numerous case studies), but seemingly also in researchers’ lack of focus on building on previous research (own or others) when designing new studies. I hope that this does not discourage you from conducting systematic reviews in the future. There are still many research topics that would benefit from a systematic evaluation and synthesis of current knowledge. SJHT encourages researchers to look at their own research areas and decide whether it is time for a systematic knowledge compilation.

Acknowledgements

I would like to thank Prof. Dr. Annet de Lange, Prof. Jens Kristian Steen Jacobsen, Prof. Håvard Hansen, PhD student Thea Steen Skogheim, Academic Librarian Else Sauge Torpe, and Specialized Librarian Kari Hølland for commenting on earlier drafts of this manuscript.