Systematic literature reviews over the years

ABSTRACT

Purpose: Nowadays, systematic literature reviews (SLRs) and meta-analyses are often placed at the top of the study hierarchy of evidence. The main objective of this paper is to evaluate the trends in SLRs of randomized controlled trials (RCTs) throughout the years.

Methods: Medline database was searched, using a highly focused search strategy. Each paper was coded according to a specific ICD-10 code; the number of RCTs included in each evaluated SLR was also retrieved. All SLRs analyzing RCTs were included. Protocols, commentaries, or errata were excluded. No restrictions were applied.

Results: A total of 7,465 titles and abstracts were analyzed, from which 6,892 were included for further analyses. There was a gradual increase in the number of annual published SLRs, with a significant increase in published articles during the last several years. Overall, the most frequently analyzed areas were diseases of the circulatory system (n = 750) and endocrine, nutritional, and metabolic diseases (n = 734). The majority of SLRs included between 11 and 50 RCTs each.

Conclusions: The recognition of SLRs’ usefulness is growing at an increasing speed, which is reflected by the growing number of published studies. The most frequently evaluated diseases are in alignment with leading causes of death and disability worldwide.

KEYWORDS:

• Systematic literature review
• randomized controlled trial
• ICD-10 classification
• rapid review

Introduction

Presenting background information about a subject or documenting the growth of knowledge over time can be achieved with narrative reviews of the literature. However, they tend to be subjective as they rely on the author’s expertise on discussed topic, and offer a condensed presentation of a subject rather than an extensive one. Furthermore, they are frequently based on articles chosen selectively from the available material, which puts them at risk for systematic bias [1]. Typically, narrative reviews don’t describe how the review process was carried out [2]. As a result, they usually do not provide a thorough foundation for theory development and testing [3]. In 1979, British epidemiologist, Archie Cochrane wrote: ‘It is surely a great criticism of our profession that we have not organised a critical summary, by speciality or subspecialty, updated periodically, of all relevant randomised controlled trials’ [4]. That is why researchers in the field of healthcare have been working on a program of systematic reviews on the efficacy of therapies starting in the 1980s. In order to collect, assess, and promote research information, the Cochrane Collaboration was established in 1993. Since then, an extensive set of guidelines for conducting systematic reviews has been produced [5]. Other organizations have also joined this effort to convert the knowledge gained by health experts into practice, the main aim being to assist evidence-based medicine (EBM) practitioners in decision-making [6]. Nowadays, systematic literature reviews (SLRs) and meta-analyses are often placed at the top of the evidence hierarchy, usually depicted as a pyramid, ordered by the design and risk of bias of included studies [7]. In contrast to narrative reviews, systematic reviews address a specific research question [8]. This includes collecting all primary research applicable to the established review question and critically evaluating and synthesizing the data [9]. There are a few stages of conducting an SLR. Defining the review question, establishing hypotheses, and coming up with a review title are all part of the first stage. Titles should ideally be succinct and descriptive, e.g. intervention for the population with a given condition. One should always a priori define inclusion and exclusion criteria (according to PICO: P – population, I – intervention, C – comparison, O – outcomes), and study type (i.e RCTs). The development of a search strategy is another key step in performing a good quality SLR. Searching typically involves using several electronic databases (such as MEDLINE, EMBASE, or Cochrane CENTRAL), but they can also include consulting article reference lists, manually scanning important journals (hand-searching), or speaking directly with experts and scholars [10]. Once all abstracts are found, the following step is their screening – the process of identifying articles for inclusion and removing duplicates [8]. Then, appropriate full-text articles are gathered. Data from selected studies are then extracted. Data analysis should be carried out after quality assessment. Alternatively, some of these methods may be streamlined or omitted to produce evidence in a resource-efficient manner, in a form of rapid review, which is less comprehensive than a traditional SLR [11]. The first phase of this procedure includes a straightforward descriptive review of each study, usually referred to as qualitative analysis. If it is possible to combine results from different studies, the second phase – quantitative analysis, or meta-analysis – can be performed [12]. If used appropriately, meta-analysis will increase the accuracy of estimates of treatment outcomes, reducing the likelihood of false positive or negative findings and possibly allowing for the earlier implementation of successful therapies [13]. The number of SLRs seems to be exploding over the years while no quantification of this phenomena has been described to the authors knowledge. The main objective of this paper was to evaluate the volume trends in SLRs of RCTs throughout the years.

Methods

To analyse the overall increase of the number of SLRs over the years, the broad search in PubMed was performed in May 2023, using the following search string: ‘randomised controlled’ OR ‘randomised clinical’ OR ‘randomized controlled’ OR ‘randomized clinical’ OR RCT*. Later, an appropriate filter was applied in order to retrieve only studies with an SLR design. The total numbers of retrieved studies stratified by publication years were exported into an Excel file.

To run a detailed analysis of the trends in RCT SLRs over the years, a rapid review was conducted in Medline database on a smaller, representative sample of references, and results were compared to the ones from PubMed, to analyse consistency between them, and check if the same trend in the number of published SLRs would be observed. Medline database was searched via Ovid in May 2023 using a highly focused search strategy: (systematic review* or systematic literature review*).ti AND randomi?ed controlled trial×.ti. The search results were then imported to EndNote 20 (Clarivate) program and analysed using the Eppi-Reviewer Web software [14,15]. A single screening of titles and abstracts was performed. Additionally, based on information provided in titles and abstracts, each paper was coded according to a specific ICD-10 code (depending on the analysed disease area or procedure, Table 1) or, if no ICD-10 code was applicable (for example, the SLR analysed healthy subjects), to ‘Treatments’ code, consisting of ‘Pain treatment’, ‘Anaesthesia’, ‘Supplements/diet’ and ‘Other treatments/interventions’ subcategories. When appropriate, two or more codes were selected. Data regarding the number of RCTs included in each analysed SLR was also retrieved and divided into six categories: 1–10, 11–50, 51–100, 101–200, >200, and not reported (in abstract/title, NR).

Table 1. Disease area according to ICD-10 classification.

Code	Title	Code	Title
A00–B99	Certain infectious and parasitic diseases	L00–L99	Diseases of the skin and subcutaneous tissue
C00–D48	Neoplasms	M00–M99	Diseases of the musculoskeletal system and connective tissue
D50–D89	Diseases of the blood and blood-forming organs and certain disorders involving the immune mechanism	N00–N99	Diseases of the genitourinary system
E00–E90	Endocrine, nutritional and metabolic diseases	O00–O99	Pregnancy, childbirth and the puerperium
F00–F99	Mental and behavioral disorders	P00–P96	Certain conditions originating in the perinatal period
G00–G99	Diseases of the nervous system	Q00–Q99	Congenital malformations, deformations and chromosomal abnormalities
H00–H59	Diseases of the eye and adnexa	R00–R99	Symptoms, signs and abnormal clinical and laboratory findings, not elsewhere classified
H60–H95	Diseases of the ear and mastoid process	S00–T98	Injury, poisoning and certain other consequences of external causes
I00–I99	Diseases of the circulatory system	V01–Y98	External causes of morbidity and mortality
J00–J99	Diseases of the respiratory system	Z00–Z99	Factors influencing health status and contact with health services
K00–K93	Diseases of the digestive system	U00–U99	Codes for special purposes

All SLRs analysing RCTs were included, without any restrictions on population, interventions, or outcomes. Protocols, commentaries, or errata were excluded. No restrictions on date or language were applied.

Results

Database analysis

The PubMed search for the SLR of RCTs yielded 86,765 results (Figure 1). The first identified article was issued in 1990 and compared the effects of corticosteroid administration to no corticosteroid treatment before preterm delivery based on data from 12 RCTs [16]. Later, another 10 articles were published in 1994, and since then, new articles were being released annually. The number of newly published SLRs gradually increased with each year: 100 articles were published in 1999, and 1000 in 2005. In recent years, a couple of thousands of new RCT SLRs were being published annually; what is more, 37% of all identified records were published since the year 2020 (n = 32,174). We observe an exponential growth of published RCT SLRs. The largest number of new records was observed in the year 2022, with more than 10,000 publications released. As presented in Figure 2, the number of published RCTs was also growing; however, it reached a maximum in 2014 and has been staying on a similar level nowadays.

Figure 1. The number of RCT SLRs published over the years. Source: PubMed (search run in May 2023).

Figure 2. The number of RCTs published over the years. Source: PubMed (search run in May 2023).

Rapid review

The highly targeted search conducted in Medline yielded 7,534 records. After deduplication, 7,465 titles and abstracts were analysed, from which 6,892 were included for further analyses (Figure 3). The oldest retrieved publications date back to 1994. The gradual increase in the number of published RCT SLRs was consistent with the one observed in the PubMed analysis: more than 200 articles were released in 2013, and more than 600 in 2019; there was an intensification of the increase during the last several years: from over 800 publications in 2020 to over 1,400 published in 2022. Therefore, it was assumed that the identified records provided a representative sample for further analysis.

Figure 3. Distribution of RCT SLRs over the years (search run in May 2023).

The distribution of all identified RCT SLRs by the evaluated disease area and the number of included RCTs is presented in Figure 4. Overall, the most frequently analysed area in the identified articles were diseases of the circulatory system (I00-I99, n = 750; 10.9% of included articles), such as heart failure or stroke, closely followed by endocrine, nutritional, and metabolic diseases (E00-E90, n = 734; 10.7%), mainly diabetes. Additionally, 7.8% of SLRs were focused on assessing the impact of various supplementations or diets (n = 535). The relatively low number of SLRs assessing the following disease areas were identified: congenital malformations, deformations and chromosomal abnormalities (Q00-Q99, n = 10), diseases of the ear and mastoid process (H60-H95, n = 21) and external causes of morbidity and mortality (V01-Y98, n = 26).

Figure 4. Number of RCT SLRs stratified by disease area and by the number of included RCTs.

It was shown that the majority of SLRs summarised data from a moderate number of RCTs (between 11 and 50, n = 3,194; 46.4%); furthermore, one-third of analysed reviews included a lower number of studies (between 1 and 10). Larger SLRs, including more than 51 trials, were fewer in number: 5.2% of them included between 51 and 100 trials (n = 361), and 1.9% – between 101 and 200 (n = 131); only 87 out of 6,892 identified reviews analysed more than 200 trials. Furthermore, in 6.9% of articles, the number of incorporated trials was not reported in the abstract (n = 474). This proportion was consistent throughout the years and in various disease areas.

Figure 5 depicts the distribution of studies according to disease area in three distinct periods: from 1994 to 2015 (A), from 2016 to 2019 (B) and from 2020 onward (C).

Figure 5. The number of RCT SLRs stratified by disease area, published in 3 distinct periods: from 1994 to 2015, from 2016 to 2019 and from 2020 to March of 2023.

A total of 1,624 identified RCT SLRs were published between 1994 and 2015. Diseases affecting the circulatory system (I00-I99; n = 176; 10.8%) and the musculoskeletal system and connective tissue (M00-M99, n = 138; 8.5%) were the main areas of focus. With 7.5% and 7.4% of studies, respectively, neoplasms (C00-D48, n = 121) and mental and behavioural disorders (F00-F99, n = 120) were also among the most commonly studied topics. Interestingly, in highest number of cases, studies could not be assigned to any specific ICD-10 code (n = 223; 13.7%).

In the publishing period between 2016 and 2019, 1,880 RCT SLRs were identified. One significant change from the 1994–2015 period is that reviews on endocrine, nutritional, and metabolic diseases (E00-E90, n = 232; 12.3%) outnumbered the SLRs focused on circulatory system diseases (I00-I99, n = 212; 11.3%). Additionally, the number of studies analysing the impact of supplementations or diets increased twofold.

Half of all analysed RCT SLRs were published in recent years (2020 to March 2023; n = 3,416). Similar to the previous interval, the most frequently analysed disease area was endocrine, nutritional, and metabolic diseases (n = 394; 11.5%), closely followed by the diseases of the circulatory system (n = 363, 10.6%) and of the musculoskeletal system and connective tissue (n = 305; 8.9%). The number of newly published studies analysing the impact of supplementations or diets further increased by twofold in comparison to the 2016–2019 period. In the recent years, threefold increase in number of trials concerning nervous system diseases (G00-G99) such as Alzheimer’s disease was observed. The first appearance of codes for special purposes (U00-U99; n = 80), related to the COVID-19 pandemic, is also worth noting.

Discussion

In recent years, evidence synthesis became more crucial than individual studies. It helps with comparing similar studies, combining their findings, and making evidence more accessible, as well as with the identification of the most cost-effective treatments and future research to be better designed. Back in 1995, the Cochrane Group released the Cochrane Database of Systematic Reviews (CDSR) which consisted of 50 reviews [17,18]. In comparison, the number of reviews in 2015 was above 6,000. In 1998, CDSR was made accessible on the internet. Out of 2,500 reviews released each year, 20% are written by Cochrane [17]. Overall, in 2010, approximately 75 trials and 11 systematic literature reviews were published every day [19]. However, the number of issued SLRs did not exceed the number of narrative, non-systematic reviews, which growth is even higher. There are also much more journals publishing them [19]. Furthermore, SLRs and trials are lower in the number of published works than case reports [19]. Interestingly, data shows that 95% of all articles and 98% of core clinical journals were produced by just 30 nations globally. By 2018, there was an increase in all publication types; however, the most significant increase could be noticed in terms of publications of meta-analyses from China, which was leading the chart (n = 4,659); the United States of America was leading in the case of systematic reviews (n = 3,654), clinical trials (n = 11,095) and RCTs (n = 7,953) [20].

With so many SLRs published nowadays, it is crucial to use trustworthy data. The quality of trials was defined in the literature as ‘the likelihood of the trial design to generate unbiased results’ [21]. The quality of individual included studies is affecting the quality of the entire SLR; therefore, a proper bias assessment is a crucial step and key component. This is especially relevant if the evidence of medical treatment effectiveness is inconclusive. There are many tools that help with performing a quality assessment of RCTs such as Jadad scale [8], or Risk of Bias tool for randomized trials 2.0 (RoB 2.0), which is the suggested method for evaluating bias of studies that are part of Cochrane Reviews [22]. Its structure is made out of five domains: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported results [22]. Just like in the case of RCTs, quality of SLRs varies. A Measurement Tool To Assess Systematic Reviews (AMSTAR) was published in 2007 to make it possible for health professionals and policymakers to quickly evaluate the quality of SLRs of interventional RCTs. However, due to some criticisms, such as being focused mainly on RCTs and considering articles written in languages other than English as ‘grey literature’, the second, current version was developed in 2017. AMSTAR-2 takes additionally non-RCTs into account for assessment with the goal to determine if the most crucial information is reported in SLRs [23–25]. CASP Systematic Review Checklist is also commonly used instrument recommended by World Health Organization and Cochrane as an approachable alternative for novice qualitative researchers [26]. It consists of ten questions, divided into three sections that help to determine if the results of the study are valid (Section A), what are those results (Section B) and if the results will help locally (Section C) [27]. Lastly, while not a quality assessment instrument, the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) could be used to improve the reporting of SLRs and meta-analyses; it is also a useful tool for critical appraisal of published SLRs [28].

COVID-19 pandemic shook the world in the beginning of 2020. The need for more information about this disease was high as there were many uncertainties. Already in April 2020, quick search yielded 6831 articles from a total of 1430 journals about COVID-19 [29]. The most frequent study designs were review articles (n = 202) and SLRs (n = 43). An average of almost 59 articles were published everyday [29]. The so-called ‘covidization’ emerged – until August of 2021, in general and internal medicine publications, COVID-19 received up to 79.3% of citations and was mentioned in 98 of the top 100 most-cited articles [30]. That trend is also noticeable in the number of SLRs containing keywords for COVID-19 published in PubMed since 2020 (Figure 6, search run in May 2023 - Appendix). However, according to a systematic analysis of SLRs on COVID-19 published in 2021, methodological quality of the reviews was poor: out of 243 assessed with AMSTAR-2, 12.3% had moderate quality, 25.9% had low quality, and 61.7% had critically low quality [31]. Those conclusions were confirmed by other authors. Abbott et al. conducted an analysis of early published COVID-19 SLRs and found that 88 out of 280 reviews being assessed met SLR criteria, and only 3 of them had moderate or high quality according to AMSTAR-2. Fifty-two of those SLRs have been completed within 3 weeks, and submission and publication process took 3 weeks in 50% of cases. Publications received high attention despite being of low quality [32]. It shows that studies reported as SLRs should not be considered as of high quality from the beginning; each of reader should analyze the methodology undertaken and consider its impact on the findings.

Figure 6. The number of SLRs on COVID-19 published since 2020. Source: PubMed (search run in May 2023).

The number of publications is increasing not only for RCT SLRs. Similar tendencies may be observed for different study designs. According to the performed analysis of PubMed data, the overall trend in publishing the SLRs of epidemiological studies is similar to the SLRs on RCTs: the quantity of publications from both study designs consistently grows, although RCT SLRs are more numerous (in 2022 RCTs: n = 10,061; epidemiological: n = 8,947) (Figure 7, search run in May 2023 - Appendix). Development of epidemiological studies may be a result of a recent interest in real-world evidence (RWE) and its increasing role in health-care decisions. Increasing digitisation of health records facilitating data analysis, as well as increasing focus on the importance of patient-reported outcomes supports this trend [33].

Figure 7. The number of epidemiological SLRs published over the years. Source: PubMed (search run in May 2023).

According to the Global Health Estimates published by the World Health Organization (WHO), covering the period between the year 2000 and 2019, non-communicable diseases (e.g.: chronic diseases such as heart disease, chronic respiratory disease, cancer or diabetes) made up 7 of the world’s top 10 causes of death [34,35]. Ischemic heart diseases were in the lead, accounting for 8.9 million deaths in 2019, while stroke was in second place, causing 11% of deaths [35]. This is consistent with the number of published RCT SLRs in the cardiovascular area and explains the interests that physicians and trialists across the world take in cardiovascular diseases. With such a high death rate caused by those illnesses it is essential to study all possible treatments or interventions that can allow to ease the suffering of many patients and hopefully extend their life expectancy and quality. In recent years, deaths from diabetes increased by 70% globally and represented the greatest percentage increase of all WHO regions [34,35], which was also noticeable in the SLRs’ trends. Another change from the previous years was the appearance of Alzheimer’s disease and other forms of dementia among the top 10 death causes worldwide [34], also discernible in our analysis.

The diseases in question were also impacting the quality of life and disability – heart diseases, diabetes, stroke, lung cancer and chronic obstructive pulmonary disease were collectively responsible for nearly 100 million additional healthy life-years lost in 2019 compared to the year 2000 [34]. According to both WHO [35] and the Global Burden of Disease published by The Institute for Health Metrics and Evaluation (IHME) [36], neonatal diseases were one of the leading causes of disability-adjusted life years (DALYs) in 2019; however, this was not captured in the current analysis, since there is no uniform ICD-10 code for this area (Figure 8).

Figure 8. The burden of disease by cause, measured in DALYs. Source: Institute for Health Metrics and Evaluation (IHME): GBD results [36].

The analysis identified several disease areas with a relatively low number of SLRs: congenital malformations, deformations and chromosomal abnormalities (Q00-Q99, n = 10), diseases of the ear and mastoid process (H60-H95, n = 21) and external causes of morbidity and mortality (V01-Y98, n = 26). In terms of congenital and chromosomal diseases, it could be assumed that few RCT studies are being conducted due to the low numbers of patients, as they are often rare diseases; additionally, treatment options are usually symptomatic, with limited options to treat the underlying cause, e.g. gene therapies. Treatment for diseases occurring due to external causes also tends to be mostly symptomatic; therefore, very few RCTs focused specifically on the cause itself (such as injury or poisoning) would be available. The insufficiency in the number of SLRs of RCTs on the diseases of the ear and mastoid process is in accordance with the WHO report on hearing from 2021, which underlines that there is a significant gap in services for ear and hearing worldwide – for instance, there is an 83% gap between need for and access to hearing aid use; the authors state that the reasons could include the lack of accurate information and stigmatizing mindsets surrounding ear diseases [37].

It is also worth to mention about an increasing number of SLRs reporting on food supplements and diets. Between 2010 and 2020 over 70,000 new articles on nutraceuticals became available in PubMed. COVID-19 pandemic led to even higher interest in dietary supplements in early 2020. Consumers were looking for additional protection from disease and it resulted in 44% increase in sales during the first wave of the pandemic in the US, relative to the same period in the previous year. Supplement sales in March 2020 increased by 63% and about 40–60% versus the same period in 2019 in the UK and France, respectively [38]. In some authors’ opinion, supplement market growth trends will not continue and should normalize to pre-pandemic values during the following years [39]. According to other sources, global supplement market is expected to grow, and the main factors influencing this trend are as follows: focusing on well-being, preventive healthcare and shifting from standard pharmaceuticals to supplements and diets, and the growing geriatric population [40].

Conclusions

The recognition of RCT SLRs’ usefulness for providing a synthetized unbiased information has led to increased volume of SLRs. RCT SLR publications are growing at an exponential speed. The rapid increase in the number of published RCT SLRs in the last 3 years is partly driven by the emergence of the COVID-19 pandemic. While SLR is considered as the gold standard to unequivocally address evidence, in the case of COVID-19 it was source of controversies and outcome divergence between studies. The most frequently evaluated diseases through RCT SLRs are aligned with leading causes of death and disability worldwide indicated in the reports published by WHO and IHME [35,36]. The emergence of food supplements and diets illustrate the increase interest for such interventions that may be considered at the frontier of lifestyle and medicine. Although SLR is recognized as the most rigorous way to perform review, the number of narrative review remains more important than SLR. It is interesting to notice that epidemiological SLRs are growing fast and are about to catch up the number of RCT SLRs. The development of real-world evidence to assess interventions, the larger access to historical databases, may have played a role in the development of epidemiological studies. SLR will continue to grow as the number of RCTs and epidemiological studies will grow making the need for unbiased summary increasingly important for supporting EBM.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data supporting the findings of this study are available within the article and its supplementary materials.

Supplementary Material

Supplemental data for this article can be accessed online at https://doi.org/10.1080/20016689.2023.2244305

Appendix

COVID-19 search strategy (“Systematic reviews” filter applied) run on 15.05.2023 in PubMed

(‘COVID-19’[Mesh] OR ‘SARS-CoV-2’[Mesh] OR ‘COVID-19 Vaccines’[Mesh] OR ‘COVID-19 Serological Testing’[Mesh] OR ‘COVID-19 Nucleic Acid Testing’[Mesh] OR ‘SARS-CoV-2 variants’ [Supplementary Concept] OR ‘COVID-19 drug treatment’ [Supplementary Concept] OR ‘COVID-19 serotherapy’ [Supplementary Concept] OR ‘2019-nCoV’ OR ‘2019nCoV’ OR ‘cov 2’ OR ‘COVID-19’ OR ‘sars coronavirus 2’ OR ‘sars cov 2’ OR ‘SARS-CoV-2’ OR ‘severe acute respiratory syndrome coronavirus 2’ OR ‘coronavirus 2’ OR ‘COVID 19’ OR ‘COVID-19’ OR ‘2019 ncov’ OR ‘2019nCoV’ OR ‘corona virus disease 2019’ OR ‘cov2’ OR ‘COVID-19’ OR ‘COVID19’ OR ‘nCov 2019’ OR ‘nCoV’ OR ‘new corona virus’ OR ‘new coronaviruses’ OR ‘novel corona virus’ OR ‘novel coronaviruses’ OR ‘sars coronavirus 2’ OR ‘SARS2’ OR ‘SARS-CoV-2’ OR ‘severe acute respiratory syndrome coronavirus 2’)

Epidemiologic studies search strategy (“Systematic reviews” filter applied) run on 15.05.2023 in PubMed

‘epidemiologic stud*’ OR ‘epidemiology’ OR ‘epidemiologic’ OR ‘epidemiological’ OR ‘epidemiol*’