Evidence synthesis in the time of COVID-19

A critical development in the area of evidence synthesis occurred in the late 1980s with the early work of Ian Chalmers and colleagues on synthesizing evidence related to pregnancy and birth, culminating in the establishment of the first Cochrane Centre in 1992. Prior to the development of reproducible methods for systematic reviews and meta-analysis, looking for advice about treatment for a patient meant searching for primary research reports or the latest narrative review of the area, which was likely to have been heavily influenced by the biases of the author. These days, doctors (and especially medical students!) expect that there is a systematic review and set of guidelines to cover every clinical question. A recent example in the area of midlife women’s health is a systematic review and meta-analysis on the use of testosterone in women (2019) [1,2] and the associated position statement, that was published simultaneously in four women’s health journals [3].

A perennial problem is that systematic reviews and guidelines become out of date because the review process is laborious and slow to complete and new evidence is accumulating all the time. A review of the National Institute for Health and Care Excellence (NICE) guidelines showed that 86% of guidelines were still up to date 3 years later and the median lifespan was 60 months [4]. In another study of the individual recommendations within sets of clinical guidelines from the Spanish National Health System, 92% were considered valid 1 year after development, reducing to 77% after 4 years [5]. These numbers don’t sound too bad, but the consequences of delivering sub-optimal, out-of-date care may be harmful for the patient and has the potential for cumulative inefficiencies and costs for the health-care system.

So how to solve the problem of guidelines becoming out of date? The answer is a ‘living’ approach to systematic reviews and guidelines that are updated as new evidence becomes available, rather than being updated at a pre-defined interval such as every 3 or more years.

In 2018, The Australian Living Evidence Consortium had a broad vision for creating living systematic reviews and prioritized areas such as diabetes and stroke. Their bold vision proposed using new methods such as text mining, machine learning and citizen science with the aim being to reduce the delay in updating guidelines by 75% and to reduce the person-time required to do this also by 75% [6].

And then along came COVID-19, quickly followed by a tsunami of publications about every aspect of COVID-19 and its management. Between January and June 2020, over 23,000 papers on COVID-19 were indexed on Web of Science and Scopus [7].

In March 2020, the National COVID-19 Clinical Evidence Task Force was established by the Australian Living Evidence Consortium and Cochrane Australia [8]. The Task Force includes 31 Australian or Australasian peak health professional organizations whose members are involved in caring for people with Covid-19. In order to develop and maintain up-to-date guidelines for clinicians managing COVID-19 patients, the aim of the Task Force was to review new publications and, where appropriate, incorporate new evidence into guidelines for management, on a weekly basis, a frequency that had not been attempted previously [8]. However, given the seriousness of the pandemic and the rate at which new studies were being published, a weekly cycle of review was deemed appropriate. By October of 2020, the team had gone through 24 cycles of weekly review and, as I write this, the current version of the guidelines is version 35.1 [9].

The structure for the organization of this process involves having specialized guideline panels in areas such as critical care, disease-modifying treatments and chemoprophylaxis, primary and chronic care, etc. Each panel is supported by an evidence team that identifies analyses and synthesizes the relevant evidence. Each guideline panel, with input from a consumer panel, then makes draft recommendations which go to the Guideline Leadership Group (GLG) for approval. The GLG then makes recommendations to the steering committee that includes representatives from all of the involved health professional bodies. The steering committee then endorses creation of, or changes to, management guidelines [8].

Each week the highest priority questions are chosen according to three criteria: the likely impact on patient outcomes, the proportion of the clinical population affected by the problem, and the degree of variation in current practice [8]. Areas currently covered by the guidelines are available on the website and they may well have changed between when I am writing this document and when you are reading it [9]. The guidelines currently cover areas you would expect including: the definition of disease severity, markers of clinical deterioration, disease-modifying treatments (an extensive list of individual treatments), chemoprophylaxis, respiratory support in adults and children, venous thromboembolism prophylaxis and others. It is important to avoid duplication of effort so that the work of the National COVID-19 Clinical Evidence Task Force is concentrated in areas where primary data are available but there is no systematic review, so a new review is started. Where a review might exist but it is not up to date, new studies are incorporated to update that review.

It might be of interest to readers of Climacteric that there is currently a consensus recommendation on the Australian guidelines website that clinicians consider ceasing oral menopausal hormone therapy (MHT) in women with severe or critical COVID-19. This guideline is based on consensus only, in that both oral menopausal hormone therapy (MHT) and severe COVID-19 are associated with increased risk of thromboembolic events. It is not based on direct evidence [10]. It is not known whether the prophylactic anticoagulation currently recommended for people with severe or critical COVID-19 would be adequate for women on MHT. The recommendation about oral MHT in the context of severe or critical COVID-19 may change as more information becomes available.

An issue raised in the process of reviewing and synthesizing study outcomes was that of a core outcome set. The development of the COVID-19 core outcomes set for people with COVID-19 disease has been described in detail [11]. There are three tiers of outcomes, including a critically important set for inclusion in all trials (mortality, respiratory failure, multi-organ failure, shortness of breath and recovery) along with two other tiers of outcomes for inclusion, as appropriate. The development of these core outcomes was based on the COMET (Core Outcome Measures in Effectiveness Trials) initiative. The COMET initiative was launched in 2010 [12] and, of relevance to readers of Climacteric, the aspect of COMET relevant to women at midlife is the COMMA (Core Outcomes in Menopause) initiative [13]. Recent work of the COMMA initiative has seen the development of core outcomes for trials evaluating treatments for both vasomotor and genitourinary symptoms of menopause [14], a development which should ensure greater efficiency in the future evaluation of interventions for these conditions.

Factors identified as being critical to the success of the work of the National COVID-19 Clinical Evidence Task Force include having a highly experienced team of methodologists, access to both pre-appraised evidence as well as the results of new searches for primary studies, the use of GRADE (Grading of Recommendations Assessment, Development and Evaluation) [15] as well as the online guideline platform MAGIC (making GRADE the irresistible choice) [16] and the support of clinicians who made work on the guideline panels a weekly priority.

Clearly groups other than the Australian National COVID-19 Clinical Evidence Task Force have been working on living systematic reviews and guidelines related to the management of COVID-19 including the LIVING project [17], COVID-NMA [18] and epistemonikos [19].

Nothing will be left unchanged by the COVID-19 pandemic. Along with devastating loss of life and health-care systems pushed to the limit, it has provided great challenges for human ingenuity. Who, in January 2020 would have predicted that, only 12 months later, we would be vaccinating people against the disease and be confident that we have armed our frontline clinicians with the best available evidence, updated weekly, to make decisions about their patients?

Potential conflict of interest

Professor Bell has no relevant conflicts of interest.

Additional information

Funding

Nil.