Summarizing evidence: How far should we go?

In this issue of The European Journal of General Practice, the connecting thread is undoubtedly quality improvement. The paper by Addink et al. evaluates a scheme with financial incentives for improving access to primary care practice in the UK. A Swedish study by Björkelund et al., investigates whether electronic patient records are sufficiently accurate to evaluate and improve daily practice. The study of Borrell et al., looks at the quality of the physical examination by studying how well GPs perform in recognizing an important clinical sign such as hepatomegaly. Overbeck et al., wanted to know how patients with chronic Hepatitis C are treated in Swiss general practices and whether there is room for improvement. Jan-Joost Rethans comment on the importance of communication, especially motivation, in medical education and research. They discuss how interventions aimed at improving the quality of care can best be studied. The study of Van der Wouden et al., explores a large general practice database to look whether children with asthma are adequately treated. An interesting clinical lesson advises us how to avoid complications while administering intra-articular corticosteroids in the knee joint for osteo-arthritis.

A final paper by Ernst et al., studies the effectiveness of acupuncture for insomnia. This is—just like the other papers—a very relevant topic for the general practitioner as insomnia is a common problem, especially in the elderly and acupuncture is a non-pharmacological treatment, which avoids the disadvantages of pharmacological interventions such as benzodiazepines or related products. Moreover, there are at the moment no non-pharmacological alternatives of which the effectiveness is clearly demonstrated (1).

What makes this study also interesting is the method the authors used to answer their research question: they performed a ‘review of reviews’. While most of us are quite familiar with systematic literature reviews and meta-analyses as the highest level of evidence, a review or overview of reviews is a relatively new way of summarizing evidence. It is in fact the logical next step, allowing the findings of separate reviews to be compared and contrasted (2).

However, it does not necessarily make things more clear. In 1997 Jadad et al., (3) already predicted that it would become common to find more than one systematic review addressing the same or a very similar therapeutic question and that conflicts among reviews would arise. Systematic reviews appraise critically, summarize and attempt to reconcile the published evidence concerning a particular problem. Their main purpose is to solve the problem of having to base a clinical decision on primary studies with diverging results. However, with the emergence of a plethora of published reviews (4) disagreement is again often the case and the same difficulties arise: discordant reviews will confuse rather than clarify (3).

Why do reviews disagree? Roughly there are two main reasons: the results can diverge, or the interpretations and inferences made by the review authors can be discordant (3). The latter is perhaps not that much of a problem. After all, the clinician can look at the results for himself and draw his own conclusions. Yet, it becomes more difficult when results diverge. There are numerous reasons why this could happen: the research question can be slightly different, different trials can be included because other selection criteria or another search strategy were used, the quality of included trials can be assessed in different ways, or data can be extracted differently or a different method was used to synthesize results. The biggest threat for any systematic review is missing several studies due to publication bias or incomplete searches (5).

Can a review of reviews offer a solution for divergence in the same way systematic reviews attempt to do this for diverging primary studies? Probably not.

A recent publication (2) and the Cochrane Handbook (6) describe how ‘an overview of reviews’ should be performed. It will be no surprise that the different steps are quite similar to those taken in systematic reviews of individual studies: searching in all the possible sources, selection of the reviews based on predefined criteria, quality assessment using validated instruments, etc. However, the Cochrane Handbook (6) also specifies when such an overview can be of use: it can serve to give an overview of different interventions for the same condition or problem, to review the same intervention in the same condition for different outcomes, to review the same intervention in different problems or different populations or to summarize adverse effect in different reviews studying the same intervention. The key word here is ‘different.’ Simply combining reviews on the same intervention, condition or problem and the same outcome can provide a general overview of an area, but has no added value to a well-performed and upto-date systematic review. Divergent results have to do with choices made by the authors and/or the applied methodology. The best review to answer a clinical question is the review of highest quality with a research questions closest to the clinical question, rather than the ‘mean’ result of several reviews of different quality and scope.

The paper by Ernst et al., illustrates this very well. The authors found several systematic reviews on the same intervention and condition with quite divergent results. Nevertheless, as they point out: ‘this can be explained by doubt about the quality of the primary data. Systematic reviews make little sense if they do not critically evaluate the quality and reliability of the primary studies’ and in the light of this, they cannot draw a conclusion from their overview. In every review, quality flaws are possible on every level. The quality of a review influences its conclusions (7), which in turn underpin our clinical decisions. Adding extra levels—such as in reviews of reviews—will make it very hard to check the foundation on which the conclusions are based. Therefore, we should be very careful to use them.