Appropriate use of information in therapeutic decision-making: reflections on indirect comparisons

Abstract

Although the statistical strength of direct comparative randomized controlled trials is generally acknowledged, the particular demands of therapeutic decision making will often require indirect comparisons to be made, based on pooled data from multiple trials. As for all post-hoc analyses, the process of indirect comparison runs the risk of introducing significant bias into the results and consequently a robust statistical approach is required, in order to minimise the risk. To address this problem, a range of different methodologies have been developed over the past twenty years, using both frequentist and Bayesian models. It is important to appreciate the strengths and limitations of these techniques: however, the technical complexities tend to make this type of analysis somewhat opaque to the non-specialist reader. In this article, we consider the use of a simple, non-specialist critical appraisal tool developed by ISPOR, which allows methodological and interpretive errors to be identified and flagged as potential sources of bias, even when the detailed statistical methodology is not well understood by the reader.

Keywords: :

• Bayesian mixed treatment comparison
• Meta-analyses
• Naïve indirect comparison
• Randomized controlled trials

Over the course of the past five decades, the principle that therapeutic decision-making in medicine should be determined by explicit, unbiased evidence of efficacy drawn from randomized controlled trials (RCTs) has become accepted wisdom throughout the world. The principle of evidence-based medicine underpins not only the management of individual patients, but increasingly population-level decisions on health care provision, where it sits closely alongside the discipline of health economics as part of the health technology appraisal (HTA) process. The use of data derived from published clinical trials in HTA is, however, fraught with difficulty and the potential for error or biased interpretation.

Generally, a health care purchaser will seek to compare a new treatment with existing options, in the context of the local population and health economy. It is rare for individual clinical trials to provide exactly the information needed to carry out this kind of clinical comparison – rarer still for all the necessary health economic data to be available from the same source. Inevitably, therefore, pooling and cross-comparison of multiple studies has become a central component of this type of analysis. The associated statistical methodology has evolved extremely rapidly over the past 20 years, with new approaches to ensuring that valid comparisons are made constantly emerging.

The problems that this presents for the non-specialist reader are considerable. Most clinicians can intuitively understand and accept the principles behind an RCT. The extension to simple meta-analysis may be considered reasonably accessible by most, although they may struggle to replicate the underlying statistical techniques. Trying to grasp the reality of a Bayesian mixed treatment comparison, on the other hand, can be a step too far for even for the statistically literate. Yet in order to carry out a reasoned and evidence-based assessment of competing treatment options, some level of understanding of the appropriate analytical strategy is essential.

Central to this challenge is how best to retain the statistical strengths of individual RCTs, when combining them to shed light on the answer to a question that was not originally posed in the study. This goes to the heart of the underlying principles of the randomized controlled trial. When first formulating the statistical concepts that are used within an RCT in the 1920s, the Cambridge statistician R.A. Fisher determined that, in order that the result from a sample could be extrapolated to a general population, it had to be assumed that differences between two different study results were determined by random rather than systematic errors1^,2. If, and only if this condition is satisfied, then tests of statistical significance can be considered valid. In order to achieve this, studies have to be designed such that starting conditions are as alike as possible (good matching of patient groups) and that each recruited patient is as likely to receive one treatment as another (randomization).

This principle presents us with significant problems when we attempt to combine studies in a meta-analysis. Patient populations are not necessarily homogenous between studies, and although patients will have been randomized within their own study, the same is not true if we simply pool all the results of the individual studies into a single putative über-study. To address this, methods have been evolved that allow us to retain the benefit of randomization within the component studies, while simultaneously combining the overall results3. We can even, to some extent, compensate for differences in the patient population characteristics between studies4, which would otherwise have violated Fisher’s principle of homogeneity. We must be aware, however, that each step we take down this path stretches the underlying assumptions further and we must never forget that all meta-analyses are, regardless of our statistical sophistication, post hoc analyses and therefore potentially subject to bias.

Once we move into the realm of indirect comparisons, the potential for error and bias increases dramatically. Where direct comparisons between the treatments of interest have not been carried out, but each treatment has been compared instead with a common third-party comparator, it is possible to envisage a scenario where the result of a trial comparing A vs B can be compared with the result of a trial comparing C vs B, with inference being drawn regarding a hypothetical comparison of A vs C. By directly extracting results from two separate studies and comparing these, however, the benefits of randomization are lost and the results are of no greater strength than those obtained from an observational cohort study5. This approach, which is known as naïve indirect comparison, is too crude to yield reliable inputs for an economic model, and is no longer considered acceptable for use in HTA6–8.

The simplest alternative approach is to carry out a weighted indirect treatment comparison, as described by Bucher et al.5. This approach is based on an assessment of the relative treatment effect (expressed as OR) for each included trial, with the results being weighted according to the variance of this parameter. The resulting estimate of the overall A vs C comparison therefore retains, to some extent, the benefit of the individual study randomization and tends to yield a more reliable result. The Bucher method, however, uses a fixed effect model, that is predicated on a central assumption that the relative treatment effect will be constant across all trials – an assumption that is rarely valid. There are a number of different approaches to allow this assumption to be circumvented, to some extent, by use of a random effects model or by a meta-regression approach, using either frequentist or Bayesian techniques. Even so, the precise method chosen can have a substantial impact, both on the point estimate of relative treatment effect and on the range for the confidence (or credibility) intervals9.

The situation becomes more complex still if further variables are introduced. If we are trying to compare more than two treatments, then the simple Bucher method is no longer applicable. In this circumstance we have to use instead a network meta-analytical approach. Once again, this can be either frequentist10 or Bayesian11, although the more complex the network becomes, the more likely it is that Bayesian techniques will be used. Finally, if results from direct comparisons for some or all of the treatments become available in addition to the indirect studies, a mixed treatment comparison must be used12. This is a specialized form of network meta-analysis, that will almost always be carried out using a Bayesian Monte Carlo chain approach.

It is beyond the scope of this editorial to tease apart the strengths and weaknesses of the different approaches to indirect comparisons – for those who are interested, there are a number of useful publications that can help with this6^,7^,9^,13^,14. What is important to grasp is that each indirect evidence set has its own individual challenges, with several potential approaches to meaningful analysis being available. Selecting the appropriate method for a specific situation is neither simple nor beyond dispute and requires expert advice, as it is essential that a clear and scientifically justifiable approach is used to make treatment comparisons, if the results are to have validity and credibility.

The reader of an indirect comparison is presented with something of a problem – the core question that needs answering is, as for any published research: ‘Did the authors put themselves in a position to answer the question that they set themselves?’. Given the novel and constantly changing environment in the field, this can be an extremely challenging question to answer. However, just as we have become accustomed to using CONSORT and PRISMA criteria to assist in our reading RCTs and standard meta-analysis, there are a number of tools available that allow us to apply the same approach to indirect comparisons. One that is very clearly written and easy to use has recently been published15 – this takes the form of a questionnaire that considers five distinct issues of importance to the reader: Evidence base, Analysis, Reporting, Interpretation and Conflict of interest. Based on 22 yes/no answers, an overall assessment of ‘Sufficient’ or ‘Insufficient’ credibility can be arrived at.

Whilst it is unlikely that a single negative score will invalidate an entire analysis, a structured approach such as this allows accumulated minor flaws that may impact on the validity of a comparison to be identified. This is particularly important, as although obviously fundamentally flawed analyses are now far less commonly seen in the literature than was the case five years ago14, more subtle errors that may slip past the unwary reader are still prevalent.

An example of this is seen in an indirect comparison of 63 studies evaluating the impact of various antihypertensive agents on the risk of renal disease or mortality in people with diabetes16. This provides an apparently compelling argument that ACE inhibitors should be used for preference in these patients, until one considers the paper in a structured way. It then becomes apparent that a number of factors exist that may impact on the conclusions: (a) an incomplete evidence base was used, with only pure diabetes studies included (Q1 on the ISPOR questionnaire); (b) a potentially significant treatment effect modifier (magnitude of blood pressure reduction) was not taken into account in the analysis (Q5). It may well be that the accuracy of the conclusions still stands, but the identification of these issues allows the reader to consider the results in the light of the potentially impaired validity and make a judgment accordingly.

Another example involving an economic model was published evaluating the relative cost effectiveness of two strong analgesics, recently published in this journal17. The economic model itself was robust and appeared to give valid conclusions, but derived some of its key efficacy inputs from an indirect comparison. Rather than combining studies using a method that retained the randomization benefit of the component trials, however, the authors used naïve parallel indirect comparisons, with no common comparator incorporated into the analysis. Q7 in the ISPOR questionnaire identifies this as a ‘fatal flaw’, that seriously undermines the credibility of the indirect comparison and thereby the conclusions of the entire economic model.

A third example relates to another study published in CMRO. This was a systematic review and mixed treatment comparison that set out to compare the efficacy and safety profiles of several different treatments for multiple sclerosis18. Although the methodology used came in for some robust criticism in the correspondence pages19, it appears to be a valid approach to answering the research question regarding comparative efficacy. Where the authors do seem to have slipped up, however, is that their conclusions include statements on relative safety, which was not, in fact, assessed in the indirect comparison (Q20 in the ISPOR questionnaire). Although overenthusiastic interpretation of study results is not unique to the field of indirect treatment comparison and does not invalidate the other conclusions, it is, of course, an essential issue for the reader to assess if they are to use study results to guide practice.

Conclusion

The way in which we apply statistical analytical techniques to the results of clinical trials has taken on levels of sophistication that were inconceivable at the time R.A. Fisher was defining the basic tenets of the approach 90 years ago, but the fundamental principles remain constant. It is vital to remember that the validity of the conclusions that we draw from treatment comparisons are only reliable if we retain the base assumptions that underlie the data we use to feed the analysis. As understanding advances, what was acceptable in the past no longer reaches the required standard. This is as true for indirect comparisons as it is for RCTs. Whether as authors or as readers, we all have responsibility as professionals to ensure that work we carry out continues to meet these changing standards, if we are to ensure that standards of patient care remain at the highest level.

Although, for many clinicians, understanding the detail of statistical techniques falls outside the area of their expertise, carrying out systematic appraisal of the resulting papers must now be considered an essential skill. The use of well thought out and validated questionnaires as part of this process helps to ensure that potential biases are identified, quantified and incorporated into an overall appraisal of evidence, reducing the risk of arriving at an inappropriate or unreliable conclusion.

Transparency

Declaration of funding

This editorial was not funded.

Declaration of financial/other relationships

J.D.B. has disclosed that he has no significant relationships with or financial interests in any commercial companies related to this study or article.

CMRO peer reviewers on this manuscript have no relevant financial or other relationships to disclose.