Are observational data on therapeutic outcome trustworthy?

Abstract

Randomized controlled trials have become the golden standard to assess the effectiveness of a therapeutic action. Observational studies can be used to monitor effects of routinely applied therapeutic procedures. The validity of such studies can always be questioned and the results must be assessed in light of possible sources of bias. This issue is exemplified by the analysis of more than 10,000 patients admitted for coronary artery by-pass surgery in Gothenburg early in this century.

Today the double-blinded randomized controlled trial (RCT) is seen as the most reliable method for “representing things as they really are” (1). The history of the RCTs is relatively short, even if physical and other experiments go back a long time. The most widely known medical trial with clear nutrition overtones is the one recorded in the Old Testament, where King Nebukadnezar is confronted with the results of a 10-day intervention trial on the effect of a prudent diet consisting of vegetables and water (2). The outcome was the appearance of young men meant to serve at the king's court. At the end of the 10 days they looked healthier and better nourished than any of the young men who ate the royal food. So the guard took away their choice food and the wine they were to drink and gave them vegetables instead. But more than 2000 years were to pass until the randomized controlled trial was established as means to assess therapeutic measures and preventive interventions. The first properly performed and reported trial was Austin Bradford Hill's study testing the pertussis vaccine in the 1940s (3).

Physical experiments are models in which all factors except those being tested are kept constant during the experimental period. This differs from the clinical trial, where it is impossible to keep all potential influences under control. The answer to this dilemma is to nullify the effects of the other variables by randomly distributing them across the different therapeutic strata, which was one of the major contributions to modern medical research developed by Sir Austin Bradford Hill (4). The procedure does not guarantee identical distribution of the effect determinants, but the tendency toward this increases with increasing number of observation units—the bigger the better. Having praised the RCT as the instrument of choice to assess therapeutic effect, the pertinent question is: do observational studies still have a place in evaluation of effectiveness and efficiency? The question is relevant as observational studies (i.e., nonexperimental) are inherently biased in spite of the researchers’ conscientious planning and meticulous analysis. And maybe more important, whether such bias would over- or underestimate the true effects of a particular treatment, cannot be foreseen.

It is on this background one must assess the article by Rexius et al. in the current issue of this journal (5).

The authors raise the question whether the 20-year-old truth that high risk coronary patients fare better with by-pass surgery than without, is still valid. If this were not the truth, two possibilities must be considered:

1. The effectiveness of alternative revascularization treatment avenues such as PCI in combination with best available pharmaceutical measures has improved the outcome beyond that of coronary by-pass surgery.

2. The “natural history” of coronary heart disease has changed, and the case fatality of untreated disease has declined.

The second hypothesis is for all practical purposes unresearchable today. No permission would be given to undertake a study of coronary heart disease patients where one group was to be left untreated.

The hypothesis that the authors therefore wanted to test was whether the risk of death is lower after bypass surgery than before the operation in high risk coronary patients based upon current treatment modalities by using available registries concerning admissions to by-pass surgery—in other words, observational data. The authors report on the outcome (death hazard) of a series of 10,657 consecutive patients admitted to by-pass surgery from 1 January 1995 to 31 July 2005. Clinical information on each of the patients was available and assessed as potential contributors to the death hazard functions. The major hypothesis to be assessed from these data was whether surgery affected the outcome when all other factors were taken into account.

The authors conclude that all variables reflecting angiographic severity of coronary lesions indicated lower risk of death after by-pass surgery. There is a clinically understandable consistency with increasing benefit, only underscoring that the triage and risk assessment were properly undertaken. Low risk patients on the other hand were less likely to gain due to the surgical mortality, albeit low, sufficiently large to result in more deaths in the operated group during the first 4 months. The “low risk” group of CHD patients comprised almost 20% of the total patients. Except for the absence of diabetes, left anterior descending stenosis or unstable angina, no further information was provided by the authors on these patients. One may raise the question whether this group should have been admitted to surgery in the first place.

But can we trust these results?

One major problem with case series is that one does not know the characteristics of the underlying population (the denominator) from which the cases are recruited. Will all potential coronary by-pass candidates in this population have the same probability for being diagnosed and submitted to surgery? Or is there a group of undiagnosed subjects out there, surviving or dying, but not included in the present data. If included, would it have made a difference?

Another question which may always be raised regarding observational data is whether there is a further selection mechanism that could influence these results, beyond that provided by the risk stratification?

The patients’ risk was assessed taking the known clinical factors into consideration, but we do not know whether factors other than those included in this assessment also influenced the risk categorization. Such factors may involve “subtle mechanism which contrary to best intentions may give rise to misleading results… and unconscious bias” as it was formulated almost 60 years ago at one of the first (the first?) conferences on therapy and the evaluation of new treatment (6). The argument is as valid today as it was then.

The likelihood that important and confounding factors unknown to us should be randomly distributed among these patients is small, and this does diminish the validity of the inferences drawn from the analyses. Still, the current report provides results which are plausible and understandable. They may also provide incitement for further research on the patients who do not seem to profit from surgical revascularization.

Declaration of interest: The author reports no conflicts of interest. The author alone is responsible for the content and writing of the article.