Caustics and steroids: A case of Simpson's Paradox
To the Editor:
The recent article by Fulton and Hoffman provides an excellent example of Simpson's Paradox (Citation1,Citation2). Simpson's Paradox states that “It is not necessarily true that averaging the averages of different populations gives the average of the combined population” (Citation3). In fact, comparing the average of the combined populations and the average of the averages of the individual populations can lead to conclusions which are quite the opposite.
For example, a comparison of Derek Jeter's batting average and David Justice's batting average during 1995 and 1996 seasons reveals that David Justice had a better batting average each year (see ). However, if all the at-bats in both years were pooled for the two years, then Derek Jeter would have the better batting average. Conversely, if the mean batting average for the two seasons is compared, then David Justice has the better average (Citation4).
Table 1. A comparison of Derek Jeter's batting average and David Justice's batting average during the 1995 and 1996 seasons illustrating Simpson's paradox prepared from reference (Citation4)
To demonstrate this paradox in the case of steroids and second degree caustic esophageal injuries, we can use the author's data to create an analysis that compares the average of the combined population (the rate of stricture formation from the pooled data) to the average of the averages of the different populations (the mean of the rate of stricture formation from each of the studies) (see ).
Table 2. Comparison of stricture formation rates using pooled data and mean data extracted from Table 1 of ref. (Citation1)
If the two populations are compared as the author have, that is by pooling the data without a weighting variable or statistical treatment, then the rate of stricture formation without steroids (19%) is greater than the rate of stricture formation with steroids (12%). This would appear to support the use of steroids. Paradoxically, if we compare the arithmetic means of the rates of stricture formation from each of the individual studies, the mean rate of stricture formation without steroids is 15% and the mean rate of stricture formation with steroids is 20% - leading to quite the opposite conclusion. Neither of these approaches is statistically valid because studies cannot be combined without a weighting process such as meta-analysis. Furthermore, because only three of the studies have a control group, a strict meta-analysis can only include those three studies. One prior meta-analysis of those three studies found when steroids were used to prevent stricture formation, despite the pooled analysis suggesting a possibly protective effect. (Citation5) also found steroids offered no protection. Another study that attempted to pool data (Citation6).
Since prior meta-analysis of the three investigations with both a control and a treatment group failed to support the use of steroids in second degree esophageal burns, the authors conclusions that the existing data fail to support the use of steroids is reasonable. However, the pooled data appear to demonstrate a benefit to steroids where the mean data appear to demonstrate harm from steroids and this is an example of Simpson's Paradox.
References
- Fulton JA, Hoffman RS. Steroids in second degree caustic burns of the esophagus: A systematic pooled analysis of fifty years of human data: 1956–2006. Clinical Toxicology 2007; 45: 402–408
- Simpson EH. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society 1951; 13: 238–241
- http://mathworld.wolfram.com/SimpsonsParadox.html, Weisstein, Eric W. “Simpson's Paradox.” From MathWorld—A Wolfram Web Resource.
- Ross K. A Mathematician at the Ballpark: Odds and Probabilities for Baseball Fans. Pi Press 2004; 12–13
- LoVecchio F, Hamilton R, Sturman K, Hoffman RS. A meta-analysis of the use of steroids in the prevention of stricture formation from second degree caustic burns of the esophagus (abstract). J Toxicol-Clin Toxicol 1996; 34: 579–580
- Pelclova´ D, Navra´til T. Corrosive Ingestion: The Evidence Base. Are Steroids Still Indicated in Second and Third-Degree Corrosive Burns of the Oesophagus?. J Tox Clin Tox 2004; 42: 395–564