To the editor,
We have read with interest the article by Doerr et al (Citation1) analysing prognostics systems SAPS-II and SAPS-3 in cardiac surgery patients. Our group has also published some articles on this topic, focusing on SAPS-3 calibration for intensive care unit (ICU) patients (Citation2,Citation3). We have also studied the calibration of SAPS-3 in cardiac surgery patients. This study was presented at a Congress (Citation4) and the article is in the process of publication.
In this article, Doerr et al. concluded that “performance of both SAPS models was generally poor. In this subset of patients, neither scoring system is recommended”. Bad calibration often occurs in external validation studies. The Hosmer–Lemeshow test is very sensitive to a study population with large numbers of patients (Citation5). In this study, the number of patients studied was 5,207 and small differences between predicted mortality and observed mortality could be sufficient for the Hosmer–Lemeshow test to be statistically significant and to produce a poor calibration for SAPS-3. Doerr et al. studied ICU mortality but SAPS-3 is calibrated for both hospital mortality and 28-day mortality.
In our study mentioned above (Citation4), we studied 5,383 patients. Predicted hospital mortality by SAPS-3 was 10.47%, hospital mortality was 9.7% and ICU mortality was 8%. The differences between hospital mortality and predicted hospital mortality by SAPS-3 (general equation) were inferior to 1%, with the differences having almost no statistical significance but Hosmer–Lemeshow test (H) was 15.58 and p value less than 0.05. We think that SAPS 3 performance in postoperative cardiac surgery patients is acceptable in our study.
Hospital mortality, being 2% greater than ICU mortality, is quite normal and could explain high values for the Hosmer–Lemeshow test and the poor calibration. In their manuscript the authors did not mention hospital mortality or 28-day mortality or whether they had used another equation calibrated for ICU mortality.
Mean and standard deviations for SAPS-2 and SAPS-3 are not provided. Could the authors give us these results so that readers could have a better understanding of the results?
Another major task would be to evaluate the calibration of a prognostic model on other days, in addition to the first 24 hours in ICU. One published study about the APACHE III model (Citation6), which is generalisable for others’ prognostic scores, emphasises that the system is calibrated for the first day in ICU and prediction after the initial 24-h period requires a different calibration. It can be seen in Table 4 that after Days 2 or 3, patients who remain in ICU have a higher mortality rate because those with better outcomes will have been discharged.
We think that these issues will need to be properly clarified.
References
- Doerr F, Badreldin AM, Can F, Bayer O, Wahlehrs T, Hekmat K. SAPS 3 is not superior to SAPS 2 in cardiac surgery patients. Scand Cardiovasc J. 2014; 48:111–19.
- Lopez-Caler C, García-Delgado M, Carpio-Sanz J, Alvarez-Rodriguez J, Aguiar-Alonso E, Castillo-Lorente E, et al. External validation of the simplified Acute Physiology Score (SAPS 3) in Spain. Med Intensiva. 2014;38:288–98. doi:10.1016/j.medin.2013.06.003
- Rivera-Lopez R, Aguiar-Alonso E, Lopez-Caler C, Castillo-Lorente E, Garcia-Delgado M, Arias-Verdú MD, et al. Validation of SAPS-3 and APACHE III in Mediterranean area. Acta Medica Mediterr. 2014;30:183–89.
- Delgado-Amaya M, Curiel-Balsera E, Arias-Verdú MD, Castillo-Lorente E, Carrero-Gomez FJ, Aguayo-deHoyos E, et al. Complementarity of prognosis models SAPS 3 and EuroSCORE in cardiac surgery. Critical Care. 2013;17:S176.
- Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit Care Med. 2007, 35:2212–13.
- Knaus WA, Wagner DP, Draper EA, Zimmerman JE, Bergner M, Bastos PG, et al. The APACHE III prognostic system. Risk prediction of hospital mortality for critically ill hospitalized adults. Chest. 1991;100:1619–36.