Thank you for your interest in our study. Poor calibration often occurs in external validations, and this phenomenon has been described earlier. Its main reason might be the disparity of the patients’ subsets (Citation1). Nevertheless calibration and discrimination were defined as the international gold standard for any statistical analysis of scoring systems during the 2nd European Consensus Conference in Intensive Care Medicine in 1994 (Citation2). A poor calibration of a scoring system must lead to doubt of the discrimination statistics, no matter how good the results of this analysis might be.
As you correctly say, the Hosmer–Lemeshow calibration statistics are very sensitive to the size of the study population, with large numbers of patients resulting in unreliable calibration. This fact is applicable to study populations larger than 5,000 patients, which was indeed the case in our study. We have tested the SAPS 3 in another trial on 4,054 patients of the years 2007–2009. In this analysis the score achieved similar results to that published in our study. That is the reason why we have not shifted to an alternative approach in the calibration statistics, for example the method according to Grunkemeier et al., which analyses the observed/expected ratio. Furthermore a large population must not always affect the Hosmer–Lemeshow calibration statistics. Nashef et al. achieved good results with this method on 13,302 patients when they published the EuroSCORE (Citation3).
SAPS 3 is calibrated for hospital mortality and 28-day mortality only; nevertheless we decided to study the ICU mortality. Since we started our work in the field of scoring systems several years ago, this has been a recurring issue in congress discussions, presentations, and paper reviews. After long discussions within our study group, we decided to define our outcome of interest as ICU mortality rather than hospital mortality, which is used in the SAPS models. Hospital mortality is a reasonable study endpoint, and ICU mortality as primary outcome could be seen as a potential limitation. Nevertheless the main advantage of ICU mortality as a study endpoint is that it avoids any inaccuracies related to variations in ICU discharge patterns among institutions, because nobody would discharge an endangered patient prematurely to the normal ward. ICU mortality as endpoint also excludes unrelated deaths (e.g., accidental falls) after discharge.
We also think that hospital mortality as primary outcome bears a potential pitfall in our setting, because we are a large tertiary referral university hospital. A discharge of patients from our normal ward in the university hospital to a normal ward of a smaller hospital nearby is a common scenario. A referred patient would be excluded from the database and appear as survivor, yet he/she is still in hospital care. Obviously this does not mean that a discharged/referred patient should not be admitted to a hospital. Considering the definite hospital mortality as our primary endpoint is therefore misleading.
The 28-day mortality does not exist in the literature of cardiac surgery scoring systems. The study endpoint of most scoring systems in cardiac surgery patients is either ICU mortality or 30-day mortality. Furthermore we want to refer to a statement of Dr. S. Nashef, who answered the question of the right study endpoint clearly in a personal communication (23rd Annual Meeting of the European Association for Cardio-thoracic Surgery, Vienna, Austria, October 18–21, 2009). He said ICU mortality is the best and only reasonable endpoint to test an ICU score.
Our recently published study might have direct influence on the daily ICU routine because mortality predictions with SAPS models are not reliable, due to inadequate calibration of SAPS 2 with significant p-values and low results for SAPS 3 in discrimination statistics especially during the early postoperative phase. Statistical inaccuracy during the first 48 h might have severe implications for the ongoing treatment of a recently operated patient.
The observation of a higher mortality rate in patients that are not discharged from ICU after 48 h is beyond doubt. A prolonged ICU stay stands as a surrogate for morbidity, and is a strong predictor of mortality (Citation4). Silberman et al. postulated a proportional increase of negative outcome as a consequence of prolonged ICU stay (Citation4).
In the logistic CASUS the rising influence of a prolonged ICU stay on mortality is captured by the increasing β-coefficients of the ‘ICU-day’ variable (Citation5). This might be the reason why SAPS 3, which includes cardiac surgery, patients, showed inferior statistical results in comparison to the logistic CASUS, a score that is specifically made for this patient subgroup (Citation5).
Declaration of interest: The authors report no declarations of interest. The authors alone are responsible for the content and writing of the paper.
References
- Le Gall JR, Neumann A, Hemery F, Bleriot JP, Fulgencio JP, Garrigues B, et al. Mortality prediction using SAPS II: an update for French intensive care units. Crit Care. 2005;9: 645–52.
- 2nd European Consensus Conference in Intensive Care Medicine. Predicting outcome in ICU patients. Intensive Care Med. 1994;20:390–7.
- Nashef SA, Roques F, Michel P, Gauducheau E, Lemeshow S, Salamon R. European system for cardiac operative risk evaluation (EuroSCORE). Eur J Cardiothorac Surg. 1999;16:9–13.
- Silberman S, Bitran D, Fink D, Tauber R, Merin O. Very prolonged stay in the intensive care unit after cardiac operations: early results and late survival. Ann Thorac Surg. 2013;96:15–22.
- Doerr F, Badreldin AM, Bender EM, Heldwein MB, Lehmann T, Bayer O, et al. Outcome prediction in cardiac surgery: the first logistic scoring model for cardiac surgical intensive care patients. Minerva Anestesiol. 2012;78:879–86.