SAPS 3 is not superior to SAPS 2 in cardiac surgery patients

Abstract

Objectives. Cardiac surgery patients are excluded from SAPS2 but included in SAPS3. Neither score is evaluated for this exclusive population; however, they are used daily. We hypothesized that SAPS3 may be superior to SAPS2 in outcome prediction in cardiac surgery patients. Design. All consecutive patients undergoing cardiac surgery between January 2007 and December 2010 were included in our prospective study. Both models were tested with calibration and discrimination statistics. We compared the AUC of the ROC curves by DeLong's method and calculated OCC values. Results. A total of 5207 patients with mean age of 67.2 ± 10.9 years were admitted to the ICU. The mean length of ICU stay was 4.6 ± 7.0 days and the ICU mortality was 5.9%. The two tested models had acceptable discriminatory power (AUC: SAPS2: 0.777–0.875; SAPS3: 0.757–893). SAPS3 had a low AUC and poor calibration on admission day. SAPS2 had poor calibration on Days 1–6 and 8. Conclusions. Despite including cardiac surgery patients, SAPS3 was not superior to SAPS2 in our analysis. In this large cohort of ICU cardiac surgery patients, performance of both SAPS models was generally poor. In this subset of patients, neither scoring system is recommended.

Key words::

• cardiac surgery
• mortality outcome prediction
• postoperative scoring system
• SAPS

Introduction

Today we have several scoring systems to estimate the probability of mortality for intensive care unit (ICU) patients. These scores can be used to stratify resource utilization and to control the improvement in quality of an institution. Regarding the economical aspect of medicine, an objective outcome prediction is important (1).

The current ICU systems were developed several decades ago. In 1993, Le Gall et al. published the Simplified Acute Physiology Score (SAPS) 2 (2) (Table I). This system showed a lack of prognostic performance especially in calibration analysis (3,4). To overcome this problem the study group for SAPS3 (Table II) collected data about risk factors and outcomes in a heterogeneous cohort of ICU patients. Their objective was to develop a new, improved model for risk adjustment that considers the results of previous studies and incorporates missing variables which are relevant for accurate outcome prediction (3,4).

Table I. SAPS 2.

Table II. SAPS 3.

Although cardiac surgery patients were excluded from the SAPS2 database this score is among the most widely used systems in cardiac ICUs. Cardiac surgery patients represent a difficult population for postoperative scoring systems. They show temporary pathophysiological consequences related to the heart–lung machine, which can influence the different variables used in postoperative scoring systems (5). When Metnitz et al. and Moreno et al. developed the SAPS3 in 2005 on the basis of a worldwide cohort, postoperative cardiac surgery patients were included in the population (3,4).

The first external validation of SAPS3 in surgical ICU patients was performed by Sakr et al. in 2008 (6). This validation was carried out in a mixed-case cohort of 1851 surgical patients, including 26% of cardiac surgery cases. In 2011 we postulated SAPS2 to be a poor prognostic tool in cardiac surgery (7). The performance of SAPS3 has not been validated in a purely cardiac surgery ICU population so far. Since the SAPS3 development database consideres cardiac surgery patients, we hypothesized that SAPS3 would be superior to SAPS2 in this study of cardiac surgery patient. The aim of this study was to evaluate SAPS3 in 5207 cardiac surgery patients and to compare its predictive ability to that of SAPS2.

Material and methods

General information

The study was based on an evaluation of prospectively collected data of all consecutive adult patients admitted to our ICU after cardiac surgery between 1 January 2007 and 31 December 2010. It was approved by the Institutional Review Board of our university (approval number: 2809-05/10). We considered only the first admission for patients who were readmitted to the ICU during the study period. Demographic data as well as the routinely calculated preoperative EuroSCORE were collected from our quality control system QIMS 2.5 (University Hospital of Muenster, Germany). All laboratory data were extracted from our intensive care information system COPRA 5.24 (COPRASYSTEM GmbH, Sasbachwalden, Germany), which is interfaced with the patient monitors (Philips IntelliVue MP70, Amsterdam, the Netherlands), ventilators (Draeger Evita IV, Luebeck, Germany and Hamilton Galileo, Bonaduz, Swizerland), blood gas analysis devices (ABL 800Flex Radiometer, Copenhagen, Denmark), and the central laboratories. The attending physician collected the data for the entire population. Two assigned medical clerks validated the data collection daily. A senior consultant performed a second periodical validation. Inconsistency between the evaluators was resolved by consensus. The outcome was defined as ICU mortality.

Statistical analyses

The statistical analyses were performed using SPSS (SPSS Inc, Chicago, IL, USA) and SAS (SAS Institute Inc, Cary, NC, USA). Continuous scale data are presented as mean ± standard deviation. A value of p < 0.05 was considered significant.

Discrimination (the ability of a scoring model to differentiate between survival and mortality) was evaluated with receiver operating characteristic (ROC) curves, which plot the sensitivity (true positive) against 1 – specificity (false positive). Comparison of ROC curves was performed using the method of DeLong. The area under the curve (AUC) indicates the discriminative ability of parameters, which means the ability to discriminate survivors from non-survivors. An AUC of 0.5 (a diagonal line) is equivalent to random chance, whereas an AUC of 1.0 implies perfect discrimination. Overall correct classification (OCC) (the ratio of correctly predicted survival and mortality to the total number of patients) values were calculated. Calibration (correlation of the estimated probability of mortality and the actual observed mortality) was analyzed according to the Hosmer and Lemeshow goodness of fit test (HL). A high ‘Chi²’- value indicates a poor calibration especially with a related p value < 0.05.

Daily risk scoring was supported by previous studies to monitor physiological dysfunction and to allow prognoses and therapeutic decisions to be discussed or reconsidered (7). Statistical analyses were performed from ICU Day 1 (operative day) (n = 5207) to Day 8 (n = 601) only, in order to obtain accurate statistical results and to avoid a small number of patients. A precise overview with the number of patients at risk on each day can be found in the first column of Table IV.

Table III. Patients’ data and operations during the study period.

Age	67.2 ± 10.9 years
Length of ICU stay	4.6 ± 7.0 days
Additive EuroSCORE	6.3 ± 3.7
Logistic EuroSCORE	9.9 ± 12.9
Category	Number	%
Male	3249	62.4
Female	1958	37.6
ICU mortality	305	5.9
CABG	2755	52.9
Isolated valve surgery	1187	22.8
Combined CABG and valve surgery	692	13.3
Aortic surgery	323	6.2
Cardiac transplantation	52	1.0
Others (including congenital heart defect, tumors, pulmonary embolectomy, assist device implantation)	198	3.8
Total operations	5207

Proportion of length of ICU stay and the respective mortality
Days on ICU	Total number of patients	Survivors in %	Non-survivors in %
1 or 2	2824	97.8	2.2
3–5	1482	93.5	6.5
6–8	388	90.7	9.3
9–12	156	80.8	19.2
≥ 13	357	77.3	22.7

CABG, coronary artery bypass grafting; ICU, intensive care unit.

Table IV. Statistical results.

Day (Number of patients at risk)	Scoring model	OCC^e %	ROC-analysis^g		DeLong p value^f	Calibr. HL test^c
			AUC^b	95%-CI^a		Chi²	p value
ICU-day^d 1 (5207)	SAPS^h 2	94.4	0.804	0.780–0.828	0.010	39.968	0.000
	SAPS 3	94.4	0.757	0.725–0.789		46.746	0.000
ICU-day 2 (5159)	SAPS 2	94.3	0.875	0.851–0.890	0.358	57.752	0.000
	SAPS 3	95.2	0.893	0.861–0.925		15.175	0.056
ICU-day 3 (2372)	SAPS 2	89.4	0.873	0.858–0.888	0.310	27.799	0.001
	SAPS 3	91.0	0.861	0.844–0.879		16.411	0.037
ICU-day 4 (1612)	SAPS 2	87.7	0.849	0.837–0.860	0.026	23.296	0.003
	SAPS 3	87.8	0.828	0.814–0.842		10.175	0.253
ICU-day 5 (1164)	SAPS 2	86.0	0.838	0.828–0.849	0.002	20.356	0.009
	SAPS 3	85.9	0.815	0.804–0.826		13.142	0.107
ICU-day 6 (889)	SAPS 2	84.2	0.809	0.800–0.818	0.008	16.759	0.033
	SAPS 3	84.5	0.825	0.817–0.834		2.860	0.943
ICU-day 7 (713)	SAPS 2	83.0	0.787	0.778–0.796	0.007	12.528	0.129
	SAPS 3	83.0	0.803	0.796–0.811		2.043	0.980
ICU-day 8 (601)	SAPS 2	78.9	0.777	0.769–0.784	0.140	23.055	0.003
	SAPS 3	82.7	0.784	0.777–0.792		4.366	0.823

Results of overall correct classification, discrimination (receiver operating characteristic), AUC comparison by DeLong's method, and calibration (HL test) for the two SAPS models from Day 1 until Day 8.

^a95%-CI: 95%-confidence interval.

^bAUC: area under ROC-curve.

^cCalibr. HL test: calibration Hosmer–Lemeshow test.

^dICU-day: intensive care unit day

^eOCC: overall correct classification

^fp value of DeLong's method, which compares the SAPS 2 and SAPS 3 models.

^gROC: receiver operating characteristic.

^hSAPS: Simplified Acute Physiology Score.

Results

Population characteristics

This study includes 5207 ICU patients admitted over a period of 4 years; 37.6% were female. The mean age was 67.2 ± 10.9 years. The mean length of ICU stay was 4.6 ± 7.0 days and ICU mortality was 5.9% (n = 305). The preoperative mean additive EuroSCORE was 6.3 ± 3.7 and mean logistic EuroSCORE was 9.9 ± 12.9. The types of surgical procedures, the patients’ data and the proportion of length of ICU stay and the respective mortality are shown in Table III.

Results of statistical analyses

Table IV summarizes the OCC, the discriminatory power, the comparison of the two models’ AUC curves by DeLong's method, and the calibration of both SAPS models. Both scores had acceptable discrimination on all days, with AUCs of ≥ 0.777 for SAPS2 and ≥ 0.757 for SAPS3. The best results of both scores were on Day 2 (SAPS2: 0.875; SAPS3: 0.893). SAPS3 discriminated the poorest on Day 1, whereas SAPS2 showed the poorest discrimination on Day 8. The DeLong analysis showed a significant p value in favor of the SAPS2 model on Days 1, 4, and 5, whereas SAPS3 was significantly superior on Days 6 and 7. For the ICU Days 2, 3, and 8 the differences between AUCs of both score models were not significant. In general, SAPS2 showed better AUC curves during the early postoperative phase. Figure 1 shows the ROC curves of both models on the first two postoperative days. The best OCC level for SAPS2 was on the first day, with 94.4%, and for SAPS3, the best OCC level was on the second day, with 95.2%. However, both models showed the minimum OCC value on Day 8 (SAPS2: 78.9%; SAPS3: 82.7%). The calibration of SAPS2 was inadequate, with significant p values in the HL test on Days 1–6 and 8. In contrast, SAPS3 showed a significant p value only on Days 1 and 3.

Figure 1. SAPS 2 and SAPS 3 on ICU Days 1 and 2.

We analyzed the AUC of both EuroSCORE models on the total population of 5207 patients. The preoperative additive EuroSCORE had an AUC of 0.734 (95%-confidence interval: 0.699–0.769) and the preoperative logistic EuroSCORE of 0.732 (95%-confidence interval: 0.697–0.767). No gender-based differences or missing data were present in this study.

Discussion

General information

ICU scoring systems found their place in critical care medicine. The report of an individual patient's severity of illness on the one side and the impact of ICU-related factors on patient outcome, such as ICU organization and management on the other side can be provided by a scoring system (3). The reliability of a scoring system can be assessed using calibration and discrimination tests, considered by the European Society of Intensive Care Medicine (ESICM) to be the best methods to validate scoring systems and prognostic parameters. It is known that no model can have perfect calibration and perfect discrimination at the same time. Focusing on an individual patient only, the discrimination is important; however, for clinical trials or for the comparison of ICUs, a good calibration is mandatory (6).

To our best knowledge, this is the first external validation of the global SAPS3 admission model on a large database of an independent population of 5207 cardiac surgery patients. We think it is necessary to inform the reader that our ICU was one of four German centers that contributed patient data to the original SAPS3 database.

SAPS2

SAPS2 was developed in 1993 (2) based on a European/North American database, which includes 13,152 patients. Cardiac surgery patients were originally excluded from the score's target. Nevertheless, the score is currently used in many cardiac surgery ICUs. SAPS2 focuses on data from the first 24 h after ICU admission (4). The score has been extensively validated in external studies in different patient cohorts during the last two decades. The general findings in these studies were good discrimination but poor calibration, and significant differences between observed and expected mortality rates (3,8–11). In the year 2011, we were able to confirm the poor calibration of SAPS2 in a purely cardiac surgical population (7). The performance of SAPS2 was tested in the original SAPS3 paper, where the score also showed acceptable discrimination but lack of calibration.

There are several potential reasons for these findings, such as user-dependent problems as well as patient-dependent and model-dependent problems (3). A main bias might be the old database of SAPS2 from the early 1990s. Since then, there have been changes in the prevalence of major diseases and in the availability and use of major diagnostic and therapeutic methods (4,12). All that leads to a poor calibration of the old model. Furthermore, the SAPS2 database includes only patients from Europe and North America, which limits the ability to represent the reality of intensive care medicine in the rest of the world (4).

SAPS3

SAPS3 was published in 2005 (3,4), in order to have a new, improved model for risk adjustment and to overcome the shortcomings related to different case mixes and lead-time bias of SAPS2. The development of SAPS3 was based on the largest prospective, epidemiological, multicenter, multinational study with a high-quality population of 16,784 patients consecutively admitted to 303 ICUs from 35 countries around the world. Heterogeneity is reflected in an ICU case mix that also includes cardiac surgery patients. SAPS3 considers only physiological data within the first hour after ICU admission. Beside a global model, several regional models, which were also derived from the original database during the developmental phase of SAPS3, were presented in 2005. The global model allows a broad comparison of worldwide ICUs, but that obviously has less relevance to local conditions. However, the different regional models focus on higher local accuracy, since they are related to the origin of the contributed data.

In the original SAPS3 manuscript, Moreno et al. reported good discrimination (AUC: 0.848) and satisfactory calibration (Chi²: 10.56; p value: 0.39) of the global model, but found a poor fitting of the Central and Western Europe ICUs in mortality prediction. Hence, it is not surprising that the SAPS3 calibration and discrimination set was shown to vary widely around the world (13). The SAPS3 28-day score was published by the official SAPS3 study group in 2008 as a new model that was derived from the original database to provide a specific long-term outcome prediction instrument (14).

External validations of SAPS2 and SAPS3

Several authors directly compared SAPS2 and SAPS3 in external validations. Soares and Salluh (15) validated both models in a study of 952 ICU cancer patients. The statistical analyses showed excellent discrimination for both models and acceptable calibration for SAPS3, but the SAPS2 calibration was poor. Ledoux et al. (16) reported an inadequate mortality prediction of the global SAPS3 in their mixed-case study of a Western European ICU, despite very good discriminative power, and concluded that SAPS3 was not significantly better than SAPS2. SAPS3 was first validated for mixed surgical patients by Sakr et al. (6). They compared in their study the performance of the two SAPS models in 1851 surgical ICU patients. The authors reported poor calibration for SAPS2 and SAPS3, while the discrimination was generally good for both models. Poole et al. (17) postulated in a recently published multicenter study in 3661 mixed-case ICU patients that both SAPS models were unreliable tools for hospital mortality prediction. In this study, SAPS3 showed poorer statistical results than SAPS2. These findings stand in total contradiction to another study that was published by Juneja et al. in 2012 (18). The authors compared the accuracy of ICU outcome prediction between general established ICU scoring systems and their recent counterparts. The results showed clearly that the younger generation scores were superior and had better accuracy.

Why is the calibration of SAPS3 and especially SAPS2 inaccurate?

Commonly, the inaccuracy of prognostic models is due to a possible disparity between the case mix of an external population and the original reference database. Several SAPS2 and SAPS3 variables might contribute to the limited predictive ability of the models in a cardiac surgery population. Cardiac surgery patients show temporary adverse effects and pathophysiological consequences related to the heart–lung machine, which can influence several variables in postoperative scoring systems, including SAPS2 and SAPS3, among others (5). However, most of these pathophysiological changes, such as changes in electrolyte metabolism are temporary and do not have much influence on the outcome of these patients (19). Furthermore, postoperative sedation limits the role of the Glasgow Coma Scale (GCS) as a prognostic parameter (20). The GCS is affected by sedation, anesthesia, and paralysis, and the calculation requires clinical evaluation, which may be biased by subjective interpretation (7). The patients’ age might also be an inconsistent variable, due to positive selection regarding the patients’ preoperative state and the surgeons’ experience. Nevertheless, it was recently verified that several variables of the two SAPS models were good predictors of outcome (21).

What compromises the predictive ability of SAPS3 during the first day in the ICU?

The decrease of a study population due to the discharge of uncomplicated cases is a common and known phenomenon in cardiac surgery (Table III). It is, therefore, important that an ICU score is reliable during the critical hours of the admission day to identify which patient is at higher risk (22). At risk patients can be further observed in the ICU and the discharge of an endangered patient to the general floor, which might lead to a higher mortality due to ICU readmission or prolonged hospital stay, can be avoided.

One of the main reasons for the moderate discriminatory performance of SAPS3 on the admission day could be due to the different weights of the score's variables. Though SAPS3 considers only post-ICU admission data of the first hour (Box 3 of Table II), the incorporation of these data improves the prognostic capacity of SAPS3 by only 27.5%, whereas 50% of the predictive power of SAPS3 is derived from pre-ICU admission data (Box 1 of Table II). This is generally in contrast to Knaus et al. (23) and to our observation in a cardiac surgery population during the last years (24,25). The major event in cardiac surgery patients is the operation itself. The small contribution of post-ICU admission data to the predictive power of SAPS3 might be a strong bias in a cardiac surgery population and could be the main weakness of SAPS3 during the critical early ICU period of cardiac surgery patients.

What conclusions could be drawn from this study?

Although cardiac surgery patients were excluded from the SAPS2 this score is among the most widely used systems in cardiac ICUs. We postulate, based on the results of this study, not to use this poor prognostic tool in a cardiac surgery population.

The SAPS3 model specifically includes postoperative cardiac surgery patients. This study is the worldwide first external validation of SAPS 3 in a purely cardiac surgery ICU population. Especially during the early time after ICU admission, the SPAS3 is inferior to SAPS2 in risk evaluation.

This knowledge may have direct influence on the cardiac surgical population and the daily ICU routine since mortality prediction with SAPS models is not reliable. The treating clinician in cardiac surgery ICU should question the use of SAPS2 or SAPS3 in clinical trials and the comparison of ICUs of different health institutions based on these scores.

Declaration of interest: The authors declare that they have neither financial nor non-financial competing interests.

The authors alone are responsible for the content and writing of the paper.