ABSTRACT
Background
In 2011, the European System for Cardiac Operative Risk (EuroSCORE) II was created as an improvement of the additive/logistic EuroSCORE for the prediction of mortality after cardiac surgery.
Objective
To validate EuroSCORE II in predicting the mortality of open cardiac surgery patients in Indonesia.
Methods
We performed a multi-center retrospective study of cardiac surgery patients from three participating centers (Dr. Sardjito Hospital, Kariadi Hospital, and Abdul Wahab Sjahranie Hospital) between January 1st, 2016, and December 31st, 2020. Discrimination and calibration tests were performed.
Results
The observed mortality rate was 9.5% (73 out of 767 patients). The median EuroSCORE II value was 1.13%. The area under the curve for EuroSCORE II was 0.71 (95% CI: 0.65–0.77), suggesting fair discriminatory power. Calibration analysis suggested that EuroSCORE II underestimated postoperative mortality. Gender, age, chronic pulmonary disease, limited mobility, NYHA, and critical pre-operative state were significant predictors of post-cardiac surgery mortality in our population.
Conclusion
This study suggested that the EuroSCORE II was a poor predictor for postoperative mortality in Indonesian patients who underwent cardiac surgery procedures. Therefore, EuroSCORE II may not be suitable for mortality risk prediction in Indonesian populations, and surgical planning should be decided on an individual basis.
1. Introduction
Open heart surgery procedures are showing an increasing trend in Indonesia. There was a 22.31% increase in open-heart surgery procedures between 2015 and 2016 [Citation1]. Cardiac surgery procedures have been considered one of the most expensive surgical procedures. Perioperative complications are not uncommon in cardiac surgery and can increase the cost significantly, especially in high-risk surgical procedures. Risk stratification may reduce the risk of complications in these patients. The rationale is, that when faced with a patient with a high risk of mortality, we should anticipate and make preparations to prevent complications and improve outcomes. Risk stratification is important in the management of cardiac surgery patients. Risk stratification may be associated with better surgical outcomes, and it also allows us to improve the quality of care and make comparisons of outcomes [Citation2].
Most scoring systems were developed in Western countries, such as the additive/logistic EuroSCORE, EuroSCORE II, Parsonnet, and STS, which can be considered as ‘Western’ risk scoring systems. On the contrary, no single prominent scoring system was developed specifically for large populations in Asia or Africa [Citation3]. For example, the Sino System for Coronary Operative Risk Evaluation (SinoSCORE), published in China, is not widespread outside China [Citation4,Citation5].
The EuroSCORE model is one of the most commonly used risk scoring systems in cardiac surgery. The original EuroSCORE was developed in 1999 and had the highest discriminatory power among risk models of its time and it worked well in predicting 30-day postoperative mortality of cardiac surgery patients in many European countries as well as in the United States [Citation6–8]. The EuroSCORE II was developed as an improvement to the original model’s performance [Citation9]. The application of the EuroSCORE system for populations outside Europeans should be used with caution. Although the physiological characteristics are similar among all cardiac surgery patients, there are different interactions and impacts of various risk factors on individuals in different populations. Interactions between risk factors and a patient’s genetic makeup may be varied between populations [Citation10].
There are still only a few studies that assess the validity or performance of the risk prediction system for cardiac surgery patients in Indonesia, especially from a large-scale multicenter population. Therefore, the purpose of this study was to assess the calibration and discrimination performance of EuroSCORE II in a multicenter population.
2. Methods
2.1. Study design
We conducted a multicenter retrospective study on cardiac surgery patients at three participating centers between January 1st, 2016, and December 31st, 2020. The participating centers in this study were Kariadi Hospital, Semarang; Abdul Wahab Sjahranie Hospital, Samarinda and Dr. Sardjito Hospital, Yogyakarta. Preoperative characteristics, surgery types, and outcomes were extracted from the medical records, and EuroSCORE II was calculated. Since the EuroSCORE II is applicable for all types of adult cardiac surgery, all adult patients who underwent open cardiac surgery procedures were eligible as subjects in this study.
2.2. Subjects
We retrieved medical records for all adult open cardiac surgery patients who underwent procedures with or without cardiopulmonary bypass (CPB) at the participating centers between January 1st, 2016, and December 31st, 2020. Patients under 18 years of age, pericardiotomy procedures, pacemaker surgery without CPB, thoracic aortic surgery, and transcatheter valve implantation, were excluded from the study.
2.3. Data collection
Data that corresponded with EuroSCORE II components and postoperative mortality outcomes were extracted from medical records. We used the same definition of in-hospital mortality applied in the EuroSCORE II model. A customized Android application was created for data collection. Data were then kept in a password-protected spreadsheet.
Data sent by individual centers were filtered for missing data and incoherencies. The entire dataset was reviewed and participating centers were asked to resend data when errors or missing data were detected. All data from every participating center were pooled in a single database and SPSS version 24 (IBM Corp., Armonk, NY, USA) was used for statistical analysis.
2.4. Statistical analysis
Continuous variables were presented as mean and standard deviation (SD), while categorical variables were presented as frequencies and proportions. All categorical variables were compared between groups using the chi-square test. Meanwhile, for continuous variables, an unpaired t-test was performed for variables with normal data distribution; otherwise, the Mann-Whitney U test was performed.
Discriminatory power and calibration tests were performed to assess the predictive performance of EuroSCORE II. An area under the receiver operating characteristics (ROC) curve was conducted to estimate the discriminatory performance of EuroSCORE II in predicting in-hospital mortality. Area under the curve (AUC) > 0.7 suggested fair discrimination while AUC > 0.8 suggested excellent discrimination. The calibration curve was produced and Hosmer–Lemeshow goodness-of-fit (H–L) test was performed to assess calibration.
3. Results
3.1. Patients’ characteristics
The demographics and pre-operative characteristics of subjects are shown in . A total of 847 cardiac surgery patients were identified. Of these, 80 subjects did not have complete data for EuroSCORE II calculation, thus we only included 767 subjects in the study. Subjects in our study had a mean age of 44.18 ± 14.10 years, and 50.5% of the subjects were female. Most of the subjects were NYHA class II (60.1%) and I (25.8%). The majority of the subjects (82.4%) had a good left ventricular function. Regarding surgery type, over half of the subjects had 1 non-coronary procedure (52.2%). Most subjects did not have any history of previous cardiac surgery (98.9%).
Table 1. Demographic and comorbidity analysis.
3.2. Observed and predicted in-hospital mortality
The actual in-hospital mortality rate was 9.5% (73 out of 767 patients), while the median predicted mortality rate was 1.13%. This result showed that the actual mortality rate was higher than the EuroSCORE II predicted. When EuroSCORE II results were divided into quartiles, observed mortality rates were 0.80% in the first quartile and 1.62% in the third quartile, as shown in .
Table 2. Observed and predicted mortality based on the risk category and surgery category.
3.3. Discrimination and calibration
The ROC curve of EuroSCORE II for the prediction of in-hospital mortality is presented in . The AUC of ROC curve was 0.71 (95% CI: 0.65–0.77, p = 0.000). This result suggested that the EuroSCORE II had fair or reasonable discriminatory power.
Figure 1. ROC Curve for EuroSCORE II.
We found a significant difference between predicted and observed mortality based on the calibration curve and Hosmer-Lemeshow (HL) goodness-of-fit test results. EuroScore II had a CITL value of 2.044 and a slope of 1.19 (). These values suggested that EuroScore II had poor calibration for our population. Cross-tabulation suggested that EuroSCORE II underestimated mortality in all decile risk groups as presented in .
Figure 2. Calibration curve of EuroSCORE II.
We also calculated the predicted and observed in-hospital mortality based on surgery types and predicted risk. Subjects were divided into isolated CABG, valve or congenital surgeries, and mixed procedure groups based on surgery types. EuroSCORE II underestimated mortality in all these subgroups. Based on the predicted risk, subjects were divided into very low risk (<1%), low risk (1%–2.99%), moderate risk (3%–4.99%), and high risk (>5%) groups. Likewise, EuroSCORE II underestimated mortality in all subgroups. The discriminatory power and calibration of the total sample and subgroups are presented in . The EuroSCORE II had fair calibration in all of the surgical subgroups. The values of the AUC of ROC curves were between 0.70 in the valve or congenital group and 0.75 in the mixed group.
Table 3. Calibration and discrimination of European system for cardiac operative risk evaluation.
4. Discussion
Advances in surgical and anesthesiology care aim to increase the surgery success rate and minimize mortality rates of the patients. Postoperative mortality is affected by at least three factors: preoperative, intraoperative, and postoperative factors. Preoperative factors refer to demographic patient data (age, gender, etc) and underlying medical condition before the operation, intraoperative factors refer to factors that occurred or were conducted during surgery, and postoperative factors refer to events or complications which occur after the surgery is conducted. The mortality risk prediction model is usually required in order to weigh the benefit and risk of the procedures to the patients, which will then influence the clinical decision making. It is also used as means of surgical risk communication to the patients and their families, as well as in resource allocation, quality control, and hospital audits [Citation3]. The model that was first developed for the international population was additive and logistic EuroSCORE in late 1990 which utilizes preoperative factors to predict postoperative mortality risks [Citation11].
Prediction models are beneficial in current cardiac surgical practice. However, it is necessary to validate these risk models with local populations. The original additive and logistic EuroSCORE models are now considered unsuitable for the prediction of in-hospital mortality in cardiac surgery [Citation12,Citation13]. These scoring systems overestimated the mortality risk and may have adverse consequences on the critical assessment of clinical outcomes that lead to a false sense of reassurance and may compromise patient outcomes [Citation14]. As the result, the EuroSCORE investigators then improved these scoring systems and published a revision. The EuroSCORE II was an updated and modified risk scoring method that was created to enhance and optimize its usefulness for the current cardiac surgical practice [Citation9].
The EuroSCORE II had fair discriminative power in our study. However, EuroSCORE II in our study exhibited miscalibration and underprediction. A study by Barili et al which validated EuroSCORE II in multicenter European (Italian) population showed that both predicted and observed in-hospital mortality was 3,9%, showing a good prediction [Citation3]. However, our study demonstrated a remarkable difference. The median predicted mortality rate of our study was 1.13% according to EuroSCORE II, while the observed mortality was 9.5%. The difference between observed and predicted mortality in our study was 8.37% and the Hosmer–Lemeshow test found poor calibration of EuroSCORE II.
There are several potential explanations for differences between observed and expected mortalities, i.e. variations of cardiovascular disease epidemiology between European and Asian populations, hypothetical mistakes in the methodology in the developmental or validation study, changes in case-mix, or differences in surgical activity for socioeconomic reasons, protocols, surgical technique, postoperative care, etc. Of these, special characteristics of the Indonesian cardiac patients and healthcare system are the most plausible explanations for the differences between EuroSCORE II-predicted and observed mortality in the current study [Citation10].
Our study found that cardiac surgical patients in Indonesia had relatively low-risk EuroSCORE profiles, younger age, low NYHA classification level, higher incidence of endocarditis, pulmonary hypertension, and were more likely to undergo elective and valve surgery than EuroSCORE II. CABG surgery accounts for only 19.6% of all procedures. The incidence of chronic lung disease in our population was less than half of the EuroSCORE II population.
Age was a significant predictor of mortality in the EuroSCORE II analysis. However, our dataset had a significantly lower mean age than EuroSCORE II. Females constituted 50.5% of our sample, a higher proportion than in the EuroSCORE II database (30.9%), while in another study by Borde et al. [Citation15], only 19.89% of the patients were female. Our dataset had fewer subjects with poor mobility compared to EuroSCORE II, but it was a significant predictor of mortality. In the Indonesian population, the incidence of endocarditis is higher than EuroSCORE II. The incidence of chronic lung disease was lower in our population than in EuroSCORE II, which may be attributed to the younger age of patients in our sample. Insulin-dependent diabetes mellitus (DM) was a risk factor in the EuroSCORE II database; however, it did not consider the DM duration, which may modify the effect of the disease on coronary arteries. It is common for early-onset DM in the Indonesian population. Based on our experience, many patients had a long duration of uncontrolled diabetes and diffuse disease of coronary arteries. These may affect the success of revascularization and postoperative outcomes [Citation16].
There were also concerns about whether populations included in EuroSCORE were representative of Asians, especially the Indonesian population. EuroSCORE II database was derived from 22,381 patients (16,828 as developmental set and 5553 as validation set) across 154 units in 43 countries. The participating countries were Argentina, Austria, Belarus, Belgium, Bosnia, Brazil, Canada, China, Croatia, Denmark, Finland, France, Germany, Greece, Holland, Hungary, India, Ireland, Israel, Italy, Japan, Lithuania, Montenegro, New Zealand, Norway, Poland, Portugal, Russia, Saudi Arabia, Serbia, Slovenia, South Africa, Spain, Sudan, Sweden, Switzerland, Syria, Taiwan, Turkey, UAE, UK, Uruguay, and the USA. Indonesia, a country located in Southeast Asia, was not among these 43 countries. There were no Southeast Asian countries included and there were only 8 Asian countries included (18,6%): 3 East Asian, 1 South Asian, and 4 Middle East Asian countries [Citation9]. Similar concerns about external validation have been reported from studies in Europe and Asia [Citation17–21]. In Indonesia, earlier studies have attempted to validate EuroSCORE II. One study was from Jakarta and another from Yogyakarta, but these were single-center studies. The present study, however, is more likely to be representative of the general Indonesian population as it was performed in multiple centers.
Further assessment was done by comparing epidemiology data and EuroSCORE II validation in other countries. India, one of the countries participating in the EuroSCORE database, turned out to have different demographic data than the EuroSCORE database and showed similarities to our study. Malik [M] found that the mean age of the patients undergoing open-heart surgery in India was 45 years (Europe: 60 years) and about 92% of the patients were less than 60 years of age (Europe: 30%). The same study reported that 60% of the study patients underwent valvular heart surgery contrary to only 30% of the European population. Most valvular heart surgeries in the European population were due to age-related degenerative valvular heart disease as opposed to rheumatic valvular heart disease in India. In India, rheumatic heart disease was still the leading cause of valvular heart disease, and the patients usually presented at a young age. Since the patients who underwent cardiac surgery were young, they were less likely to have risk factors mentioned in the EuroSCORE model. This was congruent with the findings of a lower prevalence of chronic obstructive pulmonary disease, extracardiac arteriopathy, elevated serum creatinine, critical preoperative state, recent myocardial infarction, and unstable angina [Citation22]. Valvular disease of rheumatic origin was also still prevalent in Indonesia. Besides elevated risk due to the medical condition, these patients were often from lower socioeconomic status, malnourished, and had limited access to healthcare; this socioeconomic background aggravated the risk even further.
There was a relatively higher incidence of pulmonary hypertension in our study (20,7%) as well as Indian study (21,93%) compared to the European study (2,0%). Pulmonary hypertension was observed in 21.93% of Indian patients as against only 2% of patients in Europe. Pulmonary hypertension alone increased mortality risk in valvular heart disease and was usually the leading pathophysiology in symptomatic valvular heart disease patients [Citation22]. Findings of a higher incidence of valvular heart surgery triggered another question on whether the higher-than-predicted mortality in the Indonesian population was only caused by the higher incidence of valvular heart surgery. Hence, research analysis on EuroSCORE II validation in CABG patients was conducted. A study in the Singapore population that compared EuroSCORE II, Society of Thoracic Score (STS), and Asian System for Cardiac Operative Risk Evaluation for predicting mortality after isolated CABG (ASCORE-C) found that EuroSCORE II was a poor discriminator of prognosis particularly amongst moderate and high-risk patients in the cohort with different Asian ethnic groups, particularly worse in 1-year mortality prediction (H-L test p < 0.05). The same study found that factors that significantly affected postoperative mortality were estimated glomerular renal filtration, congestive heart failure, aortic atherosclerosis, emergency, abnormal heart rhythm, critical preoperative status, peripheral vascular disease (PVD), Malay, age >65 years. Abnormal heart rhythm, congestive heart failure, and abnormal heart rhythm were not included in EuroSCORE II variables, suggesting other affecting variables were not recognized by the EuroSCORE II database due to epidemiology data differences [Citation23].
Concerning mismatching, existing risk models usually predict outcomes more accurately in the setting in which the risk model was originally developed. Socio-economic conditions, living standards, health resources, and geographic and ethnic characteristics influence the usefulness of risk models in different areas. The clinical profiles of patients differ between Asian and European people. The differences exist in patient demographics, patient severity at hospital visits for socio-economic, cultural, and geographic reasons, unequal distribution of medical facilities, and different patterns of therapy [Citation10].
These results instigated important clinical implications. They suggested that EuroSCORE II should not be used to guide clinical risk assessment and mortality risk stratification in cardiac surgery patients in Indonesia. Future studies should be performed to develop a local mortality risk model that is suitable for the Indonesian population [Citation3].
5. Conclusions
EuroSCORE II had a fair calibration in total cardiac surgery patients, but poor discrimination in CABG, valve, congenital and mixed procedures. It also underestimated the risk of in-hospital mortality. Therefore, EuroSCORE II may not be safely used for risk assessment of cardiac surgery in Indonesia, and clinical considerations should be applied on an individual basis.
Declaration of Interest
The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.
Ethics approval
Ethical approval was obtained from the Medical and Health Research Ethics Committee of the Faculty of Medicine, Public Health and Nursing, Universitas Gadjah Mada, Yogyakarta, Indonesia with Ref. No: KE/FK/1277/EC/2019.
Reviewer disclosures
Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.
Additional information
Funding
References
- Rumah Sakit Jantung dan Pembuluh Darah Harapan Kita. Laporan tahunan badan layanan umum rumah sakit jantung dan pembuluh darah harapan kita tahun 2016. [Internet]. Rumah Sakit Jantung dan Pembuluh Darah Harapan Kita; 2017 [cited 2017 December 20]; https://www.pjnhk.go.id/LAKIP_2016
- Kolh P. Importance of risk stratification models in cardiac surgery the opinions expressed in this article are not necessarily those of the Editors of the European heart journal or of the European society of cardiology. Eur Heart J. 2006;27:768–769.
- Hote M. Cardiac surgery risk scoring systems: in the quest for the best. Heart Asia. 2018;10:e011017.
- Zheng Z, Zhang L, Li X, et al. SinoSCORE: a logistically derived additive prediction model for post-coronary artery bypass grafting in-hospital mortality in a Chinese population. Front Med. 2013;7:477–485.
- Ma X, Wang Y, Shan L, et al. Validation of SinoSCORE for isolated CABG operation in East China. Scientific Reports [Internet]. 2017 [cited 2018 May 10]; 7. http://www.nature.com/articles/s41598-017-16925-x
- Nashef S. Coronary surgery in Europe: comparison of the national subsets of the European system for cardiac operative risk evaluation database. Eur J Cardiothorac Surg. 2000;17:396–399.
- Nashef SAM, Roques F, Hammill BG, Peterson ED, Michel P, Grover FL, Wyse RK, Ferguson TB, et al. Validation of European system for cardiac operative risk evaluation (EuroSCORE) in north American cardiac surgery. Eur J Cardiothorac Surg. 2002;22:101–105.
- Roques F, Nashef SA, Michel P, Gauducheau E, De Vincentiis C, Baudet E, Cortina J, David M, Faichney A, Gavrielle F, Gams E, et al. Risk factors and outcome in European cardiac surgery: analysis of the EuroSCORE multinational database of 19030 patients. Eur J Cardiothorac Surg. 1999;15(6):816–822. discussion 822-823.
- Nashef SAM, Roques F, Sharples LD, et al. EuroSCORE II. Eur J Cardiothorac Surg. 2012;41:734–744. discussion 744-745.
- Rabbani MS, Qadir I, Ahmed Y, et al. Heart valve surgery: Euroscore vs. EuroSCORE II vs. Soc Thorac Surg score Heart Int. 2014;9:53–58.
- Sanaiha Y, Benharash P. Cardiovascular Risk Assessment in Cardiac Surgery. Perioperative Med. [2022 Jan 1]; 46–56.
- Lebreton G, Merle S, Inamo J, et al. Limitations in the inter-observer reliability of EuroSCORE: what should change in EuroSCORE II? Eur J Cardiothorac Surg. 2011;40:1304–1308.
- Parolari A, Pesce LL, Trezzi M, et al. EuroSCORE performance in valve surgery: a meta-analysis. Ann Thorac Surg. 2010;89:787–793.e2.
- Choong CK, Sergeant P, Nashef SAM, et al. The EuroSCORE risk stratification system in the current era: how accurate is it and what should be done if it is inaccurate? Eur J Cardiothorac Surg. 2009;35:59–61.
- Borde D, Gandhe U, Hargave N, et al. The application of European system for cardiac operative risk evaluation II (EuroSCORE II) and Society of thoracic surgeons (STS) risk-score for risk stratification in Indian patients undergoing cardiac surgery. Ann Card Anaesth. 2013;16:163–166.
- Raza S, Blackstone EH, Sabik JF. The diabetes epidemic and its effect on cardiac surgery practice. J Thorac Cardiovasc Surg. 2015;150:783–784.
- Barili F, Pacini D, Capo A, et al. Does EuroSCORE II perform better than its original versions? A multicentre validation study. Eur Heart J. 2013;34:22–29.
- Grant SW, Hickey GL, Dimarakis I, et al. How does EuroSCORE II perform in UK cardiac surgery; an analysis of 23 740 patients from the society for cardiothoracic surgery in Great Britain and Ireland national database. Heart. 2012;98:1568–1572.
- Howell NJ, Head SJ, Freemantle N, et al. The new EuroSCORE II does not improve prediction of mortality in high-risk patients undergoing cardiac surgery: a collaborative analysis of two European centres. Eur J Cardiothorac Surg. 2013;44:1006–1011. discussion 1011.
- Kunt AG, Kurtcephe M, Hidiroglu M, et al. Comparison of original EuroSCORE, EuroSCORE II and STS risk models in a Turkish cardiac surgical cohort†. Interact Cardiovasc Thorac Surg. 2013;16:625–629.
- Wang L, Han -Q-Q, Qiao F, et al. Performance of EuroSCORE II in patients who have undergone heart valve surgery: a multicentre study in a Chinese population. Eur J Cardiothorac Surg. 2014;45:359–364.
- Pillai BS, Baloria KA, Selot N. Validation of the European system for cardiac operative risk evaluation-II model in an urban Indian population and comparison with three other risk scoring systems. Ann Card Anaesth. 2015;18:335–342.
- Luo HD, Teoh LK, Gaudino MF, et al. The Asian system for cardiac operative risk evaluation for predicting mortality after isolated coronary artery bypass graft surgery (ASCORE‐C). J Card Surg. 2020;35:2574–2582.