Mortality Risk Factors and Validation of Severity Scoring Systems in Critically Ill Patients with Acute Renal Failure

Abstract

Background. Risk stratification and prediction of outcome in acute renal failure patients in the intensive care unit are important determinants for improvement of patient care and design of clinical trials. Methods. In order to identify mortality risks factors and validate general and specific predictive models for acute renal failure (ARF) patients in the intensive care unit (ICU), 324 patients were prospectively evaluated. Multivariate analysis by logistic regression was utilized for identification of mortality risk factors. Discrimination and calibration were used to evaluate the performance of the following models at referral to nephrologist and at initiation of renal replacement therapy: APACHE II, SAPS II, LODS, and ATN-ISI. Organ failure was assessed by SOFA and OSF. Results. The hospital mortality rate was 85%. The identified mortality risk factors were: age ≥ 65 yr, BUN ≥ 70 mg/dL, ARF of septic origin, and previous hypertension. Serum creatinine ≥ 3.5 mg/dL, systolic blood pressure ≥ 100 mm Hg, and normal consciousness were associated with mortality risk reduction. Performance of all prognostic models was disappointing with unsatisfactory calibration and underestimation of mortality on the day of referral to the nephrologist and at initiation of renal replacement therapy. Conclusions. Cross-validation of prognostic models for ARF resulted in poor performance of all studied scores. Therefore, a specific model is still warranted for the design of clinical trials, comparison of studies, and for prediction of outcome in ARF patients, especially in the ICU.

Keywords:

• acute renal failure
• intensive care unit
• mortality
• prognostic scores

Introduction

In the last decades, significant findings on the understanding of the pathophysiological mechanisms of experimental acute renal failure (ARF) have been reported. However, few of them resulted in clinical measures to prevent, treat, or accelerate renal function recovery in patients with ARF. Despite the apparent improvement in survival rates resulting from new dialysis strategies, the mortality rate of ARF remains high and may reach 80% in intensive care unit (ICU) patients.[1-4]

There are multiple difficulties involved in the comparison of clinical studies in ARF patients: use of different definitions of ARF, study design, different evaluated outcomes, heterogeneity of patients, and associated therapies. Moreover, the assessment of severity of ARF patients by prognostic mortality models or severity scores systems has not been standardized yet. Prognostic scores may be used for disease severity stratification, comparison of patients, therapeutic response follow-up, or evaluation of medical assistance among different institutions.

Although the general prognostic scores utilized in the ICU have not been originally developed for specific populations, they are still widely used in patients with ARF.[4-6] Mortality prediction for ARF patients is likely to be more accurate if specific prognostic scores were used for this population.[7&8] Controversial results have been reported on the ability of prognostic scores to predict mortality in ARF and which model for these patients is more appropriate.[2], [9] Moreover, good performance of a prognostic model in one institution does not indicate that it will be as effective in other institutions or in different groups of patients.[10] On this background, the aims of this study were to identify mortality risk factors and determine the performance of general and specific prognostic models in severely ill patients with acute renal failure.

Materials and Methods

Study Cohort

From June 2000 to June 2001, adult patients (≥ 18 years old) diagnosed with ARF in the ICU, and referred to the nephrologists, were prospectively evaluated. Acute renal failure was defined as serum creatinine increase to at least 2.0 mg/dL or an increase of 1.0 mg/dL in patients with baseline serum creatinine lower than 1.5 mg/dL. For patients with baseline creatinine ranging from 1.5 mg/dL to 3.0 mg/dL, ARF was defined as an increase of 50% or more.

Exclusion Criteria

Patients with baseline serum creatinine higher than 3.0 mg/dL, ARF etiology other than acute tubular necrosis (ATN), and those who received nephrologic care prior to ICU admission were excluded. The cause of ARF was considered to be ATN when a prerenal cause, urinary tract obstruction, acute interstitial nephritis, glomerulopathies, vascular, or systemic diseases were ruled out as the etiology of ARF.

ARF Classification

Each patient's ARF was classified as nephrotoxic or ischemic. Nephrotoxic ARF was classified when associated with nephrotoxic drugs and ischemic ARF when caused by hypotension, hemorrhage, low cardiac output, or sepsis. Sepsis was defined according to the American College of Chest Physicians/Society of Critical Care Medicine Consensus Conference.[11] The ARF was also classified as surgical (ARF within 72 hours of surgery), obstetric (HELLP syndrome, eclampsia, post-delivery hemolytic-uremic syndrome, and other obstetric complications), and medical (patients not included in previous categories).

Initiation of renal replacement therapy and modality were indicated by the nephrologist. Renal function recovery was defined as a return to baseline serum creatinine values and partial recovery as a decrease of serum creatinine above baseline levels.

Prognostic Scores

Acute Physiology Age and Chronic Health Evaluation II (APACHE II), Simplified Acute Physiology Score II (SAPS II), Logistic Organ Dysfunction System (LODS), Sequential Organ Failure Assessment (SOFA), Acute Tubular Necrosis Individual Severity Index (ATN-ISI), and Organ System Failure (OSF) were calculated by the same investigator on the day of referral to the nephrologist and at initiation of renal replacement therapy, according to the recommendations of the original studies.[7], [12-16] The definition of renal failure of the original OSF score was modified (described earlier) and hepatic failure was defined as prothrombin activity < 50% and/or bilirubin > 2.0 mg/dL. The highest number of organ failures was six (cardiovascular, respiratory, renal, hematologic, hepatic and neurological). If a variable was not measured for a patient, it was assumed to be within the range of normal.

Statistical Analysis

Performance of prognostic models that predict the risk of hospital mortality (APACHE II, SAPS II, LODS, and ATN-ISI) was tested by means of discrimination and calibration. Discrimination is the ability of the score to predict the outcome of patients. Discrimination was assessed by the ROC (receiver operating characteristic) curve area, which shows the rate of sensibility (correct identification of nonsurviving or true positive patients) and 1-specificity (incorrect identification of surviving or false positive patients). The values of the area under the ROC curve indicate the discrimination power of the score. Prognostic models with no discrimination power have an area under the ROC curve lower or equal to 0.5; scores with an area greater than 0.7 have satisfactory discrimination power, and those with an area greater than 0.9 have excellent discrimination power.[17] The area under the ROC curve was expressed in absolute values and dispersion was expressed by the 95% confidence interval (95% CI).

Calibration evaluates the ability of a prognostic model to accurately predict mortality in all risk groups. Patients were divided according to the mortality intervals predicted by the scores in risk decile and calibration was defined as the degree of agreement between model-predicted mortality and actual mortality for each risk strata. Adequacy of curve adjustments was assessed by Hosmer-Lemeshow chi-square test. A nonsignificant result suggests that the model has good calibration.

Continuous data were analyzed by Student's t test and by Mann-Whitney's U test and expressed as mean ± standard deviation or by median and twenty-fifth to seventy-fifth percentile when appropriate. Categorical variables were expressed as percents and were analyzed by Pearson's chi-squared test or Fisher's exact test when appropriate. Multivariable logistic regression was carried out to obtain mortality risk factors using variables with a trend towards significance in the univariate analysis (P ≤ 0.15) on the day of referral to the nephrologist. Results were expressed by odds ratio (OR) and 95% CI. Differences were considered statistically significant when P < 0.05. Statistical analysis was carried out by SPSS version 10.0 (SPSS Inc., Chicago, IL, USA) and BMDP programs (BMDP Statistical Software Inc., Los Angeles, USA).

Results

During the study period, 324 patients were included for analysis and the hospital mortality rate was 85%. The ARF incidence in the ICU requiring nephrologic care was 4.6% (7080 patients admitted to the ICU in this period). The ARF etiology was multifactorial, the mean age was 60.1 ± 16.7 years and the nonsurviving patients were older (Table 1). Among the comorbidities, systemic hypertension was the most prevalent (42.3%), followed by immunosuppression (20.7%), diabetes mellitus (19.7%), congestive heart failure (16.4%), neoplasias (15.2%), coronary disease (12.3%), obstructive pulmonary disease (12%), liver cirrhosis (10.2%), stroke (6.5%), and AIDS (3.4%). Only hypertension was more prevalent in the nonsurviving patients compared with surviving patients (45.1% vs. 26.5%, P < 0.05).

Table 1. Patients demographics

	All	Survivors	Nonsurvivors	P
N (%)	324	49 (15.1%)	275 (84.9%)
Age	60.1 ± 16.7	52.9 ± 17.6	61.4 ± 16.2	0.003
Sex
Male	213 (65.7%)	33 (65.3%)	180 (65.5%)	0.79
Female	111 (34.3%)	16 (32.7%)	95 (34.5%)
Type of ARF
Medical	221 (68.2%)	26 (53.1%)	195 (70.9%)	0.005
Surgical	102 (31.5%)	22 (44.9%)	80 (29.1%)
Obstetrical	1 (0.3%)	1 (2%)	0 (0%)
Etiology of ATN
Ischemia	276 (85.2%)	37 (75.5%)	239 (86.9%)	0.04
Nephrotoxins	48 (14.8%)	12 (24.5%)	36 (13.1%)

Median time between the onset of ARF and consultation with nephrologist was 2 days (1–5 days). On the day of referral to the nephrologist, most patients were on mechanical ventilation (77.5%) with higher prevalence among the nonsurvivors (81.5% vs. 55.1% in surviving patients, P < 0.001). Only 79 patients (24.3%) were conscious, 52 (65.8%) among the surviving patients and 27 (34.2%) among the nonsurviving patients (P < 0.001). Median diuresis volume in the 24 hours prior to referral to the nephrologist was greater for the surviving patients—970 mL (430–1650 mL) vs. 592 mL (256–1130 mL) for nonsurviving patients (P = 0.017). Patients with diuresis greater than 1000 mL/day presented lower mortality (75%) compared with patients with diuresis between 400 and 1000 mL/day (88%) and patients with oliguria (89%) (P < 0.05).

On the day of referral to the nephrologist, nonsurviving patients presented higher BUN levels, more intense metabolic acidosis, and lower serum albumin than did the surviving patients (Table 2). Additionally, nonsurvivors were hemodynamically unstable and required more norepinephrine than did the surviving patients (54% vs. 24.5%, P < 0.001). The degree of illness severity of these patients may be observed by the higher levels of all prognostic scores (Table 3).

Table 2. Laboratory tests at referral to nephrologist and on the day of first dialysis

	All	Survivors	Nonsurvivors	P
BUN (mg/dL)
Referral	68.7 ± 29.4	57.5 ± 26.2	70.8 ± 29.4	0.002
Dialysis	88.8 ± 35.9	79.9 ± 35.5	90.2 ± 35.5	0.23
Creatinine (mg/dL)
Referral	3.8 ± 1.3	4.0 ± 1.4	3.7 ± 1.3	0.15
Dialysis	5.2 ± 2.8	5.9 ± 3.1	5.1 ± 2.8	0.26
Sodium (mEq/L)
Referral	139 ± 8	140 ± 8	139 ± 8	0.80
Dialysis	140 ± 8	140 ± 6	140 ± 8	0.85
Potassium (mEq/L)
Referral	4.8 ± 1.2	4.6 ± 1.2	4.8 ± 1.2	0.19
Dialysis	4.9 ± 1.2	4.9 ± 1.2	4.9 ± 1.2	0.68
pH
Referral	7.29 ± 0.11	7.35 ± 0.09	7.29 ± 0.11	0.001
Dialysis	7.27 ± 0.10	7.31 ± 0.09	7.27 ± 0.10	0.06
Bicarbonate (mEqL)
Referral	17.8 ± 5.8	19.6 ± 5.3	17.4 ± 5.8	0.017
Dialysis	17.5 ± 5.4	18.3 ± 3.3	17.4 ± 5.6	0.47
Albumin (g/L)
Referral	23.7 ± 6.7	25.8 ± 6.3	23.4 ± 6.8	0.026
Dialysis	24.1 ± 6.5	25.6 ± 6.9	23.9 ± 6.5	0.33
Protrombin time (%)
Referral	47.8 ± 22.2	51.3 ± 23.9	47.2 ± 21.9	0.27
Dialysis	46.6 ± 21.0	50.1 ± 24.0	46.2 ± 20.7	0.44
Hematocrit (%)
Referral	29.9 ± 5.7	29.2 ± 5.7	30.0 ± 5.7	0.22
Dialysis	29 ± 5	28 ± 4	29 ± 5	0.97
Leukocyte (mm^3)
Referral	12945 ± 7507	11683 ± 5316	13176 ± 7829	0.24
Dialysis	13927 ± 8073	13994 ± 7516	13920 ± 8147	0.93
Platelets (× 10^3/mm^3)
Referral	133 ± 105	148 ± 127	131 ± 100	0.41
Dialysis	134 ± 103	152 ± 141	132 ± 99	0.54

Table 3. Severity scores at referral to nephrologist and on the day of first dialysis

	All	Survivors	Nonsurvivors	P
APACHE II
Referral	31 ± 8.3	24 ± 8.7	32 ± 7.7	< 0.001
Dialysis	33 ± 7.4	26.9 ± 7.5	33.7 ± 7.1	< 0.001
SAPS II
Referral	49.4 ± 14.3	37.2 ± 13.3	51.5 ± 13.3	< 0.001
Dialysis	52.4 ± 13.4	44.4 ± 12.6	53.3 ± 13.2	0.002
ATN-ISI
Referral	0.57 ± 0.20	0.42 ± 0.21	0.59 ± 0.18	< 0.001
Dialysis	0.63 ± 0.18	0.52 ± 0.20	0.64 ± 0.17	0.004
LODS
Referral	7.5 ± 2.2	6.3 ± 1.9	7.6 ± 2.1	< 0.001
Dialysis	7.9 ± 2.2	7.3 ± 1.9	7.9 ± 2.2	0.25
SOFA
Referral	12.8 ± 4.3	10.5 ± 4.1	12.9 ± 4.2	< 0.001
Dialysis	13.9 ± 4.1	12.4 ± 4.5	14.2 ± 3.9	0.07
OSF
Referral	3.0 ± 0.9	2.7 ± 0.9	3.0 ± 0.9	0.06
Dialysis	3.0 ± 0.9	2.7 ± 0.9	3.0 ± 0.9	0.06

Dialysis was required in 210 patients (64.8%). Renal replacement therapies employed were CVVHD (135 patients—64.3%), peritoneal dialysis (49 patients—23.3%), intermittent hemodialysis (16 patients—7.6%), CVVHDF (5 patients—2.4%), CVVH (4 patients—1.9%), and SCUF (1 patient—0.5%). In 87 patients (41.4%), renal replacement therapy was started on the day of nephrologic consultation. Hospital mortality rate in patients submitted to dialysis was 90%.

Among surviving patients, 32 (65.3%) completely recovered renal function, 15 patients (28.6%) presented partial recovery, and 3 patients (6.1%) were dialysis-dependent at hospital discharge.

In the final multivariable logistic regression model (variables included on the day of referral to nephrologist), the following variables were identified as mortality risk factors: age ≥ 65 years, BUN ≥ 70 mg/dL, ATN of septic origin, and history of hypertension. On the other hand, serum creatinine ≥ 3.5 mg/dL, systolic blood pressure (SBP) ≥ 100 mm Hg and normal consciousness were associated to mortality risk reduction (Table 4).

Table 4. Mortality risk factors for ARF in the ICU at referral to nephrologist

Risk factor	Odds ratio	95% CI	P
BUN ≥ 70 mg/dL	6.02	2.13–17.0	< 0.01
Sepsis ATN	3.09	1.27–7.51	< 0.05
Age ≥ 65 years	2.81	1.25–6.34	< 0.05
History of hypertension	2.53	1.08–5.95	< 0.05
Creatinine ≥ 3.5 mg/dL	0.42	0.19–0.94	< 0.05
Normal consciousness	0.27	0.13–0.57	< 0.001
SBP ≥ 100 mmHg	0.25	0.11–0.53	< 0.001

The performance of the prognostic scores was poor on the day of referral to the nephrologist and at initiation of renal replacement therapy. All prognostic models had unsatisfactory calibration and underestimated mortality on the day of referral to the nephrologist and at initiation of renal replacement therapy (Figures 1 and 2).

Figure 1 ROC curves (A) and Calibration analysis (B), constructed by plotting observed death rate (- -▪ - -) and predicted death rate (—•—) by APACHE II, SAPS II, LODS, and ATN-ISI stratified by deciles of risk at referral to nephrologist (^*Hosmer-Lemeshow chi-square test).

Figure 2 ROC curves (A) and Calibration analysis (B), constructed by plotting observed death rate (- -▪ - -) and predicted death rate (—•—) by APACHE II, SAPS II, LODS, and ATN-ISI stratified by deciles of risk on the day of first dialysis (^*Hosmer-Lemeshow chi-square test).

Discussion

Acute renal failure (ARF) has a high prevalence in the ICU, raises length of stay, increases hospital costs, and is associated with very elevated mortality rates. The identification of mortality risk factors may help to attenuate the impact of ARF on outcome of critically ill patients. In the present study, the following variables were associated with mortality: age ≥ 65 years old, BUN ≥ 70 mg/dL, ARF of septic etiology, and history of hypertension. On the other hand, serum creatinine ≥ 3.5 mg/dL, SBP ≥ 100 mmHg, and normal consciousness were associated with mortality risk reduction. Some of these variables have been already described as mortality risk factors in ARF. The association between BUN levels and mortality in patients with ARF was described on the day of referral to the nephrologist[9] and at the time of the first dialysis.[18&19] The identification of BUN levels ≥ 70 mg/dL as one of the major mortality prognostic factors may indicate that the patient was referred to the nephrologist too late. The nutritional status of these ARF patients may be related to the detection of increased serum creatinine levels as a protective mortality factor. Patients with ARF and malnutrition have lower serum creatinine levels, more comorbidities, and higher mortality rates when compared to patients with preserved nutritional status.[20]

Frequently, ARF is not the first organ dysfunction in septic patients, but it is clearly associated with the high mortality rate observed in this group of patients. Early intervention in septic patients may lead to important results in the prevention of ARF. Rivers et al. showed that early and aggressive hemodynamic resuscitation reduced the severity of organ dysfunctions (assessed by APACHE II, SAPS II, and MODS) and mortality in patients with severe sepsis and septic shock.[21]

The presence of multiple comorbidities and advanced age may also have contributed to the high mortality rate observed in this and other studies involving ARF patients in the ICU. The mean age in this study was 60.1 years whereas in a previous analysis carried out at the same institution in 1993, the mean age was 55 years and in 1983 it was 39 years.[22] However, other authors did not identify age as a prognostic factor of mortality in ARF patients.[23] Among comorbidities, only systemic hypertension was associated with mortality. This association has not been previously reported. On the contrary, Chertow et al. had reported hypertension as one of the factors associated with lower mortality rate in ARF patients.[23] In another study, hypertension was identified as one of the major risk factors for the development of ARF after cardiorespiratory arrest.[24] The authors speculate that microangiopathy and glomerular and juxtaglomerular injuries caused by hypertension might limit functional recovery after ischemic insult. As in other studies, hypotension was also identified as a prognostic mortality factor.[7] Normal consciousness had also been associated with mortality protection in ARF and might indicate lower severity and lower need for mechanic ventilation and sedation.[7]

Performance of the models evaluated in this study was worse than expected. All scores underestimated mortality on the day of referral to the nephrologist and at initiation of renal replacement therapy. The discriminatory capability (ROC curve) was just reasonable on the day of nephrologic assessment.

General prognostic scores, APACHE II and SAPS II, are the most widely used models for the evaluation of severity of patients with ARF. The APACHE II score seems to underestimate mortality in ARF patients when calculated on the day of the first dialysis.[2], [9&10] As opposed to most authors, Parker et al. reported that when applied on the day of the first dialysis, APACHE II was a good predictor of mortality and recovery of renal function in patients with ARF.[6] The APACHE II, SAPS II, and Mortality Probability Model II (MPM II) scores do not predict death when calculated on the day of admission to the ICU for patients with ARF.[5] The performance of the general scores obtained in this study confirmed similar previously reported results. Thus, these models should probably continue to be used only for stratification of severity of illness in these patients.

The utilization of prognostic models in groups of patients with different clinical characteristics than the population of the original model may cause discrepancies between the mortality predicted by the score and the actual mortality rate. Murphy-Filkins et al. evaluated the performance of MPM II varying the percentage of diseases in the original database which included 4224 ICU patients.[25] Discrimination and calibration of the model progressively worsened as the frequency of patients with each disease increased above the original database distribution. The original frequency of patients with ARF was 4.9% and the simulated increase to 16.9% was sufficient to deteriorate the performance of the prognostic model.

In addition, differences in the mortality rate among units may also be responsible for differences in the performance of prognostic models. Glance et al. simulated different case mixes varying mortality between 5% and 18% using a database of 6806 patients admitted to a surgical ICU.[26] The area under the ROC curve of APACHE II in the original population was 0.74. Depending on the simulated mortality, the ROC curve area ranged from 0.67 to 0.79. As the simulated mortality increased, calibration determined by the Hosmer-Lemeshow's test deteriorated. In general, these scores were developed to predict mortality in populations in which the majority of patients had low probability of death. Therefore, the use of these prognostic models in patients with ARF, whose mortality rate is high, may decrease their efficacy. In our institution, which is the largest tertiary hospital in Brazil, due to the great demand, only very ill patients are admitted to ICU; and some times the admission is delayed because of shortage of beds. This situation and the severity of the illness of our population (high prevalence of multiorgan failure, hypotension, and sepsis) may help to explain the high mortality rate and the poor performance of the evaluated scores.

The use of ARF-specific prognostic scores should be more appropriate to predict mortality in this population.[7] Despite the expected better performance of ATN-ISI when compared to other prognostic models, this was not observed in this study. Douma et al. reported that ATN-ISI had a better performance than most general prediction models when calculated on the day of the first dialysis.[2] Recently, Mehta et al. developed a new predictive model for critically ill ARF patients.[9] In that study, the model specifically developed for ARF showed good discriminatory ability (area under the ROC curve of 0.83) and excellent calibration for all risk groups. In contrast, other ARF-specific scores, including ATN-ISI, did not present a good performance in that study.

Other possibilities could explain the poor performance of the scores studied in this specific population. Different inclusion criteria or ARF definition than those from the original studies may directly influence model performance. Scores developed at one institution do not necessarily have a good performance when validated in other institutions.[10] In addition, the effect of previous treatment by the ICU medical team before consultation with a nephrologist—lead time bias—may also influence the performance of prognostic models. Also, the incidence of ARF in this study (4.6%) was underestimated, since only patients referred to the nephrologist were evaluated. Hospitalization prior to admission and ARF occurring during hospitalization at the ICU are associated with mortality and could also interfere with the performance of prognostic scores.[27] Another issue refers to the use of scores at different timepoints than those of the original models. APACHE II, for example, was originally described for mortality prediction in the first 24 hours after ICU admission. Therefore, several reasons may account for the poor performance of all scores evaluated in our population.

In conclusion, cross-validation of predictive models for ARF resulted in poor performance of several scores at referral to nephrologist and at initiation of dialysis. Therefore, a specific model is still warranted for the design of clinical trials, for comparison studies and for prediction of outcome in ARF patients, especially those in the ICU.