Evaluation of Prognostic Indexes in Critical Acute Renal Failure Patients

Abstract

Objective. To study different prognostic indexes in acute renal failure (ARF) patients admitted to an intensive care unit (ICU). Design. Prospective, cohort study. Individual Severity Score–Acute Tubular Necrosis (ISS‐ATN) obtained prospectively and retrospectively, Acute Physiologic and Chronic Health Evaluation (APACHE II) Score, APACHE II Risk, Lung Injury Score (LIS), and Number of Organ Failures (NOF) were calculated for each patient. The outcome analyzed was death in the ICU. Discrimination was evaluated by the area under the receiver operator characteristic curve (AUC). For calibration analysis, the chi‐square goodness‐of‐fit test was used to compare predicted mortality, calculated by ISS‐ATN (obtained prospectively or retrospectively) and APACHE II risk, with observed mortality. Setting. ICU, São Rafael Hospital, Salvador‐BA, Brazil. Patients. Seventy‐six ARF patients admitted to the ICU within 6 months. Interventions. Surgical and medical procedures. Measurements and Results. The observed AUC was 0.69 for LIS, 0.73 for prospective ISS‐ATN, 0.75 for retrospective ISS‐ATN, 0.76 for APACHE II Score, 0.78 for APACHE II Risk, and 0.88 for NOF. These areas were significantly different from 0.5 (p < .001). There was no difference between the observed and expected death rate calculated by ISS‐ATN. However, APACHE II underestimated the observed mortality (p < .001). Conclusion. The prognostic indexes studied showed good discriminative power. However, APACHE II was not well calibrated in contrast to the good calibration of ISS‐ATN.

• Acute renal failure
• Calibration
• Discrimination
• Mortality
• Prognosis
• ROC curve
• APACHE (Acute Physiologic and Chronic Health Evaluation) II
• Creatinine
• Dialysis
• Diuresis
• Intensive care unit
• Lung Injury Score
• Prognostic index
• Respiratory failure
• Positive end expiratory pressure (PEEP)
• Death rate

Introduction

Acute renal failure (RNF) in hospitalized patients is associated with high mortality rates, increased length of hospital stay,[1] and high costs.[1], [2] Some authors have more recently reported an overall improvement in outcomes.[3], [4] However, mortality has not changed significantly in groups of patients with more severe diseases.[5]

Acute Physiologic and Chronic Health Evaluation (APACHE) II[6] is a prognostic index constructed for intensive care unit (ICU) patients. It has questionable efficiency for predicting the outcome of ARF patients.[7], [8], [9] To stratify patients for quality of care control and research purposes, many authors have been searching for prognostic score indexes that could be easily and reliably collected at bedside or from the patient's chart, and that could predict outcomes with good discrimination and calibration.

APACHE II and APACHE III, as well as a number of other “general” prognostic indexes, are logistic models that included large numbers of patients.[6], [10] This is in contrast to other indexes that were developed specifically to measure the risk of death of ARF patients. The majority of these ARF‐specific prognostic indexes were developed in retrospective studies, and none included more than 1,000 patients. Furthermore, the definition of ARF was not uniform in these studies.[4], [7], [8], [11], [12], [13], [14], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24] Due to these limitations, there is no consensus as to which prognostic index should be used to predict outcomes in ARF patients.

The aim of this study was to test the performance of different prognostic indexes[6], [19] used for the prediction of outcomes of critical patients with ARF.

Patients and Methods

Patients admitted to the ICU with ARF and those who developed ARF during the ICU stay were eligible for inclusion in this study. All patients that fulfilled the following criteria were included: acute increase of serum creatinine above 1.3 mg/dL (whether observed before or after admission to the ICU), serum creatinine higher than 1.3 mg/dL with clinical evidence of previous normality of renal function, nondialytic chronic renal failure with an acute increase of serum creatinine of at least 20% above the baseline, or an absolute increase of 0.4 mg/dL in baseline creatinine value. Patients with dialytic chronic renal failure were excluded. Permission to perform the study was obtained from the Hospital Ethics Committee.

Standard ICU care was provided to every patient included in the study. Therapeutic (nutritional support, mechanical ventilation, and dialysis) and monitoring procedures were initiated by the attending physicians, without intervention of the investigators. Hemodialysis was performed with bicarbonate dialysate and polysulfone membrane filters. Arteriovenous hemofiltration or hemodiafiltration, venovenous hemofiltration or hemodiafiltration, and peritoneal dialysis were also performed.

Patients were followed until one of the following outcomes was observed: ICU discharge, death, or recovery of renal function. Eight months after the prospective data collection, patient charts were reviewed in order to calculate ISS‐ATN retrospectively.

The APACHE II score and risk of death were calculated at admission to the study, using the method proposed by Knaus.[6] The Individual Severity Score–Acute Tubular Necrosis (ISS‐ATN), developed by Liaño et al.,[19] was calculated on the day of admission with data collected prospectively at bedside (prospective ISS‐ATN) and also retrospectively from records after hospital discharge (retrospective ISS‐ATN). Lung Injury Score (LIS) was also calculated at admission to the study as described by Murray et al.[25] This score uses the alveolar‐arterial difference of oxygen, the presence of pulmonary lesions as detected in lung X‐rays, lung compliance, and level of positive end‐expiratory pressure.

Statistical Analysis

The outcome studied was death during the ICU stay. Discrimination was analyzed by measuring the area under the receiver operator characteristic curve (AUC).[26] One receiver operator characteristic (ROC) curve was constructed for each of five indexes, and the calculated AUC of each index (with its standard error) was compared to the AUC under the null hypothesis (AUC = 0.5) using the Z test.

Chi‐square goodness‐of‐fit tests were used to study the calibration of APACHE II risk and ISS‐ATN indexes. The other scores could not be tested because they do not provide a predicted mortality. For each of these two indexes, patients were divided in quartiles (in increasing order of probability of death). The predicted mortality of patients in each quartile was compared with the observed mortality for the same quartile. In addition, the Mann Whitney Test, chi‐square test, and randomization test for contingency tables were also used. Tests that yielded p values lower than .05 were considered statistically significant. The True Epistat (Richardson, Gustafson TL) computer software was used for all statistical tests.

Results

Overall, 76 patients were included (Table 1), with ages ranging from 7 to 96 years. Sixty‐three percent were men, and 55% of the patients were older than 65 years. Thirty‐nine percent were surgical patients and 39 patients (51%) had a diagnosis of infection. Forty‐six patients (60.5%) were on mechanical ventilation, 4% had mean arterial pressure less than 50 mmHg, and 24% were oliguric. Twenty‐six patients (34%) were submitted to at least one session of dialysis.

Table 1. Clinical and Demographic Data of Critical Patients Admitted in the Study

Characteristic	Number (%)
Age older than 65 years	55%
Male	63%
Previous clinical problems
• Arterial hypertension	49 (64%)
• Renal failure	13 (17%)
• Stroke	12 (16%)
• Cardiac failure	15 (20%)
• Cancer	11 (14%)
• Diabetes mellitus	22 (29%)
Mechanic ventilation	46 (60.5%)
Use of dialysis	26 (34%)
Hemodynamic monitoring	12 (16%)
Diuresis lower than 400 mL/day	24%
Diuresis higher than 1,200 mL/day	39%
Previous surgery	30 (39%)
Use of low‐dose dopamine (< 2 µg/kg/min)	32%
Use of furosemide	65%
Use of high dose of furosemide (> 400 mg/day)	20%

Upon ICU admission, mean serum creatinine level was 2.0 ± 1.4 mg/dL and 2.1 ± 1.1 mg/dL at study entry. The peak serum creatinine was 3.1 ± 1.7 mg/dL, and the mean of the lowest creatinine was 0.9 ± 0.4 mg/dL. The difference between the lowest and the highest creatinine was 2.2 ± 1.6 mg/dL. The ICU mortality was 64%, whereas the hospital mortality was 66%. Eighteen percent of the patients recovered renal function, and none of them became dialysis dependent. Data on prognostic indexes are shown in Table 2.

Table 2. Measurement of Prognostic Indexes in 76 ARF Patients

Characteristic	Mean + standard deviation
Age (median, years)	66
APACHE II Score	19.9 + 7.8
APACHE II Risk	35.3 + 23.8%
ISS‐ATN (prospective)	59.6 + 23.5%
Number of Organic Failures	2.86 + 1.34
Lung Injury Score	1.54 + 1.03

ISS‐ATN, Individual Severity Score–Acute Tubular Necrosis.

When stratified by the LIS, the higher the score, the higher the mortality (Table 3). A higher level of peak creatinine was also associated with higher mortality (Table 4). The five prognostic indexes studied were all better than chance in discriminating patients that would die from those whom would survive. The observed AUC for death during the ICU stay was 0.69 for LIS, 0.73 for prospective ISS‐ATN, 0.75 for retrospective ISS‐ATN, 0.76 for APACHE II Score, 0.78 for APACHE II Risk, and 0.88 for NOF. All were significantly different from the curve of H_o hypothesis (AUC = 0.50) (Fig. 1).

Table 3. Lung Injury Score (LIS) and Death Rate of Patients with ARF

LIS (mean)	Died (%)	Survived	Odds ratio
< 1.75 (0.75)	18 (47%)	20	1.00
1.75–2.50 (1.96)	13 (76%)	4	3.61
≥ 2.50 (2.77)	13 (86%)	2	7.22

Chi‐square = 8.8963; df = 2; p = .0117.

Table 4. Peak Serum Creatinine and Death Rate of Patients with ARF

Creatinine (mean)	Died (%)	Survived	Odds ratio
< 2 mg/dL (1.6)	7 (35%)	13	1.00
2–7 mg/dL (3.2)	20 (67%)	10	3.71
Dialysis (4.7)	22 (85%)	4	10.21

Chi‐square = 12.2532; df = 2; p = .0022.

Figure 1. Receiver operator characteristic curve of prognostic indexes in 76 acute renal failure patients at an intensive care unit. Areas under the receiver operator characteristic curve (AUC) are significantly different from AUC of null hypothesis (0.50) for all tests, p < .001. ♦, Individual Severity Score–Acute Tubular Necrosis (prospective) (AUC = 0.73); ⋄, ISS‐ATN (retrospective) (AUC = 0.75); □, APACHE II Risk (AUC = 0.78); ▪, Number of Organic Failures (AUC = 0.88); ○, APACHE II Score (AUC = 0.76); •, Lung Injury Score (AUC = 0.69).

Retrospective ISS‐ATN had an AUC (discrimination), which was not statistically different from prospective ISS‐ATN. However, retrospective ISS‐ATN (0.509 ± 0.210) was significantly lower than the prospective ISS‐ATN (0.596 ± 0.235; p = .02).

The expected mortality for each quartile analyzed as predicted by the APACHE II was lower than the observed mortality (goodness‐of‐fit chi‐square test p < .001) (Fig. 2). However, when the observed mortality was compared in each quartile with the mortality predicted by ISS‐ATN, the opposite results were found. There was no difference between observed mortality and mortality as predicted by prospective ISS‐ATN (p = .5983) and retrospective ISS‐ATN (p = .3699) (Fig. 2).

Figure 2. Calibration of prognostic indexes in patients with acute renal failure. Solid line, predicted death rate; dashed line, observed death rate. (A) Individual Severity Score–Acute Tubular Necrosis (ISS‐ATN) (prospective)–goodness‐of‐fit chi‐square test; X² = 1.8769; df = 3; p = .5983. (B) ISS‐ATN (Retrospective)–goodness‐of‐fit chi‐square test; X² = 3.1444; df = 3; p = .3699. (C) APACHE II Risk–goodness‐of‐fit chi‐square test; X² = 29.7981; df = 3; p < .001.

Discussion

The clinical and demographic characteristics of this cohort were not different from the majority of other published series. The finding of AUCs greater than 0.70 for the indexes tested (except for LIS) suggests they all had good discriminative power in this group of patients.[27] The risk of death estimated by APACHE II was poorly calibrated, and the observed mortality was also underestimated. However, ISS‐ATN was well calibrated when collected prospectively and retrospectively.

Comparing our results with those from other more recently published studies may be difficult because these studies have characterized ARF using a number of different criteria, such as serum creatinine levels,[12], [13], [14], [17], [19], [23], [24] absolute or relative increases in creatinine levels,[16] or use of dialysis.[4], [7], [8], [11], [15], [18], [21], [22] However, our case mix was similar to the others, which described predominantly male (42%–94%) patients with mean age ranging from 52 to 68 years. Fifty‐two to 93% of the patients included in these studies were oliguric, 38% to 78% received mechanical ventilation, and 48% to 100% received dialysis. They also include surgical patients in widely different proportions (ranging from 0% to 100% of the patients). Observed mortality varied between 42% and 81%.

This study suggests that widely evaluated prognostic indexes such as ATN‐ISS, APACHE II, and other less‐studied indexes such as NOF and LIS, had moderate discriminative power in this cohort of ARF patients. The superior discriminative power of NOF indicates that this index could be a candidate for the development of a logistic model for prognosis of ARF patients, as was previously attempted for ICU patients by Marshall et al.[28] Marshall's model uses the high level of serum creatinine as a marker of poor prognosis, even though existing data suggest that it is a marker of good and not bad prognosis in dialytic ARF patients.[8], [11]

The stratification of severity obtained with dimensionless markers such as LIS and NOF has, besides discriminative power, an association with higher mortality in ARF patients stratified according to LIS or NOF. This clearly suggests that severity of each organic failure can be measured in prognostic terms. Modeling prognosis with a NOF approach becomes even more attractive when one notes the fact that ATN‐ISS, the best index in ARF patients, provides higher weights to nonrenal prognostic factors than to nephrotoxicity and oliguria.[19]

Nonoliguric patients constituted the majority of patients in our series. However, oliguria was not strongly associated with mortality. Oliguria was shown to be independently associated with mortality in many studies,[11], [13], [14], [19], [20], [23], [24] but not in others.[4], [7], [8], [12], [15], [16], [17], [18], [21], [22] However, in some of the latter, there was no attempt to include oliguria in the multivariate analysis model.[12], [15], [17], [22] A more recent series of Brazilian ARF patients showed an increased mortality in oliguric patients and a progressively decreasing mortality in five groups of patients stratified by increasing levels of diuresis after controlling for other prognostics factors.[29]

Our data confirm the results of previous studies, which indicated that APACHE II is not effective in predicting death in ARF patients,[7], [8], [9] while other studies reported the opposite.[30], [31], [32] Our data are also in accordance with a study that showed poor calibration of APACHE III in ARF patients.[33] Thus, our results provide new evidence that general prognostic indexes, which are known to underestimate risk of death in patients with dialytic ARF,[33] also underestimate risk of death in patients with less severe ARF. It should be noted that a more recent paper suggests that, in fact, APACHE III underestimated the mortality of ARF patients, but when patients who acquired ARF after admission to the ICU were analyzed separately from those that were admitted already in AFR, it became clear that APACHE III performed well for the latter group and not for the former.[34]

In conclusion, the indexes studied showed a moderate discriminative power in this group of patients. LIS had a positive association with mortality, whereas NOF had the best AUC. The indexes that provide a probability of death performed differently when calibration was analyzed: APACHE II performed poorly, and ISS‐ATN performed well in subgroups of increasing mortality. Therefore, ISS‐ATN may be a useful tool for stratifying ARF patients in our hospital.

Acknowledgments

This work was supported by CNPq and Monte Tabor Foundation. The authors are grateful to Craig Andrew Milroy for assistance in preparation of the manuscript, and to Manuela Cavalcanti, Iara Lemos, Andreia Veloso, Iuri de Araujo Guimarães, and Mila Correia Gois for assistance in data collection.