Head-to-head comparison of 19 prediction models for short-term outcome in medical patients in the emergency department: a retrospective study

Figures & data

Table 1. Overview of included prediction models.

Prediction model	Original population	Items	Outcome
4C mortality [28]	COVID-19 patients	Age, sex, laboratory tests, vital signs	Mortality (in-hospital)
APACHE II [29]	ICU patients	Age, laboratory tests, vital signs	Mortality^a
Asadollahi et al. [30]	ED patients	Age, laboratory tests	Mortality (24 h or 30 days)
COPE [31]	COVID-19 patients	Age, laboratory tests, vital signs	Mortality (28 days) or ICU admission
CURB-65 [13]	Patients with pneumonia	Age, laboratory tests, vital signs	Mortality (30 days)
EWS [43]	ED patients	Vital signs	Mortality^a
MEWS [33]	ED patients	Vital signs	Mortality (60 days) or ICU admission
MSI [34]	ED patients	Vital signs	Mortality (24 h)
NEWS [35]	ED patients	Vital signs	Mortality (24 h or 30 days)
POCT [40]	ED patients	Age, laboratory tests	Mortality (30 days)
RAPS [36]	Patients in transport setting	Vital signs	Mortality (24 h)
REMS [39]	ED patients	Age, vital signs	Mortality^a
RISE UP [15]	Older patients (>65 years)	Age, laboratory tests, vital signs	Mortality (28 days)
Prytherch et al. [37]	ED patients	Age, sex, laboratory tests, mode of admission (elective vs emergency)	Mortality (in-hospital) or ICU admission
qSOFA [38]	Patients with sepsis	Vital signs	Mortality (30 days) or ICU admission
SOFA [14]	ICU patients	Laboratory tests, vital signs	Mortality^a
SPEED [41]	ED patients	Comorbidity, vital signs, laboratory tests, pneumonia	Mortality (28 days)
SSEPS [42]	ED patients	Age, vital signs	Mortality (28 days)
WPSS [32]	ED patients	Vital signs	Mortality (in-hospital)

4C: Coronavirus Clinical Characterisation Consortium; APACHE II: Acute Physiology and Chronic Health Evaluation II; COPE: COVID-19 Outcome Prediction in the Emergency Department; COVID-19 CURB-65: Confusion Urea Respiration Blood pressure; ED: emergency department; EWS: Early Warning Score; ICU: intensive care unit; MEWS: Modified Early Warning Score; MSI: Modified Shock Index; NEWS: National Early Warning Score; POCT: Point Of Care Test; RAPS: Rapid Acute Physiology Score; REMS: Rapid Emergency Medicine Score; RISE UP: Risk Stratification in the Emergency Department in Acutely Ill Older Patients; (q)SOFA: (quick) Sepsis-related Organ Failure Assessment; SPEED: Sepsis Patient Evaluation in the Emergency Department; SSEPS: Simple Sepsis Early Prognostic Score; WPSS: Worthing Physiology Scoring System.

^a Timespan not specified in the original article.

Table 2. Characteristics of the study sample.

	Total study sample (n = 2185)	Admitted to the hospital (n = 1433)
Age, median (IQR), years	67 (53-78)	70 (57-79)
Male, n%	1192 (54.6)	823 (57.4)
CCI, median (IQR)	2 (0-4)	2 (1-4)
Transport by ambulance, n%	855 (39.1)	450 (31.4)
Triage category, n%
Blue (Non urgent)	4 (0.2)	1 (0.1)
Green (Standard)	826 (37.8)	388 (27.1)
Yellow (Urgent)	1017 (46.5)	732 (51.1)
Orange (Very urgent)	301 (13.8)	277 (19.3)
Red (Immediate)	32 (1.5)	32 (2.2)
Diagnosis in the ED
Infectious disease, n%	805 (36.8)	633 (44.2)
Gastrointestinal disease, n%	397 (18.2)	258 (18.0)
Cardiovascular disease, n%	284 (13.0)	97 (6.8)
Neoplasm, n%	138 (6.3)	97 (6.8)
Respiratory disease, n%	107 (4.9)	85 (5.9)
Hematologic disease, n%	86 (3.9)	77 (5.4)
Endocrine disease, n%	73 (3.3)	50 (3.5)
Renal or urogenital disease, n%	60 (2.7)	40 (2.8)
Other, n%	235 (10.8)	96 (6.7)
Outcomes
Admission to hospital, n%	1433 (65.6)
Length of hospital stay, median (IQR), days	5 (2-10)	5 (2-10)
Treatment restrictions, n%	602 (27.6)	530 (37.0)
Mortality (1 day), n%	31 (1.4)	29 (2.0)
Mortality (7 days), n%	97 (4.4)	92 (6.4)
Mortality (31 days), n%	231 (10.6)	216 (15.1)
ICU admission (31 days), n%	154 (7.0)	148 (10.3)
Composite endpoint (31-day mortality and/or ICU admission), n%	331 (15.1)	311 (21.7)

CCI: Charlson Comorbidity Index; ED: emergency department ICU: intensive care unit; IQR: interquartile range.

Figure 1. Receiver operating characteristics (ROC) curves for predicting 31-day mortality.

Figure 2. Calibration plots of the predicted 31-day mortality (x axis) using the RISE UP score (left panel) and the COPE score (right panel). the RISE UP score shows excellent calibration. The COPE score is also well calibrated, but shows average underestimation of 31- day mortality and a slope of >1.

Table 3. Comparison of discriminatory performance of the included prediction models.

Prediction model	31-day mortality (AUC, 95% CI)	Difference with RISE UP (p value)^a	Composite endpoint (mortality and/or ICU admission) (AUC, 95% CI)	Difference with RISE UP (p value)^a
RISE UP	0.86 (0.84-0.89)	Reference	0.81 (0.78-0.83)	Reference
COPE	0.86 (0.84-0.89)	0.821	0.81 (0.78-0.83)	0.830
4C mortality	0.81 (0.78-0.83)	<0.001	0.75 (0.72-0.77)	<0.001
Prytherch et al.	0.77 (0.74-0.80)	<0.001	0.71 (0.68-0.74)	<0.001
CURB-65	0.76 (0.73-0.79)	<0.001	0.71 (0.68-0.74)	<0.001
Asadollahi et al.	0.76 (0.73-0.79)	<0.001	0.70 (0.67-0.73)	<0.001
APACHE II	0.75 (0.72-0.78)	<0.001	0.75 (0.73-0.78)	<0.001
NEWS	0.75 (0.72-0.78)	<0.001	0.78 (0.75-0.81)	0.057
SSEPS	0.75 (0.72-0.78)	<0.001	0.74 (0.72-0.78)	<0.001
WPSS	0.75 (0.71-0.78)	<0.001	0.77 (0.75-0.80)	0.034
MEWS	0.74 (0.70-0.77)	<0.001	0.77 (0.74-0.79)	0.009
POCT	0.73 (0.69-0.75)	<0.001	0.67 (0.64-0.70)	<0.001
EWS	0.72 (0.68-0.75)	<0.001	0.74 (0.71-0.77)	<0.001
REMS	0.71 (0.68-0.75)	<0.001	0.69 (0.66-0.72)	<0.001
qSOFA	0.69 (0.65-0.72)	<0.001	0.70 (0.67-0.73)	<0.001
SPEED	0.68 (0.63-0.73)	<0.001	0.67 (0.62-0.72)	<0.001
SOFA	0.67 (0.64-0.71)	<0.001	0.67 (0.64-0.71)	<0.001
MSI	0.65 (0.61-0.69)	<0.001	0.66 (0.62-0.69)	<0.001
RAPS	0.59 (0.55-0.63)	<0.001	0.62 (0.58-0.65)	<0.001

4C: Coronavirus Clinical Characterization Consortium; APACHE II: Acute Physiology and Chronic Health Evaluation II; AUC: area und the curve; CI: confidence interval; COPE: COVID-19 Outcome Prediction in the Emergency Department; CURB-65: Confusion Urea Respiration Blood pressure; EWS: Early Warning Score; ICU: intensive care unit; MEWS: Modified Early Warning Score; MSI: Modified Shock Index; NEWS: National Early Warning Score; POCT: Point Of Care Test; RAPS: Rapid Acute Physiology Score; REMS: Rapid Emergency Medicine Score; RISE UP: Risk Stratification in the Emergency Department in Acutely Ill Older Patients; (q)SOFA: (quick) Sepsis-related Organ Failure Assessment; SPEED: Sepsis Patient Evaluation in the Emergency Department; SSEPS: Simple Sepsis Early Prognostic Score; WPSS: Worthing Physiology Scoring System.

^aComparison to the AUC of the prediction model with the AUC of the RISE UP score. The RISE UP and COPE score performed equally well.

Table 4. Comparison of discriminatory performance of the included prediction models in 1-day and 7-day mortality.

Prediction model	1-day mortality (AUC, 95% CI)	Difference with RISE UP (p value)^a	7-day mortality (AUC, 95% CI)	Difference with RISE UP (p value)^a
RISE UP	0.87 (0.81–0.93)	Reference	0.87 (0.84-0.90)	Reference
COPE	0.87 (0.81–0.92)	0.918	0.87 (0.85-0.90)	0.467
4C mortality	0.79 (0.72–0.87)	0.016	0.81 (0.77-0.85)	<0.001
Prytherch et al.	0.76 (0.69–0.85)	<0.001	0.77 (0.73-0.81)	<0.001
CURB-65	0.77 (0.67–0.86)	0.010	0.79 (0.74-0.83)	<0.001
Asadollahi et al.	0.72 (0.63–0.80)	<0.001	0.75 (0.71-0.79)	<0.001
APACHE II	0.87 (0.81–0.92)	0.942	0.82 (0.78-0.86)	0.012
NEWS	0.86 (0.81–0.92)	0.816	0.82 (0.77-0.86)	0.009
SSEPS	0.75 (0.65–0.85)	0.008	0.75 (0.70-0.80)	<0.001
WPSS	0.84 (0.76–0.91)	0.377	0.81 (0.77-0.85)	0.005
MEWS	0.83 (0.76–0.90)	0.213	0.80 (0.76-0.85)	0.001
POCT	0.68 (0.60–0.77)	<0.001	0.70 (0.65-0.75)	<0.001
EWS	0.82 (0.75–0.89)	0.125	0.78 (0.74-0.83)	<0.001
REMS	0.74 (0.63–0.84)	0.007	0.77 (0.71-0.82)	<0.001
qSOFA	0.77 (0.68–0.85)	0.013	0.72 (0.67-0.77)	<0.001
SPEED	0.70 (0.56–0.84)	0.092	0.69 (0.61-0.76)	<0.001
SOFA	0.82 (0.74–0.89)	0.244	0.73 (0.68-0.79)	<0.001
MSI	0.71 (0.61–0.82)	0.006	0.67 (0.61-0.74)	<0.001
RAPS	0.68 (0.57–0.79)	<0.001	0.68 (0.62-0.74)	<0.001

4C: Coronavirus Clinical Characterization Consortium; APACHE II: Acute Physiology and Chronic Health Evaluation II; AUC: area und the curve; CI: confidence interval; COPE: COVID-19 Outcome Prediction in the Emergency Department; CURB-65: Confusion Urea Respiration Blood pressure; EWS: Early Warning Score; ICU: intensive care unit; MEWS: Modified Early Warning Score; MSI: Modified Shock Index; NEWS: National Early Warning Score; POCT: Point Of Care Test; RAPS: Rapid Acute Physiology Score; REMS: Rapid Emergency Medicine Score; RISE UP: Risk Stratification in the Emergency Department in Acutely Ill Older Patients; (q)SOFA: (quick) Sepsis-related Organ Failure Assessment; SPEED: Sepsis Patient Evaluation in the Emergency Department; SSEPS: Simple Sepsis Early Prognostic Score; WPSS: Worthing Physiology Scoring System.

^aComparison to the AUC of the prediction model with the AUC of the RISE UP score. The RISE UP and COPE score performed equally well.

Table 5. Cut off values of the RISE UP and COPE score and prognostic accuracy for 31-day mortality.

Cut off value (%)	Patients (n%)	Deaths (n%) ^a	Sensitivity (%, 95% CI)	Specificity (%, 95% CI)	PPV (%, 95% CI)	NPV (%, 95% CI)
RISE UP score
<5	1149 (52.6)	10 (0.9)	95.7 (92.2-97.9)	58.3 (56.1-60.5)	21.3 (20.4-22.4)	99.1 (98.4-99.5)
<10	1547 (70.8)	48 (3.1)	79.2 (73.4-84.3)	76.7 (74.8-78.6)	28.7 (26.7-30.9)	96.9 (96.0-97.6)
5 to 20	714 (32.7)	231 (32.4)	41.6 (35.1-48.2)	68.4 (66.3-70.4)	13.5 (11.7-15.5)	90.8 (89.8-91.7)
>20	322 (14.7)	125 (38.8)	54.1 (47.5-60.7)	89.9 (88.5-91.2)	38.9 (34.8-43.2)	94.3 (93.5-95.0)
>30	173 (7.9)	78 (45.1)	33.8 (27.7-40.3)	95.1 (94.1-96.1)	45.2 (38.7-51.8)	92.4 (91.7-93.0)
COPE score
<5	1211 (55.4)	19 (1.6)	91.8 (87.5-95.0)	61.0 (58.8-63.2)	21.8 (20.7-23.0)	98.4 (97.6-99.0)
<10	1602 (73.3)	59 (3.7)	74.5 (68.3-80.0)	79.0 (77.1-80.8)	29.6 (27.2-32.0)	96.3 (95.4-97.0)
5 to 20	665 (30.4)	231 (32.4)	37.7 (31.4-44.3)	70.4 (68.3-72.4)	13.1 (11.2-15.3)	90.5 (89.6-91.4)
>20	309 (14.1)	125 (40.5)	54.1 (47.5-60.7)	90.6 (89.2-91.8)	40.5 (36.2-45.0)	94.3 (93.5-95.0)
>30	155 (7.1)	79 (51.0)	34.2 (28.1-40.7)	96.1 (95.2-96.9)	51.0 (44.0-58.1)	92.5 (91.8-93.1)

CI: confidence interval; COPE: COVID-19 outcome prediction in the ED; ICU: intensive care unit; NPV: negative predictive value; PPV: positive predictive value; RISE UP: Risk stratification in the emergency department in acutely ill older patients.

^aThe percentages of deaths are related to the number of patients above or below that cut off value.

Table 6. Cut-off values of the RISE UP and COPE score and prognostic accuracy for 31-day mortality or ICU admission.

Cut-off value (%)	Patients (n%)	Deaths or admissions to ICU (n%)^a	Sensitivity (%, 95% CI)	Specificity (%, 95% CI)	PPV (%, 95% CI)	NPV (%, 95% CI)
RISE UP score
<5	1149 (52.6)	48 (4.2)	85.5 (81.2–89.1)	59.4 (57.1–61.6)	27.2 (25.9–28.7)	95.8 (94.7–96.8)
<10	1547 (70.8)	107 (6.9)	67.7 (62.3–72.7)	77.7 (75.7–79.6)	35.0 (32.5–37.6)	93.1 (92.0–94.1)
5 to 20	714 (32.7)	331 (46.4)	40.2 (34.9–45.7)	68.7 (66.5–70.8)	18.6 (16.4–20.9)	86.6 (85.5–87.6)
>20	322 (14.7)	150 (46.6)	45.3 (39.9–50.9)	90.7 (89.3–92.0)	46.5 (41.9–51.1)	90.3 (89.4–91.2)
>30	173 (7.9)	93 (53.7)	28.1 (23.3–33.3)	95.7 (94.7–96.6)	53.7 (46.8–90.4)	88.2 (87.5–88.9)
COPE score
<5	1211 (55.4)	59 (4.9)	82.2 (77.6–86.2)	62.1 (59.9–64.4)	27.9 (26.3–29.4)	95.2 (93.9–96.1)
<10	1602 (73.3)	116 (7.2)	65.0 (59.6–70.1)	90.2 (78.3–81.9)	36.8 (34.0–39.6)	92.8 (91.7–93.7)
5 to 20	665 (30.4)	331 (46.4)	36.3 (31.1–41.7)	70.6 (68.5–72.7)	18.0 (15.8–20.5)	86.2 (85.1–87.2)
>20	309 (14.1)	152 (49.1)	45.9 (40.5–51.5)	91.5 (90.2–92.8)	49.1 (44.4–53.8)	90.5 (89.6–91.3)
>30	155 (7.1)	93 (60.0)	28.1 (23.3–33.3)	96.7 (95.7–97.4)	59.9 (52.6–66.8)	88.3 (87.6–89.0)

CI: confidence interval; COPE: COVID-19 outcome prediction in the ED; ICU: intensive care unit; NPV: negative predictive value; PPV: positive predictive value; RISE UP: Risk stratification in the emergency department in acutely ill older patients.

^aThe percentages of deaths or ICU admissions are related to the number of patients above or below that cut off value.

Knight SR, Ho A, Pius R, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score. BMJ. 2020;370:m3339.

 PubMed Web of Science ®Google Scholar

Knaus WA, Draper EA, Wagner DP, et al. APACHE II: a severity of disease classification system. Crit Care Med. 1985;13(10):818–829. doi: 10.1097/00003246-198510000-00009.

 PubMed Web of Science ®Google Scholar

Asadollahi K, Hastings IM, Gill GV, et al. Prediction of hospital mortality from admission laboratory data and patient age: a simple model. Emerg Med Australas. 2011;23(3):354–363. doi: 10.1111/j.1742-6723.2011.01410.x.

 PubMed Web of Science ®Google Scholar

van Klaveren D, Rekkas A, Alsma J, et al. COVID outcome prediction in the emergency department (COPE): using retrospective dutch hospital data to develop simple and valid models for predicting mortality and need for intensive care unit admission in patients who present at the emergency department with suspected COVID-19. BMJ Open. 2021;11(9):e051468. doi: 10.1136/bmjopen-2021-051468.

 PubMed Web of Science ®Google Scholar

Lim WS, van der Eerden MM, Laing R, et al. Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax. 2003;58(5):377–382. doi: 10.1136/thorax.58.5.377.

 PubMed Web of Science ®Google Scholar

Morgan R, Lloyd-Williams F, Wright MM, et al. An early warning score system for detecting developing critical illness. Clin Intens Care. 1997;8:100.

 Google Scholar

Subbe CP, Kruger M, Rutherford P, et al. Validation of a modified early warning score in medical admissions. QJM. 2001;94(10):521–526. doi: 10.1093/qjmed/94.10.521.

 PubMed Web of Science ®Google Scholar

Liu YC, Liu JH, Fang ZA, et al. Modified shock index and mortality rate of emergency patients. World J Emerg Med. 2012;3(2):114–117. doi: 10.5847/wjem.j.issn.1920-8642.2012.02.006.

 PubMedGoogle Scholar

Royal College of Physicians (RCP). National Early Warning Score (NEWS): standardising the assessment of acute illness severity in the NHS. Report of a Working Party. London: Royal College of Physicians; 2012.

 Google Scholar

Brink A, Schuttevaer R, Alsma J, et al. Predicting 30-day mortality using point-of-care testing; an external validation and derivation study. PLoS One. 2020;15(9):e0239318. doi: 10.1371/journal.pone.0239318.

 PubMed Web of Science ®Google Scholar

Rhee KJ, Fisher CJ, Jr., Willitis NH. The rapid acute physiology score. Am J Emerg Med. 1987;5(4):278–282. doi: 10.1016/0735-6757(87)90350-0.

 PubMed Web of Science ®Google Scholar

Olsson T, Terent A, Lind L. Rapid emergency medicine score: a new prognostic tool for in-hospital mortality in nonsurgical emergency department patients. J Intern Med. 2004;255(5):579–587. doi: 10.1111/j.1365-2796.2004.01321.x.

 PubMed Web of Science ®Google Scholar

Zelis N, Buijs J, de Leeuw PW, et al. A new simplified model for predicting 30-day mortality in older medical emergency department patients: the rise up score. Eur J Intern Med. 2020;77:36–43. doi: 10.1016/j.ejim.2020.02.021.

 PubMed Web of Science ®Google Scholar

Prytherch DR, Sirl JS, Schmidt P, et al. The use of routine laboratory data to predict in-hospital death in medical admissions. Resuscitation. 2005;66(2):203–207. doi: 10.1016/j.resuscitation.2005.02.011.

 PubMed Web of Science ®Google Scholar

Seymour CW, Liu VX, Iwashyna TJ, et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (sepsis-3). JAMA. 2016;315(8):762–774. doi: 10.1001/jama.2016.0288.

 PubMed Web of Science ®Google Scholar

Vincent JL, Moreno R, Takala J, et al. The SOFA (sepsis-related organ failure assessment) score to describe organ dysfunction/failure. On behalf of the working group on sepsis-related problems of the european society of intensive care medicine. Intensive Care Med. 1996;22(7):707–710. doi: 10.1007/BF01709751.

 PubMed Web of Science ®Google Scholar

Bewersdorf JP, Hautmann O, Kofink D, et al. The SPEED (sepsis patient evaluation in the emergency department) score: a risk stratification and outcome prediction tool. Eur J Emerg Med. 2017;24(3):170–175. doi: 10.1097/MEJ.0000000000000344.

 PubMed Web of Science ®Google Scholar

Liu B, Li D, Cheng Y, et al. Development and internal validation of a simple prognostic score for early sepsis risk stratification in the emergency department. BMJ Open. 2021;11(7):e046009. doi: 10.1136/bmjopen-2020-046009.

 PubMed Web of Science ®Google Scholar

Duckitt RW, Buxton-Thomas R, Walker J, et al. Worthing physiological scoring system: derivation and validation of a physiological early-warning system for medical admissions. An observational, population-based single-Centre study. Br J Anaesth. 2007;98(6):769–774. doi: 10.1093/bja/aem097.

 PubMed Web of Science ®Google Scholar

Data availability statement

Additional data are available upon reasonable request.