https://orcid.org/0000-0002-5024-5108
Abstract
Objective
To compare the predictive value of the quick COVID-19 Severity Index (qCSI) and the National Early Warning Score (NEWS) for 90-day mortality amongst COVID-19 patients.
Methods
Multicenter retrospective cohort study conducted in adult patients transferred by ambulance to an emergency department (ED) with suspected COVID-19 infection subsequently confirmed by a SARS-CoV-2 test (polymerase chain reaction). We collected epidemiological data, clinical covariates (respiratory rate, oxygen saturation, systolic blood pressure, heart rate, temperature, level of consciousness and use of supplemental oxygen) and hospital variables. The primary outcome was cumulative all-cause mortality during a 90-day follow-up, with mortality assessment monitoring time points at 1, 2, 7, 14, 30 and 90 days from ED attendance. Comparison of performances for 90-day mortality between both scores was carried out by univariate analysis.
Results
From March to November 2020, we included 2,961 SARS-CoV-2 positive patients (median age 79 years, IQR 66–88), with 49.2% females. The qCSI score provided an AUC ranging from 0.769 (1-day mortality) to 0.749 (90-day mortality), whereas AUCs for NEWS ranging from 0.825 for 1-day mortality to 0.777 for 90-day mortality. At all-time points studied, differences between both scores were statistically significant (p < .001).
Conclusion
Patients with SARS-CoV-2 can rapidly develop bilateral pneumonias with multiorgan disease; in these cases, in which an evacuation by the EMS is required, reliable scores for an early identification of patients with risk of clinical deterioration are critical. The NEWS score provides not only better prognostic results than those offered by qCSI at all the analyzed time points, but it is also better suited for COVID-19 patients.
This work aims to determine whether NEWS is the best score for mortality risk assessment in patients with COVID-19.
AUCs for NEWS ranged from 0.825 for 1-day mortality to 0.777 for 90-day mortality and were significantly higher than those for qCSI in these same outcomes.
NEWS provides a better prognostic capacity than the qCSI score and allows for long-term (90 days) mortality risk assessment of COVID-19 patients.
KEY MESSAGES
Introduction
Background
In its initial stages back in February-March 2020, the current coronavirus disease 2019 (COVID-19) pandemic was a global shock to health systems all around the world [Citation1]. In the course of time, the scientific community has developed procedures to adequately screen and manage the huge number of patients generated by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) [Citation2,Citation3].
Over the course of the pandemic, several challenges have been overcome in record time, remarkably, early virus detection techniques (antibody and antigen testing) and mass vaccination [Citation4,Citation5]. Nevertheless, the early identification of patients at high-risk of clinical deterioration is still being refined. Initially, patient categorization was complex and early warning scores already implemented in health systems were used, even if they were not specifically designed for COVID-19, as was the case for instance of the National Early Warning Score (NEWS) [Citation6,Citation7].
Importance
Throughout the pandemic, specific scores have been developed to assess disease severity in COVID-19 patients [Citation8–10]. The best performing scores have in common the use of laboratory or imaging variables (C-reactive protein, urea, leukocytes, chest computed tomography) not always available in certain healthcare environments [Citation11,Citation12]. Therefore, new simple and easily applicable scores capable of providing relevant risk discrimination capacity have been developed, such as the quick COVID-19 Severity Index (qCSI) [Citation13].
COVID-19 causes an increase in primary care centre consultations, ambulance transfers, emergency department visits, hospital and intensive care unit admissions, and finally, an increase in unexpected mortality. Severe cases present with bilateral pneumonias accompanied by multiorgan disease in which the ventilatory function is especially compromised [Citation14]. Under such circumstances, the early identification of cases at risk of deterioration in the short-term is critical for the strategic management of the pandemic. It is necessary to objectively prioritize those patients with higher probability of survival amongst those at higher risk; at this crossroads, early warning scores can help in the decision-making process [Citation15,Citation16]. Both the qCSI and the NEWS scores include, with different weights, parameters related to the ventilatory function (respiratory rate, oxygen saturation and supplemental oxygen administration), which is critical in the early detection of the risk of deterioration in patients with COVID-19 [Citation7,Citation17,Citation18]. Until now, however, no direct comparison has been done between both scores. This comparison seems mandatory since NEWS is considered the gold standard of early warning scores, and such comparison could determine the added value of the new score.
Goals of this investigation
The present study aims to compare the predictive values of the qCSI and the NEWS scores for 90-day mortality amongst patients with COVID-19 transferred to the emergency department by emergency medical services
Methods
Study population
The inclusion criteria were as follows: Adult patients (>18 years) with COVID-19 infection confirmed by SARS-CoV-2 test (polymerase chain reaction) and transferred by ambulance to EDs. The exclusion criteria included: patients under 18 years of age, without analytical confirmation of infection, or cases in which the lack of any variable impeded the estimation of the scores analyzed.
Study design
This was a multicenter retrospective cohort study analyzing health data from March to November 2020, from an overall reference population of 1,166,408 inhabitants. The study was carried out in the provinces of Palencia, Salamanca, Segovia and Valladolid (Spain) with the participation of 61 ambulance services and EDs from 8 hospitals (three tertiary university hospitals and five general district hospitals). Both the Emergency Medical Services (EMS) and the hospitals are managed by the Public Health System of Castilla-León (SACYL), the principal health operator.
The study was approved by the local institutional research review board of Rio Hortega Hospital (PI 138/20) and conformed to the Declaration of Helsinki principles. The institutional review board granted a waiver of the obligation to collect consent from study participants due to the use of unidentified subjects.
Outcome
The outcome was cumulative all-cause mortality during a 90-day follow-up, with mortality assessment at the following monitoring time points: 1, 2, 7, 14, 30 and 90 days from ED attendance. All monitoring time points were also cumulative, i.e. deaths registered at a particular point also include those registered at the preceding ones, e.g. 14-day mortality includes also patients of the preceding time points 1, 2, 3 and 7 days.
Predictors and data abstraction
Epidemiological data (sex, age, rural or urban area and nursing home origin) and clinical covariates (respiratory rate, oxygen saturation, systolic blood pressure, heart rate, temperature, level of consciousness and use of supplemental oxygen) were collected by an emergency registered nurse at the triage emergency room, so these variables were blinded to the clinical investigators.
The Connex® Vital Signs Monitor (Welch Allyn, Inc., Skaneateles Falls, NY) was used to measure blood pressure, heart rate, temperature and oxygen saturation. The respiratory frequency was measured by direct observation of the respiratory cycles for 30 s; in case of irregular breathing or extreme ranges, it was measured by direct auscultation for 1 min. The level of consciousness was assessed by means of the Glasgow Coma Scale; a score of less than 15 was considered alteration of the level of consciousness. Finally, the percentage of oxygen was evaluated by means of the fraction of inspired oxygen supplied, from which the litres per minute of oxygen administered were calculated.
Ninety days from the first care, an independent clinical investigator from each hospital reviewed the patient's electronic medical record and collected mortality data, destination (discharge on site, admission for hospitalization or intensive care unit) and 17 categories of comorbidities necessary to calculate the Charlson comorbidity index.
The NEWS and qCSI scores were calculated using vital sign data and clinical observations in accordance with the Royal College of London standards for NEWS (note that NEWS or NEWS2 were used indistinctly, and both refer to NEWS2) [Citation19,Citation20] and Haimovich et al. [Citation13] for qCSI (see Supplementary Data S1 and S2). In particular, values for both scores were calculated with physiological measurements obtained from the electronic medical record (collected on the patient's arrival at the ED). Mortality data at 1, 2, 3, 7, 14, 30 and 90 days were also obtained from the electronic health records. The score values were subsequently used to analyze the predictive capacity of both scores for mortality at the different time points.
Primary data analysis
Categorical variables were represented by absolute values and percentages. Continuous variables were represented by median and interquartile range (IQR) since they did not follow a normal distribution. For the characterization of the total sample and to analyze the association between each independent variable and the primary outcome (90-day mortality), the Mann-Whitney U test or chi-squared test was performed as appropriate.
The discriminatory validity of the scores was assessed by the area under the receiver operating characteristic (ROC) curve (AUC), calculating in each case the p value of the hypothesis contrast. The graphs of the ROC curves show the confidence interval (95% CI) obtained by resampling (or bootstrapping) 2000 realizations. Finally, the specificity, sensitivity, positive predictive value, negative predictive value, positive likelihood ratio and negative likelihood ratio of the score obtained were calculated. With the objective of comparing ROCs, a Delong’s test and a decision curve analysis were used. The effect of confounding factors (i.e. age, sex, and comorbidities, evaluated by the Charlson Age Comorbidity Index [CACI]) on the predictive value of both scores was assessed by a multivariate analysis. Data were analyzed using our own codes and base functions in R, version 4.0.3 (http://www.R-project.org; the R Foundation for Statistical Computing, Vienna, Austria).
Results
Baseline characteristics
A total of 3183 patients met the inclusion criteria. After applying the exclusion criteria, the final sample for analysis consisted of 2961 patients with confirmed SARS-CoV-2 infection (see Supplementary Figure S1).
Patients were mostly older adults (median age 79 years, IQR 66-88, range 18–104), with an almost uniform gender balance (49.2% females). The hospitalization rate was 78.6%, with 5.5% requiring intensive care unit (ICU) admission. The cumulative all-cause mortality at the monitoring time points 1, 2, 7, 14, 30 and 90 days was 5.7% (169 cases), 8.2% (243 cases), 18.1% (537 cases), 24.2% (718 cases), 27.9% (827 cases) and 32% (948 cases), respectively ().
Table 1. Demographic and clinical characteristics for 90-day mortality.
Scores’ discrimination
The discrimination capacity of each score for 1-day and 90-day mortality was assessed analyzing the distribution of survivors and non-survivors and the predicted probability of mortality according to different values of the score (). For NEWS, higher score values included a higher proportion of non-survivors and predicted a higher probability of mortality, both for 1-day () and 90-day () mortality. Moreover, lower score values predicted higher probability of death at 90 days than at 1 day. qCSI showed a similar distribution for 1-day () and 90-day () mortality, although it provided lower probabilities of death than NEWS throughout the whole range of score values. Similar discrimination results at the other time points (2, 7, 14 and 30 days) can be found in the Supplementary Data S3.
Figure 1. Predicted probability of death and observed distribution of patients across score value of NEWS for 1-day (A) and 90-day mortality (B) and qCSI for 1-day (C) and 90-day mortality (D). The grey area of the trend line corresponds to 95% confidence interval of the predicted probability of death (trend line). The bars correspond to the number of patients alive (grey) or dead (black) in the training cohort. The values within parenthesis refer to the range of score values included in each bar.
Scores comparison
The predictive capacity of each score for mortality was assessed by ROC curves analysis and by decision curves. The comparison of results for both scores for mortality at 1 day and 90 days is shown in . Similar comparisons were carried out for the intermediate time points (Supplementary Data S4). Although AUCs decreased with increasing times for both scores, AUC values were higher for the NEWS score than for qCSI throughout the whole range of time points (), indicating that NEWS has a better performance for mortality prediction. Finally, regarding statistical parameters related to AUCs, NEWS showed better sensitivity, positive predictive value, negative predictive value and positive likelihood ratio than qCSI, whereas its specificity and negative likelihood ratio were worse ().
Figure 2. Predictive validity results of each model. Decision curve analysis for 1-day (A) and 90-day mortality (C) and the area under the receiver operating characteristic (ROC) curves for 1-day (B) and 90-day mortality (D) of qCSI and NEWS. Grey line corresponds to qCSI results and the black line to NEWS results.
Table 2. Predictive validity of qCSI and NEWS for different mortality time points.
Table 3. Statistical details of the qCSI and NEWS for different point times analyzed.
Finally, we carried out a multivariate analysis to assess the effect of putative confounding variables such as age, sex and comorbidities (CACI) on the predictive ability of both scores for mortality (Supplementary Data 5). For short mortality times (1 and 2 days) no effect was found. For 7-day mortality, being >89 years was a risk factor for both NEWS and qCSI and being male for NEWS. For longer times all confounding variables were found as risk factors for mortality.
Discussion
In the present study, the NEWS score showed a better ability to predict mortality in patients with COVID-19 than the qCSI score at all-time points analyzed during a 90-day follow-up period.
In an emergency situation, such as the current COVID-19 pandemic, it is critical to accurately predict the risk of clinical deterioration in order to balance provisions of life-saving care and to steward precious resources [Citation14,Citation21]. In this sense, early warning scores are reliable tools in such critical decisions [Citation22]. For instance, NEWS is commonly used in several clinical situations [Citation23–26] and has been validated for COVID-19 [Citation7] . The qCSI, instead, was only internally validated [Citation13], and there is limited experience in its implementation [Citation27].
In the cohort of the present study, NEWS had an AUC of 0.825 for 1-day mortality, in line with recent studies [Citation16,Citation17,Citation28,Citation29], performing better than qCSI, which showed an AUC of 0.769. In the study by Haimovich et al. [Citation13], where qCSI was established and validated, AUC of this score was 0.82, though the cohort used was smaller than ours. Importantly, in our study, the performance of both scores was not affected by age, sex or comorbidities up to 2 days, revealing their short-term potential. However, at longer mortality times, from 7 to 90 days, all the confounding factors were significantly related to mortality. This result was somehow expected, since at shorter times, initial deterioration of the patients may weigh more than intrinsic factors such as age sex and comorbidities.
Severe cases of SARS-CoV-2 are characterized by significant compromise of ventilatory function with breathlessness, desaturation and tachypnoea often requiring the administration of supplemental oxygen or invasive mechanical ventilation [Citation30]. Both scores account (although with different weights) for ventilatory measurements, but NEWS additionally includes the cardio-vascular function (systolic blood pressure and heart rate), neurological function (Glasgow coma scale), and temperature. The addition of these parameters, a priori, makes it a more complete score than qCSI. On the other hand, unlike NEWS, qCSI focuses only on the analysis of ventilatory function, which could be seen as an advantage, especially considering that it only requires three easy-to-obtain variables.
In critical circumstances with a volume of patients exceeding the operational capabilities of health systems, an appropriate selection of those cases with the highest risk of clinical deterioration is mandatory. In this sense, qCSI is a simple score, easy to apply, but with a worse performance than NEWS. However, NEWS has been implemented in multiple health systems, tested in a wide variety of clinical contexts, and has a high prognostic performance [Citation31–34]. On the other hand, NEWS requires collection of more variables than the qCSI, an issue that should be considered a handicap during the pandemic. In short, both scores have strengths and weaknesses. However, based on more comprehensive scientific evidence, its implementation in multiple clinical environments, its robustness from a statistical point of view, and its better prognostic performance, NEWS could be considered a better suited score than qCSI for the EDs. This statement should be considered however with the warning message that this was a retrospective study; NEWS requires additional efforts [Citation18,Citation29,Citation35,Citation36], and it was not developed exclusively for COVID-19 patients.
During the first and second waves of the ongoing COVID-19 pandemic the initial patient/resource ratio was very high. This overload revealed the necessity of appropriate triage systems to apply the limited resources to critical patients. For instance, being classified as a high-risk patient (NEWS ≥ 7) is a strong predictor of early clinical impairment. Therefore, the use of scoring systems, such as NEWS, is critical to assist in initial triage, both at the scene and in the ED, helping to manage hospital and ICU admissions in a more efficient way, and to guide decisions on transfer to the ED. Nevertheless, its usefulness goes beyond this, because at extraordinary times like the present, it will allow us to know with certainty and in a simple way the short-term vital perspective of the patient. The availability of these scores will help the health system to better organize and manage limited resources, and also will allow us to inform the relatives of seriously ill patients of the possible outcome.
This study has several limitations. First, this is a retrospective study carried out during a pandemic. Due to the extreme burden of care at certain stages, the completion of clinical histories could have been compromised with an increase in the amount of missing data. To avoid bias, a multicenter study with a significant number of patients was planned. Second, due to the multiplicity of existing scores, a partial selection was made. Of all the early warning scores, NEWS is the one supported by the most scientific evidence, the highest degree of implementation, and by applications in diverse clinical contexts and pathologies including patients with COVID-19 [Citation7,Citation16,Citation17]. Third, we avoided using scores that require analytical inputs (e.g. CURB-65) or parameters that are complex to quantify during the first care. Fourth, it should be highlighted that this study was developed by considering only those patients requiring EMS, which are arguably in a poorer clinical state as compared with those non-evacuated by ambulance. Finally, qCSI grounded its prognostic capacity on early clinical worsening during the first 24 h (necessity of high-flow oxygen, invasive or non-invasive mechanical ventilation, and death), whereas in the present study 90-day mortality was taken as the primary outcome variable. To solve this, we included 1-day mortality as an outcome, because qCSI was established and validated at this time point [Citation13].
In summary, patients with SARS-CoV-2 infection can rapidly evolve to bilateral pneumonias with multiorgan disease; in these cases, an early identification of patients at high-risk of clinical deterioration should be prioritized. The NEWS score outperformed the qCSI score in predicting mortality at all studied time points, from 1 day to 90 days. The standardized use of early warning scores in patients evacuated by the EMS can aid in the complex decision-making process, assisting healthcare workers in the initial identification of the most severe patients and supporting the best allocation of resources.
Transparency declaration
The corresponding author on behalf of the other authors guarantee the accuracy, transparency and honesty of the data and information contained in the study, that no relevant information has been omitted and that all discrepancies between authors have been adequately resolved and described.
The following work has not been previously published and is not under consideration by any other scientific journal.
Author statement
Drs Martín-Rodríguez and López-Izquierdo had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. Study concept and design: All authors. Acquisition, analysis, or interpretation of data: Delgado Benito, Mazas Pérez-Oleaga and García Villena. Drafting of the manuscript: Martín-Rodríguez and Sanz-García. Final approval, critical revision of the manuscript for important intellectual content: All authors. Statistical analysis: Sanz García and Ortega Rabbione. Obtained funding: Martín-Rodríguez. Administrative, technical, or material support: Mazas Pérez-Oleaga and García Villena. Study supervision: Castro Villamor. All authors agree to be accountable for all aspects of the work. Drs Martín-Rodríguez and López-Izquierdo are joint first authors.
Supplemental Material
Download Zip (1.3 MB)Disclosure statement
No potential conflict of interest was reported by the authors.
Data availability statement
Data are available upon a reasonable request to the corresponding author in https://figshare.com/s/445db8346441bd09aba3.
Additional information
Funding
References
- Acter T, Uddin N, Das J, et al. Evolution of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) as coronavirus disease 2019 (COVID-19) pandemic: a global health emergency. Sci Total Environ. 2020;730:138996.
- Piva S, Filippini M, Turla F, et al. Clinical presentation and initial management critically ill patients with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infection in Brescia, Italy. J Crit Care. 2020;58:29–33.
- Giangola M, Siskind S, Faliks B, et al. Applying triage principles of mass casualty events to the SARS-CoV-2 pandemic: from the perspective of the acute care surgeons at Long Island jewish medical center in the COVID epicenter of the United States. Surgery. 2020;168(3):408–410.
- Cheng MP, Papenburg J, Desjardins M, et al. Diagnostic testing for severe acute respiratory Syndrome-Related coronavirus 2: a narrative review. Ann Intern Med. 2020;172(11) :726–734.
- Maltezou HC, Theodoridou K, Poland G. Influenza immunization and COVID-19. Vaccine. 2020;38(39):6078–6079.
- Fernando SM, Fox-Robichaud AE, Rochwerg B, et al. Prognostic accuracy of the Hamilton early warning score (HEWS) and the national early warning score 2 (NEWS2) among hospitalized patients assessed by a rapid response team. Crit Care. 2019;23(1):60.
- Kostakis I, Smith GB, Prytherch D, et al. The performance of the national early warning score and national early warning score 2 in hospitalised patients infected by the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Resuscitation. 2021;159:150–157.
- Liang W, Liang H, Ou L, et al. Development and validation of a clinical risk score to predict the occurrence of critical illness in hospitalized patients with COVID-19. JAMA Intern Med. 2020;180(8):1081–1089.
- Bartoletti M, Giannella M, Scudeller L, et al. Development and validation of a prediction model for severe respiratory failure in hospitalized patients with SARS-CoV-2 infection: a multicentre cohort study (PREDI-CO study). Clin Microbiol Infect. 2020;26(11):1545–1553.
- Covino M, De Matteis G, Burzo ML, et al. Predicting in-Hospital mortality in COVID-19 older patients with specifically developed scores. J Am Geriatr Soc. 2021;69(1):37–43.
- Knight SR, Ho A, Pius R, et al. Risk stratification of patients admitted to hospital with covid-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score. BMJ.2020;370:m3339.
- Qin L, Yang Y, Cao Q, et al. A predictive model and scoring system combining clinical and CT characteristics for the diagnosis of COVID-19. Eur Radiol. 2020;30(12):6797–6807.
- Haimovich AD, Ravindra NG, Stoytchev S, et al. Development and validation of the quick COVID-19 severity index: a prognostic tool for early clinical decompensation. Ann Emerg Med. 2020;76(4):442–453.
- Maves RC, Downar J, Dichter JR, et al. Triage of scarce critical care resources in COVID-19 an implementation guide for regional allocation: an expert panel report of the task force for mass critical care and the American college of chest physicians. Chest. 2020;158(1):212–225.
- Covino M, Sandroni C, Santoro M, et al. Predicting intensive care unit admission and death for COVID-19 patients in the emergency department using early warning scores. Resuscitation. 2020;156:84–91.
- Gidari A, De Socio GV, Sabbatini S, et al. Predictive value of national early warning score 2 (NEWS2) for intensive care unit admission in patients with SARS-CoV-2 infection. Infect Dis (Lond). 2020;52(10) :698–704.
- Myrstad M, Ihle-Hansen H, Tveita AA, Andersen EL, et al. National early warning score 2 (NEWS2) on admission predicts severe disease and in-hospital mortality from covid-19 – a prospective cohort study. Scand J Trauma Resusc Emerg Med. 2020;28(1):66.
- Carr E, Bendayan R, Bean D, et al. Evaluation and improvement of the national early warning score (NEWS2) for COVID-19: a multi-hospital study. BMC Med. 2021;19(1):23.
- Royal College of Physicians (London). National Early Warning Score (NEWS): standardising the assessment of acute illness severity in the NHS. Report of a working party London; 2012.
- Royal College of Physicians (London). National Early Warning Score (NEWS) 2: standardising the assessment of acute-illness severity in the NHS. Updated report of a working party London; 2017.
- Kirkpatrick JN, Hull SC, Fedson S, et al. Scarce-Resource allocation and patient triage during the COVID-19 pandemic: JACC review topic of the week. J Am Coll Cardiol. 2020;76(1):85–92.
- Meylan S, Akrour R, Regina J, et al. An early warning score to predict ICU admission in COVID-19 positive patients. J Infect. 2020;81(5):816–846.
- Silcock DJ, Corfield AR, Gowens PA, Rooney KD. Validation of the national early warning score in the prehospital setting. Resuscitation. 2015;89:31–35.
- Hodgson LE, Dimitrov BD, Congleton J, et al. A validation of the national early warning score to predict outcome in patients with COPD exacerbation. Thorax. 2017;72(1):23–30.
- Churpek MM, Carey KA, Dela Merced N, et al. Validation of early warning scores at two Long-Term acute care hospitals. Crit Care Med. 2019;47(12):e962–e965.
- Martín-Rodríguez F, López-Izquierdo R, Del Pozo Vegas C, et al. Accuracy of national early warning score 2 (NEWS2) in prehospital triage on in-Hospital early mortality: a Multi-Center observational prospective cohort study. Prehosp Disaster Med. 2019;34(6):610–618.
- Rodriguez-Nava G, Yanez-Bello MA, Trelles-Garcia DP, et al. Performance of the quick COVID-19 severity index and the Brescia-COVID respiratory severity scale in hospitalized patients with COVID-19 in a community hospital setting. Int J Infect Dis. 2021;102:571–576.
- Fan G, Tu C, Zhou F, et al. Comparison of severity scores for COVID-19 patients with pneumonia: a retrospective study. Eur Respir J. 2020;56(3):2002113.
- Baker KF, Hanrath AT, Schim van der Loeff I, et al. National early warning score 2 (NEWS2) to identify inpatient COVID-19 deterioration: a retrospective analysis. Clin Med. 2021;21(2) :84–89.
- Lentz S, Roginski MA, Montrief T, et al. Initial emergency department mechanical ventilation strategies for COVID-19 hypoxemic respiratory failure and ARDS. Am J Emerg Med. 2020;38(10):2194–2202.
- Su Y, Ju MJ, Xie RC, et al. Prognostic accuracy of early warning scores for clinical deterioration in patients with COVID-19. Front Med (Lausanne). 2020;7:624255.
- Martín-Rodríguez F, López-Izquierdo R, Del Pozo Vegas C, et al. Can the prehospital national early warning score 2 identify patients at risk of in-hospital early mortality? A prospective, multicenter cohort study. Heart Lung. 2020;49(5):585–591.
- Hwang JI, Chin HJ. Relationships between the national early warning score 2, clinical worry and patient outcome at discharge: retrospective observational study. J Clin Nurs. 2020;29(19–20):3774–3789.
- Martín-Rodríguez F, Sanz-García A, Melero Guijarro L, et al. Comorbidity-adjusted NEWS predicts mortality in suspected patients with COVID-19 from nursing homes: Multicentre retrospective cohort study. J Adv Nurs. 2021;1–14.
- Richardson D, Faisal M, Fiori M, et al. Use of the first national early warning score recorded within 24 hours of admission to estimate the risk of in-hospital mortality in unplanned COVID-19 patients: a retrospective cohort study. BMJ Open. 2021;11(2):e43721.
- Martín-Rodríguez F, Sanz-García A, del Pozo Vegas C, et al. Time for a prehospital-modified sequential organ failure assessment score: an ambulance–based cohort study. Am J Emerg Med. 2021;49(49):331–337.