https://orcid.org/0000-0002-3037-122X
?Mathematical formulae have been encoded as MathML and are displayed in this HTML version using MathJax in order to improve their display. Uncheck the box to turn MathJax off. This feature requires Javascript. Click on a formula to zoom.Abstract
Background
There is a paucity of prognostic models for COVID-19 that are usable for in-office patient assessment in general practice (GP).
Objectives
To develop and validate a risk prediction model for hospital admission with readily available predictors.
Methods
A retrospective cohort study linking GP records from 8 COVID-19 centres and 55 general practices in the Netherlands to hospital admission records. The development cohort spanned March to June 2020, the validation cohort March to June 2021. The primary outcome was hospital admission within 14 days. We used geographic leave-region-out cross-validation in the development cohort and temporal validation in the validation cohort.
Results
In the development cohort, 4,806 adult patients with COVID-19 consulted their GP (median age 56, 56% female); in the validation cohort 830 patients did (median age 56, 52% female). In the development and validation cohort respectively, 292 (6.1%) and 126 (15.2%) were admitted to the hospital within 14 days, respectively. A logistic regression model based on sex, smoking, symptoms, vital signs and comorbidities predicted hospital admission with a c-index of 0.84 (95% CI 0.83 to 0.86) at geographic cross-validation and 0.79 (95% CI 0.74 to 0.83) at temporal validation, and was reasonably well calibrated (intercept −0.08, 95% CI −0.98 to 0.52, slope 0.89, 95% CI 0.71 to 1.07 at geographic cross-validation and intercept 0.02, 95% CI −0.21 to 0.24, slope 0.82, 95% CI 0.64 to 1.00 at temporal validation).
Conclusion
We derived a risk model using readily available variables at GP assessment to predict hospital admission for COVID-19. It performed accurately across regions and waves. Further validation on cohorts with acquired immunity and newer SARS-CoV-2 variants is recommended.
KEY MESSAGES
A general practice prediction model based on signs and symptoms of COVID-19 patients reliably predicted hospitalisation.
The model performed well in second-wave data with other dominant variants and changed testing and vaccination policies.
In an emerging pandemic, GP data can be leveraged to develop prognostic models for decision support and to predict hospitalisation rates.
Introduction
When the worldwide spread of SARS-CoV-2 started, many research initiatives emerged, almost exclusively in secondary care settings [Citation1,Citation2]. However, most patients initially contact their general practitioner (GP) and research initiatives in this specific setting could have the largest impact [Citation3]. Risk assessment based on evaluating patient’s signs and symptoms can be complex and uncertain due to the highly variable clinical course of COVID-19 [Citation4–6]. Multivariable prognostic models can help by providing patient-specific risks based on multiple predictors. Many prognostic models for COVID-19 exist but the vast majority are developed for prognostication after presentation in secondary care and of low quality [Citation2]. The few models created on primary care data are not suitable to assist in prognostication following a physical assessment of symptomatic patients, as they do not take any symptoms or vital signs into account or are intended for remote assessment [Citation7,Citation11].
During the first COVID-19 wave, Dutch GP COVID-19 centres provided 24/7 care for their region [Citation12]. During daytime hours, patients were triaged by regular practices, and those in which further physical assessment was needed were referred to the adjacent GP COVID-19 centre. During out-of-hours, the telephone team of the regular GP out-of-hours cooperative triaged patients and referred them to the COVID-19 location if deemed necessary. GPs in COVID-19 centres routinely assessed and recorded symptoms and vital parameters such as temperature, oxygen saturation, and respiratory rate. This provided a rich data source on the early symptoms and natural course of illness.
Despite high levels of protection within the community against SARS-CoV-2 due to acquired immunity, novel SARS-CoV-2 variants of concern continuously emerge and cause seasonal surges in hospital admissions. A model consisting of routinely available predictors that can accurately estimate hospitalisation risk due to COVID-19 after physical assessment by the GP may aid in selecting high-risk COVID-19 patients that might benefit most from early interventions aimed to reduce complications and prevent hospital admission. Integrating such models in routine care electronic health records systems would allow for continuous validation of model performance and recalibration of the model’s coefficients where needed by linking predicted risks based on GP data to hospital admission data.
This study aimed to develop a model to estimate 14-day hospital admission risk based on a combination of comorbidities, vital signs and symptoms of patients presenting with COVID-19 for a face-to-face consultation in general practice and to geographically cross-validate and temporally validate the model in a later wave.
Methods
Development and validation cohorts
The development database describes a cohort of adult patients presenting in eight Dutch GP COVID-19 centres from three regions (Western South Limburg, Eastern South and Central Limburg, and North Limburg) between 01/03/2020 and 31/05/2020, merged with the hospital admission records in the same regions until 15/06/2020. Inclusion criteria were face-to-face GP consultations with codes related to COVID-19, using the Dutch College of General Practitioners’ adaptation of ICPC (international classification of primary care) codes (see Tables S1–S3) [Citation13]. Patients were excluded if a home visit to determine the death of a patient was not preceded by a COVID-19-related consultation in the study period. Other exclusion criteria were visits unrelated to COVID-19 complaints and home visits, resulting in patients deciding jointly with their GP not to visit the hospital due to a poor prognosis. Only the first visit for each patient is included in the current analysis.
The temporal validation database describes a cohort of adult patients presenting in 55 general practices in North Limburg between 13/03/2021 and 05/06/2021, merged with the hospital admission records from VieCuri (Venlo) until 19/06/2021. Only patients with confirmed COVID-19 were included; otherwise, in- and exclusion criteria were outlined above. Widespread testing and vaccination were available in the validation cohort and the dominant strains were alpha and delta, whereas it was the original (Wuhan) strain in the development cohort.
Patient data was pseudonymised after an independent third party hashed GP and hospital records and before collecting data in the electronic case report form.
Outcomes
The primary outcome was hospital admission with COVID-19 within 14 days of the COVID-19-related consultation by the GP. The COVID-19 diagnosis was established according to local hospital protocol, which could include (repeated) SARS-CoV-2 RT-PCR testing and/or CoRADS-score 4 or 5 at pulmonary CT scans [Citation14]. We also included GP-established (all-cause) mortality within 14 days but before hospital admission as an event in a composite outcome for the development and validation of the prognostic model as a safeguard against replicating potential inequities in case patients were denied access to hospital-based care at the peak of the pandemic, e.g. due to old age or comorbidity. For brevity, and because GP-established death within 14 days was infrequent, we will refer to the prediction of the composite endpoint as the prediction of hospital admission within 14 days. The secondary outcomes were hospital admission with COVID-19, ICU admission and in-hospital mortality at any time during the data collection period (minimum 14 days to maximum 106 and 98 days in the development and validation cohort, respectively). No blinding occurred but radiologists interpreting CT scans were unaware of GP findings per usual clinical practice. Those deciding on admissions were aware of GP findings per usual clinical practice.
Predictors
This study used an electronic case report form in CASTOR EDC to extract data on repeated visits within the study period from GP databases, blinded for hospital data. The candidate predictors were selected based on the literature and prioritised based on frequency of occurrence [Citation2,Citation9,Citation15], availability in general practice and subjective impression of importance among general practitioners (based on a poll at Maastricht University Primary Care Department). We considered age (in years), sex, overweight (BMI > 25, patient or physician reported), and current smoker as demographic and lifestyle predictors. Vital functions measured during the first GP visit included temperature, heart rate, respiratory rate, peripheral oxygen saturation, auscultation, duration of complaints, newly emerging confusion, chest pain or pressure, cough, sputum, haemoptysis, digestive complaints, fall event, dyspnoea, headache, and sudden worsening of complaints. The following comorbidities, events in the medical history, and use of drugs were candidate predictors: diabetes, history of cerebrovascular accident or transient ischaemic attack, chronic kidney disease, treatment for current malignancy, treatment for malignancy in the past, pre-existing hypertension, history of myocardial infarction or heart failure, chronic lung disease (incl. COPD and asthma), dementia, use of anticoagulants, use of immunosuppressives, and chronic use of corticosteroids.
Statistical analysis
Sample size calculations assumed a hospital admission percentage of 5% at the start of the project, which was adjusted to 6.6% after data collection. Assuming a C-index of 0.80 and a shrinkage factor of 0.90, 4,790 patients would be needed to develop a model with the 44 degrees of freedom needed to test all candidate predictors and non-linear effects [Citation16]. A priori, 1,000 patients for temporal validation would enable us to estimate the C-index with an error margin (confidence interval half-width) of 0.05. Based on this calculation, the validation cohort includes all GP visits during the peak of the second wave in North Limburg.
This study performed multiple imputations to handle missing predictor data (additional material S4) and modelled hospital admission using a logistic regression model built in the development cohort. Backward predictor selection (alpha 0.157) followed the a priori selection described above. Continuous predictors were modelled using restricted cubic splines with 3 knots to allow non-linear effects, and uniform shrinkage (based on the bootstrapped calibration slope) was applied.
The model was first evaluated in the first-wave development cohort using geographic leave-region-out cross-validation summarised with random effects meta-analysis [Citation17,Citation18]. Leaving out one region at a time for model development and evaluating the model in the left-out region, this procedure evaluates the model’s generalisability across regions. The developed prediction model (on all regions) was then assessed in the second-wave temporal validation cohort after filling in the predictor values in the prediction model equation. This evaluates the predictive performance after significant temporal changes occurred but is limited to North Limburg. In both cohorts, we assessed the C-index, calibration intercept, calibration slope and flexible (loess) calibration plots [Citation17]. Pre-specified subgroup analyses were done based on patient sex and vaccination status.
We used the mice [Citation19], RMS [Citation20], psfmi and meta-packages in R [Citation21,Citation22].
Ethics and open science
This study was approved by the ethical committee of the University Hospital Maastricht and Maastricht University) (METC azM/UM), the Netherlands and reported according to the TRIPOD guidelines [Citation23]. The project metadata is publicly available at https://dataverse.nl/dataverse/pro-covid.
Results
Development cohort
A total of 4,806 patients were included in the development cohort, with a median age of 56 years and 56.4% female (, ). Most patients presented with dyspnoea or cough; the median body temperature was 37.1 degrees Celsius (). Common comorbidities were hypertension and chronic pulmonary disease (). Within 14 days after the first GP visit, 292 patients (6.1%) were hospitalised, and 23 patients (0.5%) died at home ().
Figure 1. Flow chart of in- and exclusions.
Table 1a. Patient demographics and clinical features are used for the development (n = 4,806) and validation (n = 830) cohort separately.
Table 1b. Reported symptoms and vital signs were recorded during the first GP consultation, for the development (n = 4,806) and validation (n = 830) cohort separately.
Table 2. Outcomes for the development (n = 4,806) and validation (n = 830) cohort separately.
The multivariable prediction model fitted on the development data showed that being male, auscultation abnormalities, confusion, cough, haemoptysis, digestive complaints, headache, chronic kidney disease, lower oxygen saturation and higher body temperature increased the probability of hospital admission (Box 1, Table S5). Smoking, chest pain/discomfort, sputum production, current or past treatment for malignancy, and COPD decreased the probability. The probability of hospital admission within 14 days peaks for 60-year-olds and around seven days of complaint (Figure S6). Only 1.7% of patients had predicted admission risks > 50% (Figure S7). The model could distinguish between patients who do and do not get admitted within 14 days in all three regions, with a leave-region-out cross-validated pooled C-index of 0.84 (, Figure S8). The predicted risks corresponded well to observed admission risks in each region and sex subgroups (, Figure S7).
Table 3. Model performance (leave-region-out cross-validation and temporal validation).
Validation cohort
The validation cohort consisted of 830 patients, with a similar age and gender distribution to the development cohort (median age 56, 51.8% female; , ). Cough and dyspnoea were still the most prevalent symptoms at presentation, but more patients presented with non-specific symptoms such as myalgia and fatigue (). The validation cohort appeared generally healthier than the development cohort concerning lifestyle and pre-existing comorbidities () but the frequency of hospital admissions (15.2%) and other adverse outcomes were higher (). The model performed slightly inferior in the second-wave temporal validation cohort compared to the first-wave geographic cross-validation but it was still able to estimate hospital admission risks accurately () and discriminate between those who would and would not be admitted within 14 days (C-index 0.79, ).
Figure 2. Flexible calibration curve in the temporal validation cohort (n = 830, 133 events) showing the predicted probability of hospital admission versus the observed proportion with hospital admission. The histogram on the horizontal axis shows the distribution of predicted risks among patients with (1, bars pointing upward) and without (0, bars pointing downward) hospital admission. Very short bars at predicted probabilities >0.5 indicate high predicted admission risks are rare.
During the validation period, 193 patients (23%) received one vaccination and 95 (11%) received two or more vaccinations. However, only 82 were vaccinated before their COVID-19-related GP visit, of which 15 (18%) were hospitalised. Fifty-five were vaccinated at least two weeks before their COVID-19 related GP visit, of which 7 (13%) were hospitalised. C-indexes were 0.76 and 0.72 in these subgroups. The average predicted hospitalisation risk was 19% in patients presenting at the GP at least 14 days after vaccination, while the observed risk was 13% (Table S9, Figure S10).
Discussion
Main findings
This study proposed a general practice prognostic model suitable to predict hospital admission due to COVID-19 within 14 days after a face-to-face GP consultation. The model included comorbidity and lifestyle variables, as well as current COVID-19-related symptoms, peripheral oxygen saturation, duration of complaints, body temperature and patient age and sex. The model had good discrimination (C-statistic 0.84) and calibration in the three regional cohorts. With a C-statistic of 0.79 and a calibration curve close to the diagonal, the model performed well in the second-wave temporal validation cohort despite considerable differences in overall hospitalisation risk, symptomatology, presence of comorbidities, dominant variants, and availability of tests and vaccination.
There were two unexpected findings. First, this study found that specific comorbidities (e.g. cancer and COPD) were negatively correlated with hospital admission after adjusting for vital signs and symptoms. The threshold for a face-to-face GP consultation may be lower for patients with severe comorbidities than for others, and hence, any correlation between comorbidity and COVID-19 prognosis may be weaker or reversed in the GP cohort compared to the general population. Second, second-wave patients presenting with COVID-19 to general practice appeared healthier but a higher percentage of patients were hospitalised compared to the first-wave development cohort. The increased availability of personal protective equipment (PPE) and knowledge of infection prevention may have lowered the threshold to physically assess healthier patients in general practice while the reduced strain on hospitals may have led to relatively more admissions. The hospitalisation rate in the first-wave cohort may also have been artificially lowered due to the inclusion of patients with other respiratory illnesses due to the unavailability of tests in primary care.
Comparison with existing literature
The QCOVID models are the best-known prognostic models predicting hospital admission for COVID-19 [Citation9,Citation24]. They are intended for the general population (without COVID-19) instead of patients presenting to primary care with COVID-19-like symptoms or a positive test result. General population models predicting mortality are also available [Citation9,Citation25–28]. Our model targets a population of patients with symptoms presenting to their GP via a face-to-face consultation. The few models suitable for predicting the hospitalisation of patients with COVID-19 do not include symptoms or vital signs or are intended for remote assessment via digital platforms or mobile apps [Citation7,Citation8,Citation10,Citation11]. The reported c-statistic of the general population and primary care models to predict hospitalisation varied between 0.71 and 0.86. However, these performances cannot be compared directly to the c-statistics of 0.84 and 0.79 in the current analysis due to differences in studies populations and setting and type of validation.
Strengths and limitations
This study met the a priori calculated sample size for model development, allowing for exploring a large set of relevant candidate predictors without inflating the risk of overfitting. The multicentre nature of this study increases the generalisability of the findings. However, some limitations need further discussion. First, the validation dataset predates the emergence of the omicron variant, so further validation is required. Any overestimation of hospitalisation risks could be remedied by recalibrating the model coefficients. We hypothesise the predictions conditional on vital signs and symptoms likely make our model less vulnerable to fluctuations in COVID-19 prevalence and perhaps also to changes in the severity of SARS-CoV-2 VOCs (variants of concern) and acquired immunity compared to the available models based on comorbidities alone. Second, the model does not take vaccination status into account. The vaccination campaign ran at full speed in the validation period but unfortunately, most vaccinated patients became infected before vaccination. Our estimates indicate the prediction model somewhat overestimated hospitalisation risks (19% versus 13%) in primary care patients with confirmed COVID-19 two to thirteen weeks after vaccination. However, we remain uncertain due to the small number of vaccinated patients in our dataset. Third, the use of routine care data from COVID-19 centres may have led to underreporting of comorbidities and selective reporting of symptoms. The temporal validation used patient files from regular general practice with more complete comorbidity and medication data. Fourth, the temporal validation data fell short of the target sample size but met the minimum criterion of 100 events for validation [Citation29]. Fifth, the model was developed on patients with probable COVID-19, but the validation in patients with confirmed COVID-19 showed good predictive performance.
Implications
In the past year (April 2022 to April 2023) there have been five COVID-19 waves in the Netherlands with up to 200 hospitalisations per day, compared to peaks of 300 hospitalisations per day earlier in the pandemic (October 2020 to April 2022) [Citation30]. An updated version of our model could be built into software applications or online calculators to estimate hospitalisation risk after physical assessment by the GP, should COVID-19 cause a renewed strain on Dutch healthcare. It may aid in selecting high-risk COVID-19 patients that might benefit most from early interventions aimed to reduce complications and prevent hospital admission. However, to tackle changes over time due to novel SARS-CoV-2 variants, changing health policy and clinical practice, validating the model’s performance over time is critical. Model performance could be continuously or periodically checked by linking predicted risks based on GP data to hospital admission data. In any case, the current study demonstrates it is feasible to develop a risk prediction model to assist GPs in prognosis for an unknown emerging virus, should the need arise again.
Conclusion
A model consisting of sex, smoking status, symptoms, vital signs, and comorbidities accurately estimated the risk of hospital admission due to COVID-19 in general practice, but further validation is needed to evaluate the accuracy of predicted hospitalisation risks in populations with acquired immunity and novel SARS-CoV-2 VOCs.
Box 1. Prediction model equation.
lp = -41.67 + 0.48 × man - 1.19 × smoker + 0.19 × auscultation abnormalities + 0.48 × confusion - 0.37 chest pressure or pain + 0.21 × cough - 0.50 × sputum + 0.55 × haemoptysis + 0.49 × stomach complaints (other than diarrhoea) + 0.23 × headache + 0.72 × chronic kidney disease - 1.25 × current treatment for malignancy - 0.64 × past treatment for malignancy - 0.53 × COPD + 0.06 × age (years) - 0.05 × age’ (years) + 1.11 × temperature (degrees Celsius) - 0.71 × temperature’ (degrees Celsius) - 0.06 × oxygen saturation (percent) - 0.15 × oxygen saturation’ (percent) + 0.30 × duration of complaints (days) - 0.34 × duration of complaints’ (days) probability of hospital admission within two weeks =
Continuous variables are splines for which knots were placed at the 10%, 50% and 90% quantiles. For age: 37, 56 and 81; for temperature: 36.3, 37.1 and 38.4; for oxygen saturation: 92, 98 and 99; for duration of complaints: 0, 7 and 14. Lp = linear predictor
Supplemental Material
Download Zip (704.8 KB)Acknowledgements
We thank Ron Kusters, Linda Grossfeldt, Anita Speetgens and Joost Frenken for contributing data. We thank Mascha Twellaar for her assistance in preparing the data for analysis and Luc Smits for his advice on study design and analysis.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
References
- Holm A, Møller A, Aabenhus R. Management of symptomatic patients with suspected mild-moderate COVID-19 in general practice. What was published within the first year of the pandemic? A scoping review. Eur J Gen Pract. 2021;27(1):339–345. doi: 10.1080/13814788.2021.2002295.
- Wynants L, Van Calster B, Bonten MMJ, et al. Prediction models for diagnosis and prognosis of covid-19 infection: systematic review and critical appraisal. BMJ. 2020;369:m1328. doi: 10.1136/bmj.m1328.
- Greenhalgh T, Koh GCH, Car J. Covid-19: a remote assessment in primary care. BMJ. 2020;368:m1182. doi: 10.1136/bmj.m1182.
- Rabady S, Hoffmann K, Brose M, et al. Symptoms and risk factors for hospitalisation of COVID-19 presented in primary care: an exploratory retrospective study. Wien Klin Wochenschr. 2022;134(9-10):335–343. doi: 10.1007/s00508-021-01992-y.
- Cecil E, Bottle A, Majeed A, et al. Factors associated with potentially missed acute deterioration in primary care: cohort study of UK general practices. Br J Gen Pract. 2021;71(708):e547–e554. doi: 10.3399/BJGP.2020.0986.
- Dhont S, Derom E, Van Braeckel E, et al. The pathophysiology of ‘happy’ hypoxemia in COVID-19. Respir Res. 2020;21(1):198. doi: 10.1186/s12931-020-01462-5.
- van Royen FS, Joosten LPT, van Smeden M, et al. Cardiovascular vulnerability predicts hospitalisation in primary care clinically suspected and confirmed COVID-19 patients: a model development and validation study. PLoS One. 2022;17(4):e0266750. doi: 10.1371/journal.pone.0266750.
- Gude-Sampedro F, Fernández-Merino C, Ferreiro L, et al. Development and validation of a prognostic model based on comorbidities to predict covid-19 severity. A population-based study. Int J Epidemiol. 2020;50(1):64–74. doi: 10.1093/ije/dyaa209.
- Clift AK, Coupland CAC, Keogh RH, et al. Living risk prediction algorithm (QCOVID) for risk of hospital admission and mortality from coronavirus 19 in adults: national derivation and validation cohort study. BMJ. 2020;371:m3731. doi: 10.1136/bmj.m3731.
- Espinosa-Gonzalez A, Prociuk D, Fiorentino F, et al. Remote COVID-19 assessment in primary care (RECAP) risk prediction tool: derivation and real-world validation studies. Lancet Digit Health. 2022;4(9):e646–e656. doi: 10.1016/S2589-7500(22)00123-6.
- Liu Y, Wang Z, Tian Y, et al. Design and development of COVID-19 risk assessment decision support system for general practitioners. J Med Internet Res. 2020;22(6):e19786. doi: 10.2196/19786.
- de Bont E, Broers N, Muris D, et al. Opschalen naar een bovenregionale 24/7-HAP bij de COVID-19-pandemie [upscaling to a superregional 24/4-GP in the COVID-19 pandemic]. Huisarts Wet. 2022;65(3):43–46. (Dutch) doi: 10.1007/s12445-022-1395-0.
- Nederlands Huisartsen Genootschap. Richtlijnen en praktijk: ICPC 2022 [Guidelines and practice: ICPC 2022]. [cited 2023 December 05]. Available from: https://www.nhg.org/themas/artikelen/icpc.
- Prokop M, van Everdingen W, van Rees Vellinga T, et al. CO-RADS: a categorical CT assessment scheme for patients suspected of having COVID-19-Definition and evaluation. Radiology. 2020;296(2):E97–e104. doi: 10.1148/radiol.2020201473.
- Struyf T, Deeks JJ, Dinnes J, et al. Signs and symptoms to determine if a patient presenting in primary care or hospital outpatient settings has COVID-19. Cochrane Database Syst Rev. 2022;5(5):Cd013665. doi: 10.1002/14651858.CD013665.pub3.
- Riley RD, Ensor J, Snell KIE, et al. Calculating the sample size required for developing a clinical prediction model. BMJ. 2020;368:m441. doi: 10.1136/bmj.m441.
- Steyerberg EW. Clinical prediction models: a practical approach to development, validation, and updating. New York, NY: Springer US; 2019. (Statistics for Biology and Health).
- Riley RD, Stewart LA, Tierney JF. Individual participant data meta-analysis for healthcare research. In: Individual participant data meta-analysis: a handbook for healthcare research. Oxford: John Wiley & Sons; 2021. p. 1–6.
- Buuren S, Groothuis-Oudshoorn K. Mice: multivariate imputation by chained equations in R. J Stat Soft. 2011;45(3):1–68. doi: 10.18637/jss.v045.i03.
- Harrell FE. Regression modeling strategies. Regression modeling strategies. New York (NY): Springer; 2015. p. 63–102.
- Heymans MW. psfmi: Prediction Model Selection and Performance Evaluation in Multiple Imputed Datasets. R package version 1.0.0. 2021. https://mwheymans.github.io/psfmi/
- Balduzzi S, Rücker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health. 2019;22(4):153–160. doi: 10.1136/ebmental-2019-300117.
- Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. 2015;350(jan07 4):g7594–g7594. doi: 10.1136/bmj.g7594.
- Simpson CR, Robertson C, Kerr S, et al. External validation of the QCovid risk prediction algorithm for risk of COVID-19 hospitalisation and mortality in adults: national validation cohort study in Scotland. Thorax. 2022;77(5):497–504. doi: 10.1136/thoraxjnl-2021-217580.
- Williamson EJ, Tazare J, Bhaskaran K, et al. Comparison of methods for predicting COVID-19-related death in the general population using the OpenSAFELY platform. Diagn Progn Res. 2022;6(1):6. doi: 10.1186/s41512-022-00120-2.
- España PP, Bilbao A, García-Gutiérrez S, et al. Predictors of mortality of COVID-19 in the general population and nursing homes. Intern Emerg Med. 2021;16(6):1487–1496. doi: 10.1007/s11739-020-02594-8.
- Nafilyan V, Humberstone B, Mehta N, et al. An external validation of the QCovid risk prediction algorithm for risk of mortality from COVID-19 in adults: a national validation cohort study in England. Lancet Digit Health. 2021;3(7):e425–e433. doi: 10.1016/S2589-7500(21)00080-7.
- Lyons J, Nafilyan V, Akbari A, et al. Validating the QCOVID risk prediction algorithm for risk of mortality from COVID-19 in the adult population in Wales, UK. Int J Popul Data Sci. 2020;5(4):1697. doi: 10.23889/ijpds.v5i4.1697.
- Collins GS, Ogundimu EO, Altman DG. Sample size considerations for the external validation of a multivariable prognostic model: a resampling study. Stat Med. 2016;35(2):214–226. doi: 10.1002/sim.6787.
- Rijksoverheid. Coronavirus Dashboard the Netherlands: rijksoverheid, 2023. https://coronadashboard.government.nl/landelijk/ziekenhuizen-en-zorg