Proposed clinical indicators for efficient screening and testing for COVID-19 infection using Classification and Regression Trees (CART) analysis

ABSTRACT

The introduction and rapid transmission of SARS-CoV-2 in the United States resulted in methods to assess, mitigate, and contain the resulting COVID-19 disease derived from limited knowledge. Screening for testing has been based on symptoms typically observed in inpatients, yet outpatient symptoms may differ. Classification and regression trees recursive partitioning created a decision tree classifying participants into laboratory-confirmed cases and non-cases. Demographic and symptom data from patients ages 18–87 years enrolled from March 29–June 8, 2020 were included. Presence or absence of SARS-CoV-2 was the target variable. Of 832 tested, 77 (9.3%) tested positive. Cases significantly more often reported diarrhea (12 percentage points (PP)), fever (15 PP), nausea/vomiting (9 PP), loss of taste/smell (52 PP), and contact with a COVID-19 case (54 PP), but less frequently reported sore throat (−27 PP). The 4-terminal node optimal tree had sensitivity of 69%, specificity of 78%, positive predictive value of 20%, negative predictive value of 97%, and AUC of 76%. Among those referred for testing, negative responses to two questions could classify about half (49%) of tested persons with low risk for SARS-CoV-2 and would save limited testing resources. Outpatient symptoms of COVID-19 appear to be broader than the inpatient syndrome.

Initial supplies of anticipated COVID-19 vaccines may be limited and administration of first such available vaccines may need to be prioritized for essential workers, the most vulnerable, or those likely to have a robust response to vaccine. Another priority group could be those not previously infected. Those who screen out of testing may be less likely to have been infected by SARS-CoV-2 virus thus may be prioritized for vaccination when supplies are limited.

KEYWORDS:

• COVID-19
• symptoms
• SARS-CoV-2
• screening
• classification trees

Introduction

The introduction of the SARS-CoV-2 virus into the United States (U.S.) was earlier than anticipated. Furthermore, the rapid transmission of SARS-CoV-2 resulted in implementation of methods to assess, mitigate and contain the resulting COVID-19 disease that were based on limited knowledge of its epidemiology. Given potentially insufficient testing supplies, but the need to rapidly identify cases, access to testing was based on symptoms typical of an acute respiratory infection, such as cough, fever, and shortness of breath, as has been reported to be emblematic of COVID-19 infection.^1,2 A previous report in the U.S. noted that use of cough, fever, shortness of breath, and sore throat as the sole screening criteria for COVID-19 infection among health care personnel might have missed 17% of symptomatic cases.³ Gastrointestinal and non-acute respiratory illness-typical symptoms have been noted among hospitalized COVID-19 patients.^2,4 Because outpatient symptom complexes may differ from the better-known inpatient syndrome, a data-driven set of clinical indicators for COVID-19 would help to identify outpatient symptoms and those who would most benefit in situations of limited testing availability. Furthermore, identifying those who are less likely to have been exposed to the SARS-CoV-2 virus may simultaneously identify those who may respond well to a potential vaccine.

Methods

Symptomatic individuals who had either exposure to a case of COVID-19 or a typical respiratory illness symptom were scheduled at a centralized outpatient COVID-19 testing facility. Presence of SARS-CoV-2 was detected using highly sensitive RT-PCR tests⁵ from nasopharyngeal specimens, which can detect 5 copies of virus. While at the clinic, testees received a flier describing this study. Following return of their results to the electronic medical record, a list of tested patients was sent to the research team daily, for contact by e-mail or telephone. Asymptomatic patients were not scheduled for testing at the time of this study’s data collection and were not included in the analyses. After electronic or verbal consent, participants completed a short enrollment survey either online or by phone, which included basic demographic information and symptoms at enrollment. Participants received a 20 USD debit card for participating in the study. The University of Pittsburgh IRB approved this study.

Data were captured in a RedCap data base and loaded into a SAS data set. Participants were grouped into COVID-19 cases and non-cases for comparison of demographic characteristics and symptoms. Descriptive statistics were computed for continuous data. Frequency distributions were estimated for categorical data. The appropriate parametric (e.g., t-test) or nonparametric test (e.g., chi-square, Wilcoxon tests) was used to assure a balanced distribution of the demographic and clinical characteristics among COVID-19 cases and non-cases using SAS version 9.4, Cary, NC. Significance was set at alpha = 0.05.

Classification and Regression Trees (CART) recursive partitioning, based on presence or absence of symptoms, was used to create a decision tree to correctly classify enrollees into laboratory-confirmed (RT-PCR) COVID-19 cases.⁶

Symptoms reported at enrollment – fever, chills, cough, sore throat, shortness of breath, muscle aches, abdominal pain, nausea/vomiting, diarrhea, headache, decrease or loss of taste or smell, and contact with COVID-19 case – were included in the CART model as categorical predictors. Presence or absence of SARS-CoV-2 was the target variable. The tree was built on a ≥ 10% sample for the parent node and a stopping rule of ≥5% of the sample in the terminal node. A tenfold cross-validation method was used to evaluate reliability. To avoid overfitting, a maximum acceptable difference in risk between the pruned and the sub-tree of one standard error was used for tree pruning. Missing data were handled by surrogate splits. Hosmer-Lemeshow goodness of fit test confirmed the suitability of the trees. The area under the curve (AUC) for receiver operating characteristics, sensitivity, specificity, positive, and negative predictive values were assessed.

Results

Of 7,698 individuals tested for SARS-CoV-2 at this centralized center, overall, 361 (4.7%) were positive for the virus. Only adults ≥18 years of age were included. Among all adults tested, 832 (11.3%) enrolled in this study. Among enrollees, 77 (9.3%) were positive for SARS-CoV-2. No cases had tested negative within 14 days of receiving a positive result.

Among enrollees, COVID-19 cases were more likely to be nonwhite (black, other; 22.1%) than non-cases (11.7%; P = .009) and were less likely to smoke or vape (9.3% cases vs. 20.0% non-cases; P = .024), but did not differ by sex, age group or being a health care worker (Table 1). Those who tested positive significantly more frequently reported loss of taste/smell (75.9% vs. 23.6%; P < .001), diarrhea (49.4% vs. 37.7%; P = .046, fever (81.8% vs. 67.3%; P = .009), nausea/vomiting (18.4% vs. 9.7%; P = .018) and contact with a COVID-19 case (73.7% vs 20%; P < .001) (Table 2). Conversely, COVID-19 cases compared with non-cases less frequently reported sore throat (39.5% vs. 66.5%; P < .001).

Table 1. Baseline demographics by COVID-19 status

Demographics	Non-cases N = 755 (90.7%) n (%)	COVID-19 cases N = 77 (9.3%) n (%)	P value
Age group			.620
18–49 years	65.2	62.3
≥ 50 years	34.8	37.7
Female sex	73.8	66.7	.185
White race, ref. = nonwhite	88.3	77.9	.009
Healthcare worker	39.9	44.7	.410
Current smoker or vaper, ref. = non-smoker/vaper	20.0	9.3	.024
Age, mean (SD)	43.3 (14.1)	44.4 (14.0)	.516

Table 2. Presenting symptoms for outpatients attending the COVID-19 testing clinic

Symptom	Non-cases N = 755 (90.7%) n (%)	COVID-19 cases N = 77 (9.3%) n (%)	P value
Abdominal pain	27.5%	20.8%	.208
Chills	63.9%	74.0%	.077
Cough	87.2%	86.8%	.936
Decrease or loss of taste or smell	23.6%	75.9%	<.001
Diarrhea	37.7%	49.4%	.046
Fever	67.3%	81.8%	.009
Headache	81.6%	85.7%	.371
Muscle aches	67.1%	67.5%	.934
Nausea/Vomiting	9.7%	18.4%	.018
Shortness of breath	64.8%	56.6%	.156
Sore throat	66.5%	39.5%	<.001
Contact with COVID-19 case	20%	73.7%	<.001

The CART analyses resulted in a 4-terminal node optimal tree – one terminal node with a recommendation for testing, 1 node with a recommendation not to test and two nodes with a recommendation to consider testing (Figure 1). This tree has a sensitivity of 69% and specificity of 78%, resulting in an AUC of 76%. The positive predictive value for this tree was 20% while the negative predictive value was 97%.

Figure 1. Decision tree for symptoms to determine testing needs for outpatient COVID-19

For screening of individuals with acute symptoms as a possible COVID-19 case, it is important to note the order of the questions as shown in the Figure 1. If an individual first reports having been exposed to a COVID-19 case, there is a 21% likelihood of COVID-19 infection, and testing is recommended.

If the individual denies exposure to a COVID-19 case, then the next question should be, “Have you experienced a recent loss of taste or smell?” If the answer is negative, then the need for testing is minimal, as 98% of these individuals’ tests results will be negative for COVID-19, thus the recommendation would be not to test. A negative response to these two questions followed down the left side of the Figure 1 would allow almost half (49%; 406/832) of the participants to be ruled out as likely non-cases without testing.

There are two scenarios in which testing should be considered (aqua boxes in Figure 1). For instance, if the individual confirms loss of taste or smell, with or without report of sore throat, testing should be considered because a number of positives would be identified.

Discussion

CART decision tree analysis has been used to assist clinical decision making across a wide assortment of health conditions, their diagnosis or treatment, and outcome predictions.^{7,8,9,10,11,1213} In rapid response to a pandemic novel coronavirus, the local testing center established testing criteria on knowledge of inpatient symptoms, including fever, cough and shortness of breath.¹⁴ We found that among those referred for outpatient testing, the symptoms that were significantly more frequent among cases than among non-cases differed from those typically reported among inpatients with COVID-19 infection and on which early identification of possible cases were based. In our sample, cough, headache, fever and decrease or loss of taste or smell were the most frequently reported symptoms of COVID-19 cases. These findings were similar to a report from Korea¹⁵ among outpatients who most frequently presented with cough, sputum, reduced sense of smell, and nasal congestion. In our study, there were significant differences in report of symptoms between cases and non-cases, including loss of taste or smell which was reported three times more often in cases than non-cases. Thus, less severe illness seems to be associated with a broader range of symptoms that are not limited to respiratory symptoms.

Many locales have lacked adequate resources to test for SARS-CoV-2. Of importance to such locales, we found that negative responses to two questions could classify about half of tested persons who had a minute risk for SARS-CoV-2 and, if used for screening, would save limited testing resources. These questions are: Was the patient in contact with a COVID-19 case? Has the patient experienced a loss of taste or smell? If a COVID-19 case exposure did not occur, and the patient reported neither loss of taste or smell the patient need not be tested. Our algorithm has a specificity of 69% and, in our low prevalence situation, a 97% negative predictive value.

Within the next year, it is expected that one or more SARS-CoV-2 vaccines will have become available for use in the U.S. At the time of this writing, phase III drug trials are just beginning. It is unknown which vaccine(s) will be licensed, how much vaccine will be available, which population subgroups may be specifically targeted for vaccination, and how much vaccination will cost. It is possible that vaccination will be focused on those who have not been diagnosed with COVID-19. Using the decision tree for symptomatic individuals described herein, those individuals not likely to be infected may be more easily identified for targeted vaccination.

Strengths and limitations

Strengths of this study include the use of a control group of non-cases, development of a testing algorithm using sophisticated recursive partitioning, and the finding of high specificity. Given the mathematical rules underlying testing algorithms, prevalence affects predictive value; thus, other locales may have different predictive values even with the same sensitivity and specificity. The number of positives in our sample was modest, reflecting the low prevalence in our region. The response rate was also modest, reflecting recruitment by e-mail and phone during a time when infection control measures prohibited in-person recruiting and reflecting that non-cases were enrolled primarily by e-mail (i.e., no need to make greater efforts at non-case recruitment because we recruited 755 non-cases with little effort). It was argued early in the U.S. pandemic that testing of those with mild disease may not be an appropriate use of resources due to the moderate sensitivity of the tests,¹⁶ however, the RT-PCR test used to identify COVID-19 cases had high sensitivity and specificity,⁵ meaning that few true cases were likely missed, thus strengthening the conclusions from the CART modeling.

Conclusion

The outpatient symptoms of COVID-19 appear to be broader than the well-known inpatient syndrome and prominently feature loss of taste or smell at a three-fold higher rate (76% vs. 24%) in cases than non-cases. Our algorithm has a modest specificity (69%) but high negative predictive value (97%), allowing testing to be prioritized in those locales in which testing supplies are limited. Those who screen out of testing may be less likely to have been infected with SARS-CoV-2 virus thus may be higher priority for vaccination during the time when vaccine supplies are limited.

Disclosure of potential conflicts of interest

Drs. Zimmerman, Nowalk, and Balasubramani and Ms. Eng and Ms. Sax have research funding from Merck & Co, Inc. for unrelated projects. Dr. Zimmerman has funding from Sanofi for an unrelated project. Drs. Bear and Taber and Mr. and Ms. Clarke have no conflicts to report.

Irb approval

University of Pittsburgh IRB Protocol #19070407

Additional information

Funding

This study was supported through a cooperative agreement with the Centers for Disease Control and Prevention (CDC) through grant number U01 IP000467 and the National Institutes of Health grant number 1UL1 TR001857. The US Flu VE Network is supported through cooperative agreements funded by CDC. The findings and conclusions are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention. It is subject to the CDC’s Open Access Policy.