Development of an algorithm to identify urgent referrals for suspected cancer from the Danish Primary Care Referral Database

Figures & data

Table 1 Characteristics of the four PCR-P

Populations	Inclusion criteria			Age of referred patients	Males	Urgently referred patients
	Requesters	Receivers	Patients	Median/years (IQI)	% (95% CI)	% (95% CI)
PCR-P I N=1,000	GPs in DK	Hospital (DK)	All age	52.0 (31.9–67.9)	40.3 (37.3–43.3)	9.1 (7.3–10.9)
PCR-P II N=2,000	GPs in CDR	Hospital (CDR) exc. psychiatric Private specialistsa	≥ 40 years	62.8 (51.4–72.5)	43.5 (41.3–45.7)	10.7 (9.3–12.0)
PCR-P III N=2,172	GPs in CDR Exc. out-of-hours GPs	Hospital (CDR) exc. psychiatric Private specialistsa	≥ 40 years	60.7 (50.0–71.1)	42.7 (40.6–44.8)	13.5 (12.1–14.9)
PCR-P IV N=1,949	GPs in CDR Exc. out-of-hours GPs	Hospital (CDR) exc. psychiatric Private specialistsa	≥40 years No prior cancer	59.2 (49.3–70.0)	42.5 (40.3–44.8)	12.8 (11.4–14.4)

Notes:

a Private specialists include the fields of otolaryngology, dermatology, gynecology, surgery, plastic surgery, or neurology in the Central Denmark Region.

Abbreviations: GP, general practitioner; DK, Denmark; CDR, Central Denmark Region; CRR-P, Cancer Referral Reference population; PCR-P, Primary Care Referral population; CI, confidence interval; IQI, interquartile intervals; exc., excluding.

Figure 1 The stepwise development of the algorithm.

Abbreviations: CRR-P, Cancer Referral Reference population; PCR-P, Primary Care Referral population; PPV, positive predictive value; NPV, negative predictive value; W, word; F, Fields; MW, misleading word.

Table 2 The performance of the stepped algorithms on each population

	Step-1 algorithm	Step-2 algorithm	Step-3 algorithm	Final algorithm
CRR-P Ia
Sensitivity % (95% CI)	89.8 (84.4–95.1)	96.9 (93.8–99.9)	95.3 (91.5–99.0)	95.3 (91.5–99.0)
CRR-P IIa
Sensitivity % (95% CI)	85.7 (82.6–88.8)	94.8 (92.8–96.7)	94.6 (92.6–96.6)	93.8 (91.6–95.9)
PCR-P Ib
Sensitivity % (95% CI)	63.7 (53.0–73.6)	97.8 (92.3–99.7)	94.5 (87.6–98.2)	94.5 (87.6–98.2)
Specificity % (95% CI)	90.8 (88.7–92.6)	83.2 (80.6–85.5)	92.2 (90.2–93.8)	95.7 (94.2–96.9)
PPV % (95% CI)	40.8 (32.7–49.4)	36.8 (30.7–43.2)	54.8 (46.6–62.7)	68.8 (59.9–76.8)
NPV % (95% CI)	96.2 (94.6–97.3)	99.7 (99.1–100.0)	99.4 (98.6–99.8)	99.4 (98.7–99.8)
PCR-P IIb
Sensitivity % (95% CI)	78.9 (72.8–84.2)	97.2 (94.0–99.0)	94.8 (90.9–97.4)	93.0 (88.7–96.0)
Specificity % (95% CI)	84.9 (83.2–86.6)	76.6 (74.6–78.6)	90.4 (88.9–91.7)	95.2 (94.2–96.2)
PPV % (95% CI)	38.4 (33.9–43.2)	33.1 (29.4–37.0)	54.0 (48.8–59.1)	70.0 (64.3–75.2)
NPV % (95% CI)	97.1 (96.2–97.9)	99.6 (99.1–99.8)	99.3 (98.8–99.7)	99.1 (98.6–99.5)
PCR-P IIIb
Sensitivity % (95% CI)	76.1 (70.8–80.9)	96.2 (93.4–98.1)	93.9 (90.5–96.3)	93.9 (90.5–96.3)
Specificity % (95% CI)	81.6 (79.8–83.3)	73.9 (71.8–75.8)	88.3 (86.8–89.7)	93.7 (92.5–96.3)
PPV % (95% CI)	39.2 (35.2–43.3)	36.5 (33.1–40.9)	55.6 (51.1–60.0)	69.8 (65.0–74.3)
NPV % (95% CI)	95.6 (94.5–96.6)	99.2 (98.6–99.6)	98.9 (98.3–99.4)	99.0 (98.4–99.4)
PCR-P IVb
Sensitivity % (95% CI)	N/A	N/A	N/A	93.6 (89.8–96.3)
Specificity % (95% CI)	N/A	N/A	N/A	97.3 (96.4–98.0)
PPV % (95% CI)	N/A	N/A	N/A	83.6 (78.7–87.7)
NPV % (95% CI)	N/A	N/A	N/A	99.0 (98.4–99.5)

Notes:

a The performance of the algorithms on the Cancer Referral Reference populations (known to be urgent cancer referrals) was assessed by sensitivity.

b The performance of the algorithms on the Primary Care Referral populations was assessed by sensitivity, specificity, and positive and negative predictive values; all with 95% CIs.

Abbreviations: GP, general practitioner; CI, confidence interval, CRR-P, Cancer Referral Reference population; PCR-P, Primary Care Referral Population; PPV, positive predictive value; NPV, negative predictive value; N/A, not applicable.

Table 3 True- and false-positive findings of important identifying words from the final algorithm

Important identifying words	Positive findings of algorithm applied to PCR-P III (n=394)		Positive findings of algorithm applied to PCR-P IV (n=280)
	True positives (%) n=275 (69.8)	False positives (%) n=119 (30.2)	True positives (%) n=234 (83.6)	False positives (%) n=46 (16.4)
Pakkea	82 (29.8)	4 (3.4)	74 (31.6)	3 (6.5)
Kræftb	59 (21.5)	22 (18.5)	53 (22.7)	9 (19.6)
Cancer	93 (33.8)	44 (37.0)	73 (31.2)	8 (17.4)
Malign	24 (8.7)	17 (14.3)	19 (8.1)	7 (15.2)
c.c	14 (5.1)	2 (1.7)	11 (4.7)	0 (0.0)
Pakke or kræft or cancer or c.d	143 (52.0)	64 (53.8)	126 (53.9)	24 (52.2)

Notes:

a Danish term for cancer pathway.

b Danish term for cancer.

c c. includes c., c.pulm, c.panc, c.cerv, c.uteri, c.mamma, c.cere, and c.test.

d The last row gives the combination of words that measure the appearance of “cancer wording.”

Abbreviation: PCR-P, Primary Care Referral Population.

Table 4 Subgroup categories of false-positive findings of the final algorithm

	Main category N (%)	Subgroup categories, n (%)
	Not urgent cancer referral	Earlier diagnosed cancer	Earlier suspicion for cancer rejected	Urgent referral for heart disease	Nonmelanoma skin cancer	Hereditary disposure or fear for cancer	Others
False positives	119 (100.0)	75 (63.0)	15 (12.6)	0 (0.0)	2 (1.7)	11 (9.2)	16 (13.4)

Notes: The final algorithm was applied to the Primary Care Referral population III. A false-positive finding was manually assessed as “not urgent cancer referral” and incorrectly identified by the algorithm. “Not urgent cancer referrals” were manually divided into subgroup categories.