Choosing where to set the threshold between low- and high-risk patients: Evaluating a classification tool within a simulation

Figures & data

Figure 1. Simulation process map. DNA = Did not attend.

Table 1. Illustrative two-way contingency table of lump (L) and urgency (U).

	Column j	0	1	2
Row i		Symptomatic	Suspected cancer	Other
0	No lump	μ00	μ01	μ02
1	Lump present	μ10	μ11	μ12

Table 2. Simple scorecard.

	Scorecard points
Age
Age < 30	10
30 < = Age < 40	3
40 < = Age < 50	8
50 < = Age	12
Lump
Yes	13
No or not recorded	2

Table 3. Full scorecard.

	Scorecard points
Age
Age < 29	43
29 < = Age < 42	16
42 < = Age < 47	33
47 < = Age < 52	21
52 < = Age	64
Duration of symptoms
Not applicable or not recorded	29
Less than 2 weeks	42
2 weeks–2 months	35
2–5 months	26
Over 5 months	35
Family history of cancer
No or not recorded	29
Yes	38
Lump
No or not recorded	5
Yes	54
Other symptom
No or not recorded	29
Yes	36
Unilateral pain
No or not recorded	31
Yes	33
Urgency
Suspected cancer	32
Symptomatic	31
Other or not recorded	32

Table 4. Classification performance of simple scorecard for different cut-off scores.

Cut-off score	True positives	True negatives	False positives	False negatives	Sensitivity	Specificity	Classification accuracy
26	98	0	81	0	100%	0%	55%
24	93	11	70	5	95%	14%	58%
22	83	31	50	15	85%	38%	64%
17	77	45	36	21	79%	56%	68%
15	61	58	23	37	62%	72%	66%
13	50	68	13	48	51%	84%	66%
11	37	72	9	61	38%	89%	61%
8	19	78	3	79	19%	96%	54%
4	0	81	0	98	0%	100%	45%

Table 5. Classification performance of full scorecard for different cut-off scores.

Cut-off score	True positives	True negatives	False positives	False negatives	Sensitivity	Specificity	Classification accuracy
290	98	0	81	0	100%	0%	55%
280	96	3	78	2	98%	4%	55%
270	95	7	74	3	97%	9%	57%
260	94	15	66	4	96%	19%	61%
250	90	24	57	8	92%	30%	64%
240	85	37	44	13	87%	46%	68%
230	76	45	36	22	78%	56%	68%
220	60	66	15	38	61%	81%	70%
210	51	69	12	47	52%	85%	67%
200	38	73	8	60	39%	90%	62%
190	28	74	7	70	29%	91%	57%
180	16	79	2	82	16%	98%	53%
170	0	81	0	98	0%	100%	45%
160	0	81	0	98	0%	100%	45%

Table 6. Operational performance of simple scorecard for different cut-off scores.

	Clinic efficiency		Average total time (minutes)
Cut-off score	Number runs for 5% precision	Mean [95% confidence interval]	Number runs for 10% precision	Mean [95% confidence interval]
26	48	0.24 [0.23,0.25]	18	143 [129,156]
24	60	0.26 [0.24,0.27]	29	134 [121,147]
22	59	0.27 [0.25,0.28]	35	124 [111,136]
17	69	0.27 [0.26,0.29]	42	116 [105,128]
15	106	0.27 [0.26,0.28]	47	107 [97,118]
13	112	0.26 [0.24, 0.27]	46	109 [98,119]
11	132	0.25 [0.24,0.26]	52	110 [99,121]
6	150	0.23 [0.22,0.24]	65	116 [104,127]
4	166	0.20 [0.19,0.22]	61	123 [110,135]

The tied best results for clinic efficiency are highlighted in bold, with the best average total time among these also highlighted in bold.

Table 7. Operational performance of full scorecard for different cut-off scores.

	Clinic efficiency		Average total time (minutes)
Cut-off score	Number runs for 5% precision	Mean [95% confidence interval]	Number runs for 10% precision	Mean [95% confidence interval]
165	166	0.20 [0.19,0.22]	61	123 [110,135]
170	162	0.21 [0.20,0.22]	61	123 [111,135]
175	158	0.21 [0.20,0.22]	61	123 [111,135]
180	147	0.22 [0.21,0.23]	66	117 [105,128]
185	140	0.23 [0.22,0.24]	59	115 [104,127]
190	141	0.24 [0.23,0.25]	66	113 [102,124]
195	132	0.25 [0.24,0.26]	57	113 [102,125]
200	123	0.25 [0.24,0.26]	47	105 [95,116]
205	120	0.26 [0.24,0.27]	47	107 [96,117]
210	116	0.26 [0.25,0.27]	46	106 [95,116]
215	112	0.26 [0.25,0.27]	46	108 [97,118]
220	112	0.26 [0.25,0.28]	52	109 [98,119]
225	83	0.27 [0.25,0.28]	43	110 [99,121]
230	71	0.27 [0.26,0.29]	35	114 [103,126]
235	63	0.28 [0.26,0.29]	41	113 [102,124]
240	64	0.27 [0.26,0.29]	37	120 [108,131]
245	54	0.27 [0.26,0.28]	34	125 [112,137]
250	51	0.26 [0.25,0.28]	33	131 [118,144]
255	52	0.26 [0.25,0.28]	24	127 [115,139]
260	62	0.26 [0.25,0.27]	24	132 [119,144]
265	45	0.25 [0.24,0.27]	24	134 [121,147]
270	49	0.25 [0.24,0.27]	20	138 [124,151]
275	48	0.25 [0.23,0.26]	19	141 [127,155]
280	43	0.24 [0.23,0.25]	16	142 [128,156]
285	44	0.24 [0.23,0.25]	18	143 [129,156]
290	44	0.24 [0.23,0.25]	18	143 [129,156]
295	48	0.24 [0.23,0.25]	18	143 [129,156]
300	48	0.24 [0.23,0.25]	18	143 [129,156]

The best result for clinic efficiency is highlighted in bold.

Availability of data and material

The dataset generated and analysed during the current study is not publicly available because this was a condition of patient consent. However, an anonymised version is available from the corresponding author on reasonable request.