Ankle fracture classification using deep learning: automating detailed AO Foundation/Orthopedic Trauma Association (AO/OTA) 2018 malleolar fracture identification reaches a high degree of correct classification

Figures & data

Figure 1. Data flowchart.

Table 1. Base distribution of fractures according to the AO classification. Values are count (%)

	Train (n = 4,941)			Test (n = 409)
Fracture type	Yes	Maybe	No	Yes	Maybe	No
Fracture	2,156 (44)	121 (2)	2,664 (54)	306 (75)	13 (3)	90 (22)
Malleolar (44)	1,696 (34)	63 (1)	3,182 (64)	210 (51)	6 (1)	193 (47)
Tibia distal (43)	254 (5)	6 (0)	4,681 (95)	63 (15)	2 (0)	344 (84)
Fibula (4F2–3)	129 (3)	3 (0)	4,809 (97)	37 (9)	0 (0)	372 (91)
Tibia diaphyseal (42)	88 (2)	0 (0)	4,853 (98)	27 (7)	0 (0)	382 (93)
Other bone	210 (4)	47 (1)	4,684 (95)	35 (9)	5 (1)	369 (90)

“Other bone” generally indicates a visible fracture of the foot. It was possible for an examination to have multiple fracture labels.

Table 3. Distribution of malleolar fractures by type (44A–C), specified by type, group, and subgroup. Values are count (%) for samples > 100

AO type	Train (n = 4,941)	Test (n = 409)
44A (483 train and 31 test cases)
1.1	78 (22)	6
1.2	165 (46)	7
1.3	114 (32)	9
2.1	105 (93)	5
2.2	1 (1)	–
2.3	7 (6)	2
3.1	11	–
3.3	2	2
44B (1,015 train and 136 test cases)
1.1	385 (74)	39
1.2	132 (25)	26
1.3	6 (1.1)	2
2.1	99 (44)	20
2.2	105 (47)	16
2.3	19 (8.5)	2
3.1	76 (28)	12
3.2	152 (56)	13
3.3	41 (15)	6
44C (255 train and 47 test cases)
C1
1.1	85 (67)	17
1.2	20 (16)	5
1.3	22 (17)	2
2.1	30	6
2.2	21	3
2.3	39	9
3.1	10	3
3.2	9	1
3.3	19	1

Figure 2. Distribution of AO classes in the malleolar fracture data.

Table 4. Outcome measures for the most important groups and weighted average AUC for each malleolar AO type, group, and subgroup combined

	Cases	Sensitivity	Specificity
AO type	n = 409	(%)	(%)	Youden’s J	AUC (95% CI)
44A
Base	32	73	81	0.54	0.81 (0.72–0.88)
1	22	88	75	0.63	0.87 (0.77–0.94)
1.1	6	75	93	0.68	0.87 (0.70–0.98)
1.2	7	80	83	0.63	0.79 (0.54–0.94)
1.3	9	75	88	0.63	0.84 (0.70–0.95)
2	7	100	74	0.74	0.91 (0.83–0.97)
2.1	5	100	74	0.74	0.89 (0.80–0.97)
3	2	100	86	0.86	0.90 (0.83–0.96)
44B
Base	137	89	88	0.77	0.93 (0.90–0.95)
1	67	90	88	0.77	0.93 (0.88–0.96)
1.1	39	87	84	0.71	0.89 (0.85–0.93)
1.2	26	92	85	0.77	0.90 (0.81–0.96)
2	38	82	84	0.65	0.87 (0.80–0.92)
2.1	20	100	72	0.72	0.87 (0.83–0.92)
2.2	16	88	74	0.62	0.82 (0.68–0.91)
2.3	2	100	98	0.98	0.99 (0.97–1.00)
3	32	78	90	0.68	0.90 (0.85–0.94)
3.1	12	83	75	0.58	0.79 (0.63–0.90)
3.2	13	92	82	0.74	0.91 (0.84–0.96)
3.3	6	100	91	0.91	0.96 (0.93–0.98)
44C
Base	47	74	90	0.65	0.86 (0.79–0.92)
1	24	75	79	0.54	0.83 (0.72–0.91)
1.1	17	76	85	0.61	0.86 (0.74–0.94)
1.2	5	80	92	0.72	0.89 (0.77–0.97)
1.3	2	100	88	0.88	0.92 (0.86–0.97)
2	18	100	72	0.72	0.91 (0.86–0.95)
2.1	6	83	93	0.76	0.91 (0.79–0.98)
2.2	3	100	88	0.88	0.96 (0.88–1.00)
2.3	9	100	77	0.77	0.88 (0.84–0.92)
3	5	100	88	0.88	0.95 (0.90–0.98)
Malleolar	216	86	90	0.76	0.92 (0.89–0.95)
Weighted mean AUC
A					0.84
B					0.90
C					0.87
Malleolar					0.90

Criterion based on Youden’s Index (Youden 1950, Aoki et al. 1997, Shapiro 1999, Greiner et al. 2000) defined as $$YI\left(c\right)=ma{x}_c\left(Se\left(c\right)+Sp\left(c\right)–1\right).$$

This is identical (from an optimization point of view) to the method that maximizes the sum of sensitivity and specificity (Albert 1987, Zweig and Campbell 1993) and to the criterion that maximizes concordance, which is a monotone function of the AUC.

Figure 3. The network failed to identify this image as a malleolar type A fracture. Among the malleolar fractures it was predicted as a type C fracture.

Figure 4. The examination should have been a malleolar type C fracture but the network predicted type B fracture.

Youden W J . Index for rating diagnostic tests. Cancer 1950; 3(1): 32–5.

 PubMed Web of Science ®Google Scholar

Aoki K , Misumi J , Kimura T , Zhao W , Xie T . Evaluation of cutoff levels for screening of gastric cancer using serum pepsinogens and distributions of levels of serum pepsinogen I, II and of PG I/PG II ratios in a gastric cancer case-control study. J Epidemiol 1997; 7(3): 143–51.

 PubMedGoogle Scholar

Shapiro D E . The interpretation of diagnostic tests. Stat Methods Med Res 1999; 8(2): 113–34.

 PubMed Web of Science ®Google Scholar

Greiner M , Pfeiffer D , Smith R D . Principles and practical application of the receiver-operating characteristic analysis for diagnostic tests. Prev Vet Med 2000; 45(1): 23–41.

 PubMed Web of Science ®Google Scholar

Albert . Multivariate interpretation of clinical laboratory data. Boca Raton, FL: CRC Press; 1987.

 Google Scholar

Zweig M H , Campbell G . Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinic medicine. Clin Chem 1993; 39(4): 561–77.

 PubMed Web of Science ®Google Scholar