Bridging AI and Clinical Practice: Integrating Automated Sleep Scoring Algorithm with Uncertainty-Guided Physician Review

Figures & data

Table 1 Demographic Characteristics of BSDB Subjects with Respect to Individual Data Splits

Domain	Scorers involved	Split	Number of PSGs	Age µ (σ) - median - min - max	Gender % ♂- ♀
ID	> 8 SP, > 50 AP	Train	4245	49.22 (16.40) - 51 - 2 - 88	64.28–35.72
		Validation	226	52.66 (21.45) - 60 - 8 - 84	67.71–32.29
		Test	423	50.48 (20.32) - 55 - 2 - 86	65.57–34.43
OOD1	1 SP	Test	1966	48.90 (18.60) - 52 - 0 - 91	64.65–35.35
OOD2	1 SP	Test	1972	46.93 (20.06) - 50 - 0 - 86	60.92–39.08
TOTAL	>10 SP, >50 AP	–	8832	48.82 (18.25) - 51 - 0 - 91	63.76–36.24

Abbreviations: PSG, polysomnography; µ, average age per group; σ, standard deviation of the age per group; min, minimum age per group; max, maximum age per group; %, percentage; ♂, male; ♀, female; ID, in-domain; OOD, out-of-domain; SP, senior physician; AP, assistant physician.

Table 2 Occurrence of Different Classes of Sleep Disorders Among Conclusive Diagnoses of Subjects per Individual Data Splits of BSDB

Diagnosis class	Domain					ALL
	ID train	ID Validation	ID test	OOD1 test	OOD2 test
HE	27	2	3	12	22	66
INS	106 + 15	8 + 1	17 + 2	31 + 5	43 + 4	205 + 27
SDB	247 + 156	16 + 8	34 + 17	91 + 33	124 + 18	512 + 232
CDH	171 + 30	10 + 2	22 + 5	54 + 1	115 + 10	372 + 48
CRD	11 + 1	0 + 0	2 + 0	1 + 0	5 + 0	19 + 1
PSD	75 + 9	7 + 0	6 + 0	22 + 1	44 + 0	154 + 10
SMD	74 + 5	5 + 0	7 + 0	18 + 1	33 + 0	137 + 6
IS	227 + 11	13 + 1	26 + 1	77 + 1	127 + 1	470 + 15
DSS	26 + 0	1 + 0	2 + 0	7 + 0	16 + 0	52 + 0
Multiple disorders	418	26	52	128	205	829
Single disorders	227	12	25	42	33	339
Other or unknown	3573	186	343	1784	1712	7598
TOTAL	4245	226	423	1966	1972	8832

Notes: Columns indicate individual data subsets: ID (training, validation, testing) and two OOD test sets (OOD1, OOD2), summing up to ALL. Rows indicate the number of subjects according to conclusive diagnoses class indicated by abbreviations described below. Row Multiple disorders indicates the number of subjects with multiple classes of sleep-disorders, Single disorder the number of subjects with a single sleep disorder, and Other or unknown the number of subjects with no or unknown conclusive diagnosis. TOTAL is equal to HE + Multiple disorders + Single disorder + Other or unknown. At the cell level of rows (INS to DSS), the sum refers to the number of subjects having multiple disorders including that given class plus the number of subjects having that specific class only.

Abbreviations: ID, in-domain; OOD, out-of-domain; HE, healthy controls; INS, insomnia disorders; SDB, sleep-disordered breathing; CDH, central disorders of hypersomnolence; CRD, circadian rhythm sleep-wake disorders; PSD, parasomnia-related sleep disorders; SMD, sleep-related rhythmic movement disorders; IS, isolated symptoms and normal variants; DSS, findings specific to day-time sleep studies.

Table 3 Measures Evaluating Prediction’s Uncertainty Using U-Sleep Softmax Output

	Measure	Notation	Mathematical formula
I	Average softmax-entropy
II	Average softmax-ratio
III	Average softmax-standard-deviation
IV	Maximum of majority-softmax	µ
V	Standard deviation of majority-softmax	σ
VI	Fixed % according to µ	µ%	–
VII	Fixed % according to σ	σ%	–

Notes: Uncertainty measures adapted for majority-voting mechanism of the U-Sleep classifier.

Abbreviations: M, total number of input channel-pairs used; m, index over M; k, index over 5 classes (ie, sleep stages); , probability (ie, softmax-value) of the k-th class based on the m-th input channel pair; P_m, probability vector (ie, softmax) of 5 classes based on the m-th input channel pair; MAX, maximum; SD, standard deviation.

Figure 1 Schematic overview of datasets used, their size, and purpose.

Notes: A set of 13 open-access datasets (in blue) was used for the baseline training of the U-Sleep. The middle and right parts of the schema relate to the evaluations on BSDB. Its ID part refers to PSGs each scored by one of more than 50 assistants and 10 senior physicians. The ID training and validation splits (in yellow) were used to fine-tune U-Sleep and, subsequently, to train the confidence network. Baseline evaluation of both algorithmic approaches was performed on the ID-test data (in orange). Their robustness was further evaluated on two OOD test sets (in red), each containing PSGs scored by a unique SP.

Abbreviations: ID, in-domain; OOD, out-of-domain; SP, senior physician; AP, assistant physician; PSG, polysomnography.

Table 4 Classification Performance of U-Sleep on Individual Data Splits

Domain	Metric	Epoch-Wise	Subject-wise mean	Subject-wise Median
ID-test	Acc	82.5	82.1	84.5
	F1w	82.8	82.4	85.3
	Κ	75.0	71.2	76.2
OOD1	Acc	84.2	84.5	86.4
	F1w	85.0	85.5	87.4
	Κ	77.6	76.0	78.8
OOD2	Acc	80.7	80.8	82.7
	F1w	80.5	81.4	83.4
	Κ	73.3	71.1	73.8

Notes: Epoch-wise performance calculated over all 30-second windows present in individual data splits. Mean and median subject-wise metrics are calculated as performance achieved on individual-specific hypnograms.

Abbreviations: ID, in-domain; OOD, out-of-domain; Acc, accuracy; F1_w, weighted F1-score; K, Cohen’s kappa.

Table 5 Performance of Uncertainty Measures to Identify U-Sleep Predictions Discerning from Human Scoring on Individual Data Splits

Domain	Evaluation metric	Uncertainty Measure
					µ	σ	µ%	σ%	TCP*
ID-test	AUROC	76.4	75.7	76.2	76.5	64.3	59.1	56.5	85.7*
	AUPR	39.7	41.3	41.0	42.9	30.2	36.5	31.4	63.1*
OOD1	AUROC	80.1	82.0	81.6	82.4	75.4	60.6	57.2	85.6*
	AUPR	38.8	42.0	41.0	43.5	41.0	33.3	26.8	53.6*
OOD2	AUROC	79.6	80.8	80.6	81.1	75.0	59.9	57.1	82.5*
	AUPR	43.2	45.0	44.6	45.8	34.1	36.9	31.4	50.7*

Notes: Performance assessment as the % of the AUROC and AUPR curves for the softmax-based measures from Table 3 and the True Class Probability (TCP) score based on confidence network. Bold font highlights the best performance obtained among compared metrics.

Abbreviations: ID, in-domain; OOD, out-of-domain; AUROC, area under receiver operating characteristic (curve); AUPR, area under precision-recall (curve); , Average softmax-entropy; , average softmax-ratio; , average softmax-standard-deviation; µ, maximum of majority-softmax; σ, standard deviation of majority-softmax; µ%, fixed % according to µ; σ%, fixed % according to σ, TCP, true class probability; *, outperforming measure.

Figure 2 Combined output of the predicted hypnogram (in white) and the associated confidence. TCP-scores (in the background), supplemented with the physician-scored hypnogram (in blue).

Notes: Combined output for a 44-year-old female diagnosed with hypersomnolence. On-subject (Acc, F1_w, K) of (79.2, 72.2, 61.5)%, respectively. On-subject average TCP of 0.74. For correctly and incorrectly classified epochs, the average on-subject TCP was 0.87 and 0.41, respectively.

Abbreviation: TCP, true class probability.

Figure 3 Schematic overview of the implemented pipeline.

Notes: An EEG-EOG channel-pair is used as an input for the U-Sleep classifier. Using the trained U-Sleep, several representations are extracted (softmax; binary code indexing the predicted class; hidden representations - hiddens - from the layer preceding softmax) and used as an input for the confidence network evaluating the True Class Probability (TCP) confidence score. The hypnogram predicted by U-Sleep (y) is provided jointly with the assessment of predictive uncertainty (1-TCP) to guide an efficient review by physician.

Abbreviations: N, number of epochs; TCP, true class probability; y, U-Sleep predicted sleep-stages.

Table 6 Bootstrap Confidence Intervals for Difference of on-Subject Mean-Aggregated Confidence TCP-Scores of Aligning Vs Discordant Predictions

Diagnosis Class	ID-test			Pooled OOD
	Median	95% CI	N	Median	95% CI	N
HE	NA			0.26	(0.22, 0.30)	34
INS	0.21	(0.18, 0.24)	19	0.23	(0.21, 0.25)	83
SDB	0.20	(0.17, 0.22)	51	0.21	(0.20, 0.22)	266
CDH	0.23	(0.20, 0.27)	27	0.23	(0.21, 0.24)	180
PSD	NA			0.19	(0.16, 0.21)	67
SMD	NA			0.20	(0.18, 0.23)	52
IS	0.21	(0.18, 0.25)	27	0.22	(0.21, 0.23)	206
DSS	NA			0.25	(0.21, 0.29)	23

Notes: Evaluations on ID-test data and pooled OOD data. Median stands for the estimate of the mean-difference and corresponding 95% CI are calculated as 2.5% and 97.5% quantiles of bootstrap resamples.

Abbreviations: ID, in-domain; OOD, out-of-domain; CI, confidence interval; N, number of subjects; NA, not available; HE, healthy controls; INS, insomnia disorders; SDB, sleep-disordered breathing; CDH, central disorders of hypersomnolence; CRD, circadian rhythm sleep-wake disorders; PSD, parasomnia-related sleep disorders; SMD, sleep-related rhythmic movement disorders; IS, isolated symptoms and normal variants; DSS, findings specific to day-time sleep studies.

Table 7 Bootstrap Confidence Intervals for Correlation Between on-Subject Mean-Aggregated Confidence TCP-Scores and the Performance Metrics

Diagnosis Class	Performance Metric	ID-test			Pooled OOD
		Median	95% CI	N	Median	95% CI	N
HE	Acc	NA			0.74	(0.59, 0.91)	34
	K				0.67	(0.50, 0.89)
	F1w				0.60	(0.41, 0.84)
INS	Acc	0.67	(0.46, 0.91)	19	0.58	(0.46, 0.78)	83
	K	0.56	(0.28, 0.88)		0.59	(0.47, 0.78)
	F1w	0.63	(0.39, 0.91)		0.49	(0.36, 0.72)
SDB	Acc	0.71	(0.62, 0.85)	51	0.71	(0.66, 0.80)	266
	K	0.69	(0.58, 0.85)		0.68	(0.62, 0.77)
	F1w	0.57	(0.44, 0.78)		0.65	(0.59, 0.75)
CDH	Acc	0.72	(0.55, 0.90)	27	0.75	(0.69, 0.84)	180
	K	0.64	(0.45, 0.86)		0.72	(0.66, 0.82)
	F1w	0.58	(0.35, 0.85)		0.68	(0.61, 0.79)
PSD	Acc	NA			0.81	(0.74, 0.89)	67
	K				0.81	(0.74, 0.90)
	F1w				0.78	(0.70, 0.87)
Diagnosis Class	Performance Metric	ID-test			Pooled OOD
		Median	95% CI	N	Median	95% CI	N
SMD	Acc	NA			0.63	(0.47, 0.84)	52
	K				0.54	(0.37, 0.79)
	F1w				0.55	(0.39, 0.79)
IS	Acc	0.74	(0.58, 0.90)	27	0.70	(0.64, 0.80)	206
	K	0.74	(0.59, 0.90)		0.64	(0.57, 0.76)
	F1w	0.62	(0.40, 0.87)		0.63	(0.56, 0.75)
DSS	Acc	NA			0.62	(0.34, 0.92)	23
	K				0.62	(0.36, 0.93)
	F1w				0.49	(0.21, 0.86)

Notes: Evaluations on ID-test data and pooled OOD data. Median stands for the estimate of correlation with a performance metric and corresponding 95% CI are calculated as 2.5% and 97.5% quantiles of bootstrap resamples.

Abbreviations: ID, in-domain; OOD, out-of-domain; CI, confidence interval; N, number of subjects; NA, not available; K, Cohen’s kappa; Acc, accuracy; F1_w, weighted F1-score; HE, healthy controls; INS, insomnia disorders; SDB, sleep-disordered breathing; CDH, central disorders of hypersomnolence; CRD, circadian rhythm sleep-wake disorders; PSD, parasomnia-related sleep disorders; SMD, sleep-related rhythmic movement disorders; IS, isolated symptoms and normal variants; DSS, findings specific to day-time sleep studies.

Figure 4 Performance boost with physician’s review of epochs having confidence TCP-score lower than a given threshold.

Abbreviations: ID, in-domain; OOD, out-of-domain; K, Cohen’s kappa; Acc, accuracy; F1_w, weighted F1-score.

Table 8 Rescoring Amounts Needed to Achieve Desired Levels of Sleep-Scoring Performance

Domain	Metric	Desired Scoring Performance Level
		80%	85%	90%	95%
ID-test	K	7.6	15.7	25.6	41.5
	Acc	0	6.0	16.5	32.2
	F1w	0	6.2	17.2	33.7
OOD1	K	2.3	9.1	18.8	32.9
	Acc	0	1.1	9.8	25.5
	F1w	0	0.1	10.5	25.5
OOD2	K	8.3	17.3	29.0	44.0
	Acc	0	6.7	19.0	37.0
	F1w	0	8.3	21.9	39.1

Abbreviations: ID, in-domain; OOD, out-of-domain; K, Cohen’s kappa; Acc, accuracy; F1_w, weighted F1-score.

Figure 5 Review amounts (% of epochs exported) versus the % of all discordant predictions gathered.

Abbreviations: ID, in-domain; OOD, out-of-domain.