Predicting Antibiotic Resistance in ICUs Patients by Applying Machine Learning in Vietnam

Figures & data

Table 1 Description of Datasets Used for Machine Learning Predictive Models

Characteristics	Dataset 1 (Demographic, Clinical, AST, and Complete Blood Count Data)		Dataset 2 (Demographic, Clinical, AST, Complete Blood Count, and Biochemical Data)
Number (n)
Patients	684		560
Health records	953		755
AST tests	1537		1182
Complete blood count tests	7358		5924
Biochemical tests	–		4058
Entries	10,658		61,273
	n	%	n	%
Men	440	64.4	352	63.8
Year of admission
2020	363	25.6	288	26.1
2021	649	45.7	551	49.9
2022	408	28.7	265	24.0
Primary diagnosis at admission by ICD-10
Respiratory (J00-J99)	379	26.7	273	24.7
COVID-19 (U07)	330	23.2	269	24.4
Circulatory system (I00-I99)	225	15.8	178	16.1
Infection (A00-B99)	193	13.6	155	14.0
Abnormal and not classified (R00-R99)	101	7.1	78	7.1
Other diseases*	192	13.5	151	13.7
Number of non-duplicated bacterial isolates
Klebsiella spp	418	29.4	322	29.2
Acinetobacter spp	387	27.3	308	27.9
Pseudomonas aeruginosa	206	14.5	151	13.7
Escherichia coli	71	5.0	60	5.4
Staphylococcus aureus	70	4.9	55	5.0
Other	268	18.9	208	18.8
Median [IQR]
Age at admission	61	[50–70]	61	[50–70]
Length of stay in hospital (days)	14	[9–23]	14	[9–23]
Length of stay in ICU (days)	12	[7–21]	13	[8–20]

Notes: *Other diseases included: diseases of the musculoskeletal system, endocrine, neoplasms, skin, blood, and immune mechanism, mental disorders, and external causes of morbidity.

Abbreviations: AST, Antimicrobial Susceptibility Test; COVID-19, Coronavirus disease 2019; ICD-10, International Code of Disease – 10th version.

Table 2 Comparing the Performance of Different Machine Learning Algorithms on the Dataset 1

	Performance Indicators	LightGBM	Random Forest	XGBoost	AdaBoost	Ridge Logistic Regression
Acinetobacter spp	Sensitivity	0.849	0.609	0.897	0.582	0.544
	Specificity	0.759	0.754	0.769	0.655	0.583
	Precision	0.899	0.783	0.949	0.716	0.630
	Accuracy	0.898	0.795	0.949	0.737	0.727
	F1-score	0.894	0.783	0.948	0.648	0.620
	AuROC	0.958	0.865	0.982	0.725	0.615
	PRC	0.713	0.461	0.853	0.295	0.272
	NormMCC	0.866	0.719	0.934	0.570	0.509
Klebsiella spp	Sensitivity	0.802	0.643	0.816	0.588	0.535
	Specificity	0.780	0.738	0.818	0.591	0.613
	Precision	0.875	0.776	0.927	0.722	0.629
	Accuracy	0.873	0.790	0.928	0.735	0.731
	F1-score	0.864	0.777	0.926	0.629	0.621
	AuROC	0.946	0.857	0.977	0.682	0.622
	PRC	0.642	0.445	0.789	0.276	0.269
	NormMCC	0.830	0.710	0.905	0.540	0.506
Pseudomonas aeruginosa	Sensitivity	0.801	0.623	0.871	0.591	0.513
	Specificity	0.808	0.755	0.734	0.609	0.618
	Precision	0.933	0.846	0.957	0.780	0.792
	Accuracy	0.935	0.875	0.958	0.883	0.883
	F1-score	0.927	0.855	0.955	0.828	0.828
	AuROC	0.966	0.899	0.977	0.723	0.641
	PRC	0.509	0.180	0.677	0.117	0.117
	NormMCC	0.821	0.616	0.890	0.499	0.501
Escherichia coli	Sensitivity	0.969	0.641	0.981	0.844	0.619
	Specificity	0.457	0.941	0.471	0.579	0.681
	Precision	0.992	0.973	0.992	0.961	0.954
	Accuracy	0.992	0.976	0.992	0.969	0.968
	F1-score	0.991	0.972	0.992	0.959	0.955
	AuROC	0.992	0.983	0.993	0.947	0.834
	PRC	0.755	0.308	0.775	0.111	0.062
	NormMCC	0.930	0.762	0.936	0.636	0.581
Staphylococcus aureus	Sensitivity	0.937	0.539	0.975	0.867	0.545
	Specificity	0.527	0.932	0.517	0.543	0.780
	Precision	0.987	0.966	0.988	0.955	0.938
	Accuracy	0.988	0.971	0.988	0.965	0.964
	F1-score	0.987	0.965	0.987	0.950	0.947
	AuROC	0.982	0.972	0.991	0.916	0.768
	PRC	0.671	0.246	0.683	0.061	0.038
	NormMCC	0.903	0.726	0.907	0.577	0.510

Table 3 Comparing the Performance of Different Machine Learning Algorithms on the Dataset 2

	Performance Indicators	LightGBM	Random Forest	XGBoost	AdaBoost	Ridge Logistic Regression
Acinetobacter spp	Sensitivity	0.882	0.885	0.900	0.684	0.554
	Specificity	0.649	0.867	0.583	0.642	0.640
	Precision	0.967	0.971	0.988	0.730	0.662
	Accuracy	0.967	0.971	0.988	0.747	0.713
	F1-score	0.966	0.971	0.988	0.702	0.631
	AuROC	0.996	0.990	0.999	0.812	0.685
	PRC	0.914	0.920	0.968	0.379	0.307
	NormMCC	0.959	0.965	0.985	0.643	0.555
Klebsiella spp	Sensitivity	0.891	0.869	0.946	0.691	0.562
	Specificity	0.619	0.901	0.548	0.605	0.634
	Precision	0.968	0.976	0.988	0.788	0.717
	Accuracy	0.968	0.976	0.988	0.771	0.747
	F1-score	0.967	0.976	0.988	0.703	0.652
	AuROC	0.997	0.996	0.998	0.798	0.676
	PRC	0.902	0.923	0.901	0.335	0.270
	NormMCC	0.957	0.969	0.969	0.639	0.551
Pseudomonas aeruginosa	Sensitivity	0.938	0.844	0.884	0.802	0.520
	Specificity	0.542	0.939	0.578	0.567	0.650
	Precision	0.982	0.977	0.990	0.908	0.807
	Accuracy	0.982	0.977	0.990	0.898	0.895
	F1-score	0.981	0.977	0.990	0.852	0.845
	AuROC	0.997	0.990	0.999	0.854	0.641
	PRC	0.846	0.802	0.969	0.129	0.105
	NormMCC	0.951	0.937	0.987	0.578	0.499
Escherichia coli	Sensitivity	0.992	0.817	0.996	0.970	0.656
	Specificity	0.448	0.984	0.427	0.525	0.791
	Precision	0.999	0.999	1.000	0.982	0.973
	Accuracy	0.999	0.999	1.000	0.984	0.979
	F1-score	0.999	0.999	1.000	0.980	0.972
	AuROC	1.000	1.000	1.000	0.991	0.895
	PRC	0.975	0.933	0.993	0.275	0.082
	NormMCC	0.993	0.982	0.998	0.752	0.620
Staphylococcus aureus	Sensitivity	0.981	0.807	0.993	0.951	0.563
	Specificity	0.451	0.980	0.451	0.528	0.760
	Precision	0.998	0.997	0.999	0.978	0.948
	Accuracy	0.998	0.997	0.999	0.980	0.971
	F1-score	0.998	0.997	0.999	0.976	0.958
	AuROC	1.000	0.999	1.000	0.981	0.827
	PRC	0.917	0.908	0.961	0.315	0.029
	NormMCC	0.978	0.975	0.990	0.768	0.510

Notes: Values of accuracy, F1-score, and normalized Matthews correlation coefficient range from 0 to 1, where 1 is the best possible classification, and 0 is random classification.

Abbreviations: AuROC, Area Under the Receiver Operating Characteristic curve; F1-score, harmonic mean of precision and recall; PRC, Precision–Recall curve; Norm MCC, Normalized Matthew’s Correlation Coefficient; LightGBM, gradient-boosting-based machine learning package developed by Microsoft; XGBoost, Extreme Gradient Boosting Decision Tree; AdaBoost, Adaptive Boosting Decision Tree.

Table 4 Performance of LightGBM Model (5-Fold Cross-Validation) to Predict Resistance to Antibiotic Families and Between Bacteria and Antibiotic Families in Dataset 1

	Sensitivity	Specificity	Precision	Accuracy	F1-Score	AuROC	PRC	normMCC
Aminoglycoside	0.794	0.827	0.908	0.907	0.903	0.969	0.902	0.873
Klebsiella spp	0.818	0.795	0.929	0.930	0.925	0.973	0.660	0.869
Pseudomonas aeruginosa	0.803	0.796	0.955	0.957	0.954	0.975	0.564	0.853
Escherichia coli	0.985	0.417	0.996	0.996	0.996	1.000	0.801	0.945
Staphylococcus aureus	0.978	0.376	0.995	0.996	0.995	0.994	0.699	0.916
Polymyxins	0.848	0.781	0.947	0.948	0.946	0.985	0.811	0.923
Klebsiella spp	0.859	0.765	0.957	0.958	0.956	0.982	0.675	0.890
Fluoroquinolones	0.707	0.898	0.966	0.967	0.966	0.986	0.968	0.929
Staphylococcus aureus	0.978	0.351	0.996	0.996	0.996	0.998	0.704	0.915
Carbapenems	0.742	0.862	0.952	0.953	0.951	0.984	0.957	0.902
Escherichia coli	0.920	0.443	0.998	0.998	0.997	0.978	0.771	0.937
Trimethoprim derivatives	0.733	0.888	0.932	0.932	0.93	0.981	0.926	0.914
Escherichia coli	0.985	0.405	0.996	0.996	0.996	0.998	0.826	0.952
Staphylococcus aureus	0.976	0.431	0.995	0.995	0.995	0.998	0.628	0.894
Fourth generation cephalosporin	0.791	0.879	0.958	0.959	0.957	0.986	0.961	0.910
Pseudomonas aeruginosa	0.879	0.816	0.962	0.963	0.961	0.981	0.665	0.889
Escherichia coli	0.985	0.357	0.995	0.995	0.994	0.998	0.808	0.946

Table 5 Performance of LightGBM Model (5-Fold Cross-Validation) to Predict Resistance to Antibiotic Families and Between Bacteria and Antibiotic Families in Dataset 2

	Sensitivity	Specificity	Precision	Accuracy	F1-Score	AuROC	PRC	normMCC
Aminoglycoside	0.616	0.890	0.983	0.983	0.983	0.999	0.982	0.976
Klebsiella spp	0.942	0.493	0.987	0.987	0.986	0.999	0.934	0.976
Pseudomonas aeruginosa	0.948	0.538	0.986	0.986	0.985	0.997	0.827	0.948
Escherichia coli	0.990	0.688	1.000	1.000	1.000	1.000	0.995	0.999
Staphylococcus aureus	0.992	0.670	0.998	0.998	0.998	1.000	0.889	0.971
Polymyxins	0.882	0.546	0.991	0.991	0.991	1.000	0.959	0.985
Klebsiella spp	0.921	0.482	0.991	0.991	0.991	1.000	0.923	0.977
Fluoroquinolones	0.556	0.917	0.989	0.989	0.988	0.999	0.987	0.978
Staphylococcus aureus	0.985	0.672	0.999	0.999	0.999	1.000	0.940	0.984
Carbapenems	0.494	0.961	0.993	0.993	0.993	1.000	0.995	0.985
Escherichia coli	0.998	0.550	0.999	0.999	0.999	1.000	0.878	0.968
Trimethoprim derivatives	0.568	0.856	0.990	0.990	0.990	0.999	0.989	0.988
Escherichia coli	0.983	0.688	1.000	1.000	1.000	1.000	0.992	0.998
Staphylococcus aureus	0.996	0.538	0.999	0.999	0.999	1.000	0.913	0.977
Fourth generation cephalosporin	0.519	0.963	0.986	0.986	0.985	0.999	0.991	0.958
Pseudomonas aeruginosa	0.958	0.502	0.992	0.992	0.992	0.999	0.926	0.978
Escherichia coli	0.997	0.396	0.999	0.999	0.999	1.000	0.962	0.990

Table 6 Performance of LightGBM Model (5-Fold Cross-Validation) to Predict Resistance to Antibiotic Families and Between Bacteria and Antibiotic Families in Reference Dataset

	Sensitivity	Specificity	Precision	Accuracy	F1-Score	AuROC	PRC	normMCC
Aminoglycoside	0.566	0.553	0.620	0.659	0.633	0.569	0.727	0.530
Klebsiella spp	0.456	0.550	0.711	0.778	0.741	0.469	0.161	0.472
Pseudomonas aeruginosa	0.431	0.615	0.787	0.829	0.806	0.557	0.123	0.504
Escherichia coli	0.371	0.819	0.935	0.960	0.947	0.669	0.033	0.492
Staphylococcus aureus	0.101	0.855	0.972	0.980	0.976	0.546	0.014	0.496
Fluoroquinolones	0.582	0.544	0.718	0.763	0.733	0.633	0.811	0.563
Staphylococcus aureus	0.160	0.815	0.974	0.984	0.979	0.537	0.013	0.498
Carbapenems	0.614	0.540	0.712	0.746	0.722	0.639	0.798	0.587
Escherichia coli	0.085	0.820	0.972	0.984	0.978	0.571	0.014	0.497
Trimethoprim derivatives	0.534	0.481	0.584	0.620	0.595	0.539	0.684	0.526
Escherichia coli	0.397	0.831	0.899	0.938	0.918	0.767	0.052	0.489
Staphylococcus aureus	0.173	0.847	0.968	0.981	0.975	0.547	0.016	0.497
Fourth generation cephalosporin	0.595	0.525	0.741	0.788	0.757	0.616	0.831	0.562
Pseudomonas aeruginosa	0.476	0.579	0.759	0.812	0.783	0.557	0.135	0.486
Escherichia coli	0.518	0.799	0.911	0.943	0.926	0.738	0.047	0.497

Abbreviations: TP Rate, True positive rate; FP Rate, False positive rate; normMCC, Normalized Matthew’s Correlation Coefficient; AuROC, Area Under the Receiver Operating Characteristic curve; PRC, Precision–Recall curve; F-measure, harmonic mean of precision and recall.