Evaluating the Sensitivity, Specificity, and Accuracy of ChatGPT-3.5, ChatGPT-4, Bing AI, and Bard Against Conventional Drug-Drug Interactions Clinical Tools

Figures & data

Table 1 DDIs Clinical Tools Characteristics and Standardization for Severity

Access	Drugs.com	Micromedex
	Free	Subscription
Severity Scale
○ None	No interactions were found	No interactions have been identified
○ Minor	Minor	Minor
○ Moderate	Moderate	Moderate
○ Major/Contraindicated	Major	Major, Contraindicated

Table 2 Medications Pairs Included

SGLT2 Inhibitors and Macrolides	Concomitant Medications
Empagliflozin	Atorvastatin	Escitalopram	Clopidogrel	Hydrocodone
Dapagliflozin	Levothyroxine	Acetaminophen	Prednisone	Venlafaxine
Canagliflozin	Metformin	Rosuvastatin	Citalopram	Clonazepam
Clarithromycin	Lisinopril	Bupropion	Insulin Glargine	Ethinyl Estradiol
Azithromycin	Amlodipine	Furosemide	Potassium Chloride	Norethindrone
	Metoprolol	Pantoprazole	Pravastatin	Ergocalciferol
	Albuterol	Trazodone	Tramadol	Zolpidem
	Omeprazole	Amphetamine	Aspirin	Apixaban
	Losartan	Fluticasone	Alprazolam	Glipizide
	Gabapentin	Tamsulosin	Ibuprofen	Montelukast
	Hydrochlorothiazide	Fluoxetine	Cyclobenzaprine	Meloxicam
	Sertraline	Carvedilol	Amoxicillin	Allopurinol
	Simvastatin	Duloxetine	Methylphenidate

Abbreviation: SGLT2i, sodium-glucose co-transporter 2 inhibitors.

Figure 1 The number of DDIs identified by each database categorized by the severity of the interaction. The colors of the bars represent the different levels of severity of the drug-drug interactions (DDIs). Abbreviation: DDIs, drug-drug interactions.

Table 3 Sensitivity, Specificity, and Accuracy Compared to Micromedex as a Reference

Program	TP	FN	TN	FP	Sensitivity	Specificity	PPV	NPV	Accuracy
Microsoft Bing AI (N=255)	41	6	160	48	0.872	0.769	0.461	0.964	0.788
Microsoft Bing AI (N=180)	41	2	89	48	0.953	0.650	0.461	0.978	0.722
ChatGPT-3.5 (N=255)	42	4	78	130	0.913	0.375	0.244	0.951	0.472
ChatGPT-3.5 (N=254)	42	5	77	130	0.894	0.372	0.244	0.939	0.469
ChatGPT-4 (255)	43	4	108	100	0.915	0.519	0.301	0.964	0.592
Bard (255)	39	8	116	92	0.830	0.558	0.298	0.935	0.608
Bard (240)	39	1	108	92	0.975	0.540	0.298	0.991	0.613

Abbreviations: TP, true positive; TN, true negative; FP, false positive; FN, false negative; PPV, positive predictive value; NPV, negative predictive value.

Table 4 Sensitivity, Specificity, and Accuracy Compared to Drugs.com as a Reference

Program	TP	FN	TN	FP	Sensitivity	Specificity	PPV	NPV	Accuracy
Microsoft Bing AI (N=255)	70	9	157	19	0.886	0.892	0.787	0.946	0.890
Microsoft Bing AI (N=180)	70	4	87	19	0.946	0.821	0.787	0.956	0.872
ChatGPT-3.5 (N=255)	65	14	69	107	0.823	0.392	0.378	0.831	0.525
ChatGPT-3.5 (N=254)	65	14	68	107	0.823	0.389	0.378	0.829	0.524
ChatGPT-4 (255)	59	20	92	84	0.747	0.523	0.413	0.821	0.592
Bard (255)	65	14	110	66	0.823	0.625	0.496	0.887	0.686
Bard (240)	65	3	106	66	0.956	0.616	0.496	0.972	0.713

Abbreviations: TP, true positive; TN, true negative; FP, false positive; FN, false negative; PPV, positive predictive value; NPV, negative predictive value.

Figure 2 Accuracy of AI tools to detect DDIs categorized by two drug classes (drugs.com as a standard). Abbreviations: DDIs, drug-drug interactions; SGLT2i, sodium-glucose co-transporter 2 inhibitors.

Figure 3 Specificity difference of the AI tools to detect DDIs of two medications within the same class (drugs.com as a standard). Abbreviation: DDIs, drug-drug interactions.