Assessing unknown potential—quality and limitations of different large language models in the field of otorhinolaryngology

Figures & data

Figure 1. Comparison between ORL consultants and the different LLMs (ChatGPT4, Bard 2023.07.13, Claude 2) for all evaluated categories. Data shown as a scatter dot blot with points representing absolute values and bar width representing amount of individual values. Horizontal lines represent mean (95% CI). Normality distribution was tested with the D’Agostino and Pearson test. Multi-group comparisons were performed using the Kruskal-Wallis-Test. ns > .05, *p < .05, **p < .01, ***p < .001, ^****p < .0001.

Figure 2. The number of characters per answer used by ORL consultants and the different LLMs (ChatGPT 4, Bard 2023.07.13, Claude 2). Data shown as a scatter dot blot with each point resembling an absolute value. Horizontal lines represent the median. Normality distribution was tested with the D’Agostino and Pearson test. Multi-group comparisons were performed using the Kruskal-Wallis-Test. ^****p < .001.

Table 1. Correlation Analysis for ratings for Medical Adequacy, Comprehensibility, and Coherence, Conciseness and the number of characters used. (ns >< .05, *p < .01, **p < .01, ****p < .001, ;****p < .0001).

	Med. adequacy	Comprehensibility	Coherence	Conciseness	Number of characters (median)
ORL consultants	0.42^**** (0.30 to 0.52)	0.47^**** (0.37 to 0.57)	0.47^**** (0.35 to 0.55)	−0.15* (−0.27 to −0.02)	89 (31.5 to 175.0)
ChatGPT 4	0.16 (−0.17 to 0.45)	0.12 (−0.20 to 0.42)	−0.03 (−0.34 to 0.29)	−0.34* (−0.59 to −0.03)	1229 (955.5 to 1588)
Bard 2023.07.13	0.38* (0.07 to 0.62)	0.09 (−0.23 to 0.40)	0.32* (0.00 to 0.57)	−0.24 (−0.52 to 0.08)	1748 (1568 to 2204)
Claude 2	0.15 (−0.18 to 0.44)	0.29 (−0.03 to 0.55)	0.08 (−0.24 to 0.39)	−0.05 (−0.36 to 0.27)	1088 (954.5 to 1211)