Assessing unknown potential—quality and limitations of different large language models in the field of otorhinolaryngology

References

• Cabrera J, Loyola MS, Magaña I, et al. Ethical dilemmas, mental health, artificial intelligence, and LLM-based chatbots. Bioinformatics and Biomedical Engineering. Lecture Notes in Computer Science; 2023. p. 313–326.
   Google Scholar
• Nicholas PK, Smith MF. Demographic challenges and health in Germany. Popul Res Policy Rev. 2007;25(5–6):479–487. doi: 10.1007/s11113-006-9009-2.
   Web of Science ®Google Scholar
• Van Bokkelen G, Morsy M, Kobayashi T. Demographic transition, health care challenges, and the impact of emerging international regulatory trends with relevance to regenerative medicine. Curr Stem Cell Rep. 2015;1(2):102–109. doi: 10.1007/s40778-015-0013-5.
   Google Scholar
• Dawkins B, Renwick C, Ensor T, et al. What factors affect patients’ ability to access healthcare? An overview of systematic reviews. Trop Med Int Health. 2021;26(10):1177–1188. doi: 10.1111/tmi.13651.
   PubMed Web of Science ®Google Scholar
• Tu T, Palepu A, Schaekermann M, et al. Towards conversational diagnostic AI. arXiv Preprint. 2024:arXiv:240105654.
   Google Scholar
• Radford A, Wu J, Child R, et al. editors. Language models are unsupervised multitask learners; Technical report, OpenAi, 2019.
   Google Scholar
• Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need. In: Advances in neural information processing systems. Vol. 30; 2017. Computation and Language (cs.CL); Machine Learning (cs.LG)
   Google Scholar
• Buhr CR, Smith H, Huppertz T, et al. ChatGPT versus consultants: blinded evaluation on answering otorhinolaryngology case–based questions. JMIR Med Educ. 2023;9:e49183. doi: 10.2196/49183.
   PubMedGoogle Scholar
• Dallari V, Sacchetto A, Saetti R, et al. Is artificial intelligence ready to replace specialist doctors entirely? ENT specialists vs ChatGPT: 1-0, ball at the center. Eur Arch Otorhinolaryngol. 2023;281(2):995–1023. doi: 10.1007/s00405-023-08321-1.
   PubMed Web of Science ®Google Scholar
• Chee J, Kwa ED, Goh X. “Vertigo, likely peripheral”: the dizzying rise of ChatGPT. Eur Arch Otorhinolaryngol. 2023;280(10):4687–4689. doi: 10.1007/s00405-023-08135-1.
   PubMed Web of Science ®Google Scholar
• Hoch CC, Wollenberg B, Lüers J-C, et al. ChatGPT’s quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. 2023;280(9):4271–4278. doi: 10.1007/s00405-023-08051-4.
   PubMed Web of Science ®Google Scholar
• Chiesa-Estomba CM, Lechien JR, Vaira LA, et al. Exploring the potential of Chat-GPT as a supportive tool for sialendoscopy clinical decision making and patient information support. Eur Arch Otorhinolaryngol. 2023;281(5):2777–2777. doi: 10.1007/s00405-023-08267-4.
   Web of Science ®Google Scholar
• Qu RW, Qureshi U, Petersen G, et al. Diagnostic and management applications of ChatGPT in structured otolaryngology clinical scenarios. OTO Open. 2023;7(3):e67. doi: 10.1002/oto2.67.
   PubMedGoogle Scholar
• Nielsen JPS, von Buchwald C, Grønhøj C. Validity of the large language model ChatGPT (GPT4) as a patient information source in otolaryngology by a variety of doctors in a tertiary otorhinolaryngology department. Acta Otolaryngol. 2023;143(9):779–782. doi: 10.1080/00016489.2023.2254809.
   PubMed Web of Science ®Google Scholar
• Ayoub NF, Lee YJ, Grimm D, et al. Head-to-head comparison of ChatGPT versus google search for medical knowledge acquisition. Otolaryngol Head Neck Surg. 2023;1–8. doi: 10.1002/ohn.465.
   Web of Science ®Google Scholar
• Turing AM. I.—Computing machinery and intelligence. Mind. 1950;LIX(236):433–460. doi: 10.1093/mind/LIX.236.433.
   Google Scholar
• Reineke U, Riemann R. Facharztprüfung Hals-Nasen-Ohrenheilkunde: 1000 kommentierte prüfungsfragen. Stuttgart, Germany: Thieme; 2007.
   Google Scholar
• Thomas Lenarz H-GB. Hals-Nasen-Ohren-Heilkunde. Germany, Neu-Isenburg: Springer Medizin Verlag GmbH; 2012.
   Google Scholar
• Warrier A, Singh R, Haleem A, et al. The comparative diagnostic capability of large language models in otolaryngology. Laryngoscope. 2024;1–6. doi: 10.1002/lary.31434.
   PubMed Web of Science ®Google Scholar
• Saibene AM, Allevi F, Calvo-Henriquez C, et al. Reliability of large language models in managing odontogenic sinusitis clinical scenarios: a preliminary multidisciplinary evaluation. Eur Arch Otorhinolaryngol. 2024;281(4):1835–1841. doi: 10.1007/s00405-023-08372-4.
   PubMed Web of Science ®Google Scholar
• Pugliese G, Maccari A, Felisati E, et al. Are artificial intelligence large language models a reliable tool for difficult differential ­diagnosis? An a posteriori analysis of a peculiar case of necrotizing otitis externa. Clin Case Rep. 2023;11(9):e7933. doi: 10.1002/ccr3.7933.
   PubMed Web of Science ®Google Scholar
• Zalzal HG, Abraham A, Cheng J, et al. Can ChatGPT help ­patients answer their otolaryngology questions? Laryngoscope Investig Otolaryngol. 2023;9(1):e1193. doi: 10.1002/lio2.1193.
   PubMed Web of Science ®Google Scholar
• Zalzal HG, Cheng J, Shah RK. Evaluating the current ability of ChatGPT to assist in professional otolaryngology education. OTO Open. 2023;7(4):e94. doi: 10.1002/oto2.94.
   PubMedGoogle Scholar
• Liu J, Wang C, Liu S. Utility of ChatGPT in clinical practice. J Med Internet Res. 2023;25:e48568. doi: 10.2196/48568.
   PubMed Web of Science ®Google Scholar
• Long C, Lowe K, Zhang J, et al. A novel evaluation model for assessing ChatGPT on otolaryngology–head and neck surgery certification examinations: performance study. JMIR Med Educ. 2024;10:e49970. doi: 10.2196/49970.
   PubMedGoogle Scholar
• Noda M, Ueno T, Koshu R, et al. Performance of GPT-4V in answering the Japanese otolaryngology board certification examination questions: evaluation study. JMIR Med Educ. 2024;10:e57054. doi: 10.2196/57054.
   PubMedGoogle Scholar
• Shen SA, Perez-Heydrich CA, Xie DX, et al. ChatGPT vs. web search for patient questions: what does ChatGPT do better? Eur Arch Otorhinolaryngol. 2024;281(6):3219–3225. doi: 10.1007/s00405-024-08524-0.
   PubMedGoogle Scholar
• Dhar S, Kothari D, Vasquez M, et al. The utility and accuracy of ChatGPT in providing post-operative instructions following tonsillectomy: a pilot study. Int J Pediatr Otorhinolaryngol. 2024;179:111901. doi: 10.1016/j.ijporl.2024.111901.
   PubMed Web of Science ®Google Scholar
• Eriksen AV, Möller S, Ryg J. Use of GPT-4 to diagnose complex clinical cases. NEJM AI. 2023;1(1). doi: 10.1056/AIp2300031.
   Google Scholar
• Meskó B, Topol EJ. The imperative for regulatory oversight of large language models (or generative AI) in healthcare. NPJ Digit Med. 2023;6(1):120. doi: 10.1038/s41746-023-00873-0.
   PubMed Web of Science ®Google Scholar
• Kuşcu O, Pamuk AE, Sütay Süslü N, et al. Is ChatGPT accurate and reliable in answering questions regarding head and neck cancer? Front Oncol. 2023;13:1256459. doi: 10.3389/fonc.2023.1256459.
   PubMed Web of Science ®Google Scholar