Artificial intelligence performance in clinical neurology queries: the ChatGPT model

ABSTRACT

Introduction

The use of artificial intelligence technology is progressively expanding and advancing in the health and biomedical literature. Since its launch, ChatGPT has rapidly gained popularity and become one of the fastest-growing artificial intelligence applications in history. This study evaluated the accuracy and comprehensiveness of ChatGPT-generated responses to medical queries in clinical neurology.

Methods

We directed 216 questions from different subspecialties to ChatGPT. The questions were classified into three categories: multiple-choice, descriptive, and binary (yes/no answers). Each question in all categories was subjectively rated as easy, medium, or hard according to its difficulty level. Questions that also tested for intuitive clinical thinking and reasoning ability were evaluated in a separate category.

Results

ChatGPT correctly answered 141 questions (65.3%). No significant difference was detected in the accuracy and comprehensiveness scale scores or correct answer rates in comparisons made according to the question style or difficulty level. However, a comparative analysis assessing question characteristics revealed significantly lower accuracy and comprehensiveness scale scores and correct answer rates for questions based on interpretations that required critical thinking (p = 0.007, 0.007, and 0.001, respectively).

Conclusion

ChatGPT had a moderate overall performance in clinical neurology and demonstrated inadequate performance in answering questions that required interpretation and critical thinking. It also displayed limited performance in specific subspecialties. It is essential to acknowledge the limitations of artificial intelligence and diligently verify medical information produced by such models using reliable sources.

KEYWORDS:

• ChatGPT
• artificial intelligence
• clinical neurology
• accuracy
• comprehensiveness

Disclosure statement

No potential conflict of interest was reported by the author(s).

Author contributions

Concept- EA; Design- EA; Supervision- YEF; Resource- EA; Materials- GBC, EA; Data Collection and/or Processing- EKC; Analysis and/or Interpretation-GBC, EA; Literature Search- EA, EKC; Writing- YEF, EA; Critical Reviews- EA.

Data availability statement

The data underlying this article will be shared on reasonable request to the corresponding author.

Additional information

Funding

This work was supported by the no funding associated with the work featured.