Designing and building OSCEBot ® for virtual OSCE – Performance evaluation

ABSTRACT

With AI’s advancing technology and chatbots becoming more intertwined in our daily lives, pedagogical challenges are occurring. While chatbots can be used in various disciplines, they play a particularly significant role in medical education. We present the development process of OSCEBot ®, a chatbot to train medical students in the clinical interview approach. The SentenceTransformers, or SBERT, framework was used to develop this chatbot. To enable semantic search for various phrases, SBERT uses siamese and triplet networks to build sentence embeddings for each sentence that can then be compared using a cosine-similarity. Three clinical cases were developed using symptoms that followed the SOCRATES approach. The optimal cutoffs were determined, and each case’s performance metrics were calculated. Each question was divided into different categories based on their content. Regarding the performance between cases, case 3 presented higher average confidence values, explained by the continuous improvement of the cases following the feedback acquired after the sessions with the students. When evaluating performance between categories, it was found that the mean confidence values were highest for previous medical history. It is anticipated that the results can be improved upon since this study was conducted early in the chatbot deployment process. More clinical scenarios must be created to broaden the options available to students.

KEYWORDS:

• Chatbots
• Medical education
• Natural language processing
• Technology-enhanced learning, BERT
• OSCEs

Acknowledgments

All authors contributed to the final version of the manuscript and gave their final approval of the version to be published.

Disclosure statement

No potential conflict of interest was reported by the authors.

Availability of data and materials

The datasets generated during and/or analyzed during the current study are not publicly available due confidentiality issues but are available from the corresponding author on reasonable request.

Declarations

Ethics approval and consent to participate.

Additional information

Funding

The work was supported by the Fundação para a Ciência e a Tecnologia .