Designing and building OSCEBot ® for virtual OSCE – Performance evaluation

Figures & data

Table 1. RIPE Principles. Adapted from Kumar, 2021.

	DESCRIPTION	CONTEXT
Reliability	The chatbot should be simple to use and accessible through a secure platform so learners can rely on it to get ongoing feedback securely.	Using specific software for student assessment (quizOne) improves the interaction between the student and the software while maintaining information security.
interpersonal communication	The chatbot (hence, patient) should establish a relationship with the learner using NLP, figures of speech, popular expressions, etc.	To emulate a real patient, chatbots must mimic interpersonal communication.
pedagogic	The chatbot provides learning content and activities that align with the learning. Therefore, while facilitating a personalized learning platform, the chatbot should also embody active learning and communication strategies that allow the instructor to monitorlearning progress learning progress	After the interaction between the chatbot and the student, it can be viewed by teachers and educators. In addition, at the end of the interaction, students will be able to review the clinical interview and put the most likely diagnosis hypotheses. After that, the chatbot will provide immediate feedback and identify the strengths and weaknesses of the interview.
experience	The chatbot should be used on a preferred communication platform and mimic how patients naturally communicate. To increase emotional engagement, affective contact such as greetings, comedy, emoticons, and/or empathy should be used.	The chatbot will use NLP to better engage students. Students will also be presented with opportunities for empathetic interaction, where the patient presents their concerns, and the student is expected to respond appropriately.

Figure 1. Chatbot architecture.

Table 2. Confusion Matrix.

		True Class
Hypothesized class		+	-
	+	True Positives	False Positives
	-	False Negatives	True Negatives

Table 3. Performance Measures [33,34].

Measures	Meaning	Equation
Accuracy	Efficiency of an algorithm to accurately categorize examples from a given dataset.	$Accuracy=\frac{TP+TN}{\left(TP+TN+FP+FN\right)}\left(1\right)$
Precision	Performance between the information asked and the response provided.	$Precision=\frac{TP}{\left(TP+FP\right)}\left(2\right)$
Recall/Sensitivity	How well a system performs while recovering data.	$Recall=\frac{TP}{\left(TP+FN\right)}\left(3\right)$
F Score	Evaluation of a model’s performance by considering both precision and recall.	$FScore=\frac{\left(2\left(Precision\times Recall\right)\right)}{\left(Precision+Recall\right)}\left(4\right)$
Specificity	Ability of the system to identify true negatives.	$Specificity=\frac{TN}{\left(TN+FP\right)}\left(5\right)$

Table 4. Performance measures for Youden Index and Closest top-left distance cutoff values.

	Youden Index	Closest top-left distance
Cutoff Value	0.8597	.8577
Precision	0.854	.853
Recall	0.633	.635
Accuracy	0.418	.419
F1 Score	0.727	.728
Specificity	0.790	.786
Sensitivity	0.633	.635

Table 5. Descriptive Statistics for cases.

		N	%
Clinical Case	1	186	22.9
	2	357	43.9
	3	270	33.2
	Total	813	100

Figure 2. ROC curves for each case.

Table 6. AUC and performance metrics per case.

	Case 1	Case 2	Case 3
AUC	0.83	0.88	0.86
Cutoff Value	91.50	86.50	86.50
Precision	0.91	0.87	0.86
Recall	0.51	0.57	0.61
Accuracy	0.63	0.45	0.68
F1 Score	0.66	0.69	0.72
Specificity	0.89	0.83	0.81
Sensitivity	0.51	0.57	0.61

Note: performance values computed for the optimal cutoff value.

Table 7. Frequency distribution of categories.

Category	N	%
1	17	2.1
2	495	60.9
3	114	14.0
4	143	17.6
5	35	4.3
6	9	1.1

Figure 3. ROC curves for each category.

Table 8. AUC and performance metrics per category.

	Category 2	Category 3	Category 4
AUC	.86	.78	.93
Cutoff Value	.87	.92	.87
Precision	0.86	0.88	0.86
Recall	0.62	0.59	0.61
Accuracy	0.68	0.67	0.68
F1 Score	0.72	0.70	0.72
Specificity	0.80	0.83	0.80
Sensitivity	0.62	0.59	0.62

Note: performance values computed for the optimal cutoff value.

Table A1. Example of a history taking scheme focused on the symptom ‘leg pain’.

(Fishman & Fishman, 2010; Matthys et al., 2009; Peart, 2022)
Initiating the Interview	Presentation	Ask for permission to talk with the patient after introducing yourself and identifying them (e.g., name and age). Ask the patient for their consent prior to star asking questions.
	Main complaint	To better understand the patient’s issue, pose open-ended questions regarding what brought them here, for example leg pain.
Gathering information	Explore the main complaint	Any kind of pain history can be described by the abbreviation SOCRATES. Site: Where is the pain? Onset: When did it start? Was it constant/intermittent, gradual/sudden? Character: What is the pain like (sharp, burning, tight)? Radiation: Does it radiate anywhere? Associations: Is there anything else associated with the pain (for example, trauma? Time course: How long did it last? Exacerbating: Does anything make it worse? Severity: How severe is the pain, using the 1–10 scale? Avoid asking multiple questions at once.
	Previous clinical history	Obtain details regarding a patient’s previous health issues or any previous hospitalizations. Screen for MITJTHREADS. Myocardial Infarction, Thromboembolism, Jaundice, Tuberculosis, Hypertension, Rheumatic fever, Epilepsy, Asthma, Diabetes, Stroke. Learn what medications the patient is taking, their dosage, and how frequently they take them. Find out whether the patient has any allergies and vaccination history.
	Social history	Inquire about drinking and smoking. Investigate the patient’s use of any illicit substances as well. Seek clarification about the patient’s eating and exercise routines, sexual history, employment, housing, marital status, sleeping patterns, overseas travelling and stress factors.
	Family history	Gather details regarding the patient’s family medical history, such as diabetes, heart history or oncologic conditions. Ask whether the family has any hereditary disorders.
Patients Perspective and planning	Interview closing	Review what the patient has told you repeating back the key elements, for the patient to correct you if there are any misunderstandings or mistakes. Address what the patient thinks is wrong with them and what they expect from the consultation. Use the ICE acronym: Ideas, Concerns and Expectations. After the patient has finished asking any questions they may have, and you are satisfied with all the information you need, thank them for their time.

Das S, Kumar E (2018). Determining accuracy of chatbot by applying algorithm design and defined process. 2018 4th International Conference on Computing Communication and Automation, ICCCA 2018, 1–6. 10.1109/CCAA.2018.8777715

 Google Scholar

Hidayatin L, Rahutomo F (2018). Query expansion evaluation for chatbot application. Proceedings of ICAITI 2018 - 1st International Conference on Applied Information Technology and Innovation: Toward A New Paradigm for the Design of Assistive Technology in Smart Home Care, 92–95. 10.1109/ICAITI.2018.8686762

 Google Scholar