The use of a virtual patient case in an OSCE-based exam – A pilot study

Figures & data

Table

Practice points
•Virtual Patient cases with automated scoring might be used as a complementary method for summative assessment.
•Students need to be trained in mastering the assessment system prior to exams.
•Scoring rubrics should be developed and validated before implementing computer-based assessment.
•Human assistance should be limited because of possible confounding influence on students’ outcomes.

Figure 1. ISP screenshot from the history taking. Note the free text input in natural language.

Figure 2. ISP screenshot from the physical examination.

Figure 3. Set-up for short and long stations.

Table 1.  Average session durations, number of correct diagnoses and the average of students’ own estimation of their confidence in suggesting a correct diagnosis for the ISP-OSCE pilot

				Mean no. of correct diagnoses		Mean no. of submitted diagnoses
Hospital	Hospital group	No. of examinees	Mean session time	Correct diagnosis	Correct on first trial	Only 1 subm. diagnosis	More than 1 subm. diagnosis	Confidence level
H1	1. 1	17	6 min 36 sec	94%	76%	82%	18%	88%
	Males	2	7 min 26 sec	100%	50%	50%	50%	75%
	Females	15	6 min 29 sec	93%	86%	87%	13%	90%
	1. 2	12	7 min 06 sec	100%	100%	100%	0%	95%
	Males	4	7 min 05 sec	100%	100%	100%	0%	93%
	Females	8	7 min 07 sec	100%	100%	100%	0%	96%
H2	2. 1	11	8 min 56 sec	91%	54%	64%	36%	n/a
	Males	6	9 min 39 sec	83%	50%	66%	34%	n/a
	Females	5	8 min 04 sec	100%	60%	60%	40%	n/a
	2. 2	15	8 min 22 sec	87%	67%	80%	20%	93%
	Males	n/a	n/a	n/a	n/a	n/a	n/a	n/a
	Females	n/a	n/a	n/a	n/a	n/a	n/a	n/a
H3	3.1	15	8 min 12 sec	40%	40%	100%	0%	76%
	Males	3	8 min 07 sec	33%	33%	100%	0%	85%
	Females	12	8 min 13 sec	42%	42%	100%	0%	73%
	3.2	12	7 min 43 sec	42%	42%	100%	0%	90%
	Males	7	7 min 48 sec	43%	43%	100%	0%	93%
	Females	5	7 min 37 sec	40%	40%	100%	0%	86%
H4	4. 1	12	7 min 21 sec	75%	67%	92%	8%	90%
	Males	8	7 min 08 sec	75%	75%	100%	0%	87%
	Females	4	7 min 49 sec	75%	50%	75%	25%	96%
	4.2	16	7 min 47 sec	75%	56%	81%	19%	85%
	Males	6	7 min 44 sec	83%	83%	100%	0%	n/a
	Females	10	7 min 49 sec	70%	40%	70%	30%	85%
Totals/averages		110	7 min 45 sec	74%	63%	87%	13%	88%
Males		n/a	7 min 52 sec	72%	62%	88%	12%	86%
Females		n/a	7 min 28 sec	76%	60%	84%	16%	84%

Figure 4. Mean distribution of session duration per assistant (hospitals H1, H2, H3, H4 are sub-divided into hospital groups: 1.1, 1.2, 2.1, etc.).

Table 2.  Mean values (amounts and percentages) for history questions asked and lab tests ordered

Hospital group	n	Mean no. of history questions asked	Mean % of history questions required	Mean no. of words per question	Mean no. of lab tests ordered	Mean % of lab tests required
1.1	17	6.9	78%	1.6	3.2	73%
1.2	12	7.8	77%	3.4	5.8	56%
2.1	11	10.2	61%	4.6	5.4	48%
2.2	15	9.1	62%	2.8	6.9	44%
3.1	15	10.0	60%	4.7	6.7	53%
3.2	12	9.9	56%	3.8	5.9	32%
4.1	12	7.7	50%	3.6	5.9	38%
4.2	16	8.7	62%	3.5	5.1	50%
Males	36	8.4	57%	3.7	5.8	46%
Females	59	8.5	69%	3.3	5.1	55%
Totals/averages		8.8	63%	3.5	5.6	51%
P-value		0.606	0.006	0.166	0.297	0.203

Table 3.  Median and average values for the ISP-related questionnaire

Question	68 responses: 45 females, 21 males (2 did not report gender)
1. Was the ISP designed in such a way  that you could apply your knowledge?	Median: 4 (scale ranging from 1 “Highly disagree” to 6 “Highly agree”)
	52 responses
2a. What do you think is the best thing	Ranking of the most frequently cited factors (no. of respondents, (%)):
about the ISP examination?	1. Realism/authenticity/trustworthiness of the case:			10 (15%)
	2. Fun/enjoyable/engaging:			9 (13%)
	3. Easy access to lab tests and physical exam:			8 (12%)
	4. New learning mode/instructive/educational:			7 (10%)
	64 responses
2b. What do you think is the worst thing	Ranking of the most frequently cited factors (no. of respondents, (%)):
about the ISP examination?	1. Limitation of interactive dialogue with patient:			38 (56%)
	2. Lack of time:			14 (21%)
	3. Doesn’t feel authentic:			10 (15%)
	4. Not familiar with the system/this way of solving problem:			5 (7%)
3. Should stations of ISP type be used in  the practical assessment test (OSCE)?	Yes 24 (35%)	No 21 (31%)	Not sure 23 (34%)
4. What made the ISP case engaging?	Pos 43 (63%)	Neg 11 (16%)	n/a 14 (20%)	(open free text question)
5. What contributed to some sense of  realism in the ISP case?	Pos 53 (78%)	Neg 9 (13%)	n/a 6 (9%)	(open free text question)
Examples of positive answers to Q4:
• Good structure. With history taking, status etc.
• Good to see the patient and hear her voice.
• To be able to think and reason with the help of the computer. Good complement to other teaching.
• The many possibilities to examine a patient. Great variation. Interesting to be able to try many tools as in real life and see what they would give you.
• The feeling of getting it right – the detective work – and having access to all options in an immediate way.
• The motivation to ‘solve’ the case.
• To be able to have examination done immediately and also to get results right away!
• Exciting, fun to get results from lab directly, good exercise. That the patient was able to answer my questions.
• Direct feedback, often missing in real life.
• To be able to freely choose examinations/tests.
• That the patient describes her symptoms in her own way – not just in textbook fashion.
Examples of negative answers to Q4:
• I became mostly irritated on the case.
• Nice picture, and the way examinations were done, but I don’t think it was efficient. Good pictures and examinations though.
• It was fun to try to reach the right diagnosis until I realized that she never would answer fundamental, important questions and that she began saying stuff that I already had asked without properly answering the question. That only made me irritated and I went on to do a physical examination instead.
• That I wanted to find out the problem of the patient. But it didn’t go smoothly all the time.
• Too stressful for an examination situation to become engaged. Under other circumstances can I imagine that one would become engaged because it was rather realistic.
Examples of positive answers to Q5:
• That the lady talks to me.
• To see the patient and that she talked to you.
• The voice and the pictures.
• The ‘living’ patient.
• It felt interactive, it affected the outcome.
• Real patient on the screen, with a voice. The variation of diagnostic possibilities.
• That the patient talked and moved. Not just replied with text.
Examples of negative answers to Q5:
• I think the interaction with patient didn’t work, otherwise interesting.
• A real person speaking, unfortunately she always said the same things.
• It didn’t feel realistic, unfortunately.
• The difficult history taking.

Table 4.  Examinees’ answers to the general on-line delivered questionnaire (n=110)

Question	Median hosp. 1	Median hosp. 2	Median hosp. 3	Median hosp. 4
Do you think that ISP gave you a possibility to apply your knowledge?	6	6	3	5
Do you think that ISP-like program should be used in examination?	5	5	3	4
Rate your opinion of the surgery course in general.	9	8	7	7
Was the OSCE relevant?	8	8	8	8
The ISP-program stimulated problem solving.	6	6	3	5

The scale of the answers ranged from 1 to 9, where 9 is best.

Table 5.  Video-observational variables interaction on the outcome results (% and odds ratio)

				95% C.I.
Observational variable	n	Comparison between baseline groups (n,%)^a	OR	Lower	Upper	P-value
Interaction assistant/student	47	Intensive (24, 83%) vs. Weak (5, 20%)	17.21	1.30	>500	0.026
		Intensive (24, 83%) vs. Medium (18, 67%)	2.44	0.47	14.38	0.374
		Medium (18, 67%) vs. Weak (5, 20%)	7.27	0.56	426	0.177
Assistant cluster^b	111	3 (29, 97%) vs. 1 (55, 61%)	17.37	2.48	763	<0.001
		3 (29, 97%) vs. 2 (27, 81%)	6.18	0.63	311	0.162
		2 (27, 81%) vs. 1 (55, 61%)	2.77	0.84	10.82	0.105
Flow with ISP	47	Playful (6, 100%) vs. Frustrating (7, 29%)	12.19	1.21	>500	0.033
		Playful (6, 100%) vs. Normal (34, 74%)	6.57	0.89	80.65	0.070
		Normal (34, 74%) vs. Frustrating (7, 29%)	2.72	0.34	>500	0.384
Expression of uncertainty	47	Certain (22, 86%) vs. Doubtful (7, 14%)	30.98	2.60	1834	0.002
		Certain (22, 86%) vs. Neutral (18, 72%)	13.74	1.22	771	0.028
		Neutral (18, 72%) vs. Doubtful (7, 14%)	2.38	0.055	2.6	0.474
External signs of stress	47	High/Medium (23, 65%) vs. Low (24, 75%)	1.58	0.38	6.93	0.679
Mouse handling	48	Assistant (7, 86%) vs. Student (41, 68%)	2.74	0.28	138	0.658
Keyboard handling	48	Assistant (12, 83%) vs. Student (36, 67%)	2.46	0.42	27.6	0.470

^a (n,%): No. of observed students, percentage with correct diagnosis. ^bAssistant cluster consists of 3 groups based on the following profiles: 1. Undergraduate students of Medical Informatics (4 persons) 2. Experienced researcher in Medical Simulation (2 persons) 3. Senior physician (2 persons).