A virtual surgery in general practice: Evaluation of a novel undergraduate virtual patient learning package

Abstract

Background: A suite of 10 online virtual patients developed using the IVIMEDS ‘Riverside’ authoring tool has been introduced into our undergraduate general practice clerkship. These cases provide a multimedia-rich experience to students. Their interactive nature promotes the development of clinical reasoning skills such as discriminating key clinical features, integrating information from a variety of sources and forming diagnoses and management plans.

Aims: To evaluate the usefulness and usability of a set of online virtual patients in an undergraduate general practice clerkship.

Method: Online questionnaire completed by students after their general practice placement incorporating the System Usability Scale questionnaire.

Results: There was a 57% response rate. Ninety-five per cent of students agreed that the online package was a useful learning tool and ranked virtual patients third out of six learning modalities. Questions and answers and the use of images and videos were all rated highly by students as useful learning methods. The package was perceived to have a high level of usability among respondents.

Conclusion: Feedback from students suggest that this implementation of virtual patients, set in primary care, is user friendly and rated as a valuable adjunct to their learning. The cost of production of such learning resources demands close attention to design.

Introduction

Computer-based simulations of real-life clinical scenarios are increasingly being used in medical education (Issenberg et al. 2005; Huang et al. 2007). Such simulations or virtual patients (VPs) provide learners with an interactive opportunity to engage in clinical problem solving and receive feedback on their performance (Cook & Triola 2009). Interactivity is essential to facilitate active learning (Chumley-Jones et al. 2002). VPs utilise a multimedia experience to provide a ‘more realistic’ context to learning. Cognitive theories of multimedia learning suggest that both verbal and pictorial representations have the potential to make learning more meaningful (Mayer 2010).

Table

Practice points

Increasingly VPs are being used in medical education. VPs can help to contextualise learning. Medical students find VPs, in a blended teaching approach, useful in their learning of general practice and primary care. VPs can be regarded as being highly usable by learners.

Despite the increasing popularity of VPs, it is recognised that such learning packages are resource intensive and expensive to produce (Huang et al. 2007; Cook & Triola 2009). Equally such educational strategies are only deliverable when there is an effective learner interface. Therefore, not only is there a need to consider the usefulness of such learning tools but also their usability (Sandars 2010). Usability in its broadest terms is a measure of a product's appropriateness to a defined purpose (Chisnell & Rubin 2008). Specifically, usability of a product is dependent on many factors including its content, level of interactivity and user satisfaction (Zaharias & Poylymenakou 2009). Hence, the imperative that the design and educational impact of all VP learning packages are considered carefully (Sandars 2010).

Providing students with the opportunity to develop the unique skills required to provide healthcare in the community is of importance in all undergraduate medical curricula (GMC 2009). However, to date, the availability of VP learning packages specifically set in a general practice context has been limited.

‘St Elsewhere Virtual Medical Practice’

The International Virtual Medical School (IVIMEDS – www.ivimeds.org) is a worldwide partnership of medical schools that aims to provide a means of developing and sharing a range of quality digital learning resources. ‘Riverside’ is a software tool offered by IVIMEDS that allows teachers to develop electronic, multimedia-rich VP cases. In 2009, the Department of General Practice at Queens's University Belfast produced a series of VPs entitled St Elsewhere Virtual Medical Practice (Figure 1).

Figure 1. Image of ‘St Elsewhere Virtual Medical Practice’ home page.

In this e-learning package, students are presented with 10 VPs. The clinical scenarios portrayed in each VP are based on actual cases that presented in primary care. Key primary care themes were incorporated into each case including: chronic disease management, dealing with uncertainty, spotting serious illness early, prescribing and the multidimensional biopsychosocial approach to patient health care.

Progressing through each case, the learner is guided through the stages of history taking, examination, investigation and decision making by means of various types of multimedia (e.g. text, clinical images, clinical videos and audio files). At various steps throughout the case, learners are presented with both multiple choice and free-text response questions. In terms of developing clinical reasoning skills, users are given the opportunity to discriminate key clinical features, synthesise clinical information from various sources, consider differential diagnoses, compare and contrast diagnostic probabilities and construct patient-centred management plans (Figure 2).

Figure 2. Example of a VP in the ‘St Elsewhere Virtual Medical Practice’ learning package.

At the end of each case, students were asked to reflect on what they learned and how this will change their future practice. They are also awarded an overall score on their performance.

Aim of this study

This study aims to evaluate the usefulness and usability of a novel undergraduate VP learning package set in a general practice context.

Method

Setting of study

The study was conducted in the School of Medicine, Dentistry and Biomedical Sciences at Queen's University Belfast. The undergraduate medical programme follows a five-year integrated spiralling curriculum model, with clinical training focusing in years three, four and five. Students spend 4 weeks in general practice in their fourth year of study.

Questionnaire

An online self-administered questionnaire was developed by means of a focus group of medical educationalists, clinical teachers and medical students following a review of the literature. The questionnaire aimed to capture students’:

1. Demographic details (age, sex, whether they are a graduate entrant and if they are from overseas).

2. Responses, on a five-point Likert scale (ranging from strongly agree to strongly disagree) to the following statement: Overall I felt that the use of 'St Elsewhere Virtual Medical Practice' was a useful learning tool.

3. Opinions on the usefulness of the individual components of the VP learning package.

4. Rank order of the following teaching mediums in terms of facilitating their learning: text books, small group work, lectures, e-video lecture, clinical placement and VPs.

5. Overall opinions on the usefulness of the VP learning package by means of free-text comments.

6. Perception on the usability of the VP learning package by completing the System Usability Scale (SUS) questionnaire (Brooke 1996). The SUS is a 10-item scale that provides a single global subjective assessment of usability. It covers areas such as system effectiveness, efficiency of use, user satisfaction and the need for training. The SUS has a high level of reliability (Bangor et al. 2008) and correlates well with other subjective measures of usability (Brooke 1996; Bangor et al. 2009). The SUS score allows relative judgements of usability between comparable applications (Brooke 1996).

The questionnaire was piloted on a sample of medical students and then sent by e-mail to each group of students upon completion of their module in general practice (total number = 260; six module groups ranging 40–44 students in each module group). A reminder was sent after 2 weeks. Simple descriptive statistics were used to analyse the data. Free-text comments were analysed by three of the investigators. Relevant themes were identified and approved by the investigators. Ethical approval was obtained from the School's Ethics Committee (Reference 10/02V1).

Results

One hundred and forty-nine responses were received giving a response rate of 57.3% (149/260).

Respondents’ characteristics

The age of respondents ranged 21–35 years of age, with a mean age of 22.4 years (SD = 2.92). Of the respondents, 42.3% percent (63/149) were male and 57.7% (86/149) female; 6.0% (9/149) of respondents reported to be overseas students and 10.1% (15/149) were graduate entry. Respondents’ characteristics were in keeping with the total year group (i.e. 43% male, mean age 22.2 years, 7% overseas and 11% graduate entry).

Usefulness of VP learning package

Students were asked to consider the following statement Overall I felt that St Elsewhere Virtual Medical Practice was a useful learning tool. The majority of respondents either strongly agreed (62.2%, 89/143) or agreed (32.9%, 47/143) with this statement. Only a small minority either had no opinion (2.1%, 3/143), disagreed (1.4%, 2/149) or strongly disagreed (1.4%, 2/149) with this statement.

Students were also asked to give their opinion on the usefulness of the various components of the VP learning package. Figure 3 illustrates their responses. Overall, students found the provision of answers to questions the most useful aspect of the learning package. They considered the reflective commentary to be the least useful.

Figure 3. Respondents opinions on the usefulness of the various aspects of the VP learning package (*Mean Likert scores were: strongly agree = 5–strongly disagree = 1).

Students were asked to rank the various teaching methods utilised in their general practice rotations, in terms of usefulness in their learning (Table 1). Students ranked Clinical attachments as the most useful teaching method. The VP learning package was considered to be the third most useful teaching method and e-video lectures were considered the least useful.

Table 1.  Students’ opinions on the rank order of the usefulness of various teaching methods in their general practice module

Description of teaching method
1st Clinical placement
2nd Lectures
3rd St Elsewhere
4th Textbooks
5th Small group work
6th E-video lectures

Usability of the VP learning package

Using the SUS, respondents considered the virtual learning resource to have a high level of usability (i.e. 88.8%).

Free-text comments

Of the respondents, 65.8% (98/149) gave free-text comments on the usefulness on the VP learning package. Comments were themed into the following categories: (1) ‘Practitioner experience’ (i.e. users gaining experience of being a clinical practitioner), (2) ‘Patient experience’ (i.e. users experiencing the case from the patient's perspective), (3) ‘Multisensory experience’ (i.e. comments relating to the various medium platforms offered in each case) and (4) ‘Learner centric’ (i.e. comments relating to how users felt that the package was student centred). Table 2 summarises their responses.

Table 2.  Respondents free-text comments of the usefulness of the VP learning package

Theme	N	Exemplar student comments
‘Practitioner’ experience	21	Able to bring things together and road testing what it is like to be a GP
		Good practise run in preparation for real practice
		Though not the real thing – gave me a feel of treating my 'own' patient
‘Patient’ experience	35	Opportunity to review real-life examples
		Chance to go through a case as a whole and consider situations that may arise in real life
		Made me think of the whole patient experience
‘Multisensory experience’	20	I enjoyed listening to scenarios
		“This is a good way of using e-learning. So often is just e-delivery – files on the web. Ability to interact and get feedback – just great!
		Visual files and images cemented my learning
		Images made it more in context; it was quite realistic
‘Learner centric’	66	Able to dip in and learn at my pace
		Easy to use and instant feedback made a worthwhile learning experience
		It was a useful tool for thinking through a problem, assessing my own knowledge and then reflecting on what I'd learnt
		Challenged me in a safe and constructive way

Discussion

The aim of this study was to evaluate medical students’ opinions and experiences of a novel VP learning package set in a general practice context. Overall, students found this learning activity to be of value in terms of its level of engagement, perceived usefulness and usability.

Computer-based learning has gained considerable popularity in undergraduate medical education and is now a core teaching medium in most medical schools (Cook et al. 2008). However, there is a need to move on from comparing these technologies with more traditional methods, and to investigate when and how we should be using such new technologies (Cook et al. 2008). VPs are computer-based simulations that aim to portray real-life clinical scenarios. Learners are given the opportunity to imitate the role of a clinician and progress through the process of assessing and managing ‘their own patient’. Findings from this study would indicate that students value the opportunity, albeit virtually, to experience being a ‘clinician’ and having to make diagnostic and therapeutic decisions. We are aware that students learn best when they feel they are being challenged, taking ownership of an actual case and given the opportunity to make clinical decisions (Dornan et al. 2009). Students have to feel supported and safe when in such learning encounters. VPs, of course, provide a safe environment whereby students can interrogate a patient and ultimately commit to a management plan. It appears that students value this ‘dry run’ and the deliberate practice gives them a sense of ownership of a patient case.

Without question, interacting with real-patients is the cornerstone of training the next generation of competent and compassionate doctors – an opinion shared by respondents in this study. However, students acknowledged that VPs can provide a useful adjunct to their bedside learning. For such blended learning to be effective, it is important that all teaching modalities are co-ordinated and designed to achieve common goals. VPs provide learners the opportunity to practice the technical aspects of a patient encounter. However, there are many aspects to clinical competency and it is considered that VPs are no substitute for developing the more humanistic aspects of patient care (Deladisma et al. 2007; Cook & Triola 2009).

This VP learning package was set in a primary care context. Findings from the study would indicate that VPs can provide a useful vehicle to convey principles of healthcare provided in the community setting. Key to this is the need to involve general practitioners in the design and content of the VPs.

There appears to be a wide variation in the design features of VPs (Cook & Triola 2009). However, all VPs make use of multimedia platforms to present and represent a clinical case. Evidence would suggest that such multisensory learning facilitates more meaningful learning (Mayer 2010). Individuals learn better from words and images rather than just using words alone. Common to all effective VPs is the ability to provide feedback on performance (Cook & Triola 2009). Such feedback allows the opportunity for students to close the gap between their actual performance and what would be considered a desired performance. Students in this study greatly valued the feedback provided by the VP learning package.

Findings from this study support that VPs are user-centred. Not only can learners use the package when and wherever they are but also at their own pace. This is of particular relevance given the wide geographical distribution of teaching general practices. Despite the proliferation of computer-based learning in healthcare education, usability testing is not often considered (Sandars 2010). Participants in this study considered the novel VP learning package to have a high level of usability. Bangor et al. (2008) developed an adjective scale for SUS scores (i.e. a seven-point scale: worst imaginable, awful, poor, OK, good, excellent and best imaginable) to allow for relative comparison of usability with similar applications. On this scale, our VP learning package would be considered to have an excellent level of usability. However given the multiple dimensions of product usability, this generalisation has to be considered with caution. Nonetheless, the VP learning package was considered to be user friendly by the students who participated in this study. This was also borne out in the free-text comments provided by students e.g. Easy to use and worked well and (The learning package was) Straight forward to use. There are many reasons that could explain this finding. First, right from the outset, the package was designed to be user-centred. Partnership between e-educationalists, clinical teachers and most importantly medical students helped to guide the instructional design of this package. Second, students are increasingly becoming more confident and accustomed using computer-based learning mediums (Cook & Triola 2009; Gormley et al. 2009).

The findings of this study have to be considered within its limitations. The questionnaire that we used has not been validated by prior research. While we achieved a satisfactory response rate, our sample only represents a cohort of medical students in one institution. While the SUS score provides a judgement on usability – this score is subjective and may not be truly generalisable across similar products. Furthermore, our results may not be generalisable due to variation in medical schools’ curricula and student demographics.

Conclusion

In conclusion, this study provides us with the opinions and experiences of a cohort of medical students on the use of a novel VP learning package set in primary care. Overwhelmingly, they felt that this learning activity was user-centred and a useful adjunct to their learning. It was user friendly and provided students with the opportunity and challenge of managing a clinical case in its entirety. As ever, VPs will never replace real-patient learning, but support and complement such learning in a blended teaching approach.

Acknowledgements

The authors thank all those students who participated in this study and the Centre for Medical Education at Queen's University Belfast for funding the development of the ‘St Elsewhere Virtual Medical Practice’. The authors also thank Amanda McKittrick and John Adams (Media Services, Queen's University Belfast) in the production of the Virtual Patients in ‘St Elsewhere Virtual Medical Practice’.

Declaration of interest: Dr Kieran McGlade is on the IVIMEDS steering council and helped in the design of the Riverside software. The authors report no declarations of interest.