As time goes by: Stakeholder opinions on the implementation and use of a virtual patient simulation system

Abstract

Background: Stakeholder opinions on the implementation of a particular virtual patient application might prove important for decision-making and implementation efforts in general. This study aimed to capitalize on experiences originated from two post-implementation settings of Web-based Simulation of Patients (Web-SP).

Method: We conducted a cross-sectional survey of stakeholder opinions (39 students, teachers, course directors, and university leaders) on the implementation and use of Web-SP in Colombia and Sweden, using a mixed method approach.

Results: The respondents did not show equal preference in rating the ranking of the order of importance included in the variables (Friedman's Chi square: 26.5 to 115.1, df = 6, p < 0.001, Kendall's coefficient of concordance ranging from 0.11 to 0.50). The answers provided as free comments supported the statistical findings on the importance of end-user customization, need for authenticity in software design, and use of virtual patient simulations in a curricular context, for clinical reasoning development.

Conclusions: Virtual Patient design should allow extensive editing, support case authenticity and enhance clinical reasoning abilities, in an effort for ensuring accountability and sustainable development of the field.

Introduction

Many factors play a role in the successful educational use of a Virtual Patient Simulation application. To start with, producing a Virtual Patient Simulation system (VPS) is a resource-consuming endeavor (Fall et al. 2005; Huang et al. 2007; Cook & Triola 2009; Round et al. 2009), mostly because VPS development requires careful examination of the educational goals of a given institution, combined with broad consultation among clinical experts, education specialists, IT developers, and faculty administration, and also because once implemented, the VPSs need further development, maintenance and a constant flow of new virtual cases. The lack of further development by the original institution might hinder the use of an otherwise successfully implemented application. Instead of novel VPS creation, the use of existing systems might prove advantageous in higher education institutions not willing to develop their own VPS (Haag et al. 2007); however, teachers and students of the host institution may be minimally involved in the decision of adopting a specific application and as a result reluctantly agree to test what they consider to be the latest educational “gadget”. Even if previous consultation and agreement regarding the adoption of an established VPS does exist among stakeholders, the adaptation to the local socio-cultural and medical context may prove to be a crucial step towards a successful implementation and use of the application adopted in another country.

It is thus not a surprise that VPSs are “barely now entering the mainstream of medical education” (Ellaway et al. 2009). Among the reasons cited there are early design flaws, use of improper educational models, high up-front costs, and lack of proof of educational efficacy. In our experience at Karolinska Institutet, errors in implementation are frequent as well, and may impact heavily on the perceived performance and utility of the applications. Moreover, topics such as VPS authenticity, sustainable development, and accountability have not been the target of research so far, even if they have the potential of elucidating essential implementation issues.

We believe that different stakeholder opinions on a particular VPS application are important factors for implementation efforts in general. This study aimed to explore and capitalize on such knowledge, originated from two post-implementation settings of the VP system Web-based Simulation of Patients (Web-SP).

Table

Practice points

VPS should either allow extensive editing by end users or be a subject of post-implementation development by the institution of origin. Authenticity extends well beyond the design of the interface, to encompass the realism of the case scenarios and nature/quality of feedback. Accountability of creators/users is a must for sustainable VPS development. Maintenance costs, staff development, and post-implementation activities in general should be the focus of further research.

Method

We conducted a cross-sectional survey of stakeholder opinions on the implementation and use of the Web-SP system in three universities in Colombia and Sweden (Table 1). The Faculty of Medicine at Universidad El Bosque, Bogota, participated in the survey in January 2008. The corresponding Swedish sites were Karolinska Institutet (nursing, dentistry, and medicine programs) and Halmstad University (nursing program), where the study took place in March 2009.

Table 1.  Comparison of Web-SP characteristics and implementation modalities in Colombia and Sweden

	Educational setting			Web-SP
	Curricular integration	Program type	Sites	Implementation type	Case scenarios	Language
Colombia	Y	1	1	Adoption	Real life	Spanish
Sweden	N	3	2	Creation and adoption	Not real life	Swedish or English

The virtual patient application

Web-SP is an explorative linear virtual patient application developed at the Department of Learning, Informatics, Management and Ethics (LIME), Karolinska Institutet (Zary et al. 2006) and adopted in several countries worldwide. In Web-SP, the users are required to solve a clinical case by gathering information from patient interviews, physical exams and ancillary tests, in order to arrive to diagnosis and treatment sections, after which they gain access to the feedback module, with a detailed case discussion. The Spanish version of Web-SP was localized (Fors et al. 2009) and implemented in the curriculum of the Internal Medicine course at the Faculty of Medicine at Universidad el Bosque in 2005. The cases in the Spanish version were created from real life clinical records, collected by 216 Internal Medicine students from university hospitals in Bogota. Web-SP use in the two Swedish universities was optional at the date of the survey (not formally embedded in the curriculum) and encompassed medicine, nursing, and dentistry.

The participants

In Colombia, all five faculty board members involved in decision-making around the application (referred to as “leaders” in the rest of the article) and the seven university teachers who used Web-SP in teaching and assessment at El Bosque Medical School (“professors”) were invited to answer the questionnaire. From a total population of 216 undergraduate medical students who had used Web-SP in their Internal Medicine course and had been part of a larger study on assessment with VPS, 16 were randomly chosen from the one cohort available at the time of the study.

In Sweden, 3 leaders and 8 professors from Karolinska Institutet and from Halmstad University agreed to participate in the survey (out of a total of 30 users). No students were available in Sweden for the purposes of the study, as Web-SP use had been optional and “retracing” the students posed a logistical problem.

The instrument

A semi-structured questionnaire was developed for the purposes of this study. Two experts in medical education, co-authors of this study, reviewed the questionnaire for content validity. The research questions, divided in four themes of seven items each, were explored both quantitatively and qualitatively, using a mixed method approach (Creswell 2003). The participants were asked to rank several aspects related to the implementation and use of Web-SP according to their perceived order of importance (on a 1 to 7 scale, from least to most important), while a number of open-ended questions were meant to further clarify issues raised as closed questions in each theme. The questionnaire explored items related to (1) the implementation process; (2) post-implementation activities; (3) the curricular use of the application in teaching and assessment, including desired features in support of those functions; and (4) the possibility to extend the use to educational settings beyond undergraduate courses. The aspects considered for the implementation process were the following: ease of customization by end user; design of the software; curricular plan of the target course; cost of the initial implementation; cost of the maintenance of the application; the intended curricular use and its results; and staff development around the application. As post-implementation activities to reach the proposed educational goals, we listed: keeping the system running; continous creation of new cases; peer-review of newly created cases; case use in a curricular context; exchange of cases with other HEI; research; and staff development. The intended curricular use of the application explored the teaching–learning component (ranking factual knowledge, core knowledge, clinical reasoning skills, unusual diseases/syndromes, common diseases in unusual presentations/complicated, common diseases in usual presentations, and topics not covered by the study plan) and the assessment component (as open-ended questions). Finally, the use of a virtual patient software in postgraduate settings considered the need of a different flow through the application (branched cases), a higher difficulty level, the modification of the learning objective, a higher level of detail, the type and level of feed-back, and the availability window of the system.

The survey was conducted in Spanish at Universidad el Bosque, where the instrument was subject to prior on-site peer validation before application; in Sweden, the chosen language was English and no validation was deemed necessary. No further actions were taken by the research team in order to address possible variations between the two languages.

Statistical analysis

Descriptive analysis was used to determine the median score and the inter-quartile range. Friedman statistic was performed to test the hypothesis that there was no systematic difference in ranking the order of importance among the respondents. Kendall's W test was also applied to measure the level of agreement in the ranking of ordering of the relative importance of the items. The Jonckheere–Terpstra test was employed to test the null hypothesis that the medians were the same and Friedmans's posthoc test was used for multiple non-parametric pairwise comparison. The significance level was specified at 0.05. The statistical analyses were performed using SPSS and Statistica 8.0 software; the graphical presentations were created by Statistica 8.0.

Analysis of the open-ended questions

Since the open-ended questions were naturally pertaining to a theme and a sub-theme, and the answers had been brief, no formal content analysis was undertaken. All the opinions expressed were registered.

Results

The respondents did not show equal preference in rating the ranking of the order of importance of the variables (Friedman's Chi square: 26.5 to 115.1, df = 6, p < 0.001). Kendall's coefficient of concordance ranged from 0.11 to 0.50, reflecting the ranking of ordering and level of agreement on the relative importance of the items is different among the subjects (Table 2). The range indicates lesser degree of unanimity among the various responses. Kendall's W can be interpreted as a coefficient of agreement or disagreement among raters. The coefficient W ranges from 0 to 1, with 1 indicating complete inter-rater agreement, and 0 indicating complete disagreement among respondents.

Table 2.  Friedman's test and Kendall's W test by variable group (n = 39)

Variable	Friedman's Chi-squared	df	p-value	Kendall's W^a
Implementation process	78.3	6	<0.001	0.34
Post-implementation activities	26.5	6	<0.001	0.11
Intended curricular use	107.1	6	<0.001	0.46
Use in different educational settings	115.1	6	<0.001	0. 50

Note: ^aKendall's coefficient of concordance.

The Jonckheere–Terpstra test identified a significant between-group difference in some of the items for three major variables, namely implementation process, faculty post-implementation activities, and curricular use, but failed to show statistically significant difference for “use in different educational settings”. However, significant within-group differences were observed in all variables. The median score of the ratings with the inter-quartile range is presented in the graphs (Figures 1–4). We did not observe significant between-country differences in the rating of the four major variables (p > 0.05).

Figure 1. Median rank score of responses for the variable “implementation process” (where 1 is least important; 7, most important).

Figure 2. Median rank score of responses for the variable “post-implementation activities” (where 1 is least important; 7, most important).

Figure 3. Median rank score of responses for the variable “intended curricular use” (where 1 is least important; 7, most important).

Figure 4. Median rank score of responses for the variable “use in a different educational setting” (where 1 is least important; 7, most important).

The implementation process

A statistically significant between-group difference was not observed for the variable “implementation process” in all the items, except for “cost of the initial implementation” (p < 0.001). The within-group difference showed a statistically significant difference (p < 0.001), indicating that respondents have significantly different ratings for different sub-categories of the implementation process (Figure 1). The different participant categories agreed most on the importance of easy VPS customization by the end user; a good design of the software, together with curricular integration, ranked also high in their preferences, regardless of stakeholder group.

In the open-ended questions, the participants were asked about Web-SP features that needed improvement as soon as possible. The staff directly involved in case creation commented on the need for “deep” customization (beyond the actual capabilities of Web-SP), and some pointed out the importance of further development by the institution of origin. A number of students showed preference for branched cases, instead of the linear design of Web-SP, and indicated that such a flow would support better the clinical reasoning development. Teachers and students objected the unrealistic design of the physical examination section, which subtracts from the authenticity feeling and makes case creation cumbersome and time-consuming. An example is shown below: the physical examination “doll” allows for “knee auscultation from the front”: Both professors and students expressed interest in grouping the lab section in test “batteries”, instead of alphabetical listings, and discussed the real-life improbability or necessity of ordering an unlimited number of tests on a single patient. The use of VPS as add-on to an existing curriculum was strongly discouraged by participants from both countries.

Post-implementation activities to reach the proposed educational goals

We did not observe statistically significant between-group differences in five of the seven sub-categories. However, significant between-group difference was observed in “exchange of items with other HEI” (p = 0.006) and “staff development” (p = 0.02), items which registered most disagreement among different stakeholder groups. The within-group difference was also significant. The highest level of agreement was observed around the necessity of the continuous creation of new cases. Figure 2 displays the between- and within-group differences.

In this part of the questionnaire, the participants could also freely discuss items they considered essential to post-implementation. One of the most debated was the creation of new cases. The students voiced for numerous cases within a sub-speciality, with a minimum of 2–3 cases per disease. The professors also highlighted the importance of a “critical mass” of cases, but underlined that creating numerous cases is time-consuming. Most teachers and students believed that real life cases are intrinsically better than fictitious cases.

Intended curricular use

The between-group difference for the teaching–learning component was not statistically significant for all the items, except for “factual knowledge” (p = 0.006) and “topics not covered by the study plan” (p = 0.04). The within-group difference in rating the different items (Figure 3) is statistically significant (p < 0.001). The development of clinical reasoning constitutes the main curricular use of VPS for all respondent categories, followed closely by VPS depicting unusual diseases and by common diseases in usual presentations, respectively.

The stakeholders indicated the development of clinical reasoning as the most important use of VPS for teaching purposes, while core knowledge and especially factual knowledge ranked low in the professors’ and leaders’ perspective. Common diseases in usual presentations, followed by common diseases already complicated at diagnosis or in unusual presentations were considered to make a good curricular use for VPS.

The assessment component was addressed as open-ended questions only, referring generally to VPS. The participants strongly believed that VPS systems should not be the only assessment tool used in a course. They also thought VPS should not be used for high-stakes assessment either. All stakeholder groups encouraged VPS use in ordinary course assessment, in combination with other evaluation methods.

Use in different educational settings

The between-group difference was not statistically significant (p > 0.05) for all items. The within-group difference in rating the different items was statistically significant (p < 0.001). Significant levels of agreement were registered among the responder groups regarding the need for adapting the difficulty level to the educational setting where the VPSs are used (undergraduate courses, postgraduate studies, or continuing medical education), followed by changes in design (i.e., a different flow through the application).

The various stakeholders were unanimous in indicating that learning objectives must be specific to the educational setting VPS are used for, closely followed by a different difficulty level and a different type of feedback. Most free comments pointed out that a case created for medical students is unlikely to be suitable for medical residents or for continuing medical education in nursing, for example.

Discussion

Much to our surprise, the statistical analysis showed no systematic differences between the two countries. Despite differences in educational setting, language and case origin, the participants faced the same problems and arrived at similar conclusions.

Customization versus further development

The ease of customization by end users seems to be critical for successful VPS implementation. All stakeholders directly involved in case creation also expressed in the free comments the need to extensively edit the Web-SP “shell”, in order to adapt it to specific learning objectives or to a certain sub-specialty. This implied removing features considered unattractive or unnecessary (e.g., parts of the physical exam section) and adding new features or grouping the existing ones in different ways (e.g., labs and imaging), to support the learning objectives. Completing a case in accordance with the provided matrix was regarded by the staff with heavy clinical burdens as time-consuming and ineffective. The stakeholders considered that VPS should either allow extensive editing by end users or be subject of post-implementation development by the institution of origin; the latter is in keeping with opinions expressed in the literature (Haag et al. 2007).

Design and authenticity

The importance of authentic virtual patient contexts in terms of interface and tasks was flagged in a recent article on the CAMPUS system (Huwendiek et al. 2009). The interface of the physical examination section should “help students proceed in a structured manner”; the system should also allow students to make similar decisions to those “a real doctor would make”. The participants in our study also emphasized the connection between software design, case content, and authenticity. A physical exam section that puzzles both case creators and students should be a target for improvement. All aspects of design should support clinical reasoning; in our case, the stakeholders focused on the lab and imaging section and suggested those be clustered in groups of tests. However, since such “batteries” differ from country to country, the adjustments should be subject to localization efforts.

More on authenticity

There's more to VPS authenticity than meets the eye. A need for localization of “adopted” VPSs to the medical practice and the socio-cultural context of the host country has been advocated in the literature (Fors et al. 2009). The case scenarios and the nature and quality of feedback add or subtract to the authenticity of an application. The students in our study, all Colombians, appraised case creation based on real-life patient records. The students felt that the knowledge derived from the proposed cases was immediately applicable to the clinical setting, which further contributed to enhanced motivation in using the system. Another feature highlighted by students was the quality of the feedback, as the Spanish version of Web-SP offered the real treatment course and the actual clinical evolution of each patient. For an adopted application, native language may add to the perceived realism. The Spanish-speaking students in our study indicated they would not like to use an application in English; some even objected to patient cases reflecting medical practices different from the context in which they are supposed to work as physicians.

Post-implementation

Not surprisingly, our three stakeholder groups did not agree when asked about post-implementation activities that best support the use of VPS. They were united though in considering that the continuous creation of new cases is essential to the post-implementation setting and at least the teachers believed that validation is equally important to new case creation. We observed some degree of consensus on the necessity of having a functional system (mainly as internet connection, logins, IT support); nothing was recorded though on the need to update existing cases. The leaders awarded low scores to the item “case use in a curricular context” (the highest ranking item for teachers). Leaders also gave very low scores to both case validation and case exchange with other higher education institutions (the latter finding puts into perspective the current development of VPS interoperability standards). The leaders’ lack of interest in case validation may reflect a de facto conviction, that it is the “shell” that validates the content. Without a proper case-script, aligned with educational objectives and with assessment, case creation limited to filling data into empty fields, performed by staff other than clinicians, poses a serious threat to content validity.

Staff development

Who will develop and maintain virtual patient cases is indeed no trivial matter (Cook 2009) and has bearing on VPS overall costs. Even though, a few years ago, effective training in simulation was postulated to be “the product of training resources, trained educators and curricular institutionalization” (Issenberg 2006), staff development in simulation has hardly been discussed (Steinert et al. 2006; Haag et al. 2007; McLaughlin et al. 2008). The need for resident training in simulation-based education was recently identified (McLaughlin et al. 2008). Staff development to support VPS use was surprisingly enough regarded by leaders as important to the post-implementation setting, but it was not mentioned by any of the participants as a free comment.

Curricular integration

Curricular institutionalization of simulation is the third component of the Issenberg product (training resources x trained staff x curricular integration = effective simulation training). Should those VPS applications not integrated in the curriculum be considered a priori as non-effective? In reality, the systems implemented as add-ons to the existing curriculum have few users. In their article on the experience with the CAMPUS system, the authors acknowledge that “the usage…has not fulfilled the expectations” and that “as soon as students get credit points for completing case sessions most of them will do it” (Haag et al. 2007). In opposition, the CLIPP project, where the CASUS system was used to teach the North American core pediatrics clerkship curriculum, reported more than 8000 users in 98,000 sessions (Fall et al. 2005). Our study envisaged both sides of the coin, as the same VPS was part of the curriculum in Colombia and add-on in Sweden (with compulsory and voluntary use, respectively). However, regardless of setting, curricular integration was perceived by all participants as crucial to reaching the educational goals set for a course.

Use in teaching and assessment

There is a broad literature consensus in regarding “clinical reasoning” as the best use of VPS, and our findings support it. The students, however, differed from leaders and teachers in assuming that VPS should offer a reasonable load of factual knowledge with each patient. More as a surprise came the participants’ preference for usual presentations of common diseases, while another study had previously reported student inclination for problems they were “unlikely to encounter during clinical training” (Huwendiek et al. 2009). The recent North American ED2 accreditation requirements also perceive VPS as appropriate for teaching topics not seen during clinical rotations, due e.g., to seasonality (www.lcme.org). As for the assessment component, the envisaged VPS use was course examination, in combination with other assessment methods. An immediate use of VPSs for high-stakes examinations was not obvious to the responders, a finding we link to two Web-SP features: (1) the absence of an automated score feature, to ensure reliable and reproducible assessment results; and (2) student identity blinding currently impossible in Web-SP (Figure 5).

Figure 5. Screenshots from the English version of Web-SP, illustrating the physical exam and lab sections.

Use in different educational settings

Most faculty and university board members expressed the desire of an additional use of the application in the postgraduate or continuing medical education setting. The stakeholders agreed that the learning objectives must differ according to the context (different programs at undergraduate level or undergraduate vs. post-graduate setting); the participants were consistent in finding that the level of difficulty should vary, as well as the type and depth of feedback provided; even the flow through the case should be adapted to the desired competence level (the more sophisticated the formation – CME, resident training –, the less appealing the linear design).

Costs

VPS development and maintenance costs are crucial to the successful implementation and use of an application. Production costs per case varied largely in a review on VPS use in North America (Huang et al. 2007); more than half ranged from $10,000 to $50,000. Another review article found that 85% of cases cost more than $10,000 per case (Cook 2009). In the CLIPP project, the estimated overall development cost per case was $18,000 (Fall et al. 2005); this is the only paper to mention maintenance costs, in the region of $120,000 annually for the entire suite of virtual patients. Even if we not agree that the cost for developing a single case is as high as these figures (Zary et al. 2006), costs can be a problem. In our study, the costs generated by the implementation process and by the maintenance of the application ranked low on everybody's agenda; however, the leaders seemed more cost-conscious than other stakeholders.

We consider our findings generalizable, but the study is not without limitations. Unfortunately, we did not have a student group in Sweden; even if we do not consider this to be a true limitation, since we aimed to register the range of opinions of the different stakeholders, a different student group would have been desirable. In the end, the lack of a Swedish student group only highlights the importance of curricular integration of such applications.

Conclusions

Design and authenticity

The aspect of paying attention to VPS design, which should enhance clinical reasoning abilities and support case authenticity, cannot be overemphasized. Authenticity, however, extends well beyond the design of the interface. The users are more positive to the use of an application when the case content is robust, derived from everyday practice and supported by feedback providing an exposé of actual patient treatment and evolution.

Sustainable development

End-user customization is a real practical issue, which should be addressed already in the design phase. Alternatively, the institution of origin should carry on post-implementation development. Applications should be allowed to evolve according to use and discontinued when obsolete. A VPS running as a pilot or as an add-on is not educationally effective and adds to the current confusion on virtual patient efficacy as a learning tool. How to best implement VPS, what makes a case “good”, and how many cases are needed per topic are a few questions further research should answer.

Accountability

Accountability is a must for sustainable development. VPS creators and users should inform the academic community on the less glamorous topics, such as arrested development, number of users and types of educational settings, useful lifetime of an application, direct and indirect costs of development and maintenance, localization efforts, staff development, and implementation mistakes. Otherwise, the VPS community will continue to be torn between the “band-aid/asset in the learning portfolio” disjunctive (Tworek et al. 2010).

Acknowledgment

The first author was supported by the Programme Alban, the European Union Programme of High Level Scholarships for Latin America, scholarship no. E06D100194CO.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.