ABSTRACT
Results of a CME-certified activity completed by a total of 986 cardiologists and 783 haematologists-oncologists (haem-oncs) from around the world were examined to determine whether virtual patient simulation could improve decision-making and performance within the simulation related to patient evaluation, tailoring anticoagulant therapy, and patient management to improve adherence using patient-centred care strategies. Results showed a significant overall impact of education from pre- to post-clinical guidance (CG) on correct decisions made in both cases for cardiologists, with a relative improvement of 22% for Case 1 (45% pre- to 55% post-CG, n = 475, t(474) = 14.12, P<.001, Cohen’s d =.46) and 19% for Case 2 (62% pre- to 74% post-CG, n = 245, t(244) = 11.95, P<.001, Cohen’s d =.59). Impact also was seen for haem-oncs, with a relative improvement of 27% for Case 1 (45% pre- to 57% post-CG, n = 280, t(279) = 11.91, P <.001, Cohen’s d =.60) and 19% for Case 2 (63% pre- to 75% post-CG, n = 147, t(146) = 9.52, P <.001, Cohen’s d =.58). Virtual patient simulation improved cardiologists’ and haem-oncs management of patients with pulmonary embolism in a simulated environment.
Introduction
Venous thromboembolism (VTE), which consists principally of deep vein thrombosis (DVT) and pulmonary embolism (PE), is a common cause of morbidity and mortality. Healthcare providers encounter DVT or PE in a variety of settings and clinical scenarios (e.g., acute vs chronic, provoked vs unprovoked, and cancer related), all of which have different treatment algorithms [Citation1]. While there is strong evidence that the use of non-vitamin K antagonist oral anticoagulants (NOACs) have benefits for treatment of VTE, continuing medical education (CME) is needed to address persistent clinical practice gaps that result in suboptimal use of these agents.
Among these gaps, physicians who care for patients with VTE lack confidence in selecting appropriate anticoagulation therapy based on a patient’s risk for PE and incorporating NOACs into treatment strategies to optimise patient outcomes [Citation2–5]. Second, physicians have misperceptions regarding the need to monitor adherence in patients who are using NOACs [Citation3,Citation6]. And third, physicians who manage anticoagulant therapy have limited knowledge of the latest real-world data on the use of NOACs for prevention and treatment of VTE and emerging clinical applications for NOACs, including cancer-associated thrombosis [Citation7,Citation8].
CME activities are needed to educate cardiologists and haematologists-oncologists (haem-oncs) and to address clinical practice gaps to improve patient outcomes. A Medscape virtual patient simulation (VPS) platform, known as MedSims, was chosen as the educational modality because it was expected that it would address the identified clinical practice gaps. Cardiologists and haem-oncs who treat patients with VTE are challenged to stratify patient risk for PE to support confident selection of appropriate anticoagulation therapy to optimise outcomes (gap 1); monitor patients to gauge adherence to therapy (gap 2); and apply the latest data on the use of NOACs in patients with cancer (gap 3). We sought to determine whether VPS could improve decision-making and performance within the simulation, particularly as they relate to patient evaluation (risk stratification), tailoring anticoagulant therapy (applied use of data based on patient characteristics), and patient adherence (monitoring). Because simulation is associated with real-world behaviour – by one definition, it is “a bridge between classroom learning and real-life clinical experience” [Citation9] – it was expected that if we saw improvements in the simulation, they would translate to real-world practice.
At its essence, simulation in education follows a theoretical framework of having the freedom to explore, make mistakes, and learn from them. Simulation in healthcare is defined as an educational modality that replaces or amplifies real patient experiences with scenarios that replicate real health encounters [Citation10].
VPS simulates a patient-healthcare provider visit and gives learners the perspective of the provider who is listening to a patient and then making decisions about patient assessment and care via a simulated electronic health record (EHR). Direct process and outcome feedback on learners’ decisions and actions concerning each individual patient case, combined with opportunities to review decisions and the consequences of those decisions, allow learners to make real-time clinical course corrections based on formative feedback and knowledge transfer (e.g., a review of literature in the case of the current study’s intervention). This process of review and reflection in response to feedback supports behaviour change by building a repertoire of patient encounters and clinical problem solving, which learners can transfer from virtual to real clinical settings [Citation11,Citation12].
As part of a trend towards the use of simulation, modalities such as online clinical case simulations and VPS have emerged as appealing educational resources. In 2007, only 24% of medical schools in the USA and Canada were using virtual patients in their curricula [Citation13]. By 2016, an Aquifer (then MedU) collection of virtual patients was in use at approximately 75% (130) of those medical schools [Citation14]. Furthermore, VPS is increasingly endorsed as an evidence-based, interactive learning format in continuing education for healthcare professionals [Citation15].
Effective VPS is characterised by immersive, credible, case-based, situated learning activities that address the complexity of specific clinical domains and support transfer of knowledge and skills beyond the learning situation. Learners encounter virtual patients in an adaptive, controlled, and safe environment that is designed to provide an authentic approximation of clinical practice. This environment allows learners to assume the role of an active protagonist who is treating a real patient – with real concerns and issues – and supports experiential learning by giving health professionals control over their pace of learning. A well-designed VPS requires true-to-practice, open-ended treatment decisions that allow learners to think critically about their actions and to apply their knowledge in point-of-care situations. Treatment decisions that replicate deliberate practice in a clinical environment allow us to gain objective and in-depth insight into clinician practice behaviour and treatment preferences. In the process, data regarding decisions are collected, allowing for robust evaluation of the educational programme.
A large body of research consistently associates simulation-based healthcare education, such as VPS, with better outcomes for knowledge, skill, behaviour change, speed of learning, and better long-term retention [Citation10, Citation16–22]. For example, Burgon and colleagues [Citation22] found that in a quasi-controlled experiment, in which VPS was part of the intervention with an accountable care organisation in the USA, there was a 27% improvement in evidence-based quality scores and a 55% reduction in unneeded testing in the patient simulations. Those improvements correlated with improvements in real-world quality measures and were greater than those of the quasi-control group [Citation22].
In summary, VPS provides a customised learner experience, in which targeted formative and summative feedback are provided to promote effective learning. This, coupled with substantive detailed data collection of all actions made or omitted during the course of a simulation, creates a highly stylised learning intervention.
Materials and Methods
Ethical Considerations
This study was exempt from institutional review board approval as it is research involving normal education practices; it is, therefore, exempt under 45 CRF 46.104(d)(1) according to the US Department of Health and Human Services [Citation23].
Setting
“Complex Cases in Thromboembolic Disease”, a CME-certified activity, was offered online to all Medscape.com members starting 16 December 2017, and was valid for credit for 1 year [https://www.medscape.org/viewarticle/885454]. Credit was available for a maximum of 1.25 AMA PRA Category 1 Credits™.
Intervention and Its Participants
The goal of this activity was to build cardiologists’ and haem-oncs’ confidence in applying strategies for individualised treatment of patients with PE and to ensure that best practices are applied across a variety of clinical presentations. The learning objectives were as follows:
Upon completion of this activity, participants will demonstrate improved performance associated with:
Performing appropriate workup for the risk stratification of patients with PE (aligning with gap 1)
Tailoring anticoagulant therapy in patients with PE (aligning with gap 3)
Selecting appropriate patient-centred care strategies to improve adherence to long-term anticoagulant therapy (aligning with gap 2)
The MedSims VPS is an education format that replicates a physician learner’s experience of treating patients and making point-of-care decisions. The platform captures all learner decisions and their rationales. Two cases were in this intervention: Case 1 focused on a patient with newly diagnosed PE; Case 2 focused on a patient with PE and cancer, who was on anticoagulant therapy (not a NOAC). Each case would take approximately 30 to 45 minutes to complete (). There were 18 decision points in Case 1 and 7 in Case 2 (see for case descriptions). The intervention was open to any registered Medscape member. Upon starting the activity, participants could select which case they wanted to complete, with the option of completing both cases. The case started with a video of a patient discussing their condition and history in a clinical setting.
Table 1. Case descriptions
Figure 1. Patient Simulation
The participants then interacted with the MedSims VPS platform, which simulates an EHR system. Participants make orders and receive and review test results. Clinical guidance (CG), which provides evidence-based information after selection(s) are made, serves as direct education to the participant. Guidance may include results from clinical trials, descriptions of tests, or the opinion of a leader in the clinical area in terms of their own experience and preference in the field. Upon reaching the end of a case, the participant may review all decisions made as well as important decisions omitted and change them if desired. There is also a literature review available to consult prior to finalising decisions. In this instance, the educational design is oriented to both formative and summative feedback; it can serve as best practice reinforcement or as a vehicle for behaviour change.
Data were collected from launch (16 December 2017) to programme expiration (16 December 2018). A total of 986 Ex-US cardiologists and 783 haem-oncs were learners in this online activity. Learners view the content but may not complete or make decisions in the activity. The completion rate for the first case was 54% for cardiologists and 40% for haem-oncs and 67% for cardiologists and 47% for haem-oncs for the second case. The analysis in this study includes all cardiologists (n = 475 in Case 1 and n = 245 in Case 2) and all haem-oncs (n = 280 Case 1 and n = 147 Case 2) who completed all decision points and content. Within the analytic sample, 40% were practising in Europe, 19% in Asia, 18% in North Africa/Western Asia, 14% in Latin America, 4% in Australia and New Zealand, and 5% in other countries.
Outcomes Measures
Two to 13 decision points indicated each learning objective. The pre-educational guidance (pre-CG) and the post-educational guidance (post-CG) decisions were documented for each participant. Decision points assessed with their corresponding learning objectives are in and .
Table 2. Case 1 results by learning objective and decision points
Table 3. Case 2 results by learning objective and decision points
Statistical Analysis
The statistical software that was used to conduct this analysis was the RServe Analytics extension within the Tableau server environment. Paired samples t-tests were conducted to examine change from pre- to post-CG at the participant level. McNemar’s tests were used for individual decision points for each participant. P values <.05 are considered statistically significant.
Source of Funding
This CME activity was supported by Bayer AG in 2017.
Results
Results showed a significant overall impact of education on correct decisions made in both cases for cardiologists, with a relative improvement of 22% for Case 1 (45% pre- to 55% post-CG, n = 475, t(474) = 14.12, P< .001, Cohen’s d = .46) and 19% for Case 2 (62% pre- to 74% post-CG, n = 245, t(244) = 11.95, P< .001, Cohen’s d = .59). Impact also was seen for haem-oncs, with a relative improvement of 27% for Case 1 (45% pre- to 57% post-CG, n = 280, t(279) = 11.91, P < .001, Cohen’s d = .60) and 19% for Case 2 (63% pre- to 75% post-CG, n = 147, t(146) = 9.52, P < .001, Cohen’s d = .58). See and for results by speciality, learning objective, and select decision points.
In particular, the largest improvements for cardiologists and haem-oncs under the first learning objective, performing appropriate workup for the risk stratification, were seen in ordering measurements of renal function: estimated glomerular filtration rate (eGFR), relative increase of 55% (cardiologists) and 67% (haem-oncs), and ordering estimated creatinine clearance (CrCL), relative increase of 42% (cardiologists) and 56% (haem-oncs). Overall, about half of cardiologists and haem-oncs performed the appropriate workup for risk stratification so after education.
Under the second learning objective, tailoring anticoagulant treatment, in Case 1, there was a 103% (cardiologists) and 143% (haem-oncs) relative increase in ordering appropriate oral anticoagulation therapy, with 63% of cardiologists and 51% of haem-oncs doing so after education. Of note, only 5% and 11%, respectively, ordered warfarin – a decision that was clinically appropriate for this patient after education. In Case 2, there was a 76% (cardiologists) and 80% (haem-oncs) relative increase in appropriately starting NOAC therapy, with 72% and 74%, respectively, doing so after education. There was little impact for both cardiologists and haem-oncs on appropriately discontinuing enoxaparin. Under learning objective 3, selecting appropriate patient-centred strategies to improve adherence to long-term anticoagulant therapy, orders for a follow-up appointment and patient education and counselling had a relative increase of 45% and 39%, respectively, for cardiologists and 40% and 41%, respectively, for haem-oncs.
Discussion
Overall, case-based VPS education significantly improved the percentage of appropriate decisions made for patients with PE. Other studies that used simulation with clinical guidance have also reported positive results [Citation24,Citation25]. In our analysis, cardiologists and haem-oncs showed significant improvements in all areas identified as gaps: Gap 1: risk stratification – a greater proportion are now ordering appropriate tests for risk stratification; Gap 2: monitoring patients to gauge adherence to therapy – a greater proportion are selecting patient-centred care strategies to improve adherence; Gap 3: applying the latest data on NOACs in patients with cancer – a greater proportion are now tailoring anticoagulant therapies based on patient presentation.
However, despite the significant performance improvement for nearly every decision point measured using VPS, this activity also identified opportunities for improvement:
37% of cardiologists and 49% of haem-oncs in Case 1 and 28% of cardiologists and 26% of haem-oncs in Case 2 showed persistent practice gaps as they did not initiate oral anticoagulation therapy or NOAC therapy, respectively, as recommended by current clinical guidelines.
36% of cardiologists and 43% of haem-oncs did not demonstrate patient-centred care strategies by ordering a follow-up appointment or patient education counselling after education.
Significance
VPS improved cardiologists’ and haem-oncs management of patients with PE in a simulated environment. The greatest improvement observed was related to treatment decisions: appropriately initiation of oral anticoagulation therapy nearly doubled for cardiologists and more than doubled for haem-oncs, and starting NOAC therapy increased by over 70% for both cardiologists and haem-oncs. In addition, the method of collecting outcomes in simulation by means of embedded assessments using open-ended platforms is quite novel in CME, compared with the use of multiple-choice questions. Future research will examine the correlation and impact of the MedSims VPS platform on real-world practice.
Limitations
All cardiologists or haem-oncs who viewed the activity content did not make all of the decision points; therefore, they are not included in the results. The completion rate for Case 1 was 54% for cardiologists and 40% for haem-oncs and for Case 2 was 67% for cardiologists and 47% for haem-oncs. This may bias the results towards cardiologists who were motivated to participate fully in the activity. Second, although the MedSims VPS platform provides a consequence-free environment to make clinical decisions, therapy selection may be limited in the real world, depending on location or institution. Therefore, selection of therapies in the intervention may not fully reflect what a clinician would choose in actual practice.
These results are indicative of performance in a simulated environment. However, research shows simulations are predictive of real-world practice [Citation22]. To understand whether practice changes translated to real-world performance, future studies should examine the simulation behaviours and their correlation with real-world behaviours. Future work should also look at the impact of education on those who participated in both cases and assess whether the magnitude of effect is greater for those participants compared with those who completed only 1 case.
Conclusions
The results of this study are relevant to educators in that they demonstrate that a VPS, which immerses and engages clinicians in an authentic and practical learning experience, can significantly improve decision making of physicians with simulated patients, with potential implications for clinical care. This is also relevant to physicians who wish to practice decision-making in a consequence-free environment while receiving evidence-based guidance and supplemental information. The authors have plans to conduct follow-up surveys with participants in future simulations to understand both the correlation and impact of the simulation with real-world behaviour. The MedSims VPS platform has been used successfully in other therapeutic areas to improve decision-making [Citation26–28]. Given that physicians prefer online education that is case-based and 15 to 30 minutes in length [Citation29]. It may be worthwhile to explore simulations that are shorter in duration to examine their impact. Such cases could be presented in a series, where diagnosis is one portion of the activity and treatment and future management is the second.
Acknowledgments
Activity Authors:
Faculty -
Jan Beyer-Westendorf, MD
Assistant Professor, Internal Medicine and Vascular Medicine at Technical University Dresden, Dresden, Germany
Alok A. Khorana, MD
Professor of Medicine, Cleveland Clinic Lerner College of Medicine, Case Western Reserve University; Sondra and Stephen Hardis Chair in Oncology Research; Vice Chair, Clinical Services; Director, GI Malignancies programme, Taussig Cancer Institute, Cleveland Clinic, Cleveland, Ohio
Editors -
Niyati Mukherjee, MD
Scientific Director, Medscape, LLC
Terra M. Sumstine
Managing Editor/Writer, Chichester Associates, Allendale, New Jersey
Continuing Medical Education Reviewer -
Nafeez Zawahir, MD
CME Clinical Director, Medscape, LLC
Manuscript Copy Editing and Response to Reviews -
Joyce Waskelo, MA
Director, Proposal Development, Medscape, LLC
Disclosure statement
The authors have no disclosures regarding the financial interest or benefit that has arisen from the direct applications of this research.
Additional information
Funding
References
- Streiff MB, Agnelli G, Connors JM, et al. Guidance for the treatment of deep vein thrombosis and pulmonary embolism. J Thromb Thrombolysis. 2016;41:32–8.
- Arepally G, Bauer KA, Bhatt DL, et al. The use of antithrombotic therapies in the prevention and treatment of arterial and venous thrombosis: a survey of current knowledge and practice supporting the need for clinical education. Crit Pathw Cardiol. 2010;9:41–48.
- Bauer KA. Pros and cons of new oral anticoagulants. Hematology Am Soc Hematol Educ Program. 2013;2013:464–470.
- Evans NS How do you treat your patients with VTE? Medscape education clinical practice assessment. Data on file. February 24, 2015.
- Medscape Education Survey. Factor Xa inhibitors in cardiology practice: assess your knowledge. [.pdf file]. SurveyMonkey.com. Data on file. March 29, 2017.
- Pollack CV Jr. Coagulation assessment with the new generation of oral anticoagulants. Emerg Med J. 2016;33:423–430.
- Ageno W, Mantovani LG, Haas S, et al. Safety and effectiveness of oral rivaroxaban versus standard anticoagulation for the treatment of symptomatic deep-vein thrombosis (XALIA): an international, prospective, non-interventional study. Lancet Haematol. 2016;3:e12–e21.
- Medscape Education. Clinical advances in anticoagulation and thrombosis management: a summative report. Data on file December 2016.
- Society for Simulation in Healthcare (SSH). About Simulation. https://www.ssih.org/About-SSH/About-Simulation. Accessed September 25, 2020.
- Passiment MSH, Huang G. Medical simulation in medical education: results of an AAMC survey. America Association of Medical Colleges; 2011.
- Konia M, Yao A. Simulation-a new educational paradigm? J Biomed Res. 2013;27:75–80.
- Downing SM, Yudkowsky R. Assessment in health professions education. New York, NY: Routledge; 2009.
- Huang G, Reynolds R, Candler C. Virtual patient simulation at US and Canadian medical schools. Acad Med. 2007;82:446–451.
- Berman NB, Durning SJ, Fischer MR, et al. The role for virtual patients in the future of medical education. Acad Med. 2016;91:1217–1222.
- Forristal C, Russell E, McColl T, et al. Simulation in continuing professional development of academic emergency physicians, a Canadian national survey. J Soc Sim Healthcare. 2020. epub ahead of print. DOI:10.1097/SIH.0000000000000482
- Kononowicz AA, Woodham LA, Edelbring S, et al. Virtual patient simulations in health professions education: systematic review and meta-analysis by the digital health education collaboration. J Med Internet Res. 2019;21:e14676.
- Pottle J. Virtual reality and the transformation of medical education. Future Healthc J. 2019;6:181–185.
- Quail NPA, Boyle JG. Virtual patients in health professions education. Adv Exp Med Biol. 2019;1171:25–35.
- Rudran B, Logishetty K. Virtual reality simulation: a paradigm shift for therapy and medical education. Br J Hosp Med (Lond). 2018;2(79):666–667.
- Ryall T, Judd BK, Gordon CJ. Simulation-based assessments in health professional education: a systematic review. J Multidiscip Healthc. 2016;9:69–82.
- Sakakushev BE, Marinov BI, Stefanova PP, et al. Striving for better medical education: the simulation approach. Folia Med (Plovdiv). 2017;59:123–131.
- Burgon TB, Cox-Chapman J, Czarnecki C, et al. Engaging primary care providers to reduce unwanted clinical variation and support ACO cost and quality goals: a unique provider-payer collaboration. Popul Health Manag. 2019;22:321–329.
- US Department of Health and Human Services (HHS). Human subject regulations decision charts: 2018 requirements. Last reviewed June 30, 2020. Accessed September 23, 2020. https://www.hhs.gov/ohrp/regulations-and-policy/decision-charts-2018/index.html#c2.
- Jha AK, Larizgoitia I, Audera Lopez C, et al. The global burden of unsafe medical care: analytic modelling of observational studies. BMJ Qual Saf. 2013;22:809–815.
- Oravetz P, White CJ, Carmouche D, et al. Standardising practice in cardiology: reducing clinical variation and cost at Oschsner health system. Open Heart. 2019;6:e000994.
- Apotheker ST, Bleins D, Schoonheim P, et al. Can participation in virtual patient simulation improve dry eye diagnosis and management? Presented at Association for Research in Vision and Ophthalmology in Vancouver, Canada (April 2019).
- Dermer S, Chatterjee P, Lubarda J, et al. Management of pediatric patients with atopic dermatitis: a virtual patient simulation assessment of practice decisions. Presented at European Academy of Dermatology and Venereology in Madrid, Spain (October 2019).
- Lucero K, Cohen J, Norton A, et al. Asthma patient simulation and clinician outcomes across two patient visits: outstanding outcomes in CE. Presented at Alliance for Continuing Education in the Health Professions in San Francisco, CA (January 2020).
- Lucero K, Cason C, Marshall T EU doctors’ learning preferences and implications for continuing medical education. Presented at the 11th Annual European CME Forum in London, England. (November 2018).