https://orcid.org/0000-0002-4456-6423
ABSTRACT
This study aims to evaluate the effectiveness of an online course to enable orthopaedic surgeons to acquire the core competencies necessary to prevent and treat fracture-related infections (FRI). This study included orthopaedic surgeons and residents from Latin American countries who attended an online course focused on FRI. The online course included: didactic lectures, small-group clinical case discussions, and panel case discussions. The course was delivered using Zoom® platform and designed to address four core competencies: prevention, definition and diagnosis, antimicrobial therapy, and surgical treatment. An online questionnaire was created distributing 16 questions through six clinical scenarios. Participants were invited to answer the questionnaire before and after the course. Sixty of the 78 course participants answered the pre-course, and 42 the post-course assessment. Relative to before the course, the mean post-course assessment score rose significantly for prevention of FRI (4.1 before and 4.5 after; p = 0.014), definition and diagnosis (2.4 before and 3.4 after; p = 0.001), and surgical treatment (2.2 before and 2.8 after; p = 0.011). The final score encompassing all four core competencies also rose significantly (2.7 before and 3.3 after; p = 0.001). The online course on FRI was feasible and effective, significantly increasing course users’ knowledge of overall competency in managing FRI.
Introduction
Medical education is essential for patient care and online learning, such as videoconferencing and eLearning platforms, has been used extensively to deliver lectures, tutorials, and courses remotely via handheld devices and laptops[Citation1]. The recent coronavirus outbreak (COVID-19) has created profound changes in healthcare systems and considerable challenges for medical education programmes [Citation1–3]. Consequently, a change in medical education has been necessary to deliver education in a pandemic setting.However, this rapid movement towards online educational platforms has resulted from an emergent response to a health crisis and little is known about the effectiveness of these tools.
To be effective, medical education programmes must have measurable outcomes and be based on competencies. O’Malley et al. defined competency as “the unique combination of knowledge, skills, and attitudes that enables one to perform in practice; this describes what a physician must be able to do to diagnose and treat patients with a specific clinical problem or issue”[Citation4]. Competency-based medical education is based on clearly-defined learning objectives, an explicit curriculum, and an assessment framework to ensure that learners have acquired the necessary knowledge and skills they need to transition into independent practice. [Citation5,Citation6]
Postoperative bone infection is a severe complication that can arise during the treatment of fractures. Several scientific organisations have collaborated to generate a definition for fracture-related infection (FRI) that addresses the unique characteristics of this clinical entity[Citation7]. Furthermore, in a series of publications, this consensus group reviewed best practices for the diagnosis and treatment of FRI [Citation7,Citation8]. Most of these guidelines were published in English over the last three years and much of this information has not yet been incorporated into classic textbooks. As such, many young surgeons and residents in non-English speaking countries remain unaware of these new concepts. Thus, an online “FRI course” appeared to us to be a perfect model to evaluate whether an online course format can be effective at updating course participants and introduce new competencies.
The objective of this study was to evaluate the effectiveness of an online course to enable orthopaedic surgeons to acquire the key competencies necessary for the prevention and treatment of fracture-related infections.
Materials and Methods
This prospective observational study included orthopaedic surgeons and orthopaedic surgery residents from several Latin American countries who attended an online course dedicated to managing FRI. The course was conducted using an online format. The research ethics committee approved the study, and all participants consented to participate.
The Study Sample
The study population was composed of orthopaedic surgeons and orthopaedic surgery residents who attended the online FRI course and answered the study’s online assessment questionnaire. Information regarding demographic data (age and gender) and each study participant’s country of residence was gathered. The participants’ clinical experience was evaluated based on the number of years they had been practicing as orthopaedic surgeons and whether they had already completed an orthopaedic residency training programme. All faculty instructors had access to the questions before the course and were invited to respond to the post-course assessment.
Course Structure
For the study, a 12-hour course was planned, divided into four 3-hour modules, one module per week for four consecutive weeks. The course involved eLearning activities divided into three major teaching methods: lectures, small-group clinical case discussions, and panel case discussions involving all the participants and faculty instructors. The course was delivered using the Zoom® platform. For small-group discussions, ten discussion groups – each having a maximum of eight participants and one or two facilitators – were created.
Definition of Competencies
The course programme was designed to address four core competencies related to FRI management: prevention, definition and diagnosis, antimicrobial therapy, and surgical treatment. To cover each of these competencies, the following activities were planned: 15-minute lectures; small group discussions involving real clinical cases; and panel discussions. Two of the authors were responsible for deciding the educational objectives for each lecture and which competencies the participants should acquire by the end of the activities.
Course Evaluation
To assess the effectiveness of this online learning tool, an online questionnaire was created based on 20 questions spanning six clinical scenarios. These questions were intended to assess the four previously-mentioned core competencies, among which ten different component competencies were tested, all related to the management of patients with FRI. These ten component competencies were, as part of the core competency prevention: (a) use of prophylactic antibiotics, n = 3 questions; for defining and diagnosing FRI (b) diagnosing an FRI and (c) classifications systems, n = 5 questions; for antimicrobial therapy (d) local and (e) systemic therapy, n = 3 questions; and for surgical treatment (f) early and (g) late infection management, (h) debridement, (i) intra-operative tissue sampling, and (j) implant removal versus retention, n = 9 questions (Supplementary Material 1). Two types of question were used: true/false questions and multiple-choice questions. All course participants were invited by email to answer the questionnaire two weeks before and three weeks after the course (first invitation to post-course assessment at two weeks and second at three weeks after the course). Participants were not allowed to answer the questionnaire more than once or change answers. The order of the clinical scenarios and questions varied between the pre- and post-test, but there were no changes in question content. The correct answers to the questions were not discussed with the participants during the course. The faculty instructors had access to the questions two weeks before the course, and to the result of the pre-course evaluation on the first day of the course. They were taught to address the competencies in the lectures and discussions, but to avoid making any direct references to the pre-course evaluation.
Statistical Analysis
All continuous variables were summarised as a mean, standard deviation, and range. For normally-distributed variables, unpaired and paired t-tests were used to compare inter- and intra-group means. Pearson chi-square analysis was used for categorical variables. Pre- and post-course responses were analysed both together and separately, depending on the course learning competency. Considering that many issues regarding FRI remain controversial, some multiple-choice questions had more than one correct answer. A given response option was considered correct or incorrect when greater than 66% agreement was established among the faculty instructors who participated in this process, and was further validated using the best-quality evidence available in the literature. Among the instructors, because more than 66% consensus was not achieved for four of the 20 questions in the original survey, these four questions were not included in the calculation of core or overall competencies and are reported separately. The four omitted questions asked about the best surgical option for early intramedullary infection; number of tissue samples to collect during surgery; where to take samples from; and the best way to manage infected non-union after plate fixation (Supplementary Material 2).
Each given competency ultimately was graded from 0 to 5 (with zero indicating no competence and 5 complete competence) by dividing the number of correct responses by the total number of questions testing that competency, then multiplying that fraction by 5. For example, if a participant had correct answers for three out of five questions covering a given competency (i.e. Competency in Surgical Treatment), then the final competency score would be: (5/5) multiplied by 3 = 3. Mean scores for competency then were compared between competencies (unpaired t tests) and within competencies before versus after the course (paired t tests). Also, the absolute frequency of each answer was measured for each question in the pre- and post-course evaluation, and the proportion of variation evaluated. Data were analysed using SPSS® version 25, with the threshold for statistical significance set at α = 0.05.
Results
A total of 78 participants signed up and concluded the course. Of these, 60 answered the pre-course assessment and 42 the post-course assessment. Thirty-four participants answered the pre-and post-course evaluations. Thus, an independent analysis was carried out comparing all 60 with these 34 regarding pre-course answers, and no statistical difference was observed (Supplementary Material 3). No significant differences were observed in the demographic characteristics (age and gender) or country of residence between course participants who completed the pre- and post-course evaluations, nor in the years of surgical practice. However, the percentage of participants who had finished residency training was lower in the post-course assessment (p = 0.002) (). Nine of the 16 faculty instructors answered the assessment used to support decisions regarding correct answer for the questions. The pre- and post-course results are summarised and correct responses for each question listed in through 6.
Table 1. Demographic characteristics and clinical experience of course participants who completed the pre- versus post-course assessment
Table 2. Comparing levels of the core competency – Infection Prevention – before and after the on-line course
For the core competency – Infection Prevention () – a 23% increase in correct responses was noted between the responses given before versus after the online course for the administration of a prophylactic antibiotic, a difference that was statistically significant. Minimal to no change was observed for either the choice or timing of that antibiotic; however, 98 and 90% of the responses were correct at baseline (before the course), leaving almost no potential for improvement. The overall core competency score increased from 4.11 to 4.56 (p = 0.014).
For the core competency – defining and diagnosing FRI () – a significant increase in correct answers was observed for four of the five questions (with p values ranging from 0.01 to < 0.001), and there was a marked increase in the overall core competency score, from 2.43 to 3.43 (p < 0.001). On the other hand, whereas 93% of the course participants who completed the baseline questionnaire correctly agreed with the statement that wound dehiscence with bone exposure is a confirmatory criterion for infection, this percentage was slightly lower after the course, at 85%.
Table 3. Comparing levels of the core competency – Definition and Diagnosis – before and after the on-line course
For the core competency antimicrobial therapy for FRI (), the percentage who correctly selected rifampicin to treat a Staphlococcus aureus infection increased by 35% after the course (p = 0.001). However, the percentage of correct answers fell for the remaining two questions, and there was virtually no increase at all in the overall core competency score after the course (2.94 to 2.97).
Table 4. Comparing levels of the core competency – Antimicrobial Therapy – before and after the on-line course
Finally, for the core competency pertaining to the surgical treatment of FRI (), the percentage of correct responses increased for all five items, with increases ranging from 12 to 25%, one of these changes statistically significant. The overall core competency score also increased significantly, from 2.25 to 2.85 (p = 0.011).
Table 5. Comparing levels of the core competency – Surgical Treatment – before and after the on-line course
Across the 16 questions asked, the numbers of questions where the percentage of correct answers increased, remained unchanged, and decreased were 12, one, and three, respectively, with six of the 12 increases statistically significant. The average score when all four core competencies were combined significantly increased between the pre- and post-course assessments, from 2.78 to 3.38 (p = 0.001). Comparing the four core competencies, however, there were significant differences both in the level of competency attained and the extent of improvement, ranging from virtually no improvement in the course participants’ competency with antimicrobial therapy to a marked 20% absolute and 41% relative increase in their competency defining and diagnosing FRI (). An analysis involving only the 34 participants who answered the pre and post-course evaluations showed similar results concerning the average score of the four core competencies and the final score with all core competencies combined (Supplementary Material 4).
Table 6. Analysis comparing the pre- and post-assessment grades for each competence and the final score
Discussion
The global COVID-19 pandemic has altered everyday interactions, due to the necessity of social distancing to reduce the spread of the virus[Citation9]. As medical education providers, like speciality societies, cancelled their meetings and conventions, a migration to technology-based virtual learning experiences was observed. As a result, many institutions have implemented eLearning platforms and virtual events to adhere to social distancing rules and continue with their mission of providing medical education [Citation9,Citation10].Electronic learning (eLearning) is generally defined as using the internet and other media distribution methods to enhance learners’ knowledge and performance[Citation11]. The widespread use of smartphones, tablets, and multimedia platforms affords new ways to deliver educational content. Electronic resources, such as WhatsApp and Twitter, are part of our lives, and most are open to using these technology-driven tools as education facilitators[Citation10]. The use of video-conferencing applications like Webex®, Google meet®, and Zoom® increased recently, and these applications have many features that make them perfect tools for meetings, webinars, and case discussion sessions [Citation9,Citation12].
Convenience is probably the most beneficial aspect of online learning[Citation9]. Other advantages are the ability to revisit recorded sessions, connect individuals in different geographic regions, and reduce costs[Citation12]. There is tremendous versatility in this new eLearning technological toolkit: allowing online systems to mimic live classroom interactions, live-group discussions, and feedback pools. Contrary to the traditional classroom, webinars can be offered either live or on-demand. The online format also can be expanded, giving participants more exposure to experts in their field than they may might had with traditional learning[Citation10].
Although virtual education may be convenient, it is unlikely to ever replace hands-on and face-to-face clinical experiences entirely[Citation10]. There are dynamic aspects that take place during in-person interactions that cannot be replicated virtually. In surgical specialities, virtual education encompasses virtual case scenarios, online tutorials, videos, and images. Furthermore, developers may incorporate various updated educational strategies like spaced-repetition learning and “blended-learning’, which combine online and face-to-face instruction. Besides this, the ability to network and develop meaningful relationships is more likely to occur during in-person encounters[Citation9]. These limitations notwithstanding, the scarce evidence available suggests that online learning is no less effective teaching clinical skills than traditional means[Citation13].
Most eLearning tools are limited to teaching the cognitive processes necessary for learners to develop the cognitive elements they need to perform psychomotor tasks[Citation14]. However, advancement needs a degree of automaticity and thinking achieved only through simulation and clinical practice[Citation14]. As such, eLearning could serve as an adjunct to improve a curriculum’s effectiveness, especially when the curriculum has a dominant cognitive component[Citation14]. The majority of studies evaluating the learning and acquisition of skills through written/oral examinations have used performance-based assessments in a non-clinical simulated environment, and an objective assessment using pre- and post-testing for email or web-based programs is a well-known method of evaluation [Citation14,Citation15]. Promising results have been demonstrated employing a standardised needs assessment protocol[Citation16]. An evaluation system focused on the relevance of the educational activities is essential and the use of pre- and post-event multiple-choice questions to estimate the extent of knowledge gained is a well-known strategy[Citation17]. In the present study, the assessment questionnaire we utilised to evaluate learning gaps was administered before the online course. Questions were designed to address the competencies that participants should learn by the end of the course. The post-course assessment revealed marked increases in almost all of the competencies, providing evidence that the online format effectively addressed knowledge gaps.
Some competencies, like those related to antibiotic prophylaxis, are well defined in the literature, and their recommendations are based on good-quality evidence [Citation18,Citation19]. Our questions were used to reinforce these concepts and ensure that most participants could assimilate these competencies. The high percentages of correct answers obtained in the post-test for antibiotic prophylaxis indicate that, after the course, participants were able to apply tasks related to this subject properly.
Regarding diagnosis and classification, after the course most of the participants were able to diagnose FRI based on clinical confirmatory criteria (wound breakdown or purulent drainage). A significant shift towards using the Willenegger and Roth classification system [Citation20] was observed in the post-course survey, though there is no evidence to favour one specific classification system over another. Still, the FRI consensus group recommends using a time-based classification system that emphasises the importance of biofilm maturation for FRI management [Citation21–24].
Antimicrobial therapy competencies approached some critical concepts in FRI, such as using rifampicin to treat S. aureus infection and the duration of administered intravenous antibiotic therapy. The standard recommendation is at least two weeks of intravenous therapy before transitioning to oral therapy. However, in a recent multicenter randomised controlled trial, earlier transition was not found to diminish treatment efficacy. This concept of a shorter course of intravenous therapy has been advocated in recent guidelines and after-course results have demonstrated that course participants can assimilate this concept [Citation8,Citation25].
The surgical treatment of FRI has two main alternatives: (a) debridement, antimicrobial therapy, and implant retention or (b) debridement, antimicrobial therapy, and implant removal. Implant stability and bone union must be considered when deciding between implant removal versus retention [Citation21,Citation23]. Our post-course survey results demonstrated an increase in the number of correct answers in competencies related to surgical treatment, with course participants exhibiting considerable improvement in their ability to correctly recommend surgical treatment to the detriment of clinical management. An increase in the correct recommendations for implant retention or removal and a staged surgical management decision was observed.
Although post-course results revealed substantial improvement, the final average competency score of 3.1 out of 5 is far from a desirable level; that said, the baseline competency level was low (2.1). Among the nine faculty instructors who took the course assessment, the final score was roughly 3.7. As among the course participants, among the instructors, the learning competence scores were high for antibiotic prophylaxis and very low for antibiotic therapy and surgical treatment. These findings may reflect the lack of consensus in some FRI topics and heterogeneity in the participants’ clinical practices.
Study limitations include the small sample size and that participants were limited to one geographic area (Latin America). Additionally, a small number of participants did not complete the pre and post-course survey. The post-course assessment also was completed three weeks after the course, so our results represent just a short-term view of the participants’ understanding, and cannot confirm any true intention for them to change their practice. The few true or false questions included in the assessment can also be considered a limitation, increasing the chance of correct answers. Our results should be interpreted with considerable caution, considering the peculiarities of the virtual format and curriculum used. Of note, this course format has been used extensively in face-to-face AO courses and was just adapted to be delivered online. The same is true for the pre and post-course evaluation format, which have also been used extensively to identify learning gaps and back-ward planning [Citation16,Citation17]. In part, this FRI course’s online transition benefited from the absence of psychomotor activities, like practical exercises on synthetic bone or time spent in a cadaver lab. Future studies comparing the virtual format against face-to-face courses could provide additional information about the benefits of each model in curriculum improvement.
Conclusions
Transitioning an FRI course from an in-person to online format was both feasible and effective. After course evaluations demonstrated a significant increase in the course participants’ knowledge levels in most of the competencies evaluated.
Supplemental Material
Download Zip (51.4 KB)Acknowledgments
The authors would like to thank AO trauma Latin America, especially the regional office, for their collaboration in this study, including supporting the online course and sending the survey to the participants.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Supplementary material
Supplemental data for this article can be accessed here.
Additional information
Funding
References
- Chang Liang Z, Wang W, Murphy D, et al. Novel Coronavirus and Orthopaedic Surgery. J Bone Joint Surg. 2020;102(9):745–7.
- Guo X, Wang J, Hu D, et al. Survey of COVID-19 Disease Among Orthopaedic Surgeons in Wuhan, People’s Republic of China. J Bone Joint Surg. 2020;102(10):847–854.
- Liang ZC, Ooi SBS, Wang W. Pandemics and Their Impact on Medical Training: lessons From Singapore. Acad Med. 2020;95(9):1359–1361.
- O’Malley NT, Cunningham M, Leung F, et al. Early Experience in Implementation of a Learning Assessment Toolkit in the AOTrauma Geriatric Fracture Course. Geriatr Orthop Surg Rehabil. 2011;2(5–6):163–171.
- Nousiainen MT, McQueen SA, Hall J, et al. Resident education in orthopaedic trauma: the future role of competency-based medical education. Bone Joint J. 2016;98(10):1320–1325.
- Wilber R, Subramaniam C, Nousiainen M, et al. The AO Trauma Competency-Based Training and Assessment Program: an Approach to Address Performance and Assessment Gaps in Surgical Training. J Orthop Trauma. 2021;S22–7. https://journals.lww.com/jorthotrauma/Documents/Learning%20in%20the%20AO_Looking%20Back%20and%20Moving%20Forward.pdf
- Metsemakers WJ, Morgenstern M, McNally MA, et al. Fracture-related infection: a consensus on definition from an international expert group. Injury. 2018;49(3):505–510.
- Depypere M, Kuehl R, Metsemakers WJ, et al., Fracture-Related Infection (FRI) Consensus Group. Recommendations for Systemic Antimicrobial Therapy in Fracture-Related Infection: a Consensus from an International Expert Group. J Orthop Trauma. 2020;34(1):30–41.
- Essilfie AA, Hurley ET, Strauss EJ, et al. Resident Fellow, and Attending Perception of E-Learning During the COVID-19 Pandemic and Implications on Future Orthopaedic Education. J Am Acad Orthop Surg. 2020;28(19):e860–864.
- Shah S, Diwan S, Kohan L, et al. The Technological Impact of COVID-19 on the Future of Education and Health Care Delivery. Pain Physician. 2020;23(8;4S):S367–380.
- Vaca SD, Warstadt NM, Ngayomela IH, et al. Evaluation of an E-Learning Course for Clubfoot Treatment in Tanzania: a Multicenter Study. J Med Educ Curric Dev. 2018;5:2382120518771913.
- Kogan M, Klein SE, Hannon CP, et al. Orthopaedic Education During the COVID-19 Pandemic. J Am Acad Orthop Surg. 2020;28(11):e456–464.
- McCutcheon K, Lohan M, Traynor M, et al. A systematic review evaluating the impact of online or blended learning vs. face-to-face learning of clinical skills in undergraduate nurse education. J Adv Nurs. 2015;71(2):255–270.
- Maertens H, Madani A, Landry T, et al. Systematic review of e-learning for surgical training. Br J Surg. 2016;103(11):1428–1437.
- Wutoh R, Boren SA, Balas EA. eLearning: a review of Internet-based continuing medical education. J Contin Educ Health Prof. 2004;24(1):20–30.
- Rüetschi U, Baumgaertner MR, Kapatkin AS, et al. The Journey to Competency-Based Education. J Orthop Trauma. 2021;S1–4. https://journals.lww.com/jorthotrauma/Documents/Learning%20in%20the%20AO_Looking%20Back%20and%20Moving%20Forward.pdf
- Ghidinelli M, Cunningham M, Uhlmann M, et al. Designing and Implementing a Harmonized Evaluation and Assessment System for Educational Events Worldwide. J Orthop Trauma. 2021;35(Suppl 2):S5–10. https://journals.lww.com/jorthotrauma/Documents/Learning%20in%20the%20AO_Looking%20Back%20and%20Moving%20Forward.pdf
- Gillespie WJ, Walenkamp GH. Antibiotic prophylaxis for surgery for proximal femoral and other closed long bone fractures. Cochrane Database Syst Rev. 2010 17;2010:3.
- Foster AL, Moriarty TF, Trampuz A, et al. Fracture-related infection: current methods for prevention and treatment. Expert Rev Anti Infect Ther. 2020;18(4):307–321.
- Willenegger H, Roth B. Behandlungstaktik und Spitalergebnisse bei Frühinfekt nach Osteosynthese. Unfallchirurgie. 1986;12(5):241–246.
- Metsemakers WJ, Kuehl R, Moriarty TF, et al. Infection after fracture fixation: current surgical and microbiological concepts. Injury. 2018;49(3):511–522.
- Obremskey WT, Metsemakers WJ, Schlatterer DR, et al., ICM Orthopaedic Trauma Work Group. Musculoskeletal Infection in Orthopaedic Trauma: assessment of the 2018 International Consensus Meeting on Musculoskeletal Infection. J Bone Joint Surg. 2020;102(10):e44.
- Metsemakers WJ, Morgenstern M, Senneville E, et al. Fracture-Related Infection (FRI) group. General treatment principles for fracture-related infection: recommendations from an international expert group. Arch Orthop Trauma Surg. 2020;140:1013–1027.
- Govaert GAM, Kuehl R, Atkins BL, et al., Fracture-Related Infection (FRI) Consensus Group. Diagnosing Fracture-Related Infection: current Concepts and Recommendations. J Orthop Trauma. 2020;34(1):8–17.
- Li HK, Rombach I, Zambellas R, et al. OVIVA Trial Collaborators. Oral versus Intravenous Antibiotics for Bone and Joint Infection. N Engl J Med. 2019;380(5):425–436.