Aligning Practice Data and Institution-specific CPD: Medical Quality Management as the Driver for an eLearning Development Process

ABSTRACT

For hospital physicians, alignment of Continuing Professional Development (CPD) with quality improvement efforts is often absent or rudimentary. The purpose of this study was to evaluate a CPD development process that created accessible learning opportunities and aligned CPD with practice data. We conducted a chart audit to identify patient safety and quality of care issues within the institution, then established an eLearning approach that supported quick and cost effective development of high-quality interactive CPD opportunities. We tested a pilot module on the management of common infections in sub-acute care settings with fifteen (68%) residents and three staff physicians to evaluate the approach. One resident and three staff agreed to a follow-up interview. The satisfaction survey indicated that participants felt the content was generally appropriate and the module well designed. Significant improvements to knowledge were reported in the multi-drug resistance (Mean Difference = 25%, p = 0.002), infection management (MD = 32%, p < 0.001), and cellulitis risk factor (MD = 22%, p = 0.02) questions, as well as in the overall score (MD = 19%, p < 0.001). In terms of confidence in their answers, the mean rating pre-module was 3.17, rising significantly to 3.92 post-module (p < 0.001). In this way, collaboration between quality management and education committees allowed for the development of relevant CPD for physicians, with eLearning providing a timely and accessible way to deliver training on emerging issues.

KEYWORDS:

• Elearning
• Continuing Professional Development
• patient safety
• quality of care
• common infections

Introduction

The delivery of safe, high quality patient care throughout a physician’s career requires dedication to ongoing learning. Continuing Professional Development (CPD) is critical to maintaining and refining existing competencies, as well as updating knowledge and skills in accordance with medical advances [1,2]. However, dissatisfaction with the Maintenance of Certification programme is widespread among physicians, and according to a recent national survey in the USA, physicians find that current CPD offerings have little value or relevance and are not well integrated into their clinical practice [3]. Moreover, many traditional CPD activities, such as conferences, courses, and lectures, have been shown to have little impact on physicians’ actual practice [4,5]. One particular difficulty with CPD in its current form is that it is predominantly self-directed and is based upon physicians’ own assessments of their skills and competencies [2], which have been shown often to be inaccurate [3,6,7].

Healthcare organisations such as hospitals also face learning needs, but these are more likely to be considered at the organisational and system levels in terms of quality improvement (QI) and patient safety than at the level of individual healthcare professionals [8]. Physicians who work in such organisations thus face institution-specific learning needs (e.g., infection control, EMR use, end-of-life care), which are often not covered by mainstream CPD. Moreover, alignment of CPD with QI efforts is typically absent or underdeveloped [5,9]. One reason for this is that CPD is often perceived to be less concerned with the delivery of care and more with education and training [10].

All of these factors suggest that targeted CPD for physicians is essential, but their steadily increasing workload – physicians in Canada work 51.43 hours a week on average [11] – makes attending traditional CPD activities (e.g., lectures, conferences) a challenge. Yet lectures remain the main delivery medium for CPD [4,9]. Although some physicians still prefer group learning activities such as lectures and conferences, online and blended activities are becoming increasingly popular, and studies have shown greater learner satisfaction for these modes of CPD delivery [12,13].

eLearning offers education at a time and place convenient to learners, supports instructional methods that may be difficult in other formats, and can be tailored to individual needs [14,15]. eLearning is growing exponentially and many medical educators argue it is here to stay [13,16–21]. The popularity of online learning has brought a shift from teacher- to learner-centred instruction [22,23], as well as a plethora of research on the effectiveness of eLearning [16,24,25]. When compared to methods such as face-to-face instruction, eLearning in health professions is consistently associated with large positive effects for outcomes of knowledge, skills and behaviour, and moderate effects for patient-related outcomes [12,26]. Relative to traditional methods, eLearning shows similar learning outcomes but is superior from a cost analysis perspective as it requires low enrolment to break even [25,27,28]. Overall, there is evidence to support eLearning as a viable modality for CPD.

Despite the explosive growth of eLearning, the process for developing online modules remains a mystery to many healthcare practitioners, and thus the prospect of introducing this mode of delivery can seem overwhelming. The purpose of the present study was to evaluate an eLearning process through which CPD can be aligned to quality improvement needs and implemented institution-wide to geographically dispersed physicians in a convenient, timely, and cost effective manner.

Methods

Context

Bruyère Continuing Care comprises two hospitals, two family medicine centres, two long-term care residences, and assisted living apartments, across three locations (www.bruyere.org/en/about). Bruyère is also an academic institution affiliated with the Faculty of Medicine at the University of Ottawa.

The quality of medical care across Bruyère is overseen by the Medical Advisory Committee (MAC), whose members include the Chairs of the Medical Quality Management (MQM) and Medical Education Committees (MEC). The MQM includes the Chief of Staff and Departmental Chiefs and is primarily responsible for monitoring quality of care and patient safety issues, as well as proposing solutions when issues arise. It accomplishes this by establishing indicators for standards of care, monitoring the quality of care through data reviews and departmental audits, making recommendations on clinical policy, and identifying continuing medical educational needs. The MEC comprises the Medical Education Coordinators for each department and, among other responsibilities, is tasked with addressing medical education opportunities and challenges specific to the organisation, as well as fostering ongoing scholarship and promoting Bruyère’s mandate as an academic institution. As such, its membership predominantly comprises clinician-educators, meaning that historically there has been little crossover in the membership of the two Committees. Although the committee Chairs provide regular updates via MAC, effective collaboration had been a challenge, which had led to a lack of coordination between the two committees. A fortunate cross-over in membership led to a synergistic solution in which the MQM’s departmental safety audits were operationalised as a needs assessment for CPD, with eLearning the delivery mechanism to address scheduling and geographic challenges.

Bruyère’s 201 physicians care for a medically complex population who require sub-acute, geriatric, or palliative care. Bruyère does not have a formal, mandatory CPD program for physicians, as CPD requirements are mandated by the regulatory Colleges (the College of Family Physicians of Canada and the Royal Society of Physicians and Surgeons of Canada). Typical institutional learning opportunities have included journal clubs, hospital and regional rounds, and “lunch & learns”, as well as external CPD events relevant to practice. However, attendance is low due to demands on time and workload. Further, physicians have tended to select CPD activities based on their own interests and strengths and not on their knowledge and skill gaps. Bruyère’s mandate to serve frail and vulnerable patients, many with highly specialised needs, also meant that mainstream CPD activities often did not address the challenges most frequently encountered by Bruyère physicians. We recognised that if our institution wanted physicians to improve their knowledge in these areas, we needed to develop and offer CPD that is both relevant and convenient.

The two committees also faced different, but complementary, challenges. For the MQM, the challenge was how best to address identified knowledge gaps given the scheduling and geographic challenges faced by a relatively small organisation with multiple sites. For its part, the MEC was struggling to obtain physician input on CPD needs, as well as witnessing low physician turnout at its scheduled educational activities. To address the combination of these challenges, we needed to find a solution that would (i) ensure we developed CPD of relevance to our physicians, (ii) address identified quality management issues in a responsive manner, and (iii) offer a flexible delivery system to accommodate physicians’ increasing workload. Circumstances provided us an opportunity to address these challenges when the director of the MEC was invited to join the MQM as the member-at-large. This allowed the MQM’s challenges to be viewed from an educational perspective, whereby it became apparent that the MQM was struggling to find effective solutions to address identified learning needs related to the quality of care in the institution, as this was an area in which its members, as clinician-administrators, did not have expertise. This presented an opportunity for the MEC to (i) offer more clinically relevant and institutionally appropriate CPD, and thereby (ii) be more congruent with the needs of the organisation as a whole.

It was therefore proposed that issues identified through the MQM Departmental Audits be used to identify CPD content for the MEC, which would then develop online CPD modules to address scheduling and geographic challenges faced by physicians. It was further recognised that eLearning is a good approach to educating physicians about issues that are encountered infrequently, as it provides easy access to learning when and where physicians need it. This approach is reflective of current trends in improving quality of care and learning. For example, the College of Family Physicians of Canada recently launched a practice improve initiative which encourages providers to use practice-level data to improve the patient and provider experience. Linking learning to practice activities such as this can be part of the College’s Maintenance of Proficiency program to support CPD.

Module Development

Six priority issues identified through the audits as having wide applicability for physicians across Bruyère were prioritised as CPD topics: Management of common infections; Advance care planning; Creating effective consultation letters; Cognitive impairment and driving; End-of-life care; and How to use the electronic patient record. The module on common infections was developed first and piloted so lessons learned could be used to inform development of the remaining modules. A team collaborated to create this module, which uses an experiential case-based approach and includes cases of common infections that emulate real patient encounters in sub-acute care settings that require learners to make clinical decisions similar to those they face in the clinical setting. For efficiency, the technical infrastructure for the modules was adapted from the award-winning iLEARN-Peds modules [29], a series developed for the University of Ottawa’s Department of Pediatrics that served as a pilot for our Faculty of Medicine’s eLearning strategy. A graphic designer rebranded the look and feel of the iLEARN-Peds modules, while maintaining the existing structure to ensure reusability of the technical infrastructure.

Subject matter experts were selected and met with an experienced instructional designer who served as a coach through the process. The educational approach for the modules was explained, a demonstration of the iLEARN-Peds modules was given, and a template for content development was provided. Working closely with the subject matter expert, the instructional designer developed content that was storyboarded for online production. The module was then produced using Articulate Storyline, an eLearning authoring tool that requires no programming knowledge to use. This software allows full flexibility in module design and functionality. Modules can be produced so that they can be hosted on a website or in a learning management system. Use of this tool has meant that updates to modules have been easy to manage as protocols and best practices are updated. The final pilot module can be found here: http://www.bruyere.org/en/emodules-infections.

Module Evaluation

Participants

All family medicine residents (N = 22) completing the core Geriatrics rotation with the Department of Care of the Elderly at Bruyère between July 2014 and July 2015 were charged with completing the pilot module as part of their educational requirements. All were invited to participate in the module evaluation and those who agreed signed an informed consent. In addition, Department Chiefs were asked to identify new physicians on staff to evaluate the modules, who were then invited to participate and sign an informed consent if they chose to participate.

Data Collection

At the start of the rotation, each resident participant was given a short multi-choice multi-answer quiz to assess their knowledge of common infections (Appendix A) and asked to rank their confidence in their answers overall (1 = not confident at all; 5 = extremely confident). Residents were then given access to the module. Upon completion, residents retook the quiz and completed a 20-item satisfaction survey designed to solicit feedback on the module (Appendix B). One month later, a random selection of residents was invited to an interview to provide further in-depth feedback on the module.

Staff physician participants were given the module URL and asked to complete it at their convenience. They had to complete the quiz before gaining access to the module and again on completion, along with the satisfaction survey. Each staff physician was interviewed three months after completion to solicit thoughts on the feasibility of implementing this type of eLearning module, determine satisfaction with the module, and solicit opinions on how useful the module is for staff and residents. All interviews were recorded and transcribed verbatim.

Data Analysis

Descriptive statistics were calculated for quantitative survey data. For each multi-choice multi-answer question in the quiz, the score was calculated as follows, with a negative score given the value of zero.

$questionscore=\frac{\mathrm{\#}correctresponsesselectedbylearner-\mathrm{\#}incorrectresponsesselectedbylearner}{\mathrm{\#}correctresponseoptions}$

For example, if a question had five response options, three of which were correct, a learner who selected two of the correct answers and one of the incorrect answers would score 33% for that question.

$questionscore=\frac{2-1}{3}=33\mathrm{\%}$

The overall test score was calculated the same way:

$overallscore=\frac{\mathrm{\#}correctresponsesselectedbylearneroveral\text{l}-\mathrm{\#}incorrectresponsesselectedbylearneroverall}{\mathrm{\#}correctresponseoptionsoverall}$

Paired-sample t-tests were conducted on the pre-post scores for each question and overall. The qualitative survey data, as well as the interview transcripts, were analysed inductively using the constant comparative method.

Results

Fifteen (68%) residents and three staff agreed to participate in the study (see Table 1). One resident (6%) and three staff (100%) agreed to an interview.

Table 1. Participant demographics

	Frequency	Percentage
Gender
Male	6	33%
Female	12	67%
Role
Residents	15	83%
PGY1	12	67%
PGY2	2	11%
PGY3	1	6%
Staff	3	17%
Computer proficiency
Basic (can perform simple word processing tasks; conduct a web search; send an email)	2	11%
Proficient (comfortable with a variety of software applications; can add/remove programs)	12	67%
Advanced (have advanced computer knowledge and skills; can problem-solve computer problems)	4	22%
Prior experience with online learning
Yes	16	89%
No	2	11%
Attitude towards online learning
Before
Very positive	6	33%
Positive	10	56%
Neutral	2	11%
Negative	0	0%
Very negative	0	0%
After
Very positive	6	33%
Positive	11	61%
Neutral	1	6%
Negative	0	0%
Very negative	0	0%

Satisfaction

Overall, participants liked the module. One resident commented, “Having done a lot of modules over the years, these are good ones. They are really to the point and you can really pull good application from it”. Similarly, a staff physician noted, “I was really impressed because I have done a lot of these learning modules from other hospitals … but this is probably the best designed module I have seen in term of interactivity and pertinence to what we do. … I would definitely recommend my residents look at it”. Responses to the satisfaction survey indicated that participants felt the content was generally appropriate and the module well designed. Moreover, they felt capable of managing common infections as a result of completing the module (see Table 2).

Table 2. Satisfaction survey findings

Question*	Mean (SD)
CONTENT
The module …
Includes content cases of appropriate depth and breadth	4.22 (0.43)
Presents realistic cases similar to those faced at Bruyère Continuing Care	4.33 (0.59)
Builds on my current experience and knowledge	4.22 (0.55)
Teaches the skills necessary to deal with the illness/issue cases presented	4.17 (0.71)
Provides information that is applicable and adaptable to new cases	4.22 (0.43)
Is missing information that I would find useful when diagnosing similar cases	3.22 (1.00)
Is missing information that I would find useful when managing similar cases	3.33 (0.91)
DESIGN
The module is aesthetically pleasing	4.39 (0.50)
The module is clear and easy to view	4.44 (0.51)
The module is organised and well laid out	4.44 (0.51)
I could easily find the information I needed	4.17 (1.04)
It was easy to navigate throughout the module	4.33 (0.77)
The module is sufficiently interactive	4.33 (0.59)
The icons, menu buttons, links, and controls did what I expected	4.00 (1.08)
I did not encounter any problems using the module	3.22 (1.17)
OUTCOMES
After completing the module I am able to …
Define an approach to hospital acquired pneumonia	4.06 (0.42)
Select appropriate antimicrobial therapy in the empiric management of pneumonia in patients admitted in continuing care hospitals, while avoiding excessive broad spectrum antibiotic use	4.00 (0.59)
Identify appropriate supportive measures for patients with pneumonia in continuing care hospitals	4.22 (0.43)
Define an approach to skin infection	4.17 (0.51)
Select appropriate antibiotic therapy in the empiric management of skin infection in patients admitted in continuing care hospitals, while avoiding excessive broad spectrum antibiotic use	4.00 (0.69)
Identify appropriate supportive measures for patients with skin infection in continuing care hospitals	4.11 (0.58)
Define an approach to urinary tract infections	4.28 (0.46)
Select appropriate antimicrobial therapy in the empiric management of urinary tract infections in patients admitted in continuing care hospitals, while avoiding excessive broad spectrum antibiotic use	3.94 (0.64)
Identify appropriate supportive measures for patients with urinary tract infections in continuing care hospitals	4.28 (0.46)
OVERALL
I enjoyed learning using this module	4.17 (0.71)
The module was engaging	4.11 (0.58)
The module met or exceeded my expectations	4.00 (0.59)
The module allowed me to meet some of my educational goals	4.28 (0.46)
I would recommend this module to residents on rotation at Bruyère Continuing Care	4.06 (0.94)
I would recommend this module to staff physicians at Bruyère Continuing Care	3.83 (0.92)

*Response options: Strongly Agree [5]; Agree [4]; Neutral [3]; Disagree [2]; Strongly Disagree [1]

Participants liked the instructional design of the module. One staff physician explained, “It was a very, very clear layout. [In] the first case you get paged about a patient and [you are asked] what you need to ask the nurse. I think that that is really, really helpful for learners because that is exactly how we train them to think … [and the case] definitely reflects a lot of our patients”. Another noted, “I like the fact that you are given a time pressure, you have got two competing demands, administrative and clinical, you have got to make a decision. So this speaks to judgement a little bit, like should I take this seriously or not, so that is helpful”. Another participant commented on the flow of the case, as depicted by a clock that changes time as the case progresses: “I like the time stamp because it allows you then to get a sense from real life, time marches on and you have got to make decisions and it allows you to see the outcomes of your actions and the time that it takes to get those results and sometimes you have to make decisions before you have information or time passes, so that was good”.

Other participants liked receiving immediate feedback after answering each question with an explanation of why a specific clinical decision is correct. They appreciated the way that the feedback given addressed biases that may be present in the physician while making a decision.

Participants felt the way the physical findings were presented, the use of x-rays, and the building of the patient chart as the case progressed made the case more helpful and more engaging than a simple text-based presentation. One noted, “I like the fact that I can still get the chart to remind me of what I was dealing with so that helps what I had forgotten”.

Participants appreciated how the module considered lesser known (or more rarely considered) options for patient care, including non-pharmacological supportive measures, and that the decisions had to be based on the specific details of the case rather than catch-all solutions intended to cover all bases. The presence of gold standards and references in the module helped to legitimise the content and support the recommendations of the content expert. On the other hand, criticisms of the content included the relative lack of emphasis on communication with the patient’s family and with medical colleagues (specifically one’s preceptor, for residents).

Participants did not always agree with the answers provided in the module and indicated that they would take a different approach in their practice. For example, one staff physician said “I would not feel confident managing this patient as an inpatient but maybe some other physicians would … I would actually transfer the patient to the emergency department”. However, this participant also indicated that the recommendation in the module is “fair”. Participants stressed the importance of establishing context in educational modules like these; the treatment of an elderly patient with advanced illness is likely to be different in a palliative care setting versus in acute care.

Change in Knowledge

The average scores for each quiz question and overall pre and post are shown in Figure 1. Paired t-tests indicate a significant change (p < 0.05) to participants’ scores in all questions except for urinary tract infections. Significant improvements to knowledge were reported in the multi-drug resistance (Mean Difference = 25%, p = 0.002), infection management (MD = 32%, p < 0.001), and cellulitis risk factor (MD = 22%, p = 0.02) questions, as well as in the overall score (MD = 19%, p < 0.001). In terms of confidence in their answers, the mean rating pre-module was 3.17, rising significantly to 3.92 post-module (p < 0.001).

Figure 1. Changes in knowledge quiz score. Mean percent correct for quizzes completed before and after completing the eLearning module ± standard error. Scores are represented for each question and for the total quiz score

Discussion

The purpose of this study was to evaluate a process for aligning CPD with practice-based data and implementing delivery institution-wide to geographically dispersed physicians in a convenient, timely, and cost effective manner. This project was the first time our MQM and Medical Education committees had collaborated to address quality of care issues. While perhaps surprising, this may be the result of the typical lack of crossover between physicians on the clinician-educator track and those on the clinician-administrator track. Each of these tracks develops expertise that is relevant to improving the quality of care but in isolation only provides a partial understanding of how and what to implement. Taken together, they enable a synergy of QI and CPD that can lead to better outcomes. In smaller institutions like Bruyère, where time and resources are highly constrained, it is especially important that clinical and educational priorities align. Collaboration between these two groups is valuable from the perspectives of both tracks. It can help clinician-educators recognise the importance of a continuous quality improvement mindset when choosing which scholarly projects to pursue. Through involvement at the grassroots level in quality improvement projects, they can then serve as champions for patient safety and quality of care initiatives. Equally, increasing the engagement of clinician-administrators in educational initiatives is vital, as they can serve as valuable subject matter experts and help raise the profile and importance of education in the institution. Therefore, better integration and reporting between education and quality management committees is recommended and the strongest source for doing so is practice-based data.

Using existing instructional design templates and programming files meant module development could be expedited and costs minimised. Subsequent modules took as little as two days of effort to design and develop. Having a robust process that supports the development of modules quickly is important when developing educational initiatives around patient safety issues that must be rapidly addressed (e.g., resulting from a coroner’s inquest). The module subject matter experts need to be carefully selected to ensure they have the time to be involved in the project, are a good fit with the module development process, and have the required knowledge. We suggest they are given protected time for content development if the module is to be developed in a timely manner.

Rather than asking the subject matter expert to develop content in isolation, we have found that scheduling face-to-face time with the instructional designer is most effective. The instructional designer acts as a coach to the subject matter expert; guiding them through the process and ensuring content development occurs with efficient use of the subject matter expert’s time. Regular check-ins allow the instructional designer to keep building on the subject matter expert’s content and minimise demands on their time. The goal is quickly to move the subject matter expert from the role of “content developer” to that of “content reviewer,” with the instructional designer taking on as much of the content development as possible. Hiring a senior instructional designer who had extensive experience developing eLearning solutions for medical education was thus key to the project’s success. Further, using expertise external to the organisation distributed some of the workload away from the Learning and Development Department, whose mandate is to support the learning needs of Bruyère’s allied health staff, and allowed us to set our own timelines.

Overall, these modules were well received by the learners and their knowledge of common infections improved. Although there was significant improvement in knowledge for the multi-drug resistance question, at 44% the mean score on the final quiz was still low. Examining the data revealed that learners indicated that “Treatment with antibiotics within the last 6 months” was a risk factor, whereas in fact the correct timeframe is 90 days. Given that this question related to the Bruyère Continuing Care Pneumonia protocol, this is somewhat concerning (Appendix A) and suggests that the message needs to be clearer in this module. As well, the multi-choice multi-answer format of the questions may be an issue for assessing changes in knowledge.

Interestingly, while the eLearning format allowed learners to complete it at a time and location convenient to them, the residents chose to do the module together. Instead of taking 20 minutes, residents spent an hour discussing the cases and consulting references on their Smartphones. In fact, one staff physician felt this approach might be beneficial, with a staff physician walking through the case with their residents to explain more thoroughly the reasoning and context behind the medical decisions made in the scenarios, thus allowing learners to show in a learning setting how to do what the activity intended them to be able to do. According to Moore Jr. and colleagues, these actions would put the activity at the level of competence (level 4), one level higher than knowledge attainment [30]. Having staff physicians complete the module together or come together after for a follow-up discussion could also be useful, as discussing their own perspectives on the management of the cases, the practice recommendations in the context of their setting, and potential barriers to behaviour change may improve the module’s effectiveness. For example, some participants did not agree with the management of the cases and suggested in one case the patient should be transferred. However, one purpose of the module was to reduce the number of unnecessary transfers – an issue identified in the audit – by teaching physicians how to manage the case within the sub-acute care setting. Without the opportunity for some kind of follow-up, it is likely that these learners left the module without any intent to change their practice as the recommendations did not align with what they currently do. These face-to-face collaborative means of completing the modules speak to the need for blended learning, whereby online and in-person activities are preferred modes of learning. We need to move away from the “bolus” approach to CPD, that views learning as distinct one hour segments, to consider learning as an ongoing – indeed, continuous – activity.

An important consideration that emerged when designing the module was making the context of the case clear, for example, placing the case in the context of an institution where laboratory tests are done externally as compared to a hospital setting where the physician has access to acute labs and imaging. One resident brought up the concept of medical stewardship, noting “I think it is a good reminder of what is actually evidence-based and what is necessary vs. there to, in some sense, cover your own ass”. Further, we know that residents struggle with context-specificity and the need to manage patients differently in different settings. This is a higher level competency that residents need to develop; these types of cases provide this opportunity.

We currently have nine modules developed (http://www.bruyere.org/en/emodules). Our next steps are to launch them officially within our organisation, as well as with other small local hospitals that have shown great interest in this project but do not have the resources or expertise to develop their own eLearning modules.

Conclusions

It is our hope that using MQM data to increase the awareness of patient care and safety issues within the institution, then providing a convenient solution to help close the knowledge gap, will help guide physicians’ choices in CPD and address gaps that arise in performance appraisals, with management directing physicians to specific eLearning modules, completion of which can be included in their objectives for the year. Further, eLearning is a good approach to educate physicians about issues that are encountered infrequently, as it provides easy access to learning when and where physicians need it. Our study has shown that eLearning can be an effective means of building knowledge in a timely manner and providing convenient access to both individual and group educational opportunities. The reuse of eLearning design and development templates, when possible, can make the process of eLearning module development more efficient. Finally, free online tools can be used to address some tracking needs if a learning management system is not available.

Authors’ contributions

DA and VFM conceived and designed the study and acquired the data. DA, JB and VFM analyzed and interpreted the data. DA, JB, MF and VFM drafted and substantively revised the manuscript. All authors read and approved the final manuscript.

Declarations

Ethics approval and consent to participate.

This study was approved by the Ottawa Health Science Network Research Ethics Board and Bruyère Research Ethics Board.

Acknowledgments

The authors would like to thank Emma J. Stodel for her contributions to this project and this article.

Availability of data and materials

The datasets used and analysed during the current study are available from the corresponding author on reasonable request.

Disclosure statement

The authors have no competing interests.

Additional information

Funding

This project was supported by the Innovation Fund of the Alternative Funding Plan for the AcademicHealth Sciences Centres of Ontario.

Appendix A

1. Which of the following are risk factors for multi-drug resistant organisms? Indicate all that apply.

•  □ Chronic wounds *

•  □ Treatment with antibiotics in the last 6 months

•  □ Indwelling devices *

•  □ Need for contact care *

•  □ Frequent emergency department visits

2. Which of the following measures should be considered in managing infections at Bruyère Continuing Care? Indicate all that apply.

•  □ Communication with the substitute decision-maker *

•  □ Hypodermoclysis for hydration *

•  □ Increase frequency of vital signs every 4 hours for the first 24 hours *

•  □ Subcutaneous heparin for DVT prophylaxis *

•  □ Suction p.r.n for increased respiratory secretions *

3. According to the Bruyère Continuing Care UTI Protocol, presence of 2 or more of which of the following clinical signs and symptoms should trigger a urine culture? Indicate all that apply.

•  □ Unexplained hyperglycaemia *

•  □ Foul-smelling urine

•  □ Acute haematuria *

•  □ Increased urine sediment

•  □ Change in behaviour *

4. Which of the following are risk factors for cellulitis? Indicate all that apply.

•  □ Alcoholism *

•  □ Obesity *

•  □ Venous disease (stasis dermatitis, lymphoedema, ulceration) *

•  □ Diabetes *

•  □ Paralysis of the affected body part *

Appendix B

Bruyère Continuing Care Quality of Care and Patient Safety

Online Learning Modules

DEMOGRAPHIC AND SATISFACTION QUESTIONNAIRE (Post-Module)

MODULE COMPLETED:

Thank you for participating in this evaluation. In this questionnaire the module development team would like to collect some background information about you and get feedback about the module you just completed.

Background information

1. Indicate your age (in years):

a. < 20

b. 21-30

c. 31-40

d. 41-50

e. > 50

2. Are you:

a. Male

b. Female

c. Choose not to disclose

3. Are you:

a. Staff

b. Resident

4. a. If you are a resident what program are you in? ______________

b. What rotation are you on ______________?

5. How would you describe your computer proficiency?

a. Basic (e.g., can perform simple word processing tasks, conduct a web search, send an email)

b. Proficient (e.g., use computers on a daily basis at work; comfortable with a variety of software applications; can add/remove programs)

c. Advanced (e.g., have advanced computer knowledge and skills and am able to problem-solve computer-related problems)

6. Did you have any experience with online learning before completing this module?

a. Yes

b. No

If yes, please describe the type and format (e.g., university course using Blackboard; professional development using web-conferencing).

7. Before completing this module, what was your attitude towards online learning?

a. Very positive

b. Positive

c. Neutral

d. Negative

e. Very negative

8. Since completing the module how would you describe your attitude towards online learning?

a. Very positive

b. Positive

c. Neutral

d. Negative

e. Very negative

9. From where did you view this module?

a. Home

b. Work

c. Other

10. What type of device did you use to view the module?

a. Desktop

b. Laptop

c. Tablet

d. Smartphone

e. Other: _______________

11. What platform did you use

a. Mac (IOS)

b. PC (Windows)

c. Other: _______________

Satisfaction Questionnaire

Please complete ALL of the following questions in this questionnaire using the following rating scale:

SD = Strongly Disagree; D = Disagree; N = Neutral; A = Agree; SA = Strongly Agree

NA = Not Applicable

Content (for clinical modules)

The module …

•  1. Addresses key clinical and patient safety issues encountered in Bruyère Continuing Care’s specific context of working with frail and vulnerable patients

•  2. Includes cases of appropriate depth and breadth

•  3. Presents realistic cases similar to those faced at Bruyère Continuing Care

•  4. Builds on my current experience and knowledge

•  5. Teaches the knowledge and skills necessary to deal with similar cases in my work

•  6. Provides information that is applicable and adaptable to new cases

•  7. Is missing information that I would find useful when diagnosing similar cases in my work

•  8. Is missing information that I would find useful when managing similar cases in my work

Do you have any suggestions that will improve the content in this module?

Do you have any comments regarding the content in this module?

Content (for EPR module)

The module …

•  1. Clearly explains MEDITECH and its features

•  2. Includes an appropriate number of tutorials

•  3. Includes tutorials that address the key tasks I have to perform using MEDITECH

•  4. Builds on my current experience and knowledge

•  5. Teaches the skills necessary to use MEDITECH

•  6. Is missing information that I would find useful when using MEDITECH

Do you have any suggestions that will improve the content in this module?

Do you have any comments regarding the content in this module?

Design

•  9. The module is aesthetically pleasing

•  10. The module is clear and easy to view

•  11. The module is organised and well laid out

•  12. I could easily find the information I needed

•  13. It was easy to navigate throughout the module

•  14. The module is sufficiently interactive

•  15. The icons, menu buttons, links, and controls did what I expected

•  16. I did not encounter any problems using the module

Do you have any suggestions that will improve the design of this module?

Do you have any comments regarding the design of this module?

Overall

•  17. I enjoyed learning using this module

•  18. The module was engaging

•  19. The module met or exceeded my expectations

•  20. The module allowed me to meet some of my educational goals

•  21. I would recommend this module to residents on rotation at Bruyère Continuing Care

•  22. I would recommend this module to staff physicians at Bruyère Continuing Care

Please provide any other comments about the module: