New directions in e-learning research in health professions education: Report of two symposia

Abstract

Background: The use of Computer Assisted Instruction (CAI) is rising across health professions education. Research to date is of limited use in guiding the implementation and selection of CAI innovations.

Aims: In the context of two symposia, systemic reviews were discussed that evaluate literature in Internet-based learning, Virtual Patients, and animations. Each session included a debate with the goal of reaching consensus on best current practices and future research.

Methods: Thematic analysis of the discussions was performed to arrange the questions by theme, eliminate redundancy, and craft them into a cohesive narrative.

Results: The question analysis revealed that there are clear advantages to the use of CAI, and that established educational theories should certainly inform the future development and selection of CAI tools. Schools adopting CAI need to carefully consider the benefits, cost, available resources, and capacity for teachers and learners to accept change in their practice of education. Potential areas for future research should focus on the effectiveness of CAI instructional features, integration of e-learning into existing curricula and with other modalities like simulation, and the use of CAI in assessment of higher-level outcomes.

Conclusions: There are numerous opportunities for future research and it will be important to achieve consensus on important themes.

Background

The use of computer assisted instruction (CAI) and e-learning technologies in health professions education has risen progressively over the past five decades. Numerous variations of Internet based learning, computer-based simulations, virtual patients, and many other innovations are being used extensively at all levels of training. These modalities both supplement and in some cases replace traditional instruction. In the past decade, ubiquitous Internet access, teachers’ experience, and learners’ comfort with these technologies have only served to increase the pace of adoption. Concurrent with the growing use, research in CAI has also flourished with the goal of informing best educational practices.

Table

Practice points

Repeated systemic reviews of literature in Computer Assisted Instruction (CAI) have found limited evidence to guide the increasing adoption of these teaching tools. Numerous advantages support the use of CAI, though they may not be an appropriate solution across the entire educational landscape. Educational theory, both from within and outside of health education, should inform the selection and use of CAI. Future CAI research should focus on features that are most effective, integration into curricula, and new assessment tools.

In the context of two symposia convened in 2009, one at the annual meeting of the American Association of Medical Colleges (AAMC), and other at the annual meeting of the Association for Medical Education in Europe (AMEE), the authors summarized three systematic reviews of central and emerging educational technologies: Internet-based learning (Cook et al. 2008; Cook et al. 2010d), computerized virtual patients (Cook et al. 2010b), and computer animations (Ruiz et al. 2009). The symposia were structured to highlight how to translate the accumulated evidence to inform implementation of CAI strategies and provide directions for future research.

The audiences comprised of medical education and informatics leaders, Deans, and educational researchers. Each session included time for a debate around these issues in an effort to reach consensus on future research. This manuscript represents a thematic analysis of the questions and concerns raised by symposia attendees as a reflection of challenges facing the science of medical education and the real-world practices of implementing CAI.

Methods

Following a summary of each of the systematic reviews we opened the floor to the audience for comments and questions. We recorded and subsequently transcribed this dialog in its entirety. We then arranged the questions by theme, merged questions to eliminate redundancy, and edited our responses to craft a cohesive narrative.

Summary of systemic reviews

Internet-based learning in the health professions: A meta-analysis

Cook et al. evaluated studies of the impact of CAI as compared to “traditional” education and to no intervention (Cook et al. 2008). Their evaluation of the literature through 2007 (130 studies) found that CAI is consistently superior to no intervention. Evidence is more mixed when CAI is compared with traditional teaching methods (76 studies), but when averaged across multiple studies there is essentially no difference between CAI and non-CAI instruction. These reviews also highlighted the fact that the majority of existing quantitative studies of CAI in health education were designed as comparisons to no intervention, a significant obstacle to the translation of their results to the practice of teaching. Reviewing the studies to date did however reinforce that CAI research is rapidly expanding (Cook et al. 2010c), and is doing so across the full spectrum of medical education topics, learners, and settings.

In the second phase of their study, Cook et al. looked at comparisons of two CAI interventions (Cook et al. 2010d). These studies (N = 51) have addressed 22 distinct research themes. Meta-analysis found that interactivity, practice exercises, feedback, and repetition of study material were associated with higher learning outcomes, and that online discussion, interactivity, and use of audio were associated with higher satisfaction. This review also found that, contrary to popular belief, CAI is not necessarily more efficient than other methods (Cook et al. 2010d) and that learning outcomes show a strong and statistically significant association with time on task.

Computerized virtual patients in health professions education: A systematic review and meta-analysis

Cook and Triola evaluated 45 years of literature describing data on Virtual Patients (VP; Cook et al. 2010b). VPs have been defined as “a specific type of computer program that simulates real-life clinical scenarios; learners emulate the roles of health care providers to obtain a history, conduct a physical exam, and make diagnostic and therapeutic decisions,”(AAMC 2007) Similar to the above study of more generic CAI interventions, research on VPs shows that these are much more effective than no intervention, have small comparative advantages to non-computer instruction, and suffer from a paucity of research identifying best practices.

Computer animations in medical education: A critical literature review

Ruiz, Cook, and Levinson conducted a systematic review of published studies evaluating the use of computer animations in medical education (Ruiz et al. 2009). Despite the rapid proliferation of animations, this review of the literature revealed very few published studies that would inform the use of these tools. Only when the investigators included studies from non-health professions education was sufficient evidence found to draw even tentative conclusions. Contrary to popular belief, there is growing evidence that animations are not always superior to still images and in some cases may impede learning. The educational benefits of animations vary according to learner characteristics, such as prior knowledge and spatial ability, implying tailoring different types of animations to individual learners may be more effective. Learning outcomes also vary for different learning tasks, indicating that different tasks may require different instructional designs. Despite the cost and effort required to produce animations, existing medical education research does little to inform their use.

Symposia themes

Theme: When and how should CAI be used?

Context

There is significant interest by educators to use CAI in their curricula (Cook et al. 2010c). However, e-learning technologies are really just tools enabling solutions to pedagogical or practical challenges. CAI features can facilitate teaching techniques not possible using traditional methods, overcome issues of scale, or provide ubiquitous availability of learning resources. For example, interactive media-enhanced Virtual Patients often include authentic video material of real patients, allow large numbers of learners to practice whenever they want, and permit computer-created customized feedback based on individual performance. Implementing a similar program using traditional Standardized Patients would prove difficult and costly in terms of both money and the time of simulated patients, faculty, and learners.

External pressures on the traditional models of health professions education are also driving the adoption of CAI. Issues such as the decreased medical student exposure to case variation in ambulatory and inpatient settings are being addressed by CAI simulations. CAI presents one method of efficiently providing learners with interactive exposure to diverse representations of clinical concepts and the opportunity to repetitively practice problem solving or clinical reasoning (Friedman et al. 1991). These features have been shown to be key ingredients to effective instructional design (Clark 2008).

What advantages does CAI provide?

Our response: CAI has advantages across the education mission, many of which are outlined above. CAI applications can scale to hundreds or thousands of learners, be accessed anytime or any place, provide rich interactivity and learner-specific feedback, and provide for both repetition and variation of a topic or problem. CAI can facilitate educational interventions. Using CAI, learners have access to the best examples of lectures, radiographic studies, heart sounds, etc.; all of which have perpetual durability in their digital formats. These tools can also give learners much more control over their educational materials, allowing them to create custom collections, organize content for individual or small group learning, and self-pace their exposure (Triola and Holloway 2011). Faculty teaching with CAI can also realize the efficiencies of preparing online lectures or digital teaching materials at their convenience and unconstrained by classroom schedules.

CAI can also facilitate assessment. Computer-graded assessments can ease the logistic hassle of test administration, and by requiring a minimum passing score can ensure mastery of key concepts prior to advancement. The use of tools such as electronic portfolios that aggregate learner data can also provide a much more complete picture of individual learner performance. Education administration can more easily use the electronically collected CAI data for usage tracking and regulatory compliance.

How can we know when to use CAI versus face-to-face (FTF) education?

Our response: Currently available evidence only incompletely informs the answer to this question. Unfortunately, given the range and variation of educational activities there is no single answer, and research to accrue relevant evidence is challenging. To address similar questions in clinical medicine, a drug study might do subgroup analysis to determine the effectiveness of a given drug in specific patient populations. Similar analyses could in theory be done in CAI research, but entail at least two layers of additional complexity: the variation in topics taught, and the variation in instructional methods employed in both the CAI and the FTF formats. Thus, instead of looking at drug-patient interactions, we are looking at modality-student-topic-CAI method-FTF method interactions. Achieving the power needed for such analyses will require sample sizes available only through research at the epidemiological level. Studying national populations of medical graduates (Andriole et al. 2005; Andriole and Jeffe 2010; Cook et al. 2010a) is possible, but even these studies may lack sufficient controls to definitively answer the question. Thus, for most researchers, qualitative research may offer the most efficient way to answer the question of “CAI vs FTF?” The answers, we believe, will be largely practical and address issues of cost, both actual and opportunity, and the logistics of overcoming barriers of time, distance, and faculty availability.

However, available evidence and anecdotal experience suggest some principles that can be used to make such decisions about CAI vs FTF. In general CAI seems to be more suited to foster skills in the knowledge domain (e.g. clinical reasoning) rather than practical skills like examining a real patient, where face-to-face contact is required, or physical tasks suited to mannequin simulators and partial task trainers. CAI also can be useful in the preparation for FTF experiences (e.g. seeing real patients, small group discussions). Finally, we believe this is a false dichotomy: educators will be best served by working to ideally blend, adapt and sequence CAI and FTF instruction than to exclusively use one or the other modality (Ruiz et al. 2006).

Can we replace the first two years of medical school with CAI? What would be the impact?

Our response: This question is of increasing relevance given the number of medical schools going through curricular reform and considering structural changes to how they deliver basic science and clinical education (Cooke et al. 2006). As we shift to new curricula across the country and as academic medical centers increasingly include remote campuses, we must embrace new models of how our students flow through medical education. Many students already perform the “CAI replacement” by not attending lectures and accessing materials using solely electronic means (Billings-Gagliardi and Mazor 2007; Kircher et al. 2010; Traphagan et al. 2010). These students are part of the social network but not part of the face-to-face education network. Embracing CAI as a complete replacement for the initial years of medical school is a complicated proposition and has pedagogical, political, financial and ethical dimensions.

With curricular reform, many schools are introducing more and more novel educational activities which do not lend themselves to CAI: continuity patient exposures, small group and team-based learning exercises, simulations with mannequins and standardized patients, etc. In this context, CAI could be viewed as a mechanism to free-up time for more of these interactive and engaging in-person activities by replacing some traditional didactic exercises like lectures. Though there are limited data, issues such as individual motivation, social engagement and the enjoyment or collegiality of face-to-face teaching and learning are likely important factors and are now being studied more extensively.

When do we best use VP's? What do VP's measure?

Our response: As we noted above, there are limited data on VP use, and many of the technologies available today did not even exist when much of this literature was written. VPs have been principally designed to foster the development of clinical reasoning skills. There is abundant evidence that clinical reasoning skills have specificity to individual cases as opposed to a global competence (Norman 2005, 2008), and that much of clinical expertise arises from the ability to rapidly and accurately apply pattern recognition, which in turn is accumulated by exposure to numerous clinical exemplars (Norman 2005). VPs can readily provide learners with multiple and varied case examples, and also provide feedback on both the accuracy of the learner's diagnosis and treatment plan, and the information-gathering and decision-making path (Cook and Triola 2009). Given these characteristics, VPs are increasingly used to accelerate the development of clinical reasoning as part of a spectrum of simulation that may also include more expensive and personnel-intensive activities involving standardized patients, physical simulators, partial task trainers, and more. VPs can also be easily used by groups of learners collaboratively reasoning through a clinical problem (Bryce et al. 1998). Similar to OSCEs, the VP can be used as an assessment tool, one that provides much richer data on a given learner's problem-solving skills.

A limited number of qualitative papers have attempted to understand the entirety of the learner's VP experience (Bryce et al. 1998; Bearman 2003; Bergin et al. 2003; Mallott et al. 2005; Huwendiek et al. 2009). Overall this question is difficult to answer given the heterogeneity of VP applications studied to date, and the lack of standardized measures of quality across studies. Recent collaborations such as the eVIP effort (eViP 2011) in Europe provide environments that overcome barriers such as lack of standardization of the assessment and intervention and small sample sizes. These new laboratories may be the solution to answering fundamental VP research questions.

What instructional design factors or learning strategies are most appropriate for CAI? What theories (conceptual frameworks) support the use of CAI and VPs?

Our response: Theory-based selection and implementation of CAI is essential. While the evidence base deriving from health professions education is incomplete, researchers in other fields have shed substantial light on this topic. Richard Mayer, for example, has programmatically studied CAI for over 40 years using the lens of cognitive psychology (Mayer 2010). Building on other theories and his own empirical research he has developed the Theory of Multimedia Learning, and about a dozen Principles of Multimedia Instruction (Mayer 2009). Other relevant theories come from the constructivist approach such as cognitive flexibility theory, which has been applied across health professions education (Loving 1993; Heath et al. 2008), the Systems Approach Model (Dick 1978), and Cognitive Load Theory (Sweller 1988). There remains a great need for clarification studies of CAI applications and strategies to understand how and why they work and establish clearer roles for their use (Cook et al. 2008). Clarification studies are also needed to identify the specifics of learner assessment within CAI. Answering these questions will require sophisticated evaluations and collaborative research. Overall the important question to answer, one that could be an element of many future studies, is “How do we design CAI in a way that is effective and efficient, and meets the needs for our medical students and other health professionals?”

What about action research (design research)?

Our response: Action research, or research conducted in naturally-occurring educational settings, figures prominently in the evidence base accumulated thus far (Cook et al. 2008, 2010d), and we anticipate it will continue to play a vital role going forward. Whether such studies use existing groups or randomization and quantitative or qualitative methods, they will best advance the science of e-learning when they focus on questions that clarify when and how to use CAI.

Theme: Local use of CAI

How do I convince my Dean to use CAI?

Our response: The most frequently recurring theme at each of the discussions revolved around how to increase the use of CAI within the local institution. It seemed as though people felt that the technology train was leaving the station, and that if they didn't jump on board they’d be left behind. If the Dean is concerned about the comparative effectiveness of CAI, then we suggest first responding based on existing evidence. First, from a conceptual/theoretical standpoint, what matters is not the medium but rather the instructional methods; superior learning will result from superior methods regardless of whether these methods are implemented using CAI, lecture, paper-based cases, etc. Second, empiric evidence bears this out - namely, CAI consistently has a large effect size (when compared with no intervention) and on average there is little difference in educational outcomes between CAI and traditional teaching (Cook et al. 2008). If these arguments fail, it is entirely reasonable to conduct scholarly curricular evaluations to address local needs to demonstrate that the new e-learning intervention “works.” This may require a comparison with no intervention, or with traditional methods (i.e., the “media-comparative study”). While we’ve argued extensively against such research studies as regards the global research agenda, pressing local needs often dictate the hypotheses to be tested. Such studies can accomplish important local objectives by showing (for example) a Dean/funder/stakeholder that their money and resources are going to good use and that the intervention is worth the added expense. The important thing for those who aspire to scholarship addressing CAI is to remember that such local evaluations may stimulate limited interest from others, and thus may be difficult to publish in a peer-reviewed forum. If one wishes to publish results, it might be best to attempt two goals simultaneously - conduct the media-comparative study (to satisfy the demands of the Dean) and also conduct the clarification study with intent to publish.

What is the cost? Especially for VP's

Our response: Almost all recent educational advances, technological and otherwise, are associated with higher costs, yet these are seldom reported in the literature. One study that looked specifically at the development of VP cases found that on average they cost more than $10,000 per case to develop and the amount of faculty effort involved was quite significant (Huang et al. 2007). Mannequin-based teaching and standardized patients are also associated with significant additional expense. There are limited data on how much traditional education ‘costs’ and determining this, including the nuances of lost clinical productivity by faculty, can be a complicated prospect (Mennin and Martinez-Burrola 1986). Fortunately, as technologies improve and commercial or free sources of content and cases become prevalent, the ongoing costs to schools using these advances are likely to decrease dramatically.

How can schools/programs evaluate the appropriateness of a CAI solution prior to implementation?

Our response: Like most informatics solutions, the technologies involved are far less important than the cultural and pedagogical appropriateness. It is certainly reasonable to pilot test a solution with faculty and students to judge their perception of the interface and willingness to accept the potential changes in work-flow (Beckman and Cook 2007; Cook 2010). It is also important to evaluate the capacity of local resources to support the application and develop faculty in its use. This pre-implementation evaluation is often an excellent opportunity to identify champions among the students and faculty to help with the implementation project. Without buy-in of the stakeholders, especially the faculty, even the best CAI application is doomed to failure.

How can research findings be translated from controlled settings to an individual school's environment?

The challenges to local use of CAI are similar to the challenges found in applying clinical research findings to individual patients, and in both cases the science of translational informatics aims to provide a solution. Many local factors play a role in the success of CAI: technology-specific issues such as servers and databases, support and training resources, and faculty acceptance. As noted above, the current research findings can guide local implementation but are far from providing definitive answers. These findings need to be looked at in the context of your local learning and curricular culture: is there enough evidence that this intervention is at least as good as your current practice? Will CAI solve an important problem in such a way as to justify the cost and effort? Is there an opportunity for you to structure your implementation as a study that could provide additional research findings?

Theme: Future research

What are the important targets for future research?

Our response: Descriptive studies, qualitative studies, controlled trials in an experimental setting, field trials of applications, and work-based action research will all help to inform the evidence base as we try to understand how to use these technologies effectively. The most important part of any research study is the research question, followed closely by the use of rigorous study design and analysis. We believe that coming to consensus around important research themes would greatly benefit the medical education research community. Such themes need not constrict researchers to a single lens or paradigm; indeed, diverse approaches and methods could be used to answer questions such as:

• Which instructional design features of CAI material and applications are most effective at achieving learning objectives (including evaluation of variations of interactivity, computer provided feedback, and appropriate levels of authenticity and fidelity)?

• How can CAI be integrated into existing curricula?

• How can CAI be combined and sequenced with other innovative teaching methods to maximize the trajectory of learning?

• How can CAI-based assessment tools measure clinical reasoning or other higher-level learner characteristics with validity and reliability?

• How can computer-generated feedback be used to promote formative changes within learners?

Conclusion

The use of CAI and e-learning technologies is on the rise across health professions education. The educational context has changed dramatically in the past decade in the setting of post-Flexnerian curricular reform efforts. These changes, and much of the recent literature in medical education, highlight a renewed emphasis on novel teaching and learning strategies. Outcome measures have shifted away from a focus on immediate changes in knowledge recall to more sophisticated measures of change in clinical reasoning, problem solving, and application of concepts. Investigators and educators recognize that the body of CAI evidence is often of limited use in guiding the growing number of CAI projects, is lacking in power, and has thoroughly and repetitively evaluated basic comparative questions. The research to-date has not addressed some of the most important questions, and thus the failure to find definitive evidence is an indication of the need for more (and different) evidence - not less.

The questions asked by symposia participants reflected the needs of a diverse group who are grappling with rapidly evolving technology, mounting pressures on health education delivery models, and the pedagogical changes that accompany curricular reform. Though several schools are investing significant resources to implement CAI solutions, many remain unfamiliar with evidence for their effectiveness and are unsure of their role given a lack of best practices. Educators recognized that curricular reform is introducing greater heterogeneity in local educational structures and teaching strategies; these differences can be powerful barriers to implementing CAI solutions.

In the decade to come we anticipate significant research that builds upon existing evidence and creates new knowledge that informs both the functional and pedagogical design of CAI across the education spectrum. We also foresee that large-scale education collaboratives and CAI projects (EVIP 2011) will provide new laboratories that have the power to answer sophisticated comparative questions. The underlying pedagogical and instructional approaches should remain at the center of this work. New technologies should never be the tail that wags the instructional dog. Rather, new instructional strategies and curricular reform efforts should be the requirements that define future technology development.

Ethical approval

As no human subjects were involved, ethical approval was not required.

Acknowledgments

No external funding support.

Declaration of interest: The authors have no affiliation with an organization with a financial interest in the subject matter, and are not aware of any conflicts of interest. The authors alone are responsible for the content and writing of this article.