An e-learning course in medical immunology: Does it improve learning outcome?

Abstract

Background: E-learning is used by most medical students almost daily and several studies have shown e-learning to improve learning outcome in small-scale interventions. However, few studies have explored the effects of e-learning in immunology.

Aim: To study the effect of an e-learning package in immunology on learning outcomes in a written integrated examination and to examine student satisfaction with the e-learning package.

Methods: All second-year students at a Norwegian medical school were offered an animated e-learning package in basic immunology as a supplement to the regular teaching. Each student's log-on-time was recorded and linked with the student's score on multiple choice questions included in an integrated end-of-the-year written examination. Student satisfaction was assessed through a questionnaire.

Results: The intermediate-range students (interquartile range) on average scored 3.6% better on the immunology part of the examination per hour they had used the e-learning package (p = 0.0046) and log-on-time explained 17% of the variance in immunology score. The best and the less skilled students’ examination outcomes were not affected by the e-learning. The e-learning was well appreciated among the students.

Conclusion: Use of an e-learning package in immunology in addition to regular teaching improved learning outcomes for intermediate-range students.

Introduction

E-learning, defined as educational use of information technology (Masters & Ellaway 2008a), is used in a variety of fields in modern society and has also become an important part of medical education (Masters & Ellaway 2008a). Learning aided by material accessible on the internet offers the users a way to study independent of location and learning staff availability. E-learning has also the potential to provide a cost-effective way of reaching geographically widespread students, especially if in large numbers (Ruiz et al. 2006; Masters & Ellaway 2008b).

Electronic learning material has in certain settings been shown to be more effective than face-to-face teaching within medical education (Krönke 2010). E-learning has also been reported to be easily accepted and appreciated by students and tutors involved (Gormley et al. 2009). Use of online learning material in medicine has been found to improve course grades (Romanov & Nevgi 2007).

Most studies evaluating the effect of providing e-learning to students have been performed on a small-scale basis, normally not covering a larger field of knowledge on one electronic platform (Bilham 2009). Also, much of the e-learning offered until now has been constituted of merely static text presented on a screen, making the e-learning into no more than a book-on-a-screen (Harden 2008), without emphasis on the importance of design as an important factor in educational success (Masters & Ellaway 2008b).

Table

Practice points

E-learning is well liked among students. Use of an e-learning introduction is correlated with skill level; less skilled students use e-learning introductions more than more skilled students. Intermediate-range students can improve examination outcomes when given access to an e-learning introduction to immunology.

Moreover, many studies have tested short-term knowledge retention, looking at tests performed directly after the intervention. Studies looking at long-term effect, examining outcomes from tests performed on students several months after learning intervention, have given varying conclusions: Peroz et al. (2009) showed long-term knowledge retention to be similar for e-learning and oral lectures, while Botezatu et al. (2010) concluded e-learning to be superior to traditional teaching when students were tested 5 months after intervention. Other e-learning modules in immunology exist (Colsman et al. 2006), but little has been done to evaluate how they affect learning outcomes.

During the spring of 2009, medical students at the Norwegian University of Science and Technology (NTNU) were offered an animated e-learning package as an introduction and overview to the subject immunology. The package was offered as an independent supplement to the regular teaching. The goal of the developer was that the program should provide the student with sufficient basic knowledge to pass the immunology part of questions on a written integrated end-of-the-year examination.

The aim of this study was to examine whether use of the package was associated with better scores on the immunology questions at the end-of-the-year examination. As the main focus of the package was to provide sufficient basic knowledge, limiting in-depth knowledge and details, we a priori hypothesized that the less skilled students would benefit more from using the e-learning package than the most skilled ones. Moreover, as we assumed that the most skilled students would find the e-learning package too basic, we also expected the less skilled students to use the e-learning package more. Finally, we studied the students’ perceptions of the package.

Methods

Study design and population

Eligible to participate in this study were all students completing the second year of the medical curriculum at the Faculty of Medicine at the NTNU during the academic year 2008–2009. The Faculty of Medicine follows an integrated, problem-based learning curriculum, with one summative, integrated examination at the end of each year (Ware & Vik 2009).

The main focus areas in the second year are neurosciences, embryology, growth and development, medical communication, microbiology and immunology. As a supplement to the regular teaching, the students were offered a free e-learning package in basic immunology. The time each student spent using the program was recorded (log-on-time). At the end of the second year, the students are tested in all subjects taught during the preceding 2 years in an integrated written examination. This examination includes 120 multiple choice questions (MCQs) in addition to a number of short essay questions. On the basis of the 12 MCQs concerning immunology, an immunology score was calculated for each student. This score was correlated with the students’ log-on-time. In addition, at the end of the year, the students were asked to evaluate how satisfied they were with the e-learning package.

Of 125 medical students enrolled in the second year, 120 attended the end-of-the-year integrated written examination and these 120 students comprise the study population. The student cohort consisted of 52% females and 48% males. Mean age was 23.6 years (SD = 3.2). No formal exclusion criteria were given, though any use of the e-learning package implied knowledge of the Norwegian language.

Exposure variables

The e-learning package was made in Macromedia Flash, consisted of 25 sequences and had a total play-through length of approximately 1.5 h. The e-learning package comprised animations synchronized with audio explanations and with supportive use of graphics and text. The user was free to choose any sequence, to pause, jump back and forth and to use an integrated dictionary providing mouse-hold-over explanations of immunology-specific words. The e-learning package was intended to serve as an introduction to the subject and to give a general overview of the basic concepts in immunology. Samples of the e-learning can be viewed at: http://eforelesninger.amendor.no/amendor-electure/fullscreen/3?magic=/2/3/22//bm.

To gain access to the e-learning package, the students had to log on with their individual student identification number. Each student's individual use of the e-learning package was recorded in the number of 10-s intervals and defined as log-on-time. Logging stopped when the user had stayed idle for more than 3 min, resuming only after another action being taken. Students were informed about the study both on paper and on the web-page of the course, and they approved to participate by using their unique student number to log on to the programme. However, they could not be identified as individuals because the authors of this study did not have access to personal identification data.

Manuscripts and animations were made by the first author during his third year as a medical student at NTNU. The immunology-specific content was proofread and approved by the second author (TM, professor emeritus of immunology). The author of the MCQ items, the principal lecturer of the regular course in immunology, was not aware of the content of the package. She was, however, informed about the study and had approved that it was done. None of the authors had any involvement in the regular teaching of immunology, nor were they involved in the construction of the MCQ items.

Outcome variables

The MCQ part of the examination consisted of 120 MCQs with four to five options, all with one best answer (Ware & Vik 2009). The total MCQ score for each student was recorded. Twelve MCQs were identified as immunology questions by the first author and further substantiated by two professors of immunology. The main outcome measure was the students’ scores on these MCQs. By subtracting the immunology score from the total score, a non-immunology score was also recorded.

To address our main hypothesis, the students were a priori grouped into quartiles on the basis of their overall score on the MCQ part of the examination, consistent with the practice of the quality assessment of MCQs (Ware & Vik 2009). The 25% of the students with the lowest scores were defined as ‘less skilled students’, the 25% with the highest scores were defined as ‘the best students’, while the remaining 50% of the students were defined as ‘the intermediate group’.

An anonymous short questionnaire was given to the students at the end of the year obtaining information about their general impression of the e-learning package and whether the e-learning package made them spend more or less time studying immunology.

Statistical methods

Differences in group means between users and non-users of the e-learning package were analysed by using the Mann–Whitney U-test. Linear regression was performed to study the correlation between log-on-time in hours and immunological and non-immunological scores in percent. PASW statistics version 18.0.0 was used in all analyses.

Results

The examination results and the log-on-time were obtained for all 120 students attending the examination. The median score on the immunology MCQ part of the examination was 75% (interquartile range = 67–83%, mean score = 73%, SD = 16%). On the non-immunology questions, the median score was 78% (interquartile range = 74–82%, mean score = 77%, SD = 9%). Seven students failed the MCQ part, and among these, four had results suggesting extensive guessing, with scores lower than 50%.

A quality assessment of the MCQ examination showed mean difficulty score on the immunology questions was 73% (meaning that on average 73% of the students chose the correct option on these items), while it was 77% on the non-immunology questions. In all, 55% of the immunology MCQ items were considered to assess higher cognitive skills (i.e. reasoning and/or understanding), while this was the case for 57% of the non-immunology questions. Eighty-two percent of the immunology questions discriminated (discriminating index ≥0.15; Ware & Vik 2009) between the best and the less skilled students, compared with 62% of the non-immunology questions. The overall performance of the MCQ examination was comparable to previous end-of-the-year examinations (Ware & Vik 2009).

In all, 84 (70%) of the 120 students had been logged on to the program, and the median log-on-time for these students was 2.5 h (interquartile range = 1.7–3.6 h). There was no difference in immunology score between students who had been logged on to the program (median score = 75%, interquartile range = 67–83%, mean score = 73%, SD = 16%) and non-users (median = 75%, interquartile range = 58–83%, mean score = 73%, SD = 17%; Figure 1).

Figure 1. Box plot showing per cent score on the immunology (grey boxes) and on the non-immunology (hatched boxes) MCQs among students who used (users; n = 84) and did not use (non-users; n = 36) the e-learning package in immunology.

Among all users, there was no correlation between log-on-time and the score on the immunology questions. However, students in the intermediate score group increased their immunology score by 3.6% per hour they had been logged on (p = 0.0046) and log-on-time explained 17% of the variance in immunology score. This correlation was neither found in the low-score nor in the high-score group (Table 1).

Table 1.  Linear regression results when correlating per cent score on the immunology questions with the amount of hours the students had used the e-learning program

	Number	Adjusted r^2	β (SE)	p-value
Low-score group	21	−0.048	−0.64 (2.1)	0.77
Intermediate group	41	0.17	3.6 (0.4)	0.0046
High-score group	22	−0.010	−1.8 (2.0)	0.38

Notes: The students are grouped into quartiles based on their overall score on the MCQs. Students not being recorded using the e-learning are excluded.

Regarding non-immunology questions, there was an inverse correlation between score and log-on-time (adjusted r²= 0.036, p = 0.047). When excluding the two students in the user group with total MCQ scores lower than 50%, the inverse correlation was even stronger (adjusted r²= 0.15, p = 0.00020). The median log-on-time among the users in the less skilled group was 2.4 h, compared with 1.4 h in the best group (p = 0.094).

Sixty students answered the questionnaire about user satisfaction. When asked ‘what was your general impression of the e-learning program’, 53 students (88%) evaluated the program as good or very good and 7 (12%) as average. On the question ‘how well did the e-learning program function as a supplement to the regular teaching’, 42 (70%) answered good or very good, 13 (22%) answered average and 5 (8%) answered poor or very poor. Asked ‘did the e-learning program make you spend more or less time in total studying immunology’, 9 (15%) stated that the e-learning program made them spend less time, 44 (75%) answered neither more or less and 6 (10%) answered more or a lot more.

Discussion

In this study, we found no clear support for our main hypothesis that the weakest students would benefit most from using the e-learning package. However, use of the e-learning package improved learning outcomes for intermediate-range students. We also found evidence supporting the hypothesis that the program was more used by the less skilled than by the more skilled students. The e-learning had high user satisfaction.

The results are unlikely to be due to chance, as indicated by the low p-values. On the other hand, due to the limited number of participants, especially in the low- and high-score groups, lack of statistically significant findings in these groups should be interpreted with caution. This limitation is of particular relevance for the comparison of log-on-time between the best and the less skilled students. Another limitation may be that we were unable to explore possible gender or age differences regarding usage and effect on learning outcome, because we did not have access to personal identification data.

The MCQ items in immunology were slightly more difficult than the questions not concerning immunology and a larger percentage of the immunology MCQs discriminated between the best and the less skilled students. Bias or confounding due to particularly easy MCQ items in immunology is therefore unlikely. A bias resulting from the MCQs being adapted to the e-learning is improbable because the author of the MCQs in immunology was unaware of the content of the e-learning.

A potential bias of this study is that we do not know whether the e-learning package was accessed by groups of students. In such cases, only the one student logging on would be recorded as a user. However, we would have expected this misclassification to dilute the effects of the program. Another potential bias is that students could have exchanged their student identity numbers, but this is most unlikely because these numbers are used for a number of other purposes in confidential student communication with the Faculty.

A confounder of the observed correlation between log-on-time and immunology score in the intermediate group could be that students with higher scores were more skilled users of information technology in general and of e-learning programs in particular. However, the lack of correlation between log-on-time and non-immunology score within the same group makes this explanation less likely.

There was an inverse correlation between log-on-time and non-immunology score. This suggests that the weakest students, in accordance with our secondary hypothesis, had used the e-learning package more than the more skilled students. This interpretation may also be consistent with the borderline significant trend towards longer log-on-time in the less skilled student group. Another explanation of the inverse correlation could be that use of the immunology e-learning package had resulted in less time spent on other subjects. However, this is unlikely because the average use of the e-learning package (2.5 h) constitutes a very low percentage of the total amount of 573 h teaching in the second year of the curriculum at NTNU (Karlsen et al. 2000).

Our study may be consistent with findings in other studies showing that e-learning improves examination outcomes (Romanov & Nevgi 2007; Botezatu et al. 2010). User satisfaction has also previously been reported to be high among students using e-learning (Gormley et al. 2009). However, only one study by McNulty et al. (2000) examined the association between log-on-time and examination scores and found no correlations between examination outcome and log-on-time. The main difference between their and our study is that while we studied the effects of an e-learning package on a specific subject, McNaulty et al. studied the time the students were logged on as users of a general learning material database.

The positive association between log-on-time and examination score was only observed among the intermediate-range group of students. No such effect was found in the low- or in the high-score groups of students. However, the high-score group students still performed better on the immunology questions than the intermediate-range students. The lack of a positive effect in the high-score group may be explained by the basic content of the e-learning package. Probably, the students in this group studied more thoroughly through books, lectures and group learning sessions, therefore, not profiting from the use of an e-package with limited depth. The lack of effect among the low-score students, in contrast to our original hypothesis, may be due to these students trying to compensate for lack of participating in regular teaching by spending more time, eventually in ‘the last minute’, on the e-learning course. As the latter was a supplement to regular teaching, the e-learning may not have been sufficient to compensate for lack of participation in other learning areas.

Conclusions

In conclusion, we found that an e-learning package offered as a supplement to the regular teaching improved learning outcome in immunology among intermediate skilled students. The e-learning package had no impact on the immunology examination outcomes of the best or the less skilled students. The e-learning package appeared to be more used by the less skilled then by the more skilled students. The e-learning package was readily used and well appreciated by the students.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.