Abstract
Purpose
The effect of e-learning on practical skills in medicine has not yet been thoroughly investigated. Today’s multimedia learning environment and access to e-books provide students with more knowledge than ever before. The aim of this study is to evaluate the effect of online demonstrations concerning the quality of orthodontic appliances manufactured by undergraduate dental students.
Materials and methods
The study design was a parallel-group randomized clinical trial. Fifty-four participants were randomly assigned to one of the three groups: 1) conventional lectures, 2) conventional lectures plus written online material, and 3) access to resources of groups one and two plus access to online video material. Three orthodontic appliances (Schwarz Plate, U-Bow Activator, and Fränkel Regulator) were manufactured during the course and scored by two independent raters blinded to the participants. A 15-point scale index was used to evaluate the outcome quality of the appliances.
Results
In general, no significant differences were found between the groups. Concerning the appliances, the Schwarz Plate obtained the highest scores, whereas the Fränkel Regulator had the lowest scores; however, these results were independent of the groups. Females showed better outcome scores than males in groups two and three, but the difference was insignificant. Age of the participants also had no significant effect.
Conclusion
The offer that students could use additional time and course-independent e-learning resources did not increase the outcome quality of the orthodontic appliances. The advantages of e-learning observed in the theoretical fields of medicine were not achieved in the educational procedures for manual skills. Factors other than e-learning may have a higher impact on manual skills, and this should be investigated in further studies.
Introduction
Quality and innovation in university teaching are continuously evaluated. Currently, these aspects are dominated by the Internet because devices such as smartphones and tablet PCs have become part of the daily lives of students. This ever-increasing mechanization is also present at the university when students are taught over the Internet. It has been shown that from 2008 to 2012, the use of mobile devices for learning increased fivefold.Citation1 The task of any academic institution is therefore to evaluate the benefits of such media tools with respect to universities and their students.
Numerous studies have been conducted on e-learning in the field of medical and dental training in which participants are asked about their attitudes toward this subject. Results of these studies show that the respondents enjoy taking new media course offerings,Citation2–Citation7 without missing the traditional style of teaching.Citation8
The majority of studies focus on theoretical education. There is a positive trend that e-learning leads to an increase in theoretical knowledge,Citation9–Citation14 and only a few studies have shown no difference in terms of knowledge improvementCitation15 or that e-learning alone leads to poorer results.Citation16
A questionnaire on the topic of e-learning is a valuable tool but does not reflect the quality of the practical outcomes in medicine and dentistry. Only a few studies have assessed the transition from e-learning to practical skills. Investigations of life support training show similar resultsCitation17 or moderate improvementsCitation18 in practical outcomes when e-learning is compared with traditional didactic teaching. In physiotherapy education, students with access to e-learning sources needed less time for musculoskeletal palpation but more time in obtaining ultrasound images.Citation19 Dental students showed better results in carious lesion excavation when traditional lectures are accompanied with live demonstrations.Citation20
The effect of e-learning on practical skills is yet not well known. There is a lack of data, specifically in disciplines where manual dexterity is required. Therefore, the present investigation aimed to evaluate the influence of online video material on students’ manual skills in manufacturing orthodontic appliances. The hypothesis is that live demonstration plus access to an online demonstration would be more effective than live demonstration alone.
Materials and methods
Ethical approval
The ethics commission of the Medical Faculty reviewed the research project (Ref No: 2014-524-f-N) and gave permission to commence the study. A copy of the Institutional Review Board letter was sent to the Editor-in Chief. All participating students freely gave their informed consent to participate in this trial.
Study design
This study used a semi-blind three-parallel-group randomized controlled trial. The primary outcome was the students’ practical skill measured by the quality of the appliances that were manufactured during a preclinical orthodontic course.
Participants
The inclusion criterion for study enrolment was that students were eligible to attend the first preclinical course in orthodontics at the Dental School of the University of Münster, Germany. This included all students in the sixth semester who had passed the Physikum (preliminary examination). The exclusion criteria were failing the course (excluded from analysis), repeating the course, and violation of the course guidelines (). All course students gave their informed consent to participate in the study prior to the beginning of the trial.
Figure 1 Flow diagram of the assessment for eligibility.
Intervention
Regular attendance at the lectures was mandatory. Conventional lectures on orthodontic theoretical background and live demonstrations about appliance manufacturing were presented to all students. Lectures and live demonstrations were held in the lecture hall of the Dental School. Manufacturing of the appliances by the students took place in the student laboratory also located in the Dental School.
During the course, students were required to make three orthodontic appliances (). The manufacturing of the appliances was demonstrated by an experienced dental technician. The live demonstrations were supported with cameras and monitors installed in the lecture hall. All procedures were video recorded by professional camera operators and prepared for viewing via the e-learning portal of the department. The video material was divided into sections and labeled for clarity ().
Figure 2 Appliances that students were required to make during the course. (A) Schwarz Plate (B) Karwetzky’s U- Bow Activator Type 1 (C) Fränkel’s Functional Regulator Type 3.
Table 1 Film processing
At the beginning of the course, participants were randomly divided into three groups. Randomization was performed by the investigator SS-B using a dice roller. Group one received conventional lectures and live demonstrations during the course schedule. Group two attended the lectures and also had access to an online blog. The blog contained general information about the course, handouts of oral lectures, links to relevant literature and a step-by-step description of the appliances’ manufacturing process.
Group three had access to all resources of groups one and two plus access to the online video material. Students in group three were allowed to download the videos on their devices for offline use.
Evaluation
At the end of the course, all custom-built appliances made by the students were collected and pseudonymized by the investigator SS-B. Tutors (postgraduate students of the Department of Orthodontics), who were blinded to the groups and the students’ names, rated the appliances according to a predetermined point system. It was based on a 0–15 point scale in which 0 represents the lowest and 15 the highest achievable score (outcome measure). The evaluation protocols for the different appliances are shown in –.
Table 2 Grading system Schwarz Plate
Table 3 Grading system Karwetzky’s U-Bow Activator Type 1
Table 4 Grading System Fränkel’s Functional Regulator Type 3
Measurement error
Eighteen pseudonymized appliances were randomly selected and both tutors evaluated them independently according to the evaluation protocol.
Statistics
The statistical analysis of the data was performed using the statistical program GraphPad Prism 4 (GraphPad Software, San Diego, CA, USA). Variance analysis and a non- parametric t-test were utilized. To estimate inter-rater agreement, Cohen’s kappa values were calculated. To further differentiate Cohen’s kappa test, Spearman’s rank correlation coefficient was calculated. Means were compared using Dunnett’s test. A significance level with a 5% error probability level (P<0.05) was observed.
Results
Participants
The first orthodontic course at the Dental School has a fixed group size of 55 students. All students were included in the study and randomized into one of the three groups. The trial started with the first lecture of the course and ended on the closing date for the completion of the appliances. During the course, five students violated the course guidelines and were excluded from the analysis. There were no further dropouts. There are no significant group differences concerning age and sex ().
Table 5 Representation of group distribution
Measurement error
Cohen’s kappa for inter-rater agreement was 0.212, different from 0, which means that this is not a random match (P=0.029). According to Landis and Koch,Citation21 this is a fair agreement. However, a more reliable measure for ordinal data is the Spearman’s rank correlation coefficient, which was 0.834 (P<0.001). Cohen’s kappa measures the agreement between the evaluations of two raters when both are rating the same object on categorical scale. The Spearman’s coefficient is a parameter-free measure of correlations, which means it measures how well an arbitrary monotonic function to describe the relationship between two variables, without making any assumptions about the probability distribution of the variables.Citation22
Group comparisons
Concerning the different appliances made, there was no significant difference in the outcome score for all students (). The distribution according to the appliance type and group is shown in –.
Figure 3 Result-specific differences in the Schwarz Plate on a 15- point scale.
Figure 4 Result-specific differences in Karwetzky’s U-bow on a 15- point scale.
Figure 5 Result- specific differences in Fränkel’s functional regulator Type 3 on al 15- Point scale.
Table 6 Mean outcome scores and standard deviations concerning groups and appliances
The Schwarz Plate received the highest score, whereas the Fränkel Regulator had the lowest score (). Within the groups, there was a similar pattern without any significant difference.
Figure 6 Device-specific differences on a 15- point scale.
Abbreviations: SP, Schwarz Plate; UBA, Karwetzky’s U-Bow Activator; FR, Fränkel’s Functional Regulator Type 3.
Considering the different appliances, the results show that group two had slightly higher scores than the other groups, except for the Fränkel Regulator. However, the differences are not significant.
Concerning sex differences, females showed better outcome scores than males in groups two and three. The highest difference was found in the Fränkel group with 4.3±1.2 points for males and 8.3±2.0 points for females. Given the small group size, a statistical test was not applied.
To assess whether age has an effect on the outcome scores, the data were split according to participants’ age: ≤24 years (n=23) and ≥25 years (n=27). Although there are differences in the outcome scores because of the small numbers of the subgroups, the differences are not significant.
Discussion
In this study, we assessed the effect of e-learning resources on the outcome quality of orthodontic appliances manufactured by dental students in their first orthodontic course. Development of e-learning content is a time- and resource-consuming process aimed to improve students’ theoretical and manual skills in various fields of education. It is therefore mandatory to evaluate the benefit of new teaching methods. Particularly in disciplines where manual dexterity is required, we think that additional resources like e-learning content can improve students’ ability to convert what they learn during lectures into practical knowledge. Compared with theoretical knowledge, manual outcome quality in medical education is difficult to measure and there is a lack of data in the literature.
Strengths and limitations of the study
Chen et alCitation23 discussed teaching studies and suggested that methods are valid only if a) students are assigned randomly, b) quality and performance of the demonstrators are identical, and c) the content of the lectures and the learning environments are the same. All of these conditions were ensured in this study.
However, one limitation of the study may be the restricted number of participants assessed. The first orthodontic course at the Dental School in general is limited to students in the sixth semester who have passed the Physikum (preliminary examination). The course size depends on the number of laboratory work places available. Allocating students from higher semesters would bias the study because of their gained knowledge and skills in orthodontics. For example, FoleyCitation24 showed that the outcome quality between semesters varies. On the other hand, restriction to beginners could be seen as strength of the study because they have the same level of knowledge. Johnson et al showed that when students have a common subject but come from other disciplines, this leads to different behaviors in terms of the commitment to the learning progress.Citation25
Randomization of participants is also a strength, but the procedure used should be discussed. Dice tossing is not truly random but pseudorandom.Citation26 However, in this study, the bias of pseudo-randomization should be neglected.
Grading systems that are based on point scales are used in orthodontics to assess malocclusions and treatment outcomes.Citation27,Citation28 There are no validated indices available to evaluate the quality of orthodontic appliances. However, the validated American Board of Orthodontics Objective Grading System has a comparable reliability to our score (). The 15-point scale has been used for years at the Dental School, and this score is equivalent to a passing grade in the overall school system. Rater and participating academic staff were familiar with the application of the score. We therefore assume that the score system did not bias the results.
Table 7 Comparisons of reliability measurement between the American Board of Orthodontics Objective Grading System and a 15-point scale system
Students were explained the study design and the hypothesis of the study to be able to give their free informed consent to participate. This involves the risk of bias with regard to the Hawthorne effect. We assume that this type of bias was minimized by the fact that the evaluation of the appliances (score) had no effect of passing the course. Moreover, participants were told that they will be unaware of their particular score. Another discussion point would be whether the Hawthorne effect may have been responsible for the student performance in the courses. On the one hand, it was inevitable in this study to let the students ignorant as you could communicate with each other during the course. On the other side, it is also improbable that the knowledge has changed over the study outcome, as the students were told that their performance will have no benefit for them. Which in turn leads to the hypothesis that the students carefully carried out the production of the devices using online courses when no advantages in terms of knowledge queries or university classroom benefits were present. In the sixth dental semester at the University of Münster, students are required to undertake many tasks and manufacturing devices is only a small part of their training. The students were not informed that the manufactured devices had been evaluated, or that a knowledge test of the manufactured equipment was not performed. Thus, whether a knowledge check would lead to significant differences in this field remains unclear. The theory was that students would prefer to use only the easily accessible, quickly viewable online answers for their exams.
A further bias could be the knowledge exchange between students because it was not possible to separate participants during the course. This problem was encountered by explaining the study protocol and behavior during the course prior to the study.
A further strength of the study could be seen in the low dropout and the statistical power, although fixed group size was fixed. Age and sex of the groups did not differ significantly, and all participants have completed the same basic studies in dentistry and have technical courses of undergraduate successfully passed. Further, all participants were skilled in the use of the e-learning resources that were maintained by the university’s IT structure.
E-learning is more than desire among students. This aspect was evident in the surveys of students where over 50% of respondents took advantage of the e-learning offerCitation29 and 95% responded that e-learning was helpful.Citation30
Moreover, the economic benefit of e-learning has been shown. The University of Chicago conducted a study to determine whether online lectures about sleep medicine could replace the traditional classroom setting. After 1 year, the cost of the online course was paid off, while the cost of employing the teaching staff was incurred repeatedly.Citation31
Cohen et alCitation32 conducted a meta-analysis on the impact of visual-based instruction in education. The authors compared the effects of visual-based and conventional teaching in five areas: achievement, retention, aptitude-achievement correlation, student rating, and course completion. In the area achievement 37 of 65 studies, visual based was better than examination performance in a conventional class and a clear majority of studies favored visual based. In the area retention, the difference, based on six studies, was not statistically significant. By 10 aptitude-achievement studies no reported a significant difference. In the point student attitudes five of six studies favoring visual based. Only 10 studies examined the effect of visual based on course completion. The average withdrawal rate was 13.1% in visual based, and the average rate in the conventional classes was 13.2%. Given these results, it could be speculated that information technology development facilitates e-learning.
The impact of e-learning on theoretical knowledge gain has been well investigated.Citation9–Citation14
However, there are only a few studies that evaluate e-learning with respect to teaching practical skills in comparison with traditional lectures. Preston et alCitation33 divided graduate-level physiotherapy students into two groups to evaluate whether e-learning leads to an improvement in the practical skills applied to patients when coupled with traditional teaching methods. Although the results showed no significant differences, the group with the additional e-learning module tended to achieve slightly better results.Citation33
Hards et alCitation17 divided 20 anesthesia residents into two groups to examine the management of maternal cardiac arrest by using high-fidelity simulation and comparing the subsequent performance following either didactic teaching or e-learning. The results showed no significant differences; however, they discovered that e-learning and didactic teaching offer comparable benefits.Citation17
In a study by Perkins et al,Citation18 3,732 health care professionals received an advanced life support. One group of participants completed a 1-day course with e-learning; the other group participated in a conventional 2-day course. In comparison with traditional advanced life support, the e-learning group had a lower pass rate for cardiac arrest on the simulation tests, equal scores on a knowledge test, and diminished costs.Citation18
Rosa et alCitation20 assessed the influence of e-learning on students’ ability to remove artificial decayed dentine by hand. Twenty-seven students were divided into two groups in order to compare the traditional cavity removal lecture with a traditional lecture that included a practical, live demonstration. An evaluation of the practical cavity removal skills between the two groups showed no significant difference, although the group that received the live demonstration enjoyed a temporary benefit. However, students who participated in the live demo were able to remove nearly twice as much carious dentine in the same time amount of time as the other group.Citation20
Chen et alCitation23 investigated two groups of 16 dental assistants who were educated at the orthodontic department of the University of London. A test was conducted to determine whether the dental assistants obtained a better understanding by watching a 15-minute video demonstration on practical bracket positioning instead of listening to a 30-minute lecture.
No significant difference was found between the two groups, although with respect to bracket angulation, the group that viewed the video demonstration tended to show better results. Chen et al concluded from this study that it is slightly more effective to show video demonstrations of practical movements.Citation23
There is hardly any published study that addresses e-learning effects on the quality of manufacturing processes in education. The results of the few studies on the practical outcome quality suggest that e-learning has only minor effects. Further studies will be necessary to proof or disproof the benefits of e-learning on manual skills.
Conclusion
Our study suggests that providing students with time and location-independent e-learning resources additional to the classical lectures did not increase the outcome quality of the orthodontic appliances. The advantages of e-learning observed in the theoretical fields of medicine were not achieved in the educational procedures for manual skills. Factors other than e-learning may have a higher impact on the outcome quality. Further studies will be necessary to proof or disproof the benefits of e-learning on manual skills.
Author contributions
SS-B and TS designed the study. SS-B, CL, DW, and TS analyzed the data. CL and DW contributed to writing the paper. CL and TS supervised the clinical procedures and data collection. SS-B and TS wrote the main part of the paper. All authors gave useful comment on the analysis of data and text of the manuscript. All authors approved the final version of the manuscript.
Acknowledgments
We acknowledge support by Deutsche Forschungsgemeinschaft and Open Access Publication Fund of University of Münster.
Disclosure
The authors report no conflicts of interest in this work.
References
- EggermontSBloemendaalPMvan BaalenJME-learning any time any place anywhere on mobile devicesPerspect Med Educ2013229598
- JangHWKimKJUse of online clinical videos for clinical skills training for medical students: benefits and challengesBMC Med Educ2014145624650290
- GohJClaphamMAttitude to e-learning among newly qualified doctorsClin Teach2014111202324405914
- MeckfesselSStühmerCBormannKHIntroduction of e-learning in dental radiology reveals significantly improved results in final examinationJ Craniomaxillofac Surg2011391404820452231
- LongmuirKJInteractive computer-assisted instruction in acid-base physiology for mobile computer platformsAdv Physiol Educ2014381344124585467
- AlbarrakAIMohammedRAbalhassanMFAlmutairiNKAcademic satisfaction among traditional and problem based learning medical students. A comparative studySaudi Med J201334111179118824252898
- DavisCRBatesASEllisHRobertsAMHuman anatomy: let the students tell us how to teachAnat Sci Educ20147426227224249485
- NkenkeEVairaktarisEBauersachsAAcceptance of technology-enhanced learning for a theoretical radiological science course: a randomized controlled trialBMC Med Educ2012121822463757
- MoazamiFBahrampourEAzarMRJahediFMoattariMComparing two methods of education (virtual versus traditional) on learning of Iranian dental students: a post-test only design studyBMC Med Educ2014144524597923
- WormBSJensenKDoes peer learning or higher levels of e-learning improve learning abilities? A randomized controlled trialMed Educ Online2013182187724229729
- HeartyTMaizelsMPringMOrthopaedic resident preparedness for closed reduction and pinning of pediatric supracondylar fractures is improved by e-learning: a multisite randomized controlled studyJ Bone Joint Surg Am20139517e1261e126724005210
- KononowiczAAKrawczykPCebulaGEffects of introducing a voluntary virtual patient module to a basic life support with an automated external defibrillator course: a randomised trialBMC Med Educ2012124122709278
- MorgulisYKumarRKLindemanRVelanGMImpact on learning of an e-learning module on leukaemia: a randomised controlled trialBMC Med Educ2012123622640463
- DeBateRDSeversonHHCragunDRandomized trial of two e-learning programs for oral health students on secondary prevention of eating disordersJ Dent Educ201478151524385519
- PerozIBeucheAPerozNRandomized controlled trial comparing lecture versus self studying by an online toolMed Teach200931650851219811166
- BainsMReynoldsPAMcDonaldFSherriffMEffectiveness and acceptability of face-to-face, blended and e-learning: a randomised trial of orthodontic undergraduatesEur J Dent Educ201115211011721492347
- HardsADaviesSSalmanAErik-SoussiMBalkiMManagement of simulated maternal cardiac arrest by residents: didactic teaching versus electronic learningCan J Anaesth201259985286022777579
- PerkinsGDKimaniPKBullockIImproving the efficiency of advanced life support training: a randomized, controlled trialAnn Intern Med20121571192822751757
- Arroyo-MoralesMCantarero-VillanuevaIFernández-LaoCGuirao-PiñeyroMCastro-MartínEDíaz-RodríguezLA blended learning approach to palpation and ultrasound imaging skills through supplementation of traditional classroom teaching with an e-learning packageMan Ther201217547447822579034
- RosaQFBarcelosTMKaizerMRDo educational methods affect students’ ability to remove artificial carious dentine? A randomised controlled trialEur J Dent Educ201317315415823815692
- LandisJRKochGGThe measurement of observer agreement for categorical dataBiometrics1977331159174843571
- SpearmannCThe proof and measurement of association between two thingsInt J Epidemiol20103951137115021051364
- ChenMSHorrocksENEvansRDVideo versus lecture: effective alternatives for orthodontic auxiliary trainingBr J Orthod19982531911959800017
- FoleyJIDental students consistency in applying the ICDAS system within paediatric dentistryEur Arch Paediatr Dent201213631932223235133
- JohnsonSPurkissJHoladayLSelvigDHortschMLearning histology – dental and medical students’ study strategiesEur J Dent Educ2015192657324809952
- NaglerJRichterPHow random is dice tossing?Phys Rev E Stat Nonlin Soft Matter Phys2008783 Pt 203620718851121
- SongGYBaumrindSZhaoZHValidation of the American Board of Orthodontics Objective Grading System for assessing the treatment outcomes of Chinese patientsAm J Orthod Dentofacial Orthop2013144339139723992811
- BergoldMStrametzRWeinbrennerSEvidence-based Medicine online for young doctors – a randomised controlled trialZ Evid Fortbild Qual Gesundhwes20131071364323415342
- LevineAEvaluation of World Wide Web-based lessons for a first year dental biochemistry courseMed Educ Online2002713
- WarneckeEPearsonSMedical students’ perceptions of using e-learning to enhance the acquisition of consulting skillsAustralas Med J20114630030723386892
- BandlaHFrancoRASimpsonDBrennanKMcKanryJBraggDAssessing learning outcomes and cost effectiveness of an online sleep curriculum for medical studentsJ Clin Sleep Med20128443944322893775
- CohenPAEbelingBJKulikJAA meta-analysis of outcome studies of visual-based instructionECTJ19812912636
- PrestonEAdaLDeanCMStantonRWaddingtonGCanningCThe Physiotherapy eSkills Training Online resource improves performance of practical skills: a controlled trialBMC Med Educ20121211923176318