https://orcid.org/0000-0003-3791-4669
ABSTRACT
Background
During the COVID-19 pandemic, pre-clerkship medical education, including all physiology classes, was obliged to change to online teaching due to limitations of on-site (face-to-face) classes. However, the effectiveness of online teaching in non-lecture physiology topics during the COVID-19 pandemic has not been thoroughly investigated.
Method
We conducted a prospective study to evaluate the students’ academic achievement and opinions on online teaching during the COVID-19 academic year. Academic achievement of 312 students in the COVID-19 year was compared with that of 299 students in the pre-COVID-19 year. Student opinions regarding social interactions and the preferred learning method were also collected.
Results
We found that student academic achievement in the non-lecture physiology topics, assessed by summative scores, was 4.80±0.92 percent higher in the pre-COVID-19 year than in the COVID-19 year (P < 0.01, Cohen’s d = 0.42). Students rated that online classes tended to reduce their interactions with peers and teachers; however, students preferred online learning over traditional on-site learning.
Conclusions
This study pointed out that students’ academic performance related to the physiology topics taught by online non-lecture methods during the COVID-19 pandemic was lower than their performance when the topics were taught by the traditional (on-site) methods, although students reported that they preferred the online teaching. Hence, we suggest that medical teachers should deliberately plan and utilise a variety of tools and techniques when developing online non-lecture classes to preserve the interactivity of the classes, which might overcome this gap in students’ academic performance.
Background
Teaching medical students during the coronavirus disease 2019 (COVID-19) pandemic has been a challenge for medical schools globally [Citation1]. Thailand encountered the first COVID-19 case in January 2020, and the infection continued to spread to every province by the end of March. In concern of the viral spreading, the Thai government has reacted accordingly with national lockdown, travelling restrictions, physical distancing, and closure of public spaces, including schools and universities [Citation2].
Meanwhile, medical education cannot cease or postpone for a long time. Medical schools worldwide responded by switching from traditional on-site (face-to-face) teaching methods to online teaching methods, particularly for pre-clerkship education, and most of them were reported to have an overall negative impact on undergraduate medical education [Citation3,Citation4] or reported that this required further validation [Citation5]. Medical schools in Thailand must also continue teaching during the lockdown with the restriction of on-site teaching methods, especially in pre-clerkship classes in which basic scientific knowledge rather than clinical experience is emphasised. Traditional on-site physiology teaching in medical schools has a variety of teaching methods, such as lectures, laboratory demonstrations, basic medical skill practical classes, small group discussions, case conferences, and flipped classrooms with student presentations [Citation6,Citation7]. Some medical schools successfully combined on-site and online teaching methods even before the pandemic [Citation8,Citation9]. In 2020, most on-site physiology classes were forced to switch to online classes using various teaching methods, such as recorded/asynchronous lectures (so-called electronic lectures, e-lectures, or e-learning), recorded/asynchronous laboratory demonstrations, live interactive lectures, live small group discussions, and live case conferences; and the outcome of online teaching has been shown to be considerably effective [Citation10–12].
Second-year pre-clerkship medical students from our institution (the Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand) in the year 2020 were in the midst of the transition from on-site to online teaching since they did not encounter the pandemic in their first year. As they experienced both teaching methods (the on-site methods in their first year and the online methods in their second year), it raised our interests and concerns whether their academic achievement was influenced by the switch in teaching methods due to the COVID-19 pandemic. We have reported previously that asynchronous online lecture was not as effective as the traditional on-site lecture for teaching cardiovascular physiology [Citation13]; however, such an effect on non-lecture classes has not been investigated. We also questioned whether online teaching during the COVID-19 pandemic could substitute traditional on-site teaching. Thus, this study was primarily aimed to evaluate the effectiveness of online teaching in non-lecture physiology topics in the second-year pre-clerkship medical students during the COVID-19 pandemic by analysing student academic achievement using their summative scores. In addition, we also explored the students’ opinions on online teaching in the non-lecture physiology topics in the COVID-19 year.
Methods
Study population and study design
This study design was a prospective cohort with historical control. We prospectively gathered the data of the second-year medical students in the academic year 2020 (COVID-19 year) and used the data of the second-year medical students from the previous academic year (pre-COVID-19 year) as a control. This is under the assumption that the only difference between these two academic years was the shift from on-site to online teaching, and a similar academic background could be presumed from the similar enrolment criteria of our undergraduate medical program.
We aimed to assess the effectiveness of the online non-lecture classes in topics related to physiology in the Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand. This study included 7 physiology topic-containing courses taught in the first semester of the second-year medical curriculum. The list of the courses is shown in . Students who enrolled in all the 7 courses and took all summative examinations were included in the study unless they opted out.
Figure 1. List of the courses taught in the first semester of the second-year medical curriculum.
In general, each course aimed to teach basic medical knowledge relevant to the organ system and contained topics of anatomy, physiology, and biochemistry. All topics in each course were designed and approved by the course-coordinating committees, consisting of academic staffs from all relevant departments. Each topic was taught using either lecture or non-lecture methods. Non-lecture methods included laboratory, practicum of physical examination (basic clinical skills), small group discussion, large group discussion, and flipped classroom with student presentation. All physiology topics in the pre-COVID-19 year were taught on-site, whereas all topics in the COVID-19 year were taught online. The transition from on-site to online physiology teaching was planned in a 3-day seminar for all academic staff in the department before the beginning of the semester. To preserve the learning objectives of each class, almost all of the online non-lecture classes were delivered with similar class activities and class materials to those used in the pre-COVID-19 year. Physical examination and clinical investigation classes were taught with live or recorded demonstrations followed by interactive discussions instead of practicums due to limitations in the availability of medical devices and peer physical examination during remote online learning. To preserve the interactivity of traditional on-site non-lecture classes, we organised the online classes using various online interactive tools, such as Zoom meeting with breakout rooms and polls, Kahoot!, Mentimeter, Google Forms, and Google Sheets. All non-lecture physiology topics, duration of the class (in hours), and teaching methods in the pre-COVID-19 year and the COVID-19 year are listed in .
Table 1. List of the non-lecture physiology topics and teaching methods in the pre-COVID-19 year (on-site teaching) and the COVID-19 year (online teaching) and their duration in hours.
Assessment of student academic achievement
Student academic achievement was assessed using summative scores. Summative examinations were scheduled every 7 weeks, as demonstrated in . The examinations of both academic years were held on-site by an iPad-based closed-book examination using the application ExamPod (Learning Innovation Center, Chulalongkorn University). The summative examination items of physiology-relevant topics in all courses were 5-option single best answer (SBA). All physiology-relevant examination items were gathered and blindly assessed by 5 medical teachers to determine whether the questions could assess students’ knowledge based on topics taught by non-lecture methods. The items agreed upon by at least 3 out of 5 medical teachers were included in this study. The numbers of included items in each course are listed in . The summative score of each student, presented as a percentage, represented an individual academic achievement.
Table 2. List of physiology classes during the pre-COVID-19 and the COVID-19 years and their numbers of summative items included in the study.
Item analysis
To confirm the equivalent standard between included summative items of the two academic years, we analysed percent-correct value, discrimination index, and minimal passing level. Percent-correct value (p-value, so-called the difficulty index) of each item was calculated from a proportion of students who chose the correct answer to the total number of responders [Citation14]. We categorised each item into three groups according to the range of the p-value. The items with a p-value of <0.3, 0.3–0.7, and >0.7 were considered difficult, moderate, and easy items, respectively [Citation15].
Discrimination index (DI) was calculated by the difference in the correctness-to-total score ratio of the highest 27th percentile group and the lowest 27th percentile group of the students in each academic year [Citation14]. Ranging from −1 to 1, DI indicates the discrimination power of each item. We categorised items, depending on their discrimination index, into four groups. The items with a DI of <0.15, 0.15–0.24, 0.25–0.34, and >0.35 were considered poor, marginal, good, and excellent items, respectively [Citation15].
Minimal passing level was calculated by the modified Ebel method. All questions were pooled, randomly ordered, and blindly assessed by 7 academic faculties. Each question was assessed for relevance (essential, important, acceptable, or questionable) and easiness (easy, medium, or hard). The expected correct percentage of each question was categorised into a 4 × 3 relevance by easiness table and calculated from an average score from all assessors [Citation16]. An average of the expected correct percentage of all questions indicated the minimal passing level of the examination.
Class evaluation
At the end of each non-lecture class in the COVID-19 year, a class evaluation form was sent to students using Google Forms comprising two major parts. The first part was the formative assessment which consisted of 10 true/false questions to assess students’ understanding of the topic. The second part of the form was used to assess students’ opinions on whether they achieved the class objectives (a yes/no question), which were informed at the beginning of each class and stated on the first page of the class material. Immediately after submitting the form, students received their formative scores, and the correct answers with explanations of all formative questions were provided. Students were aware that the formative assessment scores did not affect their grades. Student attendance, defined and approved by the course-coordinating committee, in non-lecture classes was recorded by the number of formative evaluation forms submitted within 24 hours after the scheduled date of each class.
To evaluate if better understanding in each non-lecture class could lead to higher academic achievement in the summative exam, we performed a correlation analysis of the formative score with the summative score. The overall formative score was reported as a weighted average score, using the proportion of summative items from each course (the last column in ) as a weighting factor. Students who failed to meet the 80% attendance were not included in the correlation analysis.
Assessment of student opinions regarding online teaching
At the end of the 7th week (one-third of the semester) and the end of the 23rd week (the end of the semester) in the COVID-19 year, we launched a survey to ask students’ opinions regarding online teaching. Students anonymously and voluntarily rated the level of their interaction with peers and with teachers (lower/similar/higher) when learning non-lecture classes by online methods compared with the on-site methods they had experienced in the previous academic year (before the COVID-19 pandemic). In addition, students were asked if they preferred learning non-lecture topics by online methods compared with the on-site methods they had experienced in the previous academic year. Content validity of the survey was determined by 5 independent experts. The index of item-objective congruence was 0.93, indicating an acceptable quality of the survey [Citation17].
Ethical considerations
This study was approved by the Institutional Review Board of the Faculty of Medicine, Chulalongkorn University, Bangkok, Thailand (COA no. 900/2020), and the study protocol was carried out in compliance with the international guidelines for human research protection as the Declaration of Helsinki, the Belmont Report, and the CIOMS Guideline.
Statistical analysis
Statistical analyses were performed using Microsoft Excel (Version 2111), GraphPad Prism (version 9.3.1), and IBM SPSS (version 29.0.0.0). Categorical data were shown as frequency and percentage. Continuous data were shown as mean and standard deviation (SD). The differential distributions of p-value, DI, and student opinions on interactions and the preferred learning method between the 7th and 23rd weeks were analysed using Chi-square test. The difference in summative scores and minimal passing levels between the two academic years were analysed using unpaired t test. The standardised effect size (Cohen’s d) was calculated as the mean difference divided by the pooled SD. In addition, the difference in summative scores adjusted for possible covariates, i.e., baseline characteristics of students, between the two academic years were analysed with ANCOVA. The correlation between the weighted average formative scores and summative scores was analysed using Pearson correlation. A P value of <0.05 was considered statistically significant.
Results
There were 299 and 312 second-year medical students in the pre-COVID-19 year and the COVID-19 year enrolled in the study, respectively. Both groups showed no difference in baseline characteristics, including the baseline academic performance assessed by average summative scores of the two courses that contained physiology topics and were taught by the same on-site methods in their first-year curriculum (). Our medical faculties agreed to include a total of 126 summative items from the pre-COVID-19 year and 131 items from the COVID-19 year. The quality of the summative examinations between the two academic years was equivalent in terms of the percent-correct value and the discrimination index shown in . Also, the minimal passing level calculated by the modified Ebel method did not significantly differ between the two academic years (65.37 ± 9.21% in the pre-COVID-19 year vs. 66.05 ± 10.10% in the COVID-19 year, P = 0.20, . Student academic achievement in the non-lecture physiology topics assessed by summative scores was 4.80 ± 0.92% higher in the pre-COVID-19 year compared to scores in the COVID-19 year (71.23 ± 10.46% vs. 66.42 ± 12.13%, P < 0.01, Cohen’s d = 0.42, ). Of note, the difference in student academic achievement between the two academic years remained significant after being adjusted for baseline characteristics (mean difference 4.99 ± 0.58%, P < 0.01).
Table 3. Baseline characteristics of the students enrolled in the study.
Figure 2. Item analysis.
Figure 3. Student academic performance.
In the COVID-19 year, 303 out of 312 students (97%) attended at least 80% of all non-lecture physiology classes. An average of formative scores, the number of class attendance, and the number of students stating that they achieved the class objectives for each topic are shown in . The overall weighted average of the formative score was equal to 78.19 ± 9.22%. Pearson correlation analysis between the formative and summative scores was weakly positively correlated (r = 0.29, p < 0.01, graph not shown).
Table 4. Formative score, class attendance, and class objective achievement of each non-lecture physiology topic.
Students’ opinions on interactions with peers and with teachers and their preference between online and on-site methods at the 7th and 23rd weeks are shown in . In general, students reported that online classes resulted in a lower or similar level of interaction with peers, but a similar level of interaction with teachers, compared to the on-site teaching methods. It should be noted that the patterns of responses in the 23rd week (the end of the semester) were similar to the patterns in the 7th week. However, the distributions of answers towards lower levels of student-student and student-teacher interactions were potentiated in the 23rd week (P < 0.01, ). Interestingly, students reported that they preferred online learning to on-site learning, although they reported lower levels of interactions.
Table 5. Student opinions on the interactions and learning method preference.
Discussion
Teaching physiology during the COVID-19 pandemic has been challenging due to the restrictions that limit face-to-face interactions between teachers and students [Citation5]. In this study, we found that the effectiveness of teaching non-lecture physiology topics by online methods was apparently lower than teaching by on-site methods, as indicated by the lower overall academic performance in the COVID-19 year. This finding confirmed the concern of many medical teachers that online learning methods would interfere with and limit students’ academic achievement [Citation18].
Although various online interactive tools and teaching methods were utilised to preserve social interactions in learning experiences during online classes, students reported decreased levels of interactions between them and peers or teachers. It was reported that student interaction in an online class was a determinant of student perceived learning outcome and satisfaction [Citation19]. However, we cannot conclude the causal relationship between students’ perceived interaction and their academic achievement by this study design due to the anonymity of questionnaire responses. Interestingly, we found that while students admitted decreased levels of interactions during online classes, most of the students still preferred online learning. We presumed that this could be because of the benefits of online learning, i.e., self-scheduled learning, repeatability, and travel-free routine [Citation20]. Accordingly, teachers could aim to improve student interaction in online classes through various strategies, such as using flipped classrooms, breakout online conference rooms, and encouragement techniques which will ultimately improve students’ soft skills [Citation21–23].
Formative assessment with prompt feedback has generally been used in medical education. In the COVID-19 year, we used online formative assessment to help teachers assess students’ attendance and class objective achievement. As expected, we found that formative assessments could partly reflect students’ academic performance since the formative score showed a weakly positive correlation with the summative score. This finding added to its well-known benefits demonstrated in many previous studies that online formative assessment could improve students’ understanding, engagement, and self-awareness of the learning outcomes [Citation24–26]. However, developing online formative assessment with online teaching to maximise student learning success is still challenging.
Assessment of student academic achievement has also been influenced by the pandemic, leading to a rapid increase in the implementation of technology-enhanced assessment [Citation27]. For example, the examination during the COVID-19 academic year was shifted from an on-site closed-book examination to an online open-book examination [Citation10] or an online platform with remote invigilation [Citation4,Citation28]. Since it might be inaccurate to directly compare the students’ academic achievement from different modes of examination, we used the same mode of examination (on-site closed-book examination) in our study to avoid this disparity between the two academic years.
To our knowledge, this was the first prospective study that was designed prior to the beginning of the COVID-19 semester; evaluated the academic achievement of students from the two academic years with identical baseline characteristics; and compared the academic achievement in the nearly identical curriculum structure, course-coordinating committees, and mode of summative examination. Moreover, this study contained a large sample size as we collected data from an entire class of students from one of the biggest medical schools in Thailand.
A limitation of this study is that assessment of student knowledge is not the only aspect to be evaluated to fully assess the success of online teaching during the COVID-19 pandemic. Laboratory skills, clinical skills, and other soft skills were not included in this study since they could not be fully assessed by summative scores. Students’ academic achievement by online teaching in other subjects, e.g., anatomy, biochemistry, pharmacology, and pathology, might differ from this study due to a difference in the nature of the subjects [Citation29]. Other factors which were not investigated in this study could possibly affect students’ learning outcomes, so our finding of lower academic performance during the COVID-19 year might not solely be attributable to changing from on-site to online teaching. Since the COVID-19 pandemic suddenly shifted our pre-clerkship classes to online teaching, the academic staff might have insufficient time or inadequate experience in organising online classes, particularly for a large group of students like in our medical program, which might be the cause of the diminished academic performance. In addition, stress levels among medical students have been reported to be another important factor contributing to students’ academic performance [Citation30,Citation31]. However, since our study did not evaluate levels of stress and anxiety among medical students and teachers during the COVID-19 pandemic, we cannot exclude the direct effects of stress and anxiety on students’ academic performance. Hence, further studies are required to evaluate the overall effects of the pandemic on medical students’ academic performance.
Conclusion
In terms of the academic performance of medical students, online teaching for non-lecture physiology topics during the COVID-19 pandemic appeared to be less effective than the traditional (on-site) teaching methods. Although there are various advanced technologies to assist online learning for pre-clerkship medical courses, it could not completely substitute traditional on-site teaching. Moreover, since students reported a preference for online learning over on-site learning, we suggest that medical teachers should deliberately plan and utilise a variety of tools and techniques when developing online non-lecture classes to preserve the interactivity of the classes, which might overcome this gap in the academic performance of students. However, further research is still required to fully demonstrate the contributing factors to the lower academic performance influenced by online teaching.
Acknowledgments
The authors would like to thank senior medical teachers in the Department of Physiology, Faculty of Medicine, Chulalongkorn University, who had developed teaching materials and helped teach physiology in both academic years of the study, namely Emeritus Prof. Dr. Bungorn Chomdej, Emeritus Prof. Dr. Ratree Sudsuang, and Assoc. Prof. Dr. Chanvit Kotheeranurak. We also thank Dr. Tanat Tabtieang, Dr. Kornvalee Meesilpavikkai, Dr. Thachamai Smitasiri, Dr. Namthip Witayavanitkul, and Dr. Natthawan Sanguanwong for assessing the questionnaire and general feedbacks. We thank Dr. Sawitree Boonpatcharanon for the advice on statistical analysis and Asst. Prof. Dr. Danai Wangsaturaka for the feedback relating to medical education.
Disclosure statement
No potential conflict of interest was reported by the author(s).
Additional information
Funding
References
- Ahmed H, Allaf M, Elghazaly H. COVID-19 and medical education. Lancet Infect Dis. 2020;20(7):777–9.
- World Health Organization. Thailand: how a strong health system fights a pandemic [Internet]. 2020 Sep 20 cited 2022 Jan 19. Available from 2022. https://www.who.int/docs/default-source/coronaviruse/country-case-studies/thailand-c19-case-study-20-september.pdf.
- Wanigasooriya K, Beedham W, Laloo R, et al. The perceived impact of the COVID-19 pandemic on medical student education and training – an international survey. BMC Med Educ. 2021;21(1):566.
- Abdul Razzak R, Al-Shaibani T, Naguib Y. Do students effectively learn physiology through distance online instruction? Medical students’ perceptions and academic performance. Adv Physiol Educ. 2022;46(1):65–70.
- Rose S. Medical student education in the time of COVID-19. JAMA. 2020;323(21):2131–2132.
- Casotti G, Rieser-Danner L, Knabb M. Successful implementation of inquiry-based physiology laboratories in undergraduate major and nonmajor courses. Adv Physiol Educ. 2009;32:286–296.
- Ji M, Luo Z, Feng D, et al. Short- and long-term influences of flipped classroom teaching in physiology course on medical students’ learning effectiveness. Front Public Health. 2022;10:835810.
- Anderson LC, Krichbaum KE. Best practices for learning physiology: combining classroom and online methods. Adv Physiol Educ. 2017;41(3):383–389.
- Felder E, Fauler M, Geiler S. Introducing e-learning/teaching in a physiology course for medical students: acceptance by students and subjective effect on learning. Adv Physiol Educ. 2013;37(4):337–342.
- Memon I, Feroz Z, Alkushi A, et al. Switching from face-to-face to an online teaching strategy: how anatomy and physiology teaching transformed post-COVID-19 for a university preprofessional program. Adv Physiol Educ. 2021;45(3):481–485.
- Casotti G, Beneski J, Knabb M. Teaching physiology online: successful use of case studies in a graduate course. Adv Physiol Educ. 2013;37:65–69.
- Mahanta A, Sundareswaran L, Krishnan S, et al. Sharing a preliminary experience of an effective online teaching strategy for physiology practicals. Cureus. 2021;13(11):e19388.
- Kositanurit W, Vivatvakin S, Kaikaew K, et al. Asynchronous online lecture may not be an effective method in teaching cardiovascular physiology during the COVID-19 pandemic. BMC Med Educ. 2022;22(1):162.
- Item Analysis and Interpretation of Results. In: Paniagua MA, Swygert KA editors. Constructing written test questions for the basic and clinical sciences, 4th, 21–26. National Board of Medical Examiners: Philadelphia PA. 2016.
- D’Sa J, Visbal-Dionaldo M. Analysis of multiple choice questions: item difficulty, discrimination index and distractor efficiency. Int J Nurs Educ. 2017;9:109.
- Wangsaturaka D. Standard setting. in: medical education unit faculty of medicine Chulalongkorn University, editor. In: Medical education: learning evaluation and assessment. Bangkok: Chulalongkorn University Printing House; 2011. p. 152–167.
- Turner RC, Carlson L. Indexes of item-objective congruence for multidimensional items. Int J Test. 2003;3(2):163–171.
- Yusoff MSB, Hadie SNH, Mohamad I, et al. Sustainable medical teaching and learning during the COVID-19 pandemic: surviving the new normal. Malays J Med Sci. 2020;27(3):137–142.
- Baber H. Determinants of students’ perceived learning outcome and satisfaction in online learning during the pandemic of COVID-19. J Educ e-Learn Res. 2020;7(3):285–292.
- Syed S, Rastogi A, Bansal A, et al. Future of e-learning in medical education—perception, readiness, and challenges in a developing country. Front Educ. 2021;6:598309.
- AlQhtani A, AlSwedan N, Almulhim A, et al. Online versus classroom teaching for medical students during COVID-19: measuring effectiveness and satisfaction. BMC Med Educ. 2021;21(1):452.
- Hew KF, Lo CK. Flipped classroom improves student learning in health professions education: a meta-analysis. BMC Med Educ. 2018;18(1):38.
- Wut T-m, Xu J. Person-to-person interactions in online classroom settings under the impact of COVID-19: a social presence theory perspective. Asia Pac Educ Rev. 2021;22(3):371–383.
- Kibble J. Use of unsupervised online quizzes as formative assessment in a medical physiology course: effects of incentives on student participation and performance. Adv Physiol Educ. 2007;31(3):253–260.
- Marden NY, Ulman LG, Wilson FS, et al. Online feedback assessments in physiology: effects on students’ learning experiences and outcomes. Adv Physiol Educ. 2013;37(2):192–200.
- Gupta S, Ojeh N, Sa B, et al. Use of an adaptive e-Learning platform as a formative assessment tool in the cardiovascular system course component of an MBBS programme. Adv Med Educ Pract. 2020;11:989–996.
- Khan RA, Jawaid M. Technology enhanced assessment (TEA) in COVID-19 pandemic. Pak J Med Sci. 2020;36(Covid19–s4):S108–10.
- Elsalem L, Al-Azzam N, Jum’ah AA, et al. Stress and behavioral changes with remote e-exams during the COVID-19 pandemic: a cross-sectional study among undergraduates of medical sciences. Ann Med Surg (Lond). 2020;60:271–279.
- Ben-Peretz M. The impossible role of teacher educators in a changing world. J Teach Educ. 2001;52(1):48–56.
- Lampe LC, Müller-Hilke B. Mindfulness-based intervention helps preclinical medical students to contain stress, maintain mindfulness and improve academic success. BMC Med Educ. 2021;21(1):145.
- Abdulghani HM, AlKanhal AA, Mahmoud ES, et al. Stress and its effects on medical students: a cross-sectional study at a college of medicine in Saudi Arabia. J Health Popul Nutr. 2011;29(5):516–522.