Introduction to Medical Statistics Software Using the Flipped Classroom: A Pilot Study

Abstract

Teaching practical skills is of particular interest in the study of human medicine. With regard to medical statistics this means the use of statistical software, which may be effectively taught by a flipped classroom approach. As a pilot study, we designed and implemented an elective course on medical statistics that focused on hands-on data analysis in SAS Studio. Students independently prepared for class using materials such as pretaped asynchronous lectures, and worked on exercises during synchronous class sessions. On course evaluations completed by n $=$ 15 students out of 26, students rated their satisfaction with the course a mean of 1.3 (SD 0.6) on a scale where 1 $=$ best and 6 $=$ worst. Twelve (80%) indicated that they processed all materials provided, and 11 students (73%) rated the frequency of direct contact with the instructor as sufficient. Nearly all (14 out of 15) viewed the course as an adequate substitute for a full face-to-face course. Our results suggest that the proposed course design is well-accepted. The flipped classroom format offers high flexibility and can be implemented easily online. Our pilot data are encouraging regarding the aim of designing a prospective follow-up study which compares the flipped classroom approach to a teaching format based on attendance.

Keywords:

• Flipped classroom
• Independent learning
• Medical statistics
• SAS
• Statistical software

Introduction

A basic understanding of medical statistics is without doubt an essential skill for emerging physicians (MFT Medizinischer Fakultaetentag der Bundesrepublik Deutschland e. V., GMA Gesellschaft für Medizinische Ausbildung e.V 2015; Ärzteblatt 2019). Despite its importance, students of human medicine often have a certain apprehension of this subject because of the mathematical content (da Silva and Moura 2020). Hence, teaching biostatistics should focus on the application of statistical methods to real-world problems of data analysis in medical research in order to relieve reservations.

Practical components of teaching are characterized above all by the use of statistical software. The know-how to handle programs that enable an evaluation of scientific data is of great importance, especially for young professionals in the field of medicine; after all, in Germany the proportion of graduates with a Ph.D. is still highest in the field of medicine (Holzapfel 2017; Forschung und Lehre 2019). Hence, getting familiar with essential data analysis techniques is an indispensable skill for this particular group of students in order to manage their individual Ph.D. research projects. Planned innovations according to the “Masterplan Medizinstudium 2020” (Ärzteblatt 2019) even stipulate that all students have to draft a small scientific paper in the course of their studies. Against this background, it seems to be even more important for teaching of medical statistics to regularly implement the use of statistical software in the future, potentially using novel educational approaches (Perry et al. 2014; GAISE College Report ASA Revision Committee 2016).

When choosing software for this purpose, different aspects have to be considered and weighed against each other. Suitable for a broad use in teaching are programs that offer a sufficient spectrum of statistical methods, allow a low-threshold entry, are characterized by a good availability, and are preferably free of charge (Hayat et al. 2013; Davidson et al. 2019). Even more important is the choice of an appropriate and efficient teaching format. In courses of shared focus on both the theoretical aspects of the content as well as their technical implementation the available teaching time plays a significant role for successful learning. Our experience from former attendance courses accompanying our curricular lecture on medical statistics showed that especially technical aspects can be very time-consuming, and thus, less time is available for theory comprehension and guided practice (Muche and Babik 2008; Muche et al. 2014). To counter this problem, a course format based on a flipped classroom approach was developed and implemented. This didactic method follows the idea of an initial phase of independent learning at home followed by a subsequent phase of guided learning with the help of an instructor (O’Flaherty and Phillips 2015; Loux, Varner, and VanNatta 2016; Farmus, Cribbie, and Rotondi 2020). Successful independent learning requires a variety of “asynchronous” learning materials (e.g., scripts, instructional videos, exercises) to be provided. The flipped classroom has already proven to be useful for teaching biostatistics in various study programs (Phillips and Phillips 2016; Schwartz et al. 2016; Farmus, Cribbie, and Rotondi 2020; Kayaduman 2021), though information on its acceptance in the study of human medicine is sparse.

In order to evaluate the potential of a flipped classroom course in software-based introduction to medical statistics in terms of a pilot study, an elective course for students of human medicine was designed and implemented. We intend to use this course as an accompaniment to our curricular lecture on medical statistics in the future. In this article we would like to present the concept, its implementation and an initial acceptance evaluation. The findings from this course will enable us to further develop this course format and to plan a cluster-randomized teaching intervention study. Efficacy of the flipped classroom concept needs to be proven before this course could potentially replace the attendance course used as common standard up to now.

Material and Methods

Flipped Classroom Method

The flipped classroom describes a didactic concept that reverses the usual sequence of the teaching process (knowledge transfer in the group with subsequent individual learning time and post-processing on the basis of exercises or homework) (Walker et al. 2020). According to this teaching method, students first approach a certain topic in self-study on the basis of suitable teaching materials and then work on it accompanied by a course instructor. In this way, each student has the opportunity to capture the contents of the lecture as well as the practical aspects depending on the individual learning speed, but the lecturer still has the possibility to react to possible comprehension problems or follow-up questions within the guided group learning phase. With regard to knowledge transfer in a software course in medical statistics, in which the focus is on teaching practical skills in addition to theoretical content, the flipped classroom method appears to be a promising approach (Farmus, Cribbie, and Rotondi 2020).

Participants

The elective was offered to students of human medicine. Students either had to complete a curricular lecture on medical statistics before, or they had to attend this curricular lecture and the elective simultaneously. This was due to the fact that basic statistical terms and theoretical concepts underlying the above mentioned topics were assumed to be known, such that the focus of the elective could be the practical application of methods.

Concept of the Course

At Ulm University the elective “Introduction to Medical Statistics with SAS Studio” was designed for the course of study in human medicine and it was implemented for the first time in the winter semester 2020/21. The conception of the course included, as the most important component, teaching of practical skills on medical statistics. This was achieved through the use of a real sample dataset as a basis for both the weekly exercises as well as the mandatory homework (in teams of maximum four students) in order to fulfill the requirements of passing the elective. The data stemmed from a finalized cohort study investigating the prevalence of type II diabetes mellitus among obese children and adolescents (Wabitsch et al. 2004), and it included numerous demographic as well as clinical variables describing the disease status of the young patients.

With regard to the requirements of a statistical software discussed in the introduction, the SAS Studio application under SAS OnDemand for Academics (https://www.sas.com/en_us/software/on-demand-for-academics.html) represents a promising choice among the established software products on the market. SAS Studio provides the full range of statistical data analysis methods implemented in SAS on a web-based application. SAS Studio provides a menu-driven as well as a code-based approach to run analyses, whereas the former was focused during the presented course.

All learning materials were provided from the beginning of the elective in order to enable trained students, that is, those who already participated the mandatory lecture on medical biometry, to organize themselves more individually with respect to their practical skills. Nevertheless, in general we proposed a schedule to ideally process the content of the elective, especially regarding the particular topics to be addressed in each meeting. Information was provided on the general handling of SAS Studio (registration for SAS OnDemand for Academics, data import from other file formats, etc.), and on the sample dataset as well. Further, the topics “Descriptive Statistics,” “Event Time Analysis & Confidence Ranges,” “Statistical Hypothesis Testing” and “Correlation of Two Variables: Correlation & Regression” were presented both theoretically and practically with corresponding instructions for implementation in SAS Studio. The learning objectives for this course were that students are familiar with the most important (i) descriptive measures (e.g., frequencies, arithmetic mean, standard deviation, median, range), (ii) statistical tests (t-test, chi-square test, Mann-Whitney-U test), and (iii) association measures and techniques, respectively (Pearson and Spearman rank correlation, simple linear regression), as well as their appropriate application in particular analysis situations.

For each topic, there were classroom sessions in the sense of the flipped classroom concept. However, because of the SARS-CoV-2 pandemic these sessions were offered exclusively in the form of (synchronous) online teaching (90 min each). After completion of the above-mentioned topics, an additional attendance date was offered in order to address specific questions regarding the preparation of the homework, if necessary. During the synchronous sessions the lecturer presented the solution of every week’s exercises in SAS Studio via shared screen using the video conferencing tool BigBlueButton (www.bigbluebutton.org). Students were asked to actively participate during these sessions in interpreting the results and assessing the appropriateness of the statistical methods applied.

Learning Material

The entire course was implemented via the Moodle learning platform. All teaching materials used included both asynchronous and synchronous elements. Asynchronous teaching elements are materials which students can access at any time and from any location. In our elective, students were provided with a SAS Studio script (Buechele, Rehm, and Muche 2019) and custom-fit PDF presentations for each topic. In addition, instructional videos for implementing the practical aspects of the respective topic in SAS Studio were provided, some of which were created by ourselves (Open Broadcaster Software, www.obsproject.org) and some of which were already available from SAS. Furthermore, an optional multiple choice-based quiz was offered for all topics in order to give the students an opportunity to independently check their learning success. The classroom sessions as a synchronous teaching element were realized online via the web conferencing system BigBlueButton, which was also integrated into Moodle. In addition, all course participants had access to a forum on Moodle, which was used in particular to discuss questions of which the answers contained potentially helpful information for all students. The assessment in this elective was done by writing a homework paper in groups of maximum four students. For this purpose, predefined scientific questions had to be addressed on the basis of the dataset already used for the exercises.

Evaluation and Statistical Analysis

All participants of the elective were asked for an evaluation after the course ended. The aim was to obtain initial indications of satisfactory aspects of the newly designed elective or aspects in need of improvement. The survey therefore, focused in particular on satisfaction with regard to the technical implementation, the teaching materials provided and their use, and the comprehensibility of the teaching content (Table 2). For this, students were asked to rate their compliance with various statements concerning the elective (e.g., “the course was well organized”) using a five point scale ranging from 1$=$ “I completely disagree” to 5 $=$ “I fully agree.” An overall evaluation of the course was requested by giving a German school grade (1 $=$ best to 6 $=$ worst). In addition, individual criticism could be expressed in free text format. A reminder was sent by email two times after the initial invitation to evaluate the elective. The survey was conducted electronically with complete anonymity using the EFS Survey program (www.unipark.com) from Questback GmbH. Categorical characteristics were described using frequencies and percentages. For continuous parameters, the arithmetic mean, standard deviation (SD), and median were calculated. Spearman’s Rho was calculated for correlation analyses. All analyses were performed using SAS Studio.

Table 2 Quantitative results of acceptance evaluation.

Item	$--$	–	0	+	$++$	n.r.
Course schedule was reasonable	0 (0%)	0 (0%)	1 (7%)	2 (13%)	12 (80%)	0 (0%)
Learning material well available	0 (0%)	0 (0%)	0 (0%)	0 (0%)	15 (100%)	0 (0%)
Sufficient contact to the lecturer	0 (0%)	1 (7%)	3 (20%)	5 (33%)	6 (40%)	0 (0%)
Substituted a seminar in presence very well	0 (0%)	0 (0%)	0 (0%)	5 (33%)	9 (60%)	1 (7%)
Well organized	0 (0%)	0 (0%)	0 (0%)	4 (27%)	11 (73%)	0 (0%)
Common theme was identifiable	0 (0%)	0 (0%)	1 (7%)	3 (20%)	11 (73%)	0 (0%)
Learning success is important for lecturers	0 (0%)	0 (0%)	0 (0%)	3 (20%)	12 (80%)	0 (0%)
Students got motivated	0 (0%)	0 (0%)	3 (20%)	2 (13%)	10 (67%)	0 (0%)
Demands were adequate	0 (0%)	0 (0%)	1 (7%)	5 (33%)	9 (60%)	0 (0%)
Learning goals were transparent	0 (0%)	0 (0%)	0 (0%)	5 (33%)	10 (67%)	0 (0%)
Relevance was clear	0 (0%)	1 (7%)	2 (13%)	1 (7%)	11 (73%)	0 (0%)
Perceived learning success	0 (0%)	1 (7%)	1 (7%)	4 (27%)	9 (60%)	0 (0%)
Contents were well imparted	0 (0%)	0 (0%)	1 (7%)	3 (20%)	11 (73%)	0 (0%)
Lecture was vivid	0 (0%)	0 (0%)	1 (7%)	3 (20%)	11 (73%)	0 (0%)

NOTE: $--$: not the case; –: rather not the case; 0: neutral; +: rather the case; $++$: definitely the case; n.r.: not relevant.

Results

Sample Description and Qualitative Part

Twenty-six students participated in the elective, but only 15 (58%) participated in the acceptance evaluation (14 females, 12 males). There were 13 students of the seventh semester who attended the elective parallel to the curricular compulsory course Q1 and two students of the ninth semester.

Of 15 students, 12 (80%) indicated that they paid attention to the entire learning material provided, though without stating to which extent specifically. The qualitative evaluation of the course based on free-text feedback gave a very positive feedback, in particular the overall organization and structure of the elective was praised (Table 1). In addition, the learning tools provided, such as videos and the quiz for self-assessment of learning comprehension, were described as very helpful. Two students of the seventh semester would have wished for a better coordination with the compulsory Q1 lecture.

Table 1 Free-text feedback* of acceptance evaluation.

Item	Feedback
Positive aspects of the online lecture	The lecturers spent a lot of care to design this elective. Good communication, well organized. I liked it that all materials were provided already at the start of the course, so I was able to deal with the contents whenever I liked. Even people without any experience in programming were able to attend. This elective will help me analyzing the data for my doctoral thesis on my own. The synchronous sessions (using shared screen presentation of the exercises’ solutions) were very helpful. The instructional videos on SAS Studio are perfectly helpful. Also, live sessions were helpful. Overall, I learned a lot in this elective and would recommend it to others. Well-organized, useful script.
Negative aspects of the online lecture	Lecture on Medical Statistics and this elective need to be better synchronized. Contents of the accompanying lecture and the elective sometimes not synchronous. Nothing!
Which learning materials were most helpful?	Script and BigBlueButton meetings. BigBlueButton meetings. Instructional videos and script. BigBlueButton meetings. Videos, script and BigBlueButton meetings (actually everything). PDF presentations including multiple choice questions.
Suggestions for improvement	Synchronize lecture and elective such that the topics of the elective follow those of the lecture, not vice versa. This would improve learning. Elective should be scheduled after the lecture. I would have used SPSS instead of SAS, but only because most clinicians suggest SPSS, not because I am personally preferring it.

NOTE: ${}^{*}$Free-text feedback has been translated (German to English) and summarized occasionally (without changing the initial sense though).

Evaluation of the Elective

Overall, the course was rated very positively with regard to the aspects of organization and structure, teaching commitment of the lecturers, learning objective and teaching content, as well as didactic implementation (Table 2, all of these measured using a 5 point Likert scale). On average, the elective was rated 1.3 (SD 0.6) in terms of an overall grade (1 $=$ best, 6 $=$ worst). Eleven students (73%) indicated that the frequency of direct contact with the instructor was sufficient. There was a moderate, positive relationship between subjectively perceived learning success and the percentage of course material utilized by the students (rho $=$ 0.495). Almost all (14 of 15) viewed the course as an adequate substitute to a face-to-face course due to the format provided. The quizzes provided for self-monitoring were perceived as less helpful than the other instructional materials, especially when compared to the instructional videos and BigBlueButton events.

Consequently, more than 80% of the students (13/15) answered “no action required” when being asked if there is a need for redesigning or further developing the elective. This assessment related in particular to the structural/organizational, didactic and content aspects.

Discussion

The flipped classroom offers both students and teachers a very attractive and flexible approach of teaching. In terms of design freedom, it is obviously superior to the classic teaching approach (knowledge transfer in plenary sessions followed by individual post-processing) (Ramnanan and Pound 2017; Hew and Lo 2018). The advantages for students are that they can learn the topics of a course independently and at their own pace. At the same time, however, the lecturer has the opportunity to support the students with questions and to implement guided learning through an appropriate overall conception of the course. The overall conception (teaching material, implementation of the guided, synchronous teaching phase) can be adapted to the specific teaching and learning environment. Especially in times of the current Covid-19 pandemic, for example, the contact between students and lecturers is more difficult and limited to online teaching, yet the additional use of video conferencing software enables an adequate exchange between lecturers and students. This was also evident in this way during our acceptance evaluation.

The basic prerequisite for successful implementation and broad acceptance of the flipped classroom approach in teaching is, of course, careful planning of the course with asynchronous and guided learning phases, as well as the provision of an extensive portfolio of teaching materials (Puppe and Nelson 2019). A problem with this teaching approach can be the fact that not all students may have the same standard of technical equipment. In addition, the learning success of the flipped classroom approach relies heavily on the students’ own initiative and motivation, which may need to be supported by additional measures on the part of the lecturers. The acceptance evaluation of our pilot study showed that the students predominantly regarded the elective course as a good substitute for a face-to-face course. In general, evaluation of our pilot data suggests that the positive ratings which have been reported in other fields of study when applying a flipped classroom to biostatistics (Phillips and Phillips 2016; Farmus, Cribbie, and Rotondi 2020; Kayaduman 2021) also pertain for students of human medicine.

Planning a Controlled, Cluster-Randomised Study

To date our curricular lecture on medical statistics is accompanied by an attendance seminar in groups of a maximum of 24 students. However, only a small proportion of all students (approximately 30%) have the opportunity to attend one of the seminar groups in which a statistical software is used to work on the provided exercises. In these attendance courses the SPSS software has been used for years, according to the classical teaching approach. Our evaluation results of this pilot study on the applicability of the flipped classroom approach to the teaching of medical statistics using SAS Studio demonstrated that it is well-accepted among students. Thus, it seems reasonable to evaluate the effectiveness of the flipped classroom course in comparison to the established seminar format (attendance course using SPSS) in the course of a subsequent prospective teaching research study. However, self-rated perception of learning may not be used as a primary endpoint of such an effectiveness study, since results may be biased due to subjective experiences. In light of available data, the more objective endpoint of learning success should be used instead, which is measured by means of students’ final marks. Since the distribution of the final marks already achieved a very high level (mean grade 2.0 (SD 0.8) according to the German grading system with 1 $=$ best and 6 $=$ worst) in the attendance seminar, the goal of a proof of superiority does not seem to be feasible. For this reason, the required number of students for a comparative, prospective, cluster-randomised study will be estimated in a noninferiority setting.

Since the planned study will be an evaluation based on clustered data (students in different courses of 20 each), sample size estimation should be done according to the requirements for cluster-randomized trials (Dreyhaupt et al. 2017). As a primary endpoint the students’ final marks during examination shall be used, and we assume noninferiority of the presented flipped classroom approach using SAS Studio when compared to the established attendance seminar using SPSS, that is, students’ mean grade in the flipped classroom group should be equal to 2.0 (SD 0.8) or better according to the information given above. It seems reasonable to use a noninferiority bound of 0.4 points here, since this would mean for students to still finish the course with the overall mark “good” (according to the German grading system). Assuming a power of 80%, a one-sided significance level of 2.5%, and a design effect (definition see below) of 1.95 to adjust the number of cases for the cluster structure of the data, the required number of students will not exceed a total of N $=$ 100 assuming that the mean grading of the flipped classroom students will not be worse than 2.0 (Chow, Shao, and Wang 2008). For the design effect $D_{\text{eff}}=1+\rho \mathrm{ }•(1+\mathrm{m})$ with $m=$ cluster size-1, a comparatively small intra-cluster correlation coefficient ρ of 0.05 was assumed based on the empirical values from the past seminars, that is, only 5% of the data variance can be explained by cluster membership. The calculated numbers could easily be included within one semester having 25 students per group, that is, four PC-based seminar groups of which two will use the flipped classroom approach and two will use the attendance format. In order to ensure an unbiased estimation of the learning effect owing to the flipped classroom intervention, of course the provided learning material has to be harmonized among the two lecture formats. Currently, not all features used for the flipped classroom approach are implemented in the attendance courses, for example, instructional SPSS videos would be additionally required.

Limitations

Due to the low number of 26 participants in general the results of the acceptance evaluation presented here are to be interpreted in terms of an uncontrolled pilot study. Results interpretation is even more limited because of the low response rate of only 58% (15 of 26), which might introduced some bias regarding the general positivity of evaluation findings. Furthermore, all courses at our university within this period were offered in a remote setting only because of the Covid-19 pandemic, so the very positive feedback on our course might be inflated to some extent simply because of the fact that the flipped classroom approach was rated superior to traditional classroom approaches. For a more comprehensive evaluation of the flipped classroom concept in teaching medical statistics, comparative studies must be implemented and can be based on the model planning mentioned above. However, this is also subject to certain restrictions. For example, the preliminary data used there on learning success in the regular seminar cannot be attributed exclusively to the PC-based (SPSS) seminar. The choice of statistical software used could have an effect on the evaluation of the flipped classroom approach if software other than SAS Studio was judged to be more practical. Also, ratings from earlier courses which were taught in-person may not be compared directly to the ones from the flipped classroom course which was taught exclusively online due to the COVID-19 pandemic. Furthermore, the usage of students’ final marks as a measure of learning success has been critically discussed due to a lack of external validity (Zieffler et al. 2008). Future acceptance evaluations should therefore, explicitly include an objective assessment of both learning success as well as satisfaction with the software used. Further, some of the features assessed during our evaluation might be assessed more precisely (e.g., extent of utilization of particular learning materials) in future assessments and could also be extended to an evaluation of the mode of examination, which is intended to be done by means of an electronic open book exam.

Outlook and Summary

Based on the pilot data presented here, the flipped classroom approach can be considered a promising teaching approach in medical statistics. A comprehensive evaluation will take place as part of a cluster-randomised study.