Click-level Learning Analytics in an Online Medical Education Learning Platform

Abstract

Theory

Learning in digital environments allows the collection of inexpensive, fine-grained process data across a large population of learners. Intentional design of the data collection can enable iterative testing of an instructional design. In this study, we propose that across a population of learners the information from multiple choice question responses can help to identify which design features are associated with positive learner engagement. Hypothesis: We hypothesized that, within an online module that presents serial knowledge content, measures of click-level behavior will show sufficient, but variable, association with a test-measure so as to potentially guide instructional design. Method: The Aquifer online learning platform employs interactive approaches to enable effective learning of health professions content. A multidisciplinary focus group of experts identified potential learning analytic measures within an Aquifer learning module, including: hyperlinks clicked (yes/no), magnify buttons clicked (yes/no), expert advice links clicked (yes/no), and time spent on each page (seconds). Learning analytics approaches revealed which click-level data was correlated with the subsequent relevant Case MCQ. We report regression coefficients where the dependent variable is student accuracy on the Case MCQ as a general indicator of successful engagement. Results: Clicking hyperlinks, magnifying images, clicking “expert” links, and spending >100 seconds on each page were learning analytic measures and were positively correlated with Case MCQ success; rushing through pages (<20 seconds) was inversely correlated with success. Conversely, for some measures, we failed to find expected associations. Conclusions: In online learning environments, the wealth of process data available offers insights for instructional designers to iteratively hone the effectiveness of learning. Learning analytic measures of engagement can provide feedback as to which interaction elements are effective.

Keywords:

• Online learning
• learning analytics
• medical education
• radiology
• instructional design

Acknowledgments

Our thanks go out to the Aquifer (formerly MedU) Team at Dartmouth University who provided the raw dataset for this study and allowed us full academic freedom to explore where the data would lead. We also thank the members of the expert focus group: Dr. Maria Shiau, Dr. Ruth Crowe, Dr. Mary Ann Hopkins, So Young Oh and the late, wonderful, Dr. Martin Nachbar. Finally, we thank the editor and reviewers for their profound impact on the final form of this manuscript.

Disclosure statement

Declaration of interests include: funding for this investigator initiated project was provided by an unrestricted education grant from Med-U/Aquifer.

Table 1. Generalized linear model of engagement variables as they predict success on the relevant MCQ.

Additional information

Funding

This work was funded by MedU – Aquifer.