“Yes, and …” Exploring the Future of Learning Analytics in Medical Education

ABSTRACT

This Conversations Starter article presents a selected research abstract from the 2017 Association of American Medical Colleges Northeastern Region Group on Educational Affairs annual spring meeting. The abstract is paired with the integrative commentary of three experts who shared their thoughts stimulated by the study. Commentators brainstormed “what's next” with learning analytics in medical education, including advancements in interaction metrics and the use of interactivity analysis to deepen understanding of perceptual, cognitive, and social learning and transfer processes.

AAMC NEGEA 2017 ABSTRACT

Click-Level Learning Analytics in an Online Medical Education Learning Platform

Matt M. Cirigliano^a, Charlie Guthrie^b, and Martin V. Pusic^c

^aEducational Communications and Technology Program, NYU Steinhardt, New York, New York, USA; ^bGraduate School of Arts and Sciences, New York University, New York, New York, USA; ^cInstitute for Innovations in Medical Education, New York University School of Medicine, New York, New York, USA

Phenomenon: Students engaged with virtual patients and expository multimedia can interact with educational content in a number of ways. The nature of these interactions can reveal important information about both the media and the learning, particularly when documented in the detail provided by digital environments.¹ Through big data and learning analytics approaches we can now explore the click-level tendencies and handling methods employed by a large population of medical students using online learning platforms like MedU, an online suite of modules teaching patient-centered approaches to clinical problem-solving skills.²

Approach: Using measures of student interaction, we sought to identify patterns that predict downstream assessment performance in a linear online module.³ We selected a MedU expository online module on the topic of musculoskeletal radiology. The module consisted of six sections, each of which had a set of four to five screens, with a variety of engagement activities, followed by a multiple-choice question (MCQ). We convened a multidisciplinary focus group of experts to identify potential learning analytic measures within the MedU system. These included hyperlinks clicked on a page (yes/no), magnify buttons clicked (yes/no), expert advice links clicked (yes/no), and time on page (seconds). Each unit's engagement activity data were correlated with the (single) succeeding, relevant MCQ. These correlations were later compared with a single expert's ratings for relevance of content to the subsequent MCQ (rating dichotomously expressed as “fully relevant: yes/no”).

Findings: We obtained click-level data describing usage of the module from July 1, 2014, to May 5, 2015, encompassing the experiences of 2,806 North American medical students. All six MCQ items showed acceptable item-total correlations. A number of the interaction behaviors were significantly correlated with likelihood that a student correctly answered the subsequent MCQ. Clicking hyperlinks (odds ratio [OR] = 1.21), 95% confidence interval (CI) [1.12, 1.31]; magnifying images (OR = 1.20), 95% CI [1.11, 1.31]; referring to the expert's answers (OR = 1.21), 95% CI [1.05, 1.39]; and spending more than 100 seconds on each instructional page (OR = 1.38), 95% CI [1.27, 1.51] all were correlated with correct MCQ answers. Rushing through the pages (< 20 seconds) was inversely correlated (OR = 0.74), 95% CI [0.66, 0.83]. For each assessment question, a unique logistic regression model could be constructed to indicate which activities/interactions would result in correctly answering the MCQ. For the five hyperlinks judged “completely relevant” by the expert, only one was in fact statistically significantly related to the subsequent MCQ. Similar ratios for the magnify (3/7) and time (5/8) interaction measures were observed.

Insights: Our intention was to demonstrate the merits of learning analytics within the online context, giving educators a new tool for improving experiences in online learning environments. Results of this analysis, where the data from thousands of learners are summarized, can serve as feedback to instructional designers as to which interaction elements are effective. It may also be useful to show students evidence that there is a statistically significant relationship between engaging with the material and performing well on assessments.

Additional information

Notes on contributors

Anna T. Cianciolo

Anna T. Cianciolo, Ph.D., is Associate Professor, Department of Medical Education at Southern Illinois University School of Medicine and Editor of Teaching and Learning in Medicine. Her research focuses on understanding and improving the performance of individuals, teams, and groups as they seek to solve the complex problems of academic health care, including diagnosis, collaborative learning, and clinical teaching and supervision.

Jennifer E. Lim-Dunham

Jennifer E. Lim-Dunham, M.D., FACR, is Professor of Radiology, Pediatrics and Medical Education at Loyola University Chicago Stritch School of Medicine. Her clinical practice focuses on ultrasound and pediatric radiology. She serves as course director of the Vertical Curriculum in Radiology at Stritch and pursues research in medical student online learning in radiology.

Anderson Spickard

Anderson Spickard III, M.D., M.S., is Assistant Dean of Education Design and Technology and Associate Professor of Medicine and Biomedical Informatics at the Vanderbilt School of Medicine. He practices Internal Medicine and serves as a Master Clinical Teacher in the medical school. His research interests include all aspects of medical education with a focus on the organization, design, and application of technology to support the medical education mission.

Valerie Terry

Valerie Terry, MPAff, Ph.D., is at the University of Texas–Rio Grande Valley School of Medicine with roles and responsibilities in a number of areas involving undergraduate and graduate curriculum design and implementation, instructional delivery and evaluation, including the use of educational technology, and faculty instruction and development. Her educational and research interests encompass the discipline of communication, health care public policy, and health literacy, with an emphasis on rhetorical theories as they inform symbolic constructions of social and political contexts. Dr. Terry is currently designing, developing, and teaching comprehensive undergraduate and graduate Communication curriculum, partnering with a child psychiatrist faculty colleague.