Crowdsourcing for assessment items to support adaptive learning

Abstract

Purpose: Adaptive learning requires frequent and valid assessments for learners to track progress against their goals. This study determined if multiple-choice questions (MCQs) “crowdsourced” from medical learners could meet the standards of many large-scale testing programs.

Methods: Users of a medical education app (Osmosis.org, Baltimore, MD) volunteered to submit case-based MCQs. Eleven volunteers were selected to submit MCQs targeted to second year medical students. Two hundred MCQs were subjected to duplicate review by a panel of internal medicine faculty who rated each item for relevance, content accuracy, and quality of response option explanations. A sample of 121 items was pretested on clinical subject exams completed by a national sample of U.S. medical students.

Results: Seventy-eight percent of the 200 MCQs met faculty reviewer standards based on relevance, accuracy, and quality of explanations. Of the 121 pretested MCQs, 50% met acceptable statistical criteria. The most common reasons for exclusion were that the item was too easy or had a low discrimination index.

Conclusions: Crowdsourcing can efficiently yield high-quality assessment items that meet rigorous judgmental and statistical criteria. Similar models may be adopted by students and educators to augment item pools that support adaptive learning.

Acknowledgments

We are grateful to those who submitted and reviewed items for this project. We would like to thank Kathie Rose (NBME) and Thasin Jaigirdar (Osmosis) for project coordination and Srinivas Prasad for support in item review.

Disclosure statement

The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Dr. Tackett receives salary support from Osmosis for research and scholarship. Dr. Desai receives salary support from Osmosis. Mr. Gaglani is co-founder and CEO of Osmosis.

Additional information

Notes on contributors

Sean Tackett

Sean Tackett, MD, MPH, is assistant professor of medicine, Johns Hopkins Bayview Medical Center; and research director, Osmosis, Baltimore, Maryland.

Mark Raymond

Mark Raymond, PhD, NBME, is director of research and principal assessment scientist, Research and Discovery, National Board of Medical Examiners, Philadelphia, Pennsylvania.

Rishi Desai

Rishi Desai, MD, MPH, is clinical instructor in pediatrics, Stanford University School of Medicine, Palo Alto, California; and chief medical officer, Osmosis, Baltimore, Maryland.

Steven A. Haist

Steven A. Haist, MD, MS, FACP, NBME, is vice president, Test Development Services, National Board of Medical Examiners Philadelphia, Pennsylvania.

Amy Morales

Amy Morales, MA, NBME, is director, test materials development, Test Development Services, National Board of Medical Examiners Philadelphia, Pennsylvania.

Shiv Gaglani

Shiv Gaglani, MBA, is a medical student, Johns Hopkins University School of Medicine; and co-founder and CEO, Osmosis, Baltimore, Maryland.

Stephen G. Clyman

Stephen G. Clyman, MD, MS, NBME, is senior vice president, Research and Discovery, National Board of Medical Examiners, Philadelphia, Pennsylvania.