What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study

Abstract

Purpose

The purpose of this study was to enrich understanding about the perceived benefits and drawbacks of constructed response short-answer questions (CR-SAQs) in preclerkship assessment using Norcini’s criteria for good assessment as a framework.

Methods

This multi-institutional study surveyed students and faculty at three institutions. A survey using Likert scale and open-ended questions was developed to evaluate faculty and student perceptions of CR-SAQs using the criteria of good assessment to determine the benefits and drawbacks. Descriptive statistics and Chi-square analyses are presented, and open responses were analyzed using directed content analysis to describe benefits and drawbacks of CR-SAQs.

Results

A total of 260 students (19%) and 57 faculty (48%) completed the survey. Students and faculty report that the benefits of CR-SAQs are authenticity, deeper learning (educational effect), and receiving feedback (catalytic effect). Drawbacks included feasibility, construct validity, and scoring reproducibility. Students and faculty found CR-SAQs to be both acceptable (can show your reasoning, partial credit) and unacceptable (stressful, not USMLE format).

Conclusions

CR-SAQs are a method of aligning innovative curricula with assessment and could enrich the assessment toolkit for medical educators.

Keywords:

• assessment
• undergraduate medical education
• open-ended questions
• medical knowledge
• preclerkship

Acknowledgments

The authors would like to thank Krista Paxton and Ayomide Johnson for their instrumental contributions in the preparation of this manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The author(s) reported there is no funding associated with the work featured in this article.

Notes on contributors

Judith M. Brenner

Judith M. Brenner, MD, is the Senior Associate Dean for Medical Education at the New York University Grossman Long Island School of Medicine, Mineola, NY.

Tracy B. Fulton

Tracy B. Fulton, PhD, is Professor of Biochemistry and Biophysics and a course director at University of California San Francisco.

Marieke Kruidering

Marieke Kruidering, PhD, is Professor in the Department of Cellular & Molecular Pharmacology and a course director at University of California San Francisco.

Jeffrey B. Bird

Jeffrey B. Bird, MA, is the Educational Research & Strategic Assessment Analyst and an assistant professor at the Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY.

Joanne Willey

Joanne M. Willey, PhD, is the Leo A. Guthart Professor of Biomedical Sciences and Chair of the Department of Science Education at the Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY.

Kelli Qua

Kelli Qua, PhD, is the Director of Evaluation and CQI Center for Medical Education at Case Western Reserve University School of Medicine.

Doreen M. Olvet

Doreen M. Olvet, PhD, is Assistant Professor and Medical Education Project Manager Department of Science Education, Donald and Barbara Zucker School of Medicine at Hofstra/Northwell, Hempstead, NY.