What have we learned about constructed response short-answer questions from students and faculty? A multi-institutional study

References

• Bierer SB, Taylor CA, Dannefer EF. 2009. Evaluation of essay questions used to assess medical students’ application and integration of basic and clinical science knowledge. Teach Learn Med. 21(4):344–350. doi:10.1080/10401330903230980.
   PubMed Web of Science ®Google Scholar
• Bird JB, Olvet DM, Willey JM, Brenner J. 2019. Patients don’t come with multiple choice options: essay-based assessment in UME. Med Educ Online. 24(1):1649959. doi:10.1080/10872981.2019.1649959.
   PubMed Web of Science ®Google Scholar
• Boscardin CK, Earnest G, Hauer KE. 2020. Predicting performance on clerkship examinations and USMLE step 1: what is the value of open-ended question examination? Acad Med. 95:S109–S113. doi:10.1097/ACM.0000000000003629.
   PubMed Web of Science ®Google Scholar
• Brauer DG, Ferguson KJ. 2015. The integrated curriculum in medical education: AMEE Guide No. 96. Med Teach. 37(4):312–322. doi:10.3109/0142159X.2014.970998.
   PubMed Web of Science ®Google Scholar
• Cannings R, Hawthorne K, Hood K, Houston H. 2005. Putting double marking to the test: a framework to assess if it is worth the trouble. Med Educ. 39(3):299–308. doi:10.1111/j.1365-2929.2005.02093.x.
   PubMed Web of Science ®Google Scholar
• Case SM, Swanson DB. 1993. Extended-matching items: a practical alternative to free-response questions. Teach Learn Med. 5(2):107–115. doi:10.1080/10401339309539601.
   Google Scholar
• Frambach JM, van der Vleuten CP, Durning SJ. 2013. AM last page. Quality criteria in qualitative and quantitative research. Acad Med. 88(4):552.
   PubMedGoogle Scholar
• Ginzburg S, Brenner J, Willey J. 2015. Integration: a strategy for turning knowledge into action. MedSciEduc. 25(4):533–543. doi:10.1007/s40670-015-0174-y.
   Google Scholar
• Gulikers JTM, Bastiaens TJ, Kirschner PA. 2004. A five-dimensional rramework for authentic assessment. ETR&D. 52(3):67–86. doi:10.1007/BF02504676.
   Web of Science ®Google Scholar
• Hanson JL, Balmer DF, Giardino AP. 2011. Qualitative research methods for medical educators. Acad Pediatr. 11(5):375–386. doi:10.1016/j.acap.2011.05.001.
   PubMed Web of Science ®Google Scholar
• Hattie J, Timperley H. 2007. The power of feedback. Rev Educ Res. 77(1):81–112. doi:10.3102/003465430298487.
   Web of Science ®Google Scholar
• Hauer KE, Boscardin C, Brenner JM, van Schaik SM, Papp KK. 2020. Twelve tips for assessing medical knowledge with open-ended questions: designing constructed response examinations in medical education. Med Teach. 42(8):880–885. doi:10.1080/0142159X.2019.1629404.
   PubMed Web of Science ®Google Scholar
• Hodges B. 2013. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 35(7):564–568. doi:10.3109/0142159X.2013.789134.
   PubMed Web of Science ®Google Scholar
• Hsieh HF, Shannon SE. 2005. Three approaches to qualitative content analysis. Qual Health Res. 15(9):1277–1288. doi:10.1177/1049732305276687.
   PubMed Web of Science ®Google Scholar
• Hubbard JK, Potts MA, Couch BA. 2017. How question types reveal student thinking: an experimental comparison of multiple-true-false and free-response formats. CBE Life Sci Educ. 16(2):1–13. doi:10.1187/cbe.16-12-0339.
   Web of Science ®Google Scholar
• Ji YA, Nam SJ, Kim HG, Lee J, Lee SK. 2018. Research topics and trends in medical education by social network analysis. BMC Med Educ. 18(1):222. doi:10.1186/s12909-018-1323-y.
   PubMedGoogle Scholar
• Kang SHK, McDermott KB, Roediger HL. III 2007. Test format and corrective feedback modify the effect of testing on long-term retention. Eur J Cognitive Psychol. 19(4-5):528–558. doi:10.1080/09541440601056620.
   Web of Science ®Google Scholar
• Khan JS, Mukhtar O, Tabasum S, Shaheen N, Farooq M, Irfan MA, Sattar A, Nabeel M, Imran M, Rafique S, et al. 2010. Relationship of awards in multiple choice questions and structured answer questions in the undergraduate years and their effectiveness in evaluation. J Ayub Med Coll Abbottabad. 22(2):191–195.
   PubMedGoogle Scholar
• King A, Hoppe RB. 2013. Best practice" for patient-centered communication: a narrative review. J Grad Med Educ. 5(3):385–393. doi:10.4300/JGME-D-13-00072.1.
   PubMedGoogle Scholar
• Kogan JR, Hauer KE. 2020. Sparking change: how a shift to step 1 pass/fail scoring could promote the educational and catalytic effects of assessment in medical education. Acad Med. 95(9):1315–1317. doi:10.1097/ACM.0000000000003515.
   PubMed Web of Science ®Google Scholar
• Konopasek L, Norcini J, Krupat E. 2016. Focusing on the formative: building an assessment system aimed at student growth and development. Acad Med. 91(11):1492–1497. doi:10.1097/ACM.0000000000001171.
   PubMed Web of Science ®Google Scholar
• Larsen DP, Butler AC, Roediger HL. 3rd. 2008. Test-enhanced learning in medical education. Med Educ. 42(10):959–966. doi:10.1111/j.1365-2923.2008.03124.x.
   PubMed Web of Science ®Google Scholar
• Lupi CS, Tempest HG, Ward-Peterson M, Ory SJ. 2018. The educational effects of a summative diagnostic reasoning examination among second-year medical students. MedSciEduc. 28(4):667–673. doi:10.1007/s40670-018-0610-x.
   Google Scholar
• Malik AS, Malik RH. 2011. Twelve tips for developing an integrated curriculum. Med Teach. 33(2):99–104. doi:10.3109/0142159X.2010.507711.
   PubMed Web of Science ®Google Scholar
• Mayer RE. 2010. Applying the science of learning to medical education. Med Educ. 44(6):543–549. doi:10.1111/j.1365-2923.2010.03624.x.
   PubMed Web of Science ®Google Scholar
• McDaniel MA, Roediger HL, 3rd, McDermott KB. 2007. Generalising test-enhanced learning from the laboratory to the classroom. Psychon Bull Rev. 14(2):200–206. doi:10.3758/bf03194052.
   PubMed Web of Science ®Google Scholar
• Melovitz Vasan CA, DeFouw DO, Holland BK, Vasan NS. 2018. Analysis of testing with multiple choice versus open-ended questions: outcome-based observations in an anatomy course. Anat Sci Educ. 11(3):254–261. doi:10.1002/ase.1739.
   PubMed Web of Science ®Google Scholar
• National Board of Medical Examiners. 2018. New information about new item formats on the medicine clinical science subject examination. [accessed January 23 2023]. https://www.nbme.org/sites/default/files/2020-01/NBME%20Updates%20New%20Item%20Formats%20on%20Medicine%20Exam.pdf.
   Google Scholar
• Norcini J, Anderson B, Bollela V, Burch V, Costa MJ, Duvivier R, Galbraith R, Hays R, Kent A, Perrott V, et al. 2011. Criteria for good assessment: consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach. 33(3):206–214. doi:10.3109/0142159X.2011.551559.
   PubMed Web of Science ®Google Scholar
• Norman GR, Smith EK, Powles AC, Rooney PJ, Henry NL, Dodd PE. 1987. Factors underlying performance on written tests of knowledge. Med Educ. 21(4):297–304. doi:10.1111/j.1365-2923.1987.tb00367.x.
   PubMed Web of Science ®Google Scholar
• Olvet DM, Bird JB, Fulton TB, Kruidering M, Papp KK, Qua K, Willey JM, Brenner JM. 2022. A Multi-institutional study of the feasibility and reliability of the implementation of constructed response exam questions. Teach Learn Med. doi:10.1080/10401334.2022.2111571.
   Web of Science ®Google Scholar
• Palmer EJ, Duggan P, Devitt PG, Russell R. 2010. The modified essay question: its exit from the exit examination? Med Teach. 32(7):e300-307–e307. doi:10.3109/0142159X.2010.488705.
   Web of Science ®Google Scholar
• Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. 2009. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health. 36(1):24–34. doi:10.1007/s10488-008-0197-4.
   PubMed Web of Science ®Google Scholar
• Roediger 3rd, HL, Butler AC. 2011. The critical role of retrieval practice in long-term retention. Trends Cogn Sci. 15(1):20–27. doi:10.1016/j.tics.2010.09.003.
   PubMed Web of Science ®Google Scholar
• Roediger III, Henry L., Adam L. Putnam, and Megan A. Smith. 2011. Ten benefits of testing and their applications to educational practice. Psychol Learn Motiv. 55:1–36.
   Web of Science ®Google Scholar
• Rotgans JI. 2012. The themes, institutions, and people of medical education research 1988-2010: content analysis of abstracts from six journals. Adv Health Sci Educ Theory Pract. 17(4):515–527. doi:10.1007/s10459-011-9328-x.
   PubMed Web of Science ®Google Scholar
• Rudolph MJ, Daugherty KK, Ray ME, Shuford VP, Lebovitz L, DiVall MV. 2019. Best practices related to examination item construction and post-hoc review. Am J Pharm Educ. 83(7):7204. doi:10.5688/ajpe7204.
   PubMed Web of Science ®Google Scholar
• Sam AH, Field SM, Collares CF, van der Vleuten CPM, Wass VJ, Melville C, Harris J, Meeran K. 2018. Very-short-answer questions: reliability, discrimination and acceptability. Med Educ. 52(4):447–455. doi:10.1111/medu.13504.
   PubMed Web of Science ®Google Scholar
• Schuwirth LW, van der Vleuten CP. 2004a. Changing education, changing assessment, changing research? Med Educ. 38(8):805–812. doi:10.1111/j.1365-2929.2004.01851.x.
   PubMed Web of Science ®Google Scholar
• Schuwirth LW, van der Vleuten CP. 2004b. Different written assessment methods: what can be said about their strengths and weaknesses? Med Educ. 38(9):974–979. doi:10.1111/j.1365-2929.2004.01916.x.
   PubMed Web of Science ®Google Scholar
• Scouller K. 1998. The influence of assessment method on students’ learning approaches: mulitple choice question examination versus assignment essay. Higher Educ. 35(4):453–472. doi:10.1023/A:1003196224280.
   Google Scholar
• Scudder DR, Sherry AD, Jarrett RT, Fernando S, Kuhn AW, Fleming AE. 2019. Fundamental curriculum change with 1-year pre-clerkship phase and effect on stress associated with residency specialty selection. Med Sci Educ. 29(4):1033–1042. doi:10.1007/s40670-019-00800-7.
   PubMedGoogle Scholar
• Shilo G. 2015. Formulating good open-ended questions in assessment. Educ Res Q. 38(4):3–30.
   Google Scholar
• Stranger-Hall KF. 2012. Multiple-choice exams: an obstacle for higher-level thinking in introductory science classes. CBE Life Sci Educ. 11(3):294–306.
   PubMed Web of Science ®Google Scholar
• Sumeracki MA, Weinstein Y. 2018. Six strategies for effective learning. Acad Med. 93(4):666. doi:10.1097/ACM.0000000000002091.
   PubMed Web of Science ®Google Scholar
• United States Medical Licensing Examination. 2021a. USMLE step 1 transition to pass/fail only score reporting.
   Google Scholar
• United States Medical Licensing Examination. 2021b. Work to relaunch USMLE Step 2 CS discontinued. [accessed 2023 January 18]. https://www.usmle.org/work-relaunch-usmle-step-2-cs-discontinued.
   Google Scholar
• Wainer H, Thissen D. 1993. Combining multiple-choice and constructed response test scores: toward a Marxist Theory of test construction. Applied Measurement in Education. 6(2):103–118. doi:10.1207/s15324818ame0602_1.
   Google Scholar
• Wlodarczyk S, Muller-Juge V, Hauer KE, Tong MS, Ransohoff A, Boscardin C. 2021. Assessment to optimize learning strategies: a qualitative study of student and faculty perceptions. Teach Learn Med. 33(3):245–257.
   PubMed Web of Science ®Google Scholar
• Yedidia MJ, Gillespie CC, Kachur E, Schwartz MD, Ockene J, Chepaitis AE, Snyder CW, Lazare A, Lipkin M. Jr. 2003. Effect of communications training on medical student performance. JAMA. 290(9):1157–1165. doi:10.1001/jama.290.9.1157.
   PubMed Web of Science ®Google Scholar
• Yudkowsky R, Szauter K. 2021. Farewell to the step 2 clinical skills exam: new opportunities, obligations, and next steps. Acad Med. 96(9):1250–1253. doi:10.1097/ACM.0000000000004209.
   PubMed Web of Science ®Google Scholar