Twelve tips for developing key-feature questions (KFQ) for effective assessment of clinical reasoning

References

• Abdulghani HM, Ahmad F, Irshad M, Khalil MS, Al-Shaikh GK, Syed S, Aldrees AA, Alrowais N, Haque S. 2015. Faculty development programs improve the quality of Multiple Choice Questions items’ writing. Sci Rep. 5:9556.
   PubMed Web of Science ®Google Scholar
• Abdulghani HM, Irshad M, Haque S, Ahmad T, Sattar K, Salah Khalil M. 2017. Effectiveness of longitudinal faculty development programs on MCQs items writing skills: A follow-up study. Plos One. 12:e0185895.
   PubMed Web of Science ®Google Scholar
• Abozaid H, Park YS, Tekian A. 2017. Peer review improves psychometric characteristics of multiple choice questions. Med Teach. 1–5.
   Web of Science ®Google Scholar
• Accreditation Council for Graduate Medical Education (ACGME) and American Board of Pediatrics. 2012. The Pediatrics Milestone Project; [accessed 2017 Aug 4]. https://acgme.org/Portals/0/PDFs/Milestones/PediatricsMilestones.pdf.
   Google Scholar
• Alamoudi AA, El-Deek BS, Park YS, Al Shawwa LA, Tekian A. 2017. Evaluating the long-term impact of faculty development programs on MCQ item analysis. Med Teach. 39(sup1):S45–S49.
   Web of Science ®Google Scholar
• Anderson LW, Krathwohl D, editors. 2001. A taxonomy for learning, teaching, and assessing: a revision of Bloom's taxonomy of educational objectives. New York, NY: Longman Publishers.
   Google Scholar
• Armstrong P. Bloom's Taxonomy; [accessed 2017 Jul 12]. https://cft.vanderbilt.edu/guides-sub-pages/blooms-taxonomy/.
   Google Scholar
• Bloom BS. 1956. Taxonomy of educational objectives: The classification of educational goals. New York, NY: McKay.
   Google Scholar
• Bordage G, Brailovsky C, Cohen T, Page GG. 1997. Maintaining and enhancing key decision-making skills from graduation into practice: An exploratory study. Seventh Ottawa Conference on Medical Education and Assessment: Advances in Medical Education, Maastricht, The Netherlands: Kluwer Academic Publishers.
   Google Scholar
• Bordage G, Brailovsky C, Carretier H, Page G. 1995. Content validation of key features on a national examination of clinical decision-making skills. Acad Med. 70:276–281.
   PubMed Web of Science ®Google Scholar
• Bordage G, Carretier H, Bertrand R, Page G. 1995. Comparing times and performances of French- and English-speaking candidates taking a national examination of clinical decision-making skills. Acad Med. 70:359–365.
   PubMed Web of Science ®Google Scholar
• Brailovsky C, Allen T, Lawrence K, Crichton T, Laughlin T, Van der Goes T. 2014. Short answer questions based on Key Features have higher discrimination indices on a certification examination in family medicine Ottawa Conference. Ottawa, ON.
   Google Scholar
• Case SM, Swanson DB. 1998. Writing written test questions for the basic and clinical sciences. Philadelphia, PA: National Board of Medical Examiners.
   Google Scholar
• Cerutti B, Blondon K, Galetto A. 2016. Long-menu questions in computer-based assessments: a retrospective observational study. BMC Med Educ. 16:55.
   PubMed Web of Science ®Google Scholar
• Coderre S, Woloschuk W, McLaughlin K. 2009. Twelve tips for blueprinting. Med Teach. 31:322–324.
   PubMed Web of Science ®Google Scholar
• Downing S, Haladyna TA. 2006. Handbook of test development. Mahwah, NJ: Lawrence Erlbaum Assoc. Inc.
   Google Scholar
• Durning SJ, Artino AR, Schuwirth L, van der Vleuten C. 2013. Clarifying assumptions to enhance our understanding and assessment of clinical reasoning. Acad Med. 88:442–448.
   PubMed Web of Science ®Google Scholar
• Escudier M, Woolford M, Tricio J. 2018. Assessing the application of knowledge in clinical problem solving: The structured professional reasoning exercise. Eur J Dent Educ. 22:e269–e277.
   PubMed Web of Science ®Google Scholar
• Eva K. 2005. What every teacher needs to know about clinical reasoning. Med Educ. 39:98–106.
   PubMed Web of Science ®Google Scholar
• Eva KW, Wood TJ, Riddle J, Touchie C, Bordage G. 2010. How clinical features are presented matters to weaker diagnosticians. Med Educ. 44:775–785.
   PubMed Web of Science ®Google Scholar
• Farmer E. 1998. Writing key feature problems. Australia: Royal Australian College of General Practitioners. [accessed 2018 June 12]. https://www.academia.edu/1749144/Writing_Key_Features_Problems.
   Google Scholar
• Farmer EA, Hinchy J. 2005. Assessing general practice clinical decision making skills: the key feature approach. Austr Fam Phys 34:1059–1061.
   PubMedGoogle Scholar
• Farmer EA, Page G. 2005. A practical guide to assessing clinical decision-making skills using the key features approach. Med Educ. 39:1188–1194.
   PubMed Web of Science ®Google Scholar
• Fischer MR, Kopp V, Holzer M, Ruderich F, Junger J. 2005. A modified electronic key feature examination for undergraduate medical students: validation threats and opportunities. Med Teach. 27:450–455.
   PubMed Web of Science ®Google Scholar
• Frank JR, Snell LS, Sherbino J. 2014. The Draft CanMEDS 2015 Milestones Guide; [accessed 2017 Aug 4]. http://www.royalcollege.ca/portal/page/portal/rc/common/documents/canmeds/framework/canmeds_milestone_guide_sept2014_e.pdf.
   Google Scholar
• General Medical Council. 2014. Good medical practice. United Kingdom: General Medical Council.
   Google Scholar
• Glover Takahashi S, Herold J, Clark M, Nayer M, Beggs C, Corbett C, Drynan D, Cho N, Dignum T, Hudson B, Corbett K. 2012. The use of key features cases to assess clinical decision-making, CanMEDS roles & competence First Montreal Conference on Clinical Reasoning. Montreal, QC.
   Google Scholar
• Glover Takahashi S, Herold J, Clark M, Nayer M, Drynan D, Cho N, Dignum T, Corbett K, Hudson B, Hynes M. 2013. Building better written exams – The use of key features cases to assess clinical decision-making, CanMEDS roles and competence International Conference on Residency Education (ICRE). Calgary, Alberta.
   Google Scholar
• Haladyna TA, Downing S. 1993. How many options is enough for a multiple-choice teste item. Educ Psychol Meas. 53:999–1010.
   Web of Science ®Google Scholar
• Haladyna TM. 2004. Developing and validating multiple-choice test items. Hillsdale, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Haladyna TM, Downing SM. 1989a. A taxonomy of multiple-choice item-writing rules. Appl Meas Educ. 2:37–50.
   Google Scholar
• Haladyna TM, Downing SM. 1989b. Validity of a taxonomy of multiple-choice item-writing rules. Appl Meas Educ. 2:51–78.
   Google Scholar
• Haladyna TM, Rodriguez MC. 2013. Developing and validating test items. New York, NY: Routledge Taylor & Francis Group.
   Google Scholar
• Hatala R, Norman GR. 2002. Adapting the Key Features Examination for a clinical clerkship. Med Educ. 36:160–165.
   PubMed Web of Science ®Google Scholar
• Hrynchak P, Glover Takahashi S, Nayer M. 2014. Key-feature questions for assessment of clinical reasoning: a literature review. Med Educ. 48:870–883.
   PubMed Web of Science ®Google Scholar
• Huwendiek S, Reichert F, Duncker C, de Leng BA, van der Vleuten CPM, Muijtjens AMM, Bosse HM, Haag M, Hoffmann GF, Tönshoff B, Dolmans D. 2017. Electronic assessment of clinical reasoning in clerkships: A mixed-methods comparison of long-menu key-feature problems with context-rich single best answer questions. Med Teach. 39:476–485.
   PubMed Web of Science ®Google Scholar
• Ilgen J, Humbert A, Kuhn G, Hansen M, Norman G, Eva KW, Charlin B, Sherbino J. 2012. Assessing diagnostic reasoning: a consensus statement summarizing theory, practice, and future needs. Acad Emerg Med. 19:1454–1461.
   PubMed Web of Science ®Google Scholar
• Jozefowicz RF, Koeppen BM, Case S, Galbraith R, Swanson D, Glew RH. 2002. The quality of in-house medical school examinations. Acad Med. 77:156–161.
   PubMed Web of Science ®Google Scholar
• Kilgour JM, Tayyaba S. 2016. An investigation into the optimal number of distractors in single-best answer exams. Adv Health Sci Educ Theory Pract. 21:571–585.
   PubMed Web of Science ®Google Scholar
• Krathwohl D. 2002. A revision of Bloom's taxonomy: An overview. Theory Pract. 41:212–218.
   Web of Science ®Google Scholar
• Lang AJ, Bronander K, Harrell H, Kovach R, Monteiro S, Bordage G. 2014. Validity evidence for a key features examination to assess clinical decision making in the internal medicine clerkship 16th Ottawa Conference. Ottawa, ON.
   Google Scholar
• Lawrence K, Allen Brailovsky T, Crichton C, Bethune T, Donoff C, Laughlin M, Wetmore TS, Carpentier M-P, Visser S. 2011. Defining competency-based evaluation objectives in family medicine: key-feature approach. Canadian Family Physician 57:e373–e380.
   PubMed Web of Science ®Google Scholar
• Leung F-H, Herold J, Iglar K. 2016. Family medicine mandatory assessment of progress: results of a pilot administration of a family medicine competency-based in-training examination. Can Fam Physician 62:e263–e267.
   Web of Science ®Google Scholar
• Medical Council of Canada. 2012. Guidelines for the Development of Key Feature Problems and Test Cases; [accessed 2017 Jan 26] http://mcc.ca/wp-content/uploads/cdm-guidelines.pdf.
   Google Scholar
• Medical Council of Canada. 2014. Blueprint Project: Qualifying examinations blueprint and content specifications. Ottawa, ON, Medical Council of Canada.
   Google Scholar
• Norman GR, Arfai B, Gupta A, Brooks LR, Eva KW. 2003. The privileged status of prestigious terminology: impact of "medicalese" on clinical judgments. Acad Med. 78:S82–S84.
   PubMed Web of Science ®Google Scholar
• Norman G, Bordage G, Page G, Keane D. 2006. How specific is case specificity? Med Educ. 40:618–623.
   PubMed Web of Science ®Google Scholar
• Page GG. 1999. Writing key feature problems for the clinical reasoning skills examination; [accessed 2017 Jan 26]. http://www.idealmed.org/workshop/SectionD-KeyFeatures.pdf.
   Google Scholar
• Page GG, Bordage G. 1995. The Medical Council of Canada's Key Features Project: A more valid written examination of clinical decision-making skills. Acad Med. 70:104–110.
   PubMed Web of Science ®Google Scholar
• Page GG, Bordage G, Allen T. 1995. Developing key-feature problems and examinations to assess clinical decision-making skills. Acad Med. 70:194–201.
   PubMed Web of Science ®Google Scholar
• Paniagua MA, Swygert KA. 2016. Writing written test questions for the basic and clinical sciences. Philadelphia, PA: National Board of Medical Examiners.
   Google Scholar
• Pharmaceutical Society of Australia. 2010. National Competency Standards Framework for Pharmacists in Australia. Australia: Pharmaceutical Society of Australia.
   Google Scholar
• Piasentin KA. 2010. Exploring the optimal number of options in multiple-choice testing. CLEAR Exam Rev (Winter). 18–22.
   Google Scholar
• Rodriguez MC. 2005. Three options are optimal for multiple-choice items: a meta-analysis of 80 years of research. Educ Meas Issues Prac (Summer). 24:3–13.
   Google Scholar
• Rotthoff T, Baehring T, Dicken HD, Fahron U, Richter B, Fischer MR, Scherbaum WA. 2006. Comparison between Long-Menu and Open-Ended Questions in computerized medical assessments. A randomized controlled trial. BMC Med Educ. 6:50.
   PubMedGoogle Scholar
• Royal Australian College of General Practitioners. 2015. Competency profile of the Australian general practitioner at the point of Fellowship. Australia: Royal Australian College of General Practitioners. [accessed 2018 June 12] https://www.racgp.org.au/download/Documents/VocationalTrain/Competency-Profile.pdf
   Google Scholar
• Royal College of Physicians and Surgeons of Canada. 2005. CanMEDS 2005 Framework. Ottawa, ON: Royal College of Physicians and Surgeons of Canada.
   Google Scholar
• Schneid SD, Armour C, Park YS, Yudkowsky R, Bordage G. 2014. Reducing the number of options on multiple-choice questions: response time, psychometrics and standard setting. Med Educ. 48:1020–1027.
   PubMed Web of Science ®Google Scholar
• Schuwirth LW, Verheggen MM, van der Vleuten CP, Boshuizen HP, Dinant GJ. 2001. Do short cases elicit different thinking processes than factual knowledge questions do? Med Educ. 35:348–356.
   PubMed Web of Science ®Google Scholar
• Tamblyn R, Abrahamowicz M, Dauphinee D, Wenghofer E, Jacques A, Klass D, Smee S, Blackmore D, Winslade N, Girard N, et al. 2007. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA 298:993–1001.
   PubMed Web of Science ®Google Scholar
• Tamblyn R, Abramowicz M, Dauphinee D, Wenghofer E, Jacques A, Klass D, Smee S, Eguale T, Winslade N, Girard N, Bartman I, Buckeridge D, Hanley J. 2010. Influence of physicians' management and communication ability on patients' persistence with antihypertensive medication. Arch Intern Med. 170:1064–1072.
   PubMed Web of Science ®Google Scholar
• The Royal College of Obstetricians and Gynecologists. Blueprint Grid for the Membership of the Royal College of Obstetricians and Gynecologists Examination; [accessed 2018 Feb 8]. https://www.rcog.org.uk/globalassets/documents/careers-and-training/mrcog-exam/part-1/ex-part-1-blueprinting-grid-new.pdf.
   Google Scholar
• University College Cork Ireland. 2015. How to Use the Draft Blueprint for the Pre-Registration Examinations (PRES) Level 3; [accessed 2018 Feb 8]. https://www.medicalcouncil.ie/Information-for-Doctors/Examinations-/How-to-use-the-Blueprint-for-the-PRES.pdf.
   Google Scholar
• van Bruggen L, Manrique-van Woudenbergh M, Spierenburg E, Vos J. 2012. Preferred question types for computer-based assessment of clinical reasoning: a literature study. Perspect Med Educ. 1:162–171.
   PubMedGoogle Scholar
• Van der Vleuten C, Newble D. 1995. How can we test clinical reasoning? Lancet. 345:1032–1034.
   PubMed Web of Science ®Google Scholar
• Wenghofer E, Klass D, Abrahamowicz M, Dauphinee D, Jacques A, Smee S, Blackmore D, Winslade N, Reidel K, Bartman I, Tamblyn R. 2009. Doctor scores on national qualifying examinations predict quality of care in future practice. Med Educ. 43:1166–1173.
   PubMed Web of Science ®Google Scholar
• World Health Organization (WHO). 2016. International Classification of Diseases, ICD10; [accessed 2018 Mar 28]. http://apps.who.int/classifications/icd10/browse/2016/en#/V.
   Google Scholar