A systemic framework for the progress test: Strengths, constraints and issues: AMEE Guide No. 71

References

• Aarts R, Steidel K, Manuel BAF, Driessen EW. Progress testing in resource-poor countries: A case from Mozambique. Med Teach 2010; 32: 461–463
   PubMed Web of Science ®Google Scholar
• Accreditation Council for Graduate Medical Education. 2012. Retrieved 24 January, 2012. Available from http://www.acgme.org/acWebsite/home/home.asp
   Google Scholar
• Al Alwan, I, Al-Moamary, M, Al-Attas, N, Al Kushi, A, ALBanyan, E, Zamakhshary, M, Al Kadri, HMF, Tamim, H, Magzoub, M, Hajeer, A et al., 2011. The progress test as a diagnostic tool for a new PBL curriculum. Educ Health. 24:493. Epub
   Google Scholar
• Albano MG, Cavallo F, Hoogenboom R, Magni F, Majoor G, Manenti F, Schuwirth L, Stiegler I, van der Vleuten C. An international comparison of knowledge levels of medical students: The Maastricht progress test. Med Educ 1996; 30: 239–245
   PubMed Web of Science ®Google Scholar
• Alnabhan M. An empirical investigation of the effects of three methods of handling guessing and risk taking on the psychometric properties of a test. Soc Behavior Personality 2002; 30: 645–252
   Web of Science ®Google Scholar
• American Educational Research Association, American Psychological Association & National Council on Measurement in Education. Standards for educational and psychological testing. American Educational Research Association, Washington, DC 1999
   Google Scholar
• Bandaranayake RJ. Setting and maintaining standards in multiple choice examinations: AMEE Guide No. 37. Med Teach 2008; 30: 836–845
   PubMed Web of Science ®Google Scholar
• Basu S, Roberts C, Newble DI, Snaith M. Competence in the musculoskeletal system: Assessing the progression of knowledge through an undergraduate medical course. Med Educ 2004; 38: 1253–1260
   PubMedGoogle Scholar
• Bennett J, Freeman A, Coombes L, Kay L, Ricketts C. Adaptation of medical progress testing to a dental setting. Med Teach 2010; 32: 500–502
   PubMed Web of Science ®Google Scholar
• Boshuizen HPA, van der Vleuten CPM, Schmidt H, Machiels-Bongaerts M. Measuring knowledge and clinical reasoning skills in a problem-based curriculum. Med Educ 1997; 31: 115–121
   PubMed Web of Science ®Google Scholar
• Bridge PD, Musial J, Frank R, Roe T, Sawilowsky S. Measurement practices: Methods for developing content-valid student examinations. Med Teach 2003; 25: 414–421
   PubMed Web of Science ®Google Scholar
• Burton RF. Misinformation, partial knowledge and guessing in true/false tests. Med Educ 2002; 36: 805–811
   PubMed Web of Science ®Google Scholar
• Butler AC. Repeated testing produces superior transfer of learning relative to repeated studying. J Experiment Psychol: Learning, Memory Cognit 2010; 36: 1118–1133
   PubMed Web of Science ®Google Scholar
• Carpenter SK, Pashler H, Wixted JT, Vul E. The effects of tests on learning and forgetting. Memory Cognit 2008; 36: 438–448
   PubMed Web of Science ®Google Scholar
• Case SM, Swanson DB. Constructing written test questions for the basic and clinical sciences. National Boad of Medical Examiners, Philadelphia, PA 2002
   Google Scholar
• Coderre S, Woloschuk W, McLaughlin K. Twelve tips for blueprinting. Med Teach 2009; 31: 322–324
   PubMed Web of Science ®Google Scholar
• Coombes L, Ricketts C, Freeman A, Stratford J. Beyond assessment: Feedback for individuals and institutions based on the progress test. Med Teach 2010; 32: 486–490
   PubMed Web of Science ®Google Scholar
• Danish KF, Khan RA. Role of effective feedback in multiple choice questons (MCQs) designing for faculty development. J Rawalpindi Med Coll 2010; 14: 98–100
   Google Scholar
• De Champlain A, Cuddy MM, Scoles PV, Brown M, Swanson DB, Holtzman K, Butler A. Progress testing in clinical science education: Results of a pilot project between the National Board of Medical Examiners and a US medical School. Med Teach 2010; 32: 503–508
   PubMed Web of Science ®Google Scholar
• Dijksterhuis MGK, Scheele F, Schuwirth LWT, Essed GGM, Nijhuis JG. Progress testing in postgraduate medical education. Med Teach 2009; 31: e464–e468
   PubMed Web of Science ®Google Scholar
• Downing SM. Threats to the validity of locally developed multiple-choice tests in medical education: Construct-irrelevant variance and construct underrepresentation. Adv Health Sci Educ 2002; 7: 235–241
   PubMed Web of Science ®Google Scholar
• Downing SM. Guessing on selected-response examinations. Med Educ 2003; 37: 670–671
   PubMed Web of Science ®Google Scholar
• Downing SM. The effects of violating standard item writing principles on tests and students: The consequences of using flawed test items on achievement examinations in medical education. Adv Health Sci Educ 2005; 10: 133–143
   PubMed Web of Science ®Google Scholar
• Downing SM, Tekian A, Yudkowsky R. Procedures for establishing defensible absolute passing scores on performance examinations in health professions education. Teach Learn Med 2006; 18: 50–57
   PubMed Web of Science ®Google Scholar
• Espinoza MP, Gardeazabal J. Optimal correction for guessing in multiple-choice tests. J Math Psychol 2010; 54: 415–425
   Google Scholar
• Finucane P, Flannery D, Keane D, Norman G. Cross-institutional progress testing: Feasibility and value to a new medical school. Med Educ 2010; 44: 184–186
   PubMed Web of Science ®Google Scholar
• Frank JR, (Ed.) 2005. The CanMEDS 2005 physician competency framework. Better standards. Better physicians. Better Care., Ottawa, The Royal College of Physicians and Surgeons of Canada
   Google Scholar
• Freeman A, Nicholls A, Ricketts C, Coombes L. Can we share quesitons? Performance of questions from different question banks in a single medical school. Med Teach 2010a; 32: 464–466
   PubMed Web of Science ®Google Scholar
• Freeman A, Ricketts C. Choosing and designing knowledge assessments: Experience at a new medical school. Med Teach 2010; 32: 578–581
   PubMed Web of Science ®Google Scholar
• Freeman A, van der Vleuten C, Nouns Z, Ricketts C. Progress testing internationally. Med Teach 2010b; 32: 451–455
   PubMed Web of Science ®Google Scholar
• General Medical Council. 2009. Good medical practice. Retrieved 24 January 2012.Available from http://www.gmc-uk.org/static/documents/content/GMP_0910.pdf
   Google Scholar
• Haladyna TM, Downing SM. A taxonomy of multiple-choice item-writing rules. Appl Meas Educ 1989a; 2: 37–50
   Google Scholar
• Haladyna TM, Downing SM. Validity of a taxonomy of multiple-choice item-writing rules. Appl Meas Educ 1989b; 2: 51–78
   Google Scholar
• Haladyna TM, Downing SM, Rodriquez MC. A review of multiple-choice item-writing guidelines for classroom assessment. Appl Meas Educ 2002; 15: 309–334
   Web of Science ®Google Scholar
• Holsgrove G, Elzubeir M. Imprecise terms in UK medical multiple-choice questions: What examiners think they mean. Med Educ 1998; 32: 343–350
   PubMed Web of Science ®Google Scholar
• International Foundations of Medicine 2011. Retrieved 20 July 2011. Available from http://www.nbme.org/Schools/iFoM/index.html
   Google Scholar
• International Partnership for Progress Testing 2011. Retrieved 18 July 2011. Available from http://ipptx.org/
   Google Scholar
• Jozefowicz RF, Koeppen BM, Case S, Galbraith R, Swanson DB, Glew RH. The quality of in-house medical school examinations. Acad Med 2002; 77: 156–161
   PubMed Web of Science ®Google Scholar
• Kerfoot BP, Shaffer K, McMahon GT, Baker H, Kirdar J, Kanter S, Corbett EC, Berkow R, Krupat E, Armstrong EG. Online “spaced education progress-testing” of students to confront two upcoming challenges to medical schools. Acad Med 2011; 86: 300–306
   PubMed Web of Science ®Google Scholar
• Koens F, Rademakers JDJ, Ten Cate TJ. Validation of core medica knowledge by postgraduates and specialists. Med Educ 2005; 39: 911–917
   PubMedGoogle Scholar
• Larsen DP, Butler AC, Roediger HL. Repeated testing improves lng-term retention relative to repeated study: A randomised controlled trial. Medical Education 2009; 43, 1174–1181
   PubMedGoogle Scholar
• Malau-Aduli BS, Zimitat C. Peer review improves the quality of MCQ examinations. Assess Eval Higher Educ 2011, DOI:10.1080/02602938.2011.586991
   Google Scholar
• Mardiastuti HW, Werdhani RA. Grade point average, progress test, and try outs's test as tools for curriculum evaluation and graduates' performance prediciton at the national baord examination. J Med Med Sci 2011; 2: 1302–1305
   Google Scholar
• McCrorie P, Boursicot AM. Variations in medical school graduating examinations in the United Kingdom: Are clinical competence standards comparable?. Med Teach 2009; 31: 223–229
   PubMed Web of Science ®Google Scholar
• McHarg J, Bradley P, Chamberlain S, Ricketts C, Searle J, McLachlan JC. Assessment of progress tests. Med Educ 2005; 39: 221–227
   PubMed Web of Science ®Google Scholar
• McLaughlin K, Lemaire J, Coderre S. Creating a reliable and valid blueprint for the internal medicine clerkship evaluation. Med Teach 2005; 27: 544–547
   PubMed Web of Science ®Google Scholar
• Muijtjens AMM, Hoogenboom RJI, Verwijnen GM, van der Vleuten CPM. Relative or absolute standards in assessing medical knowledge using progress tests. Adv Health Sci Educ 1998; 3: 81–87
   PubMed Web of Science ®Google Scholar
• Muijtjens AMM, Schuwirth LWT, Cohen-Schotanus J, van der Vleuten CPM. Origin bias of test items compromises the validity and fairness of curriculum comparisons. Med Educ 2007; 41: 1217–1223
   PubMedGoogle Scholar
• Muijtjens AMM, Schuwirth LWT, Cohen-Schotanus J, van der Vleuten CPM. Differences in knowledge development exposed by multi-curricular progress test data. Adv Health Sci Educ 2008; 13: 593–605
   PubMed Web of Science ®Google Scholar
• Muijtjens AMM, Timmermans I, Donkers J, Peperkamp R, Medema H, Cohen-Schotanus J, Thoben A, Wenink ACG, van der Vleuten CPM. Flexible electronic feedback using the virtues of progress testing. Med Teach 2010; 32: 491–495
   PubMed Web of Science ®Google Scholar
• Muijtjens AMM, van Mameren H, Hoogenboom RJI, Evers JLH, van der Vleuten CPM. The effect of a 'don't know' option on test scores: Number-right and formula scoring compared. Med Educ 1999; 33: 267–275
   PubMed Web of Science ®Google Scholar
• Muijtjens AMM, Wijnen W. Progress testing. Lessons from problem-based learning, H Van Berkel, A Scherpbier, H Hillen, C Van Der Vleuten. Oxford University Press, Oxford #2010
   Google Scholar
• Munro N, Rughani A, Foukles J, Wilson A, Neighbour R. Assessing validity in written tests of general practice – Exploration by factor analysis of candidate response patterns to Paper 1 of the MRCGP examination. Med Educ 2000; 34: 35–41
   PubMedGoogle Scholar
• Naeem N, van der Vleuten C, Alfaris EA. Faculty development on item writing substantially improves item quality. Adv Health Sci Educ 2011, DOI 10.1007/s10459-011-9315-2
   PubMedGoogle Scholar
• Norman G, Neville A, Blake J, Mueller B. Assessment steers learning down the right road: Impact of progress testing on licensing examination performance. Med Teach 2010; 32: 496–499
   PubMed Web of Science ®Google Scholar
• Nouns ZM, Georg W. Progress testing in German speaking countries. Med Teach 2010; 32: 467–470
   PubMed Web of Science ®Google Scholar
• Prideaux D, Gordon J. Can global co-operation enhance quality in medical education? Some lessons from an international assessment consortium in medical education. Med Educ 2002; 36: 404–405
   PubMed Web of Science ®Google Scholar
• Rademakers J, Ten Cate TJ, Bar PR. Progress testing with short answer questions. Med Teach 2005; 27: 578–582
   PubMed Web of Science ®Google Scholar
• Ricketts C, Freeman A, Pagliuga G, Coombes L, Archer J. Difficult decisions for progress testing: How much and how often?. Med Teach 2010; 32: 513–515
   PubMed Web of Science ®Google Scholar
• Ricketts C, Freeman AC, Coombes LR. Standard setting for progress tests: Combining external and internal standards. Med Educ 2009; 43: 589–593
   PubMed Web of Science ®Google Scholar
• Ricketts C, Moyeed R. Improving progress test score estimation using Bayesian statistics. Med Educ 2011; 45: 570–577
   PubMed Web of Science ®Google Scholar
• Roediger HL, Butler AC. The critical role of retrieval practice in long-term retention. Trends Cognit Sci 2010; 15: 20–27
   PubMedGoogle Scholar
• Roediger HL, Karpicke JD. Test-enhanced learning: Taking memory tests improves long-term retention. Psychol Sci 2006; 17: 249–255
   PubMed Web of Science ®Google Scholar
• Roex A, Degryse J. A computerized adaptive knowledge test as an assessment tool in general practice: A pilot study. Med Teach 2004; 26: 178–183
   PubMed Web of Science ®Google Scholar
• Sales D, Sturrock A, Boursicot K, Dacre J. Blueprinting for clinical performance deficiencies – Lessons and principles from the General Medical Council's fitness to practise procedures. Med Teach 2010; 32: e111–e114
   PubMed Web of Science ®Google Scholar
• Schaap L, Schmidt H, Verkoeijen PJL. Assessing knowledge growth in a psychology curriculum: Which students improve most?. Assess Eval Higher Educ 2011; 1–13
   Google Scholar
• Scharf EM, Baldwin LP. Assessing multiple choice question (MCQ) tests – A mathematical perspective. Active Learn Higher Educ 2007; 8: 31–47
   Google Scholar
• Schauber S, Nouns ZB. Using the cumulative deviation method for cross-institutional benchmarking in the Berlin progress test. Med Teach 2010; 32: 471–475
   PubMed Web of Science ®Google Scholar
• Schuwirth L. The need for national licencing examinations. Med Educ 2007; 41: 1022–1023
   PubMed Web of Science ®Google Scholar
• Schuwirth L, 2011. Personal communication
   Google Scholar
• Schuwirth L, Bosman G, Henning RH, Rinkel R, Wenink ACG. Collaboration on progress testing in medical schools in the Netherlands. Med Teach 2010; 32: 476–479
   PubMed Web of Science ®Google Scholar
• Stagnaro-Green AS, Downing SM. Use of flawed multiple-choice items by the New England Journal of Medicine for continuing medical education. Med Teach 2006; 28: 566–568
   PubMed Web of Science ®Google Scholar
• Swanson DB, Holtzman KZ, Butler A, Langer MM, Nelson MV, Chow JWM, Fuller R, Patterson JA, Boohan M, Committee M-SPT. Collaboration across the pond: The multi-school progress testing project. Med Teach 2010; 32: 480–485
   PubMed Web of Science ®Google Scholar
• Tarrant M, Knierim A, Hayes SK, Ware J. The frequency of item writing flaws in multiple-choice questions used in high stakes nursing assessments. Nurse Educ Today 2006; 26: 662–671
   PubMed Web of Science ®Google Scholar
• Tarrant M, Ware J. Impact of item-writing flaws in multiple-choice questions on student achievement in high-stakes nursing assessments. Med Educ 2008; 42: 198–206
   PubMed Web of Science ®Google Scholar
• Tombleson P, Fox RA, Dacre JA. Defining the content for the objective structured clinical examination component of the Professional and Linguistic Assessment Board examination: Development of a blueprint. Med Educ 2000; 34: 566–572
   PubMed Web of Science ®Google Scholar
• Tweed M, Wilkinson T. A randomized controlled trial comparing instructions regarding unsafe response options in a MCQ examination. Med Teach 2009; 31: 51–54
   PubMed Web of Science ®Google Scholar
• Van der Veken J, Valcke M, De Maeseneer J, Schuwirth L, Derese A. Impact on knowledge acquisition of the transition from a conventional to an integrated contextual medical curriculum. Med Educ 2009; 43: 704–713
   PubMed Web of Science ®Google Scholar
• van der Vleuten CPM, Schuwirth LWT, Muijtjens AMM, Thoben AJNM, Cohen-Schotanus J, van Boven CPA. Cross institutional collaboration in assessment: A case on progress testing. Med Teach 2004; 26: 719–725
   PubMed Web of Science ®Google Scholar
• van der Vleuten CPM, Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach 1996; 18: 103–109
   Web of Science ®Google Scholar
• van Diest R, van Dalen J, Bak M, Schruers K, van der Vleuten C, Muijtjens AMM, Scherpbier A. Growth of knowledge in psychiatry and behavioural sciences in a probelm-based learning curriculum. Med Educ 2004; 38: 1295–1301
   PubMed Web of Science ®Google Scholar
• van Herwaarden CLA, Laan RFJM, Leunissen RRM. Raamplan artsopleiding 2009. Centra, Nfvum. 2009
   Google Scholar
• Vantini I, Benini L. Models of learning, training and progress evaluation of medical students. Clin Chim Acta 2008; 393: 13–16
   PubMedGoogle Scholar
• Verhoeven BH, Snellen-Balendong HAM, Hay IT, Boon JM, Van Der Linde MJ, Blitz-Lindeque JJ, Hoogenboom RJI, Verwijnen GM, Wijnen WHFW, Scherpbier AJJA, et al. The versatility of progress testing assessed in an international context: A start fro benchmarking global standardization?. Med Teach 2005; 27: 514–520
   PubMed Web of Science ®Google Scholar
• Verhoeven BH, Van der Steeg AFW, Scherpbier AJJA, Muijtjens AMM, Verwijnen GM, van der Vleuten CPM. Reliability and credibility of an Angoff standard setting procedure in progress testing using recent graduates as judges. Med Educ 1999; 33: 832–837
   PubMedGoogle Scholar
• Verhoeven BH, Verwijnen GM, Muijtjens AMM, Scherpbier AJJA, van der Vleuten CPM. Panel expertise for an Angoff standard setting procedure in progress testing: Item writers compared to recently graduated students. Med Educ 2002a; 36: 860–867
   PubMed Web of Science ®Google Scholar
• Verhoeven BH, Verwijnen GM, Scherpbier AJJA, van der Vleuten CPM. Growth of medical knowledge. Med Educ 2002b; 36: 711–717
   PubMed Web of Science ®Google Scholar
• Wade L, Harrison C, Hollands J, Mattick K, Ricketts C, Wass L. Student perceptions of the prgores test in two settings and the implications for test deployment. Adv Health Sci Educ 2011, 01 November 2011 ed., Springer
   PubMedGoogle Scholar
• Wallach PM, Crespo LM, Holtzman KZ, Galbraith RM, Swanson DB. Use of a committee review process to improve the quality of course examinations. Adv Health Sci Educ 2006; 11: 61–68
   PubMed Web of Science ®Google Scholar
• Ware J, Torstein VK. Quality assurance of item writing: During the introduction of multiple choice questions in medicine for high stakes examinations. Med Teach 2009; 31: 238–243
   PubMed Web of Science ®Google Scholar
• World Federation for Medical Examinaton. Basic medical education. WFME Global Standards for Quality Improvement, Copenhagen 2003
   Google Scholar
• Zimmerman DW, Williams RH. A new look at the influence of guessing on the reliability of multiple-choice tests. Appl Psychol Meas 2003; 27: 357–371
   Web of Science ®Google Scholar