Twelve tips for embedding assessment for and as learning practices in a programmatic assessment system

References

• Agarwal PK, Karpicke JD, Kang SHK, Roediger HL, McDermott KB. 2008. Examining the testing effect with open- and closed-book tests. Appl Cognit Psychol. 22(7):861–876.
   Web of Science ®Google Scholar
• Artino A, Hemmer P, Durning S. 2011. Using self-regulated learning theory to understand the beliefs, emotions, and behaviors of struggling medical students. Acad Med. 86(10 Suppl):S35–S38.
   PubMed Web of Science ®Google Scholar
• Artino A, La Rochelle J, Durning S. 2010. Second-year medical students’ motivational beliefs, emotions, and achievement. Med Educ. 44(12):1203–1212.
   PubMed Web of Science ®Google Scholar
• Baghdady MT, Carnahan H, Lam EW, Woods NN. 2013. Integration of basic sciences and clinical sciences in oral radiology education for dental students. J Dent Educ. 77(6):757–763.
   PubMed Web of Science ®Google Scholar
• Bandiera G, Kuper A, Mylopoulos M, Whitehead C, Ruetalo M, Kulasegaram K, Woods NN. 2018. Back from basics: integration of science and practice in medical education. Med Educ. 52(1):78–85.
   PubMed Web of Science ®Google Scholar
• Bickerdike A, O’Deasmhunaigh C, O’Flynn S, O’Tuathaigh C. 2016. Learning strategies, study habits and social networking activity of undergraduate medical students. Int J Med Educ. 7:230–236.
   PubMed Web of Science ®Google Scholar
• Bierer SB, Colbert CY, Foshee CM, French JC, Pien LC. 2018. Tool for diagnosing gaps within a competency-based assessment system. Acad Med. 93(3):512.
   PubMed Web of Science ®Google Scholar
• Black P, Wiliam D. 2006. Assessment and classroom learning. assessment in education: principles. Policy & Practice. 5(1):7–74.
   Google Scholar
• Brauer DG, Ferguson KJ. 2015. The integrated curriculum in medical education: AMEE Guide No. 96. Med Teach. 37(4):312–322.
   PubMed Web of Science ®Google Scholar
• Butler A, Karpicke J, Roediger H. 2007. The effect of type and timing of feedback on learning from multiple-choice tests. J Exp Psychol Appl. 13(4):273–281.
   PubMed Web of Science ®Google Scholar
• Butler A, Roediger H. 2008. Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Mem Cognit. 36(3):604–616.
   PubMed Web of Science ®Google Scholar
• Chou CL, Kalet A, Hauer KE. 2014. A research agenda for remediation in medical education. In: Remediation in medical education. New York (NY): Springer Books; p. 339–348.
   Google Scholar
• Craig SD, Sullins J, Witherspoon A, Gholson B. 2006. The deep-level-reasoning-question effect: the role of dialogue and deep-level-reasoning questions during vicarious learning. Cogn Instr. 24(4):565–591.
   Web of Science ®Google Scholar
• Dann R. 2014. Assessment as learning: blurring the boundaries of assessment and learning for theory, policy and practice. Assess Educ Princ Policy Pract. 21(2):149–166.
   Google Scholar
• Dunlosky J, Rawson KA, Marsh EJ, Nathan MJ, Willingham DT. 2013. Improving students’ learning with effective learning techniques: promising directions from cognitive and educational psychology. Psychol Sci Public Interest. 14(1):4–58.
   PubMed Web of Science ®Google Scholar
• Durning SJ, Dong T, Ratcliffe T, Schuwirth L, Artino AR Jr., Boulet JR, Eva K. 2016. Comparing open-book and closed-book examinations: a systematic review. Acad Med. 91(4):583–599.
   PubMed Web of Science ®Google Scholar
• Dweck C. 2006. Mindset: the new psychology of success. New York (NY): Random House.
   Google Scholar
• Epstein RM. 2007. Assessment in medical education. N Engl J Med. 356(4):387–396.
   PubMed Web of Science ®Google Scholar
• Ericsson KA. 2004. Deliberate practice and the acquisition and maintenance of expert performance in medicine and related domains. Acad Med. 79(10 Suppl):S70–S81.
   PubMed Web of Science ®Google Scholar
• Eva KW, Bordage G, Campbell C, Galbraith R, Ginsburg S, Holmboe E, Regehr G. 2016. Towards a program of assessment for health professionals: from training into practice. Adv Health Sci Educ Theory Pract. 21(4):897–913.
   PubMed Web of Science ®Google Scholar
• Gadbury-Amyot CC, Overman PR. 2018. Implementation of portfolios as a programmatic global assessment measure in dental education. J Dent Educ. 82(6):557–564.
   PubMed Web of Science ®Google Scholar
• Gielen S, Dochy F, Dierick S. 2003. Evaluating the consequential validity of new modes of assessment: The influence of assessment on learning, including pre-, post-, and true assessment effects. In: Segers M, Dochy F, Cascallar E, editors. Optimising new modes of assessment: in search of qualities and standards. New York (NY): Kluwer Academic Publishers.
   Google Scholar
• Green ML, Moeller JJ, Spak JM. 2018. Test-enhanced learning in health professions education: a systematic review: BEME Guide No. 48. Med Teach. 40(4):337–350.
   PubMed Web of Science ®Google Scholar
• Guerrassio J. 2013. Remediation of the struggling medical learner. Irwin (PA): Association for Hospital Medical Education.
   Google Scholar
• Harks B, Rakoczy K, Hattie J, Besser M, Klieme E. 2014. The effects of feedback on achievement, interest and self-evaluation: the role of feedback’s perceived usefulness. Educ Psychol. 34(3):269–290.
   Web of Science ®Google Scholar
• Hattie J, Biggs J, Purdie N. 1996. Effects of learning skills interventions on student learning: a meta-analysis. Rev Educ Res. 66(2):99–136.
   Web of Science ®Google Scholar
• Hattie J, Timperley H. 2007. The power of feedback. Rev Educ Res. 77(1):81–112.
   Web of Science ®Google Scholar
• Hauer KE, Ciccone A, Henzel TR, Katsufrakis P, Miller SH, Norcross WA, Papadakis MA, Irby DM. 2009. Remediation of the deficiencies of physicians across the continuum from medical school to practice: a thematic review of the literature. Acad Med. 84(12):1822–1832.
   PubMed Web of Science ®Google Scholar
• Hawthorne MR, Chretien KC, Torre D, Chheda SG. 2014. Re-demonstration without remediation-a missed opportunity? A national survey of internal medicine clerkship directors. Med Educ Online. 19(1):25991.
   PubMed Web of Science ®Google Scholar
• Heeneman S, Oudkerk Pool A, Schuwirth LW, van der Vleuten CP, Driessen EW. 2015. The impact of programmatic assessment on student learning: theory versus practice. Med Educ. 49(5):487–498.
   PubMed Web of Science ®Google Scholar
• Hmelo-Silver CE, Chinn CA, O’Donnell AM, Chan C. 2013. The international handbook of collaborative learning. New York (NY): Routledge.
   Google Scholar
• Hodges B. 2013. Assessment in the post-psychometric era: learning to love the subjective and collective. Med Teach. 35(7):564–568.
   PubMed Web of Science ®Google Scholar
• Irby DM, Hamstra SJ. 2016. Parting the clouds: three professionalism frameworks in medical education. Acad Med. 91(12):1606–1611.
   PubMed Web of Science ®Google Scholar
• Johnson TR, Khalil MK, Peppler RD, Davey DD, Kibble JD. 2014. Use of the NBME Comprehensive Basic Science Examination as a progress test in the preclerkship curriculum of a new medical school. Adv Physiol Educ. 38(4):315–320.
   PubMed Web of Science ®Google Scholar
• Kalet A, Chou C. 2014. Remediation in medical education: a mid-course correction. New York (NY): Springer Books.
   Google Scholar
• Koriat A, Bjork RA. 2005. Illusions of competence in monitoring one’s knowledge during study. J Exp Psychol Learn Mem Cogn. 31(2):187–194.
   PubMed Web of Science ®Google Scholar
• Kulasegaram K, Rangachari PK. 2018. Beyond “formative”: assessments to enrich student learning. Adv Physiol Educ. 42(1):5–14.
   PubMed Web of Science ®Google Scholar
• Larsen D, Butler A. 2013. Chapter 38: test-enhanced learning. In: Walsh K, editor. Oxford textbook of medical education. Oxford (UK): Oxford University Press; p. 443–452.
   Google Scholar
• Larsen DP, Butler AC, Roediger HL. 2008. Test-enhanced learning in medical education. Med Educ. 42(10):959–966.
   PubMed Web of Science ®Google Scholar
• Larsen DP, Butler AC, Roediger HL. 2013. Comparative effects of test-enhanced learning and self-explanation on long-term retention. Med Educ. 47(7):674–682.
   PubMed Web of Science ®Google Scholar
• Little J, Bjork EL. 2015. Optimizing multiple-choice tests as tools for learning. Mem Cognit. 43(1):14–26.
   PubMed Web of Science ®Google Scholar
• Marsh EJ, Butler AC. 2013. Memory in educational settings. In: Reisberg D, editor. The Oxford handbook of cognitive psychology. New York: Oxford University Press; p. 299–317.
   Google Scholar
• Metcalfe J. 2017. Learning from errors. Annu Rev Psychol. 68:465–489.
   PubMed Web of Science ®Google Scholar
• Klymkowsky MW, Taylor LB, Spindler SR, Garvin-Doxas RK. 2006. Two-dimensional, implicit confidence tests as a tool for recognizing student misconceptions. J Coll Sci Teach. 36(3):44–48.
   Google Scholar
• Mullet HG, Butler AC, Verdin B, von Borries R, Marsh EJ. 2014. Delaying feedback promotes transfer of knowledge despite student preferences to receive feedback immediately. J Appl Res Mem Cogn. 3(3):222–229.
   Web of Science ®Google Scholar
• Nicol DJ, Macfarlane‐Dick D. 2006. Formative assessment and self‐regulated learning: a model and seven principles of good feedback practice. Stud Higher Educ. 31(2):199–218.
   Web of Science ®Google Scholar
• Norcini J, Anderson MB, Bollela V, Burch V, Costa MJ, Duvivier R, Hays R, Palacios Mackay MF, Roberts T, Swanson D. 2018. 2018 Consensus framework for good assessment. Med Teach. 40(11):1102–1109.
   PubMed Web of Science ®Google Scholar
• Norman G, Neville A, Blake JM, Mueller B. 2010. Assessment steers learning down the right road: impact of progress testing on licensing examination performance. Med Teach. 32(6):496–499.
   PubMed Web of Science ®Google Scholar
• Pennebaker JW, Gosling SD, Ferrell JD. 2013. Daily online testing in large classes: boosting college performance while reducing achievement gaps. PLoS One. 8(11):e79774.
   PubMed Web of Science ®Google Scholar
• Pluta WJ, Richards BF, Mutnick A. 2013. PBL and beyond: trends in collaborative learning. Teach Learn Med. 25 Suppl 1:S9–S16.
   PubMed Web of Science ®Google Scholar
• Reed DA, Shanafelt TD, Satele DW, Power DV, Eacker A, Harper W, Moutier C, Durning S, Massie FS, Jr., Thomas MR. 2011. Relationship of pass/fail grading and curriculum structure with well-being among preclinical medical students: a multi-institutional study. Acad Med. 86(11):1367–1373.
   PubMed Web of Science ®Google Scholar
• Roediger HL, Putnam AL, Smith MA. 2011. Ten benefits of testing and their applications to educational practice. Psychol Learn Motiv Cogn Educ. 55:1–36.
   Web of Science ®Google Scholar
• Ross PT, Keeley MG, Mangrulkar RS, Karani R, Gliatto P, Santen SA. 2019. Developing professionalism and professional identity through unproctored, flexible testing. Acad Med. 94(4):490–495.
   PubMed Web of Science ®Google Scholar
• Rowland CA. 2014. The effect of testing versus restudy on retention: a meta-analytic review of the testing effect. Psychol Bull. 140(6):1432–1463.
   PubMed Web of Science ®Google Scholar
• Schuwirth L, Ash J. 2013. Chapter 35, Principles of assessment. In: Walsh K, editor. Oxford textbook of medical education. Oxford, United Kingdom: Oxford University Press; p. 409–420.
   Google Scholar
• Schuwirth L, van der Vleuten C. 2011. Programmatic assessment: from assessment of learning to assessment for learning. Med Teach. 33(6):478–485.
   PubMed Web of Science ®Google Scholar
• Schuwirth L, van der Vleuten C. 2012. The use of progress testing. Perspect Med Educ. 1(1):24–30.
   PubMedGoogle Scholar
• Schuwirth L, van der Vleuten C. 2019. How ‘testing’ has become ‘programmatic assessment for learning. Health Prof Educ. 5(3):177–184.
   Google Scholar
• Spring L, Robillard D, Gehlbach L, Simas TA. 2011. Impact of pass/fail grading on medical students’ well-being and academic outcomes. Med Educ. 45(9):867–877.
   PubMed Web of Science ®Google Scholar
• Swan Sein A, Cuffney F, Clinchot D. 2020a. How to help students strategically prepare for the MCAT exam and learn foundational knowledge needed for medical school. Acad Med. 95(3):484.
   Google Scholar
• Swan Sein A, Daniel M, Fleming A, Morrison G, Christner J, Esposito K, Pock A, O’Connor Grochowski C, Dalrymple J, Santen S. 2020b. Identifying and supporting struggling students to prevent failures when the United States Medical Licensing Examination (USMLE) step 1 is moved to after clerkships. Acad Med. [accessed 2020 March 3]. https://doi.org/10.1097/ACM.0000000000003272
   Web of Science ®Google Scholar
• Szpunar KK, McDermott KB, Roediger HL. 3rd. 2007. Expectation of a final cumulative test enhances long-term retention. Mem Cognit. 35(5):1007–1013.
   PubMed Web of Science ®Google Scholar
• ten Cate O, Chen H. 2016. The parts, the sum and the whole-evaluating students in teams. Med Teach. 38(7):639–641.
   PubMed Web of Science ®Google Scholar
• van de Ridder JM, McGaghie WC, Stokking KM, ten Cate OT. 2015. Variables that affect the process and outcome of feedback, relevant for medical training: a meta-review. Med Educ. 49(7):658–673.
   PubMed Web of Science ®Google Scholar
• van der Vleuten C. 2016. Revisiting ‘Assessing professional competence: from methods to programmes’. Med Educ. 50(9):885–888.
   PubMed Web of Science ®Google Scholar
• van der Vleuten CP, Driessen EW. 2014. What would happen to education if we take education evidence seriously? Perspect Med Educ. 3(3):222–232.
   PubMedGoogle Scholar
• van der Vleuten C, Schuwirth L, Driessen E, Govaerts M, Heeneman S. 2015. Twelve Tips for programmatic assessment. Med Teach. 37(7):641–646.
   PubMed Web of Science ®Google Scholar
• Vansteenkiste M, Simons J, Lens W, Sheldon KM, Deci EL. 2004. Motivating learning, performance, and persistence: the synergistic effects of intrinsic goal contents and autonomy-supportive contexts. J Pers Soc Psychol. 87(2):246–260.
   PubMed Web of Science ®Google Scholar
• White CB, Fantone JC. 2010. Pass-fail grading: laying the foundation for self-regulated learning. Adv Health Sci Educ Theory Pract. 15(4):469–477.
   PubMed Web of Science ®Google Scholar
• Wilkinson TJ, Tweed MJ. 2018. Deconstructing programmatic assessment. Adv Med Educ Pract. 9:191–197.
   PubMed Web of Science ®Google Scholar
• Wrigley W, van der Vleuten CP, Freeman A, Muijtjens A. 2012. A systemic framework for the progress test: strengths, constraints and issues: AMEE Guide No. 71. Med Teach. 34(9):683–697.
   PubMed Web of Science ®Google Scholar