References
- Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Adv Health Sci Educ Theory Pract. 2009;14(4):595–621. doi:https://doi.org/10.1007/s10459-007-9090-2.
- Ramani S. Reflections on feedback: Closing the loop. Med Teach. 2016;38(2):206–207. doi:https://doi.org/10.3109/0142159X.2015.1044950.
- Maloney S, Tai JH-M, Lo K, Molloy E, Ilic D. Honesty in critically reflective essays: an analysis of student practice. Adv Health Sci Educ Theory Pract. 2013;18(4):617–626. doi:https://doi.org/10.1007/s10459-012-9399-3.
- Mak-van der Vossen MC, de la Croix A, Teherani A, van Mook WNKA, Croiset G, Kusurkar RA. Developing a two-dimensional model of unprofessional behaviour profiles in medical students. Adv in Health Sci Educ. 2019;24(2):215–232. doi:https://doi.org/10.1007/s10459-018-9861-y.
- Ambrose LJ, Ker JS. Levels of reflective thinking and patient safety: an investigation of the mechanisms that impact on student learning in a single cohort over a 5 year curriculum. Adv Health Sci Educ Theory Pract. 2014;19(3):297–310. doi:https://doi.org/10.1007/s10459-013-9470-8.
- Alizadeh M, Mirzazadeh A, Parmelee DX, et al. Leadership Identity Development Through Reflection and Feedback in Team-Based Learning Medical Student Teams. Teaching Learn Med. 2018;30(1):76–83. doi:https://doi.org/10.1080/10401334.2017.1331134.
- Driessen E, van Tartwijk J, Dornan T. The self critical doctor: helping students become more reflective. BMJ. 2008;336(7648):827–830. doi:https://doi.org/10.1136/bmj.39503.608032.AD.
- Wald HS, Borkan JM, Taylor JS, Anthony D, Reis SP. Fostering and evaluating reflective capacity in medical education: developing the REFLECT rubric for assessing reflective writing. Acad Med. 2012;87(1):41–50. doi:https://doi.org/10.1097/ACM.0b013e31823b55fa.
- Kihlgren P, Spanager L, Dieckmann P. Investigating novice doctors’ reflections in debriefings after simulation scenarios. Med Teach. 2015;37(5):437–443. doi:https://doi.org/10.3109/0142159X.2014.956054.
- Moniz T, Arntfield S, Miller K, Lingard L, Watling C, Regehr G. Considerations in the use of reflective writing for student assessment: issues of reliability and validity. Med Educ. 2015;49(9):901–908. doi:https://doi.org/10.1111/medu.12771.
- Ottenberg AL, Pasalic D, Bui GT, Pawlina W. An analysis of reflective writing early in the medical curriculum: The relationship between reflective capacity and academic achievement. Med Teacher. 2016;38(7):724–729. doi:https://doi.org/10.3109/0142159X.2015.1112890.
- Binyamin G. Growing from dilemmas: developing a professional identity through collaborative reflections on relational dilemmas. Adv Health Sci Educ Theory Pract. 2018;23(1):43–60. doi:https://doi.org/10.1007/s10459-017-9773-2.
- Humphrey-Murto S, Mihok M, Pugh D, Touchie C, Halman S, Wood TJ. Feedback in the OSCE: What Do Residents Remember? Teaching Learn Med. 2016;28(1):52–60. doi:https://doi.org/10.1080/10401334.2015.1107487.
- Burnett E, Phillips G, Ker JS. From theory to practice in learning about healthcare associated infections: Reliable assessment of final year medical students’ ability to reflect. Med Teach. 2008;30(6):e157–e160. doi:https://doi.org/10.1080/01421590802047299.
- Sandars J. The use of reflection in medical education: AMEE Guide No. 44. Med Teach. 2009;31(8):685–695. doi:https://doi.org/10.1080/01421590903050374.
- Brand S, Lancaster P, Gafson I, Nolan H. Encouraging reflection: good doctor or bad doctor?Med Educ. 2017;51(11):1173–1174. doi:https://doi.org/10.1111/medu.13449.
- Holmes CL, Harris IB, Schwartz AJ, Regehr G. Harnessing the hidden curriculum: a four-step approach to developing and reinforcing reflective competencies in medical clinical clerkship. Adv Health Sci Educ Theory Pract. 2015;20(5):1355–1370. doi:https://doi.org/10.1007/s10459-014-9558-9.
- Hayton A, Kang I, Wong R, Loo LK. Teaching medical students to reflect more deeply. Teach Learn Med. 2015;27(4):410–416. doi:https://doi.org/10.1080/10401334.2015.1077124.
- Chretien KC, Chheda SG, Torre D, Papp KK. Reflective writing in the internal medicine clerkship: a national survey of clerkship directors in internal medicine. Teach Learn Med. 2012;24(1):42–48. doi:https://doi.org/10.1080/10401334.2012.641486.
- Lew MDN, Schmidt HG. Self-reflection and academic performance: is there a relationship?Adv Health Sci Educ Theory Pract. 2011;16(4):529–545. doi:https://doi.org/10.1007/s10459-011-9298-z.
- Poole G, Jones L, Whitfield M. Helping students reflect: lessons from cognitive psychology. Adv Health Sci Educ Theory Pract. 2013;18(4):817–824. doi:https://doi.org/10.1007/s10459-012-9373-0.
- Koole S, Dornan T, Aper L, et al. Factors confounding the assessment of reflection: a critical review. BMC Med Educ. 2011;11(1):104 doi:https://doi.org/10.1186/1472-6920-11-104.
- Dewey J. How We Think. New York: BN Publishing; 1909.
- Korthagen FAJ. Reflective Teaching and Preservice Teacher Education in the Netherlands. J Teacher Educ. 1985;36(5):11–15. doi:https://doi.org/10.1177/002248718503600502.
- Kane M, Crooks T, Cohen A. Validating Measures of Performance. Educ Measurement Issues Pract. 2005;18(2):5–17. doi:https://doi.org/10.1111/j.1745-3992.1999.tb00010.x.
- Aronson L. Twelve tips for teaching reflection at all levels of medical education. Med Teach. 2011;33(3):200–205.doi:https://doi.org/10.3109/0142159X.2010.507714.
- McGuire L, Lay K, Peters J. Pedagogy of reflective writing in professional education. J Scholarship Teach Learn. 2012;9(1):93–107.
- Levine RE, Kelly PA, Karakoc T, Haidet P. Peer Evaluation in a Clinical Clerkship: students’ attitudes, experiences, and correlations with traditional assessments. Acad Psychiatry. 2007;31(1):19–24. doi:https://doi.org/10.1176/appi.ap.31.1.19.
- Nothelle SK, Christmas C, Hanyok LA. First-year internal medicine residents’ reflections on nonmedical home visits to high-risk patients. Teach Learn Med. 2018;30(1):95–102. doi:https://doi.org/10.1080/10401334.2017.1387552.
- Uygur J, Stuart E, De Paor M, et al. A Best Evidence in Medical Education systematic review to determine the most effective teaching methods that develop reflection in medical students: BEME Guide No. 51. Med Teach. 2019;41(1):3–16. doi:https://doi.org/10.1080/0142159X.2018.1505037.
- Veen M, Croix A. The swamplands of reflection: using conversation analysis to reveal the architecture of group reflection sessions. Med Educ. 2017;51(3):324–336. doi:https://doi.org/10.1111/medu.13154.
- Reis SP, Wald HS, Monroe AD, Borkan JM. Begin the BEGAN (The Brown Educational Guide to the Analysis of Narrative) - A framework for enhancing educational impact of faculty feedback to students’ reflective writing. Patient Educ Couns. 2010;80(2):253–259. doi:https://doi.org/10.1016/j.pec.2009.11.014.
- McNeill H, Brown JM, Shaw NJ. First year specialist trainees’ engagement with reflective practice in the e-portfolio. Adv Health Sci Educ Theory Pract. 2010;15(4):547–558. doi:https://doi.org/10.1007/s10459-009-9217-8.
- Driessen E. Do portfolios have a future?Adv Health Sci Educ Theory Pract. 2017;22(1):221–228. doi:https://doi.org/10.1007/s10459-016-9679-4.
- Branzetti J, Gisondi MA, Hopson LR, Regan L. Aiming Beyond Competent: The Application of the Taxonomy of Significant Learning to Medical Education. Teach Learn Med. 2019;31(4):466–478. doi:https://doi.org/10.1080/10401334.2018.1561368.
- Plack MM, Driscoll M, Marquez M, Cuppernull L, Maring J, Greenberg L. Assessing Reflective Writing on a Pediatric Clerkship by Using a Modified Bloom’s Taxonomy. Ambul Pediatr. 2007;7(4):285–291. doi:https://doi.org/10.1016/j.ambp.2007.04.006.
- Kember D, Leung D, McNaught C. A workshop activity to demonstrate that approaches to learning are influenced by the teaching and learning environment. Active Learning in Higher Education. 2008;9(1):43–56. doi:https://doi.org/10.1177/1469787407086745.
- Michels NR. Portfolio Learning and Assessing at the Workplace: development, Reliability and Validity [Doctoral thesis]. Antwerp 2012.
- Miller-Kuhlmann R, O’Sullivan PS, Aronson L. Essential steps in developing best practices to assess reflective skill: A comparison of two rubrics. Med Teacher. 2016;38(1):75–81. doi:https://doi.org/10.3109/0142159X.2015.1034662.
- Pollitt A. Comparative judgement for assessment. Int J Technol Des Educ. 2012;22(2):157–170. doi:https://doi.org/10.1007/s10798-011-9189-x.
- Weigle SC. Assessing Writing. Cambridge: Cambridge University Press 2002.
- Bacha N. Writing evaluation: what can analytic versus holistic essay scoring tell us? System. 2001;29(3):371–383. doi:https://doi.org/10.1016/S0346-251X(01)00025-2.
- Brennan RL, ed Educational Measurement. Westport, CT: American Council on Education/Praeger; 2004. No. 4.
- Michels NR, Driessen E, Muijtjens A, Van Gaal L, Bossaert L, Winter D. B. Portfolio Assessment during Medical Internships: How to Obtain a Reliable and Feasible Assessment Procedure? Education for Health. 2009;22(3):313–313.
- Jonsson A, Svingby G. The use of scoring rubrics: Reliability, validity and educational consequences. Educational Research Review. 2007;2(2):130–144. doi:https://doi.org/10.1016/j.edurev.2007.05.002.
- Stemler SE. A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Pract Assessment Res Eval. 2004;9(4):1–11.
- O’Sullivan P, Aronson L, Chittenden E, Niehaus B, Learman L. Reflective ability rubric and user guide. MedEdPORTAL. 2010;6(1):8133. doi:https://doi.org/10.15766/mep_2374-8265.8133.
- Bloxham S, den-Outer B, Hudson J, Price M. Let’s stop the pretence of consistent marking: exploring the multiple limitations of assessment criteria. Assess Eval Higher Educ. 2016;41(3):466–481. doi:https://doi.org/10.1080/02602938.2015.1024607.
- Bartholomew SR, Yoshikawa-Ruesch E. A Systematic Review of Research Around Adaptive Comparative Judgement (ACJ) in K-16 Education. In: Wells J, ed. CTETE—Research Monograph Series: Council on Technology and Engineering Teacher Education. Virginia: Council on Technology and Engineering Teacher Education. 2018;1(1):6–28.
- Buckley J, Canty D, Seery N. An exploration into the criteria used in assessing design activities with adaptive comparative judgment in technology education. Irish Educational Studies. 2020:1–19 doi:https://doi.org/10.1080/03323315.2020.1814838.
- Rezaei AR, Lovorn M. Reliability and validity of rubrics for assessment through writing. Assess Writing. 2010;15(1):18–39. doi:https://doi.org/10.1016/j.asw.2010.01.003.
- Weigle SC. Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches. Assess Writing. 1999;6(2):145–178. doi:https://doi.org/10.1016/S1075-2935(00)00010-6.
- Pollitt A. The method of Adaptive Comparative Judgement. Assess Educ Principles Policy Pract. 2012;19(3):281–300 doi:https://doi.org/10.1097/ACM.0b013e3181ed3aa7.
- Thurstone LL. A law of comparative judgment. Psychological Review. 1927;34(4):273–286. doi:https://doi.org/10.1037/h0070288.
- Jones I, Alcock L. Peer assessment without assessment criteria. Stud Higher Educ. 2014;39(10):1774–1787. doi:https://doi.org/10.1080/03075079.2013.821974.
- van Daal T, Lesterhuis M, Coertjens L, Donche V, De Maeyer S. Validity of comparative judgement to assess academic writing: examining implications of its holistic character and building on a shared consensus. Assess Educ Principles Policy Pract. 2019;26(1):59–74. doi:https://doi.org/10.1080/0969594X.2016.1253542.
- Haertel EH. Reliability. In: Brennan RL, ed. Educational Measurement. Vol 4. Westport, CT: American Council on Education/Praeger; 2006.
- Harris P, Bhanji F, Topps M, ICBME Collaborators, et al. Evolving concepts of assessment in a competency-based world. Med Teach. 2017;39(6):603–608. doi:https://doi.org/10.1080/0142159X.2017.1315071.
- Lockyer J, Carraccio C, Chan M-K, ICBME Collaborators, et al. Core principles of assessment in competency-based medical education. Med Teach. 2017;39(6):609–616. doi:https://doi.org/10.1080/0142159X.2017.1315082.
- Lesterhuis M, Verhavert S, Coertjens L, Donche V, De Maeyer S. Comparative judgement as a promising alternative to score competences. In: Ion G, Cano E, eds. Innovative Practices for Higher Education Assessment and Measurement. Hershey, PA: IGI Global; 2016.
- Settembri P, Van Gasse R, Coertjens L, De Maeyer S. Oranges and Apples? Using Comparative Judgement for Reliable Briefing Paper Assessment in Simulation Games. In: Bursens P, Donche V, Gijbels D, Spooren P, eds. Simulations of Decision-Making as Active Learning Tools: Design and Effects of Political Science Simulations. Cham: Springer International Publishing; 2018:93–108.
- Bramley T. Investigating the Reliability of Adaptive Comparative Judgment. Cambridge: Cambridge Assessment; 2015.
- McMahon S, Jones I. A comparative judgement approach to teacher assessment. Assess Educ Principles Policy Pract. 2015;22(3):368–389. doi:https://doi.org/10.1080/0969594X.2014.978839.
- Whitehouse C. Testing the Validity of Judgements about Geography Essays Using the Adaptive Comparative Judgement Method. Manchester: AQA Centre for Education Research and Policy; 2012.
- Coertjens L, Lesterhuis M, Verhavert S, Van Gasse R, De Maeyer S. Judging texts with rubrics and comparative judgement: Taking into account reliability and time investment. [Teksten beoordelen met criterialijsten of via paarsgewijze vergelijking: Een afweging van betrouwbaarheid en tijdsinvestering]. Pedagogische Studien. 2017;94(4):283–303.
- Heldsinger S, Humphry S. Using the method of pairwise comparison to obtain reliable teacher assessments. Aust Educ Res. 2010;37(2):1–19. doi:https://doi.org/10.1007/BF03216919.
- Verhavert S, De Maeyer S, Donche V, Coertjens L. Scale separation reliability: what does it mean in the context of comparative judgment? Appl Psychol Meas. 2018;42(6):428–445. doi:https://doi.org/10.1177/0146621617748321.
- Bouwer R, Lesterhuis M, Bonne P, De Maeyer S. Applying criteria to examples or learning by comparison: effects on students’ evaluative judgment and performance in writing. Front Educ. 2018;3(86):1–12. doi:https://doi.org/10.3389/feduc.2018.00086.
- Michels NR, Avonts M, Peeraer G, et al. Content validity of workplace-based portfolios: A multi-centre study. Medical Teacher. 2016;38(9):936–945. doi:https://doi.org/10.3109/0142159X.2015.1132407.
- Gwet KL. Handbook of Inter-Rater Reliability: The Definitive Guide to Measuring the Extent of Agreement among Multiple Raters. 4th ed.Gaithersburg, MD: Advanced Analytics, LLC; 2014.
- Bramley T, Vitello S. The effect of adaptivity on the reliability coefficient in adaptive comparative judgement. Assess Educ Principles Policy Pract. 2019;26(1):43–16. doi:https://doi.org/10.1080/0969594X.2017.1418734.
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. 2006;3(2):77–101. doi:https://doi.org/10.1191/1478088706qp063oa.
- Braun UK, Gill AC, Teal CR, Morrison LJ. The utility of reflective writing after a palliative care experience: can we assess medical students’ professionalism? J Palliat Med. 2013;16(11):1342–1349. doi:https://doi.org/10.1089/jpm.2012.0462.
- Driessen EW, Overeem K, van Tartwijk J, van der Vleuten CP, Muijtjens AM. Validity of portfolio assessment: which qualities determine ratings?Med Educ. 2006;40(9):862–866. doi:https://doi.org/10.1111/j.1365-2929.2006.02550.x.
- Aronson L, Niehaus B, DeVries CD, Siegel JR, O’Sullivan PS. Do writing and storytelling skill influence assessment of reflective ability in medical students’ written reflections?Acad Med J Assoc Am Med Coll. 2010;85(10 Suppl):S29–S32.
- Jacobs JL, Samarasekera DD, Shen L, Rajendran K, Hooi SC. Encouraging an environment to nurture lifelong learning: An Asian experience. Med Teach. 2014;36(2):164–168. doi:https://doi.org/10.3109/0142159X.2013.852168.
- Mortier AV, Lesterhuis M, Vlerick P, De Maeyer S. Comparative Judgment Within Online Assessment: Exploring Students Feedback Reactions. In: Ras E, Joosten-ten Brinke D, eds. Computer Assisted Assessment. Research into E-Assessment. Vol 571: Springer, Cham; 2015.
- Tekian A, Watling CJ, Roberts TE, Steinert Y, Norcini J. Qualitative and quantitative feedback in the context of competency-based education. Med Teach. 2017;39(12):1245–1249. doi:https://doi.org/10.1080/0142159X.2017.1372564.
- ten Cate O, Regehr G. The Power of Subjectivity in the Assessment of Medical Trainees. Acad Med. 2019;94(3):333–337. doi:https://doi.org/10.1097/ACM.0000000000002495.
- Ginsburg S, Vleuten CP, Eva KW, Lingard L. Cracking the code: residents’ interpretations of written assessment comments. Med Educ. 2017;51(4):401–410. doi:https://doi.org/10.1111/medu.13158.
- Gingerich A, Ramlo SE, van der Vleuten CPM, Eva KW, Regehr G. Inter-rater variability as mutual disagreement: identifying raters’ divergent points of view. Adv Health Sci Educ Theory Pract. 2017;22(4):819–838. doi:https://doi.org/10.1007/s10459-016-9711-8.
- Verhavert S, Bouwer R, Donche V, De Maeyer S. A meta-analysis on the reliability of comparative judgement. Assess Educ Principles Policy Pract. 2019;26(5):541–562. doi:https://doi.org/10.1080/0969594X.2019.1602027.