Key recommendations for improving AoL assessments: A longitudinal analysis of rater bias and reliability in embedded rubric-based measurements

Abstract

Assurance of learning assessment results are used as a basis for curriculum changes at Association to Advance Collegiate Schools of Business–accredited institutions. Because budget constraints and opportunity costs impact all schools, assessment quality is critical. This research uses a longitudinal, business school case study to highlight bias and reliability issues that can occur in course-embedded, rubric-based assessments. The results support and extend previous research on the value of training raters for improving reliability and provide the basis for recommended assurance of learning process steps that business schools can immediately implement to improve and maintain measurement quality over time.

Keywords:

• Assessment
• assurance of learning
• bias
• interrater reliability
• rubric