How to achieve accurate peer assessment for high value written assignments in a senior undergraduate course

Abstract

The psychometric measures of accuracy, reliability and validity of peer assessment are critical qualities for its use as a supplement to instructor grading. In this study, we seek to determine which factors related to peer review are the most influential on these psychometric measures, with a primary focus on the accuracy of peer assessment or how closely peer-given grades match those of an instructor. We examine and rank the correlations of accuracy, reliability and validity with 17 quantitative and qualitative variables for three senior undergraduate courses that used peer assessment on high value written assignments. Based on these analyses, we altered the single most significant variable of one of the courses. We demonstrate that the number of reviews completed per reviewer has the greatest influence on the accuracy of peer assessment out of all the factors analysed. Our calculations suggest that six reviews must be completed per reviewer to achieve peer assessment that is no different from the grading of an instructor. Effective training, previous experience and strong academic abilities in the reviewers may reduce this number.

Keywords:

• peer assessment
• accuracy
• reliability
• validity

Acknowledgements

We would like to thank the University of Guelph Teaching Support Services (now Open Learning and Educational Support (OpenEd)) for the development and use of the PEAR tool and Dr N. Richardson for critically reviewing the manuscript. This work was supported by the University of Guelph 2014–15 Scholarship of Teaching and Learning Award (SoTL).