Reliability of Reviewer Ratings in the Manuscript Peer Review Process: An Opportunity for Improvement

Abstract

Accountability to authors and readers cannot exist without proper peer review practices. Thus, the information a journal seeks from its peer reviewers and how it makes use of this information is paramount. Disagreement amongst peer reviewers can be considerable, resulting in very diverse comments to authors. Incorporating a clear scoring system for key concrete items and requiring referees to provide justification for scores may ensure that reviewers contribute in a consistently fair and effective manner. This article evaluates information collected from reviewers and proposes an example of a system that aims to improve accountability, while having the potential to make it easier for reviewers to perform a more objective review.

Keywords:

• accountability
• disagreement
• peer review
• rating