Evaluating rater training with double-pretest one-posttest designs: an analysis of testing effects and the moderating role of rater self-efficacy

Abstract

Previous research on performance appraisal in Human Resource Management has found that frame-of-reference (FOR) training can improve rating accuracy. However, both the time-consuming development of training materials and the almost exclusive use of experimental evaluation designs limit the dissemination of FOR training as well as a better understanding of how and when it works. Therefore, unlike past research, the present studies used more general rater training materials and examined improvements in rater accuracy by means of a double-pretest one-posttest design. Study 1, using a student sample (the majority being part-time employees; N = 58), demonstrated the effectiveness of the rater training over and above a testing effect. Study 2 with participants from the workforce population (N = 45) replicated these results. In addition, in Study 2, pre-training rater self-efficacy moderated rater training effectiveness such that the rating accuracy of trainees with low self-efficacy improved more than did accuracy scores of trainees high in rater self-efficacy. We conclude that an effective FOR training can be conducted with less organization-specific materials, though both testing effects must be controlled for and individual differences (i.e. rater self-efficacy) between trainees have to be taken into account.

Keywords:

• Rater training
• frame-of-reference
• accuracy
• double-pretest one-posttest design
• rater self-efficacy

Notes

1. Note that this does not mean that learning self-efficacy is at issue (e.g. Dierdorff, Surface, & Brown, 2010).

2. Following Feingold (2009), we used the betas divided through the standard deviation of the pretest scores as measures of effect sizes.

3. In order to scrutinize the robustness of our results, we additionally estimated the regression models without control variables (cf. Spector & Brannick, 2011). Only minor changes occurred, which underpins the robustness of the results. Calculation details can be requested from the first author.