Let’s Talk it Out: The Effects of Calibration Meetings on Performance Ratings

ABSTRACT

Despite their use in practice, calibration meetings of performance appraisal ratings have received little attention in the academic literature to date. The current paper addresses this gap by formally defining calibration meetings and by investigating the nature and impact of calibration meetings on performance ratings across two field studies. Results indicated that calibration meetings do occur in organizations. The nature and cadence of calibration meetings varied considerably, but managers generally perceived calibrations as attempts to improve rating quality. Calibration meetings were also associated with increases in perceived frame-of-reference, perceived accountability, and perceived total information to base ratings upon. Further, results showed that post-calibration ratings correlated more with other measures assessing the job performance domain than pre-calibration ratings.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ When assessing current employee performance, these practices might be referred to as “performance reviews” or “talent reviews”, depending on the setting.

² Research on group consensus discussions in interviews (e.g., Pulakos, Schmitt, Whitney, & Smith, 1996) and assessment centers (e.g., Pynes & Bernardin, 1992) might also apply to calibration meetings, although this body of research is mixed, and the time-frame, context, intentions, and requirements of PA ratings are different for making judgments.

³ Participants were directed to choose specific responses for several questions; excessively fast response times.

⁴ There are many ways to operationalize rating quality and accuracy within the PA literature (e.g., Sulsky & Balzer, 1988). Accuracy has been operationalized by the presence of psychometric effects (i.e., halo and leniency, which contemporary views do not consider as measures of accuracy), convergence with expert-derived scores, recall of specific behaviors that occurred during an observation period, and convergence with measures within the nomological network of job performance (though convergent validity is typically not viewed as a direct measure of accuracy). In operational settings where real employees and real performance appraisals are used, expert-derived true scores are near impossible to obtain, given the person most knowledgeable and capable of making expert ratings would be the manager. Further, operationalizing psychometric effects as indices of accuracy have mostly been abandoned (e.g., London et al., 2004; Roch et al., 2012). The current study operationalized rating quality as convergence with measures within the nomological network of job performance.

⁵ Hotelling-Williams t-tests were used to test for significance because of the dependence of the estimates.

⁶ One could argue that the improved correlations with performance metrics between pre- and post-ratings are conflated because the metrics were discussed in the calibration meetings. While this is a valid concern, the performance metrics do reflect ratee performance, and thus organizations would usually want alignment with these measures. Furthermore, given the ratings shifted in other ways (i.e., were less lenient, had less rater-specific variance) and given there was not perfect alignment with performance metrics, it appears the changes in pre-to-post ratings were affected by more than simply a force-fitting to the regularly collected performance indices.

⁷ A more precise estimate of the usage of calibration meetings would involve randomly sampling organizations and having either HR leaders or actual managers provide information.