Let’s Talk it Out: The Effects of Calibration Meetings on Performance Ratings

References

• Arthur, A., & Doverspike, D. (2005). Achieving diversity and reducing discrimination in the workplace through human resource management practices: Implications of research and theory for staffing, training, and rewarding performance. In R. L. Dipboye & A. Colella (Eds.), Discrimination at work: The psychological and organizational bases (pp. 305–327). Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Arthur, W., Bennett Jr., W., Stanush, P. L., & McNelly, T. L. (1998). Factors that influence skill decay and retention: A quantitative review and analysis. Human Performance, 11, 57–101. doi:10.1207/s15327043hup1101_3
   Web of Science ®Google Scholar
• Bernardin, H. J., & Buckley, M. R. (1981). Strategies in rater training. Academy of Management Review, 6, 205–212.
   Google Scholar
• Caruso, K. N. (2013). A practical guide to performance calibration: A step-by-step guide to increasing the fairness and accuracy of performance appraisal [White paper]. Retrieved from http://cdn2.hubspot.net/hub/91252/file-338193964-pdf/Practical_Guide_to_Performance_Calibration_October_2013.pdf
   Google Scholar
• Cascio, W. F., & Aguinis, H. (2005). Applied psychology in human resource management (6th ed.). Upper Saddle River, NJ: Prentice-Hall.
   Google Scholar
• Castilla, E. J. (2008). Gender, race, and meritocracy in organizational careers. American Journal of Sociology, 113, 1479–1526. doi:10.1086/588738
   PubMed Web of Science ®Google Scholar
• Cleveland, J. N., & Murphy, K. R. (1992). Analyzing performance appraisal as goal-directed behavior. Research in Personnel and Human Resources Management, 10, 121–185.
   Google Scholar
• Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
   PubMed Web of Science ®Google Scholar
• DeNisi, A. S., Cafferty, T. P., & Meglino, B. M. (1984). A cognitive view of the performance appraisal process: A model and research propositions. Organizational Behavior & Human Performance, 33, 360–396. doi:10.1016/0030-5073(84)90029-1
   Web of Science ®Google Scholar
• Djurdjevic, E., & Wheeler, A. R. (2014). A dynamic multilevel model of performance rating. Research in Personnel and Human Resources, 32, 147–176.
   Google Scholar
• Funder, D. C. (1995). On the accuracy of personality judgment: A realistic approach. Psychological Review, 102, 652–670.
   PubMed Web of Science ®Google Scholar
• Funder, D. C. (1999). Personality judgment: A realistic approach to person perception. San Diego, CA: Academic Press..
   Google Scholar
• Goffin, R. D., Gellatly, I. R., Paunonen, S. V., Jackson, D. N., & Meyer, J. P. (1996). Criterion validation of two approaches to performance appraisal: The behavioral observation scale and the relative percentile method. Journal of Business and Psychology, 11, 23–33. doi:10.1007/BF02278252
   Web of Science ®Google Scholar
• Goffin, R. D., & Olson, J. M. (2011). Is it all relative? Comparative judgments and the possible improvement of self-ratings and ratings of others. Perspectives on Psychological Science, 6, 48–60. doi:10.1177/1745691610393521
   Web of Science ®Google Scholar
• Gorman, C. A., Meriac, J. P., Roch, S. G., Ray, J. L., & Gamble, J. S. (2017). An exploratory study of current performance management practices: Human resource executives’ perspectives. International Journal of Selection and Assessment, 25, 193–202. doi:10.1111/ijsa.2017.25.issue-2
   Web of Science ®Google Scholar
• Harari, M. B., & Rudolph, C. W. (2017). The effect of rater accountability on performance ratings: A meta-analytic review. Human Resource Management Review, 27(1), 121–133. doi:10.1016/j.hrmr.2016.09.007
   Web of Science ®Google Scholar
• Harris, M. M. (1994). Rater motivation in the performance appraisal context: A theoretical framework. Journal of Management, 20, 735–756. doi:10.1177/014920639402000403
   Web of Science ®Google Scholar
• Hinsz, V. B. (1990). Cognitive and consensus processes in group recognition memory performance. Journal of Personality and Social Psychology, 59, 705–718. doi:10.1037/0022-3514.59.4.705
   Web of Science ®Google Scholar
• Hoffman, B. J., & Woehr, D. J. (2009). Disentangling the meaning of multisource performance rating source and dimension factors. Personnel Psychology, 62, 735–765. doi:10.1111/peps.2009.62.issue-4
   Web of Science ®Google Scholar
• Jelley, R. B., & Goffin, R. D. (2001). Can performance-feedback accuracy be improved? Effects of rater priming and rating-scale format on rating accuracy. Journal of Applied Psychology, 86, 134–144.
   PubMed Web of Science ®Google Scholar
• Keeping, L. M., & Levy, P. E. (2000). Performance appraisal reactions: Measurement, modeling, and method bias. Journal of Applied Psychology, 85, 708–723.
   PubMed Web of Science ®Google Scholar
• Lance, C. E., Hoffman, B. J., Gentry, W. A., & Baranik, L. E. (2008). Rater source factors represent important subcomponents of the criterion construct space, not rater bias. Human Resource Management Review, 18, 223–232. doi:10.1016/j.hrmr.2008.03.002
   Web of Science ®Google Scholar
• Landy, F. J., & Farr, J. L. (1980). Performance rating. Psychological Bulletin, 87, 72–107. doi:10.1037/0033-2909.87.1.72
   Web of Science ®Google Scholar
• Lerner, J. S., & Tetlock, P. E. (1999). Accounting for the effects of accountability. Psychological Bulletin, 125, 255–275.
   PubMed Web of Science ®Google Scholar
• Levy, P. E., & Williams, J. R. (2004). The social context of performance appraisal: A review and framework for the future. Journal of Management, 30, 881–905. doi:10.1016/j.jm.2004.06.005
   Web of Science ®Google Scholar
• London, M., Mone, E. M., & Scott, J. C. (2004). Performance management and assessment: Methods for improved rater accuracy and employee goal setting. Human Resource Management, 43, 319–336. doi:10.1002/(ISSN)1099-050X
   Web of Science ®Google Scholar
• Longenecker, C. O., Sims, J. H., & Gioia, D. A. (1987). Behind the mask: The politics of employee appraisal. Academy of Management Perspectives, 1, 183–193. doi:10.5465/ame.1987.4275731
   Google Scholar
• Makiney, J. D., & Levy, P. E. (1998). The influence of self-ratings versus peer ratings on supervisors‘ performance judgments. Organizational Behavior and Human Decision Processes, 74, 212–228.
   PubMed Web of Science ®Google Scholar
• Martell, R. F., & Borg, M. R. (1993). A comparison of the behavioral rating accuracy of groups and individuals. Journal of Applied Psychology, 78, 43–50. doi:10.1037/0021-9010.78.1.43
   Web of Science ®Google Scholar
• Martell, R. F., & Leavitt, K. N. (2002). Reducing the performance–Cue bias in work behavior ratings: Can groups help. Journal of Applied Psychology, 87, 1032–1041. doi:10.1037/0021-9010.87.6.1032
   PubMed Web of Science ®Google Scholar
• Mero, N. P., Guidice, R. M., & Brownlee, A. L. (2007). Accountability in a performance appraisal context: The effect of audience and form of accounting on rater response and behavior. Journal of Management, 33, 223–252. doi:10.1177/0149206306297633
   Web of Science ®Google Scholar
• Mero, N. P., Guidice, R. M., & Werner, S. (2014). A field study of the antecedents and performance consequences of perceived accountability. Journal of Management, 40, 1627–1652. doi:10.1177/0149206312441208
   Web of Science ®Google Scholar
• Mero, N. P., & Motowidlo, S. J. (1995). Effects of rater accountability on the accuracy and the favorability of performance ratings. Journal of Applied Psychology, 80, 517–524. doi:10.1037/0021-9010.80.4.517
   Web of Science ®Google Scholar
• Mero, N. P., Motowidlo, S. J., & Anna, A. L. (2003). Effects of accountability on rating behavior and rater accuracy. Journal of Applied Social Psychology, 33, 2493–2514. doi:10.1111/jasp.2003.33.issue-12
   Web of Science ®Google Scholar
• Miller, L. (2011). Dear workforce how should we conduct performance-appraisal ‘calibration’ meetings with managers? Workforce. Retrieved from http://www.workforce.com/2011/09/07/dear-workforce-how-should-we-conduct-performance-appraisal-calibration-meetings-with-managers/
   Google Scholar
• Muchinsky, P., & Culbertson, S. (2016). Psychology applied to work (11th ed.). Summerfield, NC: Hypergraphic Press Inc.
   Google Scholar
• Murphy, K., & Cleveland, J. (1995). Understanding performance appraisal: Social, organizational and goal-oriented perspectives. Newbury Park, CA: Sage.
   Google Scholar
• Murphy, K. R., Cleveland, J. N., Skattebo, A. L., & Kinney, T. B. (2004). Raters who pursue different goals give different ratings. Journal of Applied Psychology, 89, 158–164. doi:10.1037/0021-9010.89.1.158
   PubMed Web of Science ®Google Scholar
• Park, S. (2014). Motivation of public managers as raters in performance appraisal: Developing a model of rater motivation. Public Personnel Management, 43, 387–414. doi:10.1177/0091026014530675
   Web of Science ®Google Scholar
• Pulakos, E. D., Hanson, R. M., Arad, S., & Moye, N. (2015). Performance management can be fixed: An on-the-job experiential learning approach for complex behavior change. Industrial and Organizational Psychology: Perspectives on Science and Practice, 8, 51–76. doi:10.1017/iop.2014.2
   Web of Science ®Google Scholar
• Pulakos, E. D., & O’Leary, R. S. (2011). Why is performance management broken?. Industrial and Organizational Psychology, 4, 146–164. doi:10.1111/j.1754-9434.2011.01315.x
   Google Scholar
• Pulakos, E. D., Schmitt, N., Whitney, D., & Smith, M. (1996). Individual differences in interviewer ratings: The impact of standardization, consensus discussion, and sampling error on the validity of a structured interview. Personnel Psychology, 49, 85–102. doi:10.1111/peps.1996.49.issue-1
   Web of Science ®Google Scholar
• Pynes, J., & Bernardin, H. J. (1992). Mechanical vs consensus-derived assessment center ratings: A comparison of job performance validities. Public Personnel Management, 21, 17–28. doi:10.1177/009102609202100102
   Web of Science ®Google Scholar
• Pytel, L., & Hunt, S. (2017). Total workforce performance management: Using talent calibration to effectively manage the reality that all employees are valuable, but some employees are more valuable than others. Retrieved from https://www.successfactors.com/en_us/resources.html?documentTypes=white_papers
   Google Scholar
• Raymark, P. H., Balzer, W. K., & DeLaTorre, F. (1999). A preliminary investigation of the sources of information used by raters when appraising performance. Journal of Business and Psychology, 14, 319–339. doi:10.1023/A:1022151426909
   Web of Science ®Google Scholar
• Roberson, L., Galvin, B. M., & Charles, A. C. (2007). When group identities matter: Bias in performance appraisal. The Academy of Management Annals, 1, 617–650. doi:10.5465/078559818
   Web of Science ®Google Scholar
• Roch, S. G. (2006). Discussion and consensus in rater groups: Implications for behavioral and rating accuracy. Human Performance, 19, 91–115. doi:10.1207/s15327043hup1902_1
   Web of Science ®Google Scholar
• Roch, S. G. (2007). Why convene rater teams: An investigation of the benefits of anticipated discussion, consensus, and rater motivation. Organizational Behavior and Human Decision Processes, 104, 14–29. doi:10.1016/j.obhdp.2006.08.003
   Web of Science ®Google Scholar
• Roch, S. G., & McNall, L. A. (2007). An investigation of factors influencing accountability and performance ratings. The Journal of Psychology, 141, 499–523. doi:10.3200/JRLP.141.5.499-524
   PubMed Web of Science ®Google Scholar
• Roch, S. G., Woehr, D. J., Mishra, V., & Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85, 370–395. doi:10.1111/joop.2012.85.issue-2
   Web of Science ®Google Scholar
• Spence, J. R., & Keeping, L. M. (2010). The impact of non‐performance information on ratings of job performance: A policy‐capturing approach. Journal of Organizational Behavior, 31, 587–608. doi:10.1002/job.648
   Web of Science ®Google Scholar
• Spence, J. R., & Keeping, L. M. (2011). Conscious rating distortion in performance appraisal: A review, commentary, and proposed framework for research. Human Resource Management Review, 21, 85–95. doi:10.1016/j.hrmr.2010.09.013
   Web of Science ®Google Scholar
• Spence, J. R., & Keeping, L. M. (2013). The road to performance is paved with intentions: A framework for understanding managers’ intentions when rating employee performance. Organizational Psychology Review, 3, 360–383. doi:10.1177/2041386613485969
   Web of Science ®Google Scholar
• Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73(3), 497–506. doi:10.1037/0021-9010.73.3.497
   Web of Science ®Google Scholar
• Taylor, M. S., Masterson, S. S., Renard, M. K., & Tracy, K. B. (1998). Managers‘ reactions to procedurally just performance management systems. Academy of Management Journal, 41, 568–579.
   Web of Science ®Google Scholar
• Tetlock, P. E. (1983). Accountability and complexity of thought. Journal of Personality and Social Psychology, 45, 74–83. doi:10.1037/0022-3514.45.1.74
   Web of Science ®Google Scholar
• Tziner, A., Latham, G. P., Price, B. S., & Haccoun, R. (1996). Development and validation of a questionnaire for measuring perceived political considerations in performance appraisal. Journal of Organizational Behavior, 17, 179–190. doi:10.1002/(ISSN)1099-1379
   Web of Science ®Google Scholar
• Viswesvaran, C., Schmidt, F. L., & Ones, D. S. (2005). Is there a general factor in ratings of performance? A meta-analytic framework for disentangling substantive and error influences. Journal of Applied Psychology, 90, 108–131. doi:10.1037/0021-9010.90.1.108
   PubMed Web of Science ®Google Scholar
• Vollrath, D. A., Sheppard, B. H., Hinsz, V. B., & Davis, J. H. (1989). Memory performance by decision-making groups and individuals. Organizational Behavior and Human Decision Processes, 43, 289–300. doi:10.1016/0749-5978(89)90040-X
   Web of Science ®Google Scholar
• Wagner, S. H., & Goffin, R. D. (1997). Differences in accuracy of absolute and comparative performance appraisal methods. Organizational Behavior and Human Decision Processes, 70, 95–103. doi:10.1006/obhd.1997.2698
   Web of Science ®Google Scholar
• Wang, X. M., Wong, K. F. E., & Kwong, J. Y. (2010). The roles of rater goals and ratee performance levels in the distortion of performance ratings. Journal of Applied Psychology, 95, 546–561. doi:10.1037/a0018866
   PubMed Web of Science ®Google Scholar
• Weldon, M. S., & Bellinger, K. D. (1997). Collective memory: Collaborative and individual processes in remembering. Journal of Experimental Psychology, Learning, Memory, & Cognition, 23, 1160–1175. doi:10.1037/0278-7393.23.5.1160
   PubMed Web of Science ®Google Scholar
• Wherry, R. J., & Bartlett, C. J. (1982). The control of bias in ratings: A theory of rating. Personnel Psychology, 35, 521–550. doi:10.1111/j.1744-6570.1982.tb02208.x
   Web of Science ®Google Scholar
• Woehr, D. J. (1994). Understanding frame-of-reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79, 525–534. doi:10.1037/0021-9010.79.4.525
   Web of Science ®Google Scholar
• Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189–205. doi:10.1111/joop.1994.67.issue-3
   Web of Science ®Google Scholar
• Wong, K. F. E., & Kwong, J. Y. (2007). The role of anticipated regret in escalation of commitment. Journal of Applied Psychology, 92, 545–554. doi:10.1037/0021-9010.92.2.545
   PubMed Web of Science ®Google Scholar