Evaluating rater training with double-pretest one-posttest designs: an analysis of testing effects and the moderating role of rater self-efficacy

References

• Arvey, R. D., & Murphy, K. R. (1998). Performance evaluation in work settings. Annual Review of Psychology, 49, 141–168.10.1146/annurev.psych.49.1.141
   PubMed Web of Science ®Google Scholar
• Bandura, A. (1986). Social foundations of thought and action: A social-cognitive theory. Englewood-Cliffs, NJ: Prentice-Hall.
   Google Scholar
• Bernardin, H. J., Buckley, M. R., Tyler, C. L., & Wiese, D. S. (2000). A reconsideration of strategies for rater training. In G. R. Ferris (Ed.), Research in personnel and human resources management (pp. 221–274). Stamford, CT: JAI Press.
   Google Scholar
• Bernardin, H. J., & Pence, E. C. (1980). Effects of rater training: Creating new response sets and decreasing accuracy. Journal of Applied Psychology, 65, 60–66.10.1037/0021-9010.65.1.60
   Web of Science ®Google Scholar
• Bernardin, H. J., & Villanova, P. (2005). Research streams in rater self-efficacy. Group & Organization Management, 30, 61–88.10.1177/1059601104267675
   Web of Science ®Google Scholar
• Campbell, J. P. (1990). Modeling the performance prediction problem in industrial and organizational psychology. In M. Dunnette & L. M. Hough (Eds.), Handbook of industrial and organizational psychology (Vol. 1, 2nd ed., pp. 687–731). Palo Alto, CA: Consulting Psychologists Press.
   Google Scholar
• Carlson, K. D., & Schmidt, F. L. (1999). Impact of experimental design on effect size: Findings from the research literature on training. Journal of Applied Psychology, 84, 851–862.10.1037/0021-9010.84.6.851
   Web of Science ®Google Scholar
• Chirico, K. E., Buckley, M. R., Wheeler, A. R., Facteau, J. D., Bernardin, H. J., & Beu, D. S. (2004). A note on the need for true scores in frame-of-reference (FOR) training research. Journal of Managerial Issues, 16, 382–395.
   Google Scholar
• Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple regression/correlation analysis for the behavioral sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• Collins, D. B., & Holton, E. F., III (2004). The effectiveness of managerial leadership development programs: A meta-analysis of studies from 1982 to 2001. Human Resource Development Quarterly, 15, 217–248.10.1002/(ISSN)1532-1096
   Google Scholar
• Cronbach, L. J. (1955). Processes affecting scores on ‘Understanding of others’ and ‘Assumed similarity’. Psychological Bulletin, 52, 177–193.10.1037/h0044919
   PubMed Web of Science ®Google Scholar
• Dierdorff, E. C., Surface, E. A., & Brown, K. G. (2010). Frame-of-reference training effectiveness: Effects of goal orientation and self-efficacy on affective, cognitive, skill-based, and transfer outcomes. Journal of Applied Psychology, 95, 1181–1191.10.1037/a0020856
   PubMed Web of Science ®Google Scholar
• Feingold, A. (2009). Effect sizes for growth-modeling analysis for controlled clinical trials in the same metric as for classical analysis. Psychological Methods, 14, 43–53.10.1037/a0014699
   PubMed Web of Science ®Google Scholar
• Gaugler, B. B., & Thornton, G. C. (1989). Number of assessment center dimensions as a determinant of assessor accuracy. Journal of Applied Psychology, 74, 611–618.10.1037/0021-9010.74.4.611
   Web of Science ®Google Scholar
• Gorman, C. A., & Rentsch, J. R. (2009). Evaluating frame-of-reference rater training effectiveness using performance schema accuracy. Journal of Applied Psychology, 94, 1336–1344.10.1037/a0016476
   PubMed Web of Science ®Google Scholar
• Ivancevich, J. M. (1979). Longitudinal study of the effects of rater training on psychometric error in ratings. Journal of Applied Psychology, 64, 502–508.10.1037/0021-9010.64.5.502
   Web of Science ®Google Scholar
• Johns, G. (1993). Constraints on the adoption of psychology-based personnel practices: Lessons from organizational innovation. Personnel Psychology, 46, 569–592.10.1111/peps.1993.46.issue-3
   Web of Science ®Google Scholar
• Judge, T. A., & Bono, J. E. (2001). Relationship of core self-evaluations traits – self-esteem, generalized self-efficacy, locus of control, and emotional stability – with job satisfaction and job performance: A meta-analysis. Journal of Applied Psychology, 86, 80–92.10.1037/0021-9010.86.1.80
   PubMed Web of Science ®Google Scholar
• Kraiger, K., Ford, J. K., & Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78, 311–328.10.1037/0021-9010.78.2.311
   Web of Science ®Google Scholar
• Kuvaas, B. (2006). Performance appraisal satisfaction and employee outcomes: Mediating and moderating roles of work motivation. The International Journal of Human Resource Management, 17, 504–522.10.1080/09585190500521581
   Web of Science ®Google Scholar
• Lievens, F., Buyse, T., & Sackett, P. R. (2005). Retest effects in operational selection settings: Development and test of a framework. Personnel Psychology, 58, 981–1007.10.1111/peps.2005.58.issue-4
   Web of Science ®Google Scholar
• McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1, 30–46.10.1037/1082-989X.1.1.30
   Web of Science ®Google Scholar
• Murphy, K. R., & Cleveland, J. N. (1995). Understanding performance appraisal. Thousand Oaks, CA: Sage.
   Google Scholar
• Okoli, C., & Pawlowski, S. D. (2004). The Delphi method as a research tool: An example, design considerations and applications. Information & Management, 42, 15–29.10.1016/j.im.2003.11.002
   Web of Science ®Google Scholar
• Pulakos, E. D. (1986). The development of training programs to increase accuracy with different rating tasks. Organizational Behavior and Human Decision Processes, 38, 76–91.10.1016/0749-5978(86)90027-0
   Web of Science ®Google Scholar
• Putka, D. J., & Hoffman, B. J. (2015). ‘The’ reliability of job performance ratings equals 0.52. In C. E. Lance & R. J. Vandenberg (Eds.), More statistical and methodological myths and urban legends (pp. 247–275). New York, NY: Routledge.
   Google Scholar
• Raudenbush, S. W. (2001). Comparing personal trajectories and drawing causal inferences from longitudinal data. Annual Review of Psychology, 52, 501–525.10.1146/annurev.psych.52.1.501
   PubMed Web of Science ®Google Scholar
• Rhodes, N., & Wood, W. (1992). Self-esteem and intelligence affect influenceability: The mediating role of message reception. Psychological Bulletin, 111, 156–171.10.1037/0033-2909.111.1.156
   Web of Science ®Google Scholar
• Roch, S. G., & O’Sullivan, B. J. (2003). Frame of reference rater training issues: Recall, time and behavior observation training. International Journal of Training and Development, 7, 93–107.10.1111/ijtd.2003.7.issue-2
   Google Scholar
• Roch, S. G., Woehr, D. J., Mishra, V., & Kieszczynska, U. (2012). Rater training revisited: An updated meta-analytic review of frame-of-reference training. Journal of Occupational and Organizational Psychology, 85, 370–395.10.1111/joop.2012.85.issue-2
   Web of Science ®Google Scholar
• Salas, E., & Cannon-Bowers, J. A. (2001). The science of training: A decade of progress. Annual Review of Psychology, 52, 471–499.10.1146/annurev.psych.52.1.471
   PubMed Web of Science ®Google Scholar
• Schleicher, D. J., & Day, D. V. (1998). A cognitive evaluation of frame-of-reference rater training: Content and process issues. Organizational Behavior and Human Decision Processes, 73, 76–101.10.1006/obhd.1998.2751
   PubMed Web of Science ®Google Scholar
• Schmidt, F. L., & Hunter, J. E. (1990). Methods of meta-analysis: Correcting error and bias in research findings. Newbury Park, CA: Sage.
   Google Scholar
• Shadish, R. S., Cook, T. D., & Campbell, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Boston, MA: Houghton Mifflin.
   Google Scholar
• Singer, J. D., & Willett, J. B. (2003). Applied longitudinal data analysis. New York, NY: Oxford University Press.10.1093/acprof:oso/9780195152968.001.0001
   Google Scholar
• Spector, P. E., & Brannick, M. T. (2011). Methodological urban legends: The misuse of statistical control variables. Organizational Research Methods, 14, 287–305.10.1177/1094428110369842
   Web of Science ®Google Scholar
• Sulsky, L. M., & Balzer, W. K. (1988). Meaning and measurement of performance rating accuracy: Some methodological and theoretical concerns. Journal of Applied Psychology, 73, 497–506.10.1037/0021-9010.73.3.497
   Web of Science ®Google Scholar
• Sulsky, L. M., & Kline, T. J. B. (2007). Understanding frame-of-reference training success: A social learning theory perspective. International Journal of Training and Development, 11, 121–131.10.1111/ijtd.2007.11.issue-2
   Google Scholar
• Unkelbach, C., Ostheimer, V., Fasold, F., & Memmert, D. (2012). A calibration explanation of serial position effects in evaluative judgments. Organizational Behavior and Human Decision Processes, 119, 103–113.10.1016/j.obhdp.2012.06.004
   Web of Science ®Google Scholar
• Vancouver, J. B., More, K. M., & Yoder, R. J. (2008). Self-efficacy and resource allocation: Support for a nonmonotonic, discontinuous model. Journal of Applied Psychology, 93, 35–47.10.1037/0021-9010.93.1.35
   PubMed Web of Science ®Google Scholar
• Woehr, D. J. (1994). Understanding frame of reference training: The impact of training on the recall of performance information. Journal of Applied Psychology, 79, 525–534.10.1037/0021-9010.79.4.525
   Web of Science ®Google Scholar
• Woehr, D. J., & Huffcutt, A. I. (1994). Rater training for performance appraisal: A quantitative review. Journal of Occupational and Organizational Psychology, 67, 189–205.10.1111/joop.1994.67.issue-3
   Web of Science ®Google Scholar
• Wood, R. E., & Marshall, V. (2008). Accuracy and effectiveness in appraisal outcomes: The influence of self-efficacy, personal factors and organisational variables. Human Resource Management Journal, 18, 295–313.10.1111/j.1748-8583.2008.00067.x
   Google Scholar
• Yang, H., Sackett, P. R., & Arvey, R. D. (1996). Statistical power and cost in training evaluation: Some new considerations. Personnel Psychology, 49, 651–668.10.1111/peps.1996.49.issue-3
   Web of Science ®Google Scholar