Creation and Validation of the Performance Appraisal Motivation Scale (PAMS)

ABSTRACT

The current study investigated the effects of rater motivations on performance appraisals (PA). Despite rater motivations being important to PAs, there is no established and validated scale to measure rater motives. Within this study, a scale was developed to measure five rater motives. This scale, labeled the Performance Appraisal Motivation Scale (PAMS), was then examined for internal structure and validated using data from current managers. PAMS exhibited acceptable internal structure and displayed an expected pattern of relationships with individual difference variables, situational variables, other rater motives, and rating outcomes. Overall, results provide evidence in support of PAMS’ construct validity. The findings also help establish empirical linkages between rater motivations and important variables within the nomological network of rater motivation.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

¹ It is useful to note that Harris (1994) discussed how motivations might impact behavior not only during the appraisal process but also prior to when observing and encoding information. We do not argue against this. However, for the sake of measurement, the focus is on motivation proximal to the rating process.

² An expert reviewer suggested that the current taxonomy may be flawed because it was based on past conceptualizations of rater motivations from the extant literature. This might therefore fail to fully capture all possible rater motivations, and we admit there may be additional motivations beyond these that are important in the PA context. However, our approach was one of triangulation – trying to identify the communalities across existing PA motivation conceptualizations and rater motive scales. While our own work may have benefited by taking a more inductive approach (e.g., interviewing managers from many organizations), our review of the literature meant drawing from studies that did in fact already do this (e.g., Longenecker et al., 1987; Murphy et al., 2004) and from theoretical summaries based on literature reviews (e.g., Spence & Keeping, 2011, 2013). Thus, while we agree other development approaches could have been performed, we also believe enough research and theory existed to create a set of rater motives that generally encompass the important PA rater motivations.

³ By construct validity evidence, we are working within the unitarian validity framework (e.g., Binning & Barrett, 1989) that there are a variety of sources of validity evidence (e.g., content relevance judgments, correlations with important outcomes, convergent and discriminant correlations) that ultimately contribute to the degree to which evidence supports inferences drawn from test scores.

⁴ Response checks included three instructed responding questions (e.g., “Select the option “agree”) and checks for excessively fast response times. A response time cutoff was determined by having one of the study authors take the survey as fast as possible while still responding purposefully. Respondents who violated any of these checks were removed from the sample.

⁵ The demographic characteristics of MTurk participants are generally similar to US polling samples, with about 12% of respondents being in managerial functions (Huff & Tingley, 2015), and with most participants not using MTurk as a primary income source (Mason & Suri, 2012).