Does Performance-Related-Pay work? Recommendations for practice based on a meta-analysis

Abstract

Performance-related-pay (PRP) is a controversial topic. Views about its impact are mixed. Through a meta-analysis of studies in public administration, we aim to provide an evidence-based answer to the question: Does PRP work? Our meta-analysis finds a statistically significant, positive but small population effect size between PRP and employee and performance outcomes. Subgroup analyses show that PRPs impact is contingent upon the type of outcome, geographical context, government level, and data source rather than universal in nature. Effect sizes decrease when performance outcomes are measured as opposed to employee outcomes; in USA and European contexts compared to Asian contexts; at the local level compared to the federal level; and when multiple source or experimental data are used compared to single source data. Based on our findings and PRP literature, we construct a flowchart to support practitioners in deciding whether PRP may “work” for them while avoiding its many and typical pitfalls.

Evidence for practice

• Our meta-analysis demonstrates that the impact of performance-related-pay (PRP) on employee and performance outcomes in public administration is statistically significant and positive; however, the fact that the effects sizes are small leads us to conclude that PRP is not a “magic bullet”.

• Intriguingly, performance outcomes are less impacted by PRP than employee-related outcomes like work motivation and job satisfaction, implying that PRP might carry some motivational benefits while its benefits for better performance outcomes are questionable.

• PRP seemingly works better in Asia than in Europe or the USA – arguably due to existing predispositions regarding the effects of rewards and monetary incentives – and also seems more effective at the federal than the local government level.

• Much of the small positive impact uncovered in this meta-analysis seems to be the result of the type of data used in studies. Specifically, studies using multiple or experimental data show a trivial and statistically insignificant overall impact. Practitioners thus need to carefully assess how they will evaluate PRPs impact to avoid biases due to data type.

• An evidence-based flowchart including questions at the macro (cultural and societal considerations), meso (specific organizational characteristics), and micro (individual and team behavior) level is presented as a support tool for practitioners in deciding upon and designing PRP systems in their organization.

Keywords:

• Performance management
• performance-related-pay
• motivation
• human resource management
• meta-analysis

Does performance-related-pay (PRP) work? It is a seemingly straightforward question. However, even though governments across the globe have introduced various forms of PRP since the 1980s (Miller, Doherty, and Nadash 2013; Perry, Engbers, and Jun 2009; Taylor and Beh 2013; Wenzel, Krause, and Vogel 2019) – Organization for Economic Cooperation and Development (OECD) reports estimate that two-thirds of civil servants in the developed world are covered by some form of PRP – it is far from clear whether, when, and to what extent PRP actually benefits public-sector employees and organizations.

Nevertheless, PRP has been at the core of many New Public Management and Reinventing Government movements over the past decades, inspired by private sector practices (Pollitt 2011; Pollitt and Bouckaert 2017). Indeed, various forms of PRP have been common practice in the private sector for many decades, particularly at the senior management level. Intriguingly, they have continued as an industry norm despite the turmoil caused by the 2008–2009 global financial crisis and widespread criticism of the “perverse”, unethical, and unintended consequences of high individual rewards tied to the individual and firm-level performance of financial firms (Den Butter, 2009; Tang and Chiu 2003; Zalm, 2009).

Similarly, PRP in public administration has received much critique, despite its popularity. Studies suggest that PRP schemes may “crowd out” prosocial behavior and intrinsic motivation (e.g. Bellé 2015; Bénabou and Tirole 2006). Moreover, simplistic monetary performance schemes may yield unintended consequences such as goal displacement, and “creaming”, “shirking”, and “cherry picking” by managers and employees: strategic behavior prioritizing often rudimentary tasks that satisfy quantitative performance metrics and therefore result in monetary rewards and recognition (e.g. Heinrich and Marschke 2010). Such critiques suggest PRP is about “hitting the target but missing the point”. On the other hand, PRP can help to tackle concerns surrounding goal ambiguity by making it clear what is expected of managers and employees, and how they will be evaluated and rewarded when meeting those expectations. By reducing goal ambiguity, it could be argued PRP also increases employee and performance outcomes (e.g. Jung 2014; Chun and Rainey 2005).

In short: whether, when, and to what extent PRP works in public administration remains unclear. First, empirical studies typically focus on data from one country, one type of public organization, and one type of outcome – insights across these studies are needed. Second, while there have been review or overview studies on PRP in public administration these do not provide a quantitative integration of evidence across studies (Dahlström and Lapuente 2010; Forest 2008; Perry, Engbers, and Jun 2009). Meta-analytical research methods are particularly apt at uncovering insights across studies by synthesizing individual studies findings into an overall analysis, identifying population effect sizes and variation in effect sizes (e.g. George, Walker, and Monster 2019; Gerrish 2016).

Hence, to advance our thinking about the impact of PRP and formulate evidence-based recommendations for practitioners, we conduct a meta-analysis of 48 effect sizes from 13 public administration studies on PRP and employee and performance outcomes. While meta-analytical evidence on PRP has been published previously in public administration (Weibel, Rost, and Osterloh 2010), it is important to emphasize that the authors here relied on nonpublic administration evidence mostly coming from management or economics journals and the private sector – thus making our meta-analysis stand out by focusing exclusively on public administration studies.

Employee outcomes are defined as all outcomes experienced by public-sector employees, and can be clustered as motivation-related variables (i.e. work motivation, intrinsic motivation, extrinsic motivation), employee attitudes (i.e. justice and fairness perceptions, and change acceptance) and general employee outcomes (i.e. self-efficacy, autonomy, commitment, Organizational Citizenship Behavior (OCB), empowerment, satisfaction, clarity, and turnover intention). Performance outcomes relate to both the individual performance of public-sector employees and the organizational performance of public organizations. Finally, PRP is defined as any type of monetary reward incentive scheme tied to the performance of individual public-sector employees, formally embedded in performance appraisal systems.

Incentivizing performance in the public sector

Practitioners and academics alike have long wrestled with the question of how to properly motivate and incentivize government employees and their managers to put in the required effort to perform their tasks; behave ethically; and display the right amount of creativity required for the task at hand.

Indeed, negative cliché-type imagery of Machiavellian, conservative, and self-interested “bureaucrats” motivated by job security and regular office hours has long dominated both popular and academic discourse (e.g. Niskanen 1971; Van der Wal 2013). However, the recent proliferation of studies into public service motivation (PSM) emphasizing altruism and a desire to work for the greater good, display a countermovement to this cynical discourse (e.g. Perry, Hondeghem, and Wise 2010; Ritz, Brewer, and Neumann 2016).

These overview studies show that most public sector employees possess high levels of PSM. Moreover, fostering PSM results in higher performance and job satisfaction. As mentioned before, various scholars have argued that overemphasizing extrinsic and monetary incentives may not necessarily sit well with more altruistic motivations.

Context matters

However, emerging non-Western scholarship shows public sector employees in those contexts value extrinsic rewards as much as intrinsic ones (Van der Wal 2015; Van der Wal, Mussagulova, and Chen 2021). Indeed, motivational dynamics differ substantially between Western, developed settings and non-Western, often developing countries. Moreover, motivational mixes may differ within these clusters, as crude labels like “Western” and “non-Western”, or “developed”, and “developing”, depreciate their enormous diversity, even within the same geographical region (Van der Wal, Mussagulova, and Chen 2021). Stereotypes affecting administrative reform paradigms of which PRP has been a prominent feature are hard to refute as research into performance, motivation, and incentives in public administration is almost exclusively Western (Ritz, Brewer, and Neumann 2016).

Rare studies in developing or “multiple incentive” settings (Perry 2014) yield mixed results. In some cases, public servants prioritize monetary drivers like opportunities to earn additional income and love of money. Indeed, PRP and related monetary performance incentives have been implemented more vigorously and enthusiastically in Asian administrations compared to Western ones (e.g. Aoki and Rawat 2020). Intriguingly, in other cases, particularly under conditions of low pay, prosocial drivers matter even more than in Western settings (e.g. Banuri and Keefer 2016; Chen and Hsieh 2015; Houston 2014; Liu and Perry 2016).

In addition to the country context, the level of government might also matter for the extent to which PRP is effective, and in what way. So far, studies have produced mixed evidence about the extent to which PRP – often implemented as part of larger administrative reforms – is well received by federal employees (Bertelli 2006; Rainey and Kellough 2000) vis-à-vis state or local agencies (Anderfuhren-Biget et al 2010; Nigro and Kellough 2000), while crowding out intrinsic drivers, occurring mostly at the highest levels (Bertelli 2006).

Conclusively, whether, when and to what extent PRP actually works in public settings is far from settled, and a meta-analysis is needed to inject substantial evidence into this debate.

Methods and results

Data collection

We conducted a systematic literature review process to identify public administration studies focused on PRP in relation to employee or performance outcomes. First, we used Web-of-Science with the following keywords in a topic search (which includes the article title, abstract, and keywords) on September 7th 2019: performance-based pay, performance-related pay, performance-based compensation, performance-related compensation, pay for performance, merit-based pay, merit-based compensation, merit related pay, merit related compensation, and pay for merit. Only English-language journal articles from Social Sciences Citations Index-classified journals in the Public Administration category of Web-of-Science were included (all of these are filter options within Web-of-Science). This resulted in a list of 93 potentially relevant articles.

Second, we scanned the titles and abstracts of these articles to identify whether these indeed focus on PRP in relation to employee or performance outcomes and provided quantitative evidence (which could be observational or experimental). This resulted in a list of 39 articles. Finally, these articles were read in full to see whether they qualified for the meta-analysis.

In the end, 13 studies remained which cluster 48 effect sizes between PRP and employee or performance outcomes. Most of the studies were observational (11) but we also had two experimental studies in our sample. Studies we excluded from our sample focused on the adoption of PRP as opposed to its impact. In order to be included, studies needed to:

• Present quantitative evidence usable for meta-analysis (either correlations or coefficients) and

• Clearly examine PRP in relation to employee or performance outcomes – e.g. not public service motivation as this is typically considered an antecedent of these outcomes, finally studies needed to draw on an actual, real sample – actual managers, employees, people working by and large in the public sector.

We provide a list of the included articles (and their characteristics) in Table 1. As can be seen from this table, our dataset includes a wide variety of countries (including European and Asian countries, and the USA), organizational settings (including local and federal governments), and respondents (including managers and employees).

Table 1. Included studies and their characteristics.

Authors (year)	Number of effect sizes	Average effect size	95% CI	Weight in overall analysis	Country	Level
Anderfuhren-Biget et al (2010)	4	0.059	[0.027, 0.091]	7.73%	Switzerland	Civil servants in municipalities
Andersen, Boye, and Laursen (2018)	8	0.039	[-0.036, 0.115]	7.70%	Denmark	Principals and employees in schools
Bertelli (2006)	2	0.070	[0.021, 0.118]	7.72%	USA	Employees of the USA Treasury
Campbell (2018a)	2	0.430	[0.352, 0.507]	7.70%	Korea	Civil servants employed in central government ministry head offices
Campbell (2018b)	1	0.030	[−0.048, 0.108]	7.70%	Korea	Employees across 16 central government ministries
Choi and Whitford (2017)	6	−0.043	[−0.048, −0.039]	7.74%	USA	Employees of federal agencies represented on the President’s Management Council
Daley, Vasu, and Weinstein (2002)	5	−0.044	[−0.133, 0.045]	7.69%	USA	North Carolina county social service professionals
Hodgkinson et al. (2018)	4	0.143	[0.026, 0.261]	7.65%	England	Facility managers in local government-owned leisure facilities
Jacobsen and Jensen (2017)	3	0.084	[−0.016, 0.185]	7.67%	Denmark	University students with work experience (experiment)
Lee (2019)	4	0.783	[0.776, 0.790]	7.74%	USA	Federal employees taking part in Merit Principle Survey
Stazyk (2013)	4	0.005	[−0.083, 0.093]	7.69%	USA	Managers in USA local government jurisdictions
Taylor and Beh (2013)	2	0.100	[−0.010, 0.210]	7.66%	Malaysia and Australia	Federal government employees
Weibel, Rost, and Osterloh (2010)	3	0.273	[0.128, 0.418]	7.60%	Switzerland	Part-time executive MBA students (experiment)

The effect sizes we obtained from these articles are r-based effect sizes. Specifically, we looked for correlation coefficients between PRP and one of our outcomes in the specific study. If correlation tables were presented, we took the correlation from there, if not we took the correlation from regression coefficients using the formulas discussed by Ringquist (2013). As a robustness check, we tested whether r-based effect sizes collected from correlation tables versus from regression coefficients significantly differed in our sample – they did not.

Moreover, Figure 1 shows three outlying effect sizes with a particularly high coefficient (i.e. above 0.70). Again, as a robustness check, we ran all analyses again excluding these three effect sizes and similar results emerged. In addition to collecting the r-based effect sizes, we also coded for each article’s specific outcome, country sample, and level of government. Doing so allowed us to identify potential sources of variation in effect, sizes using subgroup analysis. Coding was initially done by the first author, and replicated by the second author for intercoder reliability. No issues emerged in this process.

Figure 1. Forest plot of effect sizes.

Figure 1 offers a forest plot for all identified effect sizes.

Data analysis and results

We conducted a random-effects meta-analysis on the identified effect sizes. Random-effects are preferred over fixed-effects when working with real-world data typically used in our field of study. Importantly, the r-based effect sizes underwent Fisher’s Z-transformation as is recommended for meta-analyses in our field (Ringquist 2013). Figures 2 and 3 offer funnel plots of all effect sizes and below the figures we discuss several tests of publication bias. Table 2 shows the results of the analysis. Importantly, we used a subgroup analysis to identify sources of variation in effect sizes and not a meta-regression. This choice resulted from the limited amount of data which hampers the potential of meta-regression. In such instances, subgroup analysis can provide an initial exploration of potential sources of variance. However, an additional robustness check including a meta-regression with one moderator at a time showed the same significant moderators compared to our subgroup analysis.

Figure 2. Funnel plot of effect sizes.

Figure 3. Contour funnel plot of effect sizes.

Table 2. Meta-analysis results.

Data	Population effect size	95% CI	z
All effect sizes (48)	0.130	[0.032, 0.228]	2.60**
1 average effect size per study (13)	0.149	[−0.150, 0.447]	0.97
Which outcome?
Work motivation (6)	0.139	[−0.219, 0.498]	0.76
Other employee outcomes^a (15)	0.138	[−0.066, 0.342]	1.32
Performance^b (11)	0.087	[−0.040, 0.215]	1.34
Intrinsic motivation (4)	0.167	[−0.065, 0.399]	1.41
Extrinsic motivation (2)	0.130	[0.047, 0.212]	3.08**
Employee attitudes^c (10)	.141	[−0.088, 0.370]	1.20
Which region?
US (21)	0.136	[−0.012, 0.284]	1.80^+
European (22)	0.098	[0.068, 0.128]	6.38***
Asia (5)	0.222	[0.013, 0.431]	2.09*
Which level of government?
Federal (17)	0.244	[0.079, 0.408]	2.90**
Local (25)	0.044	[0.020, 0.068]	3.56***
Other (6)	0.181	[0.041, 0.321]	2.53*
Which type of data?
Single source (22)	0.247	[0.145, 0.348]	4.772***
Multiple source/experiment (26)	−0.015	[−0.031, 0.002]	−1.779^+

⁺p < .10, * p < .05, ** p < .01, *** p < .001

^aThis includes self-efficacy, autonomy, commitment, OCB, empowerment, satisfaction, clarity, and turnover intention (reverse coded).

^bThis includes individual and organizational performance.

^cThis includes justice and fairness perceptions, and change acceptance.

Both the funnel plot and the contour funnel plot display observations in every quadrant of the plots, which is an argument against publication bias. However, while the funnel plots provide visual tests for asymmetry (which are sometimes hard to interpret as clear evidence for no publication bias), the Egger test (using the metabias command in Stata) provides a statistical test. The test confirms that we cannot reject the null hypothesis of no small-study effects (p = .418), arguing against publication bias. Taken together with our visual tests, this leads us to say that it is unlikely that publication bias is of strong influence on our findings (which is supported by the many small effect sizes and non-significant coefficients we came across when analyzing the studies).

Next, we calculated the overall population effect size. This showed that PRP has a statistically significant, positive, and small impact (<0.20) on the identified employee and performance outcomes. This analysis was replicated but this time on the average effect size per study to control for studies grouping large numbers of effect sizes. The strength of the population effect size remained the same, although significance was lost. Finally, three subgroup analyses were conducted – based on the type of outcome, region, and level of government.

These subgroup analyses demonstrated that PRP has a weaker positive impact on performance compared to the other outcomes, that PRP’s positive impact in Asia is stronger than in Europe and the US, and that PRP has a stronger positive impact at the federal level of government compared to the local level. Importantly, for Asian countries and federal levels of government, the strength of the impact of PRP actually becomes closer to moderate than small (>0.20). The data show effect sizes coming from a single source are much stronger compared to those coming from multiple or experimental sources, which indicates that much of the (still small) positive impact of PRP identified in the other analyses could actually be the result of common source bias that inflated correlations.

Evidence-based recommendations

Our study aimed to answer the question: Does PRP work? As often in our field of study, the answer is not as straightforward as one might hope. Indeed, PRP yields benefits at the federal government level, in Asian countries, and for employee outcomes. However, in general, its impact is small at best.

So, what do these findings imply for public managers (thinking of) using PRP in their organizations? Based on our observations, inspired by other systematic reviews on PRPs impact in public organizations (Dahlström and Lapuente 2010; Forest 2008; Perry, Engbers, and Jun 2009; Weibel, Rost, and Osterloh 2010), we constructed an evidence-based support tool for practice (shown in Figure 4). Our flowchart acknowledges the inherent multilevel nature of public management by focusing on macro, meso, and micro-level questions public managers ought to reflect on when thinking of using PRP. Rather than being an exhaustive checklist, this flowchart offers useful points of orientation for a comprehensive discussion on the value of PRP in public organizations.

Figure 4. A flowchart for designing effective PRP systems.

It is important to emphasize this is a preliminary meta-analysis based on a first set of public administration studies on PRP. Ideally, a meta-analysis also includes meta-regression. Due to the limited dataset, we conducted subgroup analyses instead. While it is important to take stock of PRP literature for practical as well as scholarly purposes, we acknowledge more work is needed to allow us to make broadly generalizable statements on the impact of PRP.

Nonetheless, the face validity of our findings is high, adhering to our theoretical expectations while also being supported by meta-analyses from other disciplines (e.g. Pham, Nguyen, and Springer 2021). To conclude, while PRP is not a magical bullet for increasing employee and performance outcomes, we hope our flowchart can support strategic thinking and more careful implementation of PRP, aiding public managers to move beyond adopting PRP because it seems fashionable by acknowledging key environmental, organizational, and individual considerations.

Avenues for future research

Based on the meta-analysis, we suggest three avenues for future study. First, more comprehensive comparative survey studies and case studies should be conducted into the nature and origin of the variation in PRP impact between Western and Eastern public organizations. Recent studies into the extent to which reward reforms have been introduced in Asian countries (Aoki and Rawat 2020), and motivational profiles and cultures for civil servants are more extrinsically oriented in the East compared to the West (Mussagulova and Van der Wal 2021) provide rudimentary input for building theoretical propositions for the variations found in our study.

Second, conceptual and empirical endeavors are needed to aid scholars in distinguishing more specifically between different types of (desired and intended) performance outcomes, ranging from motivational and HR-related employee outcomes to quantifiable, results-oriented outcomes. Indeed, the results of the current study show how the impact of PRP varies significantly for various types of performance outcomes. In addition to quantitative studies, qualitative research in particular can further situate how and why this impact varies in relation to the applied performance regime and its underlying performance logics (see, Virani and Van der Wal 2023).

Third and final, the context of PRP seemingly becoming a less widespread practice in the private sector due to increased public scrutiny and the move toward environmental stewardship, social responsibility, and corporate governance (ESG) based reporting and accountability expectations, merits a new wave of comparative studies into PRPs performance impact across sectors. Traditional monetary rewards tied to individual (financial) performance may become less important and desirable over time, which begs the intriguing question regarding what business may learn from the government here, rather than the other way around.

Disclosure statement

No potential conflict of interest was reported by the author(s).