No praise without effort: experimental evidence on how rewards affect Wikipedia's contributor community

Abstract

The successful provision of public goods through mass volunteering over the Internet poses a puzzle to classic social science theories of human cooperation. A solution suggested by recent studies proposes that informal rewards (e.g. a thumbs-up, a badge, an editing award, etc.) can motivate participants by raising their status in the community, which acts as a select incentive to continue contributing. Indeed, a recent study of Wikipedia found that receiving a reward had a large positive effect on the subsequent contribution levels of highly-active contributors. While these findings are suggestive, they only pertained to already highly-active contributors. Can informal rewards also serve as a mechanism to increase participation among less-active contributors by initiating a virtuous cycle of work and reward? We conduct a field experiment on the online encyclopedia Wikipedia in which we bestowed rewards to randomly selected editors of varying productivity levels. Analysis of post-treatment activity shows that despite greater room for less-active contributors to increase their productive efforts, rewards yielded increases in work only among already highly-productive editors. On the other hand, rewards were associated with lower retention of less-active contributors. These findings suggest that the incentive structure in peer production is broadly meritocratic, as highly-active contributors accumulate the most rewards. However, this may also contribute to the divide between the stable core of highly-prodigious producers and a peripheral population of less-active contributors with shorter volunteer tenures.

Keywords:

• peer production
• collective action
• volunteering
• Wikipedia
• status
• reputation

Acknowledgments

We thank Damon Centola for helpful discussion, Ori Heffetz, Georgi Kossinets, Michael Macy, Ian Roxborough, and Michael Schwartz for comments, the Wikipedia research community for guidance, and the Wikimedia Foundation for facilitating access to public Wikipedia data. We also express our gratitude to the anonymous reviewers and editor for their insightful comments and guidance. We declare no conflicts of interest and received no funding to perform this research.

Notes on contributors

Michael Restivo is a doctoral candidate at Stony Brook University and will be an Assistant Professor at SUNY Geneseo beginning in Fall, 2014. [email: [email protected]]

Arnout van de Rijt is Associate Professor in the Department of Sociology and Institute for Advanced Computational Science at Stony Brook University. His work on cumulative advantage explores novel empirical strategies for studying success-breeds-success dynamics in various tournaments for resources and status. His research has been published in American Sociological Review and American Journal of Sociology. [email: [email protected]]

Notes

1. We caution the reader that when we use the term ‘less active’ contributors, we are referring their volume of work performed relative to the most prolific contributors. Thus, less active contributors are still quite active overall. The distribution of work in Wikipedia approximates a power-law distribution, with a very long tail of infrequent contributors. However, as Anthony et al. (2009) show, these one-off contributors often add the most reliable contributions to the project, giving Wikipedia its characteristic distinction as the ‘encyclopedia that anyone can edit.’

2. This study's research protocol was approved by the Committees on Research Involving Human Subjects (IRB) at the State University of New York at Stony Brook (CORIHS #2011–1394). The experiment presented only minimal risks to subjects and was designed and conducted so as to be non-disruptive to the community; following the IRB protocol, we maintain the confidentiality of individual records and identities of all research subjects.

3. We tested whether the lack of treatment effect was due to contributors possibly being unaware that they had received a reward. This could happen if they never logged in to Wikipedia after the treatment, in which case they would not have seen the prominently displayed message signaling their editing barnstar award. To explore this possibility, we excluded all subjects who made zero edits post-treatment. This rate of ‘drop out’ varies across tiers: in the 91st–95th percentiles, 67 out of 200 subjects made zero post-treatment edits, while 37 out of 200 subjects made zero post-treatment edits in the 95–99th percentile, and only 19 out of 200 of the top-tier subjects made zero edits after treatment. This is consistent with our intuition that less active contributors are less likely to see their reward because their editing careers are marked by more frequent spells of temporary or permanent discontinuation. After excluding users who made zero post-treatment edits, we again tested productivity across conditions. The results remained consistent: the treatment effect is large and statistically significant only among the top 1% of contributors. The Mann-Whitney U test results were substantively unchanged (100th percentile: z = 2.562, p = .010; 96th–99th percentiles: z = 0.783, p = .436; 91st–95th percentiles: z = −1.184, p = .236).

4. Since the samples were drawn from the same population, we appropriately weighted the data to correct for inter-tier differences in probability of case selection.

5. The amount of work performed on the day immediately prior to treatment was not a statistically significant predictor that an editor would make zero edits post-treatment. We also tested a model where we included the variable for immediacy of feedback as an independent variable predicting edit count. However, there was no difference between control and treatment groups in this regard, which suggests that the treatment effect was not affected by how immediately the treatment came after activity; consequently, we retained this variable as a predictor in only the equation for the zero edit count.

6. Because we are interested in discontinuation, we excluded all editors who had no discernible activity after the treatment date. As discussed above, we cannot be certain that these editors have ever ‘received’ their awards, so we exclude them from this analysis as well.

7. We also model the risk of editors' ending their volunteer tenure, using a Cox proportional hazards model (Andersen & Gill, 1982; Cox, 1972). Using the same predictors of volunteer retention as the regression models estimating productivity (Table 1), we find that contributors in the lower tiers who received a reward are at a higher risk of ceasing their volunteer tenure. The results confirm the robustness of what we visually represent in Figure 3: the treatment effect in the lower tiers is a higher risk of contributors discontinuing their efforts, whereas in the treatment effect in top tier yields a lower risk. These results are available upon request; we omit the full analyses here because of space considerations.

8. Hill et al. (2012) also note that users often receive barnstars at periods of local maxima in their contribution histories. Our research suggests that it is also possible that the receipt of social recognition for one's work may not merely occur at a local maxima, but may also create such local maxima if contributor activity declines after receiving a barnstar.