Payment schemes in online labour markets. Does incentive and personality matter?

ABSTRACT

Online labor markets have gained significant importance in recent years, drawing considerable attention in academia and practice. These platforms enable workers worldwide to sell their labor services to a global pool of clients. However, the challenge lies in motivating workers effectively to enhance their productivity. To address this issue, we employ the self—determination theory and present a model that elucidates motivation's impact on productivity across various payment schemes. Additionally, we leverage the psychological trait theory and its suggested taxonomy to explore how compensation policies in online labor markets affect incentives differently based on individual differences. Our experiment tests predictions from a formal labor supply and productivity model for workers with varying compensation levels. The results indicate that intrinsic workers exhibit higher productivity when bonus rewards are introduced. Furthermore, our study confirms the presence of heterogeneous personality effects, emphasizing that increased worker productivity is primarily associated with conscientiousness and agreeableness traits. These findings illuminate the intricate mechanisms governing worker motivation and engagement in paid crowdsourcing environments. They provide valuable theoretical and managerial insights for researchers and crowdsourcing practitioners aiming to enhance worker productivity in online tasks.

KEYWORDS:

• Productivity
• incentives
• online labour markets
• crowdsourcing
• personality

JEL CODES:

• C93
• D01
• J3
• J40
• D26

1. Introduction

Online labour markets are rapidly growing means for companies and individuals to seek, find and hire workers with a particular skill set to complete a variety of online jobs (Randhawa, Wilden, and West 2019). These platforms that facilitate the connection between employers and workers to complete crowdsourcing tasks. Following this way, crowdsourcing communities can provide organisations with a cost-effective solution to exploit the ‘wisdom of crowds’ (Autor 2001; Howe 2008). This new concept of crowdsourcing, as an emerging IT innovation which promotes development more easily, efficiently and without boarder restrictions, attracts growing attention from scholars and practitioners (Mourelatos and Tzagarakis 2018; Ye and Kankanhalli 2015).

In fact, crowdsourcing workers possess the autonomy to choose tasks that are of interest to them and exert effort levels according to their discretion. Consequently, the efficacy of crowdsourcing is reliant upon optimal task design and high levels of public engagement to ensure quality outcomes (Li et al. 2022). As the online environment becomes increasingly saturated with a plethora of options for workers to allocate their limited resources, such as time, expertise, and skills, these resources are becoming scarcer for task requesters. Individuals’ involvement in crowdsourcing tasks is contingent and dynamic. Therefore, it is crucial to attract the participation of the crowd and incentivize their commitment to input additional resources (Chandler and Kapelner 2013). This requires a comprehensive comprehension of their motivation and an understanding of why people behave as they do, thus allowing organisations to create crowdsourcing tasks in a suitable and effective manner. This necessitates a holistic approach to designing crowdsourcing tasks that incorporates insights from motivational research to maximise engagement and output from participants. These factors have emerged as critical determinants of success in crowdsourcing endeavours.

Driven by that, our study focuses on online labour marketplaces, because they differ from their traditional counterparts in a few important aspects. A major difference is that the global reach of these platforms enables employers to access a larger pool of workers than before and enjoy a wider diversity of hiring choices (Brynjolfsson, Hu, and Smith 2003). Another notable difference lies in the offered wages. Because employers are uncertain ex ante whether the workers, they hire can offer high or low quality of work, economic theory predicts that employers are likely to offer wages between the wages of high-quality workers and those of low-quality workers. In turn, high quality workers are likely to exit the labour pool, causing wages and labour quality to spiral downwards (Farrell, Grenier, and Leiby 2017). In this rationale, the low compensation rates of some online labour markets reflect the low quality of work by the labour pool. Research has not yet addressed these fundamental issues and nowadays, there is a growing debate on the determinants of online productivity and its relationship with the offered reward (Horton and Chilton 2010; Hua et al. 2017; Syverson 2011). Also, purchasers of labour in these markets confront severe adverse selection issues – including uncertainty about workers’ effort and working behaviour. Thus, despite its success, this new form of ‘peer production’ faces several practical challenges in relation to workers behaviour (Beck et al. 2022).

To date, there are body of studies related to crowdsourcing workers’ incentives conceptually and empirically (Martinez 2017; Rogstadius et al. 2011), by analysing mainly survey data. These papers have primarily focused on examining the influence of perceived benefit on individuals’ motivation to participate and engage in crowdsourcing. Recent research, though, has also highlighted the interconnectedness of motivation and personality in influencing working behaviour. Pee, Koh, and Goh (2018) and Badura et al. (2020) have reported on findings that suggest a relationship between motivation and personality traits. Additionally, research by McIlroy et al. (2015) and Donato et al. (2017) have further emphasised the importance of personality in shaping working behaviour. Furthermore, Butschek et al. (2022) have noted that the relationship between personality and motivation is well-established, with personality traits being a significant predictor of motivation. Yet there are few attempts that identify and experimentally link incentives with online productivity under difference payment schemes. This represents an important gap in the literature, as understanding the impact of incentives on productivity and how this may differ based on individual characteristics is critical for the effective management of crowdsourcing tasks.

Motivated by these knowledge gaps, this research aims, experimentally, to address the following questions: (i) How does the motivation of a worker drive online productivity? (ii) what are the differential effects of intrinsic and extrinsic incentives on online productivity under different payment schemes? (iii) To what extent does personality moderate the relationship between incentives and workers’ inputs in crowdsourcing tasks?

Drawing upon the self-determination theory, we put forth a conceptual framework to elucidate the impact of motivation on productivity under varying payment arrangements. Moreover, we leverage the psychological trait theory and its prescribed typology to infer whether remuneration policies in online labour markets differentially impact incentives based on individual differences. To directly examine these relationships, we used a pure productivity-computational task. Our paper is based on data drawn from an experiment designed in Amazon Mechanical Turk, which measures and assesses personality and motivation parameters to directly predict workers’ productivity under several performance-pay and fixed term wage scenarios (Goncalves et al. 2015; Rogstadius et al. 2011). We propose that online workers have different motivations than do other workers – including different concerns about wages and different degrees of intrinsic enjoyment of computer tasks. During the first experimental round, with fixed payments for all participants, intrinsic workers, were performing worse than extrinsic ones. After our bonus payment -per -correct -answer treatment took effect in the second round, the intrinsic workers increased their performance by approximately 3%. Also, when we took into account that, intrinsic motivation may be confounded by other personality traits, several effects of personality traits were revealed. Our attempt represents a significant contribution because employers-requesters are naturally interested in the extent to which the working behaviour of employees - workers change during an online job and how. Also, a better understanding of wage policies in online environments will enhance their ICT components, making them more efficient and providing a positive effect on online labour productivity (Relich 2017).

To the best of our knowledge, ours is one of the first studies directly aimed at unbundling, initially, the relationship between incentives and online working behaviour, under different payment conditions within an experimental framework, by exploring the self-determination theory as a basis for interpretation. A second objective is to reveal further evidence that personality traits are fundamental drivers of online working economic decisions. However, incentive-enhancing preferences are irreducibly heterogeneous and their link with non-cognitive behaviour needs further investigation (Bowles, Gintis, and Osborne 2001; Huffman and Bognanno 2018; Lindqvist and Vestman 2011). Hence, by taking all the above into consideration, our findings support Horton’s and Jain and Parkes’ arguments, that a further investigation in online labour markets is needed and advances in crowdsourcing as a methodology will require broadly applicable, performance predictive models, for revealing the key role of personality heterogeneity in explaining wage dispersion in online job environments.

2. Theoretical background

2.1 Compensation policy and worker performance

The role of compensation policy in influencing worker performance is still extensively under investigation by both economists and psychologists. Of particular interest in recent empirical research are the effects of payment schemes on worker productivity, often referred to as ‘incentive effects’. More specifically, there is a small body of empirical literature investigating whether paying workers on the basis of output induces them to produce more (i.e. performance pay policy) (Franceschelli, Galiani, and Gulmez 2010). Thus, the primary motivation behind introducing a piece rate scheme is, firstly, to further engage workers by increasing their efforts (Lazear 2000). Standard economic theories and empirical papers have pointed out that without adopting performance pay policies, extrinsic incentives erode intrinsic, and predict higher performance (Lazear 1992). However, recent research also revealed evidence that, under several working conditions, a link can establish between intrinsic motivation and performance (Cerasoli, Nicklin, and Ford 2014).

It is generally believed that an easy way to increase an individual’s productivity is to pay based on measured performance. Previous studies however have pointed out several difficulties with pay-for-performance wage structures in terms of measurement.¹ The lack of data and the difficulty to find appropriate conditions in which some workers are in a performance-pay state, while others are paid a fixed amount regardless of productivity level² bring complex issues into the examination of the choice of payment scheme and its effect on individuals’ outputs in relation to cognitive, personality and demographic characteristics (Dohmen and Falk 2011; Franceschelli, Galiani, and Gulmez 2010; He, Qiu, and Cheng 2022; Lazear 2000). The analysis in this study does not suffer from this traditional problem. Our paper is based on data drawn from an experiment designed in an online labour market, which measures and assesses personality and motivation parameters to directly predict workers’ productivity under several performance-pay and fixed term wage scenarios (Goncalves et al. 2015; Rogstadius et al. 2011).

2.2 Motivation crowding out

Motivation crowding theory was developed to examine how worker utility functions are determined. This theory suggests that an external intervention via monetary incentives can undermine and -under different identifiable conditions strengthen- intrinsic motivation (Frey and Jegen 2001). Within this theoretical context, through a field experiment Kuvaas et al. 2017 showed that intrinsic motivation is positively related to affective commitment, resulting in positive employee outcomes. In the same direction, research studies investigating traditional labour markets confirmed the productivity crowding-out effects of intrinsic incentives and the role of workers’ self-selection mechanism in the labour markets (Barigozzi and Burani 2016; Barigozzi, Burani, and Raggi 2018; Cunyat 2019; Kim 2020). Lastly, Cassar and Meier 2018, have shown that nonmonetary intrinsic incentives and nonpecuniary aspects of one's job have substantial impacts on job satisfaction, productivity, and labour supply, meaning that, for many workers, work is a source of meaning. But do all the above findings also apply to online labour markets? Several papers have examined the link between incentives and compensation policies in online labour platforms from both economic and IT perspectives. For example, according to Faradani, Hartmann, and Ipeirotis 2011, just increasing the reward for a task decreases demand for it. High reward tasks are a signal for complex and involved tasks that decrease the utility of high rewards for the worker. Furthermore, overpriced tasks are also undesirable since requesters may invest excess capital qualifying a deluge of data from eager respondents (Gonen et al. 2014). More studies have also revealed that an unconditional reward policy can play an important role in productivity outcomes only for extrinsic driven participant workers (Dube et al. 2020; Liang et al. 2018). Improper pricing or scheduling often results in task starvation and loss of capital for these markets (Wu 2019). Consequently, the literature tries to explore the psychological determinants³ of workers’ underlying incentive mechanism (Benabou and Tirole 2003; Kreps 1997), and how the crowding-out effect holds within a working environment, especially for intrinsic driven individuals, resulting in higher productivity and efficiency levels (Frey and Jegen 2001). While previous studies have examined the influence of motivation across different remuneration frameworks, such analyses have tended to adopt a relatively stable approach. When payment schemes are altered dynamically during a job task, individuals with different types of motivation may exhibit varying levels of adaptability to these changes. For example, individuals with intrinsic motivation while may demonstrate greater flexibility in their working behaviour, rewards can have a detrimental effect on their performance towards the task (Hendijani et al. 2016). On the other hand, those with extrinsic motivation may be more responsive to changes in compensation (Benabou and Tirole 2003). Understanding these dynamics is critical to developing compensation schemes that are aligned with individuals’ motivations, which can ultimately lead to improved task performance and job satisfaction. Motivation is a dynamic process. For that reason, our research utilises a dynamic approach in which compensation is adjusted during the crowdsourcing process over the course of two experimental rounds.

2.3 Personality traits

In recent years, economists have been using psychological skills (personality traits and their relative aspects) to link personality to economic preference parameters which are fundamental drivers of decisions in economic theory (Borghans et al. 2008; Cadsby, Song, and Zubanov 2019; Heckman and Rubinstein 2001; Nyhus and Pons 2005). However, some of the measures used by researchers are likely to suffer from endogeneity issues and require a very careful approach (Drago 2011). For this reason, economists and psychologists have recently employed in their experiments the Five-Factor Model of Personality, which measures the so-called ‘Big Five’ personality traits (Costa and McCrae 1992). This model includes the traits of openness, conscientiousness, extraversion, agreeableness and neuroticism and is very efficient because these traits are unlikely to experience ordinal changes, particularly after school (Roberts and DelVecchio 2000). The stability and robustness of the Big Five Personality Test allowed several researchers to deepen into the study of the determinants of performance in labour markets (Cobb-Clark and Schurer 2012).

Thus, we utilise the psychological trait theory and its suggested taxonomy to draw conclusions on whether compensation policies in online labour markets affect, incentives based on individual differences differently. Personality is already linked with working behaviour in both offline (Cubel et al. 2016; Gensowski 2018; Heckman, Jagelka, and Kautz 2021) and online working environments (Kazai, Kamps, and Milic-Frayling 2011; Mourelatos and Tzagarakis 2016). Thus, we embed the Big Five personality traits measurements in our analysis. Robert’s, (2009) and Roberts and DelVecchio (2000), definitions of personality (i.e. non-cognitive skills) suggest that all psychological measurements are calibrated on measured behaviours or broadly defined ‘tasks’. Thus, in our case, to dynamically monitor participants’ real time response to monetary rewards, we adopted a task with several performance-related pay strategies. This technique is a standard element for behavioural economics researchers and business practitioners, who dynamically investigate and measure various productivity outcomes.

Already in the literature, within traditional (i.e. offline) markets and controlled laboratory environments, several attempts revealed relations between the ‘Big Five’ inventory and earning differences (Fletcher 2013; Nyhus and Pons 2005). For example, there is evidence that conscientiousness and emotional stability (the inverse of neuroticism) have, in general, a robust positive relationship with job performance (Burks et al. 2015; Donato et al. 2017; Hogan and Holland 2003; Witt et al. 2002). There are also some indications of a positive impact of extraversion and openness to experience on wages (Heckman, Jagelka, and Kautz 2021).

Regarding online labour markets, first attempts also give evidence that personality traits are key factors of workers’ behaviour. Mourelatos, Giannakopoulos, and Tzagarakis 2022, using a transcription task, showed that conscientiousness and agreeableness is associated with online job performance, while Kazai, Kamps, and Milic-Frayling (2011; 2012), using a label task, showed a negative effect of extraversion and neuroticism in workers’ job accuracy. Shi et al. (2022), linked conscientiousness, neuroticism and extraversion with solver online engagement. Nevertheless, there is a lack of research providing evidence from labour-oriented experiments on online labour markets (Kazai, Kamps, and Milic-Frayling 2012).

By taking all the above-mentioned into consideration, we, mainly, expect that our treatment will trigger the intrinsically- driven individuals to perform better than the extrinsic ones and personality traits will moderate this effect, especially for individuals who exert high levels of conscientiousness and agreeableness.

2.4 Self-Determination theory

At this point, a theoretical framework to collaborate all the mentioned theories is needed to our research questions to tie together. In this direction, self-determination theory (SDT hereafter) provides a framework for understanding the factors that promote motivation and psychological and behavioural functioning. SDT supports a basic distinction between intrinsic and extrinsic motivation and suggests a mechanism through which the treatment’s effect heterogeneity may have operated: by boosting the autonomous motivation of more intrinsically motivated workers (while reducing the controlled motivation of more extrinsically motivated individuals) (Ryan and Deci 2017). Autonomy involves acting with a sense of volition and having the experience of choice (Ryan et al. 2019). Intrinsic motivation is an example of autonomous motivation because when individuals engage an activity because they find it interesting, they are doing the activity wholly volitionally. Although the theory is psychological, research has also given attention to behavioural underpinnings of these processes and places them in an evolutionary perspective in various fields.

Many authors discuss SDT as a more global theory of motivation for the workplace (Baard, Deci, and Ryan 2004). Although is well established that the use of salient extrinsic rewards to motivate work behaviour can be deleterious to intrinsic motivation, Gagne, Deci, and Ryan (2017), build on the SDT framework and they argue that also alternative ways exist, in which tangible rewards can be used so as not to be detrimental to intrinsic motivation. Employers should be careful not to overuse extrinsic rewards. Too many rewards can undercut intrinsic motivation (a phenomenon known as the over justification effect), yet too few can cause employees to feel unappreciated. Experimental laboratory studies in social psychology have already pointed out that when individuals are offered performance-contingent monetary rewards, it can make the external controlling aspect of the reward more prominent, which can undermine their intrinsic motivation. This occurs because the emphasis on the external reward can reduce individuals’ sense of autonomy and control over their work, which can diminish their intrinsic motivation (Hagger and Chatzisarantis 2011). In other words, the prominence of the performance-contingent monetary reward can crowd out the intrinsic motivation that individuals may have had for the task, leading to a decline in overall motivation and performance.

Recent experimental research on gig and online economies highlights the need for personalised payment schemes that consider individual differences in intrinsic motivation and task preferences, and suggests that the relationship between intrinsically motivated workers, extrinsic rewards, and productivity is complex and may vary depending on individual factors. Butschek et al. (2022), found sizeable heterogeneity effects on online productivity with respect to intrinsic motivation and personality: Their proposed treatment⁴ reduced performance for less intrinsically motivated workers and it increased performance among conscientious workers with a high task motivation. By drawing on this evidence and relating it to the literature in psychology, Figure 1 suggests that economic decisions are strongly influenced by a rich interplay of preferences, biases, and skills as well as the other factors.

Figure 1. Determinants of online job performance.

3. Method

3.1 Amazon mechanical Turk

We recruited participants from the U.SA. through Amazon Mechanical Turk, an online labour market (AMT hereafter). AMT involves two major categories of participants. Employers, known as requesters, who post online jobs⁵ for potential individuals to do for pay, and individuals, known in bibliography as ‘Turkers’ who work in task-based labour in exchange for a wage set by the requesters. AMT’s price policy allows requesters either to set a flat rate wage strategy or a piece rate reward and hire Turkers using several criteria (i.e. Turkers who have previously worked in a number of HITs or have a general task completion approval rate over a set percentage). The general workflow of this crowdsourcing platform initially follows a requester, who posts the main task (i.e. which may consist of several smaller microtasks - HITs) under a set wage strategy. Depending on their work preferences, Turkers browsing offered jobs then choose the HITs they will take part in. After that, requesters review the work submitted by each participant, and accept or reject it, depending on whether its quality meets the job requirements set at the beginning of the task. In the end, AMT delivers the requester’s posted payment to the approved Turkers. The overall HITs workers complete over their working lifetime contributes to their approval rate. This index serves as a type of reputation score, offering requesters a proxy comprehension of Turkers’ work ethics (Kingsley, Gray, and Suri 2015). Thus, AMT’s design, which shapes market dynamics as an online labour market, was very compatible with our experiment’s workflow and characteristics, covering the basic components of job performance. Numerous studies have already shown that data gathered using AMT are no less quality than data collected on the standard physical lab (Arechar, Gächter, and Molleman 2018; Goodman and Paolacci 2017; Horton, Rand, and Zeckhauser 2011; Paolacci, Chandler, and Ipeirotis 2010).

3.2 General experimental design

The quasi-experimental sessions were divided into two rounds with an obligatory break in the middle. Each phase of the experiment had a duration of 150 s while the break had a duration from 120 to 150 s. In the instructions of the task, we informed potential participants that the reward was 1 $ for task completion (piece-rate pricing) and the total duration of the experiment, including the survey, was approximately ten minutes. To avoid self-selection bias, the offered wage was in line with AMT’s price policy. The real effort task uses the timed mathematical additions task of Niederle and Vesterlund and the first of which functioned as a demonstration and attention check.⁶ The task was programmed using zTree (Fischbacher 2007) and has characteristics that allow us to measure productivity as a function of cognitive abilities, soft skills and demographic characteristics, reflecting on an individual’s level of concentration, effort, stress management and adjustment. The numerical additions were designed to be reasonably simple, if repetitive. Earlier studies have deemed this a good measurement of intellectual effort and individual productivity (Niederle and Vesterlund 2007). The experiment conducted with a two-period design, to provide us with a baseline measure of online performance capturing task characteristics, intrinsic motivation to perform the task and with a measure of whether treatments affected intrinsic motivation (which can only be captured in a subsequent period without rewards) (Bradler, Neckermann, and Warnke 2019). To assure data quality, we set 2 default criteria for workers to have the opportunity to be hired in the experiment: 80%.⁷ success rate in their previous task completion activity and participation in at least 50 approved tasks in AMT (Peer, Vosgerau, and Acquisti 2014). We collected data from 465 workers only from the USA and 19,407 responds (Table 1).⁸

Table 1. Experimental design.

			Task
			T1 Period (120 s)	Break (120–150 s)	T2 Period (120 s)
Groups	Observations	Survey	Round 1	Treatment	Round 2	Payout Function
Benchmark	90	Yes	1 $ Flat Rate	Only Video/No Bonus Information	No Bonus	1 $
2	100	Yes	1 $ Flat Rate	Video with Bonus Information	1 cent/piece rate	1 $ + 0.001 * (Sum of Round 2 Correct Answers)
3	90	Yes	1 $ Flat Rate	Video with Bonus Information	5 cent/piece rate	1 $ + 0.005 * (Sum of Round 2 Correct Answers)
4	87	Yes	1 $ Flat Rate	Video with Bonus Information	10 cent/piece rate	1 $ + 0.010 * (Sum of Round 2 Correct Answers)
5	98	Yes	1 $ Flat Rate	Video with Bonus Information	15 cent/piece rate	1 $ + 0.015 * (Sum of Round 2 Correct Answers)

Notes: Authors’ calculations.

Lastly, we also run power calculations to overcome potential issues regarding the validity of our findings, which confirm that the sample size of the study is adequate to provide results with the desired power and significance.⁹ (See Appendix B for further discussion).

3.3 Measures

Before the online task commenced, workers were asked to fill in a survey. Firstly, to see how personality traits affect their performance and adaptation to salary changes, we used the Big Five Personality test, a 44-item questionnaire (McCrae and Costa 1999) providing measures for each personality trait i.e. Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism (hereinafter OCEAN¹⁰). The role of personality traits is well established in standard behavioural models on task performance and the adopted OCEAN taxonomy captures individual-specific differences in ways of thinking, feeling, and behaving (Filiz-Ozbay et al. 2018). In addition, this mid-sized test ensures an accurate and stable measurement of each personality facet (Cobb-Clark and Schurer 2012), without being excessively time-consuming, which could result in some measurement bias errors. The Big Five dimensions of personality were estimated on a scale of 1-5, where 1 = disagree, 2 = slightly disagree, 3 = neutral, 4 = slightly agree and 5 = agree. Afterwards, the OCEAN factors were constructed through a factor analysis, to render each trait orthogonal to the rest (McCrae and Costa 1999). To allow for an easier interpretation of our estimates, Big Five scores are standardised to have mean zero and standard deviation of one in all reported specifications.

We also ran a quick test through the survey, to measure participants’ level of computer competence.¹¹ Given the rapid proliferation of computer technologies (ICT), computer skills in particular become even more important for predicting individual performance especially for older subjects, eliminating age-dependent differences.

Our computer competence questionnaire (CCQ), based on the INCOBI computer expertise inventory by Richter,¹² contains 10 three-point Likert Scale items assessing theoretical and practical computer knowledge by allowing participants to express how much they agree or disagree with several computer-related aspects. Our aim was to create a measure of computer competence. The CCQ test demonstrated excellent reliability (Cronbach’s α = .873), with subscale reliabilities ranging from .852 to .870.

One last remark is that performance in the task chosen should also be influenced by the interaction of cognitive abilities with soft skills (Borghans et al. 2008). Therefore, our findings could be merely reflecting that workers with a higher level of cognitive skills also score higher in certain personality facets. For that reason, we include in our survey an IQ test, the Stanford-Binet IQ, which has been shown to include the result of cognitive and non-cognitive abilities. Terman¹³ also adapted the Stanford - Binet IQ test for use with American populations to predict academic performance. After taking into consideration an increasing body of literature that explores the relationship between productivity and a wide variety of economic and demographic outcomes, such as educational attainment and labour income, we also asked participants to report their age, gender, ethnicity, educational level, their mother’s and father’s educational level, their job status and average monthly family income.

3.4 Sample characteristics

Table 3 presents the descriptive statistics for the explanatory variables, which are grouped in the individuals’ demographic characteristics, cognitive skills, socioeconomic characteristics and personality traits.

Regarding demographics, 52.5% of our sample is female subjects, with a mean age of 37 years. Concerning ethnicity, the workers were 83% Whites, 6.9% Black and 10.1% Asian. Regarding their cognitive abilities, 83.2% of our hired crowdsourcing workers have a high IQ level and 87.7% high computer competence. Subjects have a low educational profile (i.e. only 20% have completed college education), but 82% are working (i.e. active labour force status). Lastly, half of our participants have a high family income (i.e. at least 3,000 USD per month) and come from families with parents with at least tertiary education (i.e. 51.1%). Regarding incentives, we constructed a binary variable of intrinsic and extrinsic motivation, grouping workers who reported that they participate in our task for fun, or to contribute to the crowd, or as a way to develop their skills, as intrinsic ones, and workers who participate because it is their full- or part-time job, as extrinsic ones (Liang et al. 2018). Our sample consists of approximately 40% intrinsic and 60% extrinsic participants.

With respect to their personality profile, the five variables of the Big Five Personality test while are designed to give a mean of 50 with a standard deviation of 10 for each trait, we used the normalised version of OCEAN variables in a range of 1–5, where 1 denotes a very low level of the trait and 5 a very high level respectively.

According to the observed variation in personality traits, we see that the mean score for Openness is 3.62, suggesting a high tendency towards creativity and active imagination, for Conscientiousness 4.03, signifying high levels of thoroughness, for Extraversion 2.86, signifying a moderate level of worker energetic behaviour, for Agreeableness 3.84, indicating that our workers seem more empathetic and altruistic and for Neuroticism 2.80, suggesting that workers do not tend to experience negative emotions (Table 2).

Table 2. Sample characteristics.

Variable	Mean	SD	Min	Max	N
Individual Demographic Characteristics
Female (0/1)	0.525	0.499	0	1	930
Age	36.75	10.295	18	72	930
Ethnicity (%)
Whites	0.830				772
Black	0.069				64
Asian	0.101				94
Individual Cognitive Characteristics
High IQ level (0/1)	0.832	0.373	0	1	930
High Computer Competence (0/1)	0.877	0.328	0	1	930
Individual Social Economic Characteristics
High Years of Schooling (0/1)	0.200	0.400	0	1	930
High Family Income (0/1)	0.500	0.500	0	1	930
High Parental Education (0/1)	0.511	0.500	0	1	930
Intrinsic Motivation (0/1)	0.398	0.489	0	1	930
Labor Force Status (%)
Active	0.823				766
Inactive-Unemployed	0.177				164
Individual Personality Traits (Big Five)
Openness	3.616	0.736	1	5	930
Conscientiousness	4.030	0.750	1.333	5	930
Extraversion	2.862	0.906	1	5	930
Agreeableness	3.839	0.701	1.444	5	930
Neuroticism	2.798	0.848	1.25	4.875	930

Source: Dataset with results drawn from Amazon Mechanical Turk. Authors’ calculation.

Notes: The High Parental education variable contains workers both of whose parents have at least tertiary education. The High Years of Schooling variable contains workers with at least college education. The High IQ level variable contains workers with no mistakes in the IQ tests. The High Computer Competence variable contains workers that scored at least 75% in the Computer Competence pre-tests. High Family Income includes workers with a family income of at least 3,000 Dollars per month.

Research assessing characteristics of the MTurk workers as social science research participants concludes that workers represent a large and diverse population, yet MTurk workers are a convenience sample (Paolacci and Chandler 2014). Research on online demographic characteristics has shown that, Amazon Mechanical Turk (MTurk) is a dynamic, ever-changing platform. Based on the latest evidence, approximately 57% of people on MTurk identify as female, 38% mostly 30–39 years old, 79% whites, with 23% at least college education (Coppock 2019; Litman and Robinson 2020). Hence, in relation with our study, we do not observe statistically significant differences between our sample and the general AMT population demographic attributes, confirming the internal validity and reliability of our results (Thomas and Clifford 2017) (See Appendix A for further discussion).

3.5 Empirical model

In this section, we utilise straightforward basic econometric methods to investigate whether the relationship between personality traits, motivation (intrinsic of extrinsic) and productivity is heterogeneous across subjects’ individual characteristics by following Heckman’s conceptualisation (Heckman, Jagelka, and Kautz 2021). We estimated the following baseline regression specification by ordinary least squares (OLS): (1) $\begin{array}{r}{Y}_i=\alpha +\sum _{n=1}^5{\text{β}}^n{N{C}_i}^n+\gamma M_i+{\delta }^m{E{C}_i}^m+\zeta X_i+{\hat{i}}_i+e_i\end{array}$(1) where Y_i is our productivity measurements (e.g. logarithmic values of productivity, payoffs and numbers of total correct answers and responses) by worker, NC_i is a n-vector of non-cognitive skills of worker i (where k = 1, … ,5 corresponds to Openness, Conscientiousness, Extraversion, Agreeableness and Neuroticism, respectively), M_i is the motivation of worker i (i.e. whether worker i’s incentive is intrinsic or extrinsic), EC_i is an m-vector of the experiment characteristics (i.e. where m = 1,2 denotes whether a worker belongs to the benchmark group or the treated group and the time period of the experimental phase, respectively), X_i are personal characteristics (i.e. demographic attributes, cognitive skills and socioeconomic background of worker i), î_i is a vector of location fixed effects and e_i an i.i.d. disturbance term.

Our experimental design assumes that the micro-task designer (requester) is willing to pay a bonus number of monetary values to a specific micro-worker to produce a correct outcome (Y_i), i.e. the total amount of correct adding operations of two-digit numbers. The expected outcome (Y_i) is assumed to be a stochastic function depending on observed productive characteristics (e.g. cognitive and non-cognitive skills), incentive variables (e.g. workers’ motivation in task participation) and other unobserved factors (e.g. innate ability, crowdsourcing experience).

4. Experiment

4.1 First experimental round

4.1.1 Procedure

Participants after accepting the HIT in Amazon Mechanical Turk were redirected through a web-link to the experiment, where they were asked to freely browse the detailed instructions. Once the navigation was completed, participants were asked comprehension questions as an attention check. All participants were asked to fill in the survey through a computer to prevent the use of smaller screen sizes (e.g. smartphones). Being in the actual experiments, in round 1 participants were asked to respond to as many additions of 2-digit random numbers as possible within the timeframe of 150 s.

4.1.2 Measures and task outcomes

The nature of our task allows us to investigate and measure individual productivity in several ways. Our primary outcome measure is the number of total correct answers provided, which is a direct measurement of the performance of a participant worker. Also, performance was measured as the division of total correct answers by the number of responses, effort as the total duration of task engagement in seconds, average payoffs as the mean value of earnings ($) and productivity index as the division of total payoffs by effort. Lastly, we calculated the cumulative number of responses as quantitative labour measurement. Table 3 contains the summary statistics of the abovementioned task outcomes of the first experimental round (Column 2). For the whole sample, we observe an average of 20 correct additions, reflecting an average of 91.5% of task performance. Moreover, workers had an average of 51 s of pure effort on the additions of the two-digits and an average of 22 responses.

Table 3. Task outcomes summary statistics.

	Both rounds	1st round	2nd round	Difference
	[1]	[2]	[3]	[4]: [3]-[2]
Pooled sample
Total Correct Answers	21.27	20.02	22.54	2.522***
Performance (%)	92.5	91.5	93.4	0.019***
Effort (sec)	54.09	51.24	56.95	5.71***
Productivity Index	50.07	34.92	65.19	30.27***
Responses	22.86	21.69	24.04	2.35***
Average Payoffs ($)	0.72	0.00	1.42
N	930	465	465
Not treated
Total Correct Answers	21.69	20.69	22.70	2.01
Performance (%)	91.8	90.8	91.8	0.010*
Effort (sec)	55.14	53.15	57.12	3.97
Productivity Index	33.36	34.07	32.66	−1.41
Responses	23.36	22.47	24.24	1.77
Average Payoffs ($)	0.00	0.00	0.00
N	180	90	90
Treated
Total Correct Answers	21.18	19.85	22.50	2.65***
Performance (%)	92.6	91.6	93.6	0.020***
Effort (sec)	53.84	50.78	56.91	6.13***
Productivity Index	54.09	35.13	72.99	37.87***
Responses	22.75	21.50	23.99	2.49***
Average Payoffs ($)	0.88	0.00	1.76
N	750	375	375
Treated (bonus 1%)
Total Correct Answers	22.05	20.60	23.50	2.10**
Performance (%)	92.3	91.4	93.2	0.018*
Effort (sec)	55.95	52.47	59.44	6.97**
Productivity Index	37.67	37.86	37.48	−0.38
Responses	23.72	22.30	25.13	2.83**
Average Payoffs ($)	0.12	0.00	0.24
N	200	100	100
Treated (bonus 5%)
Total Correct Answers	20.24	19.01	21.47	2.46*
Performance (%)	92.3	91.8	92.8	0.010
Effort (sec)	51.02	47.80	54.23	6.43**
Productivity Index	45.68	36.68	54.68	18.00***
Responses	21.85	20.61	23.09	2.48**
Average Payoffs ($)	0.055	0.00	1.09
N	180	90	90
Treated (bonus 10%)
Total Correct Answers	20.74	19.72	21.76	2.04*
Performance (%)	93.1	92.2	94.0	0.018
Effort (sec)	54.30	51.59	57.02	5.43*
Productivity Index	56.03	34.57	77.25	42.68***
Responses	22.44	21.64	23.24	1.60
Average Payoffs ($)	1.12	0.00	2.25
N	174	87	87
Treated (bonus 15%)
Total Correct Answers	21.53	19.97	23.08	3.11**
Performance (%)	92.7	91.0	94.3	0.031**
Effort (sec)	53.87	51.06	56.69	5.63*
Productivity Index	76.84	31.41	122.28	90.87***
Responses	22.85	21.39	24.32	2.93**
Average Payoffs ($)	1.75	0.00	3.51
N	196	98	98

Source: Authors’ calculations

Notes: The t-test is the statistics of contrast of a test of no-difference in means between workers.

*** Statistically significant at the 1% level.

** Statistically significant at the 5% level

* Statistically significant at the 10% level

4.1.3 Results

Table 4 present baseline estimates for the first experimental round. Model specifications control for several demographic, cognitive and social economic characteristics. It is noteworthy though, that following the recent literature on demographics and online productivity, we observe that female workers had statistically significant deteriorations in work performance and productivity than males, throughout all columns (Cantarella and Strozzi 2022; Straub et al. 2015). While this finding is in line with gender differences in traditional job performance studies, it is the first time that this negative relationship appears robust in the investigation of online labour markets. Regarding the estimated coefficients of personality traits, we show that extraversion is negatively associated with performance, while agreeableness is positively related to performance (Column 2). An increase of one unit in workers’ extraversion and agreeableness level is correlated with a decrease of 0.9% in performance and an increase of 1.9% in performance, respectively. Openness, conscientiousness, and neuroticism appear to be statistically insignificant.

Table 4. Determinants of outcome variables.

	Performance  =  Total Correct Answers / Responses			Total Correct Answers			Payoffs	Responses	Productivity  =  Payoffs/ Effort
	Total (1)	Round 1 (2)	Round 2 (3)	Total (4)	Round 1 (5)	Round 2 (6)	Round 2 (7)	Total (8)	Round 2 (9)
Female	−0.017** (0.007)	−0.020* (0.010)	−0.011 (0.008)	−0.102*** (0.030)	−0.094** (0.046)	−0.103** (0.041)	−0.225 (0.092)	−0.082*** (0.029)	−0.209** (0.091)
Personality Traits
Openness	−0.011** (0.004)	−0.007 (0.007)	−0.014** (0.005)	−0.044** (0.021)	−0.005 (0.032)	−0.086*** (0.029)	−0.085 (0.058)	−0.033* (0.019)	−0.004 (0.056)
Conscientiousness	0.009* (0.005)	0.008 (0.008)	0.006 (0.007)	0.028 (0.021)	0.009 (0.029)	0.035 (0.031)	0.123* (0.069)	0.017 (0.020)	−0.006 (0.070)
Extraversion	−0.006* (0.004)	−0.009* (0.006)	−0.003 (0.005)	0.013 (0.017)	−0.018 (0.026)	0.047** (0.024)	0.032 (0.053)	0.021 (0.017)	0.001 (0.050)
Agreeableness	0.016*** (0.005)	0.019** (0.009)	0.014** (0.006)	−0.010 (0.022)	−0.028 (0.032)	0.013 (0.029)	−0.013 (0.061)	−0.028 (0.020)	−0.031 (0.061)
Neuroticism	−0.001 (0.005)	0.003 (0.008)	−0.005 (0.005)	0.030 (0.021)	0.015 (0.032)	0.041 (0.028)	0.144** (0.061)	0.029* (0.019)	0.108* (0.064)
Incentives – Experiment
Intrinsic	−0.011* (0.007)	−0.001 (0.011)	−0.023*** (0.008)	−0.058** (0.028)	−0.019 (0.042)	−0.099** (0.039)	−0.129* (0.085)	−0.045* (0.026)	0.046 (0.084)
Treated	0.010 (0.007)	–	0.009 (0.010)	−0.022 (0.035)	–	0.027 (0.049)	–	−0.029 (0.034)	0.678*** (0.089)
Round 2	0.015** (0.006)	–	–	0.024 (0.025)	–	–	–	0.004 (0.024)	–
Effort (sec)	0.001* (0.001)	0.001 (0.001)	0.001 (0.001)	0.022*** (0.001)	0.020*** (0.001)	0.028*** (0.003)	–	0.021*** (0.001)	–
Constant	0.727*** (0.071)	0.705*** (0.094)	0.727*** (0.126)	−0.335 (0.254)	−0.016 (0.338)	−1.282** (0.547)	2.755*** (0.526)	0.002 (0.239)	2.664*** (0.567)
Adjusted R-squared	0.135	0.162	0.178	0.453	0.487	0.447	0.183	0.455	0.259
F-Stat	2.99	2.17	2.51	108.53	19.00	8.09	1.80	57.61	6.68

Source: Authors’ calculations. Data drawn from Amazon Mechanical Turk.

Notes: The number of responses is N = 19.407 by 465 participants. The dependent variables of Total Correct Answers, Payoffs, Responses and Productivity are in logarithmic values. Intrinsic motivation is a median split dummy from the questionnaire survey measure. The specifications control for all the individual demographic, cognitive and social economic characteristics mentioned in Table 2. Also models control for individuals’ origins i.e. Regions’ of USA fixed effects.

Robust standard errors in parentheses. Statistical Significance: *** p < 0.01, ** p < 0.05, *p < 0.10

Finally, we do not observe incentive effects on labour outcomes coming from intrinsic motivated workers, on the first experimental round.

4.2 Second experimental round

4.1.4. Procedure

Between the two rounds, subjects had a two to two-and-a-half-minute break and were not allowed to leave the task environment. During the break, workers were informed that in the second round they could be given an additional bonus per correct answer, added to the initial payment of the one dollar in the first round. An algorithm randomly assigned all participants into five new groups of treatment. Each group had a different level of a bonus reward i.e. 1, 5 cents, 10 and 15 cents. Only one group of participants, ‘the benchmark group’, received no information on additional bonus rewards during the break. Until this time, no information about their actual performance was given and the workers were not informed of the randomisation process. While receiving the new instructions, workers could watch a song video. After the end of the song, the tab for the second phase was activated, and the subjects were able to proceed. During the second phase of the experiment, the workers could monitor their current performance through a counter measuring their bonuses each time they answered correctly.

4.1.5. Measures and task outcomes

Table 3 also contains the summary statistics of the abovementioned task outcomes of the second experimental round (Column 3). For the whole sample, we observe an average of 23 correct additions, reflecting an average of 93.4% of task performance. Moreover, workers had an average of 57 s of pure effort on the additions of the two-digits and an average of 24 responses.

Column 4 includes the t-tests of the difference of the experimental rounds for each above variable. Regarding the whole sample, we observe that the increase of task labour outcomes in the second round are all 1% statistically significant, raising questions on the determinants of workers’ average improvement.

Analysing further, we divided workers into two major groups depending on whether they were assigned for treatment. The results show that average worker performance increased in round two (i.e. 2%), only for those treated with piece-rate scheme payments over basic-wage individuals. Furthermore, treated workers had a statistically significant increase in their average sum of correct answers, effort, the number of total responses, productivity and payoffs over workers who had no additional information about extra piece-rate payment in the second round. This initial finding, casts doubt on whether the magnitude of the treatment has a similarly positive effect on workers’ performance and productivity index. We thus grouped results regarding compensation schemes based on four random levels of extra bonuses.

Tellingly, participants assigned to the higher bonus group had a statistically significant improvement in performance (i.e. approximately 3%), while those in the lower extra bonus group had a statistically significant increase of 1.8% in round two.

On the other hand, the results concerning workers’ average effort and number of total responses coincide among the four categories of treatment.

4.1.6. Results.

Regarding, the findings of the second experimental round and throughout the entire experiment, Table 4 includes the coefficient estimations for equation (1).

As we expected, given the repetitive nature of the online job, we find that regarding worker performance, in general, openness has a statistically significant negative effect on performance. An increase of one unit in worker openness is correlated with a decrease of 1.1% in performance, of 4.4% in the sum of total correct answers and of 3.3% in the sum of responses, with the magnitude of the effect being higher in the second round of the experiment, when the different payment schemes were used. Moreover, the coefficient for agreeableness seems to have a positive, high statistically significant effect (1% level of significance) on performance, confirming the findings by Gill and Prowse 2016. Concerning the level of workers’ conscientiousness, our finding also supports the existing literature. Our analysis also revealed the positive and significant effect of this trait on performance and payoffs (Cubel et al. 2016; Fletcher 2013). The coefficient for conscientiousness is, albeit insignificantly, also positive and of sizeable magnitude in relation to the average total correct answers of individuals. Furthermore, in respect to neuroticism and based on literature, we expected a negative effect on performance. Nonetheless, we failed to confirm such a correlation. Our results revealed that an increase of one unit in neuroticism is correlated with an increase of 14.4% in payoffs, 2.9% in the sum of responses and 10.8% in the productivity index. Finally, we find a negative correlation between the level of worker extraversion and performance; this effect is triggered in the first experimental round but disappears in the second round (Columns 2 & 3). Taking into consideration that extraversion is related in literature to ‘explicitly rewarded’ schemes (Stewart 1996) and given the wage working condition of our task, seems that the negative impact of extraversion on performance is to be expected.

The inspection of Figures 2 and 3, which follow participants’ average total correct answers over the time of the experiment, with the vertical line separating the two rounds of the experiment, suggests that even though we have an overall improvement of performance over time, this is independent and uncorrelated with the different payment schemes groups the workers were randomly assigned to.

Figure 2. Average total correct answers between the ‘Control’ and ‘Treated’ group of workers over time.

Figure 3. Average total correct answers between the ‘Control’ and the ‘Bonus’ groups of workers over time.

Finally, as known, the relationship between rewards and workers’ performance under intrinsic motivation is already established in literature. Bonus rewards could have negative effects on performance quality in a labour activity, once a part of the reward is received (Sansone and Harackiewicz 2000) for participants with intrinsic incentives, which are also related to the degree of their task enjoyment (Tauer and Harackiewicz 2004). By taking into consideration the ‘monotonous’ nature of the task and the experimental design, which includes two rounds, the first with a fixed payment and the second with performance-payments, a strongly negative relationship between several task outcome measurements and intrinsic motivation is expected. Furthermore, as regards performance and total correct answers, the effect of intrinsic motivation seems to exist only in the second round, in which performance-payments are applied. In total specifications, the subjects with intrinsic incentives performed on average 1.2% worse, having 5.8% fewer total correct answers, 12.9% fewer total payoffs and 4.5% fewer sum of responses, at 1%, 5% and 10% levels of statistical significance.

5. Heterogeneous effects

5.1 Incentive effects

Workers can use intermediate performance information to determine how much additional performance is necessary to obtain a bonus. This creates dynamic incentive effects, where the incentive effect of the pay-for-performance plan at each point in time depends on realised performance up to then (Benabou and Tirole 2003). Incentive plans based on relative performance, where prizes are awarded for sufficiently outperforming other competitors, are particularly prone to dynamic incentive effects (Baker 1992; Delfgaauw et al. 2014; Fama 1991). To investigate these different incentive effects deeper, we analyse the response level over the experiment’s period, on our database of 465 participants in two rounds, who submitted 19,407 responses in total.

Table 5 shows the average treatment effect on three outcome variables (i.e. productivity, total correct answers, and effort- the seconds that, each worker did in each addition of two digits’ numbers) by motivation, during the experiment. Given the experimental design, it is obvious that as regards the ‘productivity index’ the treated subjects were more statistically significantly productive than the benchmark group. However, we included this outcome variable to investigate the motivation effect on response levels during the experiment. Although in baseline specifications intrinsic motivation did not influence productivity in round 2 (Table 4, Column 9), when we look at response levels, we observe a negative statistically significant effect on the productivity index (Table 5, Column 1). Concerning the sum of total correct answers, our analysis shows that there is no statistically significant average treatment effect on worker performance. However, the lack of an average treatment effect masks considerable effect heterogeneity by intrinsic motivation (Table 5, Column 2). In line with other studies on treatment effects by motivation, the coefficient on the interaction between treatment and the intrinsic incentive dummy is positive and statistically significant at 1% level of significance, although intrinsic subjects perform worse than extrinsic ones in the beginning of the task (i.e. round 1) (Butschek et al. 2022; Chen & Horton, 2016). Our regression estimates imply that during the experiment intrinsic subjects had −0.032 fewer total correct answers with bigger effort per response (0.071 s). A possible reason for this is that intrinsically motivated online workers might be less keen to maximise their performance due to differences in degrees of risk tolerance because of personality heterogeneity (Cubel et al. 2016; Müller and Schwieren 2012).

Table 5. Treatment effect on outcome variables by motivation during the experiment.

	Productivity	Total correct answers	Effort
Outcomes (Yi)	(1)	(2)	(3)
Treated	7.337*** (0.474)	−0.009 (0.006)	0.034*** (0.007)
Treated x Intrinsically motivated	4.387*** (0.703)	0.033*** (0.010)	−0.081*** (0.018)
Intrinsically motivated	−4.429*** (0.527)	−0.032*** (0.009)	0.071*** (0.017)
Constant	2.565 (1.632)	0.865*** (0.024)	2.419*** (0.022)
Adjusted R-squared	0.117	0.016	0.108
F-Stat	32.44	14.37	245.26

Source: Authors’ calculations. Data drawn from Amazon Mechanical Turk.

Notes: The number of responses is N = 19.407. Intrinsic motivation is a median split dummy from the questionnaire survey measure. The specifications control for experimental rounds and for all the individual demographic, cognitive and social economic characteristics mentioned in Table 2. Also models control for individuals’ origins i.e. Regions’ of USA fixed effects. Robust standard errors in parentheses.

Statistical Significance: *** p < 0.01, ** p < 0.05, *p < 0.10.

5.2 Personality effects

We now turn our attention on the moderating role of personality traits. This section focuses on how individual differences may interact with responses to incentives under our performance-pay intervention (i.e. treatment).

Table 6 shows the estimates of our regression using the participants’ sum of total correct answers, which is the pure performance measurement in our experiment, as an outcome variable. Firstly, we observe that the coefficient on effect heterogeneity by intrinsic motivation remains similar in size and significance at 1% level of significance, throughout this exercise, even when controlling for all traits and interactions simultaneously. Including such a range of relevant personality characteristics gives us confidence that the heterogeneity by intrinsic motivation we observe is not just the effect of an omitted personality trait we do not observe. Hence, the heterogeneous treatment effect with respect to intrinsic motivation seems very robust. Furthermore, interestingly, conscientiousness and agreeableness lead to an independent heterogeneous effect in the same direction, while neurotic and extravert treated participants seem to have a loss in their online performance per response, during the experiment (Table 6, Column 7).

Table 6. Personality-dependent treatment effects on total correct answers outcome during the experiment.

	(1)	(2)	(3)	(4)	(5)	(6)	(7)
Treated	−0.009 (0.006)	−0.005 (0.009)	−0.002 (0.008)	−0.012 (0.008)	0.001 (0.008)	−0.020*** (0.007)	−0.013 (0.012)
Treated x Intrinsic	0.033*** (0.010)	0.030*** (0.010)	0.029*** (0.010)	0.036*** (0.010)	0.026*** (0.010)	0.034*** (0.010)	0.036*** (0.010)
Intrinsic	−0.032*** (0.009)	−0.030*** (0.009)	−0.029*** (0.009)	−0.033*** (0.009)	−0.028*** (0.009)	−0.035*** (0.009)	−0.033*** (0.009)
Treated x Openness		−0.004 (0.004)					−0.005 (0.004)
Openness		−0.001 (0.008)					−0.004 (0.009)
Treated x Conscientious			0.011*** (0.004)				0.010** (0.004)
Conscientious			0.021** (0.009)				0.013 (0.009)
Treated*Extravert				−0.020*** (0.004)			−0.023*** (0.004)
Extravert				−0.020** (0.009)			−0.021** (0.009)
Treated x Agreeables					0.013*** (0.004)		0.011*** (0.004)
Agreeables					0.025*** (0.009)		0.015* (0.009)
Treated x Neurotics						−0.008* (0.004)	−0.008* (0.004)
Neurotics						−0.030*** (0.009)	−0.023*** (0.009)
Constant	0.865*** (0.024)	0.921*** (0.012)	0.894*** (0.013)	0.915*** (0.013)	0.893*** (0.013)	0.921*** (0.012)	0.922*** (0.016)
Adjusted R-squared	0.016	0.010	0.010	0.010	0.010	0.010	0.10
F-Stat	14.37	6.91	7.50	8.43	7.91	7.48	6.43

Source: Authors’ calculations. Data drawn from Amazon Mechanical Turk.

Notes: The number of responses is N = 19.407. Outcome variable is the number of correct answers of participant workers. Personality characteristics are median split dummies from the questionnaire survey. The specifications control for experimental rounds and for all the individual demographic, cognitive and social economic characteristics mentioned in Table 2. Robust standard errors in parentheses.

Statistical Significance: *** p < 0.01, ** p < 0.05, *p < 0.10

In our formal model, conscientiousness can be viewed as a trait that similarly affects work performance: a more conscientious person exerts a specific service level driven by a feeling of obligation, which would also lead to a lower elasticity of sales and to a reduction in marginal monetary incentives (Barrick and Mount 1991; Salgado 1997; Tett, Jackson, and Rothstein 1991), while recent studies also point out that more agreeable subjects perform better (Gill and Prowse 2016). On the other hand, the negative heterogeneous treatment effects of neurotic subjects (i.e. subjects with lack of emotional stability and predictability and with anxiety and insecurity) appears to be in line with the existing literature (Nyhus and Pons 2005), showing that high degrees of worker neuroticism impair their ability to focus on a task, especially under time pressure (Borghans et al. 2008). Given that extraversion seems to positively correlate with tasks that involve social interaction, we assume that the negative impact on the performance of extraverts is driven by the monotonous task design and its characteristics.

6. Discussion

The main objective of this study was to examine the effect of intrinsic incentives on online labour productivity. Across a novel experimental procedure consisting of two rounds and several payment schemes, we provide evidence that immediate rewards increase intrinsic motivation by strengthening the activity-goal association. Intrinsically motivated workers achieve higher productivity outcomes as compared to less motivated ones. Following also previous studies on how workers with different personality traits respond to different payment schemes regarding their online performance and individual productivity, we observe that our results confirm several economics and psychology studies and provide good practice guidance for improving the effectiveness of online labour markets. Hence, this study also explores the underlying mechanisms and boundary conditions that explain why and under what circumstances personality traits (extraversion, neuroticism, and conscientiousness) moderate the effects of intrinsic motivation relate to productivity in online crowdsourcing communities. The results can be summarised as follows.

Initially, in line with other studies on treatment effects by motivation, we found that intrinsic individuals perform worse under a flat, no performance-based wage scheme (Kreps 1997; Larkin and Leider 2012). Surprisingly, our study reveals that when intrinsic individuals performed within piece-rate wage contexts, they changed their working behaviour and had significantly higher online productivity. We confirm the reverse payoff effect of performance-based payments by regressing performance on incentives, treatment and pay scheme (Table 5). By using self-determination theory, we confirm that an external wage intervention can crowd-out intrinsic motivation. Our results suggest that the compensation framework and the online working context may be more important than previously thought (Woolley and Fishbach 2018). The reasons why workers driven by intrinsic motivation increase their productivity after a change of the payment scheme from fixed payment to piece-rate payment remains to be further explored. In our case, we tried to interpret the results by embedding individuals’ personality traits in our analysis. Our experimental results demonstrate that personality skills are likely to be underlying factors associated with person-specific determinants of working behaviour (Heckman, Jagelka, and Kautz 2021). Thus, we further extend the findings of Cubel et al. 2016; Gill and Prowse 2016; Mourelatos, Giannakopoulos, and Tzagarakis 2022, revealing that agreeable and conscientious subjects perform even better when performance-based payments are applied to an online task (Table 6).

Intensified also by the COVID-19 pandemic, online labour markets are at the core of the economic and policy debate about the future of work and the conditions under which people will work mostly in online environments (Mueller-Langer and Gomez-Herrera 2022). Thus, our findings provide also key insights on online productivity determinants and contribute to this growing debate, aiming to improve several aspects of socioeconomic development.

7. Implications

7.1 Theoretical implications

Our study contributes to the literature on the relationship between incentives, personality traits, and online labour outcomes, providing insights on how personality-specific differences and incentives influence workers’ productivity in crowdsourcing. First, this study contributes to the deeper understanding of the underlying incentives mechanism of workers. Previous research has explored the impact of motivation under various payment structures; however, these analyses were conducted in a relatively static manner. In contrast, our experimental design leverages two rounds of experimentation to dynamically adjust compensation throughout the crowdsourcing process. This novel approach allows us to investigate the effects of motivation in a more nuanced and dynamic manner. Hence, the present study complements the existing literature by highlighting that the introduction of payment schemes based on real time performance does not have similar significant effects on working behaviour for all individuals. More specifically, while intrinsically motivated individuals, in general, seem to perform worse, their engagement with our wage intervention leads to a better labour outcome. Thus, our results contribute to the literature of self-determination and motivation crowding theory in online labour contexts, by providing insights on how personality-specific differences and incentives influence workers’ labour outcomes (Dube et al. 2020; Huffman and Bognanno 2018; Rustichini et al. 2016). A better understanding of wage policies in online environments will enhance their ICT components, making them more efficient and providing a positive effect on online labour productivity.

Second, we examined the moderating role of personality in relation between workers’ incentives and online outcomes, thus revealing the psychological mechanism of online working. Most of the previous studies have seen workers as a whole to explore the factors affecting their intention to participate, to engage and perform in crowdsourcing tasks. This study considers the influence of their personality a key determinant of their performance. Thus, we categorise personality into five traits (i.e. openness, conscientiousness, neuroticism, conscientiousness, agreeableness), and separately examine the disparity in its moderating effect between incentives and several productivity outcomes, which demonstrate the importance of individual’s personality in crowdsourcing procedure.

Third, while several previous studies highlight the importance of studying online workers’ behaviour in specific contexts and scenarios, this research contributes to existing literature by extending the current research on the development of the ICT aspects of crowdsourcing, by adopting a dynamic monitoring experimental procedure (Colman, Vineyard, and Letzring 2018; Mourelatos and Tzagarakis 2016). By investigating the workers’ response in depth within a pure productivity task, we highlight the need for further personalisation and task customisation strategies in online labour platforms (Ipeirotis and Horton 2011; Liang et al. 2018). From a macro perspective, our results inform policymakers on the regulatory treatment of online crowdsourcing workers, emphasising the need for more individual-specific and personalised approaches.

7.2 Managerial implications

Online labour markers are gradually emerging as a key driver of economic sustainability. Their rapid growth has focused attention on crowdsourcing and its contribution to social and economic development (Cui et al. 2021). Participants in these online platforms can obtain and access resources without boundaries or restrictions and have also the opportunity, though several ICT technologies, such as automatic matching and sorting technologies, to improve their efficiency and socioeconomic development (Harvey, Smith, and Golightly 2017). However, even though online labour markets and crowdsourcing are focal points of current research, the existing studies lack a comprehensive investigation covering various factors from social and economic perspectives, which may lead to higher efficiency and efficacy.

Within this context, the managerial implications of this study are manifold. Incentive issues have become an important challenge for the development and progression of the evolution of crowdsourcing. Many studies, especially economic, have focused on the incentive aspects of online labour platforms, examining motivations from a game theory perspective (Faberman and Kudlyak 2016 and Banfi and Villena-Roldan 2019) while IT scientists investigate incentives in terms of quality assurance and participation (Gong 2017). The workers’ behaviour in crowdsourcing is an important research topic because it shows the crowd’s effort and quantity of contribution. Our study is one of the first attempts to include individuals’ personality differences under different payment scenarios in the incentive mechanisms. The personality insights give a better understanding of individuals’ effort and online productivity. Firstly, our experimental study confirms the existing literature by revealing that intrinsically motivated participants perform worse in online labour markets. But when bonus payments are introduced to the experiments, intrinsically motivated participants seem to respond more efficiently in several working outcomes. We extended our analysis by embedding interaction terms with personality traits in the predication models. The findings show that, intrinsic individuals, having high levels of conscientiousness and agreeableness and being within the performance-pay condition (i.e. treatment group), appear to improve their online working behaviour.

Hence, we believe that our findings are crucial inputs to the growing debate on the role of ICT through crowdsourcing technologies in socio-economic development (Ipeirotis and Horton 2011; Palvia, Baqir, and Nemati 2018). Hence, we need an in-depth understanding of the underlying motivation mechanisms of participating and working in online environments. Moreover, the aim of this article is also to lay a conceptual foundation for future research on the design of personalised task recommendation matching mechanisms that assist individuals (i.e. workers) in finding suitable tasks and thus create benefits for both contributors and requesters and algorithms that will monitor and accommodate the online market needs (Möhlmann et al. 2021). A more efficient crowdsourcing adoption from an organisation’s perspective, in terms of i.e. quality of output, reducing risk, problem solving and organisational core competences, will elaborate on the strength of adopting crowdsourcing as a business strategy and will make it a useful innovation partner (Boudreau and Lakhani 2013; Jespersen 2018). Lastly, following the unique challenges of our time, online labour markets can be seen as a way of minimising socio-economic inequities within populations, by offering the opportunity to economies to reopen. Several crowdsourcing ideas from developing nations could point us towards simple, low-tech, inexpensive solutions that might be just what the future needs.

7.3 Limitations and future research

We have attempted to understand how individuals’ psychological mechanisms can develop further aspects of labour crowdsourcing, by conducting an experiment on an ICT platform. Despite the contributions of this study, some limitations need to be acknowledged. Firstly, while experimental studies in AMT crowdsourcing platform have high internal and external validity, we cannot extend our findings to the generalised real labour market. Our data was from US citizens only with particular cognitive characteristics (Ipeirotis 2010; Ross et al. 2009). This means that the study offers quite a narrow perspective on the global phenomenon of OLMs as it lacks, for instance, international, cross-cultural, and global viewpoints. All these limits the global generalizability of the results. Secondly, although the Big Five personality traits are stable indexes for adolescents and working-age adults, they do not provide compelling causal explanations for human behaviour (Cobb-Clark and Schurer 2012; Rantanen et al. 2007). Hence, the Big Five may be viewed as one important model in personality studies but not the integrative model of personality.

Hence, to achieve generalizability in terms of the effects of worker psychology on their online productivity and understand their contextual behaviour in depth, it still needs to be tested by analysing other psychological aspects (e.g. self-esteem, mood, emotions, etc.), under various crowdsourcing initiatives in a similar setting (Goodman and Paolacci 2017). A second issue is further studying the nature of online labour markets in future studies aiming to dynamically conceptualise the behavioural tension and psychological dimensions of online workers along with the ICT aspects of crowdsourcing, for personalised recommendation ICT agents to further develop online labour process (Sutherland and Jarrahi 2018; Zhao and Zhu 2014). For example, an automated recruitment match with worker-job recommendations for crowdsourcing online jobs, based on the correlations between task characteristics and personality traits of potential workers (Geiger and Schader 2014; Moayedikia, Ghaderi, and Yeoh 2020).

8. Conclusions

With the rise and proliferation of online labour markets, newly emerging business models, such as crowdsourcing has greatly shaped the way people work, by offering the ability to co-create via an increasingly connected world. However, attracting and retaining a necessary number of high-quality workers is often regarded as the first and the most important prerequisite for the success of the crowdsourcing procedure. By integrating self-determination and psychological trait theory, this study presents an initial attempt to investigate and understand deeper the incentive mechanisms and personality traits that trigger people to produce high-quality of work. The results revealed that, workers being driven by intrinsic incentives, have higher productivity outcomes, when bonus rewards are adopted. Another interesting finding is that, mainly, the personality-orientated factors of conscientiousness and agreeableness can also partially explain the increase of productivity levels. This study also provides some major implications. First and foremost, it shows that the effect of incentive schemes hinges on specific traits and preferences of the workforce. In this case stronger investments in the screening of online workers with respect to their motivation for the respective task should become more important. Future research should continue to enrich this line of research by considering other forms of crowdsourcing communities and personality-targeted experimental design approaches.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 For example, Fama (1991) discusses payment based on measured time interval, while Baker (1992) and Lazear (1995) have a detailed investigation on defining the measurement of productivity outcomes.

2 Coupled with the condition that the comparisons do not involve several other heterogeneity factors that vary across groups.

3 A respectable number of social researchers suggest that a theoretical possibility exists, in which the motivation may be affected when a previously non-monetary relationship is transformed into an explicitly monetary one.

4 In their study, the treatment was a proposed alternative payment scheme consisting of a combination of a fixed and commission payment.

5 On Amazon Mechanical Turk, the posted jobs are known as Human Intelligent Tasks (HITs).

6 Overall, 25 participants across all conditions failed the main attention check. We dropped them from the final data set and excluded them from the analysis.

7 At the end of an AMT experiment, the experimenter can either approve or reject a participant. Rejection generally happens when a participant tries to submit the survey using a fake completion code. This could happen, for example, because the participant has failed the comprehension questions and did not receive a completion code. However, AMT allows experimenters to decide the minimum approval rate of participants to be recruited.

8 Total cost of the experiment was 654 $ (i.e 514 $*(initial flat rate) + 31$(bonus piece rate) +109$ (ATM 20%) and the experiment ran for 3 h.

9 This was to account for eliminated participants across our five conditions based on the attention check.

10 Openness refers to the tendency to be creative and unconventional, Conscientiousness to the tendency to be organized and disciplined, Extraversion to the tendency to be sociable and active, Agreeableness to the tendency to be trusting and modest and Neuroticism to the tendency to experience negative emotions.

11 As we know, the power of cognitive ability in predicting economic success and productivity is well documented (Bowles, Gintis, and Osborne 2001a; Borghans et al. 2008)

12 Richter, T., Naumann, J., and Groeben, N., 2001. Das Inventar zur Computerbildung (INCOBI).

13 Terman, Lewis M. (1916). The Measurement of Intelligence: An Explanation of and a Complete Guide for the Use of the Stanford Revision and Extension of the Binet-Simon Intelligence Scale. Boston, Houghton Mifflin Co.

14 Ipeirotis found that Turkers are slightly younger and consist of females with a slightly lower income than the general USA population.

Appendices

Appendix A

Table

Definitions of used frequently terms
Term	Definition
Online Labor markets (OLMs)	IT-mediated online platforms (i.e. markets) where individuals can provide online services that can be performed anywhere (often by engaging in spot labour), offered by organisations generally through micro-tasks, typifying the production model of crowdsourcing, in exchange for monetary compensation.
Crowdsourcing	The practice of obtaining information or input into a task or project by enlisting the services of a large number of people under a compensation context, via the Internet. Crowdsourcing is an IT-mediated engagement of crowds for the purposes of problem-solving, task completion, idea generation and production in which the dispersed knowledge of individuals and groups is leveraged through a mix of bottom-up innovative crowd-derived processes and inputs with efficient top-down goals set and initiated by an organisation.
Microtask	A small job that requires a low level of skill and takes minimal time to complete. The financial remuneration for completing microtasks is also minimal.
Requester	The term requester denotes an agent who wants to get the task solved, e.g. an institution, a researcher, etc. within online labour markets.
Worker	A person who does small tasks as part of an online labour market.
Turker	A worker who does small tasks as part of the Amazon Mechanical Turk service.
AMT	Amazon Mechanical Turk crowdsourcing platform.
HITs	A HIT represents a single, self-contained, virtual task that a worker can work on, submit an answer, and collect a reward for completing. HITs are created by requester customers in order to be completed by worker customers. (Human Intelligent Tasks)

A short note on external validity and generalizability of results

Many studies conduct experiments in Amazon Mechanical Turk, in order to reduce cost, time and inconvenience, as it is currently the most appropriate online labour market for experimentation. Horton, Rand, and Zeckhauser 2011 has shown that it is possible to quickly and inexpensively replicate findings from traditional, physical laboratory experiments in the online laboratory, by replicating several well-known experiments conducted in the context of online labour markets. The external validity of these online results relies on the representativeness and quantitative versus qualitative generalizability of the crowd (Kittur et al. 2008; Horton and Chilton 2010). It is already known that individuals who choose to participate in social sciences experiments represent a small segment of the population. The same is also true for subjects who work online. Concerning the laboratory experiments, university students, who make up the subjects in most these physical experiments, are highly selected compared to the U.S population, so too are subjects in online experiments.

Our experimental sample is in line with many surveys which have revealed that Amazon U.S-based workers are more likely to be young females with high computer competence (Ipeirotis 2010)¹⁴ and suggest that the profile of the typical Turker is not a person that participates in online tasks for a living in a developing country (Ross et al. 2009). Moreover, the online population of the AMT remains relatively stable over time and follows the economic circumstances of the real labour USA market (Difallah et al. 2015). However, even if the participants of our experiments ‘look like’ population, in terms of observable cognitive and demographic attributes, some degree of self-selection bias is unavoidable. In line with the physical laboratory experiments, issues of ‘who is willing to participate’ in an online task exist, but this matter does not determine the usefulness of such research studies (Falk and Heckman 2009).

Appendix B

Singh and Masuku (2014), indicate that the detection of sample size is a challenging procedure for each experimental study, and sufficient sample size is a crucial factor in drawing valid conclusions. Thus, to overcome potential issues regarding validity of our empirical study, we conduced power calculation to determine the sample size for the linear regression models (Dupont & Plummer, 1998). In the sample size calculations, we assume that the sample would have 95% reliability about population and a sampling error of 5%. The calculations show that a sample size of N = 283 participants is required for the exact approach to attain the target power 0.9, with the significance level of 95%. We confirm that the actual sample size of the study is adequate to achieve results with the desired power and significance. For the implementation, we use the new Stata module of samsi_reg (Mander 2019) and the powercal (Newson 2004).

Additional References

Dupont, W. D., & Plummer Jr, W. D. (1998). Power and sample size calculations for studies involving linear regression. Controlled clinical trials, 19(6), 589–601.

Mander, A. (2019). SAMPSI_REG: Stata module to calculate the sample size/power for linear regression.

Newson, R. (2004). Generalized power calculations for generalized linear models and more. The Stata Journal, 4(4), 379–401.

Singh, A. S., & Masuku, M. B. (2014). Sampling techniques & determination of sample size in applied statistics research: An overview. International Journal of economics, commerce and management, 2(11), 1–22.