Choir of believers? Experimental and longitudinal evidence on survey participation, response bias, and public service motivation

Abstract

Collecting data through surveys are a vital method in public administration research, but an overseen challenge for the validity of survey research is response bias, which is when response behavior correlates with study outcomes. Response bias can be particularly threatening in motivation research because motivation is an important determinant for behavior and most likely also for survey participation. Specifically, public service motivation (PSM) can increase the willingness to devote time and effort to respond to surveys for the benefit of others, whereas extrinsically motivated individuals may be less inclined to participate in surveys without prospects of economic compensation. This article examines motivation and response bias by use of a preregistered field experiment and two panel datasets. The experimental study shows that a monetary incentive increases the response rate, whereas a PSM-oriented appeal does not. Furthermore, the increased response rate of the incentive is achieved without any detectable response bias on PSM. Likewise, the panel studies offer little and mixed support for response bias related with PSM. This evidence suggests that (1) PSM is not a (strong) driver of response bias, and that (2) extrinsic motivational tools applied to PSM surveys are not necessarily associated with response bias.

Notes

1 In the literature, response bias and nonresponse bias are used interchangeably, since one is the residual of the other. For consistency, we use the term response bias here.

2 [For anonymity reasons the link is not inserted here – the preregistration is of course available upon request]

3 Robustness tests using only items PSM30 and PSM39 (cf. Perry 1996), which have been included in most previous PSM studies, produce similar conclusions for all three studies. See Figure F1 for a full analysis of all single items used in study 1, including PSM30 and PSM39.

4 The survey participation measure, “percentage completed”, is a continuous variable and therefore we do not use logistic regression models as is the case with binary outcomes in study 2 and 3 in order to have comparable results within Study 1. Similar results are obtained if using logistic regression on the binary outcomes in Study 1. Similarly, results are the same when applying a fractional response model to the continuous outcome "percentage completed".

5 In these analyses, we of course only look at PSM levels. Sometimes, researchers are more interested in the associations between PSM and other relevant variables. To test whether such associations might change as a result of the treatments in this study, we used a set of typical measures from the PSM literature: a self-rated job- permance index, a question about willingness to donate to good courses, job satisfaction and an index of professionalism. In four interaction models with the treatments interacted with the PSM measure, we reject the hypothesis that these relationships might change as a consequence of the treatments.

6 The invitation letter for these surveys contained a combination of the altruistic appeal and the incentive appeal (albeit with the possibility of winning of a modest USD 50 gift certificate when answering the survey).

7 To take into account that attrition can be an underlying explanation for nonresponse, we controlled whether attrition from the contact lists in 2011 and 2013 was explained by PSM in the previous survey and found no support for this.

8 Like in study 2 (see footnote 7), we found no evidence that attrition from the contact list was explained by PSM in the previous survey. Unlike study 2, respondents who changed jobs or for other reasons are not in the organization anymore during the time period, are not included in the analysis.

Additional information

Notes on contributors

Morten Hjortskov

Morten Hjortskov is a senior researcher at VIVE – The Danish Center for Social Science Research where he does research on public administration, survey methodology, child well-being and education. He received his PhD degree in political science from Aarhus University in 2016.

Christian Bøtcher Jacobsen

Christian Bøtcher Jacobsen is professor at the Department of Political Science and Crown Prince Frederik Center for Public Leadership, Aarhus University. His research interests include leadership, motivation, and performance with particular attention to the healthcare sector.

Anne Mette Kjeldsen

Anne Mette Kjeldsen is Associate professor at Department of Political Science and Crown Prince Frederik Center for Public Leadership, Aarhus University. Her research interests include public service motivation, job satisfaction, and leadership within health care, social services, and education.