Call Me Maybe: Using Incentives and Follow-Ups to Increase Principals’ Survey Response Rates

Abstract

Educational researchers regularly collect survey data from educators, but struggle to encourage survey completion in this busy population, particularly when surveys are web-based. In this study we experimentally investigated the usefulness of follow-up phone calls and increased monetary incentives to non-respondents as strategies to boost response rates. We randomly assigned 540 hard-to-reach Michigan secondary school principals to one of four conditions (control, call, incentive, or call and incentive). We find that follow-up calls more than double the likelihood of survey completion, while the offer of a larger monetary incentive has no statistically significant effect. Integrating this finding with recent literature on survey response rates in educational settings, we conclude with a series of recommendations for researchers surveying principals and other school administrators.

Keywords:

• Data collection
• experiment
• principal
• survey

Open Scholarship

This article has earned the Center for Open Science badges for Open Data and Open Materials through Open Practices Disclosure. The data and materials are openly accessible at https://osf.io/86vwa/.

Notes

1 A likelihood ratio test comparing a reduced model that includes only the call treatment to the full model confirms that the addition of the increased incentive treatment and the incentive-call interaction do not significantly improve the model fit: LR χ² = 3.01, df = 2, p = 0.222.

2 This is the total response rate we would have observed given our 14.26% response rate among the original 652 principals (yielding 93 respondents), and a 30.83% response rate among the remaining 540 hard-to-reach principals (yielding 166 respondents).

3 Throughout our analyses, we use the conventional p < 0.05 (i.e. α = 0.95) threshold for identifying statistically significant effects. The minimum odds ratio detectable as statistically significant with 70% power in our logistic regression is 3.069. Adopting the more liberal p < 0.2 (i.e. α = 0.8) threshold would lead to the conclusion that the effect of providing an increased incentive is statistically significant, with an 80% confidence interval of [1.15, 2.82].