The End of the (Research) World As We Know It? Understanding and Coping With Declining Response Rates to Mail Surveys

Abstract

Mail surveys have long been a staple of social science research. Properly conducted, they can gather representative data about a population that provides important generalizations about that population. High response rates are one crucial element of this capacity to make such generalizations. Response rates to mail surveys–especially those targeting general populations–have declined substantially over the years, even when employing standard “best practices” such as carefully crafted and pretested instruments, good explanations of research purpose, and multiple contacts. Our research group has a unique capacity to speak to declining response rates, having implemented mail surveys on natural resource-related topics–using relatively similar methods–for more than 45 years. We present the results of a longitudinal analysis of response rates to 191 surveys we conducted between 1971 and 2017 (response rates and methodological specifics carefully recorded over this time), documenting the changes in response rates over the years, and the factors associated with these changes. Variables assessed include topic saliency, the nature of the sample, timing, length and complexity of the questionnaire, and others. We find that year of implementation is a strong predictor of response rate, net of these other factors. We use these findings as a base from which to engage a larger discussion about strategies that social science researchers might use, both in terms of continued modifications to mail survey approaches and in moving towards other research methods.

Keywords:

• Alternative methods
• mail survey
• response rate

Notes

Notes

1 While survey research methods have been criticized by some for lacking depth, privileging the perspective of the researcher over the research participant, and for reinforcing the subject-object dualism, we want to stress that this paper does not engage the appropriateness of survey methods generally. Our point is that mail surveys have been an established method in the social sciences, but their effectiveness is potentially challenged by declining response rates undercutting their ability to deliver on their promises of representativeness and generalizability. Ours is a methodological—rather than epistemological—engagement.

2 Tests for collinearity among independent variables i using the variance inflation factor produced VIF scores ranging from 1.065 to 1.375, well below collinearity tolerance thresholds.