Questionnaire Survey Nonresponse: A Comparison of Postal Mail and Internet Surveys

Abstract

Rapid advances in computer technology, and more specifically the Internet, have spurred the use of the Internet surveys for data collection. However, there are some concerns about low response rates in studies that use the Internet as a medium. The question is whether the lessons learned in the past decades to improve rates in postal mail surveys can also be applied to increase response rates in Internet surveys. After all, the Internet is a completely new medium with its own “rules” and even its own (n)etiquette. This article examines 29 studies that directly compared different survey modes (postal mail, fax, e-mail, and Web-based surveys) with more than 15,000 respondents. Factors that can increase response rates and response quality when using Internet surveys, compared to mail surveys, are discussed. Finally, the research that can contribute to increase response rates in Internet surveys is examined.

Notes

¹We use the term Internet surveys for all surveys that use the Internet as a medium to conduct a survey. The term Web-based surveys is reserved for surveys that use a Web server to conduct surveys.

²Nowadays, entering the keyword survey (as Kaye & Johnson did in 1998) in a Google search results in 251,000,000 hits; entering the keywords Internet (and) survey results in 164,000,000 hits, and entering Internet surveys results in 736,000 hits.

³One of the reasons that there are few studies on response rates is that it is difficult to compare studies that have different topics, different samples, and different methods of surveying. Apart from that, there are also substantial differences in the method used to calculate the response rate. Most studies report the unit response rate and fail to report the item response rate. An interview or questionnaire is rarely fully completed. According to some systems (e.g., the American Association for Public Opinion Research system) an interview is completed if the respondent was cooperative and at least 80% of the questions have been reliably and validly answered. Other systems to calculate response rates (e.g., Simple Interactive Statistical Analysis) would prefer 90%. For an overview of methods to calculate response rates, see the Simple Interactive Statistical Analysis Web site at http://home.clara.net/sisa/resprhlp.htm

⁴Deliverability is the top concern for 68% of IT and e-mail marketers according to an online survey conducted by Socketware (Burns, 2005). Deliverability concerns center on a few key barriers. The biggest barriers are e-mail filters (92%), ISP blocking (73%), blacklisting (69%), commercial e-mail laws (65%), whitelisting (58%), authentication (39%), accreditation and reputation services (31%), and feedback loop processing (27%). Only 4% of respondents say they are not concerned with deliverability.

⁵Results of the Pew Internet & American Life Project Study (Fallows, 2005; Rainie & Fallows, 2004) on the effects of the CAN SPAM Act in the USA on January 1, 2004, show that 60% of employees mailers receive 10 or fewer e-mail messages on an average day, 23% receive more than 20, and only 6% more than 50. With regard to users experiences with spam after the introduction of the SPAM-CAN Act, the study found the following results: Users who say they have ever received porn spam have decreased from 71% in 2004 to 63% in 2005; 52% of Internet users consider spam a big problem; 22% of e-mail users say that spam has reduced their overall use of e-mail; 53% of e-mail users say spam has made them less trusting of e-mail, compared to 62% a year ago; and 67% of e-mail users say spam has made being online unpleasant or annoying, compared to 77% a year ago. The spam-filtering company MessageLabs has reported that in an average month during 2004, spam constituted 73% of e-mail.