Email subject lines and response rates to invitations to participate in a web survey and a face-to-face interview: the sound of silence

Abstract

This paper investigates the relationship between blank and non-blank email subject lines on levels of response to a solicitation to participate in an interview, and on participation in a web survey. Email use has grown substantially in recent years, presenting significant opportunity to the empiricist seeking research respondents. However, response to emails may be low because growth in the sheer volume of messages that individuals receive per day has led to a sense of ‘email overload’, and faced with the challenge of personal email management, many recipients choose to ignore some messages, or do not read them all fully. Drawing on information gap theory, we expected that sending an invitation with a blank subject line would induce a sense of curiosity in recipients that would improve email response and willingness to participate in research studies. However, findings from research with two samples with different propensities to participate in research (academics and business owners) revealed that an email invitation with a blank subject line does not increase overall response rates to a web survey and a face-to-face interview over either an informative subject line or a provocative subject line, but that it does prompt a greater number of active refusals. Based on this finding, recommendations for researchers are outlined.

Keywords:

• email
• response rates
• information gap theory
• information overload theory
• curiosity

Introduction

It is impossible to conduct primary empirical investigation without the agreement, involvement and co-operation of research participants: respondents to fill in questionnaires, interviewees to offer their views in face-to-face or telephone discussions, or subjects to participate in experiments. Prior to the mass spread of the Internet, researchers had at their disposal a few stock strategies for recruiting research participants: for instance, mass mailings, tapping into informal networks, or requesting the participation of undergraduates in exchange for course credits. Many of these conventional strategies are slow, laborious, and, relative to the investment in researcher time required, yield low participation rates (Baruch & Holtom, 2008; Groves, Cialdini, & Couper, 1992; Patel, Doku, & Tennakoon, 2003). Online methods of participant recruitment offer advantages that may lead to such approaches supplanting traditional recruitment strategies (Sappleton, 2013). For example, Eaton and Lewycky (2011) were able to solicit 918 visitors to their research study website in just 25 days by posting recruitment comments to carefully selected online forums and blogs.

Similarly, the recent proliferation in email users offers huge potential to the researcher. Invitations to complete surveys (including those in email, web or paper format) or to participate in interviews and focus groups (via telephone, email, instant messenger, webcam or face-to-face) can be easily, rapidly, and cheaply sent by email (Bosnjak, Neubarth, Couper, Bandilla, & Kaczmirek, 2008; Goritz, 2004). Email allows researchers to obtain results in a fraction of the time and at minimal cost; databases can be efficiently administered, with software packages allowing syncing of contacts. At the same time, there are challenges associated with the use of email to identify and recruit research respondents. A fundamental factor affecting the use of email as a data collection instrument is the response rate (Koo & Skinner, 2005). Response to emails may be low because growth in the sheer volume of messages that individuals receive per day has led to a sense of ‘email overload’ (Dabbish & Kraut, 2006). Given the challenge of personal email management, many recipients choose to ignore some messages, or do not read them all fully (Siu, Iverson, & Tang, 2006).

One element of the solicitation and invitation process that can affect email opening, reading, response and reaction decisions is the subject line. The subject line is the first visible element of any email and therefore it has the potential to lure or to repel any target respondent. While it is known that email users utilize the subject line as a filter mechanism (Wainer, Dabbish, & Kraut, 2011), few researchers have examined the effect of the subject line on research study participant recruitment. Accordingly, we describe here the results of two research studies in which we examine variations to the email subject line on responses and reactions to an invitation to participate in a face-to-face research interview and an online survey.

Previous research

Email and information overload theory

The number of emails exchanged daily is vast and growing. According to figures calculated by the technology research organization The Radicati Group (2013) in 2013, some 182.9 billion emails were exchanged globally; a figure that is set to rise to 206.6 billion by 2017. Furthermore, it is estimated that over two thirds of global email traffic is junk mail. The average US-based office-based worker sent and received 105 messages every day in 2011, however, 19% of emails landing in workers’ inboxes was graymail (solicited bulk emails such as newsletters and offers), Unsolicited Bulk Commercial E-mail (UBCE) or spam, or attempts at phishing (fraudulent email) (The Radicati Group, 2011). The growth in unsolicited mail explains why workers report that the number of emails received into their inboxes far exceed the number that they regularly send (Fallows, 2002; Quaresma, Silva, & Marreiros, 2013).

The daily deluge of email exchanges is exacerbated by increased access to an endless array of digitally disseminated information via a growing number of telecommunications devices and is contributing to a growing sense of information overload among individuals, particularly professionals (Aikat & Remund, 2012). Information overload is defined as the situation that arises ‘when the information available exceeds the user’s ability to process it’ (Aikat & Remund, 2012, p. 16) or when ‘an individual’s efficiency in using information in their work is hampered by the amount of relevant, and potentially useful, information available to them’ (Waller & Ragsdell, 2012, p. 1). Email overload is just one facet of information overload and is prompted by the sense of having to deal with and prioritize a considerable volume of incoming messages (Szóstek, 2011). Moreover, the tasks involved in email management are myriad: once a message arrives into a users’ inbox, the users must attempt to evaluate the content and urgency of the message based on visible cues such as sender name and subject line, must decide whether and when to read, reply and/or file the email, or indeed, to do nothing at all, and must engage in the actions required to respond to the email (Siu et al., 2006; Szóstek, 2011; Whittaker, Matthews, Cerruti, Badenes, & Tang, 2011). The challenges posed by the attempt to organize, store, manage and retrieve this overabundance of information can provoke sense of stress, inhibit decision-making abilities, impair work-life balance and lower productivity (Barley, Meyerson, & Grodal, 2011; Mazmanian, Orlikowski, & Yates, 2013; Waller & Ragsdell, 2012).

Use of email for recruiting research participants

In light of the mammoth task of inbox management and the growing cacophony of digital information and unsolicited mail, wariness about whether or not to open an individual item of email appears to have grown (Goodman, Cormack, & Heckerman, 2007; Rennecker & Derks, 2013; Wainer et al., 2011). At the same time, researchers are increasingly taking to email in their attempts to solicit research participants, particularly to direct them to web-based surveys (Smith & Spitz, 2010) Thus, there is a danger that solicitations to participate in research sent by email will be lost, jeopardizing response rates. There is, however, very little published research available on use of emailto solicit research participants (Hansen & Pedersen, 2012). The evidence that does exist is mixed. In one Canadian study, an attempt to recruit teenage respondents to participate in an online questionnaire about smoking cessation, yielded only 5 recipients out of a total of 2109 – a response rate of just .24% (Koo & Skinner, 2005). In Australia, a study comparing email, web-based and Internet advertising as means of recruiting cannabis users to participate in a survey found that the email strategy was the least successful in converting survey visitors to survey completers (Temple & Brown, 2011). Goritz (2004) compared the use of email, fax, direct letters and flyer drops in recruiting German participants to an online access panel (a ready group of research participants that are willing to be contacted in order to take part in occasional web surveys) and while email was the most successful contact strategy of the four, that research preceded the current situation in which both solicited and unsolicited email message are prevalent. More recently, and also in the German context, Bosnjak et al. (2008) compared email and SMS invitation response rates to invitations sent to university students to complete web surveys on topics purportedly of interest to this demographic, such as online auctions, university life and personal leisure activities. The authors noted significantly higher responses rates among individuals that received an email invitation. However, respondents had already completed a paper questionnaire for the researchers and thus the email invitation cannot be considered to be completely unsolicited.

Subject lines and information gap theory

How can researchers improve the response rate to email solicitations in the context of email overload? One factor on which it seems reasonable to focus efforts aimed at preventing non-response is the email subject line. There is good reason to believe that recipients might be sensitive to variations in subject lines. The subject line serves a similar purpose as the introductory statement in telephone interviews and the opening paragraph in a cover letter for mailed surveys in communicating to the reader the possible content of the message. Accordingly, the subject line may be thought of as a sort of mechanism that signals to users the content of the message, enabling them to make quicker decisions about whether to read or discard it. Typically, a subject line might seek to legitimize the research, offer a reward, pose a question, spark interest in the study, appeal for assistance, stress the usefulness of the investigation, or personalize the communication by using the respondent’s name (Porter & Whitcomb, 2005). However, very little published research has explicitly addressed the links between the email subject line and research response rates. The existing research examining the role of subject lines has generally been undertaken in the context of messages posted to online discussion forums but has yielded inconsistent results. An analysis of over 40,000 messages posted to 99 Usenet newsgroups found that inclusion of a question mark in the subject line increased the likelihood of response (Burke, Joyce, Kim, Anand, & Kraut, 2007). However, ten experiments by Callegaro Kruse, Thomas, and Nukulkij (2009) found no difference in public affairs survey completion rates between emails with customized and those with generic subject lines. Similarly, in a case study examining the role of the subject line in an asynchronous online discussion among university students, Skogs (2013) could find no link between subject line categories (subject lines reflecting the content of the message, those describing the function of the message and those including greetings or salutations) and the number of times the message was read by forum users. However, a qualitative survey of users did reveal that students used the subject line as a filtering mechanism.

Perhaps the most well cited study examining the effect of the email subject line on survey viewing and survey response was conducted by Porter and Whitcomb (2005). The authors compared the click-through and response rates of a number of web survey email invitations sent to two samples of US high school students and university undergraduates. Students were asked to participate in a survey relating to their university life and given the relationship of the potential participants to the university, high response rates were expected. Four different subject lines were used: a subject line including the word ‘survey’, a line with the name of the university, a plea for assistance and a blank subject line. Of the four conditions, survey response and click-through rates were highest when the subject line was blank and significantly lower when the word ‘survey’ was included. Porter and Whitcomb (2005) suggested that a blank subject line sparks curiosity and interest in readers, while requests for assistance appear to users as spam. Similarly, Wainer et al. (2011) found that emails with an ‘information gap’ in the subject line, or no subject line at all increased the likelihood that a recipient would read a message, over emails where the subject line contained information about the content. However, this was a controlled laboratory experiment, and not a real-life setting.

The idea that silence in communication can provoke a response is not a new one. For instance, studies of mailed cover letters have found that letters which strike a balance between the provision of information about the research study and provoking curiosity about it can produce increases in response rates (Dillman, Gallegos, & Frey, 1976). Information gap theory (Loewenstein, 1994) holds that a gap in knowledge elicits in individuals a ‘need to know’ or curiosity which induces them to seek information to close that gap, even if such behavior is contrary to their best interests. The potential of the information gap has long been exploited by advertisers, but researchers have thus far been slow to harness its potential (Menon & Soman, 2002). Importantly, in the context of email, it is not known whether the growth in both email and junk messages has hindered the ability of researchers to utilize a blank subject line to precipitate responses to research.

To summarise, while the literature on response rates to web surveys is extensive, there is a scarcity of research examining the use of email in the research solicitation and invitation process, and specifically the impact of the subject line. Certainly, we could find no published research that examines the role of the subject line in motivating respondents to participate in a face-to-face research interview. It would be fruitful to compare responses to an email invitation to a web survey and to an in-person interview because the level of burden experienced by an interviewee is arguably higher than that experienced by an online survey participant. Therefore our purpose in this study is to address the existing gaps in the literature by exploring the relative impact of email subject lines on web survey response and interview participation rates. Specifically, we are interested in whether provoking curiosity in the recipient is related to (1) the tendency to respond to the email (termed the ‘reaction rate’) and (2) the willingness to participate in the research (the conventional definition of the ‘response rate’). While few studies have directly addressed the question of email subject lines and email response in the context of social research, hypotheses can be reasoned from some of the wider literature. Drawing on information gap theory, we hypothesize that in both the web survey and interview conditions a blank subject line would increase response to the solicitation, even if to refuse participation. However, there is no reason to expect that a blank subject line would boost the propensity to participate in research. Stated as hypotheses:

H1a: A blank subject line increases reactions to email solicitations to a web survey compared to a non-blank subject line, whether the reaction is positive or negative.

H1b: A blank subject line does not increase response to a web survey compared to a non-blank subject line.

H2a: A blank subject line increases reactions to email solicitations to a face-to-face interview compared to a non-blank subject line, whether the response is positive or negative.

H2b: A blank subject line does not increase response to a face-to-face interview compared to a non-blank subject line.

Study 1

Data and method

Data for study 1 comes from a web survey of New York City men and women business owners examining gender segregation in business sectors. This sample is useful for the purposes of this study because the proclivity of business owners to participate in academic research is notoriously low (Bartholomew & Smith, 2006; Dennis, 2003). Business owners were randomly selected from the Dun & Bradstreet’s Selectory and Reference USA databases. Email addresses were purchased from ReferenceUSA or located through an online search. Only email addresses targeting owner names were used (rather than a generic administrative email address). If a direct email address could not be found for a business owner, that individual was deleted from the sample. The final sample comprised 700 business owners in four sectors. The sectors were targeted on the basis of the sex composition of the industry: the construction and sound recording industries (male-dominated), the childcare industry (female-dominated) and the publishing industry (sex integrated). The sample was randomly divided into two groups. The first group (n = 350) received an email with a blank subject line. The second group (n = 350) received an email with an intentionally provocative subject line that highlighted the research question. The subject line read:

Why are there so few [men/women] business owners in the [construction/childcare/sound recording/publishing] sector?

The subject line was altered slightly to reflect the gender that is a minority in the business sector. For instance, business owners in the childcare sector received an email that stated ‘Why are there so few men business owners in the childcare sector?’ while recipients in the construction industry received an email with the subject line ‘Why are there so few women business owners in the construction sector?’ Owners in the sex integrated sector received an email with a subject line that also highlighted the lower numbers of women business owners in this industry since while that industry is numerically integrated, the number of men-owned publishing businesses still exceeds the number owned by women. The content of the emails sent to both groups were identical and contained a description of the study, a plea for assistance, the contact details of the researcher, and a link to the online survey, which was posted on the Bristol Online Surveys system. The first question of the survey solicited the name of the business so that the respondent could be tracked. A reminder email was sent two weeks following the initial communication. Of the 700 emails, 49 bounced back immediately so that the number of respondents totalled 651. Of this number, 325 respondents received the email with the blank subject line and 326 received the email with the provocative subject line.

Analysis

The descriptive data is presented in Table 1. Of those receiving the email with the blank subject line, 168 did not react at all and 28 reacted to the email by sending a message in return, but did not end up completing the survey. One-hundred twenty-nine respondents (39.7%) participated in the survey (this figure includes those that responded by email and went on to complete the survey). Of those receiving the email with the provocative subject line, 197 did not react and 3 reacted by actively refusing their participation. 126 (38.7%) recipients did take part in the survey. In accordance with hypothesis 1a, we expected the blank subject line to increase reactions regardless of whether that reaction was positive (i.e. eventual completion of the survey) or negative (active refusal). To test this hypothesis, we recoded the outcome variable into a dichotomous variable where 0 = no reaction and 1 = reaction (i.e. responded to participate or actively refused). A Pearson chi-square test of significance shows that the blank subject line elicited a higher level of reaction than the provocative subject line, χ² = 5.04, p < .05. Hypothesis 1a is supported.

Table 1. Descriptive statistics, study 1.

	Outcome
Group	Did not react n (%)	Reacted, of which n (%)		Total reaction rate n (%)
		Participated in survey	Actively refused
Blank subject line (n = 325)	168 (51.7)	129 (39.7)	28 (8.6)	157 (48.3)
Provocative subject line (n = 326)	197 (60.4)	126 (38.7)	3 (.9)	129 (39.6)
Total (n = 651)	365 (56.1)	255 (39.2)	31 (4.8)	286 (43.9)

Note: Excludes emails that bounced (n = 651).

Hypothesis 1b states that a blank subject line does not increase response to a web survey compared to a non-blank subject line. As shown in Table 1, the response rate (that is, those that participated in the survey) of those receiving the email with the blank subject line was 39.7% while the response rate of those receiving the provocative email was 38.7%. The difference is marginal and does not reach statistical significance, χ² (1) = .074, p = .79. Hypothesis 1b is supported.

In summary, the results of study 1 show that, in this sample comprised of individuals that are typically unmotivated to participate in empirical research, an invitation to complete a web-based survey sent by email yielded an overall response rate of 39.2%. An email sent with a provocative subject line and one sent with a blank subject line yielded statistically equivalent results; however, the blank subject line elicited a greater overall reaction from respondents, when those who replied to actively refuse to participate in the research are included.

Study 2

Data for the second study comes from an investigation into retirement planning of academics. The sample for this study was drawn from academic employees based in seven universities in the north of England that were local to the study investigators. This sample was chosen for several reasons. Firstly, while there are still many people who do not have email access (Smith & Spitz, 2010), academics use email on a daily basis, to communicate with students and collaborate with colleagues. Secondly, it is relatively simple to construct a database of academic email addresses by visiting the appropriate university webpage. A further advantage is that the university email addresses of academics are likely to be fairly accurate because they tend to be published online by the University for teaching and research purposes. Thirdly, academics have been identified has a group that suffer from considerable email overload (Hole, 2008; Jerejian, Reid, & Rees, 2013). Fourthly, since academics tend also to have research profiles, we expected their willingness to cooperate in other research projects to be higher than that of the general population.

Two randomly selected Faculty pages from each of the seven universities were visited. All academics listed on the webpages were added to the database, yielding a total of 121academics. The sample was randomly split into two sub-samples. The first sample (n = 59) received the email with the blank subject line. The second group (n = 58) received an email with the subject line ‘Retirement Research’. Following the recommendations of Porter and Whitcomb (2005), efforts were made to avoid a subject line that included a plea for assistance or the name of the sponsoring institution. We also hoped that academics would respond to a subject line that utilized the word ‘research’ given that this population is particularly interested in research. Using email merge, the emails were sent on a single day in October 2011. A reminder email was sent ten days later. No emails bounced, although we did receive 4 out of office messages suggesting that these recipients did not receive the request immediately.

Analysis

The descriptive data for study 2 is presented in Table 2. Of the 59 academics sent the email with the blank subject line, 30 (50.8%) did not react. The remaining 29 did react, equivalent to a reaction rate for the blank subject line condition of 49.2%. Of the 58 academics sent the email with the information subject line, 40 (69%) did not react. The total reaction rate for the information subject line condition was thus 31%. Hypothesis 2a stated that a blank subject line increases reactions to email solicitations to a face-to-face interview compared to a non-blank subject line, whether the reaction is positive or negative. As in study 1, we recoded the outcome variable into a dichotomous variable where 0 = no reaction and 1 = reacted to participate or actively refuse. In support of hypothesis 2a, a Pearson chi-square test of significance shows that the blank subject line elicited a higher level of reaction than the information subject line, χ² (1) = 4.00, p < .05.

Table 2. Descriptive statistics, study 2.

	Outcome
Group	Did not react n (%)	Reacted, of which n (%)		Total reaction rate n (%)
		Participated in interview	Actively refused
Blank subject line (n = 59)	30 (50.8)	11 (18.6)	18 (30.5)	29 (49.2)
Informing subject line (n = 58)	40 (69.0)	9 (15.5)	9 (15.5)	18 (31.0)
Total (n = 117)	70 (57.9)	20 (16.5)	27 (22.3)	47 (38.8)

Note: Excludes emails that bounced (n = 117).

In hypothesis 2b, we posited that blank subject line does not increase response to a face-to-face interview compared to a non-blank subject line. The response rate (that is, those that eventually participated in an interview as a proportion of those that were sent an email which did not bounce back) in the blank subject line condition was 18.6%, while in the information subject line condition it was 15.5%. The difference is insignificant, χ² (1) = .202, p = .65 and hypothesis 2b is supported.

Generally, the results of study 2 mirror those of study 1. A solicitation to participate in a research interview sent by email to academics yielded statistically equivalent response rates when the email contained a blank subject line as when it contained an information subject line. However, the blank subject line evoked a greater reaction from recipients, even if this was merely to refuse participation. As a final point, it is also interesting to note that despite our expectation that academics, as fellow researchers, would be more accommodating to a request to participate in research, the overall response rate for the academic cohort (34.5%) was actually lower than the response rate for the cohort of business owners (39.2%).

Conclusions

Given the predominance of email as a platform of communication among adults, it seems logical that researchers are increasingly turning to email to solicit research participants, irrespective of the data collection tool that is eventually used to conduct the research. However, the factors that prompt respondents to agree to, or turn down these invitations are not well understood. In this paper, we examined the role of the subject line of the email in eliciting reactions and responses from potential participants. Drawing on information gap theory, we hypothesized that a blank subject line would elicit a greater reaction to invitations than either a subject line designed to provoke comment, or a subject line that simply informed recipients about the purpose of the study. However, we did not expect the blank subject line condition to increase overall response rates. We examined the role of the blank subject line with reference to an invitation to participate in research with relatively low perceived respondent burden (a web survey) and research with relatively high perceived burden (a face to face interview), and with a sample that is typically expected to be motivated to take part in empirical research (academics) and a sample that is general hard to motivate (business owners). These studies therefore provide us with a means of exploring the influence of the blank subject line in a number of contexts.

The results of both studies suggest that the blank subject line is good bait, but it does not necessarily land the fish. An email invitation sent with a blank subject line does seem to provoke an increased reaction from participants, but that response is not necessarily a positive one. In fact, the blank subject line seemed to prompt several of our recipients to actively refuse to participate in the research. We suggest that the ‘silence’ represented by the blank subject line induces a sense of curiosity in recipients that prompts them to reply to the email. Earlier research by Temple and Brown (2011) found that email solicitations were generally poor in converting visitors to an online survey into survey respondents. We extend this work with our finding that the strategy of silence seems an unsuccessful one in converting individual reactions into responses. On the whole, our findings present to researchers both an opportunity and a challenge: how can the subject line be effectively used to provoke not just a reaction, but a positive one? In other words, what strategies (for instance, incentives, or pre-notification) can be adopted alongside the blank subject line in order to increase response rates? Clearly, greater empirical investigation is required to identify those additional factors that can be used to increase positive response rates.

We do not suggest that this research is without limitations. The findings described above are drawn from research with an unmotivated population and a low burden research method, and a motivated population invited to participate in a research study that is relatively burdensome. These findings may not, therefore, be generalizable to other populations with different characteristics. We recommend that this experiment be repeated with non-academic and non-business samples. The real-life setting of this experiment also means that we do not know for certain whether our emails were received or read. Many of those that did not respond to our invitations may never have received the email, and others may have simply discarded it without reading it. We excluded from our analysis those recipients whose emails bounced back, but we have no other means of checking receipt and digestion of our emails. Furthermore, apart from gender and job title, we have limited information about the recipients that may affect propensity to read-and-reply. Likely confounding variables such as inbox size, perceived levels of job stress and busyness and beliefs about the worth of academic research would certainly improve our understanding of how and why blank email subject lines induce negative responses. However, attempts to follow up such questions with non-respondents, perhaps unsurprisingly, yielded no response. The inability to harness some of these additional variables reduced the sophistication of the analyses that could be performed. Multivariate analysis, such as discriminant function analysis, that takes account of the possible confounding variables highlighted above, would strengthen our findings.

In spite of these limitations, this paper extends the literature on techniques and strategies for inducing involvement in potential research participants, and offers insight into the way in which email, and specifically the subject line, can be used in the research solicitation and invitation process. We encourage future research that examines how the subject line and other characteristics of email can be manipulated to convert those invited to participate in research to actual participants.

Notes on contributors

Natalie Sappleton is a senior lecturer at Manchester Metropolitan University Business School, UK. Her research interests are in the intersections of social networks, gender segregation and gender stereotyping in the labour market. She also has a keen interest in digital research methods and is the editor of Advancing Research Methods with New Technologies (2013, IGI Global).

Fernando Lourenço is an assistant professor at the Institute for Tourism Studies (Macau), teaching business and management, entrepreneurship and design courses. He has published widely on topics relating to entrepreneurship education and ethics in entrepreneurship.

Disclosure statement

No potential conflict of interest was reported by the authors.