The effect of dispatch methods on a recruiting campaign for a business survey: evidence from Germany

ABSTRACT

The ifo Institute for Economic Research has been conducting the ifo business survey since its foundation in 1949. To ensure stability of participation rates, regular sample recruitment is indispensable. Does the dispatch method matter for response rates of a recruiting campaign? To answer this question, we conducted a controlled experiment involving invitation letters sent out to over 8,000 German industrial firms in May 2019. Our results show that standard mailing significantly increases the response rate compared to ‘Dialogpost’ (bulk mail). From a cost perspective, costs for standard mailing outweigh this effect.

KEYWORDS:

• Business survey
• recruitment
• response rate
• postage

JEL CLASSIFICATION:

• C83
• C93

I. Introduction

A statistically valid response rate is an important indicator of survey quality. Real challenges in a long-term panel survey are the issues of survey fatigue and attrition as well as competition for attention in the recruitment process – business surveys are no exceptions in this respect. They have become an appropriate tool to measure the current state of economy, as usually official statistics are published with delay. The more representative the survey is in terms of economic structure, the higher the probability of a good tracking of the GDP. Representativeness goes often hand in hand with a sufficient number of participating firms, especially in economies with a diverse structure.

The ifo Institute has been conducting the ifo business survey since its foundation in 1949. The ifo Business Climate Index is a highly-regarded early indicator of economic developments in Germany published monthly (Sauer and Wohlrabe 2018). The survey is based on a panel of approximately 9,000 companies from the manufacturing, construction, wholesale, retail and services sectors. It is a ‘warm contact’ panel survey, in which respondents have prior knowledge that they will be contacted monthly. In order to fight panel attrition, new corporate participants have to be acquired regularly, mainly done through mail consignment. The response rates of such recruitment campaigns amount usually to a single-digit percentage.

In Germany, there are two dispatch ways: standard mail or ‘Dialogpost’, a form of bulk mail. We conducted an experiment whether it makes a difference using standard or bulk mail for the acquisition of new firms for the ifo business survey.

Several researchers (Gullahorn and Gullahorn 1963; Watson 1965; Kernan 1971; Brook 1978; McCrohan and Lowe 1981; Tuttle et al. 2018) examined a possible difference in survey response between US first-class outgoing postage versus bulk rate. A reason to suggest that first-class postage would generate higher response rates than bulk-rate postage is that bulk mail might be perceived as ‘junk mail’ and unimportant. A meta-analysis (Fox, Crask, and Kim 1988) revealed an aggregate difference of 1.8%. Various studies reported increased response rates by using registered, certified, or special delivery mail to send the questionnaire (Rimm et al. 1990; Del Valle et al. 1997; Gibson et al. 1999; Kasprzyk et al. 2001; Sang-Wook et al. 2005). A systematic review (Edwards et al. 2009) led to an odds ratio of 2.32 for certified/special delivery mailing.

Looking at the German case, researchers recommend the use of special issue stamps to avoid the impression of bulk mail and to increase recipient’s attention and response (Hafermalz 1976; Friedrichs 1990; Klein and Porst 2000; Porst 2001). There is little statistical evidence for German postal characteristics. Lavrakas et al. (2018) found that ”relatively little has been reported about factors that are related to the envelope and the mail delivery service […] and how these factors affect the propensity of a recipient opening it.”

We contribute to this literature by investigating if the type of postage matters within recruitment for a German business survey. In randomized controlled trail we sent out about 8,000 mailings using two different forms of appearance of the envelope, caused by the mailing type. We show that there is an effect of the type of postage on the response rate. But we also demonstrate that the costs can outweigh the benefits.

II. Background: types of mailing in Germany

‘Dialogpost’ is a form of bulk mail, which is often used for advertising mail, but also available for specific non-promotional content. The requirements by Deutsche Post AG for non-promotional content (e.g. polls, contract terms, etc.) contain identical content (i.e. quantity and type of content, packaging design and format, franking), machine readability of addresses and the creation of the address side. The minimum volume throughout Germany amounts to 4,000 ‘Dialogpost’ items. The postage rate is € 0.35/item with a weight up to 50g (2019) for basic format instead of € 0.70/item for standard mail letter (corresponds to first-class mail) (rate as at May 2019). The franking mark consists of a special ‘Dialogpost’ franking wave and can be supplemented by a customized graphic (see Figure 1). Figure 2 shows an example for standard mailing. Comparing both figures, one can clearly see optical differences between the two forms of mailing, due to franking specifications by Deutsche Post AG.

Figure 1. Envelope ‘Dialogpost’/bulk mail

Figure 2. Envelope regular mailing/first class mail

The EU General Data Protection Regulation (GDPR) governs the collection, processing and use of personal data. With the application of the GDPR in all EU member states, effective 25 May 2018, formerly often used recruiting via email got quite complicated (requirement: double opt-in by a potential survey participant before contacting). That’s one reason why postal invitation letters are still important in generating new survey participants.

III. The experiment

In May 2019, ifo Institute carried out an acquisition campaign for the business survey in the manufacturing sector. Since there is no official complete company register in Germany, addresses were taken from the database ‘Orbis’ from Bureau van Dijk, which covers a large part of German companies. We retrieved the entire mechanical engineering sector addresses and excluded companies with less than 10 employees, because they don’t become part of the ifo business survey panel. All mailings contained the same content: cover letter, questionnaire, admission form and return envelope. This envelope was prepaid, thus there were no financial costs for return. Only the form of dispatch varied. About half of the recipients got the pre-notification by ‘Dialogpost’, the other half by standard mail. In order to avoid possible regional effects on the response rate, a twofold randomized distribution was carried out. For each German federal state, companies were divided equally between the two shipping methods. A total of 8,188 letters were sent out on 05/22/2019 – 4,114 via standard mail and 4,074 via ‘Dialogpost’. Cut-off date was 07/08/2019.

We hypothesize that dispatch by ‘Dialogpost’ would generate lower response rates than standard dispatch, because it might be perceived as ‘junk mail’ and not opened at all. As Lavrakas et al. (2018) pointed out, ‘the envelope conveys the initial impression of a survey request and thus represents the first opportunity to affect the decision-making process’.

IV. Results

A total of 263 new survey participants were generated during this experiment, of which 115 were for ‘Dialogpost’ and 148 for standard mail. The response rate was 3.2% overall, whereby standard mail generated a higher response rate (3.6%) than ‘Dialogpost’ (2.8%, see Table 1). A simple Chi2 test shows that this difference is significant at the 5% level. This led to an odds ratio of 1.28 for standard dispatch, which means that the chance of positive response is 1.28 times higher in the standard mailing group than in the ‘Dialogpost’ group. In Table 2 we show that our results are mainly driven by medium-sized firms (100–249 employees). This is the only category where we find a significant difference in response rates. For the group with the largest firms, we find the most sizable odds-ratio. However, this difference is not significant due to a smaller available sample compared to the other categories.

Table 1. Response rates

	Standard	Dialogpost	Total
Participation	3.6%	2.8%	3.2%
Not participating	96.4%	97.2%	96.8%

Pearson chi2(1) = 3.9516, p-value = 0.047.

Table 2. Response rates by size categories

Employees	Standard	Dialogpost	p-Value	Odds-Ratio	N
less than 100	3.0%	2.7%	0.563	1.10	2,530
100–249	4.1%	2.5%	0.038	1.64	1,198
250–499	8.4%	8.0%	0.889	1.06	178
at least 500	4.3%	2.2%	0.244	2.04	208

Based only on these facts, it could be concluded that standard letter dispatch is preferable to ‘Dialogpost’. However, delivery charges should not be ignored: each standard letter costs 70 cents postage – each ‘Dialogpost’ item 35 cents (rate as at May 2019).

As can be seen in Table 3, a new survey participant involves noticeably less postage costs for a dispatch by ‘Dialogpost’ than for standard dispatch. Here, the total postage costs per shipping method were distributed among the generated participants.

Table 3. Comparison of costs

	Number of mails sent out	New Participants	Postage cost per sending	Costs per participant
Dialogpost	4,074	115	35 Cent	12.40 €
Standard	4,114	148	70 Cent	19.46 €

V. Conclusion

In this note, we provide evidence that the type of mailing matters for response rates of invitation letters in the recruitment process of a business survey. Standard mailing generated a significantly higher response rate compared to ‘Dialogpost’. What are the reasons for our findings? ‘Dialogpost’ is a form of bulk mailing and is often used to send standardized advertising. Questionnaires sent out in this way could be therefore interpreted as advertising or ‘junk mail’ before the letter is opened. As a consequence, it would possibly be sorted out directly. Furthermore, the receiver might have the impression that a letter sent by ‘Dialogpost’ is unimportant, as it can contain only standardized and not individual content. This might happen in our experiment where the need to open the bulk mail letter was quite lower compared to standard mail. From a cost perspective, this positive effect is outweighed by the actual postage costs per new participant. With focus on cost-efficiency, the recommendation is that mailing by ‘Dialogpost’ should be the preferred method.

However, this recommendation is subject to a restriction which applies to several economic sectors. As soon as the population of companies is quite small, it is possible to contact the entire population during a single recruitment campaign. Survey fatigue and attrition are permanent challenges to long-term survey projects, but companies should not be contacted too frequently, to avoid a feeling of annoyance. Therefore, it should be considered how important the need for new participants in a specific economic sector is for the representativeness of the survey results.

The final recommendation is accordingly that ‘Dialogpost’ should be chosen when the population is large, as the cost per new participant is lower. If the population is quite small and there is a strong need for new panel members, the aim should be to achieve the highest possible response rate with standard mailing.

Disclosure statement

No potential conflict of interest was reported by the authors.