Comparing the mobilising effects of in-person canvassing to postal reminders – experimental evidence from a longitudinal election study

ABSTRACT

For hard-to-survey populations such as ethnic minorities and immigrants, increasing survey response rates is a crucial element of the fieldwork as these populations often show a higher likelihood of not participating compared with the native population. However, no study has so far compared different strategies for mobilisation within this group. Using data from the Immigrant German Election Study II, this experiment systematically compares the effects of home visits and postal reminders for the mobilisation of immigrant-origin non-respondents (i.e. persons classified as being from Turkey or from the former Soviet Union and its successor states) from a randomly drawn sample regarding the likelihood of participation in the first wave, signing up for a multi-wave panel, and taking part in all three panel waves. Multivariate analyses show that those in the treated home-visit group were more likely to take part in all stages of the survey design. Even though costs are higher than conventional postal reminders, home visits might be a useful strategy for cases in which increasing the response rate is an important goal, given a fixed, small number of potential respondents.

KEYWORDS:

• Immigrants
• Germany
• survey research
• in-person contact
• experiment

Introduction

What is the potential of mobilizing immigrant-origin non-respondents into surveys by conducting in-person home visits? Numerous strategies for mobilising respondents to participate in quantitative surveys, as well as different survey modes, have been researched extensively among many populations (Bosnjak et al., 2018; Edwards et al., 2009; Pötzschke & Braun, 2017). While postal reminders are a standard strategy to improve response rates in mail or face-to-face surveys after initial, unsuccessful contact attempts, personal contact with non-respondents is used less due to its high costs: sending out interviewers to remind non-respondents to take part in a survey is time consuming and thus economically often not feasible (Deding et al., 2008; Rasinski et al., 2012). However, there are cases where such an investment might be justified, or even needed, to reach the required number of interviews. This might especially be the case for surveys that focus on populations such as ethnic minorities and immigrants, which sometimes make up comparatively small numbers within the overall population or fulfil the definition of hard-to-survey populations for several other reasons (Tourangeau et al., 2014). Since these groups are hard-to-sample – and are often also harder to interview due to socio-cultural aspects, language barriers or to their distrust in the anonymity of the interview in particular – it is of central importance to increase response rates as much as possible (see Feskens et al., 2006; Kappelhof, 2015; Reichel & Morales, 2017 for an overview). Using in-person canvassing, which allows the canvassers to explain the objective, to emphasise the importance of a study, and to show appreciation by visiting non-respondents at home, might increase the willingness to participate, especially within this group (e.g. Krist et al., 2021). Even though existing studies use many different strategies to mobilise non-respondents, they rarely test them systematically against each other. One of the few previous studies (Krist et al., 2021) found increased participation when non-respondents were visited by native speakers in their homes, compared to receiving another postal reminder, although this increase was less than one percentage point. However, the two strategies studied were implemented in different waves/years and are thus not easily comparable. Surprisingly, no study so far has systematically analysed the effects of home visits compared to postal reminders, neither for the majority population nor for ethnic minorities and immigrants. In this experimental study, we aim to fill the current research gap.

For our purposes, we rely on data of our own Immigrant German Election Study II from the postal set-up of a probability-based telephone panel in the city of Duisburg, Germany (for an overview of that study see Goerres et al., 2022). This telephone panel was conducted before and after the 2021 German federal election. The recruitment phase spanned a partial COVID-19 lockdown, when most shops were closed and schools were in alternating between classroom and online teaching mode.

We compare results from our analysis of two samples of immigrant-origin individuals: Germans of Turkish origin, and Germans of post-Soviet origin, the two biggest immigrant-origin communities in Germany.¹ Our results show that in-person canvassing significantly increases the probability of target persons (TP) to take part in the initial survey and to participate in the first wave of a longitudinal CATI survey. Furthermore, within the group of canvassed TPs, direct personal contact with a canvasser at the doorstep causes the highest increase in response rates, compared to personalised letters.

Method

Experimental design

All persons, pre-classified by onomastic procedures as either of Turkish origin or from countries of the former USSR² who had not responded to the postal survey invitation within three weeks after the field start, were randomly assigned to one of two groups (see Figure 1 for an overview).³ The experimental condition (N = 1,940, made up of 970 from each immigrant-origin group) was scheduled for a home visit by a canvasser. The control group (N = 6,369) received a postal reminder. This extends previous research such as Krist et al. (2021) because we experimentally vary recruitment strategies within the same wave, while we hold external conditions constant. Ten percent of all persons in the experimental condition were randomly assigned to the soft-launch group, which was visited first to identify potential problems as early as possible. After the soft-launch revealed no major problems, canvassing was started for all TPs. Canvassing took place from April 12 to May 20. After the canvassing period, all remaining non-respondents, from both the control and experimental groups, received a last reminder via postcard.

We analyse the effect of home visits versus postal reminders for three major criteria that are important for panel surveys: taking part in the initial survey, taking part in the first wave of the panel, and taking part in all three telephone panel waves.

Procedure for the home visits

Eleven student canvassers were recruited and trained in two sessions of four hours each, what included training on how to handle previously designed vignettes of possible interview situations as well as discreet and responsible handling of interview data.⁴ They were instructed to follow hygiene measures at all times and not to enter homes. All canvassers were provided with FFP-2 masks, disinfectant, and COVID-19 lateral flow tests which were to be used before each day in the field. The main goal of the canvassing strategy was to establish direct personal contact with the TP or at least with a household member in order to ask them if they had received the survey documents. This allowed canvassers to emphasise the importance of the survey, explain data protection measures and to point out the possibility of contacting the research team directly. For the home visits, non-respondents were provided with a survey package that contained all survey documents (invitation and questionnaire) as well as 5 € in cash. Canvassers did not conduct the interview themselves (due to COVID-19 restrictions at the time), and were instructed to talk to the TPs only on the intercom or at the door. If they could not reach anybody, they were to leave the survey documents in the letterbox with a personal note which included their name, the date of delivery, and a short text explaining the failed contact attempt. Each target address was visited only once. Canvassers were generally free to visit their assigned addresses everyday between 10 am and 6 pm, except on Sundays and public holidays in order to minimize respondent burden.

Of the 1,940 home visits, 623 were successful, in the sense that the canvassers either talked to the TP or to a household member. We found that canvasser characteristics (age, gender, experience) had no effect on any of the three outcomes (see Online Appendix, Table OA 1).

Data

The present study was part of the recruitment stage for a multi-wave telephone survey in the context of the German federal election and was carried out from April to May 2021. Recruitment was based on a random sample of inhabitants from a major German city, Duisburg, with an oversampling for immigrants with German citizenship and their German-citizenship descendants. The recruitment stage, which relied on postal mailings, was used to gather telephone numbers and acquire participants’ consent to take part in the three-wave telephone survey that would follow. The telephone survey waves were conducted by an external company, BIK ASCHPURWIS + BEHRENS GMBH, between June and November 2021.

The population register of the city of Duisburg was used as the sample frame from which a random sample was drawn of all eligible voters aged 18 and above on election day (N = 70,000). In the next step, individual names were classified – using an onomastic approach from a specialised company – as belonging with at least medium certainty to persons either of Turkish origin, or from one of the successor states of the USSR, or from any other country, or to Germans without a migration background. All persons in the sample who were classified as being from a country other than Germany were invited by mail to take part the survey.⁵ Another 4,500 randomly drawn persons classified as Germans with no migration background were also invited by mail to take part. After the first round of invitations, it became apparent that the response was less than half among Germans of Turkish descent and Russian Germans than among persons classified as having no migration background. We thus decided to pursue an experimental strategy for increasing participations for the two main groups of immigrant origin by testing the effects of home visits and postal reminders. Neutral losses – such as invalid addresses, or persons who were unable to be interviewed due to illness or death – in both the postal reminder and the canvassing groups were excluded from our analyses. For a detailed methodological description of the study, we refer to (A. Goerres, J. Elis, S. J. Mayer, & D. Spies, 2021).

Results

We first analysed whether home visits versus postal reminders increased the likelihood of taking part in the initial survey. As displayed in Table 1, the proportion of those responding to the initial survey in any way – either taking part online or sending back the questionnaire – increases by about 21 percentage points in both of the immigrant-origin groups.

Table 1. Average treatment effect for all groups and for the treated and untreated groups regarding the three outcomes of interest.

		Proportion taking part in:
		the recruitment survey		the first panel wave		all three panel waves
Turkish origin	Control condition (N = 2,526)	12.4	12.4	4.4	4.4	2.4	2.4
	Experimental condition (N = 940)	33.0		9.6		4.0
	Δ Treatment vs. Control	+20.6***		+5.2***		+1.6**
	Exp.: not reached (N = 659)		22.1		7.5		2.9
	Exp.: directly reached (N = 311)		34.4		13.8		6.4
	Δ Experiment (not treated) vs. Control		+9.7***		+3.1**		+0.5
	Δ Experiment (treated) vs. Experiment (not treated)		+12.3***		+6.4***		+3.6**
Post-Soviet origin	Control condition (N = 879)	17.2	17.2	6.3	6.3	3.6	3.6
	Experimental condition (N = 939)	38.6		9.1		4.3
	Δ Treatment vs. Control	+21.4***		+2.8**		+0.7
	Exp.: not reached (N = 658)		24.4		6.5		3.2
	Exp.: directly reached (N = 312)		41.2		14.1		6.4
	Δ Experiment (not treated) vs. Control		+12.0***		+2.1		+0.8
	Δ Experiment (treated) vs. Experiment (not treated)		+21.4***		+7.6***		+3.2*

*** p < 0.001, ** p < 0.01 (t-test one-sided), ANOVA, Bonferroni correction for multiple comparisons, comparison: Control vs. treated not displayed.

Only every third or fourth person that took part in the recruitment survey for the first panel wave was reached by telephone. We also found that home visits increased the likelihood of TPs taking part in the first panel wave by between three and five percentage points. All these differences are statistically significant at the 0.01 % level. These results also hold true when we control for TP characteristics such as age, gender and local borough; information which we also acquired from the population register (see Online Appendix, Tables OA 2 and 3). TP gender did not matter for any of the groups. We find a similar positive relation for home visits and participation for taking part in all three panel waves, regardless of whether TPs were classified as being of Turkish or of post-Soviet origin. However, this difference only reaches conventional levels of statistical significance for the group of participants of Turkish origin.

Our analyses so far have only focused on the average treatment effect. We now divide the experimental condition according to the codes assigned by the canvassers: direct contact with TPs or no direct contact (i.e. leaving a personalised note in the letter box). This also allows us to determine whether the process of leaving all documents with a personal note and a 5 € incentive had an effect, independent of the canvassing contact itself.

Table 1 also shows that the probability of TPs taking part in the recruitment wave is significantly higher for all those in the experimental condition, whether treated or not, compared to the control condition. However, the treatment effect is the highest for those who were in contact with a canvasser. For the other two outcomes, there are no significant differences between those in the control group and those in the experimental group who were untreated. However, we still find a significant average treatment effect for the treated for taking part in the first panel wave (p < 0.01), and for taking part in all three panel waves (at least p < 0.05).

Discussion and conclusion

When setting up a panel study, we experimentally analysed whether home visits, paired with an unconditional, small cash incentive, made a difference among non-respondents of immigrant-origin from two different immigrant groups. We found that those in the home-visit group who were treated were more likely to take part in the initial survey, in the first wave of the panel study, as well as in all three waves. Furthermore, we found no meaningful differences between the two immigrant groups we studied, i.e. persons classified as being from Turkey or from the former Soviet Union and its successor states.

There are two limitations to this study that provide avenues for future research. First, the recruitment stage of the survey was administered by a research team from the local university, an institution known and trusted by the local population. This might have additionally affected the improvement of the response and participation rates after canvassing. Second, the context of the COVID pandemic could have biased our findings, as those TPs who were willing to talk to strangers on their doorstep might in general also be more open and more likely to be mobilised. However, as the canvassers also talked to the TPs on intercoms and never entered the houses and apartments themselves, our home visit strategy was the least intrusive as possible.

Our findings show that activating non-respondents from hard-to-survey populations with home visits might be a feasible strategy to increase initial and all-wave participation in a panel survey. Furthermore, even simple personal contact was enough to increase participation substantially, and this might be less costly compared to conducting the whole interview as part of the home visits. Even though their costs are higher than conventional postal reminders, home visits might be useful for cases in which increasing the response rate is an important goal, given a fixed, small number of potential targets. This approach should especially be considered in contexts with hard-to-survey populations that are clustered or densely distributed.

Acknowledgments

Prof Dennis C. Spies, Heisenberg Chair of Political Economy, passed away unexpectedly aged 40 on 19 June 2021, when the interview field period was in full swing. We are missing a fantastic political scientist, a wonderful collaborator, a reliable and caring mentor, and a great friend. PIs of the project: AG, SJM, DCS; experimental design: JE, AG, SJM, DCS, training of canvassers: AG (lead), JE; monitoring of fieldwork: JE; statistical analysis: JE, first draft: JE, final draft: JE, AG, SJM. 185 We thank the German Research Foundation (DFG) for financing the Immigrant German Election Studies II under grant numbers GO 1833, MA 8816, SP 1619.

Disclosure statement

No potential conflict of interest was reported by the authors.

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/13645579.2023.2189804

Additional information

Funding

This work was funded by the German Research Foundation under grant numbers GO 1833, MA 8816, SP 1619

Notes on contributors

Jonas Elis

Jonas Elis is a PhD candidate at the University of Duisburg-Essen and survey manager of the Immigrant German Election Study II.

Achim Goerres

Achim Goerres is a full professor for political science at the University of Duisburg-Essen. His research focuses on the welfare state, political solidarities as well as immigrants as political actors.

Sabrina J. Mayer

Sabrina Jasmin Mayer is a full professor for political science at the Otto-Friedrich-Universität Bamberg. Her research interests are electoral research, social group belongings and immigrants as political actors.

Dennis C. Spies

Dennis Christopher Spies was a full professor for political science at the University of Düsseldorf. His research focusses among others on political economy and integration and migration.

Notes

1. The Turkish community of any citizenship is the biggest immigrant-origin community in Germany. Russian Germans are the biggest immigrant-origin group among German voters.

2. Since we only have data on the country of birth of the respondents (for those who responded to the postal survey invitation) and their parents, we must rely on the onomastic classification to identify a TP’s migration background. Responses from the survey invitation allow us to compare information on migration backgrounds with onomastic classification. Among those who responded, the onomastic classification was correct for 92.76 % classified as Turkish and for 80.3 % of those classified as coming from a successor state of the USSR. Accordingly, onomastic classification is not a completely accurate indicator of migration background, but we might be able to add valuable information to compare the sampled groups.

3. These two groups were already our target groups in the Immigrant German Election Study I whose field and work report are available (Goerres et al., 2020). For an overview of its results, see Goerres et al. (2018).

4. Given the uncertainty of the volatile situation at the time, we self-committed to a procedure about how to monitor and adjust our field work in order to guarantee the complete psychological and physical safety of our interviewers and interviewees (A. Goerres, J. Elis, S. J. Mayer, & D. Spies, 2021).

5. Originally, we only invited persons of Turkish or post-Soviet origin and persons without a migration background. We only added the group of persons with other migration background after the home visits had already been conducted.