Investigation of alternative interface designs for long-list questions–the case of a computer-assisted survey in Germany

ABSTRACT

This study aims to address the questionnaire design challenges in cases wherein questions involve a large number of response options. Traditionally, these long-list questions are asked in open-ended or closed-ended formats. However, alternative interface design options are emerging in computer-assisted surveys that combine both interface designs. To investigate trade-offs of these alternative designs, a split-ballot experiment was conducted with a) a long list of radio buttons, b) a search tree (nested list of response options), and c) a combo box (combination of a text box and a drop-down box). Based on the question on the highest educational qualification attained from the Innovation Sample of the German Socio-Economic Panel, we investigated the interface design that facilitates respondents optimally and enhances the measurement quality. The findings indicate that combo boxes reduce the response burden and increase measurement details, whereas search trees and long lists reduce post-coding efforts.

KEYWORDS:

• Instrument design
• questionnaire design
• look-up database
• education
• measurement

Acknowledgments

The author gratefully acknowledge support from the SOEP Innovation Sample at the DIW, Berlin and the Collaborative Research Center (SFB) 884 “Political Economy of Reforms” at the University of Mannheim, Germany. I especially thank Silke Schneider, Verena Ortmanns, Beatrice Rammstedt, and David Richter for their support in making this experiment possible. I am grateful for comments on a previous version from Annelies G. Blom, Frauke Kreuter, Edith D. de Leeuw, Florian Keusch, and Barbara Felderer.

Data availability

The data used in the analyses of this article are freely available as part of the Scientific Use Files (SUFs) of the DIW Berlin (SOEP-IS) survey data. They can be requested from the SOEP Data Archive at https://www.diw.de/en/diw_02.c.222829.en/access.html. The data sets used are catalogued under the following DOI numbers: 10.5684/soep.is.2011, 10.5684/soep.is.2012, 10.5684/soep.is.2013, and 10.5684/soep.is.2014.

Disclosure statement

No potential conflict of interest was reported by the author.

Supplementary material

Supplementary data for this article can be accessed here.

Notes

1. The implemented search algorithm uses text string matching without a fuzzy search. However, the search algorithm ignores special characters, the number of space characters, as well as upper and lower case letters. The suggestions are presented in alphabetical and hierarchical orders.

2. The sample size did not allow more experimental conditions (for information on power analysis see Döring & Bortz, 1995; Lachin, 1981).

3. German response options are presented because translating educational qualifications is prone to errors (Schneider, Joye, & Wolf, 2016). The combo box and search tree are based on Windows Presentation Foundation (WPF) technology and were recalled by the Kantar survey software nipo developed by TNS infratest, Germany. For further information on the tools see www.surveycodings.org/education.

4. Client-side paradata are collected at the level of the respondent’s computer (Heerwegh, 2003).

5. The responder’s educational level was harmonized with their answers in the panel waves 2012 and 2013 to avoid mismatches between the interface designs owing to changes in the respondent’s personal educational history.

6. This was based on a screening question that pertained to multiple educational qualifications.

7. However, no multivariate models for the data quality indicators of code-ability and number of educational qualifications are estimated, as these indicators have low-cell frequencies for many respondent characteristics and low-variations in some experimental conditions.

8. The estimates are also robust when estimating fixed-effect models to account for respondents who are clustered in interviewers.

9. In line with the recommendations of Best and Wolf (2015, p. 157), only the direction and statistical significance of the beta coefficients is interpreted in the cases of the logistic regression models 2 and 3.

10. These human coded responses were included in the database for later use.

Additional information

Funding

This work was supported by the Leibniz-Association through the Leibniz competition under grant number [SAW-2013-GESIS-5 249].

Notes on contributors

Jessica M. E. Herzing

Jessica M. E. Herzing is a postdoctoral researcher at the Life Course and Inequality Research Center (LINES) of the University of Lausanne and is associated with the Swiss Centre of Expertise in the Social Sciences (FORS).