Plain language in web questionnaires: effects on data quality and questionnaire evaluation

ABSTRACT

In web surveys, no interviewer is present to clarify question comprehension problems, which can be particularly prevalent among respondents with low literacy skills. Although plain language is used in various contexts to improve text comprehensibility, its use in social science questionnaires has not been investigated to date. Using a between-subjects design, we compared the use of plain language and standard language in a web questionnaire in terms of data quality and respondents’ evaluation of the questionnaire. Plain language did not generally have a positive effect on data quality and questionnaire evaluation. However, in the plain-language condition, we found less item nonresponse and more response differentiation among respondents who spoke a language other than German at home, which suggests that especially respondents who likely have low literacy skills benefit from plain language. Based on our findings, we outline opportunities for future research and give practical recommendations.

KEYWORDS:

• Plain language
• question comprehension
• questionnaire evaluation
• question design
• data quality

Introduction

Survey researchers face various challenges with web surveys, including not having an interviewer present to help respondents to answer the questions. Respondents generally go through four cognitive steps when answering survey questions (e.g. Tourangeau et al., 2000). First, they have to comprehend the question by reading all its components, which implies processing the structure of the question and understanding the syntax and the words used. Second, they have to retrieve the relevant information from memory, and third, they have to use that information to make the required judgments. And finally, fourth, they have to provide an appropriate answer by selecting a response option or giving a response in their own words (Sudman et al., 1996; Tourangeau et al., 2000). The first step – question comprehension – is essential for the subsequent steps. If respondents are not conscientious about reading and understanding all components of a survey question, they will not know what it is about and what information is needed for an appropriate answer. Thus, comprehension problems in the first step of the cognitive response process may negatively affect the subsequent steps (Wenz et al., 2021).

To perform the four steps of the cognitive response process conscientiously and comprehensively – which Krosnick (1991, 1999) called ‘optimizing’ – respondents must have certain cognitive abilities and be motivated to make the necessary effort. Otherwise, they may employ a ‘satisficing response strategy,’ which means they may execute all four steps, but less conscientiously and comprehensively (weak satisficing); or they may skip some or all the steps (strong satisficing; Krosnick, 1991, 1999). In addition to respondent ability and motivation, task difficulty also influences the likelihood of satisficing (Krosnick, 1991, 1999). Satisficing can take various forms, all of which affect data quality. For example, respondents may speed through the questionnaire, answering the questions very quickly without investing the time necessary to properly read and process them (e.g. Conrad et al., 2017; Zhang & Conrad, 2014); they may straightline by selecting the same or almost the same answer options for all items in a battery (e.g. Kim et al., 2019; Roßmann et al., 2018); they may choose the ‘don’t know’ option (e.g. Deutskens et al., 2004; Krosnick et al., 2002); they may leave one or more questions unanswered (e.g. Wenz et al., 2021; Yan & Curtin, 2010); or they may break off the survey (e.g. Mavletova & Couper, 2015; Steinbrecher et al., 2015).

In contrast to respondent characteristics, task difficulty is a satisficing-fostering aspect that survey researchers can directly influence. One way to make the response task easier, thereby counteracting comprehension problems and reducing the effort necessary to process the questions, may be to use plain language – that is, a simplified version of standard language, which is intended to be easy to read and understand.

In the present study, we aimed to fill a research gap by examining the use of plain language in web questionnaires. We conducted a web survey among respondents from an online access panel to explore the following research questions: (1) Does the use of plain language in a web questionnaire improve data quality compared with using standard language? (2) Does the use of plain language in a web questionnaire improve respondents’ evaluation of the questionnaire compared with using standard language?

Background

Plain language

Plain language is intended to be easy to read and understand; for weak readers or non-experts on a particular topic, it offers an alternative to standard-language texts (Maaß, 2020). Accessible and inclusive communication is a high-priority issue in many countries. The United Nations Convention on the Rights of People with Disabilities (CRPD) identifies plain language as a means of providing information in an accessible format (United Nations, 2017). Plain language has been the subject of extensive discussion, research, and legislation in the United States, the United Kingdom, Canada, and Australia since the 1960s (Petelin, 2010). In recent years, other countries around the world have become involved in the plain language movement, and many of them are now members of the Plain Language Association International (PLAIN), a nonprofit organization that aims inter alia to increase public awareness of plain language and to promote the use of and research and development in plain language (Plain Language Association International PLAIN, n.d.). While there are several handbooks and practical guidelines on the use of plain English (e.g. Cutts, 2020), ‘plain language’ (Einfache Sprache) is a relatively new concept in Germany (Maaß, 2020), where it is not defined by strict rules but rather by a set of recommendations (Pottmann, 2019).

Plain language is characterized by short and concise sentences using the active rather than the passive voice. Using simple syntax and providing explanations or examples for rare terms or abstract matters is recommended; so, too, is avoiding foreign words, metaphors, and abstract terms. Beyond that, plain language aims not only to simplify the wording but also to adapt the structure and design of the written text. Thus, plain language can be considered a ‘complete package’ characterized by clarity of wording (e.g. word choice, length of sentences), structure (e.g. thematic sequence), and design (e.g. visual layout), so that ‘intended readers can easily find what they need, understand what they find, and use the information’ (International Plain Language Federation, n.d..)

Survey questions in plain language

The idea of simplifying the language of survey questions to improve data quality and respondent experience is not new. Previous studies have examined the impact of different question wording – often using cognitive pretests or experimental designs. Cognitive pretesting is a well-established method in survey research. To assess the comprehensibility of individual questions and to identify problems in answering the questionnaire before it is fielded, respondents are asked to express their associations with the survey questions and words used (Lenzner et al., 2016; Presser et al., 2004). However, one shortcoming of cognitive pretesting is that due to time and resource constraints, usually only a small number of questions can be tested with a small group of respondents. In addition, as Eisner et al. (2019) noted, cognitive pretesting does not offer systematic ways of finding alternatives for problematic words and phrases.

Previous experimental studies on survey data quality have also focused mainly on question wording. For example, Lenzner (2012) found that survey questions with low comprehensibility in terms of different psycholinguistic text features (e.g. low-frequency words, vague terms, or complex syntax) decreased data quality. Blasius and Friedrichs (2009) tested the use of high-brow (or elaborate) versus low-brow (or everyday) question wording and its effects on data quality. They showed that using low-brow language resulted in more diverse responses; however, the distribution of responses differed between the two language versions. Slavec and Vehovar (2015) used either low-frequency (unfamiliar) or high-frequency (familiar) words in survey questions. They found evidence of lower break-off rates and lower perception of difficulty when the high-frequency question wording was used. These three studies show that the choice of words in survey questions can affect data quality and the respondents’ perception of the survey questions. However, to our knowledge, the use of plain language as a complete package that goes beyond the variation of individual words and terms has not yet been systematically studied in survey research.

Plain language can be used in web questionnaires to help respondents understand the questions as quickly, simply, and completely as possible. It has the potential to counteract comprehension problems that may occur in the first step of the cognitive response process. Because reading and understanding a question requires a certain level of language proficiency, comprehension problems when answering survey questions are more likely among respondents with low literacy skills (Lenzner, 2012; Wenz et al., 2021). According to the results of the second LEO – Living with Low Literacy survey, a large-scale survey of adult reading and writing skills in Germany, 12% of the adult population are considered to have low literacy and are able to read and write only simple sentences or even single words (Grotlüschen et al., 2020). Previous research in Germany has found that individuals with low formal education or a low family educational background are more likely to have lower literacy levels, and that a first language other than German is also a predictor of low literacy skills (Grotlüschen et al., 2020). Plain language can be of benefit not only to respondents with low literacy skills; it can also make it easier for all other respondents to complete web questionnaires. We assume that making questions easier to understand and reducing cognitive effort through plain language can reduce satisficing response behavior and improve data quality and respondents’ survey experience in general.

Data and methods

Sample and survey

The web survey was conducted in Germany from November 26 to December 4, 2020. The questionnaire included questions about family, relationships, and values. Respondents were sampled from a German online access panel. We used quotas for gender, age, and education. Of the 5,332 panelists who started answering the questionnaire, 30 were screened out because they were not members of the target population (age 18–69 years), and 1,179 were excluded due to filled quotas. Ninety-eight panelists broke off the survey, resulting in a break-off rate of 2.4% (Callegaro & DiSogra, 2008). Following the American Association for Public Opinion Research (The American Association for Public Opinion Research AAPOR, 2016) definition, we classified as ‘break-offs’ those cases that answered less than 50% of all applicable questions. A total of 4,025 respondents completed the questionnaire, with an average completion time of 10.7 min (median = 8.8 min).

Experimental design

Our questionnaire comprised three parts: the mandatory quota questions, the main section with 47 questions, and the evaluation questions at the end of the questionnaire. The between-subjects design was part of the main section of the questionnaire, with respondents being randomly assigned either to the control group with questions in standard language (n = 1,024) or the experimental group with questions in plain language (n = 981). Random assignment to the two language versions was successful, as indicated by nonsignificant differences between the two groups in terms of gender (${\chi }^2$(2) = 0.29, p =.865), age (${\chi }^2$(2) = 0.175, p =.916), and education (${\chi }^2$(2) = 4.844, p =.089). The questionnaire included another experiment that is not reported in the current study, which is why the total number of cases was greater than the sum of the numbers reported above.

The 47 questions in the main section of the questionnaire were adapted from the German version of the international Generations and Gender Survey (GGS; Gauthier et al., 2018), which is implemented in Germany as part of the Family Research and Demographic Analysis (FReDA) panel study (Schneider et al., 2021). These questions and the evaluation questions at the end of the questionnaire were translated into plain language by Lebenshilfe Bremen e.V., an organization that provides professional translations of texts into plain language (see https://leichte-sprache.de/uebersetzungen/leichte-sprache/). The translation included changes in the wording (e.g. avoiding complex words), sentence structure (e.g. converting one sentence into two sentences), and visual design (e.g. adding line breaks) of the questions. Some changes resulted in longer question texts (especially introductions) and response option labels (see Figure A1 in the Appendix). Due to longer response option labels in the plain-language version, we presented all item batteries in both language versions in a vertical item-by-item format instead of a grid format. In doing so, we also followed the recommendation that the visual presentation of questions in plain language should make the response task easier for respondents. To ensure comparability between the two language versions, the visual design of the questions in the main section was the same for all respondents. For the same reason, respondents in both the experimental and the control condition received the evaluation questions in plain language (for the English translations of the questions in both language versions, see Table A1 in the Appendix).

Dependent variables

We computed various data quality indicators based on the 47 questions in the main section of the questionnaire, counting each item in an item battery as a single question. In addition, we calculated an evaluation index based on seven items asked at the end of the questionnaire (see Table A2 in the Appendix).

Break-offs

We created a binary variable that indicates whether a respondent broke off the survey during the main section of the questionnaire (0 = no break-off; 1 = break-off).

Item nonresponse

Respondents could skip questions without being forced to answer. For each respondent, we calculated the proportion of questions with missing answers. This indicator could take values from 0 to 1, where 0.5 means a respondent left 50% of the questions unanswered.

Completion time

Completion times are a proxy measure for the cognitive effort a respondent spent on answering the questions. Using the open-source Embedded Client Side Paradata (ECSP) script developed by Schlosser and Höhne (2020), we collected page-wise timestamps in milliseconds for each survey page. We calculated the absolute completion time for the main section of the questionnaire for each respondent by adding up the page-wise response times. To account for differences in question length between the two language versions, we additionally calculated the relative completion time as the quotient of the page-wise response times divided by the number of characters in the respective question. We excluded outliers page-wise at two standard deviations above the group mean. Completion times are reported in seconds.

Time to first click

The time to first click captures the time respondents initially spent reading and comprehending the question before they selected an answer. Unlike the completion time measures, this indicator does not consider subsequent revisions of an initially answered question. And in the case of multiple items on a web page, this indicator is limited to the first response to one of the items. Using the ECSP script (Schlosser & Höhne, 2020), we measured the time until the first click was made on a web page and averaged these times across all pages in the main section. Again, we excluded outliers at two standard deviations above the group mean and reported the time to first click in seconds.

Speeding

Speeding means that respondents answer a question too quickly to be able to fully read and comprehend it (Conrad et al., 2017). To identify speeding respondents, we used the ‘scanning’ threshold method proposed by Andreadis (2014), which assumes that 39.375 characters can be read per second. In addition to reading a question, a respondent needs additional time to process it. Thus, a constant is added that takes the value of 1.4 seconds for simple attitude questions and 2.0 seconds for more complex attitude questions. For each respondent, we calculated the proportion of questions in which speeding was present. The indicator can take values from 0 to 1, with higher values indicating more speeding.

Nondifferentiation

Nondifferentiation means that respondents use the same or nearly the same response option for all items in a battery. As the items cover very different aspects, a low degree of differentiation is undesirable, or even implausible. We used three indices to measure different aspects of nondifferentiating response behavior (Kim et al., 2019). For all three indices, we calculated the average value at respondent level across the six item batteries in the main section of the questionnaire.

First, we computed straightlining (Zhang & Conrad, 2014) as the proportion of item batteries in which respondents used the same scale point for each item in the battery. This indicator can range from 0 to 1, where a value of 0.5 means a respondent chose the same scale point in 50% of the item batteries.

Second, we calculated the coefficient of variation as $CV=\frac{s}{\bar{x}}$ (McCarty & Shrum, 2000), where s represents the standard deviation and $\bar{x}$ the mean of a respondent’s answers across the items in a battery. This indicator is a measure of the distance between the scale points used by the respondents. It can take values from 0 to greater than 0, where 0 means that each item was answered with the same scale point, and higher values indicate a greater distance between the scale points used by a respondent.

Third, we calculated the probability of differentiation as a measure of variation in the scale points that respondents used to answer item batteries. According to McCarty and Shrum (2000), the probability of differentiation is defined as $P_d=1-\sum _{i=1}^n{P}_i$, where n is the number of scale points and $P_i$ is the proportion of the answers at each point of the scale. As the item batteries differed in terms of the number of items as well as the number of scale points, the maximum value of $P_d$ differed between the item batteries. Thus, we standardized the values for $P_d$ by scaling the values of the individual batteries to a range between 0 and 1, with larger values indicating that a respondent used a greater number of different scale points to answer an item battery. Respondents who answered less than three items in an item battery were excluded from analyses of $P_d$.

Questionnaire evaluation

We calculated an evaluation index based on (a) six items covering different aspects of the questionnaire (i.e. ‘interesting,’ ‘varied,’ ‘important for science,’ ‘long,’ ‘difficult,’ and ‘too personal’), which respondents rated on a 5-point scale from 1 (no, not at all) to 5 (yes, always; Kaczmirek et al., 2014); and (b) an additional item asking respondents how they found the questionnaire overall, which they rated on a 5-point scale from 1 (very good) to 5 (very bad; see Table A2 in the Appendix). The items ‘long,’ ‘difficult,’ and ‘too personal’ were reverse coded before analysis, so that higher index values indicate a positive evaluation across all seven items. Respondents who left one or more questionnaire evaluation items unanswered were excluded from analyses of this dependent variable.

Methods

We employed logistic regressions for ‘break-off’ as a binary dependent variable and the language version offered as independent variable. For the continuous dependent variables (i.e. item nonresponse, completion time, time to first click, speeding, nondifferentiation, and questionnaire evaluation), we used t tests for independent samples.

In addition to the full-sample analyses, we conducted analyses for subgroups with low literacy skills. As the assessment of literacy skills was not part of our study, we drew on previous research on individuals in Germany who tend to have low literacy skills (Grotlüschen et al., 2020). According to Grotlüschen et al. (2020), low formal education, first language other than German, and low family educational background are the strongest predictors of low literacy skills in Germany. As we did not collect data on family educational background, we were limited to the respondents’ level of education. For the first language spoken, we relied on information about the language that respondents predominantly speak at home and on information about whether respondents were born in Germany and had German citizenship. We distinguished subgroups by level of education (low, medium, and high), German citizenship (yes or no), Germany as country of birth (yes or no), and language predominantly spoken at home (German vs. other language). We re-ran the analyses for each of these subgroups.

Results

Full-sample analyses

Table 1 shows our results for data quality and questionnaire evaluation depending on whether the questions were provided in plain language or standard language.

Table 1. Results* on data quality and questionnaire evaluation.

	Overall	Standard language	Plain language
	M (SD)	M	M	Value^†	p
Break-off	0.022 (0.146)	0.022	0.021	−0.10	.921
n		1,043	981
Item nonresponse	0.007 (0.022)	0.007	0.006	1.57	.058
Completion time (absolute)	412.601 (188.478)	402.680	423.479	−2.49	.007
Completion time (relative)	0.195 (0.958)	0.186	0.210	−0.54	.294
Time to first click	7.534 (3.982)	7.675	7.390	1.59	.056
Speeding	0.130 (0.144)	0.130	0.133	−0.50	.310
Straightlining	0.168 (0.178)	0.169	0.167	0.20	.420
Coefficient of variation	0.343 (0.105)	0.325	0.361	−7.48	.000
Probability of differentiation	0.606 (0.147)	0.610	0.600	0.91	.362
Questionnaire evaluation	4.147 (0.571)	4.131	4.163	−1.22	.112
n		1,021	969

*For the analysis of break-offs, a logistic regression was conducted. For the remaining dependent variables, t tests for independent samples were conducted. †z values for break-offs, t values for the remaining dependent variables. Statistically significant effects in bold (p < .05).

We found significant effects of language version on absolute completion time. Respondents who received the questions in plain language took on average about 20 seconds longer to answer than respondents who answered the questions in standard language. Regarding the relative completion time, which takes into consideration that the plain language questions were generally longer (i.e. the question texts and response options had more characters), the differences between the two language versions were no longer significant. Regarding nondifferentiation, we found significant differences in the coefficient of variation (CV); providing questions in plain language rather than standard language resulted in more differentiated responses in terms of a larger distance between the scale points used to answer the items in a battery. We did not find significant effects of language version on straightlining, probability of differentiation, break-off, item nonresponse, and time to first click. Further, we found no significant effects of language version on questionnaire evaluation, irrespective of whether the evaluation items were considered together or individually.

Subsample analyses

Table 2 shows the results for the subgroups that are generally at higher risk of lower literacy.

Table 2. Results* for subgroups on data quality and questionnaire evaluation.

	Education						German citizenship				Germany as country of birth				Language spoken at home
	Low		Medium		High		Yes		No		Yes		No		German		other
	Value^†	p	Value^†	p	Value^†	p	Value^†	p	Value^†	p	Value^†	p	Value^†	p	Value^†	p	Value^†	p
Break-off	0.66	.508	−1.21	.225	0.10	.921	0.03	.974	−0.58	.564	0.01	.991	−0.42	.673	/	/	/	/
n		643		726		655		1,935		82		1,881		143		1,811		116
Item nonresponse	0.76	.224	0.87	.193	1.09	.138	1.18	.119	0.75	.227	1.12	.131	1.53	.064	1.01	.155	2.48	.007
Completion time (absolute)	−1.03	.152	−1.85	.032	−1.51	.066	−2.05	.020	−1.67	.050	−2.28	.011	−1.25	.107	−2.50	.006	−0.66	.254
Completion time (relative)	−0.60	.280	0.84	.200	−0.39	.350	−0.52	.300	−0.56	.290	−0.55	.290	0.41	.340	0.63	.260	−0.32	.370
Time to first click	1.56	.059	1.11	.134	−0.05	.481	1.86	.032	−0.74	.231	1.45	.073	0.55	.292	1.35	.089	0.69	.245
Speeding	−0.28	.390	0.06	.476	−0.40	.343	−0.66	.255	0.32	.373	−0.14	.444	−1.43	.078	−0.45	.326	−0.41	.340
Straightlining	−1.13	.129	−0.11	.458	2.01	.022	0.00	.499	1.37	.088	0.04	.484	0.87	.193	−0.25	.402	2.35	.010
Coefficient of variation	−2.37	.009	−4.72	.000	−6.03	.000	−7.7	.000	−0.16	.436	−7.26	.000	−1.84	.034	−7.51	.000	−0.45	.326
Probability of differentiation	1.64	.051	−0.25	.402	0.03	.488	1.17	.120	−1.42	.080	1.05	.148	−0.65	.260	1.31	.095	−2.28	.012
Questionnaire evaluation	−0.48	.317	−1.27	.103	−0.51	.306	−1.34	.090	0.16	.436	−1.00	.158	−0.93	.178	−1.38	.084	0.20	.421
n		632		715		643		1,902		81		1,849		141		3,620		116

* For the analysis of break-offs, a logistic regression was conducted. For the remaining dependent variables, t tests for independent samples were conducted. ^†z values for break-off, t values for the remaining dependent variables. Significant effects in bold (p < .05).

For respondents who speak a language other than German at home, we found significantly less item nonresponse when the questions were presented in plain language compared with standard language. For this subgroup, we also found significantly less straightlining and a higher probability of differentiation for questions in plain language compared with standard language.

We found significantly longer absolute completion times for plain language compared with standard language only among respondents who presumably had higher literacy skills (i.e. higher formal education, German citizenship, born in Germany, speaking German at home), but not for respondents who were more likely to have had low literacy skills (i.e. low formal education, not born in Germany, and speaking a language other than German at home). For the remaining indicators of data quality, as well as for questionnaire evaluation, we found no additional or differing results in the subsample analyses compared with the full sample.

Conclusion

Summary of results and discussion

In the present study, we focused on the use of plain language in web questionnaires and its impact on data quality and questionnaire evaluation. Although plain language is used in various contexts to improve text comprehensibility and ensure inclusion and accessibility, to our knowledge its use in population-wide social science surveys has not been investigated to date. Using plain language involves more than just explaining or replacing individual words; it is a complete package that includes simplifying the wording and sentence structure and clarifying the visual design of written text. In general, plain language is intended to make texts easier to read and understand. In self-administered (web) surveys, no interviewer is present to help respondents if they have problems comprehending the questions. Using plain language might help to prevent comprehension problems and thus ensure high-quality responses – especially among respondents with lower literacy skills, who are more likely to have comprehension problems. We further reasoned that making the first step of the cognitive response process easier for all respondents might help to reduce the effort required to process the questions, which might lead to a more positive evaluation of the questionnaire.

The results of our full-sample analyses were mixed. We found that absolute completion times were slightly longer when plain language was used compared with standard language. As we know from previous research, longer response times may indicate more conscientious response behavior, because comprehensive cognitive question processing takes more time (e.g. Callegaro et al., 2009; Lenzner et al., 2010; Toepoel et al., 2008). This result can therefore be interpreted as an indication that data quality was higher because respondents took more time to thoroughly comprehend the question and think about an answer. An alternative explanation for this finding may be the increased length of the question stems and response options in plain language, which required respondents to spend more time reading them. This interpretation is supported by the fact that when the number of characters was considered, there was no longer a difference in relative completion times between the two language versions. Regarding response differentiation in item batteries, we found a slightly higher coefficient of variation for plain language, but no statistically significant differences for straightlining and probability of differentiation between the two language versions. Thus, simplifying the language may help respondents take clearer positions on various issues, as evidenced by greater distances between the scale points used to answer the items in a battery. However, it does not necessarily result in respondents using a greater number of different scale points to answer item batteries. Break-off, item nonresponse, and time to first click, as further data quality indicators, were not affected by the different language versions, nor was questionnaire evaluation. Thus, using plain language in web questionnaires does not appear to be a panacea for improving data quality or respondent experience.

Our analyses for subgroups that were more likely to have low literacy skills (e.g. respondents with low formal education or with a first language other than German) yielded some promising findings. Our results suggest that plain language can potentially reduce item nonresponse and straightlining among respondents who are at risk of lower literacy skills. Thus, plain language may make it easier for some respondents to understand the questions and thereby reduce their likelihood to engage in satisficing behavior.

Implications for questionnaire design

Our findings have several practical implications for questionnaire design. First, we found no indication that plain language triggered detrimental response behavior in the full sample for any of the data quality indicators. We interpret this finding as a positive sign for those who fear that respondents with higher literacy skills might perceive more simplified language as unsophisticated or even taxing. Although we stress that more research is needed on the use of plain language in social science questionnaires, our findings may be a first indication that using plain language is feasible and does not hamper data quality.

Second, we found that developing a plain-language version of our standard questions was in itself helpful and beneficial for question development. In trying to simplify the question wording, the complexity of our questions became apparent, which was something we had taken for granted before their translation into plain language. It also helped us to shed a different light on question development and explicitly link it to the cognitive response process. Moreover, taking the perspective that plain language is a complete package led us to acknowledge that easing question comprehension involves not only the wording (e.g. word choice, length of sentences) but also the structure (e.g. thematic sequence) and design (e.g. visual layout) of written text. We see a benefit in routinely incorporating the exercise of creating an alternative question version in plain language into question development – if time and resource constraints allow.

Third, using plain language appears to make the cognitive response process easier for respondents with lower literacy skills. Thus, providing an alternative questionnaire for those respondents might be a helpful tool for researchers who have detected data quality issues in these respondent groups.

Limitations and future research opportunities

We designed the present exploratory study to initiate a discussion on the use of plain language in questionnaires to make the response process easier for respondents with low literacy and to make surveys more inclusive. Thus, our study has several limitations that are at the same time opportunities for further research on using plain language in questionnaires. First, we tested our questionnaire based on a sample from a non-probability online access panel. Online access panelists are likely to be experienced respondents, which may have masked the observed differences between the two language versions and limited the generalizability of our findings. However, we see such samples as a conservative test case, because the detection of effects that depend on literacy is less likely in a sample composed of individuals who have a basic level of literacy (i.e. the ability to sign up for an access panel) compared with a general population sample. While we acknowledge that non-probability samples can be used for exploratory studies such as ours to pave the way for more resource-intensive testing, we strongly recommend testing the use of plain language in probability-based samples of the general population.

Second, the relevant subgroups of respondents with low literacy skills were comparatively small, thus limiting our possibilities to conduct more in-depth analyses. Anticipating this issue when relying on respondents from an online access panel, we used quota sampling with education as one of our quota variables to ensure at least some variation across subgroups. In addition, we did not assess respondents’ actual literacy skills, but rather created subgroups based on assumptions derived from previous findings about predictors of low literacy skills. Consequently, we strongly recommend that future studies investigate plain-language versions in a sample that includes more individuals with (potentially) low literacy skills. More profound insights could be gained by assessing respondents’ literacy skills prior to investigating the effect of plain language on their response behavior.

Third, the process of translating survey questions into plain language holds potential for further research. There are no translation processes and standards specifically designed and used for adapting survey questions to plain language. We relied on professionals who specialize in translating texts into plain language for general purposes, but not specifically for survey questions. Thus, we see two major avenues for future research. First, measurement equivalence between the translated and the original language versions needs to be assessed. Our study was aimed at testing the feasibility of using plain language in web questionnaires and did not attempt to assess measurement equivalence. Assessing measurement equivalence is especially relevant when adapting well-established scales or questionnaires to plain language in ongoing longitudinal surveys. Second, we strongly encourage future research on standardized ‘checklists’ that provide information on which elements of a survey question need to be translated into plain language and how to create a version of the question that is easier for respondents to understand and answer.

Building on our exploratory study, we see potential to further investigate the use of plain language in questionnaires. Using plain language not only has the potential to improve the quality of responses by making the first step of the cognitive response process – question comprehension – easier; it can also make surveys more inclusive. Questionnaires that are more accessible because of plain language might motivate and empower individuals to participate who could not or would not respond to questionnaires in standard language.

Ethics declaration

Because our study was non-interventional (i.e. a survey), ethical approval was not required. However, we obtained informed consent from our respondents.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplemental data

Supplemental data for this article can be accessed online at https://doi.org/10.1080/13645579.2023.2294880.

Additional information

Notes on contributors

Irina Bauer

Irina Bauer is a PhD student at GESIS – Leibniz Institute for the Social Sciences in Mannheim, Germany. Her research interests include (visual) questionnaire design and data quality in web surveys. Email: [email protected]. Postal address: GESIS – Leibniz Institute for the Social Sciences, B6.4-5, 68159 Mannheim, Germany.

Tanja Kunz

Tanja Kunz is a senior researcher at GESIS – Leibniz Institute for the Social Sciences in Mannheim, Germany. Her current research interests include the design and implementation of mixed-mode survey data collection, with a focus on questionnaire design and data quality. Email: [email protected]. Postal address: GESIS – Leibniz Institute for the Social Sciences, B6.4-5, 68159 Mannheim, Germany.

Tobias Gummer

Tobias Gummer is a senior researcher and team leader at GESIS – Leibniz Institute for the Social Sciences in Mannheim, Germany, and apl. Professor at the University of Mannheim. His methodological research interests include survey design, data quality, nonresponse, and correction methods for biases. Email: [email protected]. Postal address: GESIS – Leibniz Institute for the Social Sciences, B6.4-5, 68159 Mannheim, Germany