“Who says what” in multiple choise questions. A comprehensive exploratory analysis protocol

ABSTRACT

Multiple-response questions are a common feature of survey-based research, as its advantages of information collection are well known. Statistical analysis of the responses, however, tends to be either one or two-dimensional, with the restrictions that this entails. This paper presents a multidimensional analysis protocol that provides the researcher with tools to identify more and better profiles about ‘who says what’. The strategy begins by coding the response options as a set of metric binary variables. The ideal methodological duo for the exploration of the resulting data is Principal Component Analysis coupled with an Ascending Hierarchical Cluster Analysis, incorporating, in addition, supplementary variables. When applied to the analysis of three different multiple-response questions included in a Spanish National Survey, this proposal provides evidence not only of the interpretation potential of the coding/analysis protocol but also of the limitations of some multiple-response question formats.

KEYWORDS:

• Forced-choice questions
• check-all-that-apply questions
• Principal Component Analysis
• clustering
• supplementary variables

Disclosure statement

No, potential conflict of interest was reported by the authors.

Supplementary material

Supplemental data for this article can be accessed here.

Notes

1. ‘In grid questions, a series of items is presented (usually in rows), sharing a common set of response options (usually in columns), asking one or more questions about each item’ (Couper et al., 2013).

2. Also known as ‘mark all that apply’, ‘multiple response format’, or ‘check all’ (Callegaro, Murakami et al., 2015).

3. This phenomenon will not occur in the presence of mutually-exclusive response options.

4. There is a wide range of classification techniques, including Hierarchical, K-Means or acombination of the two. The choice is up to the analyst, depending on the specific characteristicsof the problem (Lebart et al., 2006).

5. The survey was conducted by means of personal interviews, since the universe is the Spanish population over the age of 18, the final sample comprised 2,492 individuals, sampling error (assuming simple random sampling) is ± 2%, for a confidence level of 95.5% (two sigmas and P = Q). The questionnaire, technical details, data file, and the values for each variable and some cross-paired variables are available at http://www.cis.es/cis/export/sites/default/-Archivos/Marginales/3160_3179/3179/cues3179.pdf

6. The first two factors capture 26.30% of the total information (18.11% on the first factor and 8.19% on the second).

7. Since this prior analysis, which is included only for illustration purposes, exceeds the scope of this study, there is no need to describe the basic differences between MCA and PCA, which derive mainly from the nature of the data. Readers interested in obtaining further details are referred to Escofier and Pagès (2016) and Lebart et al. (2006).

8. Overall, few respondents checked these options, as can be verified in the published CIS results which can be accessed by visiting http://www.cis.es/cis/export/sites/default/-Archivos/Marginales/3160_3179/3179/Cru3179_enlace.html

Additional information

Notes on contributors

M. Landaluce-Calvo

M. Isabel Landaluce-Calvo is working in the Department of Applied Economics, University of Burgos, Burgos, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-3963-7485

Ignacio García-Lautre is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. His interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-4151-8477

Vidal Díaz de Rada is working in the Department of Human Sciences and Education, Public University of Navarre, Pamplona, Spain. The author’s research interest includes the study of survey error and data quality issues, with a special emphasis on questionnaire design, question testing strategies, interviewing techniques, survey non response, and survey sampling. ORCID: 0000-0002-9638-3741

Elena Abascal is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0001-5661-5483

Ignacio García-Lautre

M. Isabel Landaluce-Calvo is working in the Department of Applied Economics, University of Burgos, Burgos, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-3963-7485

Ignacio García-Lautre is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. His interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-4151-8477

Vidal Díaz de Rada is working in the Department of Human Sciences and Education, Public University of Navarre, Pamplona, Spain. The author’s research interest includes the study of survey error and data quality issues, with a special emphasis on questionnaire design, question testing strategies, interviewing techniques, survey non response, and survey sampling. ORCID: 0000-0002-9638-3741

Elena Abascal is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0001-5661-5483

Vidal Díaz de Rada

M. Isabel Landaluce-Calvo is working in the Department of Applied Economics, University of Burgos, Burgos, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-3963-7485

Ignacio García-Lautre is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. His interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-4151-8477

Vidal Díaz de Rada is working in the Department of Human Sciences and Education, Public University of Navarre, Pamplona, Spain. The author’s research interest includes the study of survey error and data quality issues, with a special emphasis on questionnaire design, question testing strategies, interviewing techniques, survey non response, and survey sampling. ORCID: 0000-0002-9638-3741

Elena Abascal is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0001-5661-5483

Elena Abascal

M. Isabel Landaluce-Calvo is working in the Department of Applied Economics, University of Burgos, Burgos, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-3963-7485

Ignacio García-Lautre is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. His interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0002-4151-8477

Vidal Díaz de Rada is working in the Department of Human Sciences and Education, Public University of Navarre, Pamplona, Spain. The author’s research interest includes the study of survey error and data quality issues, with a special emphasis on questionnaire design, question testing strategies, interviewing techniques, survey non response, and survey sampling. ORCID: 0000-0002-9638-3741

Elena Abascal is working in the Department of Statistics, Computer Science, and Mathematics, Public University of Navarre, Pamplona, Spain. Her interests include mainly in multivariate data analysis, descriptive factorial analysis, classification techniques, and their application to social science data and, more generally, to the analysis of survey data. ORCID: 0000-0001-5661-5483