Who responds inconsistently to mixed-worded scales? Differences by achievement, age group, and gender

Figures & data

Figure 1. Display of example consistent and inconsistent responses.

Note. Items stem from the TIMSS 2019 student questionnaire (Martin et al., 2020). Figure adapted from Steinmann, Strietholt, et al. (2022).

Table 1. Mixed-worded mathematics self-concept scale administered in TIMSS 2019 Grades 4 and 8.

	Wording
1. I usually do well in mathematics	+
2. Mathematics is harder for me than for many of my classmates^1	-
3. I am just not good at mathematics^2	-
4. I learn things quickly in mathematics	+
5. Mathematics makes me nervous	-
6. I am good at working out difficult mathematics problems	+
7. My teacher tells me I am good at mathematics	+
8. Mathematics is harder for me than any other subject	-
9. Mathematics makes me confused	-

Note. The response categories were 1 = agree a lot, 2 = agree a little, 3 = disagree a little, and 4 = disagree a lot. The items were administered in both TIMSS 2019 Grades 4 and 8.

¹In Grade 8, the exact wording was ‘Mathematics is more difficult for me than for many of my classmates’. ²In Grade 8, the exact wording was ‘Mathematics is not one of my strengths’.

Figure 2. Prevalence of inconsistent respondents in Grades 4 and 8.

Note. Displayed are the shares of inconsistent respondents (x-axes) across countries (y-axes). Vertical dashed lines indicate mean shares of inconsistent respondents across countries.

Figure 3. Results of regressing inconsistent responding variable on mathematics achievement, student age, language at home, and gender in grades 4 and 8.

Note. Displayed are log odds ratios (log (OR)) (x-axes) across countries (y-axes). A positive/negative log(OR) indicates that the odds of being classified as an inconsistent respondent is higher/lower for students scoring 100 scale score points higher on the mathematics achievement test, for students who are one year older, who (almost) always speak the language of the test at home, and for girls. The grey dashed vertical line is drawn at log(OR) = 0, corresponding to independence between the covariate and the random respondent classification. Horizontal whiskers indicate 95% confidence intervals. *Overall: The rectangle represents the 95% confidence interval around the estimated average log odds ratio across countries, and the whiskers extending the rectangle represent the corresponding 95% prediction interval.

Figure 4. Results of regressing shares of inconsistent respondents on mean mathematics achievement in Grades 4 and 8.

Note. Displayed are the results of regressing the shares of inconsistent respondents (y-axes) on the mean mathematics achievement scores (x-axes) in simple linear regressions (blue lines) in 38 countries in Grades 4 and 8 in scatterplots.

Figure 5. Results of regressing shares of inconsistent respondents on share of students who (almost) always speak test language at home in Grades 4 and 8.

Figure A1. Results of regressing inconsistent responding variable on mathematics achievement in Grades 4 and 8.

Note. Displayed are log odds ratios (log (OR)) (x-axes) across countries (y-axes). A positive/negative log(OR) indicates that the odds of being classified as an inconsistent respondent is higher/lower for students scoring 100 scale score points higher on the mathematics achievement test. The grey dashed vertical line is drawn at log(OR) = 0, corresponding to independence between the covariate and the random respondent classification. Horizontal whiskers indicate 95% confidence intervals. *Overall: The rectangle represents the 95% confidence interval around the estimated average log odds ratio across countries, and the whiskers extending the rectangle represent the corresponding 95% prediction interval.

Figure A2. Results of regressing inconsistent responding variable on student age in Grades 4 and 8.

Note. Displayed are log odds ratios (log (OR)) (x-axes) across countries (y-axes). A positive/negative log(OR) indicates that the odds of being classified as an inconsistent respondent is higher/lower for students that are one year older. The grey dashed vertical line is drawn at log(OR) = 0, corresponding to independence between the covariate and the random respondent classification. Horizontal whiskers indicate 95% confidence intervals. *Overall: The rectangle represents the 95% confidence interval around the estimated average log odds ratio across countries, and the whiskers extending the rectangle represent the corresponding 95% prediction interval.

Figure A3. Results of regressing inconsistent responding variable on language at home in Grades 4 and 8.

Note. Displayed are log odds ratios (log (OR)) (x-axes) across countries (y-axes). A positive/negative log(OR) indicates that the odds of being classified as an inconsistent respondent is higher/lower for students who (almost) always speak the test language at home than for those who do not. The grey dashed vertical line is drawn at log(OR) = 0, corresponding to independence between the covariate and the random respondent classification. Horizontal whiskers indicate 95% confidence intervals. *Overall: The rectangle represents the 95% confidence interval around the estimated average log odds ratio across countries, and the whiskers extending the rectangle represent the corresponding 95% prediction interval.

Figure A4. Results of regressing inconsistent responding variable on gender in Grades 4 and 8.

Note. Displayed are log odds ratios (log (OR)) (x-axes) across countries (y-axes). A positive/negative log(OR) indicates that the odds of being classified as an inconsistent respondent is higher/lower for girls than for boys. The grey dashed vertical line is drawn at log(OR) = 0, corresponding to independence between the covariate and the random respondent classification. Horizontal whiskers indicate 95% confidence intervals. *Overall: The rectangle represents the 95% confidence interval around the estimated average log odds ratio across countries, and the whiskers extending the rectangle represent the corresponding 95% prediction interval.

Martin, M. O., Von Davier, M., & Mullis, I. V. S. (Eds.). (2020). Methods and procedures: TIMSS 2019 technical report. TIMSS & PIRLS International Study Center, Lynch School of Education and Human Development, Boston College and International Association for the Evaluation of Educational Achievement (IEA).

 Google Scholar

Steinmann, I., Strietholt, R., & Braeken, J. (2022). A constrained factor mixture analysis model for consistent and inconsistent respondents to mixed-worded scales. Psychological Methods, 27(4), 667–702. https://doi.org/10.1037/met0000392

 PubMed Web of Science ®Google Scholar