How Randomly are Students Random Responding to Your Questionnaire? Within-Person Variability in Random Responding Across Scales in the TIMSS 2015 Eighth-Grade Student Questionnaire

Figures & data

Figure 1. Mixture IRT model framework to define and operationalize random responders in terms of independence and uniformity of item responses.

Note. Symbols follow standard path diagram conventions, with squares representing observed variables (i.e., item responses); circles, latent variables (i.e., trait to be measured by the scale of items); arrows indicating dependence relations; and vertical lines, response category thresholds. Reprinted under the terms of CC-BY-NC from “Random responders in the TIMSS 2015 student questionnaire: A threat to validity?” by van Laar and Braeken (2022), Journal of Educational Measurement.

Table 1. Overview of scales in the TIMSS 2015 student questionnaire.

Scale	I	RO	$\mathit{\lambda }$	$\mathit{E}$	$Pr$	$\mathrm{\%}Pr<.6$
Sense of school belonging	7	1–4	.68	.91	.97	1.17
Student bullying	9	1–4${ }^{\ast }$	.72	.98	1.00	.24
Like learning mathematics	9	1–4	.84	.91	.98	1.04
Views on engaging teaching in mathematics lessons	10	1–4	.81	.93	.98	1.10
Confident in mathematics	9	1–4	.81	.83	.95	2.56
Value mathematics	9	1–4	.79	.91	.98	1.19
Like learning biology	9	1–4	.79	.93	.98	.86
Views on engaging teaching in biology lessons	10	1–4	.87	.94	.98	.84
Confident in biology	8	1–4	.81	.85	.96	2.19
Like learning earth science	9	1–4	.82	.91	.98	1.27
Views on engaging teaching in earth science lessons	10	1–4	.89	.94	.98	.93
Confident in earth science	8	1–4	.83	.84	.96	2.11
Like learning chemistry	9	1–4	.82	.90	.97	1.23
Views on engaging teaching in chemistry lessons	10	1–4	.90	.93	.98	1.07
Confident in chemistry	8	1–4	.87	.85	.96	2.13
Like learning physics	9	1–4	.82	.89	.97	1.49
Views on engaging teaching in physics lessons	10	1–4	.90	.93	.98	1.10
Confident in physics	8	1–4	.85	.83	.95	2.56
Value science	9	1–4	.86	.90	.97	1.44

I = number of items within a scale; RO = response options. The response options range from 1 (agree a lot) − 4 (disagree a lot), except for the ‘Student bullying’ scale${ }^{\ast }$ where they range from 1 (at least once a week) − 4 (never). $\mathit{\lambda }$ = cross-country average of the average standardized factor loading for items of a scale; $\mathit{E}$ = cross-country average entropy of the mixture classification; $Pr$ = cross-country average of the average of the students’ maximum posterior class membership probability; $\mathrm{\%}Pr<.6$ = cross-country average of the percentage of students having a maximum posterior class membership probability below .60.

Figure 2. Normalized Bayesian information criteria as a function of the number of latent classes.

Note. The Bayesian information criteria (BIC) were normalized, where in each country the normalized BIC = [BIC − min(BIC)]/[max(BIC) − min(BIC)]. As a result, the latent class model with the highest BIC has a value of 1 and that with the lowest BIC has a value of 0. The model with the lowest BIC (indicated by the cross symbol in the plot) has a better balance between goodness-of-fit to the data and model complexity and is to be selected for inference and generalization purposes.

Figure 3. Random responder status probability profiles for the four-class solution.

Note. $Pr(\mathit{RR}=1)$ = class-specific probability of a positive random responder status (i.e., student classified as a random responder by the mixture IRT model for the scale). The scales on the horizontal axis appear in order of occurrence in the TIMSS 2015 student questionnaire. The dotted vertical lines divide questionnaire scales by subject: G = General, M = Mathematics, B = Biology, E = Earth Science, C = Chemistry, P = Physics, and S = Science.

Table 2. Class sizes and class averages of within-person statistics across seven countries.

	Consistent		Frequent	Intermittent
	non-RR	RR Conf	RR	RR
Class Size	3295	490	186	641
Class Proportion	[.53, .76]	[.07, .24]	[.03, .07]	[.10, .18]
Pr(RR=1)	[.01, .04]	[.17, .24]	[.40, .51]	[.15, .22]
Number of Runs (RR=1)	[0, 1]	[3, 4]	[4, 5]	[2, 3]
Number of Runs (RR=0)	[1, 2]	[4, 5]	[4, 5]	[3, 4]
Maximum Run Length (RR=1)	[1, 1]	[1, 2]	[3, 5]	[1, 2]
Maximum Run Length (RR=0)	[15, 18]	[6, 9]	[4, 5]	[8, 10]
$Pr(\mathit{R}{R}_{\mathrm{scale}}=1|\mathit{R}{R}_{\mathrm{previous}\mathrm{scale}}=1)$	[.00, .03]	[.05, .12]	[.43, .58]	[.15, .25]
$Pr(\mathit{R}{R}_{\mathrm{scale}}=0|\mathit{R}{R}_{\mathrm{previous}\mathrm{scale}}=0)$	[.96, .99]	[.67, .81]	[.49, .59]	[.77, .85]
Guttman Error	[0, 1]	[5, 6]	[6, 8]	[4, 6]

The median class size across countries is reported. The reported intervals provide the range of average values across the seven countries. A run is liberally defined here as a within-person sequence of constant random responder status across scales. A missing status for a scale was ignored and considered to not break up a run.

van Laar, S., & Braeken, J. (2022). Random responders in the TIMSS 2015 student questionnaire: A threat to validity? Journal of Educational Measurement, 59(4), 470–501. https://doi.org/10.1111/jedm.12317

 Web of Science ®Google Scholar