Animated videos in assessment: comparing validity evidence from and test-takers’ reactions to an animated and a text-based situational judgment test

Figures & data

Figure 1. Sample SJT scenario and practice statements. Adapted from ‘The socially skilled teacher and the development of tacit knowledge”, by Elliott, J. G., Stemler, S. E., Sternberg, R. J., Grigorenko, E. L., & Hoffman, N., 2011, British Educational Research Journal, 37(1), p. 93. © 2011. Julian G. Elliott, Steven E. Stemler, Robert Sternberg, Elena Grigorenko, and Newman Hoffman. All Rights Reserved. Reproduced with permission.

Figure 2. Sample scene from the animated version of a scenario.

Table 1. Performance on the English reading comprehension test.

	N	Minimum	Maximum	M	SD
Overall	129	0.00	9.00	3.84	2.01
Subgroups
Advanced English speakers	79	0.00	9.00	4.53	1.92
Non-advanced English speakers	50	0.00	7.00	2.74	1.63

Table 2. Performance on the practical knowledge SJT.

	N	Minimum	Maximum	M	SD
Overall	129	46.00	77.00	65.64	6.02
Treatment groups
Animated SJT	66	49.00	77.00	66.67	5.41
Text-based SJT	63	46.00	76.00	64.57	6.47

Table 3. Performance on the SJT, by English proficiency.

		N	Minimum	Maximum	M	SD
Animated SJT	Advanced speakers	39	59.00	77.00	68.97	3.94
	Non-advanced speakers	27	49.00	71.00	63.33	5.55
Text-based SJT	Advanced speakers	40	57.00	76.00	67.10	4.81
	Non-advanced speakers	23	46.00	72.00	60.17	6.71

Table 4. Regression models with construct-irrelevant factors.

Predictors	B	SE B	β	R^2
Text-based SJT				.368**
English proficiency (Advanced)	5.047*	1.499	0.379
Reading comprehension	1.045**	0.342	0.344
Animated SJT				.333**
English proficiency (Advanced)	4.155*	1.272	0.381
Reading comprehension	0.825*	0.331	0.290

* p < .05, **p < .01.

Table 5. Perceptions and invested effort indices.

Index	N	Minimum	Maximum	M	SD
Face validity	129	3.00	5.00	3.85	0.37
Enjoyment	129	3.33	5.00	4.48	0.47
Invested effort	129	2.60	5.00	4.24	0.56

Figure 3. Perceived difficulty of the language used in the assessment.

Note. Data labels for percentages smaller than 5% are not included in the graph.