Can Culture Be a Salient Predictor of Test-Taking Engagement? An Analysis of Differential Noneffortful Responding on an International College-Level Assessment of Critical Thinking

ABSTRACT

The objective of this study was to evaluate whether differential noneffortful responding (identified via response latencies) was present in four countries administered a low-stakes college-level critical thinking assessment. Results indicated significant differences (as large as .90 SD) between nearly all country pairings in the average number of noneffortful responses per test taker. Furthermore, noneffortful responding was found to be associated with a number of individual-level predictors, such as demographics (both gender and academic year), prior ability, and perceived difficulty of the test, though, these predictors were found to differ across countries. Ignoring the presence of noneffortful responses was associated with: (a) model fit deterioration as well as inflation of reliability, and (b) the inclusion of non-invariant items in the score linking anchor set. However, no meaningful differences in relative performance were noted once accounting for noneffortful responses. Implications for test development and improving the validity of score-based inferences from international assessments are discussed.

Disclosure Statement

No potential conflict of interest was reported by the authors.

Notes

¹ Goldhammer, Martens, and Lüdtke (2017) conducted subgroup comparisons of test-taking effort based on moderators, however, the data were aggregated across all countries and the moderators were not jointly modeled.

² As part of the agreement to use this secondary dataset, country names could not be exposed; however, as of June 2018, each of the four countries sampled belonged to the G20 international forum and possessed economies that were in the top 12 in the world. Two of the four countries were tested in non-English languages.

³ Unidimensional models were fit separately for each country’s data using confirmatory factor analysis. Based on the recommendations of Hu and Bentler (1999), all data were found to adequately fit the model. This finding provided evidence of construct equivalence across the participating countries.

⁴ The lambda values employed were .525, .375, and .275 for Countries 2, 3, and 4, respectively.

⁵ Expected a posteriori (EAP) proficiency estimation was employed using the standard normal distribution as a prior with 21θ values and quadrature points ranging from −4 to 4. The maximum number of iterations was set to 1,000 with a convergence criteria of 0.01. EAP was chosen for ability estimation as it has been shown to be one of the most robust IRT estimators to atypical response behaviors (Kim & Moses, 2016).

⁶ The RMSEA is a standardized index of the discrepancy between the observed and the model based item response curves. An RMSEA value of zero indicates no differences between the two item response curves, while higher values represent increasingly poorer levels of model-data fit (Oliveri & von Davier, 2014, p. 12)

⁷ Across the SEC 3 (Country 2: 9.20%; Country 3: 3.03%; Country 1: 1.69%; Country 4: 1.12%) and NT10 (Country 2: 9.79%; Country 3: 7.90%; Country 4: 5.19%; Country 1: 3.09%) procedures, Country 2 was found to have the highest rate of noneffortful responding.