Evaluating item response theory linking and model fit for data from PISA 2000–2012

ABSTRACT

Based on concerns about the item response theory (IRT) linking approach used in the Programme for International Student Assessment (PISA) until 2012 as well as the desire to include new, more complex, interactive items with the introduction of computer-based assessments, alternative IRT linking methods were implemented in the 2015 PISA round. The new linking method represents a concurrent calibration using all available data, enabling us to find item parameters that maximize fit across all groups and allowing us to investigate measurement invariance across groups. Apart from the Rasch model that historically has been used in PISA operational analyses, we compared our method against more general IRT models that can incorporate item-by-country interactions. The results suggest that our proposed method holds promise not only to provide a strong linkage across countries and cycles but also to serve as a tool for investigating measurement invariance.

KEYWORDS:

• Item response theory (IRT)
• scale linking approach
• Programme for International Student Assessment (PISA)
• item-by-group interactions

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. Plausible values are multiple imputations drawn from a posterior distribution obtained from a latent regression model (also referred to as population modeling or conditioning model) using IRT item parameters from the cognitive assessment and principal components from the student background questionnaire. In PISA, each respondent receives 10 plausible values for each cognitive domain that can be used as test scores to produce group level statistics (never as individual test scores). For more information on plausible values and population modeling see Mislevy et al. (1992), von Davier et al. (2009), von Davier et al. (2006) or Yamamoto et al. (2013, updated 2016).

Additional information

Notes on contributors

Matthias von Davier

Matthias von Davier: Senior Research Director, ETS

Kentaro Yamamoto

Kentaro Yamamoto: Principal Research Scientist, ETS

Hyo Jeong Shin

Hyo Jeong Shin: Associate Research Scientist, ETS

Henry Chen

Henry Chen: Senior Research Scientist, ETS

Lale Khorramdel

Lale Khorramdel: Managing Research Scientist, ETS

Jon Weeks

Jon Weeks: Associate Research Scientist, ETS

Scott Davis

Scott Davis: Principal Research Data Analyst, ETS

Nan Kong

Nan Kong: Senior Research Data Analyst, ETS

Mat Kandathil

Mat Kandathil: Manager Data Analysis, ETS.