Improving the Comparability and Local Usefulness of International Assessments: A Look Back and A Way Forward

Figures & data

Table 1. Correlation between parent and student report of number of books in the home.

	Raw correlation		Unattenuated correlation			Raw correlation		Unattenuated correlation
Country	r	SE	rcorr	SEcorr	Country	r	SE	rcorr	SEcorr
Indonesia	0.25	0.03	0.26	0.01	Dubai	0.48	0.01	0.79	0.02
Kuwait	0.25	0.03	0.35	0.03	Italy	0.49	0.02	0.73	0.03
Azerbaijan. Republic of	0.28	0.02	0.35	0.02	Iran	0.49	0.02	0.69	0.02
Qatar	0.29	0.02	0.46	0.03	Poland	0.49	0.01	0.64	0.02
Malta	0.30	0.02	0.42	0.03	Slovenia	0.49	0.01	0.60	0.02
Morocco	0.31	0.02	0.34	0.02	Hong Kong	0.49	0.02	0.68	0.03
Botswana	0.33	0.03	0.36	0.02	France	0.50	0.02	0.73	0.03
Abu Dhabi	0.36	0.02	0.52	0.03	Belgium (French)	0.50	0.02	0.78	0.03
Trinidad And Tobago	0.38	0.02	0.53	0.03	Finland	0.50	0.02	0.62	0.02
Canada (Alberta)	0.38	0.02	0.52	0.03	Morocco (Grade 6)	0.51	0.02	0.51	0.01
United Arab Emirates	0.40	0.01	0.66	0.02	Spain (Andalucia)	0.51	0.01	0.69	0.02
Oman	0.40	0.01	0.56	0.02	Czech Republic	0.51	0.02	0.68	0.02
Singapore	0.43	0.01	0.57	0.02	Germany	0.51	0.02	0.74	0.03
Canada	0.44	0.01	0.53	0.01	Ireland	0.51	0.02	0.80	0.03
Canada (Quebec)	0.44	0.02	0.57	0.02	Spain	0.51	0.02	0.71	0.02
Norway	0.44	0.02	0.62	0.03	Georgia	0.55	0.02	0.92	0.03
Australia	0.45	0.02	0.63	0.03	Austria	0.55	0.01	0.82	0.03
Canada (Ontario)	0.45	0.02	0.57	0.02	Lithuania	0.55	0.01	0.71	0.02
Saudi Arabia	0.45	0.02	0.78	0.03	South Africa	0.55	0.03	0.75	0.03
Honduras. Republic of	0.45	0.03	0.49	0.02	Portugal	0.56	0.02	0.76	0.03
Netherlands	0.46	0.02	0.65	0.03	Chinese Taipei	0.56	0.01	0.91	0.03
International Average	0.46		0.66		Sweden	0.57	0.02	0.86	0.03
Northern Ireland	0.47	0.02	0.72	0.03	Croatia	0.57	0.01	0.70	0.02
Israel	0.47	0.02	0.71	0.03	Denmark	0.59	0.01	0.84	0.03
New Zealand	0.47	0.02	0.70	0.03	Romania	0.62	0.02	1.00	0.03
Colombia	0.48	0.02	0.48	0.02	Slovak Republic	0.65	0.02	0.90	0.02
Russian Federation	0.48	0.02	0.59	0.02	Hungary	0.65	0.01	1.00	0.03
					Bulgaria	0.68	0.01	1.00	0.03

Table 2. Factor loadings of sub-scale and reliability for economic, social, and cultural status indicator.

	Factor loadings			ESCS		Factor loadings			ESCS
Country	HISEI	PARED	HOME	Reliability	Country	HISEI	PARED	HOME	Reliability
Chile	0.89	0.88	0.84	0.79	Ireland	0.81	0.81	0.68	0.65
Mexico	0.86	0.86	0.81	0.79	Israel	0.82	0.82	0.67	0.65
Portugal	0.88	0.86	0.77	0.78	Switzerland	0.81	0.79	0.67	0.65
Hungary	0.84	0.85	0.74	0.75	Median	0.82	0.81	0.69	0.65
Poland	0.89	0.88	0.71	0.75	Korea	0.78	0.79	0.73	0.64
Turkey	0.83	0.85	0.78	0.75	Netherlands	0.80	0.78	0.72	0.64
Greece	0.85	0.84	0.70	0.72	France	0.80	0.78	0.70	0.62
Luxembourg	0.86	0.84	0.70	0.71	New Zealand	0.82	0.77	0.69	0.62
Slovak Republic	0.85	0.83	0.73	0.71	Canada	0.79	0.79	0.66	0.60
Slovenia	0.84	0.83	0.66	0.70	Finland	0.80	0.79	0.67	0.60
United States	0.83	0.82	0.73	0.70	Sweden	0.81	0.77	0.65	0.60
Belgium	0.84	0.81	0.70	0.69	Czech Republic	0.81	0.81	0.65	0.59
Spain	0.84	0.83	0.67	0.69	United Kingdom	0.78	0.75	0.72	0.59
Italy	0.84	0.81	0.68	0.68	Australia	0.78	0.78	0.67	0.57
Denmark	0.83	0.80	0.72	0.67	Iceland	0.81	0.80	0.59	0.57
Austria	0.80	0.81	0.67	0.65	Norway	0.80	0.79	0.63	0.56
Estonia	0.83	0.79	0.67	0.65	Japan	0.76	0.77	0.66	0.55
Germany	0.84	0.81	0.64	0.65	Median	0.82	0.81	0.69	0.65

Note: Table adapted from (OECD, 2014, p. 352). HISEI = highest occupational status of parents; PARED = highest educational level of parents; HOME = home possessions; ESCS = economic, social and cultural status indicator.

Figure 1. Factor analytic representation of partial measurement invariance where one item is allowed a different factor loading between two groups.

Table 3. Comparison of international model to increasingly tailored country-specific models.

Model	Chi-square	DF	p	RMSEA	CFI	TLI	Chi-Diff	DF	p	ΔRMSEA	ΔCFI
(1) All parameters equal	3249.33	97	<.01	0.075	0.999	1.000	–	–	–	–	–
(2) Some parameters free	2600.43	88	<.01	0.071	0.999	1.000	1098.71	9	<.01	0.004	0.00
(3a) Denmark baseline	335.61	36	<.01	0.034	0.847	0.910	–	–	–	–	–
(3b) Denmark tailored	197.11	34	<.01	0.025	0.916	0.948	307.74	2	<.01	0.009	−0.069
(3c) Denmark all free	73.17	14	<.01	0.024	0.970	0.955	159.8	20	<.01	0.001	−0.054
(4a) Finland baseline	1316.58	43	<.01	0.058	0.454	0.644	–	–	–	–	–
(4b) Finland tailored	477.96	40	<.01	0.035	0.812	0.868	847.99	3	<.01	0.023	−0.358
(4c) Finland all free	431.29	20	<.01	0.049	0.824	0.753	209.75	20	<.01	−0.014	−0.012
(5a) Iceland baseline	413.77	43	<.01	0.05	0.635	0.762	–	–	–	–	–
(5b) Iceland tailored	240.52	40	<.01	0.038	0.803	0.862	216.98	3	<.01	0.012	−0.168
(5c) Iceland all free	237.93	20	<.01	0.057	0.786	0.700	111.76	20	<.01	−0.019	0.017
(6a) Norway baseline	2067.93	36	<.01	0.08	0.196	0.531	–	–	–	–	–
(6b) Norway tailored	557.65	34	<.01	0.067	0.466	0.670	508.41	2	<.01	0.013	−0.27
(6c) Norway all free	217.43	14	<.01	0.065	0.793	0.689	433.72	20	<.01	0.002	−0.327
(7a) Sweden baseline	531.813	43	<.01	0.049	0.765	0.847	–	–	–	–	–
(7b) Sweden tailored	212.599	40	<.01	0.03	0.917	0.942	314.66	3	<.01	0.019	−0.152
(7c) Sweden all free	205.08	20	<.01	0.045	0.911	0.875	97.26	20	<.01	−0.015	0.006

Note: Chi-diff = chi-square difference test, accounting for categorical indicators. We used the DIFFTEST option in Mplus (Muthén & Muthén, 1998).

OECD. (2014). Pisa 2012 technical report. Paris: OECD Publishing.

 Google Scholar

Muthén, L., & Muthén, B. O. (1998). Mplus user’s guide (7th ed.). Los Angeles, CA: Muthén & Muthén.

 Google Scholar