The blind side: Exploring item variance in PISA 2018 cognitive domains

Figures & data

Figure 1. PISA 2018 3-level response data structure.

Figure 2. PISA 2018 reading literacy variance components (VC) for pupils, schools and items plotted against the item to person variance components ratios (VCR).

Note. The 77 countries are arranged in descending order of variance components ratio. The 8 countries that participated in the paper-based PISA 2018 are marked by an asterisk.

Figure 3. PISA 2018 mathematical literacy variance components (VC) for pupils, schools and items plotted against the item to person variance components ratios (VCR).

Note. The 77 countries are arranged in descending order of variance components ratio. The 8 countries that participated in the paper-based PISA 2018 are marked by an asterisk.

Figure 4. PISA 2018 science literacy variance components (VC) for pupils, schools and items plotted against the item to person variance components ratios (VCR).

Note. The 77 countries are arranged in descending order of variance components ratio. The 8 countries that participated in the paper-based PISA 2018 are marked by an asterisk.

Figure 5. PISA 2018 pupil variance components by domain plotted against across-countries average pupil variance components.

Note. Countries that participated in the paper-based version of PISA 2018 are denoted by an asterisk. Country-specific variance components are shown with a solid line, and across-countries average variance components with a dashed line.

Figure 6. PISA 2018 school variance components by domain plotted against across-countries average school variance components.

Note. Countries that participated in the paper-based version of PISA 2018 are denoted by an asterisk. Country-specific variance components are shown with a solid line, and across-countries average variance components with a dashed line.

Figure 7. PISA 2018 item variance components by domain plotted against across-countries average item variance components.

Note. Countries that participated in the paper-based version of PISA 2018 are denoted by an asterisk. Country-specific variance components are shown with a solid line, and across-countries average variance components with a dashed line.

Table A1. Country-wise parameter estimates of the cross-classified mixed effects model for the PISA 2018 reading literacy domain.

Country	${\mathrm{N}}_{\mathrm{r}}$	${\mathrm{N}}_{\mathrm{p}}$	${\mathrm{N}}_{\mathrm{s}}$	${\mathrm{N}}_{\mathrm{i}}$	$b_0$ (SE)	${\sigma }_{\theta }^2$	${\sigma }_{\zeta }^2$	${\sigma }_{\beta }^2$
Albania	347,571	6341	327	308	0.09 (0.09)	0.52	0.20	2.54
Baku (Azerbaijan)	339,140	6822	197	309	−0.02 (0.10)	0.52	0.10	2.99
Argentina*	255,113	11,682	455	86	0.03 (0.16)	0.71	0.38	2.19
Australia	794,697	14,236	763	309	1.13 (0.09)	1.09	0.23	2.37
Austria	372,310	6802	291	309	0.87 (0.11)	0.70	0.60	2.81
Belgium	445,698	8460	288	309	0.98 (0.09)	0.62	0.55	2.01
Bosnia and Herzegovina	337,860	6478	213	309	0.17 (0.11)	0.56	0.19	3.11
Brazil	537,973	10,675	597	309	0.12 (0.10)	0.66	0.50	2.69
Brunei Darussalam	370,513	6821	55	308	0.16 (0.13)	0.69	0.47	2.47
Bulgaria	277,569	5279	197	309	0.24 (0.11)	0.57	0.60	2.65
Belarus	313,769	5800	234	308	0.71 (0.09)	0.62	0.26	2.15
Canada	1,223,773	22,629	821	309	1.18 (0.09)	1.01	0.19	2.30
Chile	400,308	7601	254	308	0.76 (0.10)	0.72	0.42	2.71
Chinese Taipei	410,057	7243	192	308	1.00 (0.10)	0.69	0.34	2.55
Colombia	390,308	7505	247	308	0.17 (0.10)	0.61	0.38	2.47
Costa Rica	354,675	7218	205	309	0.25 (0.10)	0.54	0.25	2.81
Croatia	364,082	6605	183	309	0.89 (0.10)	0.57	0.38	2.71
Czech Republic	386,821	7016	333	308	1.14 (0.10)	0.59	0.63	2.49
Denmark	401,867	7643	348	307	1.05 (0.09)	0.92	0.19	2.52
Dominican Republic	273,416	5672	235	308	−0.70 (0.10)	0.44	0.29	2.64
Estonia	297,711	5313	230	309	1.31 (0.09)	0.80	0.19	2.02
Finland	311,207	5647	214	309	1.39 (0.10)	1.13	0.09	2.71
France	333,574	6305	252	309	0.89 (0.10)	0.60	0.60	2.31
Georgia	275,241	5561	321	309	−0.13 (0.10)	0.55	0.20	2.60
Germany	291,620	5441	223	309	1.08 (0.11)	0.69	0.68	2.67
Greece	340,543	6399	242	308	0.61 (0.09)	0.70	0.39	2.17
Hong Kong	336,627	6024	152	309	1.27 (0.09)	0.59	0.31	1.89
Hungary	283,839	5130	238	309	0.71 (0.10)	0.46	0.60	2.27
Iceland	172,755	3294	142	308	0.74 (0.09)	1.14	0.09	2.20
Indonesia	666,995	12,080	397	304	−0.06 (0.09)	0.33	0.34	2.17
Ireland	311,077	5577	157	309	1.39 (0.10)	0.89	0.15	2.95
Israel	326,156	6618	174	318	0.76 (0.11)	0.83	0.77	2.05
Italy	639,010	11,781	542	309	0.80 (0.09)	0.60	0.40	2.13
Kosovo	272,933	5057	211	309	−0.64 (0.11)	0.40	0.18	3.07
Japan	332,520	6107	183	309	1.22 (0.10)	0.60	0.40	2.54
Kazakhstan	1,038,287	19,501	616	309	−0.06 (0.08)	0.41	0.25	1.98
Jordan*	236,070	8952	313	86	−0.12 (0.16)	0.76	0.33	1.99
Korea	379,638	6647	188	309	1.23 (0.08)	0.71	0.24	1.78
Lebanon*	120,303	5554	313	86	−0.78 (0.14)	0.72	0.95	1.34
Latvia	290,121	5302	308	309	0.72 (0.09)	0.67	0.19	2.07
Lithuania	378,840	6883	362	309	0.61 (0.10)	0.68	0.42	2.48
Luxembourg	276,049	5229	44	309	0.84 (0.13)	0.90	0.39	2.23
Macao	206,295	3772	45	309	1.10 (0.11)	0.56	0.26	2.04
Malaysia	330,482	6109	191	308	0.07 (0.10)	0.56	0.25	2.47
Malta	184,947	3360	50	309	0.68 (0.12)	1.10	0.31	2.64
Mexico	389,081	7292	286	309	0.16 (0.10)	0.52	0.33	2.65
Moldova*	129,892	5345	236	87	−0.01 (0.17)	0.80	0.30	2.30
Montenegro	350,054	6658	61	309	0.30 (0.11)	0.55	0.24	2.68
Morocco	321,293	6802	179	303	−0.44 (0.10)	0.32	0.25	2.58
Netherlands	234,688	4761	156	307	0.88 (0.11)	0.60	0.73	2.13
New Zealand	342,383	6171	192	309	1.20 (0.09)	1.09	0.20	2.35
Norway	312,974	5806	251	309	1.11 (0.09)	1.09	0.10	2.22
Panama	291,269	6263	253	309	−0.37 (0.10)	0.45	0.38	2.73
Peru	283,495	6073	340	308	−0.07 (0.09)	0.47	0.34	2.19
Philippines	384,381	7229	187	307	−0.72 (0.10)	0.48	0.29	2.75
Poland	312,194	5624	240	308	1.23 (0.09)	0.85	0.18	2.28
Portugal	321,743	5928	276	308	0.98 (0.10)	0.81	0.31	2.40
Qatar	729,462	13,820	188	309	0.00 (0.10)	0.66	0.51	2.06
Romania*	130,391	5066	170	87	−0.11 (0.18)	0.72	0.72	2.55
Russian Federation	415,796	7602	263	307	0.83 (0.10)	0.71	0.26	2.50
Saudi Arabia*	161,427	6129	234	86	−0.47 (0.17)	0.69	0.42	2.32
Serbia	347,303	6605	187	309	0.5 (0.10)	0.58	0.41	2.51
Singapore	380,970	6675	166	309	1.52 (0.10)	0.84	0.39	2.03
Slovak Republic	319,138	5955	376	309	0.51 (0.10)	0.64	0.55	2.45
Slovenia	351,454	6398	345	309	0.75 (0.10)	0.58	0.52	2.59
Spain	1,948,563	35,900	1089	309	0.91 (0.08)	0.73	0.13	2.06
Sweden	288,934	5494	223	308	1.18 (0.09)	1.09	0.21	2.23
Switzerland	315,759	5817	228	309	0.93 (0.10)	0.73	0.43	2.31
Thailand	477,619	8624	290	309	0.01 (0.09)	0.39	0.39	1.89
United Arab Emirates	1,064,061	19,261	755	309	0.17 (0.09)	0.68	0.63	2.08
Turkey	377,022	6888	186	308	0.61 (0.09)	0.34	0.45	1.70
Ukraine*	156,331	5992	250	88	0.28 (0.17)	0.91	0.38	2.47
North Macedonia*	127,965	5540	117	87	−0.37 (0.16)	0.72	0.38	1.96
United Kingdom	771,581	13,791	471	309	1.17 (0.09)	0.93	0.21	2.44
United States	268,348	4828	164	309	1.16 (0.10)	1.24	0.25	2.82
Uruguay	261,161	5255	189	308	0.28 (0.10)	0.64	0.38	2.32
B-S-J-Z (China)	690,965	12,055	361	308	1.70 (0.09)	0.46	0.40	2.13

Note. The number of responses, pupils, schools, and items used in the analysis are denoted as ${\mathrm{N}}_{\mathrm{r}}$, ${\mathrm{N}}_{\mathrm{p}}$, ${\mathrm{N}}_{\mathrm{s}}$, ${\mathrm{N}}_{\mathrm{i}}$, respectively. The estimated logit for the probability of a correct response of an average pupil from an average school on an average item is denoted as $b_0$, and its standard error is denoted as SE. The variances of the random pupil, school, and item effect are denoted as ${\sigma }_{\theta }^2$, ${\sigma }_{\zeta }^2$, ${\sigma }_{\beta }^2$, respectively. Countries that participated in the paper-based PISA 2018 are marked by an asterisk.

Table A2. Country-wise parameter estimates of the cross-classified mixed effects model for the PISA 2018 mathematical literacy domain.

Country	${\mathrm{N}}_{\mathrm{r}}$	${\mathrm{N}}_{\mathrm{p}}$	${\mathrm{N}}_{\mathrm{s}}$	${\mathrm{N}}_{\mathrm{i}}$	$b_0$ (SE)	${\sigma }_{\theta }^2$	${\sigma }_{\zeta }^2$	${\sigma }_{\beta }^2$
Albania	49,855	2593	311	69	−1.02 (0.17)	0.61	0.23	1.81
Baku (Azerbaijan)	66,981	3660	197	70	−0.98 (0.17)	0.76	0.14	1.86
Argentina*	95,657	6459	453	71	−1.24 (0.19)	0.59	0.35	2.62
Australia	153,027	7645	762	70	−0.31 (0.18)	1.04	0.34	2.37
Austria	72,523	3718	289	70	−0.40 (0.18)	0.72	0.73	2.07
Belgium	89,133	4680	287	70	−0.13 (0.18)	0.77	0.82	2.08
Bosnia and Herzegovina	60,763	3498	213	70	−1.28 (0.18)	0.65	0.21	2.24
Brazil	103,849	5672	588	70	−1.68 (0.20)	0.59	0.50	2.85
Brunei Darussalam	54,572	2805	55	69	−1.02 (0.23)	0.85	0.61	2.74
Bulgaria	54,470	2839	197	70	−0.91 (0.18)	0.59	0.63	2.08
Belarus	62,328	3131	233	70	−0.65 (0.19)	0.99	0.44	2.28
Canada	181,857	9723	805	70	−0.18 (0.18)	1.13	0.24	2.17
Chile	56,410	3101	249	70	−0.93 (0.19)	0.66	0.45	2.45
Chinese Taipei	58,986	2954	192	70	0.20 (0.16)	1.02	0.62	1.63
Colombia	57,943	3051	244	70	−1.55 (0.21)	0.65	0.42	2.90
Costa Rica	54,261	3288	205	70	−1.56 (0.22)	0.49	0.26	3.40
Croatia	49,572	2678	183	70	−0.67 (0.19)	0.67	0.41	2.26
Czech Republic	74,442	3793	333	70	−0.06 (0.18)	0.73	0.79	1.98
Denmark	81,378	4326	345	70	−0.29 (0.20)	1.07	0.23	2.60
Dominican Republic	55,535	3013	235	70	−2.53 (0.24)	0.37	0.25	3.71
Estonia	58,309	2872	229	70	0.08 (0.18)	1.03	0.24	2.24
Finland	59,518	3043	212	70	−0.13 (0.20)	1.17	0.07	2.76
France	64,694	3386	251	70	−0.43 (0.19)	0.71	0.83	2.24
Georgia	52,225	2970	317	70	−1.50 (0.18)	0.71	0.32	2.26
Germany	57,417	2992	223	70	−0.20 (0.19)	0.80	0.81	2.15
Greece	49,317	2624	237	70	−0.82 (0.19)	0.86	0.37	2.35
Hong Kong	50,358	2480	152	70	0.50 (0.17)	0.85	0.60	1.73
Hungary	55,074	2769	233	70	−0.59 (0.20)	0.56	0.89	2.38
Iceland	34,358	1782	139	70	−0.38 (0.20)	1.20	0.13	2.56
Indonesia	101,284	4936	394	70	−1.67 (0.21)	0.49	0.63	2.87
Ireland	61,461	3026	157	70	−0.30 (0.20)	0.89	0.15	2.70
Israel	51,651	2805	174	70	−0.69 (0.19)	0.95	0.85	2.16
Italy	121,867	6375	537	70	−0.29 (0.18)	0.74	0.55	2.07
Kosovo	54,222	2703	210	70	−1.95 (0.20)	0.52	0.21	2.61
Japan	65,185	3295	183	70	0.22 (0.18)	0.65	0.62	1.93
Kazakhstan	153,487	7969	607	70	−1.04 (0.17)	0.60	0.42	1.87
Jordan*	93,727	4993	313	71	−1.40 (0.18)	0.58	0.23	2.15
Korea	54,049	2729	185	70	0.16 (0.17)	0.99	0.45	1.77
Lebanon*	49,481	3108	312	70	−1.22 (0.17)	0.71	0.83	1.90
Latvia	42,769	2184	306	70	−0.37 (0.18)	0.84	0.22	2.25
Lithuania	54,525	2820	348	70	−0.62 (0.19)	0.93	0.45	2.32
Luxembourg	54,537	2821	44	70	−0.32 (0.20)	0.98	0.49	2.04
Macao	42,512	2037	45	70	0.30 (0.19)	0.90	0.46	1.86
Malaysia	69,111	3282	191	70	−1.20 (0.21)	0.73	0.35	2.81
Malta	25,892	1364	50	69	−0.37 (0.21)	1.24	0.38	2.29
Mexico	76,823	3909	280	70	−1.38 (0.21)	0.56	0.34	2.85
Moldova*	48,132	2953	235	71	−0.84 (0.18)	0.92	0.24	2.13
Montenegro	63,989	3579	61	70	−1.02 (0.19)	0.67	0.23	2.12
Morocco	48,182	2754	179	69	−1.94 (0.21)	0.49	0.26	2.86
Netherlands	49,424	2925	156	70	−0.14 (0.20)	0.61	1.12	2.40
New Zealand	65,735	3296	192	70	−0.25 (0.19)	1.22	0.25	2.34
Norway	60,287	3122	250	70	−0.15 (0.18)	1.24	0.07	2.31
Panama	41,080	2476	246	70	−2.03 (0.21)	0.39	0.34	3.05
Peru	55,494	3189	336	70	−1.40 (0.19)	0.65	0.36	2.32
Philippines	57,143	2946	187	70	−2.09 (0.22)	0.49	0.23	3.20
Poland	60,749	3014	234	70	0.05 (0.18)	1.07	0.27	2.07
Portugal	61,772	3186	276	70	−0.27 (0.18)	1.09	0.34	2.05
Qatar	143,071	7383	187	70	−1.42 (0.19)	0.81	0.66	2.30
Romania*	51,144	2834	170	71	−1.03 (0.19)	0.76	0.61	2.30
Russian Federation	60,258	3125	258	70	−0.42 (0.19)	0.82	0.34	2.28
Saudi Arabia*	65,003	3409	234	71	−1.84 (0.21)	0.61	0.25	2.94
Serbia	48,432	2698	183	82	−0.63 (0.17)	0.78	0.52	2.13
Singapore	55,921	2724	166	70	0.65 (0.17)	1.10	0.54	1.70
Slovak Republic	46,738	2504	373	70	−0.55 (0.18)	0.81	0.72	2.03
Slovenia	68,261	3515	340	70	−0.39 (0.18)	0.66	0.76	2.13
Spain	276,629	14,713	1089	70	−0.32 (0.18)	0.98	0.15	2.32
Sweden	55,857	2961	220	70	−0.15 (0.19)	1.14	0.25	2.34
Switzerland	61,662	3137	227	70	−0.02 (0.17)	0.89	0.49	1.88
Thailand	71,457	3536	286	70	−1.14 (0.18)	0.63	0.97	2.11
United Arab Emirates	213,751	10,356	747	70	−1.09 (0.17)	0.77	0.82	2.00
Turkey	74,810	3715	186	70	−0.81 (0.20)	0.47	0.91	2.50
Ukraine*	59,887	3349	250	71	−0.68 (0.19)	0.99	0.35	2.35
North Macedonia*	49,293	3101	113	71	−1.12 (0.18)	0.72	0.32	2.13
United Kingdom	139,349	7002	470	70	−0.25 (0.18)	1.00	0.29	2.31
United States	54,019	2619	163	70	−0.58 (0.20)	1.25	0.32	2.56
Uruguay	49,470	2798	189	70	−1.23 (0.21)	0.65	0.35	2.86
B-S-J-Z (China)	138,662	6505	361	70	1.25 (0.15)	0.74	0.72	1.49

Note. The number of responses, pupils, schools, and items used in the analysis are denoted as ${\mathrm{N}}_{\mathrm{r}}$, ${\mathrm{N}}_{\mathrm{p}}$, ${\mathrm{N}}_{\mathrm{s}}$, ${\mathrm{N}}_{\mathrm{i}}$. The estimated logit for the probability of a correct response of an average pupil from an average school on an average item is denoted as $b_0$, and its standard error is denoted as SE. The variances of the random pupil, school, and item effect are denoted as ${\sigma }_{\theta }^2$, ${\sigma }_{\zeta }^2$, ${\sigma }_{\beta }^2$, respectively. Countries that participated in the paper-based PISA 2018 are marked by an asterisk.

Table A3. Country-wise parameter estimates of the cross-classified mixed effects model for the PISA 2018 science literacy domain.

Country	${\mathrm{N}}_{\mathrm{r}}$	${\mathrm{N}}_{\mathrm{p}}$	${\mathrm{N}}_{\mathrm{s}}$	${\mathrm{N}}_{\mathrm{i}}$	$b_0$ (SE)	${\sigma }_{\theta }^2$	${\sigma }_{\zeta }^2$	${\sigma }_{\beta }^2$
Albania	84,282	2586	304	114	−1.00 (0.12)	0.45	0.17	1.45
Baku (Azerbaijan)	108,114	3663	197	115	−1.12 (0.12)	0.46	0.09	1.69
Argentina*	122,931	6389	452	84	−0.50 (0.10)	0.42	0.21	0.87
Australia	255,968	7698	762	115	0.01 (0.12)	1.04	0.24	1.72
Austria	119,837	3696	285	115	−0.19 (0.13)	0.64	0.57	1.68
Belgium	140,286	4696	287	115	−0.02 (0.12)	0.62	0.59	1.45
Bosnia and Herzegovina	108,054	3541	213	115	−1.17 (0.12)	0.47	0.17	1.53
Brazil	164,733	5739	591	115	−1.11 (0.13)	0.53	0.44	1.74
Brunei Darussalam	88,992	2788	55	115	−0.66 (0.15)	0.71	0.54	1.54
Bulgaria	88,520	2826	197	114	−0.92 (0.13)	0.46	0.59	1.52
Belarus	100,404	3119	233	115	−0.40 (0.12)	0.69	0.27	1.55
Canada	290,324	9716	808	115	0.09 (0.12)	0.96	0.15	1.58
Chile	93,922	3120	248	115	−0.52 (0.13)	0.56	0.37	1.79
Chinese Taipei	99,069	2963	192	115	0.14 (0.13)	0.83	0.49	1.74
Colombia	89,697	3061	244	114	−0.95 (0.13)	0.60	0.32	1.77
Costa Rica	85,268	3314	205	115	−1.07 (0.14)	0.45	0.20	1.95
Croatia	85,941	2692	183	115	−0.33 (0.13)	0.59	0.41	1.53
Czech Republic	125,876	3817	333	115	0.03 (0.13)	0.58	0.68	1.60
Denmark	130,139	4335	348	115	−0.28 (0.12)	0.99	0.20	1.65
Dominican Republic	84,389	3062	235	115	−1.88 (0.14)	0.40	0.21	2.14
Estonia	95,786	2843	226	115	0.34 (0.12)	0.76	0.23	1.46
Finland	101,217	3069	213	115	0.27 (0.13)	1.18	0.07	1.81
France	105,430	3418	252	115	−0.27 (0.12)	0.62	0.60	1.45
Georgia	87,690	3001	318	115	−1.37 (0.13)	0.51	0.20	1.86
Germany	93,085	2963	223	115	0.00 (0.13)	0.66	0.71	1.58
Greece	81,421	2618	236	115	−0.57 (0.12)	0.62	0.25	1.40
Hong Kong	79,723	2475	152	115	0.37 (0.13)	0.66	0.30	1.63
Hungary	89,933	2763	234	113	−0.38 (0.14)	0.50	0.73	1.65
Iceland	55,154	1800	132	115	−0.32 (0.12)	0.97	0.07	1.56
Indonesia	164,858	4950	396	114	−1.03 (0.12)	0.33	0.35	1.65
Ireland	99,564	2994	157	115	−0.01 (0.13)	0.89	0.13	1.71
Israel	84,244	2801	174	115	−0.42 (0.13)	0.79	0.62	1.37
Italy	201,694	6292	536	115	−0.30 (0.12)	0.60	0.42	1.65
Kosovo	86,771	2745	208	115	−1.60 (0.12)	0.32	0.16	1.47
Japan	107,444	3289	183	114	0.31 (0.14)	0.64	0.49	1.90
Kazakhstan	245,654	8001	609	115	−1.15 (0.12)	0.44	0.32	1.66
Jordan*	116,859	4995	313	85	−0.41 (0.10)	0.50	0.15	0.72
Korea	88,815	2715	187	112	0.25 (0.14)	0.93	0.36	1.78
Lebanon*	61,649	3092	308	85	−0.75 (0.10)	0.40	0.41	0.65
Latvia	69,263	2167	304	115	−0.19 (0.12)	0.70	0.17	1.53
Lithuania	90,116	2791	349	115	−0.34 (0.12)	0.70	0.37	1.60
Luxembourg	88,088	2823	44	115	−0.23 (0.15)	0.79	0.41	1.53
Macao	65,582	2032	45	115	0.28 (0.15)	0.65	0.29	1.68
Malaysia	114,175	3313	191	115	−0.77 (0.14)	0.57	0.22	2.15
Malta	43,679	1366	50	114	−0.34 (0.14)	1.12	0.23	1.43
Mexico	122,116	3920	283	115	−0.97 (0.13)	0.47	0.24	1.78
Moldova*	62,798	2979	235	85	−0.39 (0.09)	0.48	0.14	0.68
Montenegro	111,472	3592	61	115	−0.93 (0.13)	0.54	0.20	1.44
Morocco	76,093	2769	179	115	−1.52 (0.13)	0.29	0.21	1.88
Netherlands	80,459	2961	156	115	−0.06 (0.14)	0.55	0.89	1.65
New Zealand	109,081	3344	192	114	0.09 (0.13)	1.13	0.21	1.61
Norway	97,690	3127	249	115	−0.07 (0.12)	1.08	0.12	1.53
Panama	63,141	2528	250	115	−1.50 (0.13)	0.40	0.29	1.69
Peru	85,183	3254	337	115	−1.14 (0.13)	0.45	0.29	1.70
Philippines	93,618	2951	187	115	−1.70 (0.14)	0.46	0.21	1.99
Poland	101,778	3045	238	115	0.16 (0.12)	0.89	0.20	1.61
Portugal	102,897	3195	276	115	−0.11 (0.12)	0.87	0.28	1.58
Qatar	232,452	7415	187	115	−0.94 (0.12)	0.69	0.51	1.41
Romania*	63,358	2821	169	85	−0.49 (0.11)	0.41	0.29	0.80
Russian Federation	98,482	3101	262	115	−0.31 (0.12)	0.68	0.24	1.63
Saudi Arabia*	80,021	3416	234	84	−1.08 (0.11)	0.45	0.17	0.99
Serbia	80,778	2701	185	115	−0.63 (0.13)	0.56	0.34	1.55
Singapore	92,250	2756	166	115	0.56 (0.13)	0.88	0.36	1.79
Slovak Republic	77,709	2534	370	115	−0.56 (0.13)	0.66	0.60	1.57
Slovenia	114,865	3533	336	115	−0.23 (0.13)	0.56	0.58	1.69
Spain	458,655	14,637	1088	115	−0.07 (0.11)	0.81	0.11	1.52
Sweden	89,911	2956	217	115	0.01 (0.12)	1.01	0.19	1.50
Switzerland	101,256	3140	226	115	−0.10 (0.12)	0.73	0.42	1.47
Thailand	115,929	3542	283	115	−0.76 (0.14)	0.51	0.67	1.90
United Arab Emirates	352,410	10,404	748	115	−0.92 (0.12)	0.65	0.66	1.60
Turkey	121,413	3716	186	115	−0.41 (0.12)	0.31	0.57	1.33
Ukraine*	76,669	3349	250	85	−0.10 (0.11)	0.60	0.20	0.94
North Macedonia*	63,066	3072	113	85	−0.56 (0.09)	0.44	0.19	0.55
United Kingdom	234,732	7029	471	115	0.02 (0.12)	0.92	0.22	1.71
United States	86,839	2617	164	115	−0.05 (0.13)	1.13	0.30	1.75
Uruguay	80,104	2834	188	115	−0.82 (0.13)	0.59	0.29	1.58
B-S-J-Z (China)	225,216	6498	361	115	0.98 (0.13)	0.50	0.47	1.72

Note. The number of responses, pupils, schools, and items used in the analysis are denoted as ${\mathrm{N}}_{\mathrm{r}}$, ${\mathrm{N}}_{\mathrm{p}}$, ${\mathrm{N}}_{\mathrm{s}}$, ${\mathrm{N}}_{\mathrm{i}}$. The estimated logit for the probability of a correct response of an average pupil from an average school on an average item is denoted as $b_0$, and its standard error is denoted as SE. The variances of the random pupil, school, and item effect are denoted as ${\sigma }_{\theta }^2$, ${\sigma }_{\zeta }^2$, ${\sigma }_{\beta }^2$, respectively. Countries that participated in the paper-based PISA 2018 are marked by an asterisk.

Table B1. Number of PISA items by domain for computer-based (CBA) and paper-based (PBA) Assessments.

Domain	Mode	Country	Items
			Unique	Common	Total
Reading	PBA	Argentina, Jordan, Lebanon,	15	$72(70{)}^a$	103
		Moldova, Romania,
		Saudi Arabia, North Macedonia
		Ukraine	16
	CBA	All CBA countries	–	$309(287{)}^b$	318
		Israel	9
Mathematics	PBA	Argentina, Jordan, Lebanon,	12	$59(58{)}^c$	83
		Moldova, Romania,
		Saudi Arabia, North Macedonia
		Ukraine	12
	CBA	$\mathrm{C}\mathrm{B}\mathrm{A}\mathrm{S}\mathrm{u}\mathrm{b}\mathrm{s}et{1}^d$	12	$58(57{)}^g$	82
		$\mathrm{C}\mathrm{B}\mathrm{A}\mathrm{S}\mathrm{u}\mathrm{b}\mathrm{s}\mathrm{e}\mathrm{t}{2}^e$	$12(11{)}^f$
Science	PBA	Argentina, Jordan, Lebanon,	–	$85(83{)}^h$	85
		Moldova, Romania,Ukraine,
		Saudi Arabia, North Macedonia,
	CBA	All CBA countries	–	$115(108{)}^j$	115

Note. All CBA countries refer to 69 out of 70 countries that took the CBA version of the PISA 2018. Excluded is Cyprus due to lack of available data.

${ }^a$ One of the common items was not administered in Jordan, Lebanon, and Saudi Arabia; one more item was not administered in Argentina.

${ }^b$ Thirty-one of the common items (one to six items per country) were not administered in one to two of the CBA countries.

${ }^c$ One of the common items was not administered in Lebanon.

${ }^d$ Baku (Azerbaijan), Brazil, Bulgaria, Chile, Colombia, Costa Rica, Dominican Republic, Kosovo, Kazakhstan, Mexico, Morocco, Panama, Peru, Philippines, Serbia, United Arab Emirates, Uruguay.

${ }^e$ Albania, Australia, Austria, Belgium, Bosnia and Herzegovina, Brunei Darussalam, Belarus, Canada, Chinese Taipei, Croatia, Czech Republic, Denmark, Estonia, Finland, France, Georgia, Germany, Greece, Hong Kong, Hungary, Iceland, Indonesia, Ireland, Israel, Italy, Japan, Korea, Latvia, Lithuania, Luxembourg, Macao, Malaysia, Malta, Montenegro, Netherlands, New Zealand, Norway, Poland, Portugal, Qatar, Russian Federation, Serbia, Singapore, Slovak Republic, Slovenia, Spain, Sweden, Switzerland, Thailand, Turkey, United Kingdom, United States, B-S-J-Z (China).

${ }^f$ One item of the subset was not administered in Albania.

${ }^g$ Three of the common items were not administered in Brunei Darussalam, Malta, and Morocco.

${ }^h$ One of the common items was not administered to Argentina, and one item was not administered to Saudi Arabia.

${ }^j$ Seven of the common items were not administered in one to two of the CBA countries.

Table C1. Sensitivity analysis resulting country-wise variance components (VC) and variance components ratios (VCR).

Country	Reading				Mathematics				Science
	$\mathrm{V}{\mathrm{C}}_{\mathrm{p}\mathrm{u}\mathrm{p}\mathrm{i}\mathrm{l}}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{o}\mathrm{l}}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{m}}$	$\mathrm{V}\mathrm{C}\mathrm{R}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{p}\mathrm{u}\mathrm{p}\mathrm{i}\mathrm{l}}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{o}\mathrm{l}}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{m}}$	$\mathrm{V}\mathrm{C}\mathrm{R}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{p}\mathrm{u}\mathrm{p}\mathrm{i}\mathrm{l}}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{s}\mathrm{c}\mathrm{h}\mathrm{o}\mathrm{o}\mathrm{l}}$	$\mathrm{V}{\mathrm{C}}_{\mathrm{i}\mathrm{t}\mathrm{e}\mathrm{m}}$	$\mathrm{V}\mathrm{C}\mathrm{R}$
Albania	8.4	3.2	38.5	3.3	10.7	3.8	29.5	2.0	8.5	3.2	27.9	2.4
Baku (Azerbaijan)	8.3	1.6	43.8	4.4	12.9	2.6	28.2	1.8	8.6	1.8	31.4	3.0
Argentina*	11.6	5.9	31.7	1.8	8.3	4.8	39.0	3.0	8.7	4.5	18.3	1.4
Australia	16.2	3.5	33.1	1.7	14.7	4.7	33.8	1.7	17.0	4.0	28.3	1.3
Austria	10.0	8.6	37.6	2.0	10.8	11.1	30.2	1.4	10.7	9.8	28.2	1.4
Belgium	10.0	8.9	30.9	1.6	11.3	11.9	29.6	1.3	10.8	10.2	25.4	1.2
Bosnia and Herzegovina	8.3	2.9	43.2	3.9	10.5	3.2	35.0	2.6	9.0	3.3	29.2	2.4
Brazil	10.1	7.6	37.6	2.1	7.7	6.4	40.9	2.9	9.1	7.5	30.2	1.8
Brunei Darussalam	10.7	7.2	35.2	2.0	11.1	7.6	35.4	1.9	12.3	8.9	26.3	1.2
Bulgaria	8.5	9.1	36.9	2.1	9.2	9.6	31.7	1.7	8.2	10.5	27.0	1.4
Belarus	10.3	4.5	34.1	2.3	13.7	5.9	32.5	1.7	12.4	4.9	27.7	1.6
Canada	15.5	2.9	33.1	1.8	16.7	3.4	31.4	1.6	16.5	2.6	27.5	1.4
Chile	10.5	6.2	37.6	2.3	9.4	6.5	36.7	2.3	9.8	6.4	31.1	1.9
Chinese Taipei	10.7	5.3	36.9	2.3	15.3	9.0	25.0	1.0	13.7	8.0	28.5	1.3
Colombia	9.5	5.9	36.6	2.4	8.4	5.5	41.3	3.0	10.4	5.7	30.2	1.9
Costa Rica	8.3	4.0	40.7	3.3	6.3	3.0	47.6	5.1	8.1	3.6	34.0	2.9
Croatia	8.6	5.7	39.1	2.7	10.1	6.1	33.8	2.1	10.6	7.3	27.4	1.5
Czech Republic	8.8	9.5	34.9	1.9	11.2	11.7	28.5	1.2	9.8	11.5	26.9	1.3
Denmark	14.0	2.9	36.3	2.1	15.0	3.1	35.4	2.0	16.9	3.4	27.4	1.3
Dominican Republic	6.9	4.7	38.8	3.3	4.3	3.1	50.7	6.9	6.9	3.7	36.5	3.4
Estonia	13.4	3.3	31.8	1.9	15.6	3.6	32.6	1.7	13.7	3.9	26.7	1.5
Finland	16.6	1.3	37.6	2.1	15.5	1.0	38.2	2.3	19.1	1.1	29.8	1.5
France	9.3	9.4	33.8	1.8	10.3	11.4	31.8	1.5	10.8	10.4	25.2	1.2
Georgia	9.0	3.3	39.0	3.2	10.9	4.8	33.8	2.2	9.2	3.5	33.0	2.6
Germany	9.9	9.9	36.1	1.8	11.3	11.7	30.3	1.3	11.0	11.8	26.2	1.1
Greece	11.2	6.2	33.0	1.9	12.9	5.4	33.6	1.8	11.7	4.7	26.1	1.6
Hong Kong	10.2	5.3	31.2	2.0	12.9	9.2	26.0	1.2	11.8	5.4	28.8	1.7
Hungary	7.4	9.9	33.9	2.0	8.2	12.4	33.2	1.6	8.3	12.1	27.6	1.4
Iceland	17.8	1.5	32.2	1.7	17.5	1.7	33.9	1.8	17.1	1.2	27.5	1.5
Indonesia	5.9	6.0	34.7	2.9	7.1	8.9	38.0	2.4	5.9	6.3	30.1	2.5
Ireland	12.7	2.2	40.0	2.7	12.6	2.1	39.1	2.7	15.2	2.3	29.3	1.7
Israel	12.7	11.7	29.4	1.2	13.6	11.8	28.5	1.1	13.4	10.6	23.6	1.0
Italy	10.0	6.8	32.9	2.0	11.4	8.4	31.1	1.6	10.4	7.3	28.6	1.6
Kosovo	6.2	3.0	44.0	4.8	7.8	2.9	39.6	3.7	6.1	3.0	28.7	3.2
Japan	9.3	6.2	36.8	2.4	10.0	9.1	29.8	1.6	10.5	8.1	30.8	1.7
Kazakhstan	7.3	4.7	33.3	2.8	9.6	6.9	30.2	1.8	7.9	5.9	30.2	2.2
Jordan*	12.2	5.1	31.2	1.8	9.6	3.8	34.8	2.6	10.8	3.2	15.6	1.1
Korea	12.2	4.1	29.9	1.8	15.1	6.4	27.6	1.3	15.0	6.0	28.4	1.4
Lebanon*	11.9	14.7	21.2	0.8	10.3	12.5	28.2	1.2	8.4	8.9	13.9	0.8
Latvia	11.4	3.2	33.4	2.3	12.7	3.3	33.7	2.1	12.8	3.1	27.3	1.7
Lithuania	10.5	6.4	35.8	2.1	13.6	6.0	32.3	1.6	11.9	6.2	28.0	1.5
Luxembourg	14.0	6.1	32.6	1.6	14.5	7.3	30.1	1.4	13.5	7.0	26.5	1.3
Macao	9.5	4.5	33.4	2.4	13.7	6.9	28.5	1.4	11.6	5.2	29.4	1.8
Malaysia	9.0	4.0	37.5	2.9	10.1	5.0	39.0	2.6	9.4	3.7	35.4	2.7
Malta	15.9	4.6	35.3	1.7	16.4	5.2	32.3	1.5	18.8	3.8	24.5	1.1
Mexico	8.0	5.1	38.7	3.0	7.6	4.8	41.8	3.4	8.4	4.3	31.9	2.5
Moldova*	12.8	4.8	34.3	1.9	14.1	3.9	32.1	1.8	10.4	2.9	15.1	1.1
Montenegro	8.8	3.9	39.5	3.1	10.8	3.5	33.1	2.3	10.4	3.8	27.6	1.9
Morocco	5.4	4.3	39.9	4.1	6.5	3.3	41.9	4.3	5.1	3.8	33.7	3.8
Netherlands	9.3	11.3	31.8	1.5	8.4	15.4	31.9	1.3	8.8	14.3	26.8	1.2
New Zealand	16.3	3.0	33.1	1.7	16.9	3.5	33.1	1.6	18.5	3.4	26.8	1.2
Norway	17.2	1.6	32.8	1.7	18.2	1.0	32.2	1.7	18.7	2.0	25.9	1.3
Panama	6.9	6.1	39.6	3.0	5.0	4.4	45.2	4.8	7.5	5.5	30.7	2.4
Peru	8.0	5.7	34.5	2.5	9.4	5.2	35.4	2.4	8.2	5.4	30.8	2.3
Philippines	7.5	4.5	38.9	3.2	6.4	3.0	46.1	4.9	8.1	3.7	33.8	2.9
Poland	13.8	3.0	34.1	2.0	16.1	4.1	31.2	1.5	15.4	3.3	28.1	1.5
Portugal	12.7	5.0	35.3	2.0	15.8	4.7	31.1	1.5	14.8	4.8	27.5	1.4
Qatar	10.9	8.3	31.3	1.6	11.5	8.9	32.3	1.6	12.1	8.9	24.7	1.2
Romania*	10.9	10.1	34.5	1.6	11.2	8.8	33.2	1.7	8.6	6.1	16.9	1.1
Russian Federation	10.9	4.1	36.9	2.5	12.3	4.9	34.0	2.0	12.0	4.2	29.0	1.8
Saudi Arabia*	10.3	5.8	35.7	2.2	8.8	3.5	43.0	3.5	9.1	3.4	20.1	1.6
Serbia	9.1	6.5	36.6	2.3	11.5	7.7	32.9	1.7	10.1	6.2	28.0	1.7
Singapore	13.6	6.2	30.7	1.6	16.9	8.0	24.5	1.0	14.5	5.9	29.6	1.5
Slovak Republic	9.5	8.3	35.1	2.0	11.8	10.4	29.0	1.3	11.2	9.9	26.5	1.3
Slovenia	8.7	7.9	37.2	2.2	10.1	11.1	30.8	1.5	9.3	9.9	28.7	1.5
Spain	12.4	2.3	32.4	2.2	14.4	2.2	35.2	2.1	14.6	2.0	27.5	1.7
Sweden	16.5	3.2	32.5	1.6	16.3	3.5	32.4	1.6	17.6	3.2	25.8	1.2
Switzerland	11.4	6.8	34.1	1.9	14.2	7.3	28.8	1.3	12.9	7.3	25.9	1.3
Thailand	6.8	7.0	31.3	2.3	8.8	13.8	30.3	1.3	8.2	10.7	30.7	1.6
United Arab Emirates	10.8	10.1	30.7	1.5	11.5	12.0	29.2	1.2	10.7	10.8	26.6	1.2
Turkey	6.0	8.3	29.5	2.1	6.9	12.4	32.9	1.7	5.8	10.8	25.2	1.5
Ukraine*	12.8	5.3	37.5	2.1	14.5	4.9	33.5	1.7	12.1	4.0	18.7	1.2
North Macedonia*	12.5	6.3	31.0	1.6	10.6	4.9	33.5	2.2	10.2	4.5	12.4	0.8
United Kingdom	14.1	3.1	34.9	2.0	14.7	4.3	33.3	1.8	15.2	3.6	29.0	1.5
United States	17.0	3.5	36.0	1.8	16.5	4.2	34.8	1.7	17.8	4.8	27.9	1.2
Uruguay	10.1	6.2	34.2	2.1	8.7	4.8	41.3	3.1	10.7	5.4	28.5	1.8
B-S-J-Z (China)	7.6	6.6	33.7	2.4	11.7	11.3	23.9	1.0	8.9	8.1	29.6	1.7