Improved Goodness of Fit Procedures for Structural Equation Models

Figures & data

Figure 1. Population and estimated eigenvalues for a ten-dimensional CFA with 34 degrees of freedom. The × represent population eigenvalues, while the boxplots represent estimated eigenvalues across 200 replications at sample size n = 1500.

Figure 2. The sorted simulated data plotted against i/d for $i=1,2,\dots ,d.$ the curve in red is the theoretical quantile function. The curve in blue is the empirical quantile function. The dotted black values are the levels of the observations.

Figure 3. Estimated eigenvalues and associated weights for EBA and regression procedures. EBA2 = 2-block EBA, pEBA2 = penalized 2-block EBA, pOLS = penalized regression, SB = Satorra–Bentler.

Table 1. Type I Error rates, normal data.

p = 10					p = 20					p = 40
	n					n					n
Test	400	800	1500	3000	Test	400	800	1500	3000	Test	400	800	1500	3000
$\text{ML}$	4.8	5.7	4.8	5.2	$\text{RLS}$	5.4	4.9	5.4	4.7	${\text{SB}}_{\text{RLS}}^{\text{UG}}$	4.8	4.5	4.8	4.5
$\text{SB}$	4.7	5.8	4.8	5.2	${\text{SB}}_{\text{RLS}}^{\text{UG}}$	4.8	4.7	5.4	4.6	$\text{RLS}$	6.2	5.3	5.0	4.5
$\text{RLS}$	4.4	5.6	4.6	4.9	${\text{pEBA}2}_{\text{RLS}}$	4.8	4.7	5.4	4.6	${\text{pEBA}2}_{\text{RLS}}$	4.5	4.0	4.5	4.3
${\text{SB}}^{\text{UG}}$	4.4	5.7	4.8	5.1	${\text{SB}}_{\text{RLS}}$	5.6	5.1	5.5	4.7	${\text{SB}}_{\text{RLS}}$	6.8	5.5	5.2	4.6
$\text{pEBA}2$	4.4	5.7	4.8	5.2	${\text{pOLS}}_{\text{RLS}}$	4.5	4.5	5.3	4.5	${\text{pEBA}2}_{\text{UG}}$	5.4	3.7	4.2	3.8
$\text{pEBA}4$	4.4	5.7	4.8	5.2	${\text{pEBA}4}_{\text{RLS}}$	4.5	4.5	5.4	4.5	${\text{pOLS}}_{\text{RLS}}$	3.4	3.3	4.2	4.0
$\text{pEBA}6$	4.4	5.7	4.8	5.2	${\text{pEBA}6}_{\text{RLS}}$	4.4	4.5	5.3	4.5	${\text{pEBA}4}_{\text{RLS}}$	3.3	3.3	4.3	4.0
$\text{pOLS}$	4.4	5.7	4.8	5.2	$\text{EBA}2$	5.0	4.1	5.3	4.6	${\text{pEBA}2}_{\text{RLS}}^{\text{UG}}$	3.3	3.2	4.2	4.0
${\text{pEBA}4}^{\text{UG}}$	4.2	5.6	4.7	5.1	${\text{pEBA}4}^{\text{UG}}$	5.9	4.7	5.6	4.8	$\text{EBA}2$	7.5	4.4	4.4	4.0
${\text{pEBA}6}^{\text{UG}}$	4.2	5.6	4.7	5.1	${\text{pEBA}6}^{\text{UG}}$	5.9	4.7	5.6	4.8	${\text{pEBA}6}_{\text{RLS}}$	3.1	3.2	4.2	4.0
${\text{pOLS}}^{\text{UG}}$	4.2	5.6	4.7	5.1	${\text{pOLS}}^{\text{UG}}$	5.9	4.7	5.6	4.8	$\text{EBA}4$	3.1	2.5	3.6	3.6
${\text{pEBA}2}^{\text{UG}}$	4.2	5.6	4.7	5.1	${\text{pEBA}2}_{\text{RLS}}^{\text{UG}}$	4.1	4.3	5.3	4.5	${\text{pOLS}}_{\text{RLS}}^{\text{UG}}$	2.5	2.7	3.7	3.7
${\text{SB}}_{\text{RLS}}$	4.4	5.9	4.6	5.1	${\text{pEBA}2}^{\text{UG}}$	6.1	5.0	5.7	4.8	${\text{pEBA}4}_{\text{RLS}}^{\text{UG}}$	2.4	2.6	3.8	3.7
$\text{EBA}2$	4.1	5.4	4.5	5.1	${\text{pEBA}2}_{\text{UG}}$	4.2	3.8	5.2	4.5	${\text{pEBA}6}_{\text{RLS}}^{\text{UG}}$	2.1	2.4	3.7	3.7
${\text{pEBA}2}_{\text{RLS}}$	4.1	5.7	4.6	5.0	${\text{pEBA}4}_{\text{RLS}}^{\text{UG}}$	3.8	4.1	5.2	4.5	$\text{EBA}6$	2.4	2.3	3.3	3.5
${\text{pEBA}4}_{\text{RLS}}$	4.1	5.7	4.6	4.9	${\text{pOLS}}_{\text{RLS}}^{\text{UG}}$	3.8	4.0	5.2	4.4	${\text{EBA}4}^{\text{UG}}$	2.2	2.2	3.2	3.4
${\text{pEBA}6}_{\text{RLS}}$	4.0	5.7	4.6	4.9	${\text{pEBA}6}_{\text{RLS}}^{\text{UG}}$	3.8	4.0	5.2	4.4	$\text{SS}$	2.0	2.2	3.3	3.5
${\text{SB}}_{\text{RLS}}^{\text{UG}}$	4.0	5.7	4.6	4.9	$\text{pEBA}6$	6.6	5.3	5.7	4.8	${\text{pEBA}6}^{\text{UG}}$	9.6	6.4	5.3	4.7
${\text{pOLS}}_{\text{RLS}}$	4.0	5.7	4.6	4.9	$\text{EBA}4$	4.0	3.6	5.2	4.5	${\text{EBA}6}^{\text{UG}}$	1.8	2.1	3.0	3.2
${\text{pEBA}2}_{\text{RLS}}^{\text{UG}}$	3.8	5.5	4.6	4.9	$\text{pOLS}$	6.7	5.3	5.7	4.8	${\text{SS}}^{\text{UG}}$	1.5	2.0	2.8	3.2
$\text{EBA}4$	3.8	5.4	4.4	5.0	$\text{pEBA}4$	6.7	5.3	5.8	4.8	${\text{EBA}2}_{\text{RLS}}$	1.4	1.6	2.8	3.1
$\text{SS}$	3.8	5.4	4.4	5.0	$\text{SS}$	3.8	3.6	5.2	4.5	${\text{pEBA}4}^{\text{UG}}$	10.4	6.8	5.5	4.7
${\text{pEBA}2}_{\text{UG}}$	3.8	5.4	4.3	5.0	$\text{EBA}6$	3.8	3.5	5.1	4.5	${\text{pOLS}}^{\text{UG}}$	10.9	6.9	5.5	4.7
$\text{EBA}6$	3.8	5.4	4.3	5.0	${\text{EBA}4}^{\text{UG}}$	3.6	3.3	5.0	4.4	${\text{EBA}2}_{\text{RLS}}^{\text{UG}}$	1.0	1.3	2.6	3.0
${\text{pEBA}4}_{\text{RLS}}^{\text{UG}}$	3.7	5.5	4.5	4.8	$\text{pEBA}2$	7.1	5.4	5.8	4.9	${\text{EBA}4}_{\text{RLS}}$	0.4	1.1	2.3	2.9
${\text{pEBA}6}_{\text{RLS}}^{\text{UG}}$	3.7	5.5	4.5	4.8	${\text{SB}}^{\text{UG}}$	7.2	5.4	5.8	4.9	${\text{EBA}6}_{\text{RLS}}$	0.3	0.9	2.2	2.9
${\text{pOLS}}_{\text{RLS}}^{\text{UG}}$	3.7	5.5	4.5	4.8	${\text{SS}}^{\text{UG}}$	3.5	3.3	5.0	4.4	${\text{SS}}_{\text{RLS}}$	0.2	0.8	2.2	2.9
${\text{EBA}2}_{\text{RLS}}$	3.6	5.4	4.5	4.8	${\text{EBA}6}^{\text{UG}}$	3.5	3.3	5.0	4.4	${\text{EBA}4}_{\text{RLS}}^{\text{UG}}$	0.2	0.8	2.1	2.8
${\text{SS}}^{\text{UG}}$	3.6	5.3	4.3	4.9	${\text{EBA}2}_{\text{RLS}}$	3.2	3.3	4.9	4.3	${\text{EBA}6}_{\text{RLS}}^{\text{UG}}$	0.2	0.7	1.9	2.7
${\text{EBA}4}_{\text{RLS}}$	3.5	5.3	4.5	4.7	${\text{EBA}2}_{\text{RLS}}^{\text{UG}}$	2.8	3.1	4.7	4.2	${\text{SS}}_{\text{RLS}}^{\text{UG}}$	0.2	0.7	1.9	2.7
${\text{SS}}_{\text{RLS}}$	3.5	5.3	4.5	4.7	$\text{ML}$	7.8	5.9	6.0	5.0	$\text{pEBA}6$	12.4	7.6	5.9	4.8
${\text{EBA}4}^{\text{UG}}$	3.6	5.3	4.3	4.9	${\text{EBA}4}_{\text{RLS}}$	2.4	3.0	4.7	4.2	${\text{pEBA}2}^{\text{UG}}$	13.0	7.6	5.8	4.8
${\text{EBA}2}_{\text{RLS}}^{\text{UG}}$	3.5	5.3	4.4	4.7	$\text{SB}$	8.0	6.0	6.1	5.0	$\text{pEBA}4$	13.2	7.8	6.0	4.8
${\text{EBA}6}_{\text{RLS}}$	3.5	5.3	4.4	4.7	${\text{SS}}_{\text{RLS}}$	2.3	3.0	4.7	4.2	$\text{pOLS}$	14.0	7.8	5.9	4.8
${\text{EBA}6}^{\text{UG}}$	3.5	5.3	4.3	4.9	${\text{EBA}6}_{\text{RLS}}$	2.3	3.0	4.7	4.1	$\text{pEBA}2$	16.4	8.7	6.4	4.8
${\text{SS}}_{\text{RLS}}^{\text{UG}}$	3.3	5.2	4.3	4.7	${\text{EBA}4}_{\text{RLS}}^{\text{UG}}$	2.0	2.8	4.4	4.1	${\text{SB}}^{\text{UG}}$	17.5	9.8	7.0	5.1
${\text{EBA}4}_{\text{RLS}}^{\text{UG}}$	3.3	5.1	4.3	4.6	${\text{SS}}_{\text{RLS}}^{\text{UG}}$	1.8	2.8	4.4	4.1	$\text{ML}$	20.2	10.5	7.4	5.3
${\text{EBA}6}_{\text{RLS}}^{\text{UG}}$	3.3	5.1	4.3	4.6	${\text{EBA}6}_{\text{RLS}}^{\text{UG}}$	1.8	2.8	4.4	4.1	$\text{SB}$	21.6	11.1	7.4	5.3

Within each dimension, the tests are sorted according to increasing RMSE.

Table 2. Test performance across 6 non-normal distributions and 4 sample sizes, ranked in increasing RMSE order.

p = 10				p = 20				p = 40
Test	RMSE	MAD	ARR	Test	RMSE	MAD	ARR	Test	RMSE	MAD	ARR
${\text{pOLS}}^{\text{UG}}$	0.63	0.54	100.0	${\text{pOLS}}_{\text{RLS}}$	1.25	0.93	87.5	${\text{pEBA}4}_{\text{RLS}}$	1.44	1.31	95.8
${\text{pOLS}}_{\text{RLS}}$	0.64	0.53	100.0	${\text{pEBA}4}_{\text{RLS}}$	1.25	0.96	91.7	${\text{pEBA}4}_{\text{RLS}}^{\text{UG}}$	1.66	1.46	83.3
$\text{pOLS}$	0.64	0.55	100.0	${\text{pOLS}}_{\text{RLS}}^{\text{UG}}$	1.30	1.06	91.7	${\text{pOLS}}_{\text{RLS}}$	1.71	1.52	87.5
${\text{pEBA}2}_{\text{RLS}}^{\text{UG}}$	0.65	0.54	100.0	${\text{pEBA}6}_{\text{RLS}}$	1.30	1.11	95.8	${\text{pEBA}6}_{\text{RLS}}$	1.76	1.55	87.5
$\text{pEBA}6$	0.65	0.56	100.0	${\text{pEBA}4}_{\text{RLS}}^{\text{UG}}$	1.30	1.08	91.7	${\text{pEBA}2}_{\text{RLS}}^{\text{UG}}$	1.80	1.29	83.3
${\text{pEBA}4}^{\text{UG}}$	0.65	0.56	100.0	${\text{pEBA}6}_{\text{RLS}}^{\text{UG}}$	1.45	1.25	95.8	${\text{pOLS}}_{\text{RLS}}^{\text{UG}}$	2.14	1.94	70.8
${\text{pEBA}4}_{\text{RLS}}$	0.66	0.54	100.0	${\text{pEBA}2}_{\text{RLS}}^{\text{UG}}$	1.46	1.02	87.5	${\text{pEBA}6}_{\text{RLS}}^{\text{UG}}$	2.16	2.00	70.8
$\text{pEBA}4$	0.66	0.57	100.0	${\text{pEBA}4}^{\text{UG}}$	1.61	1.31	87.5	${\text{EBA}2}_{\text{RLS}}$	2.34	2.13	41.7
${\text{pEBA}2}_{\text{RLS}}$	0.67	0.57	100.0	${\text{pOLS}}^{\text{UG}}$	1.63	1.30	87.5	${\text{pEBA}6}^{\text{UG}}$	2.58	2.17	54.2
${\text{pEBA}2}^{\text{UG}}$	0.68	0.58	100.0	${\text{pEBA}6}^{\text{UG}}$	1.64	1.42	83.3	${\text{EBA}2}_{\text{RLS}}^{\text{UG}}$	2.62	2.41	37.5
${\text{pEBA}6}_{\text{RLS}}$	0.69	0.57	100.0	$\text{pEBA}6$	1.66	1.36	87.5	${\text{pOLS}}^{\text{UG}}$	2.62	2.20	54.2
${\text{pEBA}6}^{\text{UG}}$	0.70	0.57	100.0	${\text{pEBA}2}_{\text{RLS}}$	1.72	1.14	79.2	${\text{pEBA}2}_{\text{UG}}$	2.63	2.37	45.8
${\text{pOLS}}_{\text{RLS}}^{\text{UG}}$	0.72	0.60	100.0	$\text{pEBA}4$	1.77	1.27	87.5	$\text{EBA}2$	2.76	2.32	62.5
${\text{pEBA}4}_{\text{RLS}}^{\text{UG}}$	0.73	0.61	100.0	$\text{pOLS}$	1.78	1.25	83.3	${\text{pEBA}4}^{\text{UG}}$	2.76	2.14	62.5
${\text{pEBA}6}_{\text{RLS}}^{\text{UG}}$	0.79	0.67	100.0	${\text{EBA}2}_{\text{RLS}}$	1.81	1.68	91.7	${\text{pEBA}2}_{\text{RLS}}$	2.77	1.70	75.0
$\text{pEBA}2$	0.80	0.66	100.0	$\text{EBA}2$	1.81	1.64	87.5	$\text{pEBA}6$	2.95	2.32	58.3
${\text{SB}}_{\text{RLS}}^{\text{UG}}$	0.89	0.72	100.0	${\text{pEBA}2}^{\text{UG}}$	1.97	1.27	83.3	$\text{pOLS}$	3.04	2.33	54.2
${\text{SB}}_{\text{RLS}}$	1.05	0.87	100.0	${\text{pEBA}2}_{\text{UG}}$	1.97	1.80	87.5	${\text{SB}}_{\text{RLS}}^{\text{UG}}$	3.33	2.49	58.3
${\text{SB}}^{\text{UG}}$	1.07	0.86	100.0	${\text{EBA}2}_{\text{RLS}}^{\text{UG}}$	2.02	1.85	75.0	$\text{pEBA}4$	3.55	2.44	58.3
$\text{EBA}2$	1.17	0.93	100.0	${\text{SB}}_{\text{RLS}}^{\text{UG}}$	2.30	1.70	70.8	$\text{EBA}4$	3.75	3.56	20.8
$\text{SB}$	1.30	1.06	100.0	$\text{pEBA}2$	2.31	1.58	75.0	${\text{EBA}4}^{\text{UG}}$	3.92	3.76	16.7
${\text{EBA}2}_{\text{RLS}}$	1.31	1.06	95.8	${\text{SB}}_{\text{RLS}}$	2.84	2.20	62.5	${\text{EBA}4}_{\text{RLS}}$	4.02	3.94	8.3
${\text{pEBA}2}_{\text{UG}}$	1.33	1.08	95.8	${\text{SB}}^{\text{UG}}$	2.92	2.16	66.7	${\text{EBA}4}_{\text{RLS}}^{\text{UG}}$	4.14	4.07	4.2
${\text{EBA}2}_{\text{RLS}}^{\text{UG}}$	1.47	1.21	95.8	$\text{EBA}4$	2.96	2.71	41.7	$\text{EBA}6$	4.16	4.05	12.5
$\text{EBA}4$	1.93	1.69	75.0	${\text{EBA}4}_{\text{RLS}}$	3.08	2.88	37.5	${\text{EBA}6}^{\text{UG}}$	4.26	4.17	4.2
${\text{EBA}4}_{\text{RLS}}$	1.98	1.75	75.0	${\text{EBA}4}^{\text{UG}}$	3.09	2.85	37.5	${\text{pEBA}2}^{\text{UG}}$	4.27	2.78	66.7
${\text{EBA}4}^{\text{UG}}$	2.06	1.82	75.0	${\text{EBA}4}_{\text{RLS}}^{\text{UG}}$	3.21	3.01	25.0	${\text{EBA}6}_{\text{RLS}}$	4.35	4.29	4.2
${\text{EBA}4}_{\text{RLS}}^{\text{UG}}$	2.10	1.88	66.7	$\text{EBA}6$	3.30	3.05	37.5	${\text{EBA}6}_{\text{RLS}}^{\text{UG}}$	4.42	4.37	4.2
$\text{EBA}6$	2.22	1.97	70.8	${\text{EBA}6}^{\text{UG}}$	3.42	3.18	29.2	$\text{SS}$	4.62	4.59	0.0
${\text{EBA}6}_{\text{RLS}}$	2.25	2.02	66.7	${\text{EBA}6}_{\text{RLS}}$	3.43	3.23	25.0	${\text{SS}}^{\text{UG}}$	4.65	4.62	0.0
$\text{SS}$	2.29	2.03	70.8	$\text{SB}$	3.50	2.69	54.2	${\text{SS}}_{\text{RLS}}$	4.72	4.70	0.0
${\text{EBA}6}^{\text{UG}}$	2.33	2.09	66.7	${\text{EBA}6}_{\text{RLS}}^{\text{UG}}$	3.54	3.33	25.0	${\text{SS}}_{\text{RLS}}^{\text{UG}}$	4.74	4.72	0.0
${\text{EBA}6}_{\text{RLS}}^{\text{UG}}$	2.34	2.11	66.7	$\text{SS}$	3.78	3.61	20.8	${\text{SB}}_{\text{RLS}}$	4.76	3.66	45.8
${\text{SS}}_{\text{RLS}}$	2.37	2.13	66.7	${\text{SS}}^{\text{UG}}$	3.85	3.69	20.8	$\text{pEBA}2$	5.82	3.79	58.3
${\text{SS}}^{\text{UG}}$	2.41	2.16	66.7	${\text{SS}}_{\text{RLS}}$	3.88	3.74	20.8	${\text{SB}}^{\text{UG}}$	6.48	4.53	45.8
${\text{SS}}_{\text{RLS}}^{\text{UG}}$	2.45	2.21	62.5	${\text{SS}}_{\text{RLS}}^{\text{UG}}$	3.95	3.81	16.7	$\text{SB}$	8.50	6.08	41.7
$\text{RLS}$	48.84	39.58	0.0	$\text{RLS}$	76.73	69.55	0.0	$\text{RLS}$	79.48	72.15	0.0
$\text{ML}$	49.24	40.10	0.0	$\text{ML}$	77.26	70.48	0.0	$\text{ML}$	80.85	75.33	0.0

RMSE: root mean square error in percentage; MAD: mean absolute deviation of rejection rates from 5%; ARR: percentage of acceptable rejection rates.

Table 3. Top ten robustified tests according to RMSE when aggregating 6 non-normal distributions, 4 sample sizes and 3 model sizes.

Test	RMSE	MAD	ARR
${\text{pEBA}4}_{\text{RLS}}$	1.16	0.94	95.8
${\text{pOLS}}_{\text{RLS}}$	1.28	0.99	91.7
${\text{pEBA}4}_{\text{RLS}}^{\text{UG}}$	1.29	1.05	91.7
${\text{pEBA}6}_{\text{RLS}}$	1.32	1.07	94.4
${\text{pEBA}2}_{\text{RLS}}^{\text{UG}}$	1.39	0.95	90.3
${\text{pOLS}}_{\text{RLS}}^{\text{UG}}$	1.50	1.20	87.5
${\text{pEBA}6}_{\text{RLS}}^{\text{UG}}$	1.57	1.31	88.9
${\text{pEBA}6}^{\text{UG}}$	1.81	1.39	79.2
${\text{pOLS}}^{\text{UG}}$	1.82	1.35	80.6
${\text{EBA}2}_{\text{RLS}}$	1.86	1.62	76.4

RMSE: root mean square error in percentage; MAD: mean absolute deviation of rejection rates from 5%; ARR: percentage of acceptable rejection rates.

Figure 4. Rejection rates in % for six selected tests. Panel columns and rows correspond to model size and distribution, respectively.

Table 4. Rejection rate in % for the bollen–stine bootstrap.

n	Distribution	p = 10	p = 20	p = 40
800	Normal	4.0	2.5	1.1
	IG1	4.7	3.4	1.1
	IG2	5.6	5.0	1.4
	PL1	4.3	5.8	69.1
	PL2	10.2	93.1	100.0
	VM1	4.7	3.8	12.2
	VM2	6.6	55.2	99.9
3000	Normal	4.5	4.3	2.4
	IG1	4.7	3.8	2.0
	IG2	4.5	5.1	3.4
	PL1	3.7	4.6	64.1
	PL2	10.7	95.5	100.0
	VM1	4.7	4.5	7.6
	VM2	5.4	56.8	100.0

Table 5. Rejection rate in % for the $\text{EBAd}$ test procedure, aggregated over all seven distributions.

	n
	400	800	1500	3000	10^4	10^5
p = 10	2.0	2.9	3.0	4.0	4.8	4.9
p = 20	0.8	1.1	1.9	2.6	3.8	4.7
p = 40	0.3	0.4	0.8	1.2	2.6	4.8