A comparison of normality testing methods by empirical power and distribution of P-values

Figures & data

Table 1. Abbreviations for each normality testing method.

Testing method	Abbreviation
Lilliefors	M.KS
Cramer von Mises	CVM
Anderson-Darling	AD
Zhang-Wu ZC	ZC
Zhang-Wu ZA	ZA
Jarque-Bera	JB
Gel-Gastwirth	R.JB
Shapiro-Wilk	SW
Shapiro-Francia	SF

Table 2. Simulation conditions: alternative distributions, sample sizes and significance levels.

Distribution		Sample size	α
	Beta(2)
Symmetric	Laplace(0,1)
	t(1)
		10,20,30,100,200,300	0.01, 0.05
	Beta(1, 2)
Asymmetric	${\chi }^2$(1)
	Gamma(2)

Table 3. Empirical power: empirical power for various normality tests depending on sample size n under each alternative distribution at $\alpha =0.01.$

distribution under Ha		n	M.KS	CVM	AD	ZC	ZA	JB	R.JB	SW	SF
symmetric	Beta(2,  2)	10	0.0086	0.0068	0.0065	0.0130	${0.0097}^{*}$	${0.0006}^{*}$	${0.0126}^{*}$	${0.0060}^{*}$	${0.0042}^{*}$
		20	0.0090	0.0085	0.0091	0.0131	${0.0096}^{*}$	${0.0003}^{*}$	${0.0028}^{*}$	${0.0048}^{*}$	${0.0019}^{*}$
		30	0.0109	0.0127	0.0142	0.0200	0.0173	${0.0001}^{*}$	${0.0008}^{*}$	0.0083	0.0019
		100	0.0349	0.0706	0.1025	0.2171	0.3091	${0.0000}^{*}$	${0.0000}^{*}$	0.1318	0.0330
		200	0.1046	0.2571	0.3976	0.7209	0.8913	0.0514	0.0011	0.6540	0.3571
		300	0.2130	0.5014	0.7134	0.9421	0.9961	0.5608	0.1595	0.9528	0.8080
	Laplace(0,  1)	10	0.0504	0.0572	0.0605	0.0867	0.0906	0.0266	0.1587	0.0646	0.0695
		20	0.0892	0.1235	0.1303	0.1638	0.1610	0.1485	0.3113	0.1312	0.1588
		30	0.1271	0.1888	0.1977	0.2263	0.2126	0.2483	0.4259	0.2034	0.2443
		100	0.4605	0.6500	0.6598	0.5660	0.5257	0.6822	0.8409	0.6362	0.6896
		200	0.8179	0.9424	0.9457	0.8440	0.8263	0.9267	0.9804	0.9292	0.9434
		300	0.9564	0.9938	0.9941	0.9538	0.9500	0.9860	0.9979	0.9902	0.9924
	t(1)	10	0.4429	0.4818	0.4832	0.4918	0.5062	0.3439	0.6013	0.4752	0.4905
		20	0.7485	0.8067	0.8093	0.7853	0.7979	0.7633	0.8887	0.7923	0.8178
		30	0.8936	0.9324	0.9342	0.9132	0.9187	0.9159	0.9681	0.9256	0.9384
		100	0.9999	1.0000	1.0000	0.9999	1.0000	1.0000	1.0000	1.0000	1.0000
		200	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		300	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
asymmetric	Beta(2,  1)	10	0.0256	0.0283	0.0306	0.0522	0.0492	0.0042	${0.0419}^{*}$	0.0326	0.0241
		20	0.0532	0.0740	0.0874	0.1183	0.1498	0.0117	${0.0413}^{*}$	0.0836	0.0510
		30	0.0944	0.1490	0.1849	0.2257	0.3433	0.0134	${0.0389}^{*}$	0.1984	0.1095
		100	0.5537	0.7888	0.9049	0.9296	0.9981	0.1669	0.1147	0.9762	0.9161
		200	0.9534	0.9962	0.9999	0.9988	1.0000	0.9806	0.8595	1.0000	1.0000
		300	0.9992	1.0000	1.0000	1.0000	1.0000	1.0000	0.9998	1.0000	1.0000
	${\chi }^2(1)$	10	0.3070	0.4445	0.4723	0.5522	0.5742	0.1754	0.4341	0.5101	0.4577
		20	0.6986	0.8646	0.8987	0.9148	0.9622	0.5841	0.7562	0.9248	0.8920
		30	0.9203	0.9793	0.9894	0.9883	0.9988	0.8289	0.9030	0.9956	0.9898
		100	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		200	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		300	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
	Gamma(2,  2)	10	0.0605	0.0774	0.0860	0.1368	0.1413	0.0345	0.1530	0.1013	0.0947
		20	0.1412	0.2186	0.2530	0.3454	0.3913	0.1991	0.3161	0.2986	0.2785
		30	0.2351	0.3689	0.4310	0.5377	0.6365	0.3468	0.4491	0.5184	0.4714
		100	0.8292	0.9589	0.9843	0.9903	0.9997	0.9497	0.9417	0.9972	0.9942
		200	0.9957	1.0000	1.0000	1.0000	1.0000	1.0000	0.9999	1.0000	1.0000
		300	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000

Note: The boldface denotes the best performance for each sample size.

The asterisk$(*)$ shows that the empirical power decreases as the sample size increases up to moderate sample sizes.

Table 4. Empirical power: empirical power for various normality tests depending on sample size n under each alternative distribution at $\alpha =0.05.$

distribution under Ha		n	M.KS	CVM	AD	ZC	ZA	JB	R.JB	SW	SF
symmetric	Beta(2,  2)	10	0.0447	0.0435	0.0439	0.0538	0.0386	${0.0027}^{*}$	${0.0215}^{*}$	0.0419	${0.0328}^{*}$
		20	0.0487	0.0549	0.0566	0.0694	0.0433	${0.0011}^{*}$	${0.0064}^{*}$	0.0532	${0.0234}^{*}$
		30	0.0572	0.0713	0.0766	0.1027	0.0684	${0.0005}^{*}$	${0.0022}^{*}$	0.0774	0.0277
		100	0.1527	0.2434	0.3157	0.5802	0.5703	0.0148	${0.0002}^{*}$	0.4522	0.2170
		200	0.3369	0.5516	0.7081	0.9679	0.9783	0.6135	0.2529	0.9232	0.7532
		300	0.5340	0.7921	0.9189	0.9992	0.9998	0.9633	0.7929	0.9970	0.9739
	Laplace(0,  1)	10	0.1409	0.1564	0.1594	0.1620	0.1811	0.0631	0.2069	0.1540	0.1868
		20	0.2172	0.2653	0.2707	0.2582	0.2785	0.2163	0.3821	0.2609	0.3189
		30	0.2894	0.3633	0.3708	0.3348	0.3490	0.3353	0.5087	0.3566	0.4269
		100	0.7066	0.8227	0.8273	0.7049	0.7021	0.7791	0.8984	0.7966	0.8457
		200	0.9444	0.9829	0.9836	0.9271	0.9273	0.9628	0.9907	0.9748	0.9825
		300	0.9920	0.9988	0.9988	0.9859	0.9867	0.9948	0.9993	0.9975	0.9984
	t(1)	10	0.5756	0.6089	0.6098	0.5839	0.6144	0.4249	0.6466	0.5894	0.6317
		20	0.8425	0.8784	0.8804	0.8459	0.8649	0.8162	0.9109	0.8659	0.8927
		30	0.9434	0.9633	0.9643	0.9450	0.9527	0.9407	0.9766	0.9582	0.9686
		100	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		200	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		300	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
asymmetric	Beta(2,  1)	10	0.0984	0.1144	0.1238	0.1507	0.1326	0.0144	0.0627	0.1319	0.1111
		20	0.1709	0.2286	0.2614	0.3300	0.3331	0.0306	0.0706	0.3054	0.2162
		30	0.2622	0.3621	0.4260	0.5406	0.5935	0.0429	0.0771	0.5150	0.3720
		100	0.8211	0.9386	0.9820	0.9984	0.9999	0.7439	0.4392	0.9987	0.9922
		200	0.9962	0.9998	1.0000	1.0000	1.0000	0.9999	0.9991	1.0000	1.0000
		300	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
	${\chi }^2(1)$	10	0.5329	0.6627	0.6951	0.7468	0.7532	0.2722	0.4985	0.7329	0.7051
		20	0.8775	0.9514	0.9688	0.9837	0.9901	0.7162	0.8177	0.9836	0.9732
		30	0.9822	0.9957	0.9983	0.9996	0.9999	0.9199	0.9426	0.9996	0.9991
		100	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		200	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		300	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
	Gamma(2,  2)	10	0.1692	0.2047	0.2226	0.2599	0.2656	0.0766	0.1971	0.2407	0.2427
		20	0.3190	0.4193	0.4651	0.5449	0.5742	0.2888	0.3921	0.5313	0.5039
		30	0.4593	0.5981	0.6600	0.7559	0.7974	0.4746	0.5458	0.7492	0.7068
		100	0.9529	0.9912	0.9974	0.9998	1.0000	0.9916	0.9838	0.9998	0.9993
		200	0.9997	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000
		300	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000	1.0000

Note: The boldface denotes the best performance for each sample size.

The asterisk$(*)$ shows that the empirical power decreases as the sample size increases up to moderate sample sizes.

Figure 1. The empirical distributions of p-values under $H_0:N(0,1)$ and $H_a:N(1,1).$

Table 5. Expected P-values: The expected value of the distribution of p-values for various normality tests depending on each sample size under each alternative distribution.

distribution under Ha		n	M.KS	CVM	AD	ZC	ZA	JB	R.JB	SW	SF
symmetric	Beta(2,  2)	10	0.5004	0.4792	0.4683	0.4618	0.5041	0.6863	0.7181	0.4748	0.5201
		20	0.4855	0.4586	0.4478	0.3977	0.4757	0.6063	0.6695	0.4287	0.5059
		30	0.4588	0.4265	0.4090	0.3203	0.4040	0.5347	0.6139	0.3695	0.4745
		100	0.2977	0.2369	0.1868	0.0769	0.0947	0.2095	0.2992	0.1104	0.2058
		200	0.1613	0.0964	0.0546	0.0137	0.0077	0.0539	0.1033	0.0158	0.0425
		300	0.0894	0.0389	0.0156	0.0035	0.0005	0.0138	0.0355	0.0022	0.0078
	Laplace(0,  1)	10	0.3992	0.3799	0.3748	0.4302	0.3869	0.5906	0.5111	0.4146	0.3815
		20	0.3261	0.2986	0.2953	0.3418	0.2821	0.4401	0.3443	0.3334	0.2771
		30	0.2662	0.2341	0.2300	0.3026	0.2481	0.3424	0.2457	0.2620	0.2077
		100	0.0688	0.0429	0.0421	0.1022	0.0826	0.0737	0.0319	0.0541	0.0380
		200	0.0116	0.0039	0.0038	0.0160	0.0134	0.0092	0.0022	0.0057	0.0040
		300	0.0022	0.0004	0.0003	0.0036	0.0034	0.0012	0.0001	0.0006	0.0004
	t(1)	10	0.1619	0.1478	0.1466	0.1997	0.1603	0.2964	0.2000	0.1710	0.1455
		20	0.0487	0.0386	0.0378	0.0541	0.0397	0.0779	0.0407	0.0466	0.0340
		30	0.0147	0.0099	0.0096	0.0242	0.0155	0.0217	0.0090	0.0125	0.0085
		100	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		200	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
asymmetric	Beta(2,  1)	10	0.4103	0.3743	0.3533	0.3237	0.3592	0.6051	0.6286	0.3414	0.3854
		20	0.3182	0.2665	0.2348	0.1736	0.1946	0.4362	0.4861	0.1915	0.2481
		30	0.2408	0.1838	0.1476	0.1005	0.0986	0.3199	0.3778	0.1026	0.1521
		100	0.0325	0.0122	0.0044	0.0040	0.0002	0.0381	0.0647	0.0013	0.0036
		200	0.0019	0.0003	0.0000	0.0004	0.0000	0.0018	0.0052	0.0000	0.0000
		300	0.0001	0.0000	0.0000	0.0001	0.0000	0.0001	0.0004	0.0000	0.0000
	${\chi }^2(1)$	10	0.1279	0.0873	0.0738	0.0602	0.0631	0.2836	0.2393	0.0621	0.0736
		20	0.0244	0.0100	0.0065	0.0047	0.0024	0.0585	0.0504	0.0040	0.0061
		30	0.0041	0.0011	0.0005	0.0008	0.0001	0.0124	0.0111	0.0003	0.0006
		100	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		200	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
	Gamma(2,  2)	10	0.3513	0.3175	0.2991	0.2892	0.2977	0.5114	0.4934	0.2932	0.3117
		20	0.2315	0.1843	0.1598	0.1243	0.1238	0.2664	0.2674	0.1302	0.1487
		30	0.1534	0.1070	0.0840	0.0622	0.0548	0.1432	0.1497	0.0567	0.0705
		100	0.0096	0.0022	0.0008	0.0008	0.0000	0.0021	0.0028	0.0002	0.0003
		200	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000

Note: The boldface denotes the best performance for each sample size.

Table 6. Median P-values: The median value of the distribution of p-values for various normality tests depending on each sample size under each alternative distribution.

distribution under Ha		n	M.KS	CVM	AD	ZC	ZA	JB	R.JB	SW	SF
symmetric	Beta(2,  2)	10	0.5015	0.4682	0.4620	0.4474	0.4909	0.6903	0.7537	0.4582	0.5182
		20	0.4789	0.4363	0.4298	0.3405	0.4276	0.5965	0.6710	0.3902	0.5020
		30	0.4420	0.3903	0.3734	0.2155	0.3120	0.5167	0.6018	0.3168	0.4604
		100	0.2345	0.1600	0.1120	0.0369	0.0336	0.1812	0.2675	0.0596	0.1417
		200	0.0948	0.0392	0.0175	0.0073	0.0006	0.0395	0.0810	0.0050	0.0180
		300	0.0435	0.0099	0.0028	0.0027	0.0000	0.0086	0.0243	0.0005	0.0024
	Laplace(0,  1)	10	0.3527	0.3263	0.3275	0.4290	0.3800	0.6596	0.6383	0.3792	0.3141
		20	0.2398	0.2009	0.1964	0.2268	0.1772	0.4562	0.2190	0.2369	0.1597
		30	0.1642	0.1176	0.1134	0.1866	0.1136	0.2456	0.0432	0.1355	0.0818
		100	0.0133	0.0026	0.0022	0.0115	0.0135	0.0001	0.0000	0.0025	0.0017
		200	0.0006	0.0000	0.0000	0.0007	0.0004	0.0000	0.0000	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0001	0.0001	0.0000	0.0000	0.0000	0.0000
	t(1)	10	0.0210	0.0128	0.0127	0.0151	0.0148	0.1390	0.0000	0.0148	0.0112
		20	0.0001	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		30	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		100	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		200	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
asymmetric	Beta(2,  1)	10	0.3751	0.3235	0.3022	0.2314	0.2664	0.6274	0.7059	0.2795	0.3289
		20	0.2487	0.1856	0.1515	0.1024	0.1174	0.4430	0.5334	0.1132	0.1694
		30	0.1585	0.0983	0.0714	0.0530	0.0457	0.3135	0.4026	0.0470	0.0843
		100	0.0072	0.0012	0.0003	0.0029	0.0000	0.0288	0.0569	0.0002	0.0008
		200	0.0001	0.0000	0.0000	0.0004	0.0000	0.0010	0.0035	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0002	0.0000	0.0000
	${\chi }^2(1)$	10	0.0408	0.0154	0.0124	0.0175	0.0138	0.2621	0.0515	0.0092	0.0131
		20	0.0021	0.0002	0.0001	0.0005	0.0000	0.0024	0.0000	0.0002	0.0004
		30	0.0001	0.0000	0.0000	0.0002	0.0000	0.0000	0.0000	0.0000	0.0000
		100	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		200	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
	Gamma(2,  2)	10	0.2854	0.2404	0.2203	0.2315	0.2414	0.5750	0.6350	0.2064	0.2147
		20	0.1340	0.0809	0.0618	0.0453	0.0370	0.2389	0.1746	0.0415	0.0489
		30	0.0624	0.0263	0.0170	0.0121	0.0050	0.0626	0.0255	0.0087	0.0122
		100	0.0005	0.0000	0.0000	0.0002	0.0000	0.0000	0.0000	0.0000	0.0000
		200	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000
		300	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000	0.0000

Note: The boldface denotes the best performance for each sample size.

Figure 2. The distributional features of p-values when using Lilliefors’ modified Kolmogorov-Smirnov test statistic with varying sample size under the asymmetric alternative distributions. The dotted line and solid line are the expected value and the median value of the p-values, respectively.

Figure 3. The distributional features of p-values when using Cramer von-Mises and Anderson-Darling test statistics with varying sample size under the asymmetric alternative distributions. The dotted line and solid line are the expected value and the median value of the p-values, respectively.

Figure 4. The distributional features of p-values when using Zhang-Wu test statistics with varying sample size under the asymmetric alternative distributions. The dotted line and solid line are the expected value and the median value of the p-values, respectively.

Figure 5. The distributional features of p-values when using Jarque-Bera and Gel-Gastwirth (R.JB) test statistics with varying sample size under the asymmetric alternative distributions. The dotted line and solid line are the expected value and the median value of the p-values, respectively.

Figure 6. The distributional features of p-values when using Shapiro-Wilk and Shapiro-Francia test statistics with varying sample size under the asymmetric alternative distributions. The dotted line and solid line are the expected value and the median value of the p-values, respectively.