CMARS: a new contribution to nonparametric regression with multivariate adaptive regression splines supported by continuous optimization

Figures & data

Figure 1. The solutions obtained for different Z values for problem (24).

Figure 2. Norm of Lθ versus for the solutions of methods (*: MARS solution; o: CMARS solution).

Figure 3. GCV values for the solutions of methods (*: MARS solution; o: CMARS solution).

Table 1. Data sets used in the comparison.

Data Sets			Features
Number	Name	Source	n	p	Size	Scale
1	Concrete Slump Test	[25]	103	7	Small	Small
2	Uniform Sampling	[27]	160	10	Small	Medium
3	Metal Casting	[28]	92	35	Small	Large
4	PM10	[26]	500	7	Medium	Small
5	Forest Fires	[25]	517	11	Medium	Medium
6	Parkinsons Telemonitoring	[25]	578	21	Medium	Large
7	Concrete Compressive Strength	[25]	1030	8	Large	Small
8	Red Wine Quality	[25]	1599	11	Large	Medium
9	Communities and Crime	[25]	879	24	Large	Large

Table 2. Averages of performance results of the models built for the data sets.

		Training		Test		Stability
Data set number	Performance Measures	MARS	CMARS	MARS	CMARS	MARS	CMARS
1	MAE	0.4852	0.4518	0.6549	0.6946	0.7399	0.6692
	MSE	0.3814	0.3520	0.6584	0.8507	0.5486	0.4734
	R^2	0.6254	0.6757	0.4121	0.3962	0.6259	0.5594
	PWI	0.9984	0.9968	1.0000	0.9970	0.9984	0.9938
	PRESS	27.9504	97.7900	27.5762	35.6460	0.6944	0.5748
2	MAE	0.0004	0.2029	0.0005	0.2093	0.7568	0.9433
	MSE	0.0000	0.0592	0.0000	0.0625	0.5113	0.9150
	R^2	1.0000	0.9982	1.0000	0.9984	1.0000	0.9997
	PWI	0.9885	1.0000	0.9852	1.0000	0.9883	1.0000
	PRESS	0.0000	6.7330	0.0002	3.9259	0.5512	0.5891
3	MAE	0.2444	0.2145	0.8039	0.7362	0.3352	0.3156
	MSE	0.1780	0.2097	4.6815	7.3187	0.0991	0.0568
	R^2	0.8558	0.8880	0.1598	0.1921	0.1907	0.2194
	PWI	0.9819	0.9819	0.9748	0.9711	0.9710	0.9743
	PRESS	10.9591	10.1713	672.5287	685.3657	0.1388	0.1267
4	MAE	0.4984	0.4888	0.6878	0.6881	0.7295	0.6616
	MSE	0.4275	0.4590	0.8773	0.9519	0.5313	0.3989
	R^2	0.5910	0.6026	0.2936	0.2923	0.5027	0.3933
	PWI	0.9964	0.9934	0.9902	0.9898	0.9927	0.9910
	PRESS	142.4560	124.8675	156.0576	172.4247	0.8128	0.7461
5	MAE	0.2106	0.2083	1.5833	1.3309	0.2566	0.2176
	MSE	0.2283	0.2797	155.7983	108.1064	0.0746	0.0781
	R^2	0.7436	0.7266	0.0115	0.0118	0.0147	0.0163
	PWI	0.9819	0.9855	0.9818	0.9851	0.9866	0.9925
	PRESS	79.1786	87.7969	30595.9500	19218.3500	0.0494	0.1199
6	MAE	0.2550	0.2421	0.3839	0.3715	0.6732	0.6560
	MSE	0.1183	0.1097	0.5961	0.3970	0.3057	0.3063
	R^2	0.8848	0.8973	0.6250	0.7109	0.7069	0.7928
	PWI	0.9899	0.9904	0.9871	0.9851	0.9924	0.9935
	PRESS	46.7619	41.8522	17092.2200	5279.0400	0.3535	0.2721
7	MAE	0.1628	0.1591	0.2437	0.2434	0.5979	0.5897
	MSE	0.0503	0.0511	0.2872	0.2891	0.2884	0.2849
	R^2	0.9525	0.9534	0.7930	0.7933	0.7439	0.7462
	PWI	0.9906	0.9898	0.9888	0.9883	0.8952	0.8943
	PRESS	33.3882	32.8392	100.8555	97.7900	0.5579	0.5464
8	MAE	0.5290	0.5081	0.6745	0.6849	0.7885	0.7467
	MSE	0.4694	0.4493	0.9931	1.1457	0.5079	0.4287
	R^2	0.5403	0.5706	0.2360	0.2059	0.4515	0.3778
	PWI	0.9929	0.9923	0.9891	0.9885	0.9962	0.9960
	PRESS	498.1073	465.5898	550.9740	629.8354	0.7948	0.6794
9	MAE	0.1968	0.1543	0.6566	0.6952	0.3053	0.2231
	MSE	0.0770	0.0556	2.1762	2.2735	0.0585	0.0275
	R^2	0.9294	0.9542	0.3244	0.2784	0.3501	0.2918
	PWI	0.9875	0.9868	0.9796	0.9818	0.9903	0.9937
	PRESS	44.2664	28.5964	766.1493	727.2282	0.0956	0.0459

Table 3. Overall performances (mean ± standard deviation) of MARS and CMARS methods.

	Training		Test		Stability
	MARS	CMARS	MARS	CMARS	MARS	CMARS
MAE	0.288* ±	0.292 ±	0.632 ±	0.628* ±	0.576 ±	0.559* ±
	0.174	0.143^**	0.427	0.345^**	0.222^**	0.251
						(p-value = 0.094)
MSE	0.215*±	0.225 ±	18.452±	13.488*±	0.325*±	0.330 ±
	0.175^**	0.186	52.056	42.145^**	0.215^**	0.274
				(p-value = 0.042)
R^2	0.790 ±	0.807* ±	0.428 ±	0.431* ±	0.509* ±	0.488 ±
	0.169	0.168^**	0.310^**	0.317	0.303^**	0.309
		(p-value = 0.000)
PWI	0.990 ±	0.991* ±	0.986 ±	0.987* ±	0.979 ±	0.981* ±
	0.006	0.006^**	0.009	0.009	0.056	0.056
						(p-value = 0.001)
PRESS	99.44 ±	91.51* ±	5551 ±	2983* ±	0.450 ±	0.410* ±
	155.10	142.90^**	11252	7578^**	0.304	0.272^**
		(p-value = 0.004)		(p-value = 0.001)		(p-value = 0.042)
Notes: * Indicates better performance with respect to mean; p-value of LSD test is recorded within the parentheses when there is a statistically significant difference in means.
^** Indicates better performance with respect to standard deviation.

Table 4. Average of accuracy measures with respect to different sample sizes.

		Training		Test		Stability
Measures	Size	MARS	CMARS	MARS	CMARS	MARS	CMARS
MAE	Small	0.243*	0.290	0.486*	0.547	0.611	0.643*
	Medium	0.321	0.313*	0.885	0.797*	0.553*	0.512
	Large	0.296	0.274*	0.525*	0.541	0.564*	0.520
MSE	Small	0.186*	0.207	1.780*	2.744	0.386	0.482*
	Medium	0.258*	0.283	52.424	36.485*	0.304*	0.261
	Large	0.199	0.185*	1.152	1.236*	0.285*	0.247
R^2	Small	0.827	0.854*	0.524	0.529*	0.606*	0.593
	Medium	0.740	0.742*	0.310	0.338*	0.408*	0.401
	Large	0.807	0.826*	0.451*	0.426	0.515*	0.472
PWI	Small	0.990	0.993*	0.987	0.989*	0.986	0.989*
	Medium	0.989	0.990*	0.986	0.987*	0.991	0.992*
	Large	0.990	0.990	0.986	0.986	0.961	0.961
PRESS	Small	12.970*	38.231	233.368*	241.646	0.461*	0.430
	Medium	89.466	84.839*	15948.076	8223.272*	0.405*	0.379
	Large	191.921	175.675*	472.660*	484.951	0.483*	0.424
Note: * Indicates better performance.

Table 5. Average of measures for different problem scales.

		Training		Test		Stability
Measures	Size	MARS	CMARS	MARS	CMARS	MARS	CMARS
MAE	Small	0.382	0.367*	0.529*	0.542	0.689*	0.640
	Medium	0.247*	0.306	0.753	0.742*	0.601	0.636*
	Large	0.232	0.204*	0.615	0.601*	0.438*	0.398
MSE	Small	0.286*	0.287	0.608*	0.697	0.456*	0.386
	Medium	0.233*	0.263	52.264	36.438*	0.365	0.474*
	Large	0.124*	0.125	2.485*	3.330	0.154*	0.130
R^2	Small	0.723	0.744*	0.500*	0.494	0.624*	0.566
	Medium	0.761	0.765*	0.416*	0.405	0.489*	0.465
	Large	0.760	0.785*	0.340	0.370*	0.450	0.463*
PWI	Small	0.995*	0.993	0.993*	0.992	0.962*	0.960
	Medium	0.987	0.992*	0.985	0.991*	0.957	0.962*
	Large	0.986	0.986	0.981*	0.979	0.985	0.987*
PRESS	Small	67.932*	85.166	94.830*	101.954	0.688*	0.622
	Medium	192.429	186.707*	10382.308	6617.370*	0.465*	0.463
	Large	33.996	26.873*	6176.966	2230.545*	0.196*	0.148
Note: * Indicates better performance.

Figure 4. Interaction plots of size and scale for the complexity measure, MSE, for MARS and CMARS.

Table 6. Classification of run times (in seconds) of the methods with respect to size and scale.

		Scale
		Small	Medium	Large
	Small	MARS: < 2 s	MARS: < 2 s	MARS: < 2 s
		CMARS: < 21 s	CMARS: < 20 s	CMARS: < 21 s
Sample size	Medium	MARS: < 4 s	MARS: < 10 s	MARS: < 5 s
		CMARS: < 46 s	CMARS: < 35 s	CMARS: < 43 s
	Large	MARS: < 20 s	MARS: < 26 s	MARS: < 13 s
		CMARS: < 89 s	CMARS: < 140 s	CMARS: < 103 s

Table 7. Performance results of the methods on noisy data.

			Measures
	Data sets	Methods	MAE	MSE	R^2	PWI	PRESS
Noise-free data	Training	MARS	0.4318	0.3780	0.9999	0.9900	31.737
		CMARS	0.3188*	0.2670*	1.0000*	0.9900	16.127*
	Test	MARS	0.5573	0.6838*	0.9999	0.9900	58.029
		CMARS	0.5496*	0.7771	0.9999	1.0000*	47.534*
Noisy data	Training	MARS	6.2843	69.4419*	0.9869	1.0000*	6588.309
		CMARS	6.2281*	69.6126	0.9870*	0.9900	6522.154*
	Test	MARS	7.6158	102.1720	0.9895	0.9900	9630.179
		CMARS	7.3908*	97.1906*	0.9896*	1.0000*	9058.165*
Noisy fit vs noise-free data	Training	MARS	3.2556	21.3454	0.9963	1.0000*	2042.576
		CMARS	3.2156*	20.1152*	0.9966*	1.0000*	1898.328*
	Test	MARS	3.3300	21.8986	0.9991	0.9800	2070.795
		CMARS	3.0013*	15.2437*	0.9993*	1.0000*	1421.167*
Note: * Indicates better performance.