Discussion of Professor Bradley Efron’s Article on “Prediction, Estimation, and Attribution”

Figures & data

Fig. 1 Diagrams of four neural network models: (a) true ${\mu }_0(·)$; (b) partial ${\mu }_1(·)$; and (c, d) over-parameterized ${\mu }_2(·)$ and ${\mu }_3(·)$ of (20 nodes in each layer) × L layers, with L = 20 and 100, respectively.

Table 1 Mean square error of each parameter in μ₀ (training data n = 300; repetition = 10).

MSE	a11	a12	a13	a21	a22	a23	b1	b2
Opt-MSE	0.07	0.059	0.31	0.154	0.109	0.124	0.06	0.101
Neuralnet	4.87	5.9	1.53	1.14	2.23	2.08	4.87	0.84

Table 2 Performance of 95% predictive intervals under five different learning models and in two scenarios: coverage rates (before brackets) and average interval lengths (inside brackets) (training data size = 300; testing data size = 20; repetition = 10).

	True model		Wrong model
	${\mu }_0(·)$		${\mu }_1(·)$	${\mu }_2(·)$	${\mu }_3(·)$	${\mu }_4(·)$
	Opt-MSE	Neuralnet	Nueralnet	Nueralnet	Nueralnet	Nueralnet
IID scenario	0.995 (4.462)	0.99 (4.608)	0.99 (4.809)	0.99 (5.212)	0.99 (5.201)	0.985 (5.26)
Non-IID scenario	0.955 (4.52)	0.985 (9.327)	0.98 (9.77)	0.71 (5.899)	0.695 (5.201)	0.685 (5.277)

Fig. 2 Plots of predictive curves for (a) ${\mathbf{x}}_{\mathrm{ }\text{new}}\stackrel{\mathrm{ }\text{iid}}{\sim }{\mathbf{x}}_i$ and (b) ${\mathbf{x}}_{\mathrm{ }\text{new}}\cancel{\sim }{\mathbf{x}}_i$. In each plot, the red solid curve is the target (oracle) predictive curve ${\mathrm{ }\text{PV}}_n(y)=2\max \{\Phi (y-{\mu }_{\mathrm{ }\text{new}}),1-\Phi (y-{\mu }_{\mathrm{ }\text{new}})\}$, obtained assuming that the distribution of $y_{\mathrm{ }\text{new}}\sim N({\mu }_{\mathrm{ }\text{new}},1)$ is completely known. The two predictive curves obtained using ${\mu }_0(·)$ are in black (solid line for Opt-MSE; dashed line for Neuralnet). The other predictive curves (all in a dashed or broken line and in various colors) are obtained using the other four wrong working models.