Frequentist, Bayes, or Other?

Figures & data

Table 1. Estimated coefficients and test error results, for different subset and shrinkage methods applied to the prostate data.

Term	LS	Best Subset	Ridge	Lasso	PCR	PLS
Intercept	2.465	2.477	2.452	2.468	2.497	2.452
lcavol	0.680	0.740	0.420	0.533	0.543	0.419
lweight	0.263	0.316	0.238	0.169	0.289	0.344
age	−0.141		−0.046		−0.152	−0.026
lbph	0.210		0.162	0.002	0.214	0.220
svi	0.305		0.227	0.094	0.315	0.243
lcp	−0.288		0.000		−0.051	0.079
gleason	−0.021		0.040		0.232	0.011
pgg45	0.267		0.133		−0.056	0.084
Test Error	0.521	0.492	0.492	0.479	0.449	0.528
Std Error	0.179	0.143	0.165	0.164	0.105	0.152

The blank entries correspond to variables omitted. Table and caption from Hastie, Tibshirani, and Friedman (2009).

Figure 1. Density and rug plots of the profile likelihood for the 256 models of the prostate cancer data in Example 1.

Figure 2. Profile log likelihoods of the 80 models in Example 1 that include at least three of lcavol, lweight, lbph, and svi. The base model indicates which other predictors are included. The five lines are for models that exclude either one or none of lcavol, lweight, lbph, and svi.

Figure 3. $log\left(R_s\right)\text{versus}{T}_s$ in the Harvard Forest. Over 100,000 data points.

Table 2. Models and measures of fit as calculated by Wang et al. (2012) in Example 2.

		Model structures
Type	Model no.	Fixed effects	Random effects	AIC	BIC
Logistic	1	MI × SUM06		7086	7112
	2	MI × SUM06 + N100 + PDSI		7069	7109
GAM	3	s(MI) + s(SUM06)		6946	7073
	4	s(MI) + s(SUM06, k = 5)		6950	7043
	5	s(MI) + s(SUM06, k = 5) + s(N100) + s(PDSI)		6884	7098
	6	s(MI, by = REG) + s(SUM06, k = 5, by = REG)		6936	7116
Mixed	7	s(MI) + s(SUM06, k = 5)	(1|CLM_DIV)	6864	6904
	8	s(MI) + s(SUM06, k = 5)	(1|SITE)	6626	6666
	9	s(MI) + s(SUM06, k = 5)	(1|STATE)	6311	6351
	10	s(MI) + s(SUM06, k = 5)		6310	6350
	11	s(MI) + s(N100)		6306	6346
	12	s(PDSI) + s(SUM06, k = 5))	(1|ECO_SEC)	6359	6399
	13	s(PDSI) + s(N100)		6355	6395
	14	s(MI) + s(SUM06, k = 5) + s(PDSI)		6295	6348
	15	s(MI) + s(SUM06, k = 5) + s(N100) + s(PDSI)		6290	6357

SUM06 is the “sum of all hourly average ozone concentrations ⩾ 60 ppb;” N100 is the “number of hours with O₃ concentrations at or above 100 ppb;” REG refers to four regional units, Northern, Southern, Interior West, and Pacific Northwest; CLM_DIV is a categorical variable indicating areas of similar temperature and precipitation; SITE identifies an ozone biosite; STATE identifies a US state; and ECO_SEC is a subdivision of a larger domain of similar climate based on vegetation, terrain features, and elevation. s( · ) indicates a nonparametric smoother, in this case a spline, of the enclosed term.

Figure 4. Example 4. Data generated from linear logistic model. Comparison of linear and quadratic logistic regression. (a) Jittered y_i vs. x_i. Small points are data. Dashed curve is the linear logistic fit. Dotted curve is the quadratic logistic fit. (b) Points are fitted values from quadratic and linear logistic regression. Line is y = x. (c) $log{P}_{\widehat{\theta }}\left(y_i\right)$ from quadratic and linear logistic regression. Line is y = x. (d) Boxplot of log-likelihood ratio $log\frac{P_{{\widehat{\theta }}_{\text{lin}}}\left(y_i\right)}{P_{{\widehat{\theta }}_{\text{quad}}}\left(y_i\right)}$

Figure 5. Example 4. Data generated from linear logistic model with one additional point ( − 20, 1). Comparison of linear and quadratic logistic regression. (a) Jittered y_i vs. x_i. Small points are data. Dashed curve is the linear logistic fit. Dotted curve is the quadratic logistic fit. (b) Points are fitted values from quadratic and linear logistic regression. Line is y = x. (c) $log{P}_{\widehat{\theta }}\left(y_i\right)$ from quadratic and linear logistic regression. The additional point is the one near ( − 17, −3.5). Line is y = x. (d) Boxplot of log-likelihood ratio $log\frac{P_{{\widehat{\theta }}_{\text{lin}}}\left(y_i\right)}{P_{{\widehat{\theta }}_{\text{quad}}}\left(y_i\right)}$.

Figure 6. Fitted values from the quadratic model vs. fitted values from the linear model in Example 4. Left panel: y = 0; right panel: y = 1. Circles: points generated from the linear model. Triangle: the added point.

Hastie, T. J., Tibshirani, R. J., and Friedman, J. H. (2009), The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.), Springer Series in Statistics, New York: Springer.

 Google Scholar

Wang, P., Baines, A., Lavine, M., and Smith, G. (2012), “Modelling Ozone Injury to U.S. Forests,” Environmental and Ecological Statistics, 19, 461–472.

 Web of Science ®Google Scholar