Adaptive Inference for Multi-Stage Survey Data

Abstract

Multi-level models can be used to account for clustering in data from multi-stage surveys. In some cases, the intraclass correlation may be close to zero, so that it may seem reasonable to ignore clustering and fit a single-level model. This article proposes several adaptive strategies for allowing for clustering in regression analysis of multi-stage survey data. The approach is based on testing whether the PSU-level variance component is zero. If this hypothesis is retained, then variance estimates are calculated ignoring clustering; otherwise, clustering is reflected in variance estimation. A simple simulation study is used to evaluate the various procedures.

Keywords:

• Adaptive estimation
• Cluster sampling
• Huber–White variance estimator
• Multi-level models
• Variance components

Mathematics Subject Classification:

• 62D05
• 62J05

1. Introduction

Two-stage sampling is used in many surveys of social, health, economic, and demographic topics. Final population units are grouped into primary sampling units (PSUs). The first stage of selection is a sample of PSUs and the second stage is a sample of units within selected PSUs. For example, PSUs might be schools and units might be students in schools, or PSUs might be households and units might be people, or PSUs might be geographic areas and units might be households (for example, see, Cochran, 1977; Kish, 1965).

Two-stage sampling is typically used because of the following.

	There is no sampling frame of final units, but a frame of PSUs (e.g., a list of suburbs) is available.
	Cost efficiency—for example, it is much cheaper to draw a two-stage sample of 100 students from 10 schools than draw a simple random sample of 100 students, as those students might be dispersed over 100 schools (Snijders, 2005).
	Within-group correlations may be of interest in their own right. For instance, the correlation between values for students in the same school might be of interest.

A complication of two-stage sampling is that values of interest may tend to be more similar for units from the same PSU than for units from different PSUs. If so, this should be reflected in the analysis procedure. One way of doing this is by fitting a multi-level model.

Multi-level models are generalization of regression models. Let y _ij be a dependent variable of interest, and x _ij a vector of covariates for unit j in PSU i. The two-level linear mixed model (LMM) (Goldstein, 2003) is given by

where c denotes the number of PSUs in the sample, m _i denotes the number of observations selected in PSU i, β is the vector of unknown regression coefficients, is a PSU specific random effect, and e _ij is assumed to be . Therefore, , with variance . Variances of regression coefficient estimates can be estimated by either standard likelihood theory based on model (1) (West et al., 2007), or by using the robust Huber–White estimator (Huber, 1967; White, 1982). We assume that the sampling design is ignorable (Sugden and Smith, 1984) so that a simple LMM can be applied to the sample. The issues associated with the effect of more complex sampling designs on multilevel models are discussed by Pfeffermann et al. (1998).

The intra-class correlation (ρ) is a measure of the association between the regression residuals for members of the same PSU. It is the ratio of the population variance between PSUs and the total variance. Consequently, it is given by the formula (Commenges and Jacqmin, 1994).

In practice, the intra-class correlation is often quite small. For example, if units within PSUs are no more homogenous than units over all PSUs, then the intra-class correlation is zero. On the other hand, if units from the same PSU have equal values then the intra-class correlation is 1. Generally, the intra-class correlation is positive, but in the case of equal number of observations in each PSU, it is usually less than 0.1 when PSUs are geographic areas and final units are households in these areas (Verma et al., 1980). When PSUs are households and final units are people in households it is typically between 0 and 0.2 (Clark and Steel, 2002).

There are number of possible approaches to estimating the regression coefficients and their variances when the intraclass correlation (ρ) is thought to be small or has been estimated as a small value. One is to fit a linear mixed model regardless. Another is to fit a linear model assuming independent observations. However, if the sample design is relatively clustered, that is a large number of final units are selected from each PSU, the estimated variances resulting from a linear mixed model can be a lot larger than those obtained from a linear model assuming independent observations, leading to wider confidence intervals. Moreover, a linear mixed model is more complicated to fit and explain than a simple linear model, so the latter is preferable provided it does not give misleading inference. A third alternative is to use an adaptive strategy based on testing the null hypothesis that the random effect variance component, , is zero. If the null hypothesis is accepted we use the linear model for estimating the variances of the estimated regression coefficients . On the other hand, if the null hypothesis is rejected we use the estimated variance for either using the standard likelihood theory variance estimator for the LMM or the Huber–White method.

Adaptive confidence intervals calculated in this way do not incorporate the model selection uncertainty, and so may not have the correct coverage rates. The extent of this problem will be evaluated by simulation. An alternative approach would be to fit both the LM and LMM and base estimates and inference on model averaging of these two models (Hoeting et al., 1999; Yuan and Yang, 2005). We do not develop this approach here, as one of our aims is to simplify the modeling process when the intraclass correlation is small.

This article is divided into five sections. Section 2 will describe the linear mixed model including an outline of the standard likelihood theory estimators of β and and the Huber–White estimator of . Section 3 will describe an adaptive strategy. A simulation study of the adaptive and other methods will be described in Sec. 4. In Sec. 5, we will draw conclusions.

2. Fitting the Linear Mixed Model

2.1. The Model

Let X be the n × p design matrix, which is assumed to be of rank p, and be the complete set of observations in the c PSUs, where y _i  = (y _i1,…, y _{im i} )′ is the observed vector for the ith PSU.

Model (1) can also be written as

where V is a block diagonal matrix, V = diag(V _i , i = 1,…, c), and

where J _{m i} is an m _i  × m _i matrix with all entries equal to 1, and I _{m i} is the m _i  × m _i identity matrix. The variance components and are generally not known, and are usually estimated by Restricted Maximum Likelihood (REML), giving estimates of V _i .

REML was first introduced by Patterson and Thompson (1971) as a modification of Maximum Likelihood. The REML method is often presented as a technique based on maximization of the likelihood of a set of linear combinations of the elements of the response variable y, say k′y, where k is chosen so that k′y is free of fixed effects. One of the attractive aspects of REML is that it takes into account the degrees of freedom used up by the estimation of the fixed effects (Diggle et al., 1994). There is also no loss of information about the variance components when the inference is derived from k′y rather than y.

The restricted log-likelihood function is given by

where V = diag(V _i ) and V _i are given by (3). Maximizing (4) with respect to and gives the REML estimates of these parameters. The REML estimate of is given by

The simple special case of model (1) where the model includes just a grand mean parameter will be used in the simulation study in Sec. 4. This model is given by defining x _ij to be 1 for all i:

In the intercept-only model, the REML estimates are defined by the following system of equations:

(Sahai and Ojeda, 2005), where , is the variance of the mean for PSU i, and is the estimate of λ _i . Equation (7) must be solved numerically with respect to and . In the balanced data case (m _i  = m for all i), the REML estimates have a simpler form. Let , the system of Eqs. (7) becomes

(Sahai and Ojeda, 2004).

2.2. Likelihood Theory Estimator of

In this section, we will discuss the variances of the estimated regression coefficients and their estimators. The estimated variance of the REML , defined by expression (5), is given by

where . For the special case (6) of an intercept-only model, this simplifies to

In the balanced data case, the variance estimator simplifies further to

A confidence interval for could be constructed using the formula

where . However, it is not clear how the degrees of freedom in (12) should be defined for mixed models. Faes et al. (2009) suggested the following approximate confidence interval based on a scaled t-distribution: , where , and with defined in (9). Faes et al. (2009) did not specify how V(T) should be estimated; we will use a parametric bootstrap in our simulation in Sec. 4. Other approaches have been suggested; for example, see Satterthwaite (1941) and Kenward and Roger (1997). The method of Faes et al. (2009) has the advantage that it extends naturally to non Gaussian models, unlike the other approaches.

2.3. Huber–White Estimator

The estimator in (9) will be approximately unbiased provided the variance model (3) is correct. If this is not the case, will be biased and inference will be incorrect. An alternative to ML or REML estimates of is the robust variance estimation approach described by Liang and Zeger (1986), in the context of modeling longitudinal data using generalized estimating equations (GEE). This approach can be applied to the analysis of data collected using PSUs, where observations within PSUs might be correlated and the observations in different PSUs are independent.

This approach can be referred to as robust or Huber–White variance estimation (Huber, 1967; White, 1982). It will be used as an alternative approach to estimating in this article. The method yields asymptotically consistent covariance matrix estimates even if the variances and covariances assumed in model (2) are incorrect. It is still necessary to assume that observations from different PSUs are independent.

In Eq. (9) in Sec. 2.2, we suggested estimating the variance of based on substituting REML estimates of and into V _i . An alternative estimator of V _i is , where . is approximately unbiased for V _i even if (3) does not apply:

Note that

One way to construct a robust estimator of is to substitute the robust estimator in (14) as follows:

When there is only an intercept in the model (x _i  = 1), (14) becomes

(Al-Zou'bi et al., 2009). In the balanced data case (i.e., m _i  = m), from Eq. (16) and since is constant, this estimator becomes

Exact confidence intervals can then be calculated using (12) with degrees of freedom equal to c − 1 (MacKinnon and White, 1985).

2.4. REML Likelihood Ratio Test (RLRT) for Testing

Suppose we want to test vs. . The REML approach can be used to derive the likelihood ratio test (LRT) statistic for this test.

Visscher (2006) showed that the REML-based likelihood ratio test (RLRT) for testing in model (1) is given by:

where . The large sample distribution of the likelihood ratio Δ is a 50:50 mixture of χ² distribution with 0 and 1 degrees of freedom as the parameter values fall on the boundary of the parameter space (Self and Liang, 1987).

3. Adaptive Strategies

In this article, we consider two adaptive strategies. Both of them rely on the idea of testing the variance component in model (1). In the first adaptive strategy, if we reject , we use the LMM estimators of defined in Eq. (9). On the other hand, if we accept H ₀, then we assume that and we fit the standard linear model with independent errors. This strategy is explained in Fig. 1, where is the estimator of using the LM strategy, is the estimator of using the LMM strategy and is the adaptive estimator.

Figure 1 Flowchart explaining the adaptive procedure using the estimated variance extracted from the LMM.

The second adaptive strategy, explained in Fig. 2, is identical, except that the robust Huber–White estimator is used instead of when H ₀ is rejected.

Figure 2 Flowchart explaining the adaptive procedure using Huber–White estimator.

The advantage of the adaptive strategy is that we use the simple linear model to derive variance estimators, unless there is strong evidence that . This has the benefit of simplifying the model and may also give tighter confidence intervals. However, it is not clear whether the adaptive approaches will give valid confidence intervals for β, because the confidence intervals assume non adaptive procedures.

4. Simulation Study

A simulation study was conducted to compare the adaptive and non adaptive methods for estimating . Data were generated from model (6), with m _i  = m and an intercept only model. The values of ρ, m, and c were varied. One thousand samples were generated in each case. The values of and were set to and 1 respectively, to ensure that the intraclass correlation was ρ.

For each sample the estimated regression coefficients and the estimators of were calculated for the LMM and LM models using the lme and lm packages (Pinheiro and Bates, 2000) in the R statistical environment (R Development Core Team, 2007). The true variance of was determined by calculating the variance over all 1,000 simulations.

The hypothesis was tested as described in Sec. 2.4. The two adaptive strategies, ADM and ADH, were as defined in Sec. 3. 90% confidence intervals were calculated for the LMM method using the method of Faes et al. (2009), as described in Sec. 2.2. Huber–White confidence intervals were calculated as discussed in Sec. 2.3, and the adaptive confidence intervals were calculated as discussed in Sec. 3.

Tables 1-4 show the ratio of the mean estimated variance of to the true variance using the four strategies of estimation (ADM, ADH, LMM, and Huber) with values of ρ of 0, 0.025, 0.05, and 0.1. In all tables we used β = 0 and significance level α = 0.1 for testing . The tables show the non coverage rates of 90% confidence intervals for β and the lengths of these confidence intervals. The proportion of samples where was rejected are also shown. Results on non coverage and 90% confidence intervals length are also shown in graphical form in Figs. 3-8. In the graphs we also include the LM strategy of estimation so that the effect of completely ignoring the clustered nature of the data can be examined.

Figure 3 Confidence interval non coverage rates (%) for different variance estimation methods and various numbers of PSUs (c) and units per PSU (m) for intraclass correlation (ρ) of 0.

Figure 4 Confidence interval non coverage rates (%) for different variance estimation methods and various numbers of PSUs (c) and units per PSU (m) for intraclass correlation (ρ) of 0.05.

Figure 5 Confidence interval non coverage rates (%) for different variance estimation methods and various numbers of PSUs (c) and units per PSU (m) for intraclass correlation (ρ) of 0.1.

Figure 6 Confidence interval lengths for different variance estimation methods and various numbers of PSUs (c) and units per PSU (m) for intraclass correlation (ρ) of 0.

Figure 7 Confidence interval lengths for different variance estimation methods and various numbers of PSUs (c) and units per PSU (m) for intraclass correlation (ρ) of 0.05.

Figure 8 Confidence interval lengths for different variance estimation methods and various numbers of PSUs (c) and units per PSU (m) for intraclass correlation (ρ) of 0.1.

Table 1 Variance ratios, length, and non coverage of the 90% confidence intervals for β, and power of testing with ρ = 0

PSUs	Obs					Non coverage of CI for β				P (reject H 0)	Confidence interval length
c	m	ADM	ADH	LMM	Hub	ADM	ADH	LMM	Hub	Lrt	ADM	ADH	LMM	Hub
2	2	1.175	1.175	1.360	0.964	9.9	9.9	13.0	10.6	10.5	5.071	2.935	5.658	4.961
2	5	1.224	1.224	1.452	0.991	9.9	9.8	10.1	8.9	5.8	1.252	1.537	1.341	3.131
2	10	1.316	1.316	1.568	1.065	8.4	8.4	9.1	9.9	5.2	0.866	1.062	0.947	2.235
2	25	1.155	1.155	1.412	0.947	10.5	10.5	10.4	11.1	3.8	0.528	0.619	0.585	1.393
5	2	1.078	1.078	1.195	1.013	9.7	9.7	10.4	10.7	10.5	1.189	1.193	1.214	1.282
5	5	1.164	1.164	1.281	1.061	9.0	9.0	8.8	9.1	8.0	0.720	0.727	0.734	0.806
5	10	1.196	1.196	1.319	1.081	7.9	7.9	8.1	8.3	7.5	0.502	0.506	0.512	0.569
5	25	1.039	1.039	1.159	0.931	10.1	10.1	10.3	11.2	7.4	0.316	0.320	0.327	0.360
25	2	0.949	0.949	0.989	0.924	11.1	11.1	11.6	11.1	10.9	0.478	0.478	0.474	0.478
25	5	1.011	1.011	1.021	0.975	10.2	10.2	10.3	10.2	9.4	0.303	0.302	0.300	0.303
25	10	1.050	1.050	1.061	0.994	9.1	9.1	9.8	9.7	7.6	0.213	0.213	0.210	0.213
25	25	1.079	1.079	1.093	1.032	8.7	8.8	9.5	9.8	8.7	0.135	0.135	0.134	0.136

Table 2 Variance ratios, length, and non coverage of the 90% confidence intervals for β, and power of testing with ρ = 0.025

PSUs	Obs					Non coverage of CI for β				P (reject H 0)	Confidence interval length
c	m	ADM	ADH	LMM	Hub	ADM	ADH	LMM	Hub	Lrt	ADM	ADH	LMM	Hub
2	2	1.202	1.202	1.414	1.023	8.3	8.3	12.5	10.7	11.1	4.910	2.964	5.351	5.009
2	5	1.227	1.227	1.472	1.097	10.8	10.3	11.4	9.2	8.4	1.373	1.763	1.495	3.510
2	10	1.125	1.125	1.376	1.009	11.7	11.7	11.8	8.8	7.0	0.900	1.160	1.009	2.469
2	25	1.035	1.035	1.247	1.001	17.8	17.8	17.7	9.8	11.6	0.661	0.937	0.744	1.790
5	2	0.984	0.984	1.091	0.919	10.7	10.7	10.8	11.6	11.6	1.207	1.218	1.233	1.301
5	5	1.001	1.001	1.115	0.958	11.2	11.2	10.9	11.0	11.2	0.743	0.752	0.764	0.851
5	10	0.973	0.973	1.089	0.966	13.0	12.9	12.7	9.9	15.3	0.549	0.557	0.575	0.643
5	25	0.961	0.961	1.079	1.006	14.9	14.8	13.0	10.3	24.3	0.381	0.389	0.407	0.455
25	2	1.025	1.025	1.077	1.011	9.8	9.8	10.2	9.5	11.7	0.488	0.489	0.488	0.492
25	5	0.934	0.934	0.947	0.944	11.6	11.8	12.4	11.2	15.8	0.312	0.312	0.310	0.320
25	10	0.985	0.985	1.006	1.028	11.0	11.4	10.8	11.0	27.3	0.231	0.230	0.231	0.240
25	25	0.896	0.896	0.909	0.929	11.1	11.6	11.3	10.8	64.9	0.170	0.168	0.170	0.173

Table 3 Variance ratios, length, and non coverage of the 90% confidence intervals for β, and power of testing with ρ = 0.05

PSUs	Obs					Non coverage of CI for β				P (reject H 0)	Confidence interval length
c	m	ADM	ADH	LMM	Hub	ADM	ADH	LMM	Hub	Lrt	ADM	ADH	LMM	Hub
2	2	1.111	1.111	1.337	0.987	8.9	8.9	12.6	10.3	10.2	5.676	3.064	6.404	5.362
2	5	1.127	1.127	1.388	1.060	10.7	10.4	10.9	9.2	8.6	1.324	1.756	1.462	3.605
2	10	1.031	1.031	1.243	0.975	15.0	15.0	15.2	10.7	11.5	1.003	1.455	1.132	2.732
2	25	1.135	1.135	1.299	1.141	19.0	19.0	15.5	9.4	19.4	0.831	1.318	0.918	2.167
5	2	0.957	0.957	1.060	0.900	11.9	11.9	12.4	12.6	10.9	1.215	1.226	1.244	1.316
5	5	0.983	0.983	1.099	0.980	11.7	11.7	11.0	10.1	13.1	0.774	0.784	0.805	0.905
5	10	0.949	0.949	1.060	0.988	14.0	13.9	13.9	10.0	22.7	0.590	0.603	0.620	0.704
5	25	0.975	0.975	1.049	1.023	13.8	13.7	12.8	8.7	43.6	0.477	0.487	0.501	0.547
25	2	0.990	0.990	1.043	0.992	10.3	10.2	11.0	10.4	14.4	0.499	0.499	0.501	0.507
25	5	0.938	0.938	0.953	0.972	11.5	11.5	11.0	9.9	30.0	0.332	0.330	0.332	0.342
25	10	0.979	0.979	0.995	1.018	11.0	11.2	11.7	9.9	57.4	0.259	0.257	0.261	0.266
25	25	1.017	1.018	1.020	1.025	9.3	9.0	9.2	8.4	92.2	0.207	0.205	0.207	0.206

Table 4 Variance ratios, length, and non coverage of the 90% confidence intervals for β, and power of testing with ρ = 0.1

PSUs	Obs					Non coverage of CI for β				P (reject H 0)	Confidence interval length
c	m	ADM	ADH	LMM	Hub	ADM	ADH	LMM	Hub	Lrt	ADM	ADH	LMM	Hub
2	2	1.096	1.096	1.322	1.016	9.4	9.4	13.8	11.0	10.2	5.112	3.145	5.791	5.538
2	5	1.102	1.102	1.335	1.103	15.7	15.3	13.8	9.4	13.4	1.607	2.238	1.775	4.127
2	10	0.887	0.887	1.055	0.900	20.3	20.2	17.7	10.0	15.2	1.119	1.748	1.266	3.141
2	25	0.949	0.949	1.058	0.996	28.7	28.6	21.9	7.1	29.1	1.130	1.920	1.249	2.812
5	2	1.063	1.063	1.189	1.062	11.0	10.9	10.5	9.7	16.6	1.294	1.310	1.342	1.434
5	5	0.929	0.929	1.042	0.962	13.4	13.1	13.2	10.6	20.5	0.837	0.851	0.882	0.984
5	10	1.010	1.010	1.111	1.070	14.0	13.7	12.9	10.1	37.7	0.711	0.727	0.755	0.828
5	25	0.966	0.966	0.999	0.993	15.4	14.9	13.5	10.4	67.1	0.643	0.652	0.662	0.694
25	2	0.971	0.971	1.021	0.983	10.5	10.3	10.7	10.2	21.9	0.522	0.523	0.526	0.533
25	5	0.985	0.985	1.001	1.027	12.1	11.9	12.3	10.1	53.5	0.369	0.366	0.371	0.380
25	10	0.974	0.974	0.979	0.989	11.2	11.4	10.8	10.0	86.5	0.308	0.306	0.309	0.310
25	25	0.974	0.974	0.974	0.974	9.8	9.4	9.8	9.4	99.9	0.264	0.263	0.264	0.263

The variance estimators are approximately unbiased as all ratios are approximately 1. The only exception is the variance estimator using the LMM strategy, it tends to be biased when we have 2 PSUs with all numbers of observations per PSU for values of ρ of 0, 0.025, and 0.05. For ρ = 0.1, it tends to be biased when there are 2 PSUs with 5 or less observations per PSU. It tends to be biased when we have 5 PSUs with all numbers of observations per PSU in case of ρ = 0, as well.

Non coverage rates for confidence intervals for β were close to the nominal rate of 10% when ρ = 0 for all methods.

For ρ ≠ 0, Huber non coverage was close to 10% in all cases. When ρ was non zero, the LMM non coverage was high when c was 2 and m was high. LMM non coverage rates were reasonably close to nominal in all other cases.

The ADM and ADH methods generally had similar non coverage rates. Both had high non coverage when ρ and m were large and c was small. In particular, when c = 2 and m was 5 or more, or when c = 5 and m was 10 or more, the two adaptive methods had high non coverage (around 15–20%). In all other cases, the ADM and ADH non coverage rates were close to nominal.

Figure 3 shows that LM non coverage was close to 10% when ρ = 0. It was very high otherwise as shown by Figs. 4 and 5. Hence, use of LM without at least checking is not a viable strategy.

Figures 6-8 show that confidence intervals using the LM strategy are the shortest, however this strategy is not viable because of its high non coverage when ρ ≠ 0. The Huber-based approach gives the widest confidence intervals in general. The ADM and ADH confidence intervals are almost always shorter than the LMM and Huber ones, respectively. When there were 2 and 5 PSUs it is very clear that ADM and ADH confidence intervals are much shorter than LMM and Hub confidence intervals, for all values of ρ ≠ 0. In the case where there were 25 PSUs, these lengths become closer.

5. Conclusions

1. Adaptive confidence intervals are non robust to intraclass correlation in designs with few PSUs and large sample sizes in each PSU. In these designs, even a small intraclass correlation will substantially inflate the variance of the mean, however the PSU-level variance component is unlikely to be statistically significant even if the intraclass correlation is as high as 0.1. As a result, when the number of PSUs (c) is 2 or 5, and the number of observations per PSU (m) is 25, both of the adaptive estimators have higher than desirable non coverage when the ICC is non zero, of the order of 15–20%. It appears that for extreme design where c is 5 or less and m is 25 or more, clustering must be allowed for in variance estimates, even if it is not statistically significant. However, provided c is 10 or more, clustering need only to be allowed for when is significant, even if m is large.

2. In comparing the Linear Mixed Model (LMM) with the adaptive version (ADM), we find that:

Both the LMM and ADM approaches have close to nominal non coverage, except for extreme designs of the kind discussed in 1. For these designs, the adaptive and non adaptive LMM methods both have high non coverage. In the case of the adaptive method, this is presumably because there is not much power to detect the PSU-level variance component, even when it is substantial. For the non adaptive LMM, the problem seems to be that the confidence intervals used are not exact and do not do well for very small sample sizes.

The ADM confidence intervals are noticeably narrower for c equal to 2 and 5, but there is not much to choose between ADM and LMM for c = 25.

3. In comparing the robust Huber–White approach with the adaptive version (ADH), we find that:

	The Huber approach has close to nominal non coverage in all cases. So does the ADH approach, except for the extreme designs mentioned in 1.
	The Huber method gives wide confidence intervals when c is small (2 or 5) even though the non coverage is close to the nominal 10%. This is because the degrees of freedom for this method is equal to (c − 1). ADH has much narrower confidence intervals, because its degrees of freedom are equal to (n − 1) rather than (c − 1) if the PSU-level variance component is not significant.

4. This leads to the following recommendations:

	Designs with fewer than 10 PSUs, and a large sample size in each PSU should be avoided, even if the intraclass correlation is believed to be low.
	Provided this advice is followed, clustering can be ignored if the PSU-level variance effect is insignificant. This gives close to correct coverage, while giving shorter confidence intervals (at least slightly).

5. Future research will focus on whether different criteria (other than significance for the PSU-level variance component) give better adaptive confidence intervals, and on unbalanced designs and non normal data. It is possible that model averaging of the linear and linear mixed models would improve on the coverage properties of the adaptive approach, even for highly clustered designs, and this approach would also be worth investigating.

Acknowledgments

An anonymous referee made detailed suggestions which considerably improved the article. The authors also thank Dr. Steven Roberts of the School of Finance and Applied Statistics at the Australian National University for his advice on model averaging.