Measuring what's missing: practical estimates of coverage for stochastic simulations

Abstract

Stochastic sensitivity analyses rarely measure the extent to which realized simulations cover the search space. Rather, simulation lengths are typically chosen according to expert judgement. In response, this paper recommends a novel application of Good-Turing estimators of missing distributional mass. Using the United Nations Development Programme's Human Development Index, the empirical performance of such coverage metrics are compared to alternative measures of convergence. The former are advantageous – they provide probabilistic estimates of simulation coverage and permit calculation of strict bounds on estimates of pairwise dominance (for all possible weight vectors, how often country X dominates country Y).

Keywords:

• sensitivity analysis
• uncertainty analysis
• Monte Carlo
• simulation coverage
• HDI

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1. The arbitrary nature of the choice is underlined by the fact that the working paper version of the same paper uses four million regressions.

2. $\mathcal{X},\mathcal{B}$, respectively refer to the support of the exogenous inputs and parameters; $\mathcal{F}$ is the set of permissible model structures.

3. In the context of uncertainty analysis, it is often the case that there is insufficient prior information to specify $\mathcal{H}$. Thus, Laplace's principle of insufficient reason typically recommends use of independent uniform (marginal) distributions or similar pseudo-random sequences to achieve even coverage of the search space.

4. A standard result from the optimization literature is that pure random (blind) search of the sample space will converge to an arbitrarily small neighbourhood ε of a specific point $Y=y$ with probability one.[39]

5. This derivation draws on the exposition in [36].

6. See [36] for discussion of the convergence properties of this estimator (also [40,41]).

7. The education and life expectancy sub-indexes are themselves composite indexes. However, for simplicity each of the three high-level components is taken as given datum. Thus, this exercise ignores the implicit weights associated with how the high-level components are constructed, as well as other sources of uncertainty such as due to measurement error or normalization techniques (for elaboration see [13,34]).

8. This is necessary given the weights must sum to one. Sampling independently from a uniform distribution will not yield even coverage of the weight space.

9. See http://hdr.undp.org/en/media/HDR_2013_FAQ_HDI_EN.pdfhttp://hdr.undp.org/en/media/HDR_2013_FAQ_HDI_EN.pdf.

10. In the case of the MCQE, normalization by specific quantiles also is sensible. The mean is chosen for simplicity, however.

11. This procedure is only possible when the outcome variable is bounded/censored – here between 0 and 1. In more general continuous cases, additional assumptions regarding the location of missing data would be required. I am grateful to an anonymous referee for pointing this out.