Heuristics for the regression of stochastic simulations

Abstract

Modelling and simulation environments that are stochastic in nature present a multitude of problems in the creation of meta-models, notably the reduction of the quality of fit due to statistical error. This report addresses this issue by first determining the minimum number of repetitions required for interval estimates to hold true regardless of interval width. These intervals are then used for meta-model regressions. Four measures were examined. Sample mean and variance interval coverage accuracy was studied by varying skewness, kurtosis, desired coverage, and sample size within the Pearson family of distributions. Binomial proportion and quantile interval coverage accuracy was studied with the standard normal with varying sample size, desired coverage, and quantile level. Finally, heuristic measures to determine how repetitions are related to the quality of the meta-model fit were developed based on the experimentation with a canonical problem. The ratio of confidence interval width to the range of sample measures was found to be an indicator of the impact of statistical error on the quality of model fit. Regression methods of weighted least squares (WLS), ordinary least squares for constant sample sizes, and constant interval widths were compared. The WLS method is suggested for stochastic regressions of simulations.

Keywords:

• meta-models
• heuristics
• interval estimation
• statistics

Notes

1 The training data set is used in the regression to determine the coefficients of the model. The validation data set is not used in the regression; however, it can be used to determine the appropriate model form. The test data set is independent of the model creation and is used to determine the model's predictive ability.