A preposterior analysis to predict identifiability in the experimental calibration of computer models

ABSTRACT

When using physical experimental data to adjust, or calibrate, computer simulation models, two general sources of uncertainty that must be accounted for are calibration parameter uncertainty and model discrepancy. This is complicated by the well-known fact that systems to be calibrated are often subject to identifiability problems, in the sense that it is difficult to precisely estimate the parameters and to distinguish between the effects of parameter uncertainty and model discrepancy. We develop a form of preposterior analysis that can be used, prior to conducting physical experiments but after conducting the computer simulations, to predict the degree of identifiability that will result after conducting the physical experiments for a given experimental design. Specifically, we calculate the preposterior covariance matrix of the calibration parameters and demonstrate that, in the examples that we consider, it provides a reasonable prediction of the actual posterior covariance that is calculated after the experimental data are collected. Consequently, the preposterior covariance can be used as a criterion for designing physical experiments to help achieve better identifiability in calibration problems.

Keywords:

• Gaussian process models
• Kriging
• Bayesian calibration
• computer experiments

Funding

The grants that supported this work from the National Science Foundation (CMMI-1233403, CMMI-0928320 and CMMI-0758557) and also the U.S. Army Tank-Automotive Research Development & Engineering Center (TARDEC) (contract number W911NF11D0001-0037) are gratefully acknowledged.

Additional information

Notes on contributors

Paul D. Arendt

Paul D. Arendt is an Analytics Senior Consultant in the Predictive Analytics group at CNA Financial Corporation, Chicago, Illinois. He obtained B.S. and M.S. degrees in General Engineering from the University of Illinois at Urbana–Champaign and obtained a Ph.D. in Mechanical Engineering from Northwestern University. His current work is to help create predictive analytics to segment and price commercial insurance risks.

Daniel W. Apley

Daniel W. Apley is a Professor of Industrial Engineering and Management Sciences at Northwestern University, Evanston, Illinois. He obtained B.S., M.S., and Ph.D. degrees in Mechanical Engineering and an M.S. degree in Electrical Engineering from the University of Michigan. His research interests lie at the interface of engineering modeling, statistical analysis, and predictive analytics, with particular emphasis on manufacturing variation reduction applications in which very large amounts of data are available. His research has been supported by numerous industries and government agencies. He received the NSF CAREER award in 2001, the IIE Transactions Best Paper Award in 2003, and the Wilcoxon Prize for best practical application paper appearing in Technometrics in 2008. He is Editor-in-Chief Elect of Technometrics and has served as Editor-in-Chief for the Journal of Quality Technology, Chair of the Quality, Statistics & Reliability Section of INFORMS, and Director of the Manufacturing and Design Engineering Program at Northwestern.

Wei Chen

Wei Chen is the Wilson-Cook Professor in Engineering Design at Northwestern University, Evanston, Illinois. She received her B.S., M.S., and Ph.D. degrees in Mechanical Engineering at the Shanghai Jiao Tong University, the University of Houston, and the Georgia Institute of Technology, respectively. Directing the Integrated Design Automation Laboratory, her current research involves issues such as simulation-based design under uncertainty, model validation, stochastic multi-scale analysis and design, robust shape and topology optimization, multidisciplinary optimization, consumer choice modeling, and enterprise-driven decision-based design. She is a Past Chair of the ASME Design Engineering Division Executive Committee and was an elected Advisory Board member of the Design Society (2007–2013). She is a review editor of Structural and Multidisciplinary Optimization and twice served as an Associate Editor of the ASME Journal of Mechanical Design. In addition, she serves as the Associate Editor of SIAM/ASA Journal on Uncertainty Quantification and a Department Editor for IIE Transactions. She was the recipient of the 1996 NSF Faculty Early Career Award, the 1998 American Society of Mechanical Engineers (ASME) Pi Tau Sigma Gold Medal achievement award, and the 2006 SAE Ralph R. Teetor Educational award. She is a Fellow of American Society of Mechanical Engineers and an Associate Fellow of American Institute of Aeronautics and Astronautics.