Validation and calibration of structural models that combine information from multiple sources

ABSTRACT

Introduction: Mathematical models that attempt to capture structural relationships between their components and combine information from multiple sources are increasingly used in medicine.

Areas covered: We provide an overview of methods for model validation and calibration and survey studies comparing alternative approaches.

Expert commentary: Model validation entails a confrontation of models with data, background knowledge, and other models, and can inform judgments about model credibility. Calibration involves selecting parameter values to improve the agreement of model outputs with data. When the goal of modeling is quantitative inference on the effects of interventions or forecasting, calibration can be viewed as estimation. This view clarifies issues related to parameter identifiability and facilitates formal model validation and the examination of consistency among different sources of information. In contrast, when the goal of modeling is the generation of qualitative insights about the modeled phenomenon, calibration is a rather informal process for selecting inputs that result in model behavior that roughly reproduces select aspects of the modeled phenomenon and cannot be equated to an estimation procedure. Current empirical research on validation and calibration methods consists primarily of methodological appraisals or case-studies of alternative techniques and cannot address the numerous complex and multifaceted methodological decisions that modelers must make. Further research is needed on different approaches for developing and validating complex models that combine evidence from multiple sources.

KEYWORDS:

• Mathematical models
• structural models
• structural relationships
• simulation
• calibration
• validation

Acknowledgments

This paper is based on a Methods Research Report funded by the Agency for Healthcare Research and Quality, contract number HHSA 290 2007 10055 I. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

Declaration of interest

The authors have no relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript. This includes employment, consultancies, honoraria, stock ownership or options, expert testimony, grants or patents received or pending, or royalties.

Notes

1. An exact count was not provided in the main text of the paper and the supplementary Appendix was not downloadable from the journal Website.

Additional information

Funding

This manuscript received no funding.