Do different decision-analytic modeling approaches produce different results? A systematic review of cross-validation studies

Abstract

When choosing a modeling approach for health economic evaluation, certain criteria are often considered (e.g., population resolution, interactivity, time advancement mechanism, resource constraints). However, whether these criteria and their associated modeling approach impacts results remain poorly understood. A systematic review was conducted to identify cross-validation studies (i.e., modeling a problem using different approaches with the same body of evidence) to offer insight on this topic. With respect to population resolution, reviewed studies suggested that both aggregate- and individual-level models will generate comparable results, although a practical trade-off exists between validity and feasibility. In terms of interactivity, infectious-disease models consistently showed that, depending on the assumptions regarding probability of disease exposure, dynamic and static models may produce dissimilar results with opposing policy recommendations. Empirical evidence on the remaining criteria is limited. Greater discussion will therefore be necessary to promote a deeper understanding of the benefits and limits to each modeling approach.

Keywords:

• compartmental models
• decision analysis
• decision trees
• health economic evaluation
• Markov model
• microsimulation
• state-transition models
• systematic review
• validity

Acknowledgements

The authors would like to thank K Campbell for her assistance in developing the search strategy for this systematic review.

Financial & competing interests disclosure

B Tsoi is supported through awards from the Father Sean O’Sullivan Research Centre, St. Joseph’s Healthcare Hamilton and Pfizer Graduate Student Award in Health Technology Assessment. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Key issues

• Each modeling approach will impose certain constraints and assumptions that may impact the results and consequently the resource allocation decision it seeks to inform. As such, specific circumstances may exist on when each approach would be the most appropriate.

• Decision criteria have been published to distinguish the features characterizing each specific modeling approach. However, few studies have addressed whether and how these decision criteria are important empirically. Cross-validation studies can provide insight into this topic.

• Only nine studies were identified that have conducted cross-validation between different modeling approaches. Among the decision criteria studied, relatively consistent themes emerged from the empirical work regarding the impact of a decision criteria on the construction and overall results to a model.

• Aggregate- and individual-level models often produce similar results and qualitative conclusions. Rather, the choice on a model’s resolution is important in terms of the potential trade-off between model validity and feasibility.

• Interaction must be captured when indirect effects are important and are expected to be large. Studies consistently show different results estimated by dynamic (i.e., captures both direct and indirect effects) and static models (i.e., captures only direct effects) that often lead to conflicting policy recommendations.

• Overall, there has been limited work done on this topic as only nine studies were identified from a systematic literature review of bibliographic databases and gray literature. Additional research will thus be necessary to not only verify the existing understanding of the aforementioned decision criteria that have been studied more extensively but also to address the importance of the remaining less studied decision criteria (e.g., time advancement mechanism, resource constraints).

• Understanding the implications of the various structural features specific to a modeling approach is necessary to foster knowledge on the benefits and limits of each modeling approach. This would provide insight into how to select a suitable modeling approach for a particular decision problem.