Methods to construct a step-by-step beginner’s guide to decision analytic cost-effectiveness modeling

Figures & data

Table 1 Methods used to develop the beginner’s guide (BG) to decision analytic cost-effectiveness modeling

Research	Phase I	Phase II	Phase III	Phase IV
Method	Systematic review	Submethods review	Nominal group technique (consensus agreement)	Validation
Objective	Review of best practice modeling guidelines	Review of submethods literature	To reach agreement on the draft BG	To test the usability and completeness of the BG
Outcome	An exhaustive list of steps undertaken to develop a model	The guidance, recommendation, or instructive statement to support each step identified in Phase I	A finalized BG	Revised guide to include reviewer feedback
Output (result)	Framework for developing the BG	Draft BG	BG ready for testing	Tested and improved BG

Table 2 Consolidated summary of descriptions of the model development process

Briggs 2000 (alongside RCT)^Citation15	Sonnenberg (1994)^Citation14	Stahl (2008)^Citation13	Sargent (2010)^Citation12	Chilcott (2010)^Citation8 Tappenden (2012)^Citation66
What will be modeled?
–	1. Biological truth to be modeled	2. Describe the system under study	1. Problem entity	1. Understanding the decision problem
How will it be modeled? (conceptual)
1. Setting a reference case of methods	2. Theoretical model (represents an understanding of the biological truth) 3. Practical model (“most detailed model which can be constructed given the limitations of the available data and the need for the model to be understood”)	1. Describe the development of a simulation model	2. Conceptual model	2. Conceptual modeling (conceiving the model, cognitive processes of thinking about the model and the potential methods to be used, information available, etc). “Defining the boundary and depth of a model”^Citation8 and identifying critical factors for inclusion in the model are important during this phase.
Building the model
2. Specifying clinical/demographic patient characteristics	4. Implementation model (actual model in a software package)	–	3. Computerized model	3. Implementation of the model (the actual programming of the model software)
Running the model
3. Applying Bayesian methods to estimate data and distributions for the running of PSA	–	3. Evaluate the consequences of a given strategy 4. Explore the model	–	4. Model checking phase is undertaken to verify that the model is working as it should be and “includes all activities used to check the model”^Citation8
Disseminating the model results
–	–	5. Predict or forecast the behavior of the system and persuade decision-makers through consensus and evidence	–	5. Engaging with the decision (reporting of the model results and answering the original decision problem)

Note: The numbering in this table (read vertically within each column) corresponds to the order of the task described by the original (referenced) author.

Abbreviation: RCT, randomized controlled trial; PSA, probabilistic sensitivity analysis.

Table 3 Scope of review of eight submethods to inform the content of the beginner’s guide

Scope of submethod review	Literature reviewed
1. Evidence
Literature search retrieval and selection, selecting evidence for input parameters, evidence grading if relevant	Booth (2010)^Citation16; Paisley (2010)^Citation17; Braithwaite et al (2007)^Citation18; Nuijten (1998)^Citation19; Cooper (2007)^Citation20
2. Model structure
General methods for selecting model structure	Barton et al (2004)^Citation9; Cooper et al (2007)^Citation21; Brennan et al (2006)^Citation10
3. Resource valuation
Identifying and valuing resources, discounting	Miners (2008)^Citation22, Hay et al (2010)^Citation23; Shi et al (2010)^Citation24; Brouwer et al (2005)^Citation25; Claxton et al (2006)^Citation26; Claxton et al (2011)^Citation27; Gravelle et al (2007)^Citation28; Nord (2011)^Citation29
4. Effectiveness (health outcomes)
Clinical health (efficacy/effectiveness), methods to describe health-related quality of life (disease-specific; disease- and symptom-specific; generic measures (Euro-QOL EQ-5D, SF-6D, Health Utilities Index). Methods to value health-related quality of life (standard gamble; time trade off; rating scales), eliciting preferences (patient; medical experts; general population), discounting outcomes	Gray et al (2010)^Citation30; Craig et al (2009)^Citation33; Brazier (2008)^Citation31; Neumann et al (2000)^Citation32; Gold et al (2002)^Citation35; McDonough and Tosteson (2007)^Citation34
5. Uncertainty
Types of uncertainty (parameter; methodological, model). Methods to characterize uncertainty: deterministic (univariate and multivariate), probabilistic; model averaging; model selection; parameterizing subfunctions	Briggs (2000)^Citation15; Briggs and Gray (1999)^Citation36; Briggs and Gray (1999)^Citation37; Claxton (2008)^Citation38; Groot Koerkamp et al (2007)^Citation57; Sculpher (2008)^Citation40; Andronis et al (2009)^Citation43; Limwattananon (2008)^Citation44; Bojke et al (2006)^Citation45; Strong (2012)^Citation67; Brisson and Edmunds (2006)^Citation41; Groot Koerkamp et al (2010)^Citation42
6. Validity
Types of validity (face; internal; external; convergent; predictive) and validation methods	Schlesinger (1979)^Citation47; Sargent (2010)^Citation12; McCabe and Dixon (2000)^Citation48; Kim and Thompson (2010)^Citation49; Sendi et al (1999)^Citation50; Halpern et al (1998)^Citation51; Weinstein (2001)^Citation68; Weinstein et al (2003)^Citation52; Chilcot et al (2010)^Citation8; Stahl (2008)^Citation13
7. Reporting
Incremental cost-effectiveness ratio and confidence intervals; cost-effectiveness plane; cost-effectiveness acceptability curve; cost-effectiveness acceptability frontier; net benefit approach; value of information analysis	Black (1990)^Citation69; Briggs and Gray (1999)^Citation36; Briggs et al (1997)^Citation58; Fan and Zhou (2007)^Citation59; Dinh and Zhou (2006)^Citation60; Polsky et al (1997)^Citation61; Barton et al (2008)^Citation53; Briggs (2000)^Citation15; Fenwick and Briggs (2007)^Citation54; Fenwick et al (2001)^Citation55; Fenwick et al (2004)^Citation56; Groot Koerkamp et al (2007)^Citation57; Claxton (2008)^Citation38; Groot Koerkamp et al (2010)^Citation42
8. General
Methods to achieve transparency, selecting time horizons, subgroup analysis	Sculpher (2008)^Citation40; Eddy (2006)^Citation62; Cooper (2007)^Citation20

Table 4 Validators’ feedback on understanding, usefulness, and timing of the steps in the beginner’s guide

Question	V1		V2
	n	(%)	n	(%)
Total steps	156	100	156	100
Did you understand this step?
Yes	125	80	105	67
No	9	6	1	1
Missing	22	14	50	32
Missing values – input mean values for repeated steps*	10	6	38	24
Rate the usefulness of this step on a scale of 1 to 3 (1 = not useful; 2= useful; 3= very useful)
3= very useful	92	59	102	65
2= useful	19	12	3	2
1= not useful	4	3	0	0
Missing values treated as missing	41	26	51	33
Missing values – input mean values for repeated steps*	29	19	39	25
Did you perform the step here (or at another stage in the modeling process)?
Yes	103	66	91	58
No	27	17	17	11
Missing values treated as missing	26	17	48	31
Missing values – input mean values for repeated steps*	14	9	30	19

Note:

* Mean entries have been inserted for literature search, source, selection, and evidence grading steps, which were rated as done at that time point by both validators and useful by V1 and very useful by V2.

Abbreviations: V1, validator 1; V2, validator 2.

BriggsAHHandling uncertainty in cost-effectiveness modelsPharmacoeconomics200017547950010977389

 PubMed Web of Science ®Google Scholar

SonnenbergFARobertsMSTsevatJWongJBBarryMKentDLToward a peer review process for medical decision analysis modelsMed Care1994327 SupplJS52JS648028413

 PubMed Web of Science ®Google Scholar

StahlJEModelling methods for pharmacoeconomics and health technology assessment: an overview and guidePharmacoeconomics200826213114818198933

 PubMed Web of Science ®Google Scholar

SargentRVerification and validation of simulation modelsProceedings of the 2010 Winter Simulation Conference5–8 December, 2010Baltimore, Maryland, USA Available from: http://student.telum.ru/images/6/66/Sargent_VV_2010.pdfAccessed January 15, 2011

 Google Scholar

ChilcottJTappendenPRawdinAAvoiding and identifying errors in health technology assessment models: qualitative study and methodological reviewHealth Technol Assess20101425iiixii1107

 Web of Science ®Google Scholar

TappendenPConceptual modelling for health economic model developmentHEDS discussion paper 12.05 Available from: http://www.shef.ac.uk/polopoly_fs/1.172572!/file/HEDSDP1205.pdfAccessed September 9, 2016

 Google Scholar

BoothAHow much searching is enough? Comprehensive versus optimal retrieval for technology assessmentsInt J Technol Assess Health Care201026443143520923586

 PubMed Web of Science ®Google Scholar

PaisleySClassification of evidence in decision-analytic models of cost-effectiveness: a content analysis of published reportsInt J Technol Assess Health Care201026445846220923588

 PubMed Web of Science ®Google Scholar

BraithwaiteRSRobertsMSJusticeACIncorporating quality of evidence into decision analytic modelingAnn Intern Med2007146213314117227937

 PubMed Web of Science ®Google Scholar

NuijtenMJThe selection of data sources for use in modelling studiesPharmacoeconomics199813330531610178656

 PubMed Web of Science ®Google Scholar

CooperNJSuttonAJAdesAEPaisleySJonesDRUse of evidence in economic decision models: practical issues and methodological challengesHealth Econ200716121277128618034447

 PubMed Web of Science ®Google Scholar

BartonPBryanSRobinsonSModelling in the economic evaluation of health care: selecting the appropriate approachJ Health Serv Res Policy20049211011815099459

 PubMedGoogle Scholar

CooperKBrailsfordSCDaviesRChoice of modelling technique for evaluating health care interventionsJ Oper Res Soc2007582168176

 Web of Science ®Google Scholar

BrennanAChickSEDaviesRA taxonomy of model structures for economic evaluation of health technologiesHealth Econ200615121295131016941543

 PubMed Web of Science ®Google Scholar

MinersAEstimating ‘costs’ for cost-effectiveness analysisPharma-coeconomics2008269745751

 Web of Science ®Google Scholar

HayJWSmeedingJCarrollNVGood research practices for measuring drug costs in cost effectiveness analyses: issues and recommendations: the ISPOR Drug Cost Task Force report – Part IValue Health20101313719874571

 PubMed Web of Science ®Google Scholar

ShiLHodgesMDrummondMGood research practices for measuring drug costs in cost-effectiveness analyses: an international perspective: the ISPOR Drug Cost Task Force report – Part VIValue Health2010131283319883403

 PubMed Web of Science ®Google Scholar

BrouwerWBNiessenLWPostmaMJRuttenFFNeed for differential discounting of costs and health effects in cost effectiveness analysesBMJ2005331751444644816110075

 PubMed Web of Science ®Google Scholar

ClaxtonKSculpherMCulyerADiscounting and cost-effectiveness in NICE – stepping back to sort out a confusionHealth Econ20061511416365910

 PubMed Web of Science ®Google Scholar

ClaxtonKPauldenMGravelleHBrouwerWCulyerAJDiscounting and decision making in the economic evaluation of health-care technologiesHealth Econ201120121521154521

 PubMed Web of Science ®Google Scholar

GravelleHBrouwerWNiessenLPostmaMRuttenFDiscounting in economic evaluations: stepping forward towards optimal decision rulesHealth Econ200716330731717006970

 PubMed Web of Science ®Google Scholar

NordEDiscounting future health benefits: the poverty of consistency argumentsHealth Econ2011201162621154522

 PubMed Web of Science ®Google Scholar

GrayAMClarkePMWolstenholmeJWordsworthSApplied Methods of Cost-effectiveness Analysis in HealthcareUSAOxford University Press2010

 Google Scholar

CraigDMcDaidCFonsecaTStockCDuffySWoolacottNAre adverse effects incorporated in economic models? An initial review of current practiceHealth Technol Assess2009136217197181iii

 Web of Science ®Google Scholar

BrazierJValuing health States for use in cost-effectiveness analysisPharmacoeconomics200826976977918767897

 PubMed Web of Science ®Google Scholar

NeumannPJStonePWChapmanRHSandbergEABellCMThe quality of reporting in published cost-utility analyses, 1976–1997Ann Intern Med20001321296497210858180

 PubMed Web of Science ®Google Scholar

GoldMRStevensonDFrybackDGHALYS and QALYS and DALYS, Oh My: similarities and differences in summary measures of population HealthAnnu Rev Public Health2002231153411910057

 PubMed Web of Science ®Google Scholar

McDonoughCMTostesonANMeasuring preferences for cost-utility analysis: how choice of method may influence decision-makingPharmacoeconomics20072529310617249853

 PubMed Web of Science ®Google Scholar

BriggsAHGrayAMHandling uncertainty when performing economic evaluation of healthcare interventionsHealth Technol Assess1999321134

 PubMedGoogle Scholar

BriggsAHGrayAMHandling uncertainty in economic evaluations of healthcare interventionsBMJ1999319721063563810473486

 PubMed Web of Science ®Google Scholar

ClaxtonKExploring uncertainty in cost-effectiveness analysisPharmacoeconomics200826978179818767898

 PubMed Web of Science ®Google Scholar

Groot KoerkampBHuninkMGStijnenTHammittJKKuntzKMWeinsteinMCLimitations of acceptability curves for presenting uncertainty in cost-effectiveness analysisMed Decis Making200727210111117409361

 PubMed Web of Science ®Google Scholar

SculpherMSubgroups and heterogeneity in cost-effectiveness analysisPharmacoeconomics200826979980618767899

 PubMed Web of Science ®Google Scholar

AndronisLBartonPBryanSSensitivity analysis in economic evaluation: an audit of NICE current practice and a review of its use and value in decision-makingHealth Technol Assess20091329iiiixxi161

 Web of Science ®Google Scholar

LimwattananonSHandling uncertainty of the economic evaluation result: sensitivity analysisJ Med Assoc Thai200891Suppl 2S596519253488

 PubMedGoogle Scholar

BojkeLClaxtonKPalmerSSculpherMDefining and characterising structural uncertainty in decision analytic models2006 Available from: http://www.york.ac.uk/che/pdf/rp9.pdfAccessed August 8, 2011

 Google Scholar

StrongMManaging Structural Uncertainty in Health Economic Decision Models Available from: http://etheses.whiterose.ac.uk/2205/Accessed June 26, 2012

 Google Scholar

BrissonMEdmundsWJImpact of model, methodological, and parameter uncertainty in the economic analysis of vaccination programsMed Decis Making200626543444616997923

 PubMed Web of Science ®Google Scholar

GrootKBWeinsteinMCStijnenTHeijenbrok-KalMHHuninkMGUncertainty and patient heterogeneity in medical decision modelsMed Decis Making201030219420520190188

 PubMed Web of Science ®Google Scholar

SchlesingerSTerminology for model credibilitySimulation197932103

 Google Scholar

McCabeCDixonSTesting the validity of cost-effectiveness modelsPharmacoeconomics20001750151310977390

 PubMed Web of Science ®Google Scholar

KimLGThompsonSGUncertainty and validation of health economic decision modelsHealth Econ2010191435519206080

 PubMed Web of Science ®Google Scholar

SendiPPCraigBAPflugerDGafniABucherHCSystematic validation of disease models for pharmacoeconomic evaluations. Swiss HIV Cohort StudyJ Eval Clin Pract19995328329510461580

 PubMed Web of Science ®Google Scholar

HalpernMTLuceBRBrownREGenesteBHealth and economic outcomes modeling practices: a suggested frameworkValue in Health1998113114716674361

 PubMedGoogle Scholar

WeinsteinMCToyELSandbergEAModeling for health care and other policy decisions: uses, roles, and validityValue Health20014534836111705125

 PubMedGoogle Scholar

WeinsteinMCO’BrienBHornbergerJJacksonJJohannessonMMcCabeCLuceBRPrinciples of good practice for decision analytic modeling in health-care evaluation: Report of the ISPOR Task Force on Good Research Practices – modeling studiesValue Health20036191712535234

 PubMed Web of Science ®Google Scholar

BlackWCThe CE plane: a graphic representation of cost-effectivenessMed Decis Making19901032122142115096

 PubMed Web of Science ®Google Scholar

BriggsAHWonderlingDEMooneyCZPulling cost-effectiveness analysis up by its bootstraps: a non-parametric approach to confidence interval estimationHealth Econ1997643273409285227

 PubMed Web of Science ®Google Scholar

FanM-YZhouX-HA simulation study to compare methods for constructing confidence intervals for the incremental cost-effectiveness ratioHealth Serv Outcomes Res Method200771–25777

 Google Scholar

DinhPZhouXHNonparametric statistical methods for cost-effectiveness analysesBiometrics200662257658816918923

 PubMed Web of Science ®Google Scholar

PolskyDGlickHAWillkeRSchulmanKConfidence intervals for cost-effectiveness ratios: a comparison of four methodsHealth Econ1997632432529226142

 PubMed Web of Science ®Google Scholar

BartonGRBriggsAHFenwickEAOptimal cost-effectiveness decisions: the role of the cost-effectiveness acceptability curve (CEAC), the cost-effectiveness acceptability frontier (CEAF), and the expected value of perfection information (EVPI)Value Health200811588689718489513

 PubMed Web of Science ®Google Scholar

FenwickEBriggsACost-effectiveness acceptability curves in the dock: case not proven?Med Decis Making2007272939517409358

 PubMed Web of Science ®Google Scholar

FenwickEClaxtonKSculpherMRepresenting uncertainty: the role of cost-effectiveness acceptability curvesHealth Econ200110877978711747057

 PubMed Web of Science ®Google Scholar

FenwickEO’BrienBJBriggsACost-effectiveness acceptability curves – facts, fallacies and frequently asked questionsHealth Econ200413540541515127421

 PubMed Web of Science ®Google Scholar

EddyDMAccuracy versus transparency in pharmacoeconomic modelling: finding the right balancePharmacoeconomics200624983784416942119

 PubMed Web of Science ®Google Scholar