How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3)

Figures & data

Figure 1. Overview of 5-step approach for preparing a systematic review of economic evaluations of healthcare interventions.

Table 1. Items and explanation for the data extraction of economic evidence.

Number	Type of items	Explanation
	Checklist completed by	State the name of person who has filled out the data extraction sheet.
	General study characteristics
1	First author and year of publication	Report: First author, title, journal name, publication date, volume, issue, page numbers, and link to the publication.
2	Sources of funding	Report: The source of funding cited in the article: write ‘stated’ or ‘not stated’ and specify. If any, give name(s) of organization or corporation. Specify if possible the source type (public research funds, NGO, government, academic/university, health-care industry, or other).
3	Competing interests	Competing interests: write “stated” or “not stated” and specify if any.
4	Publication type	Describe type of publication^b (journal paper and HTA report)
5	Setting	List the country/countries, setting, and/or locations for economic evaluation.
6	Patient characteristics	Summarize inclusion and exclusion criteria (eligibility criteria/demographics).
7	Type of intervention	Describe the experimental treatment (service, program).
8	Control treatment	Describe the control treatment (service, program).
9	Eligibility criteria	Describe the eligible population and the population used for effect/cost data.
10	Study perspective	State the viewpoint of the analysis (society, health care, insurer, care provider, patient, and family).
11	Type of EEs	Specify the form of economic evaluation being used (e.g. CEA, CUA, CBA, and CCA).
12	Analytic approach	Describe the analytic approach: trial based or model based.
	Methods and outcomes of economic evaluations
13	Time frame of the analysis (time horizon)	State the time horizon for both costs and benefits.
14	Discount rate	Was discounting performed?
15	Discount rate for costs	What was the discount rate for the costs(s)?
16	Discount rate for effects	What was the discount rate for the effects(s)? (i.e. the rate used to account for different timing of costs and effects)
17	Inflation rate	Was adjustment for inflation performed if unit costs stemmed from different years?
18	Reference year	What was the reference year of the analysis?
19	If model based	Detail any model used (Markov, Decision Tree, and Discrete Event Simulation)^c
20	Type and category of costs	Describe the different cost types and categories used (e.g. direct in health care, indirect health care, and intangible costs).
21	Data source of resource use	Describe the data for resource use (e.g. clinical trials, administrative data, clinical databases, medical records, and published literature).
22	Methods for identifying resource use	Describe the methods used to identify resource use (questionnaire, survey, cost dairies, expert consultation, and formal consensus methods).
	Assumptions of the measurement of resources	Describe, for instance, method of imputation when incomplete measurement occurred.
22	Costs (in reported currency or in converted currency)	Present relevant costs and outcomes in both disaggregated and aggregated form (with confidence intervals and measures of significance).
24	Methods used to calculate unit costs	Describe the methods used to identify relevant unit costs (guidelines, own cost price calculations, and literature).
25	Costs^a	Present relevant costs in disaggregated and aggregated form (with confidence intervals and measures of significance).
27	Data source of effects	Specify where utilities or benefits came from (literature values and elicited in the study).
28	Methods of measurement of effects	Specify source of effectiveness estimates (stated WTP, revealed WTP, and conjoint analysis).
29	Methods of valuation of effects	Specify methods of valuation of effects (indirect or direct measurement).
30	Effects^a	Present relevant effects (utilities, [health] benefits, and outcomes) in disaggregated and aggregated form (with confidence intervals and measures of significance).
31	Incremental cost–effectiveness ratios	State the summary measure of benefit for CUA (e.g. QALY or DALY).
32	Analyses of uncertainty (e.g. sensitivity analyses)	Describe the analyses of uncertainty (e.g. statistical comparison, bootstrapping, sensitivity analysis [one-way, multiway], threshold analysis, analysis of extremes, and best/worst case analysis) and probabilistic sensitivity analysis.
34	Outcome(s) of analyses of sensitivity analyses
35	Authors’ conclusions	Report the conclusions of the authors

^aWhen reporting the study outcomes, it is preferred to report the degree of uncertainty; therefore, in addition to reporting the mean (or median), a standard deviation (or range) should be reported [5].

^bDuplicate publications of the same study need to be linked together.

^cCheck for details: Philips et al. [21].

CEA: cost–effectiveness analyses; CUA: cost–utility analyses; CCA: cost–consequence analyses; CBA: cost–benefit analyses; EEs: economic evaluations; QALY: quality-adjusted life year; NOG: Non-governmental organization; HTA: Health Technology Assessment; WTP: Willingness to pay; DALY: Disability adjusted life years.

Table 2. Example of data extraction for multipurpose review: general, RCT-related, and economic characteristics.

Author	Year	Disease	N	Intervention	Trial- or model-based EE	Comparator(s)	Outcome measure(s)	Perspective	Intervention cost-effective
De Kinderen et al.	2016	Refractory epilepsy	48	Ketogenic diet	Trial-based EE	Care as usual	QALY gain	Health-care perspective	No, based on cost per QALY
Etc.
Etc.

EE: economic evaluations; QALY: quality-adjusted life year; RCT: randomized controlled trial.

Based on De Kinderen et al. 2016.

Table 3. Example of data extraction for multipurpose review: cost–effectiveness results.

Author	Year	Disease	Main outcome	ICER/ICUR
De Kinderen et al.	2016	Refractory epilepsy	The results show that the KD reduces seizure frequency. The study did not find any improvements in quality of life and, therefore, unfavorable cost per QALY ratios resulted.	1) CEA: €18,422.21 per responder; 2) CUA: not presented but 5% probability of being cost-effective at the ceiling ratio of €50,000 per QALY
Etc.
Etc.

KD: ketogenic diet; CAU: care as usual; ICER: incremental cost–effectiveness ratio; ICUR: incremental cost–utility ratio; QALY: quality-adjusted life year; CEA: cost–effectiveness analyses; CUA: Cost-utility analysis; RCT: randomized controlled trial.

Based on De Kinderen et al. 2016.

Table 4. Example of critical appraisal of the quality of the economic evaluation using the CHEC checklist.

	CHEC-extended items (Evers et al. 2005; Odnoletkova et al. 2014)	De Kinderen et al. (2015)
1	Is the study population clearly described?	Yes (see in method section, study population and sample size section, and result section; Table 2)
2	Are competing alternatives clearly described?	Yes, sufficient details provided on usual care group vs. ketogenic diet group.
3	Is a well-defined research question posed in answerable form?	Yes, last sentence of the introduction.
4	Is the economic study design appropriate to the stated objective?	Yes, an appropriate study design is used. The economic study design is a full economic evaluation (comparison of costs and effects of two interventions).
5	Are the structural assumptions and the validation methods of the model properly reported?	NA (trial-based EE)
6	Is the chosen time horizon appropriate in order to include relevant costs and consequences?	No, a 3-month time horizon is too short; for a societal perspective, a time frame of at least 1 year is generally accepted.
7	Is the actual perspective chosen appropriate?	Yes, the societal perspective is preferred in The Netherlands.
8	Are all important and relevant costs for each alternative identified?	Yes (see method section: protocol-driven intervention costs, health-care costs, patient and family costs, and productivity costs are included.
9	Are all costs measured appropriately in physical units?	Yes, cost diaries were used.
10	Are costs valued appropriately?	Yes, sources of valuation including the reference year mentioned.
11	Are all important and relevant outcomes for each alternative identified?	Yes, outcomes are relevant, and they fit the research question and perspective.
12	Are all outcomes measured appropriately?	Yes, measurement instruments are relevant and described.
13	Are outcomes valued appropriately?	Yes (see method section) EQ-5D using a Dutch utility-tariff was used.
14	Is an appropriate incremental analysis of costs and outcomes of alternatives performed?	Yes, ICER/ICUR is calculated.
15	Are all future costs and outcomes discounted appropriately?	NA when using a time frame for analyses of less than 1 year, discounting is not needed.
16	Are all important variables, whose values are uncertain, appropriately subjected to sensitivity analysis?	Suboptimal, not all parameters.
17	Do the conclusions follow from the data reported?	Yes, conclusions are supported by the data.
18	Does the study discuss the generalizability of the results to other settings and patient/client groups?	Yes, mentioned and (briefly) discussed in the discussion section of the article.
19	Does the article/report indicate that there is no potential conflict of interest of study researcher(s) and funder(s)?	Yes, the authors report having no conflict of interest.
20	Are ethical and distributional issues discussed appropriately?	Yes, the study was approved by the Medical Ethics Committee of Maastricht University.

Based on De Kinderen et al. 2016.

NA: not applicable; EE: economic evaluation; ICER: incremental cost–effectiveness ratio; ICUR: incremental cost–utility ratio; CHEC: consensus on health economics criteria checklist; EQ-5D: EuroQol 5-dimensions.

Table 5. Example using the Welte checklist to determine the transferability of the study of the economic evaluation of the ketogenic diet from The Netherlands to the UK setting.

			Correspondence between study (The Netherlands) and decision country (UK)
General knockout criteria
	1. The evaluated technology is not comparable to the one that shall be used in the decision country.		NA	Passed
	2. The comparator is not comparable to the one that is relevant to the decision country.		NA	Passed
	3. The study does not possess an acceptable quality.		NA	Passed
			Correspondence between study (The Netherlands) and decision country (UK)	ICER of decision country based on ICER of study country is:
Methodological characteristics
	Perspective		Low (health care vs. societal)	Too low or too high
	Discount rate		Medium (3.5% UK vs. 4% NL)	Unbiased (short time horizon)
	Medical cost approach		High	Unbiased
	Productivity cost approach		Low (friction costs method in NL vs. no productivity costs in UK	Too low or too high
Health-care system characteristics
	Absolute and relative prices in health care		High	Unbiased
	Practice variation		Medium	Too low or too high
	Technology availability		High	Unbiased
Population characteristics
	Disease incidence/prevalence		High	Unbiased
	Case-mix		High	Unbiased
	Life expectancy		High	Unbiased
	Health-status preferences		High	Unbiased
	Acceptance, compliance, and incentives to patients		High	Unbiased
	Productivity and work-loss time		High	Unbiased
	Disease spread		High	Unbiased

NA: not applicable; ICER: Incremental cost–effectiveness ratio; NL: The Netherlands; UK: United Kingdom.

Based on De Kinderen et al. 2016.

Table 6. Recommendations on data extraction, risk of bias, and transferability for a systematic review of EEs for SR-EEs for multipurpose and CPG development.

Step 3: Data extraction, risk of bias, and transferability
Step 3.1: Data extraction for SR-EEs for multipurpose and CPG development
Data extraction sheet	Adapt the data extraction for every specific study; include all relevant items from the list in Table 1.
	Include the selected risk of bias and transferability checklists in the data extraction sheet.
Data extraction	It is recommended to use a picklist to choose the different answers. Describe items to increase transparency of scoring.
	Disaggregated presentation of the results is highly recommended, as well as incremental cost–effectiveness ratios, in order to facilitate the interpretation of the results.
Step 3.2: Risk of bias assessment
Risk of bias assessment for multipurpose SR-EEs and CPG development	Use CHEC-extended [24] or BMJ checklists [23].
	In cases where one is specifically interested in model-based EE and if the expected number of included studies is low (e.g. <10 studies; pragmatic decision), the Philips checklist [18] could be used.
	In cases in which the number of included model-based EEs is high (e.g. >10 studies; pragmatic decision), the ISPOR checklist is likely to be more practical for reviewing purposes [19].
Risk of bias assessment for CPG development	In general, full EEs should be preferred over partial EEs. However, in CPG development, partial EEs can be used to examine financial implications or the budget impact of a treatment.
	In the absence of a full EE, partial EEs may represent important intermediate stages in our understanding of the costs and consequences of health service programs and therefore might be convenient.
	Whenever possible, when developing CPGs, the GRADE approach should be used to systematically incorporate economic evidence of (partial) EE. In addition, it is important to perform a complementary assessment on study applicability and limitations using the NICE checklist [27].
Step 3.3: Transferability for SR-EEs for multipurpose and CPG development
Transferability	Use the Welte checklist.
	It is important to discuss the transferability of a particular study with clinicians as clinical practice might vary between countries.
Usability for SR-EEs for multipurpose and CPG development
Appraisal process	Raters are a relevant source of variability.
	It is important to have multiple raters (at least two) so that discrepancies can be resolved through consensus meetings.
	It is recommended that a few studies (i.e. two or three) should be used to pilot the assessment between multiple raters, after which discrepancies should be discussed to ensure a more uniform assessment strategy.

EEs: economic evaluations; SR: systematic review; CPG: clinical practice guideline; ISPOR: International Society of Pharmacoeconomics and Outcomes Research; GRADE: Grading of Recommendations Assessment, Development, and Evaluation; NICE: National Institute for Health and Care Excellence; BMJ: Britisch medical journal; CHEC: Consensus on health economics criteria checklist.

Drummond, M.F., Sculpher, M.J., Claxton, K, et al. Methods for the economic evaluation of health care programmes. Oxford: Oxford university press; 2015.

 Google Scholar

Philips, Z, Ginnelly L, Sculpher M, et al. Good practice guidelines for decision-analytic modelling in health technology assessment. Pharmacoeconomics. 2006;24(4):355–371.

 PubMed Web of Science ®Google Scholar

Evers S, Goossens M, De Vet H, et al. Criteria list for assessment of methodological quality of economic evaluations: consensus on health economic criteria. Int J Technol Assess Health Care. 2005;21(2):240–245.

 PubMed Web of Science ®Google Scholar

Drummond MF, Jefferson T. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ. 1996;313(7052):275–283.

 PubMed Web of Science ®Google Scholar

De Kinderen RJ, Lambrechts DAJE, Wijnen BFM, et al. An economic evaluation of the ketogenic diet versus care as usual in children and adolescents with intractable epilepsy: an interim analysis. Epilepsia. 2016;57(1):41–50.

 PubMed Web of Science ®Google Scholar

Walker, D.G., Wilson, R.F, Sharma, R, et al. Best practices for conducting economic evaluations in health care: a systematic review of quality assessment tools. The Johns Hopkins University Evidence-based Practice Center. Rockville (MD): Agency for Healthcare Research and Quality (US); 2012.

 Google Scholar

Excellence, N.i.f.C. Developing NICE guidelines: the manual; 2014; Available from: http://www.nice.org.uk/article/PMG20/chapter/7-Incorporating-economic-evaluation

 Google Scholar