How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: a five-step approach (part 1/3)

Figures & data

Figure 1. An overview of the 5-step approach for preparing a systematic review of economic evaluations for Clinical Practice Guidelines.

*Described in detail in paper by Thielen et al. [18], **Described in detail in paper by Wijnen et al. [19].

Table 1. Defining eligibility criteria, based on a real example of an SR-EEs for guideline development.

Inclusion criteria	Example	Explanation [13]
Population or patients’ characteristics^a	Adult patients with drug-resistant epilepsy medication	1) Reflect on the target audience of the intervention for which cost-effectiveness is being examined in the SR. 2) Is the entire world/international population to be considered or only the population (or subpopulation) of a particular country?
Intervention^a	Epilepsy surgery	What is the intervention and the comparator(s) for which the economic evidence is to be sought?
Comparator or Control treatment^a	Any type of antiepileptic drugs Placebo
Setting, country, or jurisdiction	All countries All settings	What are the health setting(s) to be considered; this should be specified. For example, primary health care, community-based, or hospital-based.
Outcomes^a	Incremental cost-effectiveness ratio (ICER)	Specify the outcomes which will be considered in the SR
Design of study (Types of EEs)	Two types of full EEs (CUA, CEA), Model-based or trial-based EEs	1) Specify the type of EE research design(s) to be considered. 2) Specify whether the review will focus on model-based or trial-based studies or both.
Exclusion criteria	Example	Explanation [13]
		If applicable, exclusion criteria should focus on excluding on the basis of a clear policy or scientific justification rather than on an unsubstantiated personal or clinical justification.
Types of EEs	Partial EEs	Specify which type of EEs will be excluded from the SR. Based on the inclusion criteria, only full EEs will be assessed.
Language restrictions	None	It is not recommended to use any restrictions regarding publication language [12,36].
Time frame of the analysis (time horizon)	Short term (<12 months)	Specify the time frame of the EE analysis being considered. In our review, studies reporting on a follow-up period of less than 12 months, thus having a short analytical time frame, were excluded
Publication type	Systematic reviews, Commentaries (letters to the editors, editorials) Congress abstracts	Selection of studies can also be based on publication type; it is common practice to include only original studies in an SR-EEs.
Design	Animal studies

^aPICO: P (population), I (intervention or comparator), C (context/setting), O (Outcomes) [37].

Table 2. Topics related to reporting multipurpose SR-EEs.

Topics	Description
Summary of evidence^a,b	Summarize the main findings including the strength of evidence for each main outcome; consider their relevance to key groups (e.g., health-care providers, users, and policy makers).
Heterogeneity^b	A consequence of clinical or methodological diversity, or both, among the studies [12]
Transferability	The ability to extrapolate results obtained from one setting or context to another [75–77]
Generalizability^b	The extent to which (both clinical and economic) results can be extrapolated to either a patient group with different characteristics or to similar patient group treated in a different geographic, political, or time structure [75,83]. The basic knowledge factors which affect the generalizability of EEs include population aspects (e.g. age, gender, education), disease characteristics (for instance, incidence and prevalence), care provider and system factors (e.g. variations in practice, wages of health professionals) [28,29].
Implementation problems/budget impact	Discuss possible barriers to implementation. Are there any expected changes in the expenditure of a health-care system after the adoption of a new intervention (Budget impact) [23].
Strengths	Discuss strengths of the study and outcome level, also at review level
Limitations^a	Discuss limitations at study and outcome level (e.g. risk of bias), and at review level (e.g. incomplete retrieval of identified research, reporting bias).
Recommendations	What are the knowledge gaps in relation to the investigated topic and what are therefore the recommendations for future research?
Update of the review^c	A statement that the guideline will be updated. State the explicit time interval or explicit criteria to guide decisions about when an update will occur (e.g. take into account identified ongoing studies or expected changes in treatment regimens); methodology for the updating procedure must be reported
Conclusions^a	Provide a general interpretation of the results in the context of other evidence, and implications for future research.
Source of funding^b	Describe how the study was funded and the role of the funding party in the identification, design, conduct, and reporting of the analysis. Describe other nonmonetary sources of support.
Conflict of interest^b	Describe any potential for conflict of interest among study contributors in accordance with journal policy. In the absence of a journal policy, we recommend that authors comply with International Committee of Medical Journal Editors’ recommendations.

^a Items of the PRISMA statement [2].

^b Items of the CHEERS [29].

^c AGREE II [16].

Table 3. Decision table to support the development of recommendations for SR-EEs for guideline development, including an example of a study considered for the Dutch National Guideline for epilepsy.

Study (author, year)	de Kinderen et al. 2016 [84]
Intervention	Ketogenic diet
Population	Children and adolescents with intractable epilepsy
Factor	Decision	Comments	Explanation
Based on the quality appraisal, what is quality of the EE?^^	High quality^a Moderate quality^b Low quality^c Very Low quality^d	High quality	High score on CHEC-extended Number of yes answers: 17 out of 20, number of no answers: 2, number of Not applicable answers: 1, number of sub optimal answers: 1
Is the new intervention cost-effective?	Yes or No	No	Based on the results of an interim analysis of 4 months of data
Are the outcomes of the EE uncertain and/or variable?^^	Acceptable variability/uncertainty High variability/uncertainty	High variability/uncertainty	Based on a 4-month follow-up (interim analyses) and population of 47 children
Balance between health benefits, side effects, and risk^^	Health benefits clearly outweigh side effects, and risk. Health benefits, side effects, and risk are balanced Side effects, and risk clearly outweigh health benefits	No	Not reported
Transferability of the study results	Transferable Probably transferable Not transferable	Transferable	This study was performed in the Netherlands, so no specific analyses of transferability are necessary.
Generalizability of the study results.	Generalizable Probably generalizable Not generalizable	Generalizable	Patients recruited form one of the two specialist tertiary epilepsy centers; the study population is representative for Dutch population.
Implementation problems	Unlikely Likely No implementation problems	Likely	Specialist knowledge of ketogenic diet and treatment of epilepsy is needed for healthcare professionals.
Overall conclusion of the findings	A high quality Dutch trial-based EE showing nondominance of the intervention in comparison with care as usual. Findings are on based on interim analysis at 4 months.
Overall strength of evidence^^	Not enough evidence
Research gaps	Longer follow-up time is needed, >12 months

^aHigh quality: further research is very unlikely to change confidence in the estimate of effect; ^bModerate quality further research is likely to have an important impact on confidence and is likely to change the estimate of effect; ^cLow quality; further research is very likely to have an important impact on confidence and is likely to change the estimate of effect; ^dVery low quality; any estimate of effect is very uncertain. ^[26].

Table 4. Summary of all recommendations for all five steps of the review process for both multipurpose SR-EEs and SR-EEs for CPG development.

Type of review	General recommendations
Both review types	Methods for SR-EEs		Follow the 5-step approach.
	Basic knowledge of EEs		For basic knowledge of EE methods look at Drummond et al. [20] Full EEs are the preferred type of EEs for both multipurpose SR-EEs and SR-EEs for CPG development [16].
	Country-specific guidelines for EEs		Check ISPOR website for country-specific EE guidelines [8].
SR-EEs for CPG development	Basic knowledge on CPG development		Obtain basic knowledge on guideline development [11,14,25,26]. Use AGREE II tool for CPG development [16].
	Country-specific regulations on CPG development		Check global or country-specific regulations for developing guidelines as stated in Appendix 1 of this paper.
Both review types	Minimize biases		Ideally, all steps critical for study selection (2.3 and 2.4) but also those for data extraction and appraisal (3.1, 3.2 and 3.3) should be performed independently by two reviewers.
	Reporting of SR-EEs		Use PRISMA statement for SRs [2]. Check CHEERS for reporting EEs [29].
	Step 1: Initiating a SR-EEs
Multipurpose SR-EEs	1.1 Compose project team		Compose a multidisciplinary project team to include the following expertise: clinical, SR methods, quantitative methods and health technology assessment methods, and library and information science.
SR-EEs for CPG development	1.1 Compose project team		Compose a multidisciplinary project team to include the following expertise: clinical, SR methods, quantitative methods and health technology assessment methods, CPG development, library and information science, and patient and public views.
Both review types	1.1 Manage any conflicts of interests		When preparing SR-EEs, conflicts of interests should be handled in an appropriate way.
Multipurpose SR-EEs	1.2 Identify and define a relevant topic or research questions		Perform a scoping review to identify the most relevant research questions.
SR-EEs for CPG development	1.2 Identify and define a relevant topic or research questions		Consult different stakeholders and perform a scoping review to identify the most relevant research questions. Start the SR-EE after the effectiveness SR is finished.
Both review types	1.3 Write a protocol of SR-EEs		Write a protocol of the SR-EEs by using the PRISMA-P checklist [27].
Both review types	1.4 Publish protocol of SR-EEs		Publish the protocol on the PROSPERO website [32] or, if there is a Cochrane review, on the Cochrane Website [33].
	Step 2: Identifying full EEs
Both review types	2.1 Select relevant data sources	Use list of databases to select those relevant for SR-EEs (see Appendix Thielen et al. [18].
		General databases	Select at least Medline (freely accessible on the internet via PubMed), Embase, Econlit, and Web of Science.
			Additional searches can be performed in NHS EED (although this has not been updated since March 2015).
		Specific and optional databases	Select specific databases according to your topic (if applicable).
			Search optional database(s) and websites for HTA reports and conference proceedings.
		Grey literature	Consider including grey literature.
		Citation searching	Search in known publications for relevant citations.
			Make use of citation searching (i.e. identify articles that have cited a set of relevant articles); use Web of Science or Google scholar for this.
Both review types	2.2 Development of search strategies	Search terms	Make use of the PICO (Patient, Intervention, Comparator, Outcome) scheme to find relevant search terms for all important concepts/aspects of the research question.
			Include a wide range of free-text terms.
			Use proximity operators (e.g. NEAR) if possible.
			Employ thesauri and synonyms.
			Use truncation options for your search term.
			For English, use British or American spelling.
		Search filters	Determine whether you want to use a more sensitive or precise search filter. SRs will profit from sensitive filters because precise filters will miss some articles.
			Look for search filters that filter for publication types (e.g. economic or trial publications). Choose validated filters as much as possible. Check The InterTASC Information Specialists’ Sub-Group website for validated search filters [85]. Appendices of Cochrane SRs or other high quality SRs are also good sources for filters.
		Combine search terms and filters with Boolean (AND, OR, NOT) operators	Carefully consider on what, and if at all, you want to restrict your search results. It is not recommended to restrict on language or to choose too narrow a time frame.
Both review types	2.3 Handling searches	Document the search process	Document and report all steps of the search, including the complete search strategy for every database.
		Handle references	Use bibliographic software to keep track of downloaded references and publications.
			De-duplicate the downloaded records by using a reference management software program.
Both review types	2.4 Selection of studies	Screen references	Ideally, two reviewers should screen the references independently.
			Screen titles and abstracts of the downloaded studies based on the eligibility criteria that were set in the protocol.
	Step 3: Data extraction, risk of bias assessment and transferability
Both review types	3.1 Data extraction	Adapt the data extraction sheet for every specific study; include all relevant items from the list in Table 1 in paper by Wijnen et al. [19].
		Include the selected risk of bias and transferability checklists in the data extraction sheet.
		It is recommended that a picklist be used to choose the different response options.
		It is recommended that a few studies (i.e. two or three) should be used to pilot the assessment among multiple raters, after which discrepancies should be discussed to ensure a more uniform assessment strategy.
	3.2 Risk of bias assessment
Both review types		For trial-based and model-based EEs use the CHEC-extended [67–69] or BMJ checklist [86].
		In cases where one is specifically interested in model-based EEs and if the expected number of included studies is low (e.g. <10 studies: pragmatic decision), the Phillips checklist [71] could be used.
		In cases in which the number of included model-based EEs is high (e.g. >10 studies; pragmatic decision), the ISPOR checklist is likely to be more practical for reviewing purposes [66].
SR-EEs for CPG development		Full EEs should be preferred over partial EEs at all times.
		In the absence of full EEs, partial EEs may represent important intermediate stages in our understanding of the costs and consequences of health services programs and therefore might be convenient.
		For trial-based EEs use the GRADE approach [8]. For model-based use EEs use the NICE checklist [11].
Both review types	3.3 Transferability assessment	The Welte checklist [55] is recommended for all trial-based and model-based EEs.
	Step 4: Reporting of results
Both review types	4.1 Results section and data synthesis	Convert all different currencies reported to a common currency (e.g. US dollar, Euro) and use the same year as a reference [80].
		Graphic presentation of the data can be used if common metrics (e.g. costs and QALYs) for each study are applied [13,83].
		Due to the methodological and study-specific heterogeneity issues of EEs, a meta-analysis is not recommended.
SR-EEs for CPG development		For trial-based EEs: use GRADE evidence profiles and Summary of Findings tables [87]. For model-based EEs: see recommendations for multipurpose SR-EEs.
Multipurpose SR-EEs		For multipurpose SR-EEs (trial-based and model-based): the findings of step 3 can be presented in self-developed summary tables and also summarized in the text.
	Step 5: Discussion and interpretation of results
Both review types	5.1 Discussion section	General topics for SR-EEs: summary of evidence, heterogeneity, study limitations, study strengths, time frame for update of review, previous SR-EE findings in relation to current SR-EE findings, general conclusions, recommendations for further research, source of funding and conflict of interest.
		Specific topics for SR-EEs: transferability, generalizability and implementation problems or budget impact.
SR-EEs for CPG development	5.2.1 Recommendations; Discuss results with project team	Discuss the following seven factors for each study with the project team: study quality, cost-effectiveness, variability and uncertainty, balance between health benefits, side effects and risk, generalizability, transferability and expected implementation problems.
		Fill in Table 3 to record the group’s discussion for each EE on all seven factors.
SR-EEs for CPG development	5.2.2 Formulate and present recommendations	In addition to the seven factors discussed, important other considerations which need to be taken into account before recommendations for CPG can be written are: expert opinions, patient preferences, costs (budget impact or practicability), clinical relevance, legal consequences, availability of facilities, organization of care and security.
		Use one of the following methods for structuring the discussion: voting system, informal consensus (e.g. expert opinions), and formal consensus techniques (e.g. Delphi, nominal group techniques) to formulate the final recommendations.
		When there is no evidence available on a specific research question, a statement needs to be added on this.
		In general, the presentation of the recommendations should also be clear. More specifically, users should be able to find the most relevant recommendations easily.
SR-EEs for CPG development	5.2.3 External review before publication	The CPG should be reviewed externally before it is published.

Appendix 1. Examples of global and country-specific sources for clinical practice development.

Global	Name organization	Link
	The AGREE-II (Appraisal of Guidelines for Research and Evaluation) Next Steps Consortium	http://www.agreetrust.org/
	Guidelines International Network (GIN)	http://www.g-i-n.net
	World Health Organization (WHO)	http://apps.who.int/iris/bitstream/10665/75146/1/9789241548441_eng.pdf
Country specific	Name organization	Link
USA	The Agency for Healthcare Research and Quality (AHRQ)	http://iom.nationalacademies.org/Reports/2011/Clinical-Practice-Guidelines-We-Can-Trust/Standards.aspx
Germany	Arbeitsgemeinschaft der Wissenschaftlichen Medizinischen Fachgesellschaften (AWMF-guidance)	http://www.awmf.org/en/clinical-practice-guidelines/awmf-guidance/cpg-development.html
The Netherlands	Knowledge Institute of Medical Specialists	http://www.kennisinstituut.nl
England and Wales	National Institute for Health and Care Excellence (NICE)	http://www.nice.org.uk/article/pmg20/chapter/1%20Introduction%20and%20overviewhttp://www.nice.org.uk/article/PMG20/chapter/7-Incorporating-economic-evaluationhttp://www.nice.org.uk/article/pmg9/chapter/Forewordhttp://www.nice.org.uk/article/pmg9/chapter/5-The-reference-case
Scotland	Scotland Intercollegiate Guidelines Network (SIGN)	http://www.sign.ac.uk/guidelines/fulltext/50/
Australia	National Health and Medical Research Council (NHMRC)	https://www.nhmrc.gov.au/guidelines-publications/information-guideline-developers
Canada	Cancer Care Ontario Program in evidence-based care handbook	https://www.cancercare.on.ca/about/programs/pebc/
New Zealand	New Zealand Guidelines Group	http://www.health.govt.nz/about-ministry/ministry-health-websites/new-zealand-guidelines-group

Thielen FW, Van Mastrigt GAPG, Burgers LT, et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: database selection and search strategy development (part 2/3). doi:10.1080/14737167.2016.1246962. [Epub ahead of print]

 Google Scholar

Wijnen B, Van Mastrigt GAPG, Redekop WK, et al. How to prepare a systematic review of economic evaluations for informing evidence-based healthcare decisions: data extraction, risk of bias, and transferability (part 3/3). doi:10.1080/14737167.2016.1246961. [Epub ahead of print]

 Google Scholar

Institute JB. Joanna Briggs institute reviewers’ manual 2014. [Internet]. Adelaide: JBI; 2014. cited 2015 May 15.

 Google Scholar

Higgins J, Green S. Cochrane handbook for systematic reviews of interventions version 5.1. 0. Chichester: The Cochrane Collaboration; 2013.

 Google Scholar

Smith V, Devane D, Begley CM, et al. Methodology in conducting a systematic review of systematic reviews of healthcare interventions. BMC Med Res Methodol. 2011;11(1):1–6.

 PubMed Web of Science ®Google Scholar

Russell R, Chung M, Balk EM, et al. Issues and challenges in conducting systematic reviews to support development of nutrient reference values: workshop summary: nutrition research series, Vol. 2. (US), Rockville (MD): Agency for Healthcare Research and Quality; 2009.

 Google Scholar

Currie G, Manns B. Glossary of terms for health economics and systematic reviews. In: Donaldson C, Mugford M, Vale L, Editors. Evidence-based health economics: from effectiveness tot efficiency in systematic review. London: BMJ Books; 2002.

 Google Scholar

Walker DG, Teerawattananon Y, Anderson A, et al. Generalizability, transferability, complexity and relevance. In: Shemilt I, Mugford M, Vale L, et al., Editors. Evidence-based desisions and economics. 2 nd ed. Chichester: Blackwell Publishing; 2010.

 Google Scholar

Knies S, Ament AJ, Evers SM, et al. The transferability of economic evaluations: testing the model of Welte. Value Health. 2009;12(5):730–738.

 PubMed Web of Science ®Google Scholar

Beller EM, Glasziou PP, Altman DG, et al. PRISMA for abstracts: reporting systematic reviews in journal and conference abstracts prisma for abstracts group. PLoS Med. 2013;10(4):e1001419.

 PubMedGoogle Scholar

Husereau D, Drummond M, Petrou S, et al. Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement. Value Health. 2013;16(2):e1–5.

 PubMed Web of Science ®Google Scholar

Sullivan SD, Mauskopf JA, Augustovski F: Principles of good practice for budget impact analysis II: report of the ISPOR task force on good research practices – budget impact analysis. Value Health 2014, 175–14.

 PubMed Web of Science ®Google Scholar

Moher D, Liberati A, Tetzlaff J, et al. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Bmj. 2009;339:b2535.

 PubMed Web of Science ®Google Scholar

Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting and evaluation in health care. J Clin Epidemiol. 2010;63(12):1308–1311.

 PubMed Web of Science ®Google Scholar

De kinderen RJ, Lambrechts DAJE, Wijnen BFM, et al. An economic evaluation of the ketogenic diet versus care as usual in children and adolescents with intractable epilepsy: An interim analysis. Epilepsia. 2016;57(1):41–50.

 PubMed Web of Science ®Google Scholar

World Health Organisation. WHO handbook for guideline development. Geneva: World Health Organization; 2014.

 Google Scholar

Drummond M, Schulper MJ, Claxton K, et al. Methods for the economic evaluation of health care programmes. 4th ed. Oxford: Oxford University Press; 2016.

 Google Scholar

Brunetti M, Shemilt I, Pregno S, et al. GRADE guidelines: 10. Considering resource use and rating the quality of economic evidence. J Clin Epidemiol. 2013;66(2):140–150.

 PubMed Web of Science ®Google Scholar

Developing NICE guidelines: the manual. [http://www.nice.org.uk/article/PMG20/chapter/7-Incorporating-economic-evaluation]

 Google Scholar

Institute of Medicine. GRADE working group. [http://www.gradeworkinggroup.org/]

 Google Scholar

Medicine Io. Clinical practice guidelines we can trust. Washington: The national academies press; 2011.

 Google Scholar

Moher D, Shamseer L, Clarke M, et al. Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement. Syst Rev. 2015;4(1):1.

 Web of Science ®Google Scholar

International prospective registry of systematic reviews. [http://www.crd.york.ac.uk/PROSPERO/]

 Google Scholar

Cochrane collaboration. [www.cochrane.org]

 Google Scholar

Search Filter Resource. [[https://sites.google.com/a/york.ac.uk/issg-search-filters-resource/home]]

 Google Scholar

Consensus Health Economic Criteria - CHEC list [https://hsr.mumc.maastrichtuniversity.nl/consensus-health-economic-criteria-chec-list]

 Google Scholar

Odnoletkova I, Goderis G, Pil L, et al. 2014. Cost-effectiveness of therapeutic education to prevent the development and progression of type 2 diabetes: systematic review. J Diabetes Metab 5:438. doi:10.4172/2155-6156.1000438

 Google Scholar

Drummond MF, Jefferson T. Guidelines for authors and peer reviewers of economic submissions to the BMJ. Bmj. 1996;313(7052):275–283.

 PubMed Web of Science ®Google Scholar

Philips Z, Bojke L, Sculpher M, et al. Good practice guidelines for decision-analytic modelling in health technology assessment. Pharmacoeconomics. 2006;24(4):355–371.

 PubMed Web of Science ®Google Scholar

Caro JJ, Eddy DM, Kan H, et al. Questionnaire to assess relevance and credibility of modeling studies for informing health care decision making: an ISPOR-AMCP-NPC good practice task force report. Value in Health. 2014;17(2):174–182.

 PubMed Web of Science ®Google Scholar

Bramer W, Milic J, Mast F. Reviewing retrieved references for inclusion in systematic reviews using endNote. J Med Lib Assoc. accepted for publication.

 Google Scholar

CCEMG - EPPI-Centre Cost Converter [http://eppi.ioe.ac.uk/costconversion/default.aspx]

 Google Scholar

Brunetti M, Ruiz F, Lord J, et al. Grading economic evidence. In: evidence-based decisions and economics. Shemilt I, Mugford M, Vale L, et al., Editors. Oxford: Wiley-Blackwell; 2010.

 Google Scholar