A systematic review of pharmacoeconomic guidelines

Abstract

Objective: To review, summarize, and analyze both similarities and differences of pharmacoeconomic (PE) guidelines, to enable researchers to access their characteristics and the current state of PE guidelines; furthermore, to learn which methodological issues still remain contested and to promote the methodological development of PE guidelines.

Materials and methods: The authors performed a search for PE guidelines using PubMed, the Cochrane library database, and the websites of the International Society for Pharmacoeconomics and Outcomes Research. Information of each guideline was extracted using a pre-designed extraction template, which included 22 aspects; the guidelines were summarized in the forms of charts, and their characteristics have been described.

Results: A total of 40 PE guidelines were studied. The most common methodological issues include the types of analysis, sources for effectiveness, use of quality-adjusted life-years (QALYs) to measure outcomes, and use of incremental cost effectiveness ratios to present results. The majority of the guidelines preferred a cost utility analysis with outcomes expressed in terms of QALYs. Most of the guidelines preferred meta-analysis or meta-analysis of the randomized controlled trials, and required a systematic review of all evidence. Issues that varied most in the guidelines were the choice of the comparator, recommended costs to be included, methods related to indirect cost calculations, methods of sensitivity analysis, and discounting rate.

Conclusion: A comparison of these guidelines revealed that a number of differences exist among them in several key aspects, and some critical methodological issues still exist, for which no best solution is available. Furthermore, efforts need to be made to develop harmonious methods for the PE, and to improve the transferability of the outcomes of PE evaluations.

Keywords:

• Pharmacoeconomics
• guidelines
• comparison
• characteristics

Introduction

Health resources are undoubtedly limited, and certainly cannot provide adequate treatment for all patients in all circumstances¹. At the same time, health costs have been increasing over recent years. For example, it has been reported that national health expenditures in the US are projected to grow at an average annual rate of 5.6% from 2016 to 2025, accounting for 19.9% of the gross domestic product (GDP) by 2025². Moreover, seen from global resource allocation, total health expenditure accounts for about an average of 9.7% of the global GDP³. Considering this situation, the question arises of how to best utilize these limited health resources and, consequently, how to reduce inferior programs and control health expenditures has become a challenging and hot research topic. Health economics (HE), developed to assess the outcomes of alternative treatment scenarios and policies, and to select optimal proposals while eliminating inferior programs, has become an effective method to solve such problems⁴. Pharmacoeconomics (PE) is a branch of HE, particularly focusing on the costs and benefits of pharmacy and medical technology^3,5. PE can effectively avoid resource waste and irrational distribution of medical resources and, thus, help to effectively allocate resources to maximize net health benefits⁶. This area is also the focus of this study.

Based on this, more attention has been focused on the development of the PE theory. An enormous increase in the PE literature could be observed over the past decades. However, reviews are divided about the quality of published PE studies⁷. A reason for this is that designs and implementation within PE studies are often unscientific. PE studies face several challenges, e.g. complex research methodology, or the fact that economic data can display substantial national, regional, or institutional variations, which can limit “transferability” and “comparability”. Moreover, many researchers are unfamiliar with the research methods of PE. To address these issues, guidelines for the improvement of the quality and comparability of PE research have been developed by several countries and organizations⁸. In 1992, Australia was the first country to introduce a mandatory methodological guideline for health economic evaluations of pharmaceuticals^9,10. After that, several countries and organizations developed their own HE/PE evaluation guidelines. Nevertheless, although these guidelines share a common basis, some of them require the presentation of local data, and integrated their own opinions with regard to methodological issues, especially for those for which there is still no consensus regarding appropriate or optimal methods^11,12. As a result, requirements stated in different guidelines are not always comparable, since they have their own characteristics¹¹.

A review of national HE guidelines by Hjelmgren et al.⁹ identified a lack of consensus on the key aspects of HE evaluations, such as choices of the perspective, and costs that should be included in the analysis. Furthermore, the study by Bracco and Krol¹³ reviewed economic evaluations in European submission guidelines and showed a large variation among these guidelines in terms of recommendations regarding costs to be included and methods related to cost calculations and discounting. More recently, a similar study conducted by the EUnetHTA Joint Action et al.¹² compared existing methodological guidelines for the HE in countries that have organizations involved in the EUnetHTA. The study arrived at similar conclusions, revealing that these guidelines varied extensively in terms of recommendations regarding the sources for effectiveness and the choice of the perspective. However, most similar studies mainly focused on the HE, and the guidelines of European countries, thus ignoring guidelines for Africa, Asia, and Oceania.

We conducted a comprehensive overview and comparison of PE guidelines. The objective of this study was to update and add to previous similar research. In addition, we focus more on PE guidelines and differences among countries and organizations. We examined differences and similarities among guidelines to highlight the current state of PE guidelines, as well as methodological issues that are still contested or developed. By conducting this study, we hope to contribute to the further development of PE guidelines.

Materials and methods

Search strategy

This study is based on literature originating from various sources.

1. Relevant guidelines were obtained by searching both PubMed and the Cochrane library databases up to March 15, 2017. The following terms were used (alone and in combination): “Pharmacoeconomic*”, “Health economic*”, “Pharmaceutical”, “Economic*”, “Health technology assessment*”, “cost effectiveness*”, “cost benefit*”, “cost utility*”, “recommend*”, “checklist*”, “guide*”. We combined the Medical Subject Heading (MeSH) and Title/Abstract Fields search. The full search strategy can be found in Supplementary Appendix 1.

2. The websites of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), the primarily ISPOR’s “Pharmacoeconomic Guidelines Around The World” database was searched. All retrieved articles were imported into the bibliographic database of EndnoteX6.

Selection criteria

Inclusion criteria

Articles included satisfied the following criterion: PE or HE guidelines specifying that their area of application includes pharmaceuticals.

Exclusion criteria

1. Older versions of guidelines.

2. Papers reproducing or adapting others’ guidelines or recommendations as third-part content rather than providing original PE guidelines or recommendations.

3. Guidelines which were only used to guide specific methodological issues or a sub-section rather than complete PE analysis (e.g. PE guidelines for modeling, budget impact analysis, cost analysis, expert opinions, “Piggyback” studies).

4. Articles that describe how to read economic evaluations.

5. Guidelines for a standard reporting format or question lists/tools for the evaluation of the quality of PE.

6. Publications not in English or Chinese.

Study selection and information extraction

Initial screening by one reviewer (Ji Qu) removed all duplicates. Then, two blinded reviewers (Ye Zhao and Wen-juan Ma) assessed the titles and abstracts of the articles that were about PE/HE guidelines. After that, full-text articles of the remaining studies were browsed. Studies that met all of the selection criteria were finally reviewed. If differences occurred, consensus was sought, and, if this could not be reached, a third reviewer (Jin-hui Tian) was consulted and a final decision was achieved via discussion. The authors were contacted if more information was required to determine eligibility for inclusion. To obtain the necessary data, a template was pre-designed, which included the headings: year, primarily used for, affiliation of authors, type of guideline, purpose of the guidelines, target audience of author’s interests, study question, target population, perspective, comparator, types of analyses, preferred data sources for effectiveness, choices of outcome measures, costs to be included, modeling, time horizon, discounting rate, uncertainty, presentation of results, efficacy vs effectiveness, equity, and conflicts of interests. The template was based on items from a few examples of the existing guidelines^14,15. Two reviewers (Ye Zhao and Hai-ming Feng) independently collected and extracted the data. All data were compared between reviewers to check for accuracy and agreement prior to analysis. Inconsistencies between reviewers were reconciled via discussion. The results were tabulated under the following main topics: (1) General information about these guidelines; and (2) comparison of the main contents of these guidelines. In terms of the type of guideline in the general information section, the guidelines were divided into three groups according to their roles in the national reimbursement processes, following definitions used by Hjelmgren et al.⁹, based on a prior work by Drummond¹⁶. Formalized guidelines were defined as those whose recommendations were required or mandatory prior to reimbursement submissions. Informal guidelines were defined as those whose recommendations have been voluntary prior to reimbursement submissions. Guidelines for economic evaluations were defined as general recommendations intended to improve the methodology of PE evaluations^9,13. In addition (for recording convenience), we used the country name to replace the national PE/HE guideline listed in the ISPOR website. This does not, however, mean that the listed guideline was the only guideline for the country.

Results

A flowchart can be found in Supplementary Appendix 2, which followed a PRISMA flow diagram¹⁷ and described the selection process of studies included in the paper. The electronic searches identified 8,418 citation records (PubMed = 6,311 and Cochrane library = 2,107); in addition, PE/HE guidelines found on the ISPOR website were also included (n = 31). After removal of duplicate copies (n = 53) via ENDNOTE X6, 8,396 studies remained. After screening the titles and abstracts, 311 articles were retrieved for full-text review. Of these 311 articles, 255 articles were included (220 did not comply with the inclusion criteria, 18 were not in English or Chinese, eight did not provide the original guidelines, six only described how to read guidelines, and three were standard reporting format or question lists/tools), which resulted in 56 articles that met the inclusion criteria. Of these 56 articles, two were excluded because we found an updated version, and 14 were excluded because they did not provide a complete guideline. Finally, the literature search resulted in 40 articles that met both the inclusion and exclusion criteria.

General information of the included guidelines

A structured summary of the general information of all 40 guidelines is presented in Table 1. Among these 40 guidelines, the first was published by Guyatt et al.²² in 1986 and was aimed to guide PE evaluations. The most recent guidelines are the EUnetHTA Joint Action¹², Australia¹⁴, the Netherlands⁵¹, and the US⁵², published in 2016. Among all 40 guidelines, 19^{3,14,18,19,23,26,29,30,32,37,39,40,41,43–46,48,49} were mainly developed for evaluations of pharmaceuticals, while 21^{7,12,15,20–22,24,25,27,28,31,33–36,38,42,47,50–52} were developed for all types of technologies. Of the guidelines, 45% were formal, 22.5% were informal, and 32.5% were guidelines for economic evaluations. All details are presented in Tables 1 and 2.

Table 1. General information of the included guidelines.

Guidelines	Year	Primarily used for	Affiliation of authors	Type of guidelines
Clemens et al.^18	1995	Pharmaceuticals	Roche Pharmaceuticals	Guidelines for economic evaluations
Drummond and Jefferson^7	1996	All types of technologies	University of York	Guidelines for economic evaluations
Detsky^19	1993	Pharmaceuticals	The Toronto Hospital	Formalized
EUnetHTA Joint Action^12	2016	All types of technologies	European Network for Health Technology Assessment (EunetHTA) Joint Action	Guidelines for economic evaluations
Graf von der Schulenberg and Hoffmann^20	2000	All types of technologies	University of Hanover	Guidelines for economic evaluations
Graf von der Schulenburg et al.^21	2008	All types of technologies	University of Hanover	Guidelines for economic evaluations
Guyatt et al.^22	1986	All types of technologies	MacMaster University	Guidelines for economic evaluations
Langley^23	1999	Pharmaceuticals	University of Colorado	Informal
Task Force on Principles for Economic Analysis of Health Care Technology^24	1995	All types of technologies	Task Force on Principles for Economic Analysis of Health Care Technology	Guidelines for economic evaluations
Australia^14	2016	Pharmaceuticals	The Pharmaceutical Benefits Advisory Committee (PBAC)	Formalized
Austria^25	2006	All types of technologies	Institute for Pharmaeconomic Research (IPF)	Guidelines for economic evaluations
Baltic^26	2002	Pharmaceuticals	Experts from health authorities of the Baltic countries	Formalized
Belgium^27	2012	All types of technologies	Experts from health authorities of the Baltic countries	Formalized
Canada^15	2006	All types of technologies	Canadian Agency for Drugs and Technologies in Health (CADTH)	Guidelines for economic evaluations
China Mainland^3	2011	Pharmaceuticals	China Guidelines for Pharmacoeconomic Evaluations research group	Informal
Croatia^28	2011	All types of technologies	Agency for Quality and Accreditation in Health Care	Formalized
Denmark^29	1997	Pharmaceuticals	Danish Institute for Health Services Research and Development	Informal
Egypt^30	2013	Pharmaceuticals	General Directorate of Hospital Pharmacy Pharmacoeconomic Unit	Guidelines for economic evaluations
England & Wales^31	2013	All types of technologies	National Insititue for Health and Clinical Excellence (NICE)	Formalized
Finland^32	2015	Pharmaceuticals	Pharmaceuticals Pricing Board	Formalized
France^33	2003	All types of technologies	The French Health Economists Association	Guidelines for economic evaluations
Germany^34	2009	All types of technologies	German national institute for quality and efficiency in health care (IQWiG)	Formalized
Hungary^35	2002	All types of technologies	The Ministry of Health of Hungary	Formalized
Ireland^36	2010	All types of technologies	Health Information and Quality Authority	Formalized
Israel^37	2010	Pharmaceuticals	Pharmaceutical Administration of the Ministry of Health	Informal
Italy^38	2001	All types of technologies	Italian Group for Pharmacoeconomic Studies	Formalized
Malaysia^39	2012	Pharmaceuticals	Pharmaceutical Services Division, Ministry of Health Malaysia	Formalized
New Zealand^40	2015	Pharmaceuticals	PHARMAC, the Pharmaceutical Management Agency	Informal
Norway^41	2012	Pharmaceuticals	Norwegian Medicines Agency (NOMA)	Formalized
Poland^42	2009	All types of technologies	Agency for Health Technology Assessment	Guidelines for economic evaluations
Portugal^43	1998	Pharmaceuticals	Portuguese Drug and Pharmaceutical Institute	Informal
Russian Federation^44	2010	Pharmaceuticals	Russian State Medical University	Formalized
Scotland^45	2007	Pharmaceuticals	Scottish Medicines Consortium	Formalized
South Africa^46	2012	Pharmaceuticals	National Department of Health	Informal
Spain^47	2010	All types of technologies	Canary Islands Health Services	Guidelines for economic evaluations
Sweden^48	2003	Pharmaceuticals	The Pharmaceutical Benefits Board	Formalized
Taiwan^49	2006	Pharmaceuticals	Taiwan Society for Pharmacoeconomic and Outcomes Research (TaSPOR)	Informal
Thailand^50	2008	All types of technologies	The Medical Association of Thailand	Formalized
The Netherlands^51	2016	All types of technologies	National Health Care Institute	Formalized
USA^52	2016	All types of technologies	The Academy of Managed Care Pharmacy (AMCP)	Informal

Formalized guidelines: those whose recommendations were required or mandatory prior to reimbursement submissions.

Informal guidelines: those whose recommendations have been voluntary prior to reimbursement submissions.

Guidelines for economic evaluations: general recommendations intended for use in improving the methodology of PE evaluations.

Table 2. Comparison of main content in these guidelines, categorized by topic.

Topic	Quantity/Percentage	Guidelines
Types of guidelines
Formalized	18 (45%)	Australia, Baltic, Belgium, Croatia, England & Wales, Finland, Germany, Hungary, Ireland, Italy, Malaysia, Norway, Russian Federation, Scotland, Sweden, Thailand, The Netherlands^19
Informal	9 (22.5%)	China, Denmark, Israel, New Zealand, Portugal, South Africa, Taiwan, USA^23
Guidelines for economic evaluations	13 (32.5%)	Austria, Canada, Egypt, France, Poland, Spain^7^,^12^,^18^,^20–22^,^24
Perspective
Societal	14 (35%)	China, Denmark, France, Norway, Portugal, Spain, Sweden, Taiwan, Thailand, The Netherlands^20–22^,^24
Healthcare payer	19 (47.5%)	Australia, Baltic, Belgium, Canada, Croatia, Egypt, England & Wales, Finland, Germany, Hungary, Ireland, Israel, Malaysia, New Zealand, Poland, Scotland, South Africa^12^,^23
Societal and/or healthcare payer	2 (5%)	Italy, Russian Federation
Healthcare decision-maker	1 (2.5%)	USA
Not specified	4 (10%)	Austria^7^,^18^,^19
Comparator
The most appropriate	2 (5%)	Finland, Sweden
The most commonly used	5 (12.5%)	Croatia, Egypt, France, Ireland, Italy
The most commonly used or the most effective	4 (10%)	Taiwan, Thailand^20^,^21
The most commonly used or the least expensive	1 (2.5%)	^19
The most commonly used, the most effective or the least expensive	5 (12.5%)	Canada, Portugal, Russian Federation^24
Standard treatment	11 (27.5%)	Austria, Baltic, China, Denmark, England & Wales, Hungary, Israel, Malaysia, South Africa, Spain, The Netherlands
Therapy that will be replaced	7 (17.5%)	Australia, New Zealand, Norway, Poland, Scotland, USA^23
All relevant therapeutic alternatives	2 (5%)	Belgium, Germany
Not specified	3 (7.5%)	^7^,^18^,^22
Preferred types of analysis
CUA	12 (30%)	Australia, England & Wales, Finland, Ireland, Israel, New Zealand, Norway, Portugal, Scotland, Sweden, Thailand, The Netherlands
CUA or CEA	11 (27.5%)	Baltic, Belgium, Canada, Croatia, Denmark, Egypt, Hungary, Italy, Taiwan, USA^12
CCA and (CEA or CUA)	1 (2.5%)	Poland
CEA, CUA, CBA, CMA, CCA, depends on the research question	14 (35%)	Austria, France, Malaysia, Russian Federation, South Africa, Spain^7^,^18–24
CBA or CUA	1 (2.5%)	China
Not specified	1 (2.5%)	Germany
Preferred sources for effectiveness
Clinical trials	1 (2.5%)	^20
RCTs or meta analysis of RCTs	26 (65%)	Australia, Austria, Baltic, Belgium, Croatia, Denmark, Egypt, England & Wales, Finland, Hungary, Ireland, Malaysia, New Zealand, Norway, Poland, Portugal, Russian Federation, South Africa, Taiwan, Thailand, The Netherlands, USA^12^,^19^,^21^,^24
Pragmatic trials	3 (7.5%)	Italy, Spain^7
Not specified	8 (20%)	Canada, China, France, Germany, Israel, Scotland^18^,^23
A systematic literature search required	23 (57.5%)	Australia, Belgium, Canada, Croatia, Egypt, England & Wales, Finland, Hungary, Ireland, Israel, Malaysia, New Zealand, Norway, Poland, Russian Federation, Scotland, South Africa, Taiwan, The Netherlands, USA^12^,^20^,^23
Methods of measurement for utility
Direct methods	4 (10%)	Denmark, France^20^,^22
Indirect methods	18 (45%)	Australia, Baltic, Belgium, Canada, Croatia, Egypt, England & Wales, Hungary, Ireland, New Zealand, Norway, Poland, Scotland, Spain, Sweden, Thailand, The Netherlands^23
Direct or indirect methods	10 (25%)	Austria, China, Germany, Italy, Malaysia, Portugal, Taiwan, USA^7^,^21
Costs
Societal perspective
Direct medical costs + direct non-medical cost + indirect costs	7 (17.5%)	China, France, Italy, Taiwan, Thailand^21^,^24
Direct costs + indirect costs	4 (10%)	Denmark, Spain, Sweden^20
Direct costs + indirect costs + intangible costs	1 (2.5%)	Portugal
Costs within the healthcare system + costs for patient and family +  costs in other sectors	1 (2.5%)	The Netherlands
Healthcare payer perspective
Direct healthcare costs	12 (30%)	Australia, Baltic, Belgium, Croatia, England & Wales, Ireland, Israel, New Zealand, Russian Federation, Scotland, South Africa^23
Direct healthcare costs + comparable social welfare costs related to the therapies	1 (2.5%)	Finland
Only direct medical costs + some additional costs	1 (2.5%)	Egypt
Direct medical costs + direct non-medical costs	2 (5%)	Germany, Hungary
Direct costs (to publicly funded services, to publicly funded healthcare system, to patients and their families) + time costs to patients and their families	1 (2.5%)	Canada
Methods of measurement for lost productivity
Friction cost methods	4 (10%)	Canada, The Netherlands^20^,^21
HCM	8 (20%)	Austria, China, Germany, Italy, Poland, Sweden, Taiwan, Thailand
HCM or friction cost methods	5 (12.5%)	Belgium, Denmark, France, Malaysia, Spain
Discounting rate
Costs and benefits: 5%	11 (27.5%)	Australia, Austria, Baltic, Canada, Croatia, Hungary, Portugal, South Africa, Taiwan^21^,^24
Costs and benefits: 4%	2 (5%)	Ireland, Norway
Costs and benefits: 3.5%	4 (10%)	Egypt, England & Wales, New Zealand, Scotland
Costs and benefits: 3%	8 (20%)	Finland, Germany, Israel, Italy, Malaysia, Spain, Sweden, Thailand
Costs and benefits: 0%, 3%, 5%	1 (2.5%)	France
Costs and benefits: 5% and national guidelines require rates	1 (2.5%)	^20
Costs and benefits: 1-year treasury bond rate	1 (2.5%)	China
Costs and benefits: government recommended rate	1 (2.5%)	^7
Costs: 5%, Benefits: 3.5%	1 (2.5%)	Poland
Costs: 4%, Benefits: 1.5%	1 (2.5%)	The Netherlands
Costs: 3%, Benefits: 1.5%	1 (2.5%)	Belgium
Costs: 5%	1 (2.5%)	Russian Federation
Not specified	5 (12.5%)	Denmark, USA^12^,^18^,^22
Sensitivity analysis for parameter uncertainty
PSA preferred	6 (15%)	Croatia, England & Wales, Ireland, Norway, Thailand, The Netherlands
DSA preferred	3 (7.5%)	Australia, New Zealand, Scotland
PSA and DSA	3 (7.5%)	Belgium, Canada, China

Abbreviations. HCM, Human capital method; CBA, Cost benefit analysis; CCA, Cost consequence analysis; CEA, Cost effectiveness analysis; CMA, Cost minimization analysis; CUA, Cost utility analysis; DSA, Deterministic sensitivity analysis; PSA, Probabilistic sensitivity analysis; RCTs, Randomized controlled trials.

Comparison of main content in these guidelines

A comparison of the detailed content of the 40 guidelines is presented in Supplementary Appendix 3. To simplify the understanding of the methodological issues that were recommended by the guidelines and their variation among countries, we categorized and presented the methodological issues according to topics in Table 2.

Purpose of these guidelines

The major purpose of these guidelines can be summarized as follows: (1) to provide standards for the conduct, reporting, and quality valuation of PE research, which can in turn increase the methodological quality, transparency, comparability, and uniformity of PE evaluations; (2) to provide a basis for drug reimbursement, and to assist policymakers in decision-making. Although the guidelines have a common basis, their purposes varied slightly, which becomes apparent when they are compared with each other.

Target audience of guideline’s interests

The main target audience of the guideline’s interests can be summarized as follows: pharmaceutical companies and manufacturers, academics, consumers, decision-makers, and researchers. Every guideline has its respective target audience and the information is presented in Supplementary Appendix 3.

Study question

Among all 40 investigated guidelines, 27 guidelines (67.5%)^{3,6,14,15,18–21,24–28,30,31,34,36,38–43,45,47,50,51} indicated that the study questions should be clearly stated. Twenty-one guidelines (52.5%)^{3,4,27,30–35,37,39–41,43–46,49–52} stated that a clear description of the studied disease should be included, which may include epidemiological data, the burden of the disease, current treatments, and existing clinical guidelines for the disease. Twenty guidelines (50%)^{3,14,15,23,27,30–32,35–37,39–42,45,46,50–52} required statement of the details of the studied medicines and technologies, which may include information about pharmacology, pharmacokinetics, pharmacodynamics, contraindications, and adverse events.

Target population

Thirty-five guidelines (87.5%)^{3,14,15,18–21,23,25–28,30–52} required specification of the target population of the new health technology. Thirty guidelines (75%)^{3,14,15,18,19,21,25–28,30,31,33–36,38–43,45–52} recommended sub-group analysis if the need exists.

Perspective

Four guidelines (10%)^7,18,19,25 only required that the perspective should be clearly stated without requiring a specific recommendation. Fourteen guidelines (35%)^{3,20–22,24,29,33,41,43,47–51} recommended inclusion of societal perspectives. Nineteen guidelines (47.5%)^{12,14,15,23,26–28,30–32,34–37,39,40,42,45,46} recommended inclusion of healthcare perspectives.

Comparator

Three guidelines (7.5%)^7,18,22 stated that the utilized comparator should be described in detail without requiring specific recommendations. The majority (15) of the guidelines^{15,19–21,24,28,30,33,36,38,43,44,48–50} recommended that the comparator should mainly be chosen from the most commonly used treatment, the least expensive, or the most effective currently available of the treatments (see Table 2). Seven guidelines (17.5%)^{14,23,40–42,45,52} indicated treatments that would most likely be replaced by new technologies could be chosen as comparators. Eleven guidelines (27.5%)^{3,25,26,29,31,35,37,39,46,47,51} recommended that standard treatments should be chosen as the comparator. Two guidelines (5%)^27,34 required that all therapeutic alternatives should be included, as a necessary means to estimate a cost-effectiveness frontier for multiple interventions⁵³. Furthermore, two guidelines (5%)^32,48 recommended the use of the most appropriate treatment (e.g. the most frequently used treatments, the minimum therapy methods, or no treatment) as comparators.

Types of economic analysis

The primary viewpoints of 14 guidelines (35%)^{7,18–24,23,33,39,44,46,47} were that the cost effectiveness analysis (CEA), cost utility analysis (CUA), cost minimization analysis (CMA), or cost benefit analysis (CBA) can be chosen, where the choice mainly depends on the research question; however, the choice should be justified. Twelve guidelines (30%)^{14,31,32,36,37,40,41,43,45,48,50,51} stated the CUA as the preferred type of economic analysis. Eleven guidelines (27.5%)^{12,15,26–30,35,38,49,52} indicated either CEA or CUA as preferred analyses, and four guidelines (10%)^27,35,38,42 indicated the CBA unlikely to be helpful. Only the Chinese guideline³ stated that the CUA or CBA are preferred. The German guideline³⁴ provided no accurate description of the types of economic analysis⁵³. The CMA can only be used when no differences exist in the effects among compared technologies¹².

Sources for effectiveness

Randomized controlled trials (RCTs) or the meta-analysis of RCTs were the preferred sources for effectiveness in 26 guidelines (65%)^{12,14,19,21,24–32,35,36,39–44,46,49–52}. Although Mathes et al.⁵³ considered that the Polish guideline preferred using data from observational studies, the guideline itself stated that data from the highest available level of evidence has to be used, and systematic reviews (with or without meta-analysis) or direct comparisons of RCTs are of higher priority in terms of credibility⁴². Nine guidelines (22.5%)^{12,14,18,27,31,40,42,44,51} required assessment of the quality of evidence. Twenty-three guidelines (57.5%)^{12,14,15,20,23,27,28,30–32,35–37,39–42,44–46,49,51,52} required a systematic literature search. Three guidelines (7.5%)^7,38,47 explicitly stated the attractivity of pragmatic trials. Fifteen guidelines (37.5%)^{14,23,28,31–35,37,41,42,45,46,51,52} preferred direct comparisons. Eight guidelines (20%)^{3,15,18,23,33,34,37,45} did not further clarify which sources are preferred.

Choices of outcome measures

If the CEA was chosen as the method of PE analysis, the results should be expressed in terms of natural units, such as the years of life gained, hospital days prevented, and clinical parameters^{3,7,12,14,15,18,21–24,26–28,30,33,35,36,38,39,42,43,46,47,49,51}. Almost all of the guidelines predominantly described the CUA with outcomes expressed in terms of quality-adjusted life-years (QALYs). The QALYs combine changes in both the quantity and quality-of-life (mortality and morbidity) into one composite measure⁴⁰, which is one of the reasons why the QALYs is particularly useful. The QALYs can be calculated by multiplying the quantity of life-years gained due to the intervention by health state utilities that reflect the health-related quality-of-life (HRQoL) during that time¹⁵. Health state utilities can be directly or indirectly measured. Direct measurement is a complex and expensive task. The main methods which can be used for direct measurement include: standard gamble (SG), time trade-off (TTO), and visual analog scale (VAS). The indirect measurement exists to use some instruments to indirectly obtain utilities-such as “Off the shelf” instruments: the Health Utilities Index (HUI) and the EQ-5D¹⁵. Eighteen guidelines (45%)^{14,15,23,26–28,30,31,35,36,40–42,45,47,48,50,51} explicitly stated that indirect methods are recommended. Four guidelines (10%)^20,22,29,33 recommended direct methods. Twenty-two guidelines (55%)^{3,7,14,15,20,21,23,27,28,30,31,36,37,39,41,43,47–52} required statement of the population from which the utilities originate. The majority of guidelines recommended that the utilities should originate from the general population. Nine guidelines (22.5%)^{3,15,27,28,31,34,36,47,49} stated that the quantity and quality-of-life should be presented in disaggregated and aggregated forms. Ten guidelines (25%)^{14,27,30,31,35,36,40,41,45,50} accepted mapping from disease-specific quality-of-life (QoL) measures to preference-based values if no data from the preferred instrument was available. If the CBA was chosen, the results should be expressed in a monetary form. Eight guidelines (20%)^{3,15,19,22,33,43,47,49} recommended the willingness to pay (WTP) when the results need to be monetized. Seven guidelines (17.5%)^{3,24,29,33,36,38,40} suggested that intangible costs should be included in outcomes rather than in costs.

With regard to the type of endpoints, 28 guidelines (70%)^{3,7,14,15,18,23–25,27,28,30,31,33,35,36–38,40–44,46–50,52} stated that final endpoints are preferred and, if intermediate endpoints were to be used, the connection between them and final endpoints should be clear.

Costs

The choice of the perspective determines the costs to be included. Thirty-two guidelines (80%)^{3,7,12,15,18–21,23–27,29,30,32,34–38,40,42,43,45–51} required clear recording of the amount of resources or both quantities of resources and unit costs. Eight guidelines (20%)^{22,28,31,33,39,41,44,52} either did not clearly specify the presentation of resources or only roughly outlined that resources used should be recorded.

Even in those guidelines that recommended the same perspective, the included costs differed. Among the guidelines that recommended a societal perspective, 11 guidelines (78.5%)^{3,20,21,24,29,33,38,47–50} required that direct and indirect costs should be included. Only the Portuguese guideline⁴³ suggested that intangible costs should also be included, which was not mandatory. This is because the Portuguese guideline had also recognized that intangible costs are not strictly economic costs, and can never be quantified or valued⁴³. Direct costs can primarily be divided into medical and non-medical costs. In addition, the Netherlands⁵¹ used different requirements from the above-mentioned guidelines (see Table 2).

From a healthcare perspective, 12 guidelines (63.1%)^{14,23,26,27,28,31,36,37,40,44–46} required only direct healthcare costs to be included. Canada¹⁵, Egypt³⁰, Finland³², Germany³⁴, and Hungary³⁵ differed from the above-mentioned guidelines (see Table 2).

In addition, 10 guidelines (25%)^{7,12,18,19,22,25,39,41,42,52} merely stated that all relevant costs should be included that are related to the chosen perspective, or that had recommendations not in accordance with the choice of perspective (e.g. Poland⁴² recommended the healthcare perspective, while providing a recommendation regarding costs from a societal perspective).

Seventeen guidelines (42.5%)^{3,15,20,21,25,27,29,33,34,38,39,42,47–51} provided different methods for the measurement of lost productivity, with little consensus. Four guidelines (10%)^15,20,21,51 recommended friction cost methods. Eight guidelines (20%)^{3,25,34,38,42,48–50} recommended the human capital method (HCM). Five guidelines (12.5%)^{27,29,33,39,47} suggested that lost productivity can be evaluated using the HCM or friction cost methods. Twelve guidelines (30%)^{14,15,22,26,27,30,34–36,40,50,51} required that costs should be adjusted for inflation and/or currency conversions. For future costs (which are costs associated with patients living longer as a result of a given intervention and, hence, consume additional health resources), numerous different opinions exist, and most of the guidelines divided them into direct and indirect. Nine (22.5%)^{3,15,24,27,30,34,35,40,43} guidelines considered direct future costs (directly related to the consequence of the intervention or the treatment of the studied disease) should be included, among them, three^31,32,40 guidelines stated the inclusion about indirect future costs (unrelated to their initial diagnosis or treatment) to still be a contentious issue. However, five (12.5%)^{18,27,34,40,43} guidelines did not mention direct future costs, and only stated indirect future costs, which should not be included. The French guideline³³ stated the inclusion about future costs to be a contentious issue. Thailand⁵⁰ considered future costs should not be included, while the Netherlands⁵¹ stated that all future costs should be included.

Modeling

All guidelines, except for Guyatt et al.⁷, stated that the use of models is accepted in PE evaluations. Twenty-seven (67.5%)^{3,7,14,15,21,23,24,27,28,30,31,33,34,36–43,45–47,49,50,52} guidelines required that the details of models and justifications for the choice of modeling technique should be provided. Eight (20%)^{12,19,25,26,32,35,44,51} guidelines required that the details of models should be provided, but did not require any justification for the choice of the utilized modeling technique. Four (10%)^18,20,29,48 guidelines only mentioned that models can be used. Fifteen (37.5%)^{14,15,27,30,31,33,34,36,41,42,48–52} guidelines stated that a validation of the utilized model should be provided. Nine (60%)^{15,30,31,33,34,36,41,42,48} of these further divided validity into both internal validity (to ensure the robustness of model) and external validity (can be tested in numerous ways including a comparison of the results with those of other models or via direct empirical evidence).

Time horizon

Twenty-five guidelines (62.5%)^{3,7,12,15,19,20,26,28,30–33,35,36,39–42,45–47,49–52} recommended the use of a sufficiently long time horizon of economic evaluations to reflect all important differences in either costs or outcomes among technologies that are compared. Four guidelines (10%)^18,21,25,27 stated that the time horizon depends on the studied technology or the research question. Three guidelines (7.5%)^24,37,43 required the time horizon to coincide with clinical conditions and the effects of treatment. Australia¹⁴ required that the time horizon should capture all important differences in both costs and outcomes; however, these do not have to unnecessarily extend beyond this. Germany³⁴ stated that the time horizon should at least cover the duration of RCTs. Sweden⁴⁸ required that the time horizon should cover the period during which the main health effects and costs arise.

Discounting

Thirty guidelines (75%)^{3,7,12,14,15,20,21,24–26,28,30–41,43,45–50} recommended to use the same discount rate for both costs and outcomes, while three guidelines (7.5%)^27,42,51 recommended a higher discount rate for costs than for outcomes, and Russia⁴⁴ required only costs to be discounted at a flat 5% rate. Five guidelines (12.5%)^{12,18,22,29,52} did not further specify any rates for discounting. Most guidelines used a standard discount rate of 3–5% for both costs and outcomes. In addition, 30 guidelines (75%)^{3,7,12,14,15,20,21,24,25,27–31,33–36,38–40,42,43,45–51} required inclusion of a sensitivity analysis (SA) of the discount rate. The ranges of the discount rate mostly ranged between 0–10%.

Uncertainty

All guidelines recommended the SA to handle uncertainty. For parameter uncertainty, six guidelines (15%)^{28,31,36,41,50,51} chose the probabilistic sensitivity analysis (PSA) as a preferred method. However, three guidelines (7.5%)^14,40,45 suggested the use of the deterministic sensitivity analysis (DSA) for routine examination, and the PSA should only be used additionally. Three guidelines (7.5%)^3,15,27 stated that parameter uncertainty should be handled by the DSA and PSA. Seven guidelines (17.5%)^{3,14,15,27,36,40,41} required the model uncertainty to be handled by the DSA. The remaining guidelines did not distinguish between the types of uncertainty, implying that this can be handled by the SA, the PSA, or the DSA, while three of them^46,47,49 required 1-way SA as the basic method. The US⁵² considered both types of SA. Several guidelines recommended results to be presented using confidence intervals (CIs) around the main variable, a cost effectiveness (CE) plane, or cost effectiveness acceptability curves (CEAC).

Presentation of results

The presentation of results has been detailed by 36 guidelines (90%)^{3,7,12,14,15,18,19,21–23,25–28,30–43,45–52}. Most of the guidelines recommend presenting the results in disaggregated and/or aggregated forms. Thirty-one guidelines (77.5%)^{3,7,12,14,15,19,23,25–28,30–33,35–40,42,43,45–52} recommended using incremental cost effectiveness ratios (ICERs) for the presentation of results.

Efficacy vs effectiveness

Twenty-four guidelines (60%)^{3,7,15,19,23–25,27,28,30,33,35,36,38,39,42–44,46,47,49,52} explicitly emphasized differences between efficacy and effectiveness. Twenty-two guidelines (55%)^{3,7,15,19,23,25,27,28,30,33,35,36,38,39,42–44,46,49–52} specifically preferred effectiveness. Spain⁴² recommended that both efficacy and effectiveness should be used.

Equity

Equity mainly means that all patients should have a fair opportunity for participation and for obtaining the expected treatment outcomes³⁰. For example, to determine equity in PE, an additional QALY should receive the same weight, regardless of any other utilized characteristics (e.g. their socio-demographic characteristics, their age, or their level of health) of the people receiving the health benefit^31,36. Sixteen guidelines (40%)^{3,14,15,19,20,22,27,28,30,31,33,35,36,45,47,49,50} required that equity should be discussed.

Conflicts of interests

Thirteen guidelines (32.5%)^{15,19,21,29,33,35,39,42,43,47–50} indicated that any existing conflicts of interests should be discussed.

Discussion and conclusions

This paper presents a systematic review and comparison of PE guidelines. By analyzing similarities and differences among a total of 40 PE guidelines that were included in this study, we identified their individual characteristics to provide a way for researchers and experts in the PE to access the present methodological development of the PE. Methodological issues that substantially varied among the guidelines were mainly those that presented controversy or no consensus regarding an appropriate method⁵³. The present study is intended to provide a preliminary understanding of these issues, and to enable PE specialists to focus more on them to contribute to the next phase of PE guideline development.

According to our study results, various countries all over the world produced numerous PE guidelines. Some of these were updated every few years, such as the guidelines of Australia¹⁴, the Netherlands⁵¹, and the US⁵²; however, some guidelines have never been updated since they were produced, such as Detsky¹⁹, Guyatt et al.²², and the guideline of Portugal⁴³. However, it is important that PE guidelines develop rigorous, evidence-based, clinical, and policy-relevant processes. Therefore, PE guidelines should be reviewed periodically to ensure that they keep pace with methodological developments and changing environments⁵⁴. Many countries produced compulsory PE guidelines for use prior to pharmaceutical reimbursement submissions. However, some guidelines remain voluntary. The purpose of PE guidelines can be summarized as to provide standards for PE research, and to ultimately improve the quality of PE evaluations. Furthermore, many guidelines intend to provide assistance for policymakers in their decision-making. It is important for PE research to introduce good study questions and to provide a clear description of the studied medicines and technologies. We pre-conceived that these issues are so important that they would be mentioned in all guidelines. However, the results are unsatisfactory. Among the 40 guidelines, only 27 guidelines mentioned that study questions should be clearly stated, and 20 guidelines indicated that there should be detailed descriptions of studied interventions. We suggest that the reason might be that these issues are so important that guideline makers consider there is no need to specifically describe them. The choice of the perspective is vital, because it provides a basis for the measurement of costs and outcomes. Although the societal perspective was often recommended in the literature, since it is comparatively ideal in theory (because it takes all relevant consequences into account at a broader scope, thus providing insightful information for decision makers), only 14 guidelines recommended it^27,50. The reason might be that this perspective is very difficult to implement. Consequently, most guidelines recommended the healthcare perspective. In addition, we found that the majority of guidelines recommending the societal perspective were informal, while most formalized guidelines recommended a healthcare perspective. Bracco and Krol¹³ suggested that the reason might be that the target audience of formalized guidelines more likely cared about costs that they must bear than those falling outside their responsibility. Although recommended costs to be included depend on the choice of the perspective, some unjustifiable differences exist among the guidelines⁵³. Several guidelines did not detail which costs should be included, instead briefly describing all relevant costs related to the perspective. We do not think that such vague descriptions are scientifically valid or help in improving the quality of PE research. Even in those guidelines that recommended the same perspective, the included costs differed. For example, from the societal perspective, some guidelines recommended direct medical and non-medical costs, as well as indirect costs, while others stated that direct and indirect costs should be included just in general terms. Moreover, some guidelines, such as the Netherlands³², recommended that costs for patients and families, costs within the healthcare system, and costs in other sectors should be included. Eight guidelines specifically mentioned intangible costs, and all guidelines, except for Portugal⁴³, suggested that intangible costs should be included in the outcome to avoid double-counting. Bracco and Krol¹³ indicated that there is little agreement on methods for lost productivity evaluation. We agree with their conclusion, as our results showed that the measurement of lost productivity basically involves two methods: the HCM and friction cost methods. Most guidelines recommended the HCM. Some guidelines did not recommend friction cost methods because they considered them to be challenging for data collection, due to requiring far more empirical work before they can be applied^33,49. However, the guidelines supporting friction cost methods stated that the HCM rely on the assumption of full employment being unrealistic, leading to a possible over-estimation of lost production^15,33. Therefore, these guidelines suggest friction cost methods to be more realistic. In general, both methods have flaws, and can only be used to evaluate lost productivity of paid labor. For unpaid work or leisure time, limited guidelines exist¹³. From a healthcare perspective, some guidelines only included direct healthcare costs, while some mentioned additional costs. There are arguments about the inclusion of future costs. Most guidelines agreed with the inclusion of direct future costs. However, whether these indirect future costs should be included has not been decided or reached a general agreement. Only the Netherlands⁵¹ stated that all future costs should be included. Sanders et al.⁵⁵ agreed with the Netherlands⁵¹, and also recommended that all related and unrelated future costs should be included. Regarding the choice of the comparator, there are still some different ideas. However, most guidelines recommended that standard treatments, or treatments that would most likely be replaced by new technologies, should be chosen as a comparator. Only two guidelines^27,34 required that all therapeutic alternatives should be included. Drummond et al.⁵⁴ suggest all relevant technologies should be included in HTA, since potential inefficiencies exist in all forms of healthcare. Otherwise, decision-making concerning the use of resources may be distorted⁵⁴. Another issue that varied substantially among the guidelines was regarding the methods of SA. All guidelines recommended using SA to handle uncertainty. However, the majority of them only roughly outlined that uncertainty should be handled by SA or the DSA/PSA, but did not distinguish specific types of uncertainty to provide more targeted recommendations. In addition, there is no unanimous opinion on how to handle parameter uncertainty. However, most guidelines recommended the PSA over the DSA. Methods of SA were also based on different statistical assumptions, providing different implications for decisions. As a result, restrictions on certain types of SA can be less helpful⁵³. Not surprisingly, almost all of the guidelines accepted the use of models, because these indeed form a centrally important sub-section of PE evaluations, and most of the guidelines (87.5%) specifically required that the details of models should be provided. However, only 15 guidelines required that the validity of the model should be provided, and only six of them briefly mentioned the validity rather than divided it into internal and external validity in more detail. Such rough descriptions may confuse researchers of PE evaluations. Equity and conflicts of interests had received far less attention than other issues. PE guidelines should focus more on these issues, as they might affect the allocation of resources. All such methodological issues that vary substantially among the guidelines are likely to impact the transferability of results in PE studies. The transferability of PE researches may be a problem, which should be brought to our attention. This may be particularly important in small jurisdictions and/or regarding decision problems that affect a small number of individuals. The methodological guidelines should recognize that limited resources are available for all activities⁵⁶. Therefore, if we knew that the values individuals place on health states are fairly similar across countries, it would be unnecessary for researchers or decision-makers in a given jurisdiction to specify that health state valuations must originate from the local population, thus avoiding unnecessary duplication of work and expense^53,56.

Issues with relatively unified recommendations were the preferred type of analysis, preferred sources for effectiveness, the use of the QALYs to measure outcomes, and the ICER to present results. The majority of the guidelines preferred the CUA, with outcomes expressed in terms of the QALYs, and an important reason was that the QALYs enable one to compare outcomes of different technologies across different activities in the healthcare sector³⁶. However, there are some limitations of QALYs. A concern when using QALYs is that they do not discriminate between conditions with different severity¹⁵. For instance, QALYs may not accurately reflect the burden of the diseases, which is short-lived, but typically intense. As a result, the QALYs gained because of an intervention that generates marginal gains for many people could be equal to the QALYs gained due to an intervention that generates substantial gains for a small number of people⁵⁵. These issues should be further explored. Most guidelines preferred meta-analysis, and required a systematic review of all evidence, because meta-analysis and systematic reviews were considered to be of the highest priority⁴². Moreover, although almost all guidelines agreed that costs and outcomes should be discounted, the discount rate varied substantially among guidelines.

In conclusion, this systematic review of PE guidelines identified areas of agreements and disagreements by comparing PE guidelines. Using this comparison, it can be summarized that methodological issues still remain and require further investigation.

Limitations

We have included a substantial number of PE/HE guidelines. It is, however, possible that we have missed some guidelines, e.g. guidelines published in non-English versions. We have also excluded guidelines that were not available at the time of our analysis. Furthermore, the accuracy of the data extracted from these guidelines depends on the interpretations of individual extractors. It may be the case that some extractors did not interpret the information consistently or accurately. However, we have reduced the bias, and improved the accuracy of our results. We believe our conclusions are still sufficiently accurate to provide a valuable reference.

Transparency

Declaration of funding

This manuscript has no sponsorship/funding.

Declaration of financial/other relationships

The authors declare that they have no conflicts of interests nor financial relationships that need to be disclosed. Peer reviewers on this manuscript have received an honorarium from JME for their review work, but have no other relevant financial relationships to disclose.

Acknowledgements

No assistance in the preparation of this article is to be declared.