Assessment of the quality of pharmacoeconomic evaluation literature in China

Abstract

Objective: To evaluate the quality of Chinese pharmacoeconomic-evaluation literature published between 2012–2014 retrieved from the Chinese National Knowledge Infrastructure (CNKI) in order to assess their adherence to recommendations of the Chinese Pharmacoeconomic Guidelines.

Methods: Identified literature was screened according to pre-specified criteria to access legibility for inclusion. Each included piece of literature was systematically compared against the recommendations proposed by the relevant Chinese guidelines.

Results: After culling, 259 studies were included in the comparative analysis. When compared to a previous study evaluating the quality of similar literature published between 1997–2007, the results showed improvements in certain technical aspects over the years. Particularly, an improvement was observed in more diverse evaluation methods being used, increased use of cost-utility analysis (2.43% in 2012–2014 vs 0.26% in 1997–2007) and use of discounting (45% in 2012–2014 vs 4.35% in 1997– 2007). In addition, a small number of studies were starting to apply modeling.

Conclusion: The quality of economic evaluation literature has improved in recent years, with more researchers realizing the importance and necessity of using discounting, sensitivity analysis, and modeling when conducting economic evaluation. This study also highlights certain important areas needing further attention when conducting economic evaluations in China. These include the ICER threshold of economic analysis, more detailed guidance in performing sensitivity analysis and modeling, as well as transferability of cost data across different regions. Overall, the results would support the positive contribution of the Chinese Economic Guideline in promoting economic evaluations in China.

Keywords:

• Pharmacoeconomics
• evaluation guidelines
• quality
• literature review
• comparative analysis
• China

Introduction

In recent years, pharmacoeconomic evaluation has been attracting increasing attention in China due to many factors affecting resource allocation in the healthcare system, including strained medical resources, and less than comprehensive health insurance coverage. As drug treatment plays an irreplaceable role in maintaining public health, it is not surprising that designing an effective drug policy has become an important focus in the recent Chinese healthcare reforms, starting in 2009. Hence, national authorities are particularly interested and highly value the important role of pharmacoeconomic evaluation in essential medicines inclusion, drug pricing, and medical insurance. This is evident as, since 2009, a number of national policies and opinions, such as the “CPC Central and State Council’s opinion on deepening reform of the healthcare system” were released to confirm the significance and importance of pharmacoeconomics¹. Congruent with the increased emphasis on pharmacoeconomics, the quantity of published pharmacoeconomic evaluation literature in China also increased year by year. However, pharmacoeconomic research in China started relatively late, and, compared with those published abroad, the quality of these Chinese studies would not be considered high, and most still suffered from some methodological issues.

In order to standardize and facilitate pharmacoeconomic research, the “Pharmacoeconomic Evaluation Guidelines of China” was published jointly in 2011 by the Chinese Pharmaceutical Association, China Association for Science and Technology, and the Chinese Medical Doctor Association². This guideline is a methodology guideline aiming to provide some recommended standard advice for performing economic evaluation. The guideline provides recommendations in 10 categories, including how to conduct pharmacoeconomic evaluation or study, covering topics such as research questions, study design, cost estimation, health outcomes, evaluation techniques, modeling analysis, variability, uncertainty, generalizability, and budget impact analysis. Since there is no mandatory requirement to use the Chinese Pharmacoeconomic Evaluation Guidelines, in order to evaluate its impact, it would be informative to assess the adherence to the recommendations in pharmacoeconomic studies performed in China since its publication. This would provide valuable insights for further improvement of pharmacoeconomic research in China, and facilitate its further potential mandatory use in assisting health resource allocation.

Methods

To assess the quality of published pharmacoeconomic studies against the published Chinese Pharmacoeconomic Guidelines (2011 edition), seven Level 1 items, and 14 Level 2 items (Table 1) from the Guidelines were selected as the criteria for assessment of adherence. At the same instant, we also attempted to identify the funding sources and affiliations of the researchers.

Table 1. Assessment criteria of pharmacoeconomic evaluation literature.

Level 1	Level 2	Assessment
Research question	Objective	Clarity of objective
	Perspective	Clarity and consistency
	Target population	Selection criteria of sample, and basis for grouping
	Comparators	Selection basis of the intervention and comparators
Study design	Types of study	Prospective Study, Retrospective Cohort Study, Mixing design (clinical trials combined with retrospective or data collection under actual conditions), or secondary literature study design
	Time horizon	Description test observations and rationale for selection
Evaluation techniques	Types of evaluations	Description and rationale of the method used
Cost	Cost classification	Types of cost including in research, and consistency with the research perspective
	Cost measurement	Obtaining and identification methods, discounting, and discount rates
Health outcomes	Output Indicators	Type of indicators and consistency with perspective, measurement methods, and reasons
	Presentation	Measurement presentation, discount, and the discount rate
Modeling analysis	————	Rationale for the use and choice of model (if used)
Discussions of results	Generalizability	The extent of applicability of results beyond study population or situation
	Study limitations	Justification of the use the sensitivity analysis (if used), justification of the assumptions used, and discussion of research limitations
	Others	Equity and Budget Impact Analysis

Database searching

Full-text database Chinese National Knowledge Infrastructure (CNKI) was searched from January 2012 to September 2014, using the keywords “pharmacoeconomics”, “cost-effectiveness analysis”, “cost-benefit analysis”, “cost-utility analysis”, “cost minimization analysis”, and “cost analysis”.

The title and abstract of the identified literature were scanned for relevance and suitability. A parallel approach was used when screening the potential literature for inclusion as well in the final analysis of the included literature. Any disagreement of relevance and suitability was resolved by discussion among the researchers. Full texts of literature deemed relevant were retrieved, and the reference sections of these manuscripts were further scanned manually for any missing literature. In our current study, we only included experimental research literature; conference reports, reviews, and non-experimental studies were excluded.

Results

In total, the number of identified papers from the database for the period 2012–2014 amounted to 1,035. By systematically removing conference reports, literature reviews, duplicate studies, and non-experimental studies^3–6, 259 studies were finally included for quality assessment. The culling process is shown in Figure 1.

Figure 1. Document screening and culling process.

First, the authorship and the institute affliction of the studies were examined. The first authors of most of the published literature were clinicians and pharmacists, with relatively few (12.4%) from academic institutions. Among the retrieved studies, 35.9% originated from hospitals for the evaluation of several drugs or therapies, with most evaluation focused on respiratory diseases and tumors. A summary of the relevant information of these retrieved manuscripts is presented in Table 2.

Table 2. Summary of retrieved studies (2012–2014) and degree of adherence to recommendations in Guideline.

Year Level	Study design						Evaluation methods						Cost		Sensitivity analysis performed
	Types of study				Time horizon (year)		CEA	CMA	CUA	CBA	CEA + CMAA	CEA + CUAA	D + I	D + I + IT
	RE	PR	MI	LI	1 ≤ X < 5	5 ≤ X
2012
Number	80	3	0	–	6	1	71	8	–	1	2	1	1	–	24
Ratio (%)(%)	96.4	3.6	1.0	–	7.2	1.2	85.5	9.7	–	1.2	2.4	1.2	1.2	–	28.9
2013
Number	38	65	0	1	7	1	83	13	3	–	5	–	5	–	52
Ratio (%)(%)	36.5	62.5	–	1.4	6.7	1.0	79.8	12.5	2.9	–	4.8	–	4.8	–	50.0
2014
Number	19	52	1	–	3	0	54	7	3	1	5	2	6	1	32
Ratio (%)(%)	26.4	72.2	–	–	4.2	0.0	75.0	9.7	4.2	1.4	6.9	2.8	8.3	1.4	44.4

RE: Retrospective study; PR: Prospective study; MI: Mixed study; LI: Literature Search; D + I: Direct + Indirect cost; D + I + IT: Direct + Indirect + Intangible cost; CEA:cost effectiveness analysis; CBA:cost benefit analysis ; CUA:cost utility analysis; CMA:cost minimization analysis.

Assessment of quality (as adherence to guideline recommendations)

Research questions

All of the studies stated the objectives clearly, these included rational drug use, comparison between imported and domestic drugs, economic analysis of several pharmacotherapeutic treatments for the same disease, and comparison of different treatment options.

Costs and outcomes of most studies were obtained based on the perspective of the patient or the hospital. The few exceptions used the societal perspective⁷, the perspective of health service providers⁸, and fund payer perspective⁹.

Almost all studies contained the selecting description of the target population. However, this was not specified in 21 studies, and 45 studies included control groups.

Study design

Among the identified articles, there were 120 (46.8%) randomized controlled trials, and 137 (53.5%) retrospective studies using data from hospital electronic patient records. The time horizon used in all studies was generally short (between 1–3 months), with more than 1 year in only 18 articles (7.03%)^10–27.

Research methods

In terms of evaluation methods, most retrieved studies used cost-effectiveness analysis (CEA). However, in a number of studies, CEA was still performed even when the results showed no statistically significant difference in treatment effect^28–35. In contrast, cost-utility analysis (CUA) was rarely used and was employed in only four studies (1.56%)^20,36–38. Results of CEA were mostly expressed by cost-effectiveness ratio (C/E) and the incremental cost-effectiveness ratio (ICER), but most of the CEA did not present ICER^39–53. In addition, many ICEA generally compared C/E of the group with the least effective comparator; in other words, the ICEA produced the cost-effectiveness ratio compared with the least effective group. Other studies selected the minimum cost scheme as the standard to interpret the increment ratios obtained^54–59. In contrast, several studies compared all programs when performing ICEA^60,61, but, as there is still no accepted or recommended ICER threshold value in China², the results were interpreted based on the subjective judgments of the evaluators^62–69.

Cost estimation

Most studies had clear definitions and classification of costs, with the cost divided into direct costs, indirect costs, and intangible costs. However, most would claim difficulties in estimating indirect costs and intangible costs, and proceeded to estimate direct cost only in their studies. To be precise, direct and indirect costs were included in 10 studies^{5,23,50,54,66,70–74}, while direct costs, indirect costs and intangible costs were presented in only one study⁸. For studies comparing several drugs or treatment programs in hospitals, besides drug cost, most would assume other direct costs to be identical or negligible. Only a few studies included different medical equipment, different length of hospital stay, and diagnostic costs into direct cost estimation^75–83.

For discounting, most studies did not use the National Treasury rates or discounted rates recommended by the guidelines to adjust the costs, but simply used the year’s prices^84–91 or retail price of hospital instead^{5,9,10,29,92–103}. Even for studies with a duration of more than 1 year, many researchers did not apply discounting at all. This would hinder making meaningful comparison with these studies in future.

Output indicators

For outcomes used in the economic analysis, most research reported the treatment results with the intervention in patients in the natural clinical situation, while only 11 studies reported the efficacy of the intervention based on randomized controlled conditions^{31,32,51,62,63,71,90,104–107}. Actually this would be desirable as the effect of the intervention was estimated taking into consideration real-life factors such as doctor’s diagnostic error, complexity of the disease, complications of the situation, patient’s compliance, etc.

Use of modeling

Few studies used appropriate modeling in their evaluation, and most of these originated from academic or research institutions. Among these few studies, Markov modeling was applied in three studies^14,20,23, and decision tree modeling in five studies^{9,52,108–110}.

Handling of uncertainties

Sensitivity analysis was performed to handle uncertainties in 108 studies (41.7%). These sensitivity analyses were mainly used to evaluate the change of estimates in assumptions and certain key variables, such as the price of drugs, hospital days, the cure rate, and the discount rate. Most studies (almost 88%) applied single factor sensitivity analysis by decreasing drug prices by 10%, but only 12% of studies applied multivariate analyses using different probability of the variables^{20,50,68,71,90,101,108,111–114}. However, without clear guidance or standard recommendations from the guidelines, the range used for the different variables was subjectively decided by the authors.

Discussion and recommendations

The main purpose of the Chinese Pharmacoeconomic Evaluation Guidelines is to provide standardized recommendations about various technical aspects in conducting pharmacoeconomic research in China. Therefore, it would be informative to assess its level of impact on studies performed since its publication. For this reason, our current study evaluated recently published literature in China against the existing Chinese pharmacoeconomic guidelines. A better understanding of the discrepancies between current pharmacoeconomic research and criteria as recommended in the Chinese guidelines would facilitate wider application of guidelines and improve the quality of evaluation studies.

Before the Chinese guideline was published, Li and Ma et al.¹¹⁵ had evaluated 201 pharmacoeconomic studies from 1997–2007, studying the perspective, evaluation method, discounting, cost, modeling, and sensitive analysis of each literature. A comparison of the results of the study by Li and Ma et al. and our current study is presented in Table 3. Based on the comparison, we could observe that there were improvements in certain areas over the years. Particularly, the improvement was seen in more diverse evaluation methods being used, increased use of cost-utility analysis (2.43% in 2012–2014 vs 0.26% in 1997–2007) and use of discounting (45% in 2012–2014 vs 4.35% in 1997–2007). Furthermore, a small number of studies (n = 8) were using modeling such as Markov and decision tree models. Overall, this comparison shows the quality of economic evaluation literature in China had improved in recent years.

Table 3. Analysis results of the literatures in 1997–2007 and 2012–2014.

Year	1997–2007^a	2012–2014^b
Number	201	286
Perspective (%)	4.37%	3.12%
Evaluation method
CUA (%)	0.26%	2.43%
CBA (%)	0.68%	0.78%
Comprehensiveness^c (%)	3.08%	5.47%
Use of Discount (%)	4.35%	45.00%
Use of Modeling (%)	–	2.8%
Use of Sensitivity analysis (%)	56.87%	57.90%

^a Results from Li MH, Ma AX et al.(2008).

^b Results from current study.

^c Studies applying two or more economic evaluation techniques (e.g. CMA, CEA, CUA, etc.).

However, despite the observed positive trends observed, based on our results, there are at least five areas that would need further improvement. These are discussed briefly in the following paragraphs.

1. Our study found that most current published studies did not explicitly state its research perspective and needed to be inferred, thus greatly reducing the usefulness to other users and decision-makers. In this respect, the Chinese Guideline recommends that there should be a clear perspective such as social/medical/patient, when conducting any pharmacoeconomic research.

2. When considering the types of economic analysis performed, CMA was rarely performed. There were a significant proportion of CEA studies being performed after no significant difference in efficacy between treatments was shown. Researchers should be advised that CEA/CUA is appropriate only when a statistical and clinical difference between treatments is established. Furthermore, in most ICEA, the result was compared with C/E_min (i.e. the ratio from the least effective comparator), or interpreted based on the evaluators’ subjective judgments only. The lack of an accepted or recommended ICER threshold value in China is noteworthy, it is recommended that a threshold value or range should be developed with collaboration of government and other stakeholders based on China’s economic situation and other relevant national conditions.

3. Furthermore, in estimation of cost, many studies unrealistically assumed that, besides drug costs, all other direct costs were identical. This ignores the efficacy of newer drugs which can significantly reduce the suffering of patients, loss of working time, the length of hospital stay, etc. Such assumptions would not capture comprehensively the financial benefits of the new treatment, and would potentially misguide and misinform the policy-makers. A more comprehensive and accurate evaluation of costs involved would greatly improve the applicability and generalizability of the evaluation.

4. Most studies used drug prices to perform univariate sensitivity analysis, but the existence of quite a huge inter-regional drug price difference in China would greatly affect the external validity of the results. In fact, the range of fluctuation in drug prices in China (and, for that matter, many other countries) is affected by many factors, such as patient demand, currency exchange rate fluctuation, inflation, bidding, etc., so how to best handle this uncertainty in drug price remains to be resolved. At this juncture, it is recommended to study the interaction between at least some of the more important parameters in multi-factor sensitivity analysis to better inform decision-making.

5. Probably due to a lack of expertise, modeling was seldom attempted in most of our included studies. For those few studies where modeling was attempted, the exercises were not properly conducted. Hence, learning about modeling techniques in foreign literature and guidelines and conducting training by Chinese academic institutes and experts in the field would gradually alleviate and improve this technical insufficiency.

Certainly, there are limitations in our present study. First, the guideline was released in 2011, and usually there would be a time lag before a wider adaptation by the scientific and clinical communities. Hence, we could not rule out the fact that some of the retrieved studies were conducted before the release of the guideline. In addition, despite our best effort, there existed the probability that we did not capture fully all the relevant literature of CNKI. However, we were reasonably satisfied that the captured articles would be a good representation of the status of pharmacoeconomic studies in China. Finally, the improvement observed over the years may not be totally attributable to the Guidelines, as experience gained over time by researchers would definitely be another contributing factor.

Conclusions

By comparing our results with a previous review, we conclude that the overall quality of economic evaluation literature published in China has improved in recent years after the publication of the Chinese Guidelines. However, there is still a significant deficiency in several areas that needs to be rectified for the quality of economic evaluation to be comparable with international standard. A better adherence to the recommendations from the Chinese Guidelines would contribute to achieving this.

Transparency

Declaration of funding

The authors declared that no sponsorship/funding was received for the project.

Declaration of financial/other relationships

None of the authors has any conflict of interest to declare. JME peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgement

The authors would like to express their gratitude to the staff at the study site for their assistance during the literature collection process.