Designing economic evaluations to facilitate optimal decisions: the need to avoid bias

Dear editor

Guertin et al¹ argue in their article “Bias within economic evaluations” that if researchers fail to incorporate the future availability of generics entrants for new patented drugs, the incremental cost-effectiveness ratio (ICER) will be overestimated.¹ Before addressing the validity of this argument, it is first worthwhile to consider the nature of both bias and economic evaluation.

Bias occurs due to an “error in the conception and design of a study … leading to results or conclusions that are systematically (as opposed to randomly) different from truth.”² The concept of bias is therefore specific to the purpose of the specific study and the question or decision problem it is designed to address. Economic evaluations address decision problems specific to individual decision makers through considering the opportunity cost associated with alternative decision options relating to the adoption of a technology at the time of the analysis.

Guertin et al challenge the use of current prices of branded pharmaceuticals in evaluating whether or not to fund them. They suggest that analysis should allow for the future availability of lower-cost generic equivalents. Before challenging the premise of this argument, it is worth considering the analytical approach adopted.

The authors assume that generics will become available at the expiry of the existing patent. In their case study of dabigatran, the patent expires in 2018.³ However, the availability of generic entrants is affected by the efforts of manufacturing companies to “evergreen” the patent.⁴ In 2011, additional patents were granted for dabigatran extending patent protection through 2024.³ The authors’ assumption of the earliest possible date for generic entry leads to the lowest possible estimate of the ICER.

The authors assumed that the cost of the generic equivalent would be 25% of the brand price. In some jurisdictions (such as Ontario as cited by Guertin et al), the cost of generic equivalents is set at a fixed percentage of the brand; however, as is the case in Ontario, this is frequently a function of the number of generic entrants approved for funding. The authors’ assumption of the lowest possible cost for a generic equivalent leads to the lowest possible ICER.

The authors addressed the latter two issues with limited sensitivity analysis. Further, assumptions relating to persistence and mortality rates used in the case study were also biased in favor of lowering the ICER, which despite the illustrative nature of the care study for the authors’ arguments may speak to larger issues of bias in economic evaluations.

Thus, on further exploration, assumptions made in the article led to the lowest possible estimate of the incremental cost of dabigatran, suggesting bias in the approach adopted.

Although we have demonstrated the problems in conducting such an analysis, it should be highlighted that the approach is fundamentally flawed in that it ignores the true context of the decision problem – whether or not to fund the technology now at the current price. The decision problem facing decision makers is specific to a particular timepoint – decision makers are concerned with whether a technology currently represents an appropriate use of scarce health care resources. Decision makers can make alternate decisions at later timepoints, but this relates to alternate decision problems that can be addressed by further evaluation. The decision to fund or not fund a particular technology, at a given time, should not preclude an alternate decision at a later timepoint when the context and/or the information have changed. Thus, if a product is not cost-effective at the current brand cost but may be at the generic price, it will be optimal for decision makers to not fund it at the current time but consider revising their decision at a future date, once a generic becomes available.

Guertin et al appear to raise an interesting point for discussion. However, what the authors criticize in their article is not bias in the conduct of economic evaluation but a true representation of the decision problem facing decision makers. Economic evaluations are designed to assist in making optimal decisions, thus maximizing the health benefits to be obtained from scarce health care resources. The approach suggested in this article is biased in both application and design and will lead to bias in the estimation of the true value of products, thus leading to nonoptimal resource allocation.

Disclosure

Karen Lee is a paid employee of CADTH. Doug Coyle is a member of the Ontario Ministry of Health’s Committee to Evaluate Drugs. Kathryn Coyle reports no conflicts of interest in this communication.

Authors’ reply

Jason R GuertinDominic MitchellFarzad AliJacques LeLorierCorrespondence: Jacques LeLorier, Centre de Recherche du Centre Hospitalier de l’Université de Montréal, 800 Rue St-Denis, Montréal, QC, H2X 0A9, Canada, Tel [phone] ext 12918, Fax [phone], Email [email protected]

Dear editor

We welcome the opportunity to reply to the letter by Lee et al in order to dispute some of the claims they make about our paper.¹ In their letter, Lee et al first propose a definition of the term “bias,” hinting that our choice of words within our title was inappropriate. We wish to remind Lee et al of a different definition of the term “bias”: “A range of factors that systematically influence the measures undertaken independent of the studied intervention; a tendency, intentional or unintentional, to inappropriately or unfairly favor one or more of the interventions being evaluated.”² As such, we maintain our position that bias does occur if the time to introduction of the generic entrant is applied differentially to the new drug than to the comparator drug.

Following this previous point, Lee et al go on to question our assumption that generic entrants of patented drugs would enter the market once the first patent of the drugs expire.¹ We agree with Lee et al that there is uncertainty regarding the exact moment when generic entrants of patented drugs will arrive and that this may not be at the time of expiration of their first patent. Indeed, there have been situations where the generic entrants arrived prior to the expiration of the patents and cases where they arrived later. Since the exact date of appearance of the generic entrants cannot be determined when the economic evaluation is being submitted, we believe that it would be good practice to deal with this uncertainty through the use of sensitivity analyses.

Finally, Lee et al argue that “The decision problem facing decision makers is specific to a particular timepoint.” We would like to remind Lee et al that the 2006 Canadian Guidelines for the Economic Evaluation of Health Technologies state that “It is good practice to anticipate future comparators, particularly lower cost technologies that may enter the market within the timeframe relevant to the analysis.”³ Furthermore, most decision makers require that economic evaluation be conducted with timeframes of 10 years or more and budget impact analyses which extend for 3 years or more. We thus strongly believe that the time of generic entry of a new drug should be taken into consideration within an economic evaluation and this practice is consistent with the 2006 Canadian Guidelines for the Economic Evaluation of Health Technologies. However, in the event that Lee et al still feel that the decision to fund or not a technology should be solely dependent on the current cost of both comparators, they should recommend that the next edition of the Canadian Guidelines on the Economic Evaluation of Health Technologies eliminate all reference to the future introduction of any generic entrants within the relevant time-horizon of the economic evaluation.⁴

Disclosure

The authors report no conflicts of interest in this work.