Transferability of health technology assessments and economic evaluations: a systematic review of approaches for assessment and application

Abstract

Background:

Health technology assessments (HTA) generally, and economic evaluations (EE) more specifically, have become an integral part of health care decision making around the world. However, these assessments are time consuming and expensive to conduct. Evaluation resources are scarce and therefore priorities need to be set for these assessments and the ability to use information from one country or region in another (geographic transferability) is an increasingly important consideration.

Objectives:

To review the existing approaches, systems, and tools for assessing the geographic transferability potential or guiding the conduct of transferring HTAs and EEs.

Methods:

A systematic literature review was conducted of several databases, supplemented with web searching, hand searching of journals, and bibliographic searching of identified articles. Systems, tools, checklists, and flow charts to assess, evaluate, or guide the conduct of transferability of HTAs and EEs were identified.

Results:

Of 282 references identified, 27 articles were reviewed in full text and of these, seven proposed unique systems, tools, checklists, or flow charts specifically for geographic transferability. All of the seven articles identified a checklist of transferability factors to consider, and most articles identified a subset of ‘critical’ factors for assessing transferability potential. Most of these critical factors related to study quality, transparency of methods, the level of reporting of methods and results, and the applicability of the treatment comparators to the target country. Some authors proposed a sequenced flow chart type approach, while others proposed an assessment of critical criteria first, followed by an assessment of other noncritical factors. Finally some authors proposed a quantitative score or index to measure transferability potential.

Conclusion:

Despite a number of publications on the topic, the proposed approaches and the factors used for assessing geographic transferability potential have varied substantially across the papers reviewed. Most promising is the identification of an extensive checklist of critical and noncritical factors in determining transferability potential, which may form the basis for consensus of a future tool. Due to the complexities of identifying appropriate weights for each of the noncritical factors, it is still uncertain whether the assessment and calculation of an overall transferability score or index will be practical or useful for transferability considerations in the future.

Keywords:

• costs and cost analysis
• economic evaluation
• health technology assessment
• geographic transferability
• portability
• generalizability

Introduction

Over the past two decades, decision-makers at all levels of the health care system have been faced with increasing pressure to make more efficient use of existing health care resources. As a result, public and private agencies worldwide have turned to evidence-based processes to better assess the clinical and economic benefits of both new and existing health care technologies. Although safety and efficacy are essential first considerations, health technology assessment (HTA), in general, and economic evaluation (EE), specifically, have become an integral component of the overall decision-making process regarding the assessment and adoption of both new and existing health care technologies. Full EE, whether conducted alongside a trial or as a modeling study, assesses both the costs and consequences of a health care intervention, which is evaluated in comparison to at least one other intervention. The general rule when assessing programs is that the difference in costs is compared to the difference in outcomes achieved, typically in an incremental analysis. Therefore, the basic tasks of economic evaluation are to identify, measure, value, and compare the costs and consequences of the alternatives under consideration.¹

The basic types of economic evaluation include cost-benefit analysis (CBA), cost-effectiveness analysis (CEA), and cost-utility analysis (CUA). CBA measures and values the benefits and costs of outcomes achieved from a program or intervention in monetary terms. For example a CBA may require that a dollar value be placed on expected years of life gained or expected improvements in health and wellbeing. The benefits and costs that are counted include not only those directly attributed to the program or intervention but also any indirect benefits or costs through externalities or other third-party effects.² However, much of the controversy surrounding the use of CBA derives from the fact that consumers of health care are not used to valuing health outcomes and these other intangible benefits. Given these difficulties, CEA provides a more practical approach to health care decision-making. It compares the costs of achieving particular health effects that are measured in natural units related to the objective of the program, such as average cases of disease avoided, or years of life gained. The results of such comparisons are stated in terms of cost per unit of effect. CUA is often seen as a special form of CEA that introduces measures of benefits that reflect individuals’ preferences over the health consequences of alternative programs that affect them. CUAs use a global measure of health outcome, such as quality-adjusted life-years (QALYs) by undertaking one program instead of another, and the results are often expressed as a cost per QALY gained.¹ This enables the comparison of different types of programs, which makes CUA more practical for decision-makers. Sometimes a cost-minimization analysis (CMA) may be performed if the alternatives under consideration are considered to achieve the given outcomes to the same extent. However, CMA is usually not viewed as a separate form of full economic evaluation because the original intent of the study was to conduct either a CEA or CUA.¹

Approaches to conducting full economic analyses can be categorized as either trial-based studies using patient-level data or decision analytic modeling based on secondary data. Given that randomized clinical trials are typically a necessary condition for the successful licensing of a pharmaceutical, economic data are sometimes collected by piggybacking onto the same trials. This approach is potentially more attractive for internal validity, while the main limitation is that this study may suffer from external generalizability. Decision analytic modeling brings together a range of evidence sources and allows the expansion of the comparators considered in the analysis and an expansion of the time horizon beyond the trial period. In addition, decision analytic modeling provides a framework for informing specific decision-making under conditions of uncertainty by allowing more convenient assessment of modeling assumptions, modeling structural uncertainty, and different patient subgroups (heterogeneity).¹

These evaluations, either done as a trial or modeling study and either as an EE or as part of a broader HTA report, are usually time-consuming, expensive, and demanding in terms of statistical sophistication and in terms of the researchers equipped with the necessary skills. As a result, it is not possible to conduct an EE or HTA on every intervention. Fortunately, health care decision-makers often have access to previously published EEs or HTAs on the topic of interest. Unfortunately, it is commonly the case that these EEs or HTAs are from another jurisdiction and decision-makers need to assess whether, and to what extent, the assessment and analysis from this other jurisdiction applies to their own jurisdiction. Several terms such as geographic transferability, generalizability, portability, and extrapolation have been used to describe the process of applying the analyses and results of EEs or HTAs from one jurisdiction to another jurisdiction. Before considering conducting an EE or HTA for the jurisdiction of interest, it is important to first consider whether an EE or HTA on the topic of interest may already exist from another jurisdiction, and if so, the potential applicability of transferring or using these results for the country of interest. This is especially useful for decision-makers who face budget or time constraints in health care resource allocation decisions and given the increasing number of technologies that need to be assessed. As a result of the growing demand in many jurisdictions to use evidence as part of their decision-making process, there is growing interest in the assessment of transferability of EEs and HTAs, with the aim to decide the extent to which the results of a study from another jurisdiction can be adapted locally.

There have been a series of important contributions to the published literature in recent years addressing the issue of geographic transferability of EE and HTA results. Several efforts were made to identify potential factors causing variability in EE and HTA data between locations. One of the first researchers who helped to progress the debate about transferability was O’Brien,³ who identified six threats to the transferability of economic data across countries. Since then the list of the factors affecting transferability has been further expanded through a number of review papers on the topic. For example, in a systematic review by Goeree⁴ a comprehensive classification system was developed of over 80 factors grouped into 5 broad categories of variability factors including characteristics of the patient, the disease, the health care provider, and the health care system. In addition to factors which may raise transferability concerns, authors have proposed systems, checklists or even indices to assess or measure the transferability potential of EEs and HTAs conducted in other jurisdictions to the jurisdiction of interest. However, no previous study has summarized these papers. Therefore, the purpose of this study was to review and summarize the existing approaches, systems, and tools for assessing transferability potential or guiding the actual conduct of transferability of HTAs and EEs.

Methods

Literature search strategy

Targeted literature search strategies were developed for PubMed, Ovid MEDLINE, and EMBASE databases respectively. Detailed search terms are provided in the Appendix. The search focused on relevant papers reflecting the methods, systems, or tools used to assess transferability or generalizability of economic evaluation results across jurisdictions. Studies published in English were considered, without publication year restriction. The literature search was supplemented with online searching using the Google search engine, hand searching of relevant journals, and reviewing of the bibliographies of identified articles. The search was run on 22 November 2010. All references obtained were incorporated into a reference manager database to check for duplication and to perform title and abstract screening.

Literature selection criteria

Studies meeting our selection criteria were those that had a focus on methods or tools applied to assess or evaluate transferability. The tools could consist of a checklist, toolkit, system, practice guideline, set of criteria, decision chart, or index. Articles were excluded if they were general narrative reviews, if they focused on identification of sources of variability in EEs of HTAs or factors likely to affect transferability between locations, or if they addressed the analytical approaches of adapting cost-effectiveness results to a particular jurisdiction of interest.

Synthesis and reporting

It was determined ‘a priori’ that due to the nature of the different proposed systems, a quantitative summary (eg, meta-analysis) of retrieved studies was not feasible. As a result, each paper is presented sequentially by year of publication with a qualitative description of the tool, checklist, criteria, system, or flow chart. Generalizations and conclusions across these approaches are provided following a description of each approach.

Results

In total, the comprehensive search identified 282 citations. Among them, 230 articles were identified from PubMed, and 52 additional articles were identified from Ovid MEDLINE and EMBASE. A Google search did not identify any additional citations. Title and abstract screening was carried out for relevance to the topic and resulted in the inclusion of 23 articles, which were retrieved in full text and reviewed in detail. Four additional references were identified during bibliography searching. Among the 27 articles reviewed in full text, seven papers with specific focus on approaches to assess the transferability of economic evaluations across geographic locations were included in the review. Please refer to Figure 1 for a flow diagram of study selection. A description of the system or approach from each of the seven papers, by year of publication, is provided below along with a summary of the application of system or approach if available.

Figure 1 Flow diagram of literature selection for systematic review.

Abbreviation: CE, cost effectiveness.

Heyland’s generalizability criteria (1996)

Heyland⁵ introduced a list of criteria when assessing the generalizability of economic evaluations to their own setting in order to improve the efficiency of their critical care unit. The criteria were based on ten key variability factor-based questions regarding clinical generalizability and health care system generalizability (see Table 1). Before applying these criteria, selected studies also need to meet the inclusion criteria for minimal methodologic standard, which includes a comprehensive description of competing alternatives, sufficient evidence of clinical effectiveness or efficacy, appropriate identification, measurement and valuation of all important costs, and appropriate sensitivity analysis that takes into account all estimates of uncertainty.

Table 1 Heyland’s generalizability criteria

Clinical generalizability Are the patients described in the analysis similar to those patients you see in your own setting? Systems generalizability Is the viewpoint of the analysis relevant to your clinical setting/situation? Is the intervention under study generalizable to your setting? (ie, despite good clinical evidence, is such a program available or likely to be available in your setting?) Are the costing methods applicable to the health care system in which you work? Is the unit price for drugs, physician fees, laboratory tests, etc the same? Is the mix of resources consumed the same? Is the volume of patients, and therefore, the average cost per patient, similar across systems? Can you convert exchange rates across countries appropriately? Are the outcomes measured appropriate to your setting? Was a method to measure the outcomes compatible with the current methods utilized in your setting? If a preference-based measure was used, is there evidence that the preferences of your patients are the same as those preferences used in the analysis? Is the discount rate applicable to your setting?

Source: Heyland DK, et al. Crit Care Med. 1996;24(9):15.

The authors identified 29 critical care studies (19 CEAs, 6 CMAs, 4 partial economic evaluations that only compared costs of alternative interventions) that met the minimum level of methodological rigor to be further evaluated for generalizability to their own intensive care unit in a tertiary care hospital in Hamilton, ON. Of the 29 studies, 14 adequately described both competing alternatives; 17 provided sufficient evidence of clinical effectiveness; and six identified, measured, and evaluated costs appropriately. Overall, it was determined that none of the 29 papers met the minimum methodological standard for all criteria. Four papers that provided adequate costs and clinical effectiveness/efficacy were further evaluated using the proposed generalizability criteria from a hospital perspective. Overall, it was found that the differences in costing methods precluded the generalizability of three of these studies; however, for two of them, the direction or qualitative result was probably generalizable, whereas the magnitude or quantitative result was not.

Späth’s transferability indicators (1999)

Späth⁶ defined five indicators to assess the eligibility of economic evaluations for transfer to a given health care system (see Table 2). Similar to the approach proposed by Heyland,⁵ before applying the transferability criteria, Späth proposed four criteria critical for internal methodological validity (ie, perspective of the study from national level, comparison of two or more options, description of the evaluated therapies, and the assessed therapies and/or its comparators are used in the health system of interest) that need to be satisfied before consideration of transferability indicators. Studies fulfilling the four critical criteria are then assessed for eligibility for transferability using a five-indicator checklist. The five indicators comprise three dimensions including the settings at local level (potential users of the economic evaluation and characteristics of the patient population) in which the studies might be used, the transferability of health outcome data, and the transferability of resource utilization data (health care resources used and their unit prices). Studies from other health systems and settings must fulfill all of the five indicators in order to be considered transferable.

Table 2 Späth’s transferability indicators

The setting(s) in which the studies might be used: Potential users Characteristics of the treated patient population The transferability of health outcome data: Health outcome data The transferability of resource utilization: Health-care resources Unit prices of health-care resources and discount rates

Source: Späth HM, et al. Health Policy. 1999;49:165–166.

To test the proposed checklist, 26 published economic evaluations regarding adjuvant therapy in women with breast cancer were identified to apply to the French health care system. Six of these studies met the critical appraisal of internal validity to be assessed further for transferability. These studies covered different types of economic evaluation (4 CUA, 1 CEA, 1 CMA; 3 decision analyses, and 3 clinical trial results with cost estimations). All of the evaluations included direct medical costs, but no other types of resource utilization information were included in the studies and the related unit prices were not reported. Therefore, none of the six studies were determined to be eligible for transfer to the French health care system.

Welte’s transferability decision chart (2004)

One of the best known transferability assessment tools was published by Welte⁷ a few years later. As shown in Figure 2, Welte proposed the use of a transferability decision chart that takes into account ‘knock-out’ criteria, a transferability checklist, and methods for improving transferability and for assessing the uncertainty of transferred results. The decision chart starts with general ‘knock-out’ criteria to identify studies that are deemed not transferable. These general ‘knock-out’ criteria include:

• The relevant technology is not comparable to the one that shall be used in the decision country.

• The comparator is not comparable to the one that is relevant to the decision country.

• The study does not possess an acceptable quality.

Figure 2 Welte’s transferability decision chart.

Source: Welte R, et al. Pharmacoeconomics. 2004;22(13):863.

Note: *Indicates order of magnitude can be transferred.
Abbreviation: CER, cost-effectiveness analysis ratio.

If the general ‘knock-out’ criteria have been passed, a transferability checklist is then applied to test for specific ‘knock-out’ criteria and to determine whether modeling adjustments are necessary. The checklist is based on 14 factors of methodological, health care system and population characteristics. Although not a general ‘knock-out’ criteria, each relevant transferability factor from the checklist can become a specific ‘knock-out’ criterion if the factor/criterion cannot be assessed because of lack of data from the study or the decision country. The specific criteria checklist is listed below:

• Perspective

• Discount rate

• Medical cost approach

• Productivity cost approach

• Absolute and relative prices in health care

• Practice variation

• Technology availability

• Disease incidence/prevalence

• Case-mix

• Life expectancy

• Health-status preference

• Acceptance, compliance, incentives to patients

• Productivity and work-loss time

• Disease spread

For each transferability factor, it has to be determined 1) to what extent it is relevant for the investigated technology; 2) the correspondence between the study country and the decision country; and 3) the likely effect of the transferability factor on the results.

If any of the general or specific criteria applies, then users should examine the transferability of the study design rather than the study results. For studies that pass all the ‘knock-out’ criteria, the need for modeling-based adjustment to improve transferability will be evaluated based on data availability. For studies in which modeling adjustments are necessary, the decision chart will also guide the user to identify whether the study results are transferable after adjustments either in a descriptive way (reapply the decision chart) or by doing probabilistic or multivariate sensitivity analysis.

The transferability decision chart was then applied to several international cost-effectiveness studies in the areas of interventional cardiology, vaccination, and screening. Application of the transferability checklist to an American multicenter study of stenting versus percutaneous transluminal coronary angioplasty (PTCA) by Cohen⁸ revealed that modeling adjustments would be necessary for the German Ministry of Health, however, because resource valuation was not presented in enough detail, the results of this study could not be transferred. Another Belgium Netherlands Stent Study by Serruys⁹ also showed that modeling adjustments were needed for Germany. The study presented sufficient details, which enabled the adjustment of resource valuation using data derived from the German study. The adjusted study results were then evaluated for transferability by applying the transferability checklist again, and the authors concluded that transferring the order of magnitude of the adjusted EE result seemed defendable, but not the exact value. The third case was an example where Denmark was interested in a Dutch EE that used a dynamic model instead of the common static decision-analysis model to project the cost effectiveness of a large-scale chlamydial screening program. General ‘knock-out’ criteria were fulfilled and the transferability of the study model was examined. Although it was found that transferring the economic model was laborious, it was determined that the progression component of the disease model and the productivity cost module could easily be transferred. Because these models take much more time to build than to adjust from other contexts, it was estimated that transferring the model saved approximately 6 years of work.

Boulenger’s transferability information checklist (2005)

In the article by Boulenger,¹⁰ the authors developed a transferability information checklist and used it to generate a score that represents the percentage of applicable items that were either adequately or partially addressed in economic studies. The checklist is shown in Table 3. The overall checklist comprises 42 questions and includes two parts: the first part is related to overall methodological quality and internal validity of the studies; the second part (transferability sub-checklist) contains 16 items that are more specifically focused on judging the transferability of a study. The extended checklist was divided into six main sections: the subject and key elements of the study (Q1-M2), characteristics of the methods used to measure clinical outcomes (E1-E7), the measure of health benefits used in the economic analysis (B1-B5), the costs (C1-C11), discounting (D1-D4), and discussion by the authors (S1, O1). To quantitatively evaluate how thoroughly key methodological items regarding transferability are addressed and reported in each study, a percentage score of transferability information can be derived for both the full checklist and the sub-checklist after assigning each item one of the following values: 1 for ‘yes’, 0.5 for ‘partially’, and 0 for ‘no/no information’.¹¹ Based on the responses for each item in the checklist, Boulenger generated an overall transferability score based on the assumption that each item or factor contributes equal weight to the overall score. In Boulenger’s system, there are no critical criteria as each criterion contributes equally to the overall score.

Table 3 Boulenger’s transferability information checklist

Each question is answered with ‘yes’, ‘partially’, ‘no/no information provided’ or ‘not applicable’
Q1	Is the study question clearly stated
Q2	Are the alternative technologies justified by the author(s)?
HT1^a	Is the intervention described in sufficient detail?
HT2^a	Is (are) the comparator(s) described in sufficient details?
SE1	Did the authors correctly specify the setting in which the study took place (eg, primary care, community)?
SE2^a	Is (are) the country(ies) in which the economic study took place clearly specified?
P1^a	Did the authors correctly state which perspective they adopted for the economic analysis?
SP1^a	Is the target population of the health technology clearly stated by the authors or when it is not done can it be inferred by reading the article?
SP2	Are the population characteristics described? (eg, age, sex, health status, socio-economic status, inclusion/exclusion criteria)
SP3^a	Does the article provide sufficient detail about the study sample(s)?
SP4	Does the paper provide sufficient information to assess the representativeness of the study sample with respect to the target population?
M1	If a model is used, is it described in detail?
M2	Are the origins of the parameters used in the model given?
E1	If a single study is used, is the study design described (sample selection, study design, allocation, follow-up)?
E2	If a single study is used, are the methods of data analysis described (iTT/per protocol or observational data)?
E3	If based on a review/synthesis of previous published studies, are review methods described (search strategy, inclusion criteria, sources, judgment criteria, combination, investigation of differences)?
E4	If based on opinion, are the methods used to derive estimates described?
E5^a	Have the principal estimates of effectiveness measure been reported?
E6	Are the side effects or adverse effects addressed in the analysis?
E7^a	Does the article provide the results of a statistical analysis of the effectiveness results?
B1	Do the authors specify any summary benefit measure(s) used in the economic analysis?
B2	Do the authors report the basic method of valuation of health states or interventions?
B3	Do the authors specify the source(s) of health states (eg, specific patient populations or the general public)?
B4	Do the authors specify the valuation tool used?
B5^a	Is the level of reporting of benefit data adequate (incremental analysis, statistical analyses)?
C1^a	Are the cost components/items used in the economic analysis presented?
C2	Are the methods used to measure cost components/items provided?
C3	Are the sources of resource consumption data provided?
C4	Are the sources of unit price data provided?
C5^a	Are the unit prices for resources given?
C6^a	Are costs and quantities reported separately?
C7^a	Is the price year given?
C8	Is the time horizon given for each element of the cost analysis?
C9^a	Is the currency unit reported?
C10	Is a currency conversion rate given?
C11	Does the article provide the results of a statistical analysis of cost results?
D1	Was the summary benefit measure(s) discounted?
D2	Were the cost data discounted?
D3	Do the authors specify the rate(s) used in discounting costs and benefits?
D4	Were discounted and not discounted results reported?
S1^a	Are quantitative and/or descriptive analysis conducted to explore variability from place to place?
O1^a	Did the authors discuss caveats regarding the generalizability of their results?

Note:

a Items comprising the transferability sub-checklist.

Source: Boulenger S, et al. Eur J Health Econom. 2005;4:337–338.

The checklist was applied to 25 studies to assess the level of transferability of EEs of health care programs between the UK and France. Selected studies included 17 multi-country studies that included study sites in France and the UK, along with eight single-country studies from France or the UK, which evaluated the same health technology using the same methodology. Cost and/or cost-effectiveness ratios in these studies were converted using purchasing power parities (PPP), with the Euro as the common currency. These enabled comparisons of the results between the two countries. The mean percentage transferability information score for the full checklist was 66.9% ± 13.6%, and the results obtained for the sub-checklist (68.8% ± 15.1%) were very similar to the full-checklist, suggesting a high correlation between overall study quality and a good transferability score. The potential relationship between the transferability information score and the characteristics of the study (multi-country vs single-country) was explored using χ² test; however, no statistically significant differences were found. Comparisons for the 25 economic evaluations revealed that in the vast majority of cases, the cost-effectiveness ratio for France was more favorable than that of the UK. Therefore, the authors pointed out that if this finding was confirmed by further studies, the implication would be that a decision-maker in France, on seeing a study performed in the UK, could be fairly confident that even more favorable results would be obtained for France. However, it is difficult to determine the extent to which the results would be more favorable due to the wide range of difference (1.5%–250%). For the 17 multi-country studies, five enabled direct comparisons, and the source of difference was merely a price effect (unit costs).

Drummond’s application algorithm (2009)

An International Society of Pharmacoeconomics and Outcomes Research (ISPOR) Task Force on Good Research Practices lead by Drummond,¹² proposed a 4-step application algorithm that determines the appropriate methods for adjusting cost-effectiveness estimates based on data availability. The authors considered three types of situations where cost-effectiveness results are not transferable or in other words there are three critical criteria: 1) if either the experimental technology or the comparator(s) are not relevant in the jurisdiction of interest; 2) if the methodological quality of the studies doesn’t meet local standards, which is similar to Welte’s general ‘knock-out’ criteria; 3) if the study population is different between jurisdictions. Furthermore, they developed an application algorithm (see Figure 3) to assess whether the study results are directly transferable through simple adjustment procedures for practice variations, unit health care costs, the settings, time horizon, discount rates, and productivity/time costs (ie, similar to Welte’s specific ‘knock-out’ criteria), or whether more elaborate adjustment is needed which analyzes individual patient data or uses decision-analytic modeling techniques depending on whether the jurisdiction of interest is within a multi-location study that has been conducted. The authors further addressed analytical methods of using trial-based patient data, and under what circumstances decision-analytic models should be considered.

Figure 3 Drummond’s steps for determining appropriate methods for adjusting cost-effectiveness information.

Source: Drummond MF, et al. Value Health. 2009;12:411.

Abbreviation: CE, cost effectiveness.

Turner’s transferability checklist (2009)

The next system proposed was by Turner¹³ as part of the European network for Health Technology Assessment (EUnetHTA) project.¹⁴ These authors and contributors developed an adaptation toolkit that contains a series of checklists, questions, and resources to guide the users through the process of selecting possible relevant material, assessing the relevance, reliability, and transferability of HTA reports from other settings or jurisdictions, and adapting the information for the desired context of their own setting. The toolkit is structured into two sections: 1) speedy sifting – a screening tool which enables rapid screening of existing HTA reports to assess the relevance of the HTA report for adaptation; 2) main toolkit – a more comprehensive tool with questions on reliability and issues regarding transferability. The speedy sifting section contains eight questions:

1. Are the policy and research questions being addressed relevant to your questions? (Yes/No)

2. What is the language of this HTA report? Is it possible to translate this report into your language? (Yes/No)

3. Is there a description of the health technology being assessed? (Judgment needed)

4. Is the scope of the assessment specified? (Judgment needed)

5. Has the report been externally reviewed? (Judgment needed)

6. Is there any conflict of interest? (Judgment needed)

7. When was the work that underpins this report done? Does this make it out of date for your purposes? (Judgment needed)

8. Have the methods of the assessment been described in the HTA report? (Judgment needed)

Users are supposed to undertake the speedy sifting for a quick assessment of the relevance of the report for adaptation before using the more comprehensive tool. Based on answers to the eight questions, users can make their own judgment on whether to: 1) proceed to the main part of the toolkit, 2) seek further information, or 3) end the adaptation process and consider creating a new local HTA report. To that extent, these first eight questions are similar to critical criteria proposed in early systems.

The main part of the toolkit (see Table 4) comprises five domains in the form of a checklist: technology use, safety, effectiveness or efficacy, economic evaluation, and organizational aspects. Each of the domains includes a series of questions that enable the assessment of a report’s reliability, specific relevance, and transferability. Answers to these questions help the users extract information from the corresponding sections of the HTA report, and incorporate it within an HTA report in their own setting. There may be a need to update the data and supplement with local context data. The main toolkit can be used in its entirety in all five domains or can be used to adapt information in one or more domains. If after going through the main toolkit it is found that no part of the HTA report was reliable and/or transferable, then users should not consider transferability and should consider creating a new local HTA report.

Table 4 EUnetHTA adaptation toolkit

Technology use domain To assess relevance: 1. What is the research question considered? Is the research question considered within this section of the report relevant to your question? To assess reliability: 2. Were conditions, target group, relevant interventions, or comparisons between interventions and relevant outcomes appropriately defined? 3. Is the information provided on technology use and development complete and comprehensive? Are the methods and sources used when elaborating the background information well documented? 4. Are patterns of utilization, diffusion, indications, and time trends adequately described? 5. Is an analysis of the regulatory status of the technology provided (market admission, status in other countries)? To assess transferability: 6. Is there any consideration of when and how technical characteristics affect outcomes? 7. Are there any differences in the use of this technology within the target setting (compared to the uses described in the HTA report for adaptation)?
Safety domain To assess relevance: 1. Were harms or safety assessed? 2. Is the scope of the safety assessment relevant to your question? To assess reliability: 3. Was the search for studies reasonably comprehensive? 4. Were special sources consulted?: disease registers, data routinely collected (on utilization, costs, adverse effects …), consumer associations, etc … The aspects that should be assessed concerning the sources of safety data are: 5. What are the sources of information/data? eg, surveillance databases, declaration of incidents, safety report, RCT, case reports Quality of the safety assessment (ie, appraisal of evidence) 6. Were the criteria used for deciding which studies to include in the HTA report reported? 7. Was bias in the selection of studies avoided? 8. Did the selection of studies (in particular the choice of eligible study designs) minimize the possibility of including studies with a high propensity for bias? 9. Were the criteria used for assessing the validity of the included studies reported? 10. Were the inclusion criteria used for the primary studies appropriate to the study questions posed by the HTA report? Were the criteria used to assess the validity of the primary study appropriate? 11. What risks have been reported? How were these measured? 12. Were the study outcomes valid? Were they pertinent? 13. Are the number of patients, their representativeness, and the quality of the data high enough to exclude a modest but clinically relevant rate of serious complications? ie, what is the potential for overlooking a possible serious adverse event? 14. Is there a possibility for a ‘class’ effect adverse reaction or safety problem? c) To assess transferability: 15. Does the population described for eligibility match the population to which it is targeted in the target setting? 16. Are there any reasons to expect differences in complication rates (eg, epidemiology, genetic issues, health care system [quality of care, surveillance])? 17. Are the requirements for its use (special measures needed for use/implementation, maintenance etc) available in the target setting? 18. Is the necessary expertise (knowledge and skills) available in the target setting? 19. Is safety particularly dependent on training? Are there types of teams to which the procedure should be limited for safety reasons? Is there a need for special training or certification to deliver the intervention properly? Would it be possible (affordable) to organize such training, if any?
Effectiveness (including efficacy) domain To assess relevance: 1. What is the research question considered? Is the research question considered within this section of the HTA report relevant to your HTA question? 2. Are the outcome measures relevant for your HTA question? 3. Were the search methods used to find studies relevant to the main question(s) stated? To assess reliability: 4. Was the search for studies reasonably comprehensive? 5. Were the criteria used for deciding which studies to include in the HTA report reported? 6. Was the bias in the selection of studies avoided? 7. Did the selection of studies (in particular the choice of eligible study designs) minimize the possibility of including studies with a high propensity for bias? 8. Were the criteria used for assessing the validity of the included studies reported? 9. Was the validity of all studies referred to in the text assessed using appropriate criteria (either in selecting studies for inclusion or in analyzing the studies that are cited)? 10. Were the methods used to combine the findings of the relevant studies (to reach a conclusion) reported? 11. Were the findings of the relevant studies combined appropriately with respect to the main question the HTA report addresses? 12. Were the conclusions made by the authors supported by the data and/or analysis reported in the HTA report? 13. How likely is it that the relevance of this HTA report has changed due to additional research that had started, completed or been published since this HTA report? To assess transferability: 14. Would you expect the baseline risk of patients within your own setting to be the same as the baseline risk of those patients considered within the HTA report for adaptation? (assuming that patients receive the same treatment and same comparator) We would expect the relative risk to be the same and baseline risk different. The user needs to consider the impact of local epidemiological and demographic data on the baseline risk.
Economic evaluation domain
To assess relevance and reliability: 1. Was a well-defined question posed in an answerable form? 2. What is the question being asked in the report? Is the economic question relevant? What type of economic analysis is being performed to answer the question (ie, cost-minimization, cost consequences analysis, cost-effectiveness analysis, cost-utility analysis, cost-benefit analysis)? 3. Has the viewpoint or perspective for the analysis been stated clearly, along with the reasons for this choice? Is it a societal perspective, third-party payer perspective, or patient perspective? Is the analysis presented in a disaggregated fashion showing these perspectives separately? 4. Was a comprehensive description of the competing alternatives given (ie, can you tell who did what to whom, where and how often)? 5. Has the study included a comparison of alternative treatments for patients with the same clinical condition? Are those alternatives explicitly stated? Are the alternatives chosen valid and reasonable? 6. Has the evidence of the product’s efficacy been established through randomized trials? Has the evidence of efficacy been supplemented by evidence of effectiveness applicable to the patient population or subgroups considered in the study? Has the latter evidence been derived from studies documenting routine use in clinical practice? Have all the relevant and significant variations in effectiveness for different subgroups been identified and reported? 7. Was the effectiveness of the programmes or services established? 8. Are the methods and analysis displayed in a clear and transparent manner? Are the components of the numerator (cost of each alternative) and denominator (clinical outcomes of each alternative) displayed? Are clinical outcomes expressed first in natural units and then translated into alternative units, such as benefits or utility? 9. Are all important and relevant costs and consequences (outcomes), including adverse effects for each alternative identified? 10. Were costs and consequences measured accurately in appropriate physical units (eg, hours of nursing time, number of clinician visits, lost work-days, gained life-years)? 11. How is health related quality of life (HRQoL) measured? 12. Is HRQoL an important component of an economic analysis for this question? Based on the sensitivity analysis how sensitive is the estimate of cost-utility to variations in HRQoL? 13. Were costs and consequences valued credibly? 14. Were costs and consequences adjusted for differential timing? 15. Are costs and consequences modeled (as a decision tree) with information derived from a variety of sources or estimated directly from specific patient population(s)? 16. Are capital costs and overhead costs included as well as operating costs? How are they measured? 17. How have indirect costs (ie, productivity costs, cost of lost time) been identified and estimated? 18. For variables that are difficult to measure, what method is used to handle this difficulty? Does this method slant the analysis all in favor of one intervention in order to bias the analysis against the expected result? 19. Was an incremental analysis of costs and consequences of alternatives performed? 20. Was allowance made for uncertainty in the estimates of costs and consequences? 21. Were adequate sensitivity analyses undertaken, ie, when parameters with high uncertainty were analyzed, did the direction of the results change? 22. If a stochastic sensitivity analysis was applied, are the underlying distribution functions justified? 23. What equity assumptions have been made in the analysis? eg, are QALYs gained by any individual considered equal? 24. Is the incremental cost-effectiveness ratio estimated for a specific clinical indication that represents the majority of all of its expected use by those covered under the programs operated by the decision-makers to whom the report is addressed? Are there other indications, which have not been considered which involve a large amount of utilization for which the ratio may be very different? 25. Is there an estimate of the aggregate incremental expenditure required for the decision-makers to whom the study is addressed, to provide this product to patients covered by their programs? What is the estimate of aggregate incremental costs? Does this estimate cover all of the major indications for use of the product? 26. Did the presentation and discussion of the study results include all issues of concern to users?
To assess transferability: 27. How generalizable and relevant are the results, and validity of the data and model to the relevant jurisdictions and populations? 28. Are there any differences in the following parameters? Perspective Preferences Relative costs Indirect costs Discount rate Technological context Personnel characteristics Epidemiological context (including genetic variants) Factors which influence incidence and prevalence Demographic context Life expectancy Reproduction Pre- and post intervention care Integration of technology in health care system Incentives If differences exist, how likely is it that each factor would impact the results? In which direction? Of what magnitude? Taken together, how would they impact the results and of what magnitude? Given these potential differences, how would the conclusions likely change in the target setting? Are you able to quantify this in any manner? 29. Does the economic evaluation violate your national/regional guidelines for health economic evaluation?
Organizational aspects domain (organizational aspects matrix)
Organizational levels
Organizational aspects dimensions	Inter-organizational level	Intra-organizational level	Health care system
Utilization	Type of data and methods of analysis
Work processes	Data from research (quantitative and qualitative)
Centralization/decentralization	Literature reviews
Staff	Routine data
Job satisfaction	Informal knowledge and anecdotes
Communication	Judgments
Finances	Models
Stakeholders
Organizational aspects domain additional questions: Are the dimensions assessed relevant for my own research questions? If no, adaptation of organizational aspects data from this report unnecessary. Are the theories and methods used relevant and reliable ones? A judgment will be necessary here. Is the analysis transferable (statistically or analytically)? (this will be dependent on the structure of the health care system and similarities of units of analysis). A judgment will be necessary here. 4. Are the results applicable to my context? A judgment will be necessary here.

Source: Turner S, et al. EUnetHTA Adaptation Toolkit Work Package 5, v4, 2008.

Abbreviations: HRQoL, health related quality of life; HTA, health technology assessments.

Antonanzas’ transferability index (2009)

And finally, in the paper by Antonanzas,¹⁵ the authors have constructed a numerical global transferability index based on weighted objective and subjective elements to measure the degree of transferability of EE results. The global index was derived through a mathematical formula that combines factors and weights from a general index and a specific index. Both of the partial indices considered a range of critical and noncritical factors that are similar to those considered by Boulenger¹⁰ and Welte.⁷ The factors used in the general and specific indices are shown in Table 5 and the index formulas are shown in Figure 4. In the first phase of this system, a general transferability index considers seven critical objective values and then the same 16 noncritical objective factors as proposed by Boulenger¹⁰ (see Table 5). The formula for the general index (IT₁) is shown in the first panel of Figure 4 and is intended to initially evaluate the methodological quality of the studies and then exclude studies that are deemed not transferable. The specific index, which is considered subjective, measures the applicability of each study to the specific setting of the decision-maker. In the specific index there are four critical criteria and eight noncritical criteria. The formula for the specific index (IT₂) is shown in the second panel of Figure 4. The overall global transferability index (third panel in Figure 4) then combines the general and specific indices, either as a simple or weighted arithmetic mean, the geometric mean, or a combination of indices. The overall approach for the objective factors is similar to that proposed by Boulenger,¹⁰ however, Boulenger assumed equal weights to all noncritical items in that checklist. In contrast, Antonanzas derived weights for noncritical objective factors averaged from answers of seven HTA agencies in Spain and assigned these weights to the noncritical objective factors. The theoretical value of the global index is between 0 and 1, or 0 and 2, depending on which formulation to choose. The authors do not provide a threshold above which a given study would be deemed transferable. Given that the same study could have different index values in different jurisdictions, it was felt that health authorities from different jurisdictions should select the threshold value that best fits their purposes.

Figure 4 Antonanzas’ transferability indices – detailed formulas.

Source: Antonanzas F, et al. Health Econ. 2009;18:629–643.

Table 5 Antonanzas’ general and specific transferability index factors

Global index (IT)

General index (IT1)

Critical objective factors: The relevant parameters needed to calculate the ratio cost/effectiveness are given in the study. The quality of the study is acceptable: The study objectives are presented in a clear, specific, and measurable manner. The variable estimates used in the analysis come from the best available source. The measurement of cost is appropriate and the methodology for the estimation of quantity and unit costs is clearly described. The health outcome measures are based on valid and reliable scales, when available. Otherwise, the scales used in the study must be fully justified. The economic model (including its structure), study methods, and components of the costs and effectiveness are presented in a clear manner. The conclusions and recommendations for the study are justified and based on the study results. Noncritical objective factors:The codes as well as description of items are similar to those considered by Boulenger et al HT1. Is the intervention described in sufficient detail? HT2. Is the comparator described in sufficient detail? SE2. Is the country in which the economic study took place clearly specified? P1. Did the authors correctly state the perspective for the economic analysis? SP1. Is the target population of the health technology clearly stated or can it be inferred by reading the article? SP3. Does the article provide sufficient detail about the study sample? E5. Have the principal estimates of effectiveness measures been reported? E7. Are the results of a statistical analysis of the effectiveness results provided? B5. Is the level of reporting of benefit data adequate (incremental analysis, statistical analyses)? C1. Are the cost components used in the analysis presented? C5. Are unit price for resources given? C6. Are costs and quantities reported separately? C7. Is the price year given? C9. Is the currency unit reported? S1. Are quantitative and/or descriptive analyses conducted to explore variability from place to place? O1. Did the authors discuss the generalizability of their results?

Specific index (IT2)

Critical subjective factors: The evaluated technology is used in the new health context (This factor will not be taken into account if the economic evaluation is carried out to obtain relevant information before the potential use of the technology in the new context). The comparator is available or used in the new context. Treatment and comparator data, as well as relevant epidemiological parameters for the technology, are valid in the new context. The study perspective coincides with that used in the new context.

Noncritical subjective factors: Cost components correspond to the medical practice related to the evaluated technology in the original study. If medical practice differs in the new context, additional cost components must be taken into account. (For example, if in the original study all patients were explored with the CT-Scanner but in the new context, both MRI and CT-scanner are used: a new cost component should be included which can be unavailable). The model connecting variables and parameters can be adapted to the new context. Life expectancy is similar in both contexts. Health-status preferences are similar in both contexts. (Applicable to cost/utility analyses). Productivity measures are similar in both contexts (applicable to cost minimization and cost/benefit analysis). The evolution of the disease is similar in both contexts. The applied discount rate is similar in both contexts. Costs and health effects data are presented in current and discounted units.

Source: Antonanzas F, et al. Health Econ. 2009;18:629–643.

The index was applied to 27 Spanish studies on infectious diseases from the perspective of hypothetical regional agency in charge of evaluating health technologies. The reason infectious diseases were chosen was because they deal with preventive and curative technologies and included models with diverse time horizons, which allows the checking off of different aspects included in the proposed index. In total, 11 out of 27 studies were deemed not transferable as one or more critical factors received a value of 0 and were therefore excluded. A global transferability index was generated for each transferable study using the arithmetic and geometric means (ie, Figure 4; Formula 1) and 2) where a = b = 0.5), although the computed values of the index hardly showed any variation regardless of the formulation used. The transferability of the analyzed studies was relatively low; the mean value of the index was 0.534–0.543, with 1 as the maximum theoretical value. The authors also compared the average index value with the average score generated using Boulenger’s¹⁰ approach, and the difference was not found to be statistically significant.

Summary of approaches

Although there have been seven publications to date on alternative systems, processes, or approaches for assessing the geographic transferability potential of EEs and HTAs or for guiding the conduct of transferring EEs and HTAs across jurisdictions, the proposed approaches and the factors used for assessing transferability potential have varied substantially across these publications. Some of the proposed approaches consist simply of a list of a few study quality issues/factors, while other proposed approaches have a much more extensive list of factors for consideration. Some authors have gone in a completely different direction by proposing a sequenced flow chart type approach to help guide the process of either conducting a transferability study or determining whether a primary study in the country of origin is needed.

More common amongst the proposed systems or approaches is an assessment of critical criteria for determining transferability potential first, followed by an assessment based on other noncritical factors. Although there is some agreement on what these critical and noncritical transferability factors are, there are considerable discrepancies in both the critical and noncritical factors that have been proposed across the systems and approaches. Finally some authors have gone one step further by proposing a quantitative score or index to measure transferability potential, with a higher score indicating greater transferability likelihood. A variant on this approach is a weighting for the index where if critical factors are not satisfied, then the index value returns a zero indicating the EE or HTA is not transferable.

Discussion

The increasing pressure on health care decision-makers to make more efficient use of existing health care resources has resulted in a substantial increase in the global demand for EEs and HTAs. However, conducting EEs and HTAs on every new intervention in every jurisdiction is not only unfeasible, but will likely result in a very inefficient use of scarce global assessment and evaluation resources. As a result, decision-making bodies will increasingly be looking at the applicability and feasibility of using EE and HTA evidence from other jurisdictions for their own local needs. Geographic transferability can be a very promising way of making more efficient use of existing assessment and evaluation resources and may be the only alternative for some jurisdictions who want to follow an evidence-based decision-making process but where resources or expertise to conduct these assessments are limited.

Most promising from the research to date on geographic transferability is the identification of factors that can potentially affect transferability. This has led to systems and approaches with checklists for assessing and conducting transferability, and specifically to the development of an extensive checklist of critical and noncritical factors in determining transferability potential. Although the list of factors has varied substantially across publications, some patterns have emerged which might form the basis for consensus in the development of a future tool, checklist, or system. For example, the critical transferability factors that have been proposed seem to focus on issues of study quality, transparency of methods, the level of reporting of methods and results, and the applicability of the treatment comparators to the target country. The proposed list of noncritical factors has been much more extensive and perhaps future research might focus on narrowing or refining this list. More recently, indices have been promised to measure transferability potential. However, due to the complexities of identifying appropriate weights for each of the noncritical factors, it is still uncertain whether the assessment and calculation of an overall transferability score or index will be practical or useful for transferability considerations in the future.

Researchers need to continue to drive toward consensus on developing a list of critical factors that prevent geographic transferability and on developing a list of noncritical factors that may prevent geographic transferability of EE or HTA initiatives across jurisdictions. A consensus will lead to good practice guidelines and perhaps eventually to a general scoring system or index which could be tailored to each jurisdiction based on jurisdiction-specific weights for each transferability factor. Whether or not an index measurement approach is practical or useful for decision-makers, an essential starting point is consensus across jurisdictions regarding the list (eg, checklist) of critical and noncritical factors, which are important for determining transferability potential.

Acknowledgements

Jean-Eric Tarride and Daria O’Reilly are supported by Ministry of Health and Long Term Care (MOHLTC) Career Scientist awards.

Disclosure

No funding was obtained for this research and the authors have no financial conflicts of interest to declare.

Appendix

Search details for PubMed

1. Chart*[tiab] OR toolkit*[tiab] OR checklist*[tiab] OR index*[tiab] OR indices*[tiab] OR methodologic*[tiab] OR practice*[tiab] OR guideline*[tiab]

2. Models, Economic[mh]

3. Economic evaluation*[tiab]

4. Technology transfer[mh]

5. Transferab*[tiab] OR transferrab*[tiab] OR transfer[tiab] OR transportable[tiab] OR transportability[tiab] OR portable[tiab] OR portability[tiab] OR adapt*[tiab] OR generali*[tiab]

6. 1 and (2 or 3) and (4 or 5)

Search details for Ovid MEDLINE and EMBASE

1. (chart* or toolkit* or checklist* or index* or indices* or methodologic* or practice* or guideline*).ti,ab.

2. Exp Models, Economic/use prmz

3. Economic evaluation*.ti,ab.

4. Economic Evaluation/use emez

5. Technology Transfer/

6. (transferab* or transferrab* or transfer or transportable or transportability or portable or portability or adapt* or generali*).ti,ab.

7. 1 and (2 or 3) and (5 or 6 use prmz)

8. 1 and (3 or 4) and (5 or 6 use emez)