The most important facilitators and barriers to the use of Health Technology Assessment in Canada: a best–worst scaling approach

Abstract

Background

Health Technology Assessment (HTA), which can support public drug reimbursement decisions will play a core function in the planned national Pharmacare program in Canada. To address existing barriers to the use of HTA, these must be ranked in order of priority. The goal of this study was to access the relative importance of known facilitators and barriers to the use of HTA in the context of the Canadian health care system, with attention to differences between regions and stakeholder groups.

Methods

We used the best–worst scaling object case approach to elicit a quantitative ranking of a list of 20 facilitators and 22 barriers. A sample of 68 Canadian HTA stakeholders, including members of expert committees, decision/policymakers, researchers/academics, and others participated in the study. Their task was to identify the most important and the least important item in 12 sub-sets of five facilitators and 14 sub-sets of five barriers.

Findings

Relative Importance Scores derived via hierarchical Bayes analysis revealed relations, engagement, and contact between stakeholders as most important on both the barrier and facilitator sides. Other top-ranked facilitators included the availably of credible and relevant research. Other top-ranked barriers included inconsistencies in the evidence and limited generalizability. The availability of HTA guidelines did not rank highly on either side. The main limitation of the study was the challenge with reaching the relevant respondents; this was mitigated by involving the national HTA agency in the research.

Conclusion

Canadian stakeholders consider the relationships within the HTA network among the most important. Policies should focus on strengthening these relationships. Future research should focus on the connectivity and distribution of knowledge and power within the HTA network.

Keywords:

• Health Technology Assessment
• Canadian Pharmacare
• selection of pharmaceuticals
• best–worst scaling
• relative importance scores

JEL CLASSIFICATION CODES:

• I12
• I1
• I
• I11

1. Introduction

The health insurance system in Canada is publicly financed, but it does not include a comprehensive and universal drug plan^1,2. Pharmaceutical coverage remains fragmented, despite commitments by the Government of Canada to the implementation of a national Pharmacare^2,3 and ongoing planning discussion. The government’s report on implementation stresses the central role that Health Technology Assessment (HTA) should play in the selection of drugs for public reimbursement³, but is HTA in Canada ready to support the healthcare system at this critical juncture?

HTA in Canada is embedded in a broader process of bringing new health technologies to end-users, which includes but is not limited to pharmaceuticals. This process is considered fragmented, similarly to many other aspects of the health system⁴. A network of HTA advisory bodies operates various levels of government and their recommendations are non-binding⁴. Specifically, the Canadian Agency for Drugs and Technologies in Health (CADTH) makes recommendations regarding the funding of health technologies, and these are considered by all Provinces and Territories except Quebec. Quebec has a separate HTA body, the Institut national d’excellence en santé et services sociaux (INESSS). In addition, many Provinces and Territories have their own drug advisory committees whose mandates include HTA⁵.

This network has been characterized as “… a complicated labyrinth …”⁶ facing a multitude of challenges^5–8. These arise because the system is complex, adaptive and composed of individuals with diverse perspectives^8,9. Current shortcomings in the Canadian HTA system have been linked to inequity, delays in the equitable access to health technologies arising from differences in the management of P/T healthcare systems⁹.

Internationally, the processes and actual uses of HTA vary^10,11, yet the influence of HTA has been judged as limited⁹. Factors that facilitate or hinder the use of HTA in policy making at the macro- and micro-levels have been studied for more than two decades^12–15. These vary across institutional contexts, social values related to health care, societal perceptions regarding efficiency, equity, and personal responsibility in health care¹⁶. To affect meaningful changes to the HTA process, these facilitators and barriers need to be prioritized in the specific context of their countries.

Several systematic reviews of influences on the use of evidence in policy or decision making in general¹⁷ or specific types of evidence in specific health policy-related contexts^18,19 have shown that there are commonalities in the barriers or facilitators faced by decision makers¹⁹. Specifically, these studies focus on use in the sense of uptake of HTA as a part of the process, as opposed to the interpretation or implementation of HTA results that may be influenced by political pressures²⁰ or actors’ attitudes and biases^21,22. Barriers and facilitators have been categorized along various dimensions, for example, those related to access and availability versus bureaucratic requirements²³, or those related to institutional, political, cultural, and methodological factors¹⁸. Oliver et al. offered a review of the use of evidence in policy in general, not limited to health. They grouped the reported facilitators and barriers as factors related to six categories: (i) contact and collaboration; (ii) organization and resources; (iii) research and researcher characteristics; (iv) policymaker characteristics; (v) policy characteristics; and (vi) other¹⁷. Their categorization was collapsed by Feig et al. into (a) decision-maker-related factors consisting of contact, collaboration and policymaker characteristics; (b) context-related factors consisting of organization, resources and policy characteristics; and (c) methodology related factors, consisting of research and researcher characteristics²⁴.

Fewer studies exist that identify the relative importance of influences on the use of evidence in health policymaking. To date, the relative importance of known barriers and facilitators to the use of HTA has been investigated in Austria²⁴, Colombia²⁵, Germany²⁶, France²⁶, the Netherlands^26,27, and the United Kingdom²⁶. A ranking of relative importance can serve to identify the order in which to address barriers and promote facilitators, with the assumption that any policy change is gradual and incremental. Findings from the above studies suggest that many facilitators and barriers are common across countries and contexts, but there are differences in their relative prominence. In addition, the relative importance is rated differently depending on the professional backgrounds of respondents. To date, this topic has not been explored in Canada, where an understanding of how to prioritize the removal of barriers to the use of HTA pressing.

The research goal of this study was to assess the relative importance of facilitators and barriers to the use of HTA in the context of the Canadian health care system. A supporting objective was to estimate the differences in relative importance assigned by regions and stakeholder groups.

2. Methods

2.1. Study design

2.1.1. The best–worst scaling method

The best–worst scaling (BWS) method is a type of conjoint analysis with demonstrated relevance to the assessment of preferences in health care settings^28–30. To elicit a quantitative ranking of a list of qualitative items that do not have levels the BWS object case was used in this study, in line with previous studies^29–31.

2.1.2. Identification of facilitators and barriers

This study included 20 facilitators and 22 barriers. An initial list was identified via a scoping review completed for a comparable BWS study in the Netherlands²⁷ and verified for the Canadian setting. The same initial list was used as a point of departure for similar studies in Austria²⁴, Colombia²⁵ and France, Germany, and the United Kingdom²⁶. The list was translated into French by an independent translator in Canada and translation was verified by one of the study authors (MH).

To validate the relevance and completeness of the list for the Canadian setting, the list was distributed to 10 Canadian HTA experts across the country, including HTA committee members and policymakers with Provincial drug plans, four of whom provided feedback. Feedback was discussed among study authors. Two facilitators and two barriers were unique to Canada and were added specifically for this study, and three facilitators and two barriers investigated in similar studies were removed from the Canadian list. Similarly, to Feig et al., items were classified into three categories: (i) context-related factors; (ii) decision-maker-related factors; and (iii) methodology-related factors²⁴.

2.2. Data collection

2.2.1. Survey participants

Canadian HTA stakeholders included: (i) experts who are members of drug advisory committees at both the national and Provincial/Territorial levels, (ii) employees/representatives of institutions involved in the conduct of HTA, (iii) public employees involved in Provincial/Territorial drug plan decisions or otherwise in drug reimbursement, (iv) patients or patient representatives involved in drug reimbursement decision processes at various stages, and (v) academic researchers who study methods and processes used in Canadian HTA.

Recruitment of study participants relied on a purposive and snowball approach. First, the study invitation was distributed via two main channels: (i) via two HTA organizations, CADTH and INESSS, who used their confidential mailing distribution lists; and (ii) using a list of publicly available email addresses (n = 250) that we assembled from a variety of HTA organizations and authorship lists of Canadian HTA studies (purposive sample). CADTH distributed the questionnaire to members of all its advisory committees and a broader HTA community list. Second, within the invitation email, a request was sent to distribute the invitation among the respondents’ networks (snowball sample). The number of snowball invitations was not trackable.

2.2.2. The best–worst scaling survey

The Sawtooth Software’s SSIⁱ Web platform was used to design the experiment. Fractional efficient designs were used characterized by: (i) orthogonality (items were shown and paired an approximately equal number of times), (ii) minimal overlap (minimizing the number of times that each item appeared within the same set across the designs), (iii) positional balance (items appeared approximately an equal number of times in each position, (iv) connectivity (items were directly or indirectly linked), and (v) stability. Two different versions of the questionnaire were generated for each element of the study, that is, two versions for facilitators and two versions for barriers. Respondents were randomly assigned to one of the versions of facilitators and barriers respectively. The order of choice-sets presented to respondents was random.

The online self-administrated survey was designed using Qualtricsⁱⁱ Participants were able to select an English or French version of the survey, which consisted of three sections (Appendix 1). The first section asked for demographic and professional information about respondents, such as their primary position (official designation) and role (expertise or perspective) in the HTA process. The second section asked participants to assess 14 choice sets of five barriers. For each set, respondents were asked to identify the barrier that they considered most important when thinking about policymakers’ decisions to use HTA studies, and which they considered least important. The third section asked participants to assess 12 choice sets of five facilitators by selecting the most important and least important in each set. Open-ended questions at the end of sections two and three allowed respondents to list additional barriers or facilitators respectively, or to comment more generally. Furthermore, respondents were asked to rate the difficulty of making their choices on a Likert scale from 1 (very easy) to 7 (very difficult) for both sections two and three.

The study was approved by the Dalhousie Research Ethics Board (file #2018-4635) as having been designed in accordance with the Canadian Tri-Council Policy Statement: Ethical Conduct for Research Involving Humans – TCPS 2 (2018)³². The survey was piloted with seven Canadian graduate students with expertise in health policy and economics. The estimated time for completion was ∼15 min. The survey was distributed between 8 July and 29 July 2020 with two reminders. Respondents were provided with detailed information about the study, an informed consent request, and had the option to exit the survey at any time. The survey was anonymous.

2.3. Data analysis

2.3.1. Statistical analyses

Descriptive statistics were calculated for all respondents. Questionnaires with a complete section 1, but incomplete section 2 were used for a comparison between respondents and non-respondents using Pearson’s Chi-squared tests. The preference analysis included all fully completed BWS responses.

The BWS preference experiment responses were analyzed using Hierarchical Bayes (HB) analysis^33,34 in the Sawtooth Software’s SSI Web platform. This is commonly used to analyze results from health-related BWS studies^29,30,35. Relative importance scores (RIS) were calculated for each item; these are rescaled (probability) scores that reflect the likelihood of an item being selected. The sum of scores is 100, and scores are ratio-scaled³⁶.

Subgroup analyses were conducted for all respondent categories (see Table 1) in two steps. First, we compared the ranking of each sub-group to the ranking of the full sample using Spearman’s rho non-parametric rank-order correlation coefficient. A coefficient of less than 0.8 was considered to signal potentially meaningful differences in rankings. Second, for categories with at least one different (ρ < 0.8) sub-group ranking, we compared average RIS scores using one-way ANOVA tests.

Table 1. Respondent characteristics.

Characteristic	Completed versus partial responses
	Number of full responses (percentage of total sample)	Partial responses* (percentage of all who dropped out)	Chi-squared statistic (p-value)
Full Sample	68 (100)	19 (100)
Demographics
Female	38 (56)	9 (47)	.987 (p=.690)
Male	29 (43)	10 (53)
Other	1 (1)	–
Age 30–39 years	13 (19)	2 (11)	1.708 (p=.645)
Age 40–49 years	19 (28)	8 (42)
Age 50–59 years	21 (31)	6 (32)
Age 60 years and above	15 (22)	3 (16)
Selected the French language version	6 (9)	2 (11)	.052 (p=.820)
Experiences with HTA or the HTA process
Conducted or contributed to HTA study	48 (71)	9 (47)	3.544 (p=.060)
By position in the HTA process
Current or past member of an HTA committee	31 (46)	5 (26)	19.099 (p<.001)
Decision or policy maker	11 (16)	4 (21)
Researcher or academic	19 (28)	–
Otherwise involved in HTA review/ process or other	7 (10)	10 (53)
By role in the process
Clinical expertise	14 (21)	2 (11)	6.622 (p=.146)
Economic expertise	17 (25)	4 (21)
Input on the patient perspective and/or ethical matters	11 (16)	1 (11)
Input on the payer perspective and/or organizational matters	14 (21)	3 (16)
Research and other	12 (18)	9 (47)
By province or territory
Atlantic Region	8 (12)	6 (32)	8.417 (p=.054)
Central Canada	37 (54)	11 (58)
Quebec alone	8 (12)	11 (58)
Prairie Provinces	15 (22)	1 (11)
West Coast	7 (10)	–
North	1 (1)	1 (11)

*Partial responses – profile of participants who completed the demographics section of the questionnaire but did not complete the full BWS experiment.

3. Results

3.1. Participants’ characteristics

In total, 123 respondents started the survey, of which 68 completed the entire survey, 19 completed section 1 but dropped out at the BWS stage, and 36 dropped out prior to completing section 1. We detected two statistically significant differences between those who dropped out and those who completed the experiments. First, current and past committee members, decision or policymakers and researchers were less likely to drop out compared to those otherwise involved in the process (χ² = 19.09; p < .001). Second, respondents from Quebec were more likely to drop out of the survey compared to respondents from other regions (χ² = 8.42; p = .054).

The majority of survey participants were female (56%) and were between 40 and 59 years (59%). 48 had direct experience with conducting or contributing to an HTA study (71%). The majority (54%) came from the Central Region (Ontario and Quebec). Many were members of an HTA committee (46%), policymakers (16%), or academic researchers (28%). Expertise/perspective was relatively evenly distributed across respondents (Table 1).

3.2. Relative importance of factors influencing the use of HTA in Canada as ranked by HTA stakeholders

Table 2 reports the average relative importance scores (RIS) of the facilitators of HTA use as ranked by Canadian stakeholders. Results are reported for the full sample (n = 68), as well as for Quebec (n = 8) and the rest of Canada (n = 60) separately because these regions rely on different HTA agencies (INESSS and CADTH respectively).

Table 2. Relative Importance Scores for facilitators of the use of HTA in Canada.

Description of facilitator and category*		Full sample		Quebec (n = 8) (INESSS)		Rest of Canada (n = 60) (CADTH)
		Rank	RIS (95% CI)	Rank	RIS (95% CI)	Rank	RIS (95% CI)
Sufficient engagement by stakeholders including policy makers, clinicians and patients	D	1	8.58	1	9.16	1	8.50
			(7.42–9.73)		(4.30–14.03)		(7.27–9.72)
Sufficient support within the policy organization for the use of HTA	C	2	7.33	2	8.43	2	7.17
			(6.21–8.45)		(4.68–12.19)		(5.96–8.40)
Credibility of HTA research findings	M	3	6.53	9	5.41	3	6.67
			(5.58–7.47)		(1.83–8.99)		(5.66–7.69)
Appropriate timing between HTA research and policy making	D	4	6.45	10	5.10	4	6.63
			(5.16–7.74)		(1.56–8.65)		(5.20–8.06)
Availability of relevant HTA research for policy makers	C	5	6.20	5	6.23	5	6.19
			(5.15–7.25)		(3.22–9.25)		(5.03–7.36)
Contact and interaction between HTA producers, policy makers and other stakeholders	D	6	6.05	3	7.88	6	5.81
			(4.94–7.17)		(3.45–12.31)		(4.63–6.99)
Availability of an explicit framework for the decision-making processes that rely on HTA	C	7	5.51	12	4.65	8	5.62
			(4.36–6.66)		(0.33–8.98)		(4.39–6.86)
Clear/concise presentation of HTA reports, such executive summaries or bulleted key points	M	8	5.42	17	2.76	7	5.78
			(4.29–6.55)		(0.42–5.10)		(4.53–7.02)
Access to relevant HTA research (and dissemination) for policy makers	D	9	5.27	7	5.63	10	5.23
			(4.42–6.13)		(2.55–8.71)		(4.31–6.15)
Sufficient human resources to understand and interpret relevant HTA studies	C	10	5.24	4	6.55	12	5.06
			(4.20–6.27)		(1.96–11.14)		(3.99–6.13)
Quality of HTA research findings meeting scientific requirements	M	11	5.15	11	5.08	11	5.16
			(4.25–6.05)		(1.37–8.78)		(4.20–6.12)
Appropriate incentives for the implementation of cost-effective interventions	C	12	5.01	19	1.54	9	5.48
			(3.70–6.33)		(0.30–2.78)		(4.02–6.94)
Sufficient relevant Canadian data on, for example, appropriate comparator, drug prices, market share	C	13	4.77	15	4.26	13	4.84
			(3.65–5.89)		(1.69–6.82)		(3.59–6.09)
Sufficient awareness within the policy organization of the relevance of HTA	C	14	4.70	6	6.23	14	4.50
			(3.80–5.60)		(3.36–9.10)		(3.51–5.47)
Transparency of HTA research findings and clear description of methods used	M	15	3.98	18	2.44	15	4.19
			(3.29–4.67)		(0.26–4.62)		(3.45–4.93)
Sufficient legal and legislative support for the use of HTA	C	16	3.56	14	4.30	16	3.47
			(2.49–4.64)		(–0.44–9.03)		(2.35–4.59)
Existence of an HTA agency	C	17	3.44	8	5.51	17	3.16
			(2.37–4.51)		(–0.31–11.34)		(2.12–4.20)
Longstanding relations between policymakers and HTA producers	D	18	3.00	13	4.62	18	2.79
			(2.00–4.01)		(–0.39–9.64)		(1.78–3.80)
Sufficient financial and human resources to conduct relevant HTA research	C	19	2.76	16	3.17	19	2.70
			(2.13–3.40)		(0.93–5.41)		(2.02–3.40)
Availability of adequate HTA guidelines	M	20	1.04	20	1.01	20	1.04
			(0.77–1.31)		(–0.28–2.31)		(0.77–1.32)

*Categories are: C – context-related; D – decision-maker related; M – methodology-related²⁴.

The rank order correlation between Quebec and the rest of Canada is ρ = 0.574 and is statistically significant at the 5% level.

Three of the top six facilitators (RIS > 6.0) for Canada are related to decision-makers. They include “sufficient engagement by stakeholders” (RIS = 8.58), “appropriate timing” (RIS = 6.45), and “contact and interaction” (6.05), even though “longstanding relations” rank low (RIS = 3.00). The other top facilitators are “sufficient support” (RIS = 7.33), “credibility” (RIS = 6.53) and “availability of relevant research” (RIS = 6.20). The “availability of HTA guidelines” (RIS = 1.04) and the “financial and human resource availability” (RIS = 2.76) are not perceived as strong facilitators of HTA use in Canada.

Table 3 shows the average RIS of the barriers to the use of HTA in Canada. The top four barriers (RIS > 6.0) include methodological and decision-maker-related factors. The top-ranked decision-maker barriers are the “insufficient engagement by stakeholders” (RIS = 7.61) and “insufficient contact and interaction” (RIS = 6.72). Methodology-related barriers are “inconsistent findings” (RIS = 7.08) and “limited generalizability” (RIS = 6.24). “Longstanding relations” (RIS = 2.76) and “absence of HTA guidelines” (1.45) rank low as barriers.

Table 3. Relative Importance Scores for barriers of the use of HTA in Canada.

Description of barrier and category*		Full sample		Quebec (n = 8) (INESSS)		Rest of Canada (n = 60) (CADTH)
		Rank	RIS (95% CI)	Rank	RIS (95% CI)	Rank	RIS (95% CI)
Insufficient engagement by stakeholders including policy makers, clinicians, and patients	D	1	7.61	2	9.66	1	7.34
			(6.45–8.78)		(5.37–13.94)		(6.10–8.58)
Inconsistent or unclear HTA findings, for example conflicting results	M	2	7.08	4	6.01	2	7.21
			(6.04–8.11)		(1.94–10.08)		(6.11–8.32)
Insufficient contact and interaction between HTA producers, policy makers, and other stakeholders	D	3	6.72	1	9.87	4	6.29
			(5.55–7.89)		(6.12–13.60)		(5.05–7.54)
Limited generalizability of HTA studies to the policy makers’ context	M	4	6.24	19	2.47	3	6.74
			(5.26–7.22)		(0.26–4.70)		(5.71–7.78)
Absence of appropriate incentives for the implementation of cost-effective interventions	C	5	5.72	20	2.01	5	6.21
			(4.54–6.90)		(0.18–3.84)		(4.93–7.50)
Low quality of HTA research findings meeting scientific requirements	M	6	5.37	5	5.87	8	5.30
			(4.34–6.41)		(1.70–10.06)		(4.20–6.40)
Uncertainty surrounding HTA results	M	7	5.36	21	1.31	6	5.89
			(4.35–6.37)		(0.24–2.38)		(4.81–6.98)
No explicit framework for the decision-making processes that rely on HTA	C	8	5.14	7	5.41	9	5.10
			(4.06–6.22)		(1.50–9.33)		(3.93–6.28)
Insufficient human resources to understand and interpret relevant HTA studies	C	9	5.06	11	4.75	10	5.10
			(3.99–6.14)		(0.98–8.53)		(3.94–6.27)
Insufficient relevant Canadian data on, for example, appropriate comparator, drug prices, market share	M	10	5.04	16	2.88	7	5.33
			(4.45–6.64)		(1.16–4.60)		(4.70–5.96)
Insufficient access to relevant HTA research (and dissemination) for policy makers	D	11	5.00	12	4.70	11	5.04
			(4.18–5.82)		(1.21–8.21)		(4.17–5.91)
Insufficient support within the policy organization for the use of HTA	C	12	4.88	3	6.56	12	4.65
			(3.96–5.80)		(2.99–10.12)		(3.69–5.63)
Low credibility of HTA research findings	M	13	4.69	6	5.69	13	4.55
			(3.81–5.56)		(1.86–9.52)		(3.64–5.47)
Insufficient availability of relevant HTA research for policy makers	C	14	4.39	8	5.28	14	4.27
			(3.37–5.04)		(3.15–7.40)		(3.55–4.98)
Insufficient financial and human resources to conduct relevant HTA research	C	15	3.65	13	4.04	15	3.60
			(2.77–4.53)		(0.72–7.36)		(2.65–4.55)
Insufficient awareness within the policy organization of the relevance of HTA	C	16	3.37	9	5.22	17	3.12
			(2.59–4.15)		(1.64–8.80)		(2.33–3.91)
Inadequate presentation format, for example overly long or theoretical reports, excessive use of jargon	M	17	3.07	17	2.70	16	3.12
			(2.21–3.94)		(0.17–5.23)		(2.17–4.08)
Insufficient legal or legislative support for the use of HTA	C	18	3.01	15	3.44	18	2.95
			(2.04–3.97)		(–0.38–7.25)		(1.91–3.98)
Insufficient transparency of HTA research findings and clear description of methods used	D	19	2.91	14	3.64	19	2.81
			(2.27–3.56)		(–0.29–7.56)		(2.20–3.43)
Lack of longstanding relations between policymakers and HTA producers	D	20	2.76	10	5.18	20	2.44
			(1.86 − 3.66)		(0.41–9.95)		(1.56–3.30)
Lack of a national HTA agency	C	21	1.47	18	2.60	22	1.32
			(0.91–2.04)		(–0.15–5.36)		(0.76–1.89)
No availability of adequate HTA guidelines	M	22	1.45	22	0.68	21	1.55
			(0.91–1.99)		(0.13–1.22)		(0.94–2.17)

*Categories are: C – context-related; D – decision-maker related; M – methodology-related²⁴.

The rank order correlation between Quebec and the rest of Canada is ρ = 0.321 and is not statistically significant.

3.3. Subgroup analysis

The Spearman’s rank-order comparison of rankings (Table 4) revealed several differences between ranks assigned by sub-groups relative to those assigned by the full sample. Rankings of both facilitators and barriers were different among those who indicated Quebec as their region (facilitators ρ = 0.65; barriers ρ = 0.40). Respondents from the Atlantic region also ranked the importance of facilitators and barriers differently from the full sample (facilitators ρ = 0.31; barriers ρ = 0.74). Rankings of decision or policymakers also stood out as different (facilitators ρ = 0.76; barriers ρ = 0.77). Furthermore, facilitators were ranked differently by those in the 50–59 years age group (ρ = 0.79), those providing input on the patient perspective (ρ = 0.79) and researchers (ρ = 0.79), as well as respondents from the Prairies (ρ = 0.75) and the West Coast (ρ = 0.74). Barriers were ranked differently by respondents in the 30–39 years age group (ρ = 0.78), and by those “otherwise involved” (ρ = 0.69).

Table 4. Comparisons of rankings by sub-groups to the full sample ranking.

Subgroup	Spearman’s ρ Rank Order Coefficient (p-values in brackets)
	Facilitators	Barriers
Demographics
Female	0.970***	0.974***
Male	0.878***	0.922***
Age 30–39 years	0.866***	0.781***
Age 40–49 years	0.934***	0.941***
Age 50–59 years	0.785***	0.868***
Age 60 years and above	0.820***	0.894***
Selected the French language version	0.759	0.353
Experiences with HTA or the HTA process
Conducted or contributed to HTA study	0.971***	0.948***
Did not conduct or contribute to an HTA study	0.820***	0.834***
By position in the HTA process
Current or past member of an HTA committee	0.901***	0.949***
Decision or policy maker	0.762***	0.768***
Researcher or academic	0.887***	0.940
Otherwise involved in HTA review/ process or other	0.832***	0.686
By role in the process
Clinical expertise	0.878***	0.808***
Economic expertise	0.835***	0.869***
Input on the patient perspective and/or ethical matters	0.788***	0.871***
Input on the payer perspective and/or organizational matters	0.856***	0.911***
Research and other	0.791***	0.880***
By region
Atlantic Region	0.306***	0.741***
Central Canada	0.947***	0.905***
Quebec alone	0.645*	0.401***
Prairie Provinces	0.749***	0.850**
West Coast	0.738***	0.825***

Categories with one respondent (Gender: Other. Region: North) were not included in the comparison of rankings.

***Statistically significant at the 1% level; **statistically significant at the 5% level; *statistically significant at the 10% level.

Comparison of average RIS scores by item by sub-group using ANOVA analysis revealed several statistically significant (p < .10) differences. On the barriers side, “insufficient human resources to understand and interpret” was considered more important by researchers and academics (RIS = 7.45) than by current or past members of HTA committees (RIS = 4.52) or decision and policymakers (RIS = 2.16).

There were also regional differences: as a barrier, “insufficient contact and interaction” was most important in the Prairies (RIS = 7.82) compared to the Atlantic Region (RIS = 3.39) or the West Coast (RIS = 3.38). As a facilitator, “contact and interaction” was considered most important in Central Canada (RIS = 7.52) when compared with the West Coast (RIS = 3.38) or the Atlantic Region (RIS = 1.79). As a barrier, “no HTA guidelines” was more important in the Atlantic region (RIS = 3.32) than in the Prairies(RIS = 0.87) or Central Canada (RIS = 1.10), but was overall relatively unimportant.

The sub-group comparison between Quebec and the rest of Canada showed that the rankings of facilitators (Table 4) were statistically significantly different between these regions (ρ = 0.574). Facilitators that were relatively less important to Quebecois respondents included: “appropriate timing”, “clear and concise presentation” and “appropriate incentives”, whereas facilitators that ranked higher in Quebec included “existence of an HTA agency” and “longstanding relations” (Appendix 2). Barriers that were ranked lower in Quebec included: “limited generalizability”, “absence of appropriate incentives”, and “uncertainty”, whereas barriers that ranked higher in Quebec included: “insufficient transparency”, “longstanding relations”, and “lack of national HTA agency” (Appendix 3).

4. Discussion

The importance of building a strong HTA stakeholder community was recently identified as a key consideration among HTA users³⁷. The most important Canadian issue identified in our study was “engagement of stakeholders, including policy makers, clinicians, and patients” on both the facilitators and barriers list. Respondents further ranked the extent of “contact and interaction between HTA producers, policymakers, and other stakeholders” among the top three barriers and top six facilitators. In other words, the system of relationships between stakeholder groups is considered a top priority in Canada (also in the Quebec sub-group). This is consistent with the strong emphasis that Canadian processes place on stakeholder engagement^38,39, which has long been recognized as the key step to improved HTA uptake⁴⁰. Stakeholder interactions have been described as central to the exchange and interpretation of information in the Canadian HTA process, which Lopes et al. specifically characterize as “a social enterprise involving diverse people with differing world-views”⁹. Consistent with our results, they find that stakeholders value initiatives that bring them together to formulate recommendations⁹. Potential drawbacks to the inclusion of diverse stakeholder groups include the inability to focus results appropriately for their target audiences and diffusion of responsibility with respect to implementation⁵.

Stakeholder interaction at the level of Canadian regions has been described recently as a network of collaborative horizontal relationships, which Fierlbeck et al. characterized as a “response to the complex systems thinking applied to HTA”⁴. The emphasis is to put people at the forefront of governance and enable the interactions of multiple actors. This is also consistent with Lopes et al.’s description of the Canadian HTA process as a complex system⁵. Our results suggest that stakeholder collaboration is understood as important, but requires further improvement particularly with the goal of creating consistent collaborative relationships across all regions. Similarly, MacNeil et al. suggest that HTA uptake could be further enabled through increased information sharing across jurisdictions⁹.

Curiously, the longstanding relations between HTA producers and policymakers ranked as relatively unimportant on both the barriers and facilitators list; their specific ranks varied between Quebec and the Rest of Canada and we noted that the French version of the questionnaire excluded the term “longstanding”. The relations between HTA producers and other stakeholders have been characterized as strained and lacking trust elsewhere in the literature⁶, specifically when the pharmaceutical industry is seen as the HTA producer. The nature of their involvement in the network merits careful consideration.

Important methodological barriers included inconsistent or unclear findings (e.g. conflicting results) and the limited generalizability of HTA studies. These challenges are not new and may be key contributors to the inequities in access to publicly funded pharmaceuticals across Canada⁴¹. The lack generalizability of existing studies is prominent in Canada, because of the fragmented nature of decision-making. Further to this point, the lack of generalizability ranked considerably lower in Quebec (rank 19) than the rest of Canada (rank 3), suggesting that the HTA studies prepared for INESSS do match well to the Quebecois context, whereas those prepared for CADTH are difficult to generalize across the remaining Provinces and Territories.

Decreased generalizability of results may be the trade-off resulting from centralization efforts and the building of a national HTA agency, the advantages of which include increased consistency and no duplication of efforts⁷. The existence of an HTA agency was not considered an important facilitator by the Rest of Canada (rank 17), but a relatively more important facilitator in Quebec (rank 8), suggesting that the province-focused approach may be perceived as a stronger facilitator of the use of HTA. In addition, regional subgroup analysis showed that some regions (Atlantic, Prairies) feel more disconnected from the HTA process than others (Central region), therefore centralization of processes may not have sufficiently tended to the building of networks. Further research into the perceived challenges with generalizability could shed light on the specific components of HTA studies that are less generalizable than others and spark a potential reconsideration of which parts of the HTA process best be centralized and which might be better regionally focused.

Previous BWS studies highlighted the importance that policy contexts have on HTA use. For example, Feig et al. flag the development of clear and explicit decision and legal frameworks as most important in the Austrian context. The need for explicit frameworks has been discussed elsewhere, for example, Goethebeur et al. describe and test a multi-criteria decision framework (evidence and value: impact on decision making, EVIDEM) to explore its utility to drug advisory committees⁴². The extent to which such as framework is available in Canada is ranked 7th among the facilitators and 8th among the barriers in our study. It was not considered among the most pressing issues in Canadian HTA and there does not appear to be a strong appetite for refinements to a decision framework. Tony et al. also discovered that committee members considered such a framework to be potentially useful but the one under study proved too complicated for use⁴³.

Cheung et al. highlight the importance of the interplay between science policy (e.g. high quality and adequate studies) and healthcare policy (e.g. adequate decision-making framework) in France, Germany, the Netherlands and the United Kingdom²⁶. Our results suggest that these important factors may best be framed as a network problem, where the use of HTA in the healthcare system is understood as driven by both science and politics. Future research using network analyses may shed light on the connectivity and distribution of knowledge and power between the different stakeholders of HTA when looking at the use of evidence policy-making, as opposed to the technocratic aspects of knowledge translation^44–46. Consistently, in our study, items related to knowledge translation, such as clarity, transparency of findings, or presentation format were indeed ranked as having relatively average importance.

Our results have important implications for the policy and practice of HTA in Canada. The building of an HTA community, strengthening of stakeholder networks and coordination of HTA process elements can and to a degree has been taken on by CADTH. In addition to the conduct of HTA, the agency has a strong platform for the engagement of patients⁴⁷ and implements strategies to strengthen collaboration and outreach through a variety of initiatives⁴⁸. Most recently, for example, CADTH has launched the Canadian Journal of Health Technologies⁴⁹. The existence of the pan-Canadian Health Technology Assessment Collaborative⁵⁰ demonstrates a commitment to collaborations between various HTA bodies in Canada, although our results suggest that this could be strengthened. Furthermore, CADTH’s collaborations with other bodies, such as the Patented Medicines Prices Review Board and the pan-Canadian Pharmaceutical Alliance could be strengthened; current collaborations are relatively linear and static^51,52.

Limitations of the study lie primarily in the potential for bias, which has three sources: (i) bias due to sampling strategy; (ii) bias due to drop-out; and (iii) bias due to potential small differences in meaning between the two language versions of the questions (the study was translated and the translation verified, but it was not back-translated as an accuracy check).

The sampling strategy aimed to reach as many respondents as possible from a diverse community with no formal structure using a purposive sample followed by a snowball sampling strategy. Participants may not be representative, for example, we observed an overrepresentation of women (56% in our sample compared to 51% in the general Canadian population), overrepresentation from the Atlantic region (12% in our sample compared to 7% of the general population), and underrepresentation from Quebec (12% in our sample compared to 23% of the general population)⁵³.

Of the 123 respondents who opened the questionnaire, 55 (45%) dropped out. This is comparable to previous studies, in which the drop-out rates were 72%²⁵, 54%²⁶, 42%²⁷, and 30%²⁴. Of those who dropped out, 19 completed the demographic section. A comparison between respondents who completed all sections and those who completed only the demographics section (see Table 1) reveals that respondents from Quebec and those who identified their position as “otherwise involved in the process” were more likely to drop out (based on and χ² comparisons). In general, we speculate that drop-out rates could be due to respondents not being familiar with the BWS task, which can be both cognitively challenging and time-consuming.

5. Conclusions

Our study highlighted the most important facilitators and barriers to the use of HTA in Canadian healthcare policy. Our results provide a background to important policy measures that would increase the use and appropriateness of HTA in Canada. Our results show that HTA stakeholders perceive the need for a framework to strengthen collaboration and cooperation across regions and between stakeholder groups within the HTA network. This need is perceived as more pressing than that for more explicit decision-making frameworks that connect evidence, even though the quality of the evidence is perceived as important. Canadian HTA is a strong advocate and early adopter of participatory decision-making practices, which may explain that the focus of stakeholders is on the linkages that strengthen inclusive participation. Solidifying the function of the HTA process in Canada is critical currently, while the planning of the national Pharmacare program is underway.

Transparency

Declaration of funding

This study was partially funded by the Dalhousie University Vice-President Research and Innovation International Seed Grant number [VPRIIS-2018].

Declaration of financial/other relationships

W. D. Wranik is a member of an HTA expert committee in Canada. She received honoraria for participation in meetings. The other authors declare that they have no competing interests in relation to this study.

JME peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Author contributions

MH, KLC, SM conceptualized the study in other countries; WDW validated study for the Canadian context. All authors discussed conceptualization in the Canadian context. WDW led the translation and distribution of the questionnaire. KLC, MH, RS conducted data analysis. All authors collaboratively interpreted results. SM wrote draft sections of the manuscript. WDW wrote the full draft of the manuscript. All authors worked through the draft manuscript and equally contributed to its finalization.

Ethical approval

The study obtained approval from the Dalhousie University Social Sciences and Humanities Research Ethics Board, REB# 2018-4635 under the title “Facilitators and barriers to the use of Health Technology Assessment in Canada”.

Consent to participate

All respondents consented to participate by moving forward in an online survey. After a description of the project using an approved informed consent template, respondents were prompted: “If you agree to complete the survey, please choose the language you want to complete the survey in, then click ‘Continue’.”

Data sharing

The data that support the findings of this study are available from the corresponding author, MH, upon reasonable request.

Acknowledgements

None reported.

Notes

i Sawtooth Software's SSI Web platform, Sequim, WA, USA.

ii Qualtrics, Provo, UT, USA.