Relevance of barriers and facilitators in the use of health technology assessment in Colombia

Abstract

Objectives: Several studies, mostly from developed countries, have identified barriers and facilitators with regard to the uptake of health technology assessment (HTA). This study elicited, using best-worst scaling (BWS), what HTA experts in Colombia consider to be the most important barriers and facilitators in the use of HTA, and makes a comparison to results from the Netherlands.

Methods: Two object case surveys (one for barriers, one for facilitators) were conducted among 18 experts (policymakers, health professionals, PhD students, senior HTA-researchers) from Colombia. Seven respondents were employees of the national HTA agency Instituto de Evaluación Tecnológica de Salud (IETS). In total, 22 barriers and 19 facilitators were included. In each choice task, participants were asked to choose the most and least important barrier/facilitator from a set of five. Hierarchical Bayes modeling was used to compute the mean relative importance scores (RIS) for each factor, and a subgroup analysis was conducted to assess differences between IETS and non-IETS respondents. The final ranking was further compared to the results from a similar study conducted in the Netherlands.

Results: The three most important barriers (RIS >6.00) were “Inadequate presentation format”, “Absence of policy networks”, and “Insufficient legal support”. The six most important facilitators (RIS >6.00) were “Appropriate timing”, “Clear presentation format”, “Improving longstanding relation”, “Appropriate incentives”, “Sufficient qualified human resources”, and “Availability to relevant HTA research”. The perceived relevance of the barriers and facilitators differed slightly between IETS and non-IETS employees, while the differences between the rankings in Colombia and the Netherlands were substantial.

Conclusion: The study suggests that barriers and facilitators related to technical aspects of processing HTA reports and to the contact and interaction between researchers and policymakers had the greatest importance in Colombia.

Keywords:

• Health technology assessment
• Colombia
• barriers
• facilitators
• best–worst scaling
• preferences

Introduction

In the past decades, technological innovations in the healthcare sector have resulted in significant improvements in terms of healthcare delivery and outcomes for patients. Despite evident medical advantages, it must be acknowledged that this development has led to rapidly increasing healthcare costs. For example, in 2008 the US Congressional Budget Office stated that half of the increased costs of healthcare can be associated with new technological progress¹. As healthcare budgets must be limited, prioritizing value-adding technologies is becoming a policy concern². In light of restricted healthcare resources, health-technology assessment (HTA) has been increasingly used to assess new healthcare technologies, in order to support priority setting and decision-making in the healthcare system³. In high-income countries, in particular, the use of HTA is currently at a high in determining market authorization, reimbursement, and the use of health technologies. Accordingly, several HTA agencies have been created around the world. HTA is defined by the WHO as “a multidisciplinary process to evaluate the social, economic, organizational and ethical issues of a health intervention or health technology”⁴. The term technology in this context is defined as “The drugs, devices, and medical and surgical procedures used in healthcare, and the organizational and supportive systems within which such care is provided” p.122⁵.

Policymakers in low- and middle-income countries (LMIC) are also dealing with an increase in healthcare expenditures and an increasing need for healthcare, suggesting that, as resources are scarce, rational choices have to be made⁶. However, it remains hard for policymakers to set priorities between healthcare interventions. Most LMIC lack institutions which could assess the priorities in light of national needs and circumstances⁷. Nonetheless, several HTA initiatives are now emerging in developing countries. The first regional HTA network in Latin America (LA), the Red de Evaluación de Tecnologías en Salud de las Américas (RedETSA), was established in 2011, and in the same year Colombia enacted law 1438, calling for the establishment of its own HTA agency to inform policymakers regarding coverage decisions⁸. However, although Colombia established the Instituto de Evaluación Tecnológica de Salud (IETS) in September 2012, with the aim of introducing HTA as a means of promoting cost-aware decision-making, the institute still lacks the technical capacity to conduct and assess HTA reports which could influence decision-makers^9,10. Hence, involving multiple stakeholders to optimize the use and impact of HTA in Colombia could be a worthwhile step towards making the decision-making process on coverage more fair and transparent¹.

Several studies have identified barriers to and facilitators for the uptake of HTA. Most of these studies were conducted in high-income countries¹¹. Recently, a best–worst scaling was conducted in the Netherlands to assess the most important barriers to and facilitators for the use of HTA. These results cannot necessarily be transferred to a LMIC such as Colombia³. Accordingly, to facilitate and increase the use of HTA in Colombia and other developing countries, research is, therefore, needed to assess which barriers and facilitators are important.

The goal of the research was, therefore, to determine, among HTA experts in Colombia, the most important barriers to and facilitators for the uptake of HTA, to better understand the process and value of HTA in the Colombian setting. We used a best–worst scaling that was previously developed by Cheung et al.¹² for their study (2017) in the Netherlands. A secondary objective was then to compare our findings to the Dutch study to explore whether the importance of barriers and facilitators differed between a high-income country such as the Netherlands and a middle-income country such as Colombia.

Methods

Best–worst scaling

Recently, best–worst scaling (BWS) has received increasing attention as a methodology for eliciting stated preferences in healthcare. In comparison with the traditional discrete choice experiment (DCE), BWS has the advantage that it also collects information about the least preferred option and that more attributes can be incorporated¹³. In this study, we used the BWS object case, aiming to assess the relative importance of each element from a list of attributes measured on a single level. In a typical BWS object case, the attributes from the list are arranged in different choice sets, which are presented to the participant. The participant has to indicate his/her most and least important attribute in the choice sets, which usually contain between three and five attributes. The final result is a full ranking of the attributes according to their relative importance¹⁴.

Participants

We recruited policymakers (decision-makers and policy advisors on various levels) and HTA experts (PhD students and senior HTA researchers) from Colombia. The recruited participants had many roles, for example consultant for national institutions in making buying decisions, advisor for the ministry of health in the generation of guidelines, HTA report analysts, and economists. The participants were recruited using a sampling strategy based on our network of contacts. Additional contacts were retrieved from contacting the regional Colombian chapter of the International Society for Pharmacoeconomics and Outcomes Research (ISPOR), as well as the IETS. The initial emails approaching potential participants (n = 64) about participating in the survey were sent via the in-build distribution function of Qualtrics software. This had the advantage that responses could be monitored, and persons who had not responded within 2 weeks could be sent reminders. The recruitment period lasted from May 1, 2017 to July 18, 2017.

Barriers and facilitators

The list of barriers and facilitators was similar to a previous study conducted in The Netherlands. In brief, barriers and facilitators were derived from an international literature review that included all suitable articles that were published before January 2015. Further information on the identification and selection of barriers and facilitators can be retrieved from the article describing the study¹². The final master list included a total of 22 barriers and 19 facilitators (see Supplementary Table A for the full list).

Questionnaire

The questionnaire was translated by a professional English–Spanish translator, and proofread by a native speaker with experience in HTA. In each choice set, respondents had to choose the most and least important attribute from a set of five influencing factors that were randomly drawn from the master lists of barriers and facilitators. For the barriers BWS (14 choice sets) as well as for the facilitators BWS (12 choice sets), four differently arranged versions were created to increase variability. The questionnaire started with a few demographic questions (age, gender, professional role) and continued with one version of the barriers and one version of the facilitators, each of which were randomly assigned by Qualtrics. In addition to making B/W choices, participants rated on a Likert-scale (0 = very easy, 7 = very difficult) how difficult the choice task was. After the choice tasks, participants were asked general questions about HTA in Colombia (current importance, usage level, and future role). Participants also had the opportunity to propose additional barriers and facilitators that could be added to the list. The possibility of adding barriers and facilitators was given to explore whether the master list used in The Netherlands actually captures all priorities in Colombia. The translated version of the questionnaire was piloted and validated by three HTA experts, who suggested some changes to enhance comprehensibility. In particular, technical terms were adjusted and clarified.

Analysis

In order to exclude respondents who provided purely random answers, we included responses in the analysis only from respondents who completed both barriers and facilitators questionnaires with a higher fit statistic than the recommended 0.227¹⁵. The answers to the demographic questions were analyzed using descriptive statistics to get an overview of the participants’ characteristics. The preference data stemming from the BWS was analyzed using Hierarchical Bayes analysis, which is frequently used for BWS surveys¹⁶. The Bayesian approach is ideal for constructing hierarchical models, since it allocates credibility to the parameter values that are consistent with the data and prior knowledge¹⁷. Mean relative importance scores (RIS) and 95% confidence intervals (CI) were computed separately for the attributes of the barriers and facilitators. The RIS of each response to the barriers or the facilitators sums up to 100, and, the higher the score of the attribute, the higher its relative importance. All attributes were ranked from most important to least important according to their mean RIS. Consistent with the Dutch study, a RIS of >6.0 was used as a cut-off score for the most important barriers and facilitators¹². For the least important barriers and facilitators, a cut-off score of <3.0 was introduced for interpretation purposes. As a large quantity of respondents came from the IETS (n = 7, 39%), a one-way ANOVA analysis was conducted to detect possible differences between IETS and non-IETS employees. In addition, the ranking of the facilitators and barriers was compared with the results from the Netherlands. Analyses were performed using Sawtooth.

Results

Descriptive statistics

In total, of the 64 stakeholders approached, 38 (59%) started the survey, of whom 18 (47%) fully completed both BWS questionnaires, resulting in 252 barrier and 216 facilitator choice sets. The number of respondents per version of the choice sets was (Barriers: V1 = 4, V2 = 9, V3 = 2, V4 = 3) and (Facilitators: V1 = 8, V2 = 5, V3 = 3, V4 = 2). All participants’ responses had a fit statistic higher than 0.227 and were, therefore, included in the analysis. The mean age was 39.5 years (SD = 10.7); the youngest participant was 26, and the oldest 64 years. Most respondents were male (72%). The respondents were policymakers (n = 2; 11%), health professionals (n = 5; 28%), PhD students (n = 8; 17%), or senior HTA researchers (n = 8; 44%); of all respondents, seven (39%) were employed by IETS. Both BWS experiments were rated as moderately difficult (Barrier: Mean = 4.06, SD = 1.89; Facilitator: Mean = 3.56, SD = 1.15) on a Likert scale (0 = very easy, 7 = very difficult).

Relative importance of the barriers

The complete list of RIS of the barriers can be found in Table 1. The overall fit statistic of 0.402 can be regarded as good.

Table 1. A complete list of RIS scores of the barriers.

Rank	Barrier	RIS score (95% CI)	Raw scores (95% CI)
1	Inadequate presentation format	10.54 (8.11–12.97)	2.47 (1.41–3.53)
2	Absence of policy networks	7.21 (4.61–9.80)	0.79 (–0.33–1.92)
3	Insufficient legal support	6.62 (4.90–8.33)	0.83 (0.23–1.43)
4	Lack of contact and interaction	5.83 (3.39–8.26)	0.26 (–0.88–1.4)
5	Lack of financial resources	5.47 (3.34–7.61)	0.05 (–1.11–1.21)
6	Lack of support	5.27 (3.58–6.95)	0.34 (–0.29–0.98)
7	No access to relevant HTA research	5.17 (3.63–6.71)	0.41 (–0.18–0.99)
8	Uncertainty	5.16 (2.90–7.41)	0.09 (–0.84–1.02)
9	Absence of appropriate incentives	5.06 (3.85–6.27)	0.49 (0.09–0.89)
10	Lack of longstanding relation	4.94 (2.98–6.91)	0.25 (–0.57–1.08)
11	No guidelines	4.76 (3.20–6.32)	0.30 (–0.28–0.88)
12	Lack of awareness	4.56 (2.94–6.18)	0.02 (–0.67–0.71)
13	Limited generalizability	4.40 (2.18–6.61)	–0.13 (–0.98–0.72)
14	No availability to relevant HTA research	3.83 (2.82–4.85)	0.08 (–0.28–0.44)
15	Lack of qualified human resources	3.30 (1.51–5.09)	–0.87 (–1.95–0.2)
16	Insufficient support by stakeholders	3.27 (1.82–4.72)	–0.53 (–1.3–0.25)
17	Lack of timeliness	3.21 (2.00–4.42)	–0.29 (–0.72–0.15)
18	Lack of transparency of HTA research findings	2.99 (1.86–4.12)	–0.55 (–1.22–0.13)
19	Insufficient quality	2.38 (1.61–3.15)	–0.62 (–1.04 to –0.2)
20	Lack of credibility	2.21 (0.84–3.58)	–1.00 (–1.73 to –0.27)
21	Lack of consensus between HTA findings	2.09 (1.40–2.78)	–0.83 (–1.29 to –0.37)
22	No explicit framework for decision-making process	1.75 (0.75–2.75)	–1.58 (–2.37 to –0.79)

The three most important barriers (RIS score above 6.0) were “Inadequate presentation format”, RIS = 10.5 (8.1–13.0); “Absence of policy networks”, RIS = 7.2 (4.6–9.8); and “Insufficient legal support”, RIS = 6.6 (4.9–8.3). The next most important barriers were “Lack of contact and interaction”, RIS = 5.8 (3.4–8.7) and “Lack of financial resources”, RIS = 5.5 (3.3–7.6). The five least important barriers (RIS score below 3.0) were “No explicit framework”, “Lack of consensus between HTA findings”, “Lack of credibility”, “Insufficient quality”, and “Lack of transparency”.

Relative importance of the facilitators

The complete list of RIS of the facilitators can be found in Table 2. The overall fit statistic (0.36) can be regarded as good.

Table 2. Complete list of RIS scores of the facilitators.

Rank	Facilitator	RIS score (95% CI)	Raw scores (95% CI)
1	Appropriate timing	8.14 (5.23–11.06)	0.98 (0.13–1.84)
2	Clear presentation format	7.83 (5.22–10.44)	0.73 (–0.04–1.51)
3	Improving longstanding relation	7.45 (5.02–9.88)	0.69 (–0.02–1.4)
4	Appropriate incentives	7.44 (6.16–8.72)	0.8 (0.46–1.13)
5	Sufficient qualified human resources	6.90 (4.00–9.80)	0.45 (–0.46–1.36)
6	Availability to relevant HTA research	6.04 (4.74–7.34)	0.49 (0.12–0.86)
7	Availability of guidelines	5.57 (3.66–7.49)	0.26 (–0.36–0.88)
8	Access to relevant HTA research	5.56 (4.20–6.91)	0.36 (–0.05–0.77)
9	Sufficient financial resources	5.47 (3.36–7.58)	0.01 (–0.72–0.74)
10	More personal contact and interaction	5.23 (3.40–7.06)	0.08 (–0.48–0.64)
11	Sufficient support by stakeholders	5.23 (2.96–7.49)	–0.15 (–0.94–0.64)
12	Sufficient support	4.93 (3.04–6.83)	–0.21 (–0.94–0.53)
13	Sufficient awareness	4.93 (3.12–6.74)	–0.05 (–0.66–0.56)
14	Creation of policy networks	4.63 (2.73–6.52)	–0.09 (–0.65–0.47)
15	Sufficient credibility	3.64 (1.80–5.48)	–0.54 (–1.18–0.11)
16	Sufficient legal support	3.61 (2.10–5.13)	–0.57 (–1.2–0.05)
17	Higher transparency of HTA research findings	3.08 (1.85–4.30)	–0.80 (–1.4 to –0.2)
18	Sufficient quality	2.29 (1.01–3.58)	–1.18 (–1.76 to –0.6)
19	Availability of explicit framework for decision-making process	2.03 (1.09–2.98)	–1.26 (–1.82 to –0.71)

The six most important facilitators, with a RIS score that was higher than 6.0, were “Appropriate timing”, RIS = 8.1 (5.2–11.1); “Clear presentation format”, RIS = 7.8 (5.2–10.4); “Improving longstanding relation”, RIS = 7.5 (5.0–9.9); “Appropriate incentives”, RIS = 7.4 (6.2–8.7); “Sufficient qualified human resources”, RIS = 6.9 (4.0–9.8); and “Availability to relevant HTA research”, RIS = 6.0 (4.7–7.3). The two least important facilitators (RIS score below 3.0) were “Availability of explicit framework for decision-making process” and “Sufficient quality”. The next least important facilitators were “Higher transparency” (RIS = 3.1) and “Sufficient legal support” (RIS = 3.6).

Sub-group analysis of IETS

The differences in the RIS among the barriers were all statistically insignificant to the 5% level (see Supplementary Figure A). Significant differences in the ranking (p < .05) could be observed among only two facilitators: “Sufficient financial resources” (IETS Mean = 8.2, SD = 4.4 vs Non-IETS Mean = 3.7, SD = 3.61), F(1.16) = 5.60, p < .05; and “Higher transparency of HTA research findings” (IETS Mean = 1.4, SD = 1.1 vs Non-IETS Mean = 4.2, SD = 2.7), F(1.16) = 6.79, p < .05. (see Supplementary Figure B).

Additional barriers and facilitators

Table 3 shows the full list of additional barriers and facilitators that were suggested by the respondents. In total, five additional barriers and five additional facilitators were named by the respondents, and most of these are already captured by the existing ones. For example, the barrier “Existing gap of technical knowledge” and the facilitator “training of human talent responsible for HTA” resemble “sufficient/lack of qualified human resources”.

Table 3. List of additional barriers and facilitators.

Barriers	Facilitators
•Lack of involvement of hospital managers in the implementation and prioritization of HTA	•The circumstantial relationship with the market
•Standard methodology of realization. Standard methodology for determining the subject to be studied.	•The Universities
•Times for the development of the HTA are inadequate and there are no explicit processes that guide decision-making based on HTA	•Methodological tools of condensation and updating of knowledge that is practical for defining the state of the art in relation to the HTA
•The existing gap of technical knowledge regarding the methods of HTA and their limitations	•Epidemiological profile, training of human talent responsible for HTA at all levels
•Contextual independence according to the characteristics of the environment	•The availability of contextual analysis for the adoption of policies and the possibility of implementation

Comparison to the similar Dutch study

The most and least important barriers and facilitators in the study conducted in The Netherlands were quite distinct in comparison with the Colombian study. The most important barrier in the Netherlands, “No explicit framework for decision-making process”, was regarded as the least important in Colombia. Vice versa, the most important barrier in Colombia “Inadequate presentation format” was ranked as the 4th least important in The Netherlands. The second most important barrier in Colombia, “Absence of policy networks”, was regarded as the 3rd least important in The Netherlands. The closest matches among the most important were “Insufficient legal support” (CO: 3rd, NL: 8th) and “Lack of support” (CO: 6th, NL: 3rd). The closest least important barriers were “Lack of credibility” (CO: 3rd, NL: 5th) and “Insufficient quality” (CO: 4th, NL: 7th) (see Figure 1).

Figure 1. Relative importance scores of barriers: Colombia vs The Netherlands.

The most important facilitator in The Netherlands, “Availability of explicit framework for decision-making process”, was regarded as the least important in Colombia. Vice versa, the most important facilitator in Colombia, “Appropriate timing”, was regarded as the 2nd least important in the Netherlands. Colombia’s second most important facilitator, “Clear presentation format”, was ranked in the Netherlands as the 3rd least important, and the 6th most important facilitator in Colombia, “Availability to relevant HTA research”, ranked as the least important in the Netherlands. The 4th most important facilitator in the Netherlands, “Sufficient quality”, was the 2nd least important in Colombia. There was one match among the most important facilitators: “Appropriate incentives” (CO: 4th, NL: 3rd); the next closest pairs were “Sufficient qualified human resources” (CO: 5th, NL: 10th) and “Sufficient support” (CO: 6th, NL: 11th). “Sufficient credibility” was a match for least important facilitator (CO: 15th, NL: 15th) (see Figure 2).

Figure 2. Relative importance scores of facilitators: Colombia vs The Netherlands.

General questions about HTA in Colombia

The current importance of HTA in Colombia was rated 5.5 on a Likert-scale (0 = not important, 7 = very important), and the level of usage of HTA in Colombia was rated 4.8 on a Likert-scale (0 = never, 7 = very often). All except one respondent (94%) were aware of the existence of the IETS. To the question of whether HTA will play a larger role in decision-making in the next 5 years, the majority answered “Yes, definitely have a large role in decision-making” (66.7%), followed by “Probably, a large role in decision-making” (27.8%), and “Maybe, a somewhat larger role in decision-making” (5.6%).

Discussion

This study assessed the relative importance of barriers to and facilitators for the use of HTA in Colombia. Notwithstanding the existence of many antonyms, only one pair of barriers and facilitators was among the most important: “Inadequate presentation format” (first place) and “clear presentation format” (second place). A clear presentation format is, thus, shown to be very important, as HTA reports seem to be perceived as overwhelming in terms of the amount of text and also in technical terms. A survey that explored the transferability of HTA reports in LA and the Caribbean came to the conclusion that reports from these regions had a lower methodological quality than reports from outside these regions⁶. This lack of transferability can be regarded as one possible explanation for the ranking of the sixth most important facilitator, “Availability to relevant HTA research”. The fifth most important facilitator, “Sufficient qualified human resources”, shows the need for more staff or better educated staff, which is not surprising as, even in high-income countries, agencies are not able to conduct sufficient amounts of assessments⁶. This is in line with the view of the former Executive Director of the IETS, Hector E. Castro, who named Colombia’s limited technical capacity for conducting and interpreting HTA a key challenge⁹. Creating networks and facilitating exchange among stakeholders appears to be of great importance, as can be seen in the most important facilitator: “Improving longstanding relation” (third place), the second most important barrier: “Absence of policy networks”, and the fourth most important barrier: “Lack of contact and interaction”. Pinchon-Riviere et al.¹⁸ named “active engagement with stakeholders” as a procedural principle for the robust operation of HTA, and considered it among other factors to be uniformly poor in LA. Their survey showed particularly large gaps in key principles related to the link between HTA and decision-making, which has to be regarded as a main challenge for LA. The key principle “HTA should be timely” belonging to this group, which resembles the most important facilitator “Appropriate timing” was also rated as one of the most relevant, but simultaneously as one with a large gap in application¹⁸. The findings suggest that, for the long-term, it is important to invest in the education of human resources to overcome the lack of conceptual and technical knowledge in HTA. In the meantime, more easily worded, more concise or standardized reports could help to reduce the workload and overcome the overburdening through difficult HTA reports. Other than that, dialog and cooperation between researchers and policy-makers should be promoted, which is also supported by the study conducted in LA¹⁸.

The IETS sub-group analysis showed significant differences within the facilitator, “Sufficient financial resources”, which was ranked among the most important by the IETS group (RIS >6). Another significant difference could be found in the facilitator “Higher transparency”. Despite not being among the most important in either of the groups, it is worth mentioning that the IETS group ranked it particularly low, as the second least important facilitator. Accordingly, for the IETS group, financial resources seem to be a major concern, while transparency seems to be of lesser importance. The additional barriers and facilitators did not differ substantially from those on the Dutch list, from which it can be concluded that similar barriers and facilitators appear to be important. However, since respondents expressed their need for extra barriers and facilitators, existing attributes might not be understood or interpreted properly.

Surprisingly there is just one facilitator which was regarded as most important (RIS > 6) in both The Netherlands and Colombia: “Appropriate incentives” (CO: 4th, NL: 3rd). The barriers received a completely different rating compared to The Netherlands; not a single barrier amongst the most important directly matches, suggesting that preferences in barriers and facilitators with regard to the use of HTA could differ between middle-income and high-income countries. Whereas, in The Netherlands networks and relationships among the relevant stakeholders don’t seem to be an issue, respondents in Colombia regarded the barrier “Absence of policy networks” as very important (CO: 2nd, NL 20th), along with the barrier “Lack of contact and interaction” (CO: 4th, NL: 7th) and the facilitator “Improving longstanding relation” (CO: 3rd, NL: 12th). A possible explanation for this observation is that HTA Institutes and the topic HTA, in general, are fairly new in LA and Colombia. Whereas the first EU HTA project Eur-assess started in 1993¹⁹, and European institutes and networks have existed and cooperated with each other for several years (SE: SBU*1987, UK: NICE*1999, GER: IQWiG*2004, EU: EUnetHTA*2004)⁹, the first approaches toward raising awareness about HTA in LA in the 1990s were not too successful. The first significant increase in interest in HTA could be observed during the 2008 HTAi meeting in Montreal, which was instrumental in creating the basis for RedETSA²⁰. Hector E. Castro describes the development of HTA in Colombia until the creation of the IETS as “starting from ‘scratch’ in the mid-2000s up to the formulation by law 1438 of IETS last January 2011” p.101⁹. A possible explanation for the wide discrepancy in the relevance of the barriers and facilitators between the two countries could be the different level of development of HTA, differences among stakeholders, or differences in the HTA focus, which likely are due to different priorities and needs. Generally, HTA seems to be regarded as relevant by stakeholders in Colombia, and, despite the great challenges to be overcome, most participant were positive about the future role of HTA in decision-making.

This study has several limitations, which could potentially limit the validity of the results. Despite great efforts to collect as many qualified respondents as possible, only 18 out of the 38 who started the questionnaire delivered complete data. One reason might be the length of the questionnaire, since respondents took up to 30–45 minutes to complete it. As HTA is a new topic in Colombia, it can be assumed that the availability of potential respondents is lower than in Europe. Despite the perceived difficulty of the choice tasks as moderate, this could misrepresent the people who eventually dropped out due to lack of understanding. The results of the study need to be interpreted with caution, due to the lack of power resulting from the small sample size. However, there is a lack of guidelines and justification with regard to the sample size in BWS studies. The number of participants and analyses are in line with other best–worst scaling in recent literature^12,21. Since the list of barriers and facilitators was derived from a study conducted in a developed country without being adapted to local circumstances, it does not necessarily fully represent the relevant barriers and facilitators for Colombia. Since only two countries were compared, the results are not necessarily applicable to other high-income and LMIC in general. The language barrier could also have played a role, since the main author only has a moderate level of Spanish, complicating the questionnaire translation, data collection, and correspondence with respondents, which was mainly conducted in Spanish.

In conclusion, barriers and facilitators that are related to technical aspects of processing HTA reports and to the contact and interaction between researchers and policymakers had the greatest importance. A key priority in advancing the use of HTA in Colombia would, therefore, be to train HTA specialists and create longstanding relationships with policymakers. To establish research–policy relations, it is recommended that policymakers be more connected with HTA research, and co-create interventions and evidence, to enhance commitment, strengthen the network, and enhance the transfer of information. This study, therefore, corroborates previous literature, by quantifying the importance of the barriers and facilitators reported in the literature.

Transparency

Declaration of funding

This manuscript was not funded.

Declaration of financial/other relationships

The authors have disclosed that they have no significant relationships with, or financial interests in, any commercial companies related to this study or article. JME peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgments

We would like to thank Elisa Convers for her willingness to help with the translation of the questionnaire, and all the participants for completing the survey. Moreover, we would like to acknowledge Aurelio Mejía for offering the possibility of visiting the Instituto de Evaluación Tecnológica de Salud (IETS) and providing us with many contacts for possible respondents.