Comparison of statistical analysis methods for object case best–worst scaling

Figures & data

Figure 1. Example of a BWS choice task.

Table 1. Master lists of the barriers to the uptake of HTA.

Lack of timeliness. Gap between HTA research and policy-making regarding timeliness of research

Lack of longstanding relation between professionals/policy-makers and researchers

Lack of contact and interaction, among policy-makers, HTA researchers, and other stakeholders

No availability of relevant HTA research for policy-makers

No access to relevant HTA research (or poor dissemination) for policy-makers

Lack of financial resources to conduct relevant HTA research

Lack of qualified human resources to conduct or understand relevant HTA research within the policy organization

Absence of policy networks (or observatories that promote the joint efforts of researchers and policy-makers)

No guidelines. Absence of adequate (reliable) HTA guidelines

Lack of awareness, within the organization to the relevance of HTA

Lack of support, within the organization to the use of HTA

Lack of transparency of HTA research findings, i.e. the process as to how presented research findings emerged is not clear

Insufficient quality of HTA research findings: quality not in accord with scientific requirements

Lack of credibility of the HTA research findings (industry-sponsored studies)

Lack of consensus between HTA findings, existence of different and opposing recommendations

Limited generalizability of HTA studies to the policy-maker’s context

Uncertainty surrounding HTA results

Inadequate presentation format. HTA reports overly long, too theoretical or abounding in technical jargon

No explicit framework for decision making process, that uses HTA evidence

Insufficient support by stakeholders. Policy-maker's perception of insufficient support by end-users (e.g. patient associations)

Insufficient legal support. Lack of legal or legislative support for the use of HTA

Absence of appropriate incentives for the implementation of cost-effective interventions

Table 2. Comparison of rankings for each analysis method.

	Count analysis		MNL		MXL			HB
List of barriers	Ranking	BWS score	Ranking	β	Ranking	β	η	Ranking	RIS
No explicit framework for decision-making process	1	0.26	1	0.65**	1	0.77**	1.46**	1	7.63
Limited generalizability	2	0.23	2	0.58**	2	0.72**	0.23	2	6.92
Lack of consensus between HTA findings	3	0.13	3	0.26*	3	0.35*	1.19**	4	5.86
No availability of relevant HTA research	4	0.12	4	0.24	4	0.29	1.16**	3	6.00
Lack of qualified human resources	5	0.06	6	0.09	7	0.17	1.18**	7	4.98
Insufficient quality	6	0.06	5	0.10	6	0.17	1.10**	5	5.12
Lack of support, within the organization	7	0.05	9	0.00	10	0.03	1.12**	10	4.78
Lack of awareness, within the organization	8	0.05	8	0.02	11	0.01	0.99**	8	4.83
Uncertainty surrounding HTA results	9	0.04	11	–0.03	9	0.05	1.29**	9	4.81
Lack of transparency of HTA research findings	10	0.03	7	0.05	8	0.05	0.38	12	4.46
Lack of timeliness	11	0.02	10	0.00	12	0.00	0.00	11	4.74
No access to relevant HTA research	12	0.01	12	–0.09	13	–0.06	1.30**	6	5.02
Lack of credibility	13	0.00	13	–0.10	14	–0.13	0.46	15	3.95
Insufficient support by stakeholders	14	–0.02	14	–0.20	15	–0.25	0.99**	14	4.09
Absence of appropriate incentives	15	–0.05	16	–0.28**	5	0.27	1.66**	13	4.20
Lack of contact and interaction	16	–0.05	15	–0.25**	16	–0.31*	0.91**	16	3.81
Lack of financial resources	17	–0.06	17	–0.29**	17	–0.45**	1.07**	17	3.59
Insufficient legal support	18	–0.13	18	–0.46**	19	–0.56**	1.40**	18	3.52
No guidelines	19	–0.15	19	–0.52**	18	–0.54**	1.32**	20	2.98
Lack of longstanding relation	20	–0.16	20	−0.54**	20	–0.64**	1.62**	19	3.29
Inadequate presentation format	21	–0.18	21	−0.62**	21	–0.80**	1.48**	21	2.89
Absence of policy networks	22	–0.19	22	–0.66**	22	–0.92**	0.95**	22	2.51

List of barriers are ordered according to the count analysis ranking. The ranking indicates the position of each barrier in each of the statistical methods (e.g. “Lack of consensus between HTA findings” was ranked position three in the count analysis, MNL and MXL, and position four in the HB analysis).

Abbreviations: HB, Hierarchical Bayes ranking; RIS, Relative Importance Score; MNL, Multinomial Logit ranking; MXL, Mixed Logit ranking; β, relative preference compared to the reference factor; η, standard deviation term for the factor.

* p < 0.05;

** p < 0.01.

Table 3. Latent class analysis. Latent clusters and comparison of rankings.

	C1		C2		C3		C4		C5
Barriers	R	β	R	β	R	β	R	β	R	β
1. No explicit framework for decision-making process	5	1.08**	15	–0.88*	8	0.92*	1	0.91	1	0.79*
2. Limited generalizability	3	1.20**	3	0.67	9	0.78	7	–0.57	7	–0.34
3. Lack of consensus between HTA findings	11	0.51	2	0.74	7	1.13**	13	–1.21*	8	–0.79*
4. No availability of relevant HTA research	2	1.31**	6	0.04	17	0.21	8	–0.74	20	–1.85**
5. Lack of qualified human resources	10	0.68*	21	–1.79**	3	2.14**	6	–0.49	15	–1.28**
6. Insufficient quality	1	1.37**	9	–0.14	14	0.43	16	–1.70**	18	–1.57**
7. Lack of support, within the organization	13	0.14	20	–1.56**	5	1.40**	4	–0.16	5	–0.08
8. Lack of awareness, within the organization	9	0.72*	19	–1.38**	11	0.65	2	0.41	9	–0.82*
9. Uncertainty surrounding HTA results	14	0.07	1	1.13*	20	–0.14	14	–1.27*	10	–1.07**
10. Lack of transparency of HTA research findings	7	0.75**	5	0.14	12	0.62	17	–1.87**	13	–1.16**
11. Lack of timeliness (ref)	16	0.00	7	0.00	19	0.00	3	0.00	4	0.00
12. No access to relevant HTA research	4	1.11**	8	–0.02	16	0.26	12	–1.20*	22	–2.41**
13. Lack of credibility	6	0.78**	12	–0.46	13	0.45	19	–2.06**	14	–1.21**
14. Insufficient support by stakeholders	19	–0.47	10	–0.20	6	1.35**	18	–1.94**	6	–0.08
15. Absence of appropriate incentives	17	–0.23	11	–0.31	18	0.00	21	–2.40**	3	0.20
16. Lack of contact and interaction	20	–0.94**	4	0.33	1	2.85**	10	–1.00	19	–1.58**
17. Lack of financial resources	15	0.02	17	–1.17*	10	0.72	11	–1.16	12	–1.15**
18. Insufficient legal support	12	0.16	22	–2.32**	15	0.36	22	–4.25**	2	0.36
19. No guidelines	8	0.73**	18	–1.26	22	–0.31	20	–2.21**	21	–2.10**
20. Lack of longstanding relation	22	–1.37**	13	–0.47	4	2.06**	9	–0.83	11	–1.07**
21. Inadequate presentation format	18	–0.46	14	–0.87*	21	–0.29	5	–0.29	17	–1.55**
22. Absence of policy networks	21	–1.17**	16	–1.01*	2	2.16**	15	–1.69**	16	–1.39**
Latent class probability		0.39**		0.14**		0.13**		0.11**		0.23**

List of barriers are ordered according to the count analysis ranking of Table 2 (e.g. barrier “Lack of timeliness” is ranked 16^th in C1).

Abbreviations: C1, C2, C3, C4, C5, Class 1, Class 2, Class 3, Class 4, Class 5; R, ranking.

* p < 0.05;

** p < 0.01.

Figure 2. Kernel density estimate for “Limited generalizability” (a) and “No explicit framework for decision-making process” (b).