https://orcid.org/0000-0002-1091-9139
ABSTRACT
Background
Organizational aspect is rarely considered in healthcare. However, it is gradually seen as one of the key aspects of the decision-making process as well as clinical and economic dimensions. Our primary objective was to identify criteria already used to assess the organizational impact of medical innovations. Our secondary objective was to structure them into an inventory to support decision-makers to select the relevant criteria for their complex decision-making issues.
Materials and methods
A search using the Medline database was conducted in June 2019. The records published between January, 1990 and December, 2018 were identified. The publications cited by the authors of the included articles and the websites of health technology assessment agencies, units or learned societies identified during the search were also consulted. The identified criteria were structured in an inventory.
Results
We selected 107 records of a wide range of evidence mostly published after the 2000s. We identified 636 criteria that we classified into five categories: people, task, structure, technology, and surroundings.
Conclusion
Criteria selection is a crucial step in any multi-criteria decision analysis (MCDA). This work is the first step in the development of a validated MCDA method to assess the organizational impact of medical innovations.
Introduction
Nowadays, health technology assessment (HTA) is mainly conducted based on clinical effectiveness and safety studies and medico-economic studies [Citation1]. The description of other aspects such as organizational, ethic or strategic aspect is rarely considered. Nevertheless, complex decision-making requires the consideration of all relevant aspects. The assessment of innovative drugs, medical devices, medical or surgical procedures and organizations like care pathways currently established does not take into account all the aspects, which characterize an organization in health. This pitfall results in a lack of rationality in decision-making as well as a sub-optimal use of resources. According to Leavitt’s model, an organization is made up of four interdependent entities: people, task, structure and technology [Citation2]. The modified Leavitt’s organizational model includes interactions with surroundings. Evaluating the organizational impact of an innovation includes the study of the expected results on one or more of these entities but also of the changes induced on other entities and surroundings. Its evaluation is all the more important for disruptive innovations, those that create or replace than for incremental innovations, those that improve [Citation3]. In addition, healthcare organizations are characterized by a high level of complexity and a very dynamic environment. That is why, the assessment of organizational impact is gradually seen as one of the key aspects of the decision-making process as well as clinical and economic dimensions [Citation4].
In view of this observation, there is a need for effective decision-making tools enabling a systemic approach of decision-making issues. Multi-criteria decision analysis (MCDA) methods aim to facilitate the identification of the best possible solution to a given problem that requires considering a set of aspects or criteria, which are often heterogeneous. These methods seem to meet this need [Citation5]. Developed in the 1970s and widely used in non-medical domains such as farming, energy or marketing [Citation6–Citation8], they are booming in healthcare since the 2000s [Citation9]. Core components of any MCDA method are the alternatives in competition with one another, the criteria by which alternatives are assessed, the level of performance of each alternative for each criterion and the relative weight of each criterion in relation to the other [Citation5]. The selection of relevant criteria is one of the most crucial steps of any MCDA method.
To the best of our knowledge, no specific tool exists to help decision-makers assess the organizational impact of medical innovations. The primary objective of our study was to identify all the criteria already used to evaluate it. The secondary objective of our study was to build a structured inventory of these criteria to support decision-makers select the relevant criteria for assessing medical innovations considering organizational aspect. This inventory was intended to be used regardless of the type of innovation being evaluated (drugs, medical devices, medical or surgical procedures or organizations) and regardless of the point of view adopted (from a care unit perspective to a national perspective).
Materials and methods
Literature review
We conducted a review of the published and gray literature to identify as many criteria as possible that have already been used to assess medical innovations regarding organizational aspect. All publications dealing with assessment of the organizational impact of health products or organizations, regardless of the decision-making perimeter (health care unit, institution, area, country) were included. An HTA or MCDA methodology had to be implemented and the criteria for assessing the organizational aspect had to be detailed. In order to identify as many criteria in as many different contexts as possible, no exclusion criteria regarding the type of publication were applied. Publications were excluded when their full text was not available. Due to limited language skills, only publications written in English or French were included.
First, a literature search using the Medline database (Pubmed, USA National Library of Medicine, the USA) was conducted in June 2019. Records published between January, 1990 and December, 2018 were retrieved. Indeed, HTA emerged in the 1990s. The following keywords were used to develop two search equations: “decision support tool “, “ decision support model “, “ decision support technique “, “ decision-making “ (Medical Subject Headings (MeSH) term and key word), “ health technology assessment “, “ HTA “, “ criteria “ and “ systematic review “ (). Second, the publications cited by the authors of the included articles were reviewed. Third, the websites of international, national or regional HTA agencies or units and HTA learned societies identified during the research were consulted.
Table 1. Search equations used for the literature review on the Medline database.
For each selected document, the following elements were collected: date of publication, name of the first author or institution, origin (country or continent), type of document (article, guide book, thesis, poster, website), content of the document (HTA report or model, MCDA tool, study, round table report), level (local, regional, national, international), criteria and their definition when available.
Structuring the criteria inventory
First, a standardized term or expression was associated with each criterion to facilitate their grouping by theme. We counted the number of similar criteria that were grouped within the same theme. Second, criteria were structured into five categories: people, task, structure and technology and surroundings. These are the five categories of the modified Leavitt’s organization model. We chose this model because it is widely known and for its simplicity. We used the definitions of each category proposed by the Danish Center for Evaluation and Health Technology Assessment (DACEHTA) in its handbook to classify the criteria [Citation2].
Results
After removal of duplicates and exclusion of articles whose full text was not available or not written in English or French, consulting the Medline database allowed us to identify 245 potential publications. After review of the abstract and/or full-text, 22 of these publications, that met inclusion criteria, were selected. References cited by the authors of these 22 publications as well as the websites of HTA agencies or units and HTA learned societies identified during the research were consulted to complete the data collection. Through this research strategy, 107 records were included (). The origin, the type of publication, the content of the publication, and the level if the publication presented an HTA model or MCDA tool were collected ().
Table 2. Characteristics of the selected records.
Figure 1. Research strategy flowchart.
The great majority of selected publications (n = 103) were published after the 2000s. This is consistent with the growth of HTA and MCDA methods in healthcare, previously reported.
More than half of the selected publications were European (n = 61). For example, HTA core model® from the European Network for Health Technology Assessment (EunetHTA) project and Hospital-based HTA core model® from the Adopting Hospital based Health Technology Assessment (AdHopHTA) project, as well as the health technology assessment handbook and the introduction to mini-HTA published by DACEHTA were selected [Citation2,Citation10–Citation12]. A quarter of the selected documents were published in North America (n = 31), the vast majority of which were published in Canada (n = 28). HTA units, like the one established by the McGill University Health Centre (MUHC) are widely developed in this country [Citation13]. Five selected publications came from Oceania, including four publications from Australia. The remaining documents were from Asia (n = 4), South America (n = 3) and Africa (n = 2). The poor number of publications from these regions is probably due to the recent growth of HTA in developing countries [Citation14]. Finally, a publication was written by an international organization.
Among the 107 records selected there were 71 articles, 11 guidebooks, 9 websites of HTA agencies, 7 HTA agency reports, 5 communication papers, 3 theses and one poster. Half of these publications introduced an HTA model or an MCDA tool. Among the HTA models (n = 28), two were international, ten were national, thirteen were regional and three were local. Regarding MCDA tools (n = 26), five were international, three were national, six were regional and twelve were local. For example, the Evidence and Values: Impact of Decision Making (EVIDEM) and the Valutazione delle tecnologie sanitarie frameworks as well as Matrix4Value® and Innovative Device Assessment (IDA) tools [Citation15–Citation18] were selected. Several of the selected articles (n = 44) were literature reviews or surveys (interviews, Delphi method) regarding the use of HTA or MCDA methods in healthcare. For example, a study carried out in 2012 by Guindo and al. which focused on healthcare decision criteria was included [Citation19]. Moreover, HTA reports of national or regional agencies were identified (n = 6). This was the case of reports published by the Committee for Evaluation and Dissemination of Innovative Technologies (CEDIT), established by the Greater Paris University Hospitals (Assistance Publique des Hôpitaux de Paris, AP-HP) [Citation20].
A total of 636 criteria were identified, with an average of 5.9 criteria per record. A standardized term or expression was associated with each criterion to facilitate their grouping by theme and to classify them more easily into the five categories previously defined (‘people’, ‘task’, ‘structure’, ‘technology’ and ‘surroundings’) (). This process resulted in the creation of a structured inventory of all the criteria collected ().
Table 3. Inventory of criteria for assessing the organizational impact of medical innovations. List of references used in Table 3: (Citation2, Citation10-Citation12, Citation14, Citation16-Citation19, Citation23, Citation24, Citation26-Citation121).
Figure 2. Flowchart of the criteria structuring process.
This work aimed to help healthcare professionals select relevant criteria to assess the organizational impact of a medical innovation, whether it is a drug, a medical device, a medical or surgical procedure or an organization such as a care pathway. Decision-makers should keep these four requirements in mind when selecting criteria [Citation15,Citation21]:
Completeness: all relevant criteria are selected.
Absence of redundancy.
Mutual independence: the level of performance of each criterion is independent of the level of performance of the other criteria.
Operationality: the data needed to assess performance are available.
Considering the number of criteria per category, we recommended that eight criteria be selected as follows: two criteria of the ‘Surroundings’ category, two criteria of the ‘Tasks’ category, two criteria of the ‘People’ category, one criterion of the ‘Technology’ category and one criterion of the ‘Structure’ category.
The ‘Surroundings’ category included aspects related to legislation such as the applicability of existing legislation or the risk of conflicts as well as aspects related to cooperation with other organizations. It also described aspects linked to de-centralization (the distribution of the supply of care on the territory), coordination between care providers, communication, information, vigilance system and impact on the environment. Depending on the point of view adopted, interactions were studied between different services, institutions or territories. In addition, interactions between all stakeholders including patients and health authorities could be appreciated. One of the selected models considered criteria related to the border context.
The ‘Task’ category referred to aspects related to workflow, implementation of innovation, the care process, quality assurance and health pathways. The most frequently identified criterion in the selected documents was the workflow. Six sub-criteria were identified, i.e. the characteristics of the intervention, hospital stay, associated activities, time management, activity profile and, most importantly, performance. The second criterion in terms of occurrence was the implementation of innovation. It was subdivided into three sub-criteria relating to planning, method of deployment and evaluation of its success. The process of care criterion referred to risk of misuse, quality of care and patient recruitment. The quality assurance criterion included three sub-criteria relating to controls, indicators and risk management. Finally, the criterion pathways assessed the sectors/actors involved and the chronology of care.
The category ‘Personnel’ referred to aspects related to staff training, human resources and in particular staff resources and skills management. It also dealt with the working environment as well as health and safety at work.
The ‘Technology’ category referred to the resources, infrastructure and characteristics of innovation. The ‘resources’ criterion was subdivided into two sub-criteria concerning hardware and software including their compatibility with the innovation and budget. Financial resources were discussed in a qualitative manner, in contrast to the assessment of the economic aspect.
The ‘Structure’ category referred to both formal and informal structure of an organization. The most identified criterion in the selected documents was acceptability of innovation by health professionals and/or patients. One criterion assessed if the innovation fit with the organization and/or individuals culture, missions and values. Another concerned the congruence with the organization strategic plan. The last two criteria related to management and procedures.
Discussion
The construction of the search equation was complex due to the difficulty in finding the appropriate keywords and MeSH terms. Indeed, the keywords relating to the organizational aspect did not make it possible to identify a satisfactory number of publications and to retrieve the articles identified during the preliminary searches. One of the main reasons could be the low number of publications dealing with the assessment of the organizational aspect during health care decision-making. This is why we have used more general keywords such as ‘criteria’. The completeness of the data collection can be discussed. This data collection was limited to publications that were referenced on a single database. In addition, only publications written in English or French were selected and only one reviewer made the screening. We chose to select documents without any restrictions on the level of evidence to make an inventory as complete as possible of the criteria that have already been used to assess medical innovations regarding organizational aspect. We considered that the number of documents was sufficient and that the criteria identified were representative. The distribution of the criteria into the five categories of the modified Leavitt’s model was facilitated by the use of standardized terms. The definitions of the categories in the HTA handbook published by DACEHTA also assisted us in the allocation. During this step, interpretation and/or translation errors may have led to inaccuracy in wording.
In parallel with this work, an MCDA method was developed. To this end, a group of seven experts has been brought together for a day of experimentation. It is recommended that a multi-disciplinary group carried out the selection of criteria. If a patient could not be part of this group, one of the experts was a citizen. Indeed, the analysis of patient preferences benefits from an increasing interest in healthcare decision-making [Citation22]. On this occasion, a complex decision-making issue was addressed and the inventory of criteria was used for the selection of relevant criteria. During the selection process, the experts identified criteria whose meaning was unclear. As a result, the wording of these criteria has been changed. Some criteria were flagged as implicitly positive or negative, and their wording was also changed. Regarding the number of criteria selected, a literature review of MCDA methods found a mean number of 8.2 criteria used to evaluate interventions (range 3 to 19) [Citation23]. However, there does not seem to be a consensus on the optimal number of criteria to be selected. The number of criteria is directly related to the complexity of the innovation being assessed. However, care must be taken, on the one hand, not to omit any relevant criterion and, on the other hand, to comply with the four requirements mentioned above. When selecting criteria, if redundancy is identified, the creation of a composite criterion is a common solution used in MCDA methods to solve this problem [Citation16,Citation24]. This solution is more appropriate than adding or removing criteria [Citation24]. In addition, it is recommended that each member of the group first performs an individual screening. Differences of opinion reflect the diversity of individual perceptions of participants. These differences are not a limitation, they help to identify the criteria that need to be discussed, and they encourage exchanges with the aim of reaching consensus. A transparent display of the method of selection and the criteria selected is a ‘reasonableness’ approach to decision-making [Citation24,Citation25]. Training of professionals in the MCDA method is a prerequisite.
Conclusion
Taking into account the organizational aspect is a major challenge for the evaluation of medical innovations, especially for disruptive innovations. A review of the published and gray literature was conducted to collect and classify in a structured inventory all the criteria that have already been considered to assess the organizational impact of medical innovations, whether they concern drugs, medical devices, medical or surgical procedures or organizations. The selection of relevant criteria is one of the crucial steps in any MCDA method. The inventory helps decision-makers to select the relevant criteria for their decision-making issue. This review was carried out in parallel with the development of an MCDA method. This work is the first step in the development of a criteria selection tool integrated into a validated MCDA method for assessing the organizational aspect in healthcare decision-making.
Disclosure statement
No potential conflict of interest was reported by the authors.
Additional information
Funding
References
- Fery-Lemonnier E. [Nine success paths for an effective integration of medical innovations]. Rev Hosp Fr. 2003;493:25–11.
- DACEHTA. Health technology assessment handbook [Internet]. 2007. [cited 2017 Aug 7]. Available from: http://www.sst.dk/~/media/C0ED080616D7410E8B6020B903AD0339.ashx
- Christensen C-M. The innovator’s dilemma: when new technologies cause great firms to fail. Boston, Massachussets: Harvard Business Review Press; 1997. p. 225.
- Kidholm K, Ølholm AM, Birk-Olsen M, et al. Hospital managers’ need for information in decision-making–An interview study in nine European countries. Health Policy Amst Neth. 2015;119(11):1424–1432.
- Thokala P, Duenas A. Multiple criteria decision analysis for health technology assessment. Value Health. 2012;15(8):1172–1181.
- Food and agriculture organization of the United Nations. A multi-criteria approach for better food safety decisions [Internet]. 2015 [cited 2017 Aug 9]. Available from: http://www.fao.org/3/a-i3920e/i3920e15.pdf
- Steinhilber S, Del Rio P, Toro F, et al. Multi-criteria decision analysis - Assessing policy pathways for renewables support in the EU after 2020. Intel Energy Eur. [Internet]. 2014 [cited 2017 Aug 9] p. 68. Available from:: http://www.res-policy-beyond2020.eu/pdffinal/Multi-criteria%20decision%20analysis%20(beyond2020%20-%20D6-1).pdf
- Denguir A Possibilistic framework for multicriteria and multi actors decision aiding: application for marketing and benchmarking of e-commerce websites [PhD thesis]. Chambéry, France: Savoie University; 2007.
- Diaby V, Campbell K, Goeree R. Multi-criteria decision analysis (MCDA) in health care : a bibliometric analysis. Oper Res Health Care. 2013;2(1–2):20–24.
- AdHopHTA project partners. The AdHopHTA handbook : a handbook of hospital-based health technology assessment [Internet]. 2015 [cited 2017 Jan 2]. Available from: http://www.adhophta.eu/sites/files/adhophta/media/adhophta_handbook_website.pdf
- Eunethta. HTA Core Model version 3.0 [Internet]. 2016 [cited 2017 Jan 3]. Available from: http://eunethta.eu/sites/5026.fedimbo.belgium.be/files/HTACoreModel3.0.pdf
- DACEHTA. Introduction to mini-HTA : a management and decision tool for the hospital service [Internet]. 2005 [cited 2017 Aug 9]. Available from: http://www.sst.dk/~/media/47C62A769EBC4E80A153F986C5348F55.ashx
- McGregor M, Brophy J. End-user involvement in health technology assessment (HTA) development: a way to increase impact. Int J Technol Assess Health Care. 2005;21(2):263–267.
- Youngkong S, Kapiriri L, Baltussen R. Setting priorities for health interventions in developing countries: a review of empirical studies. Trop Med Int Health. 2009;14(8):930–939.
- Goetghebeur MM, Wagner M, Khoury H, et al. Evidence and value: impact on decision making, the EVIDEM framework and potential applications. BMC Health Serv Res. 2008;8:270.
- Radaelli G, Lettieri E, Masella C, et al. Implementation of EUnetHTA core Model® in Lombardia: the VTS framework. Int J Technol Assess Health Care. 2014;30(1):105–112.
- Sampietro-Colom L, Morilla-Bachs I, Gutierrez-Moreno S, et al. Development and test of a decision support tool for hospital health technology assessment. Int J Technol Assess Health Care. 2012;28(4):460–465.
- Martelli N, Hansen P, van den Brink H, et al. Combining multi-criteria decision analysis and mini-health technology assessment: A funding decision-support tool for medical devices in a university hospital setting. J Biomed Inform. 2016;59:201–208.
- Guindo LA, Wagner M, Baltussen R, et al. From efficacy to equity: literature review of decision criteria for resource allocation and healthcare decisionmaking. Cost Eff Resour Alloc CE. 2012;10(1):9.
- AP-HP. [Opening workshop of Evaluation of the médical device (CEDM)(AP-HP)] [Internet]. 2008 [cited 2017 Jun 26]. Available from: http://www.informationhospitaliere.com/actualite-13450-atelier-d-ouverture-centre-d-evaluation-dispositif-medical-cedm-ap-hp.html
- Marsh K, IJzerman M, Thokala P, et al. Multiple Criteria Decision Analysis for Health Care Decision Making–Emerging Good Practices: report 2 of the ISPOR MCDA Emerging Good Practices Task Force. Value Health J Int Soc PharmacoEcon Outcomes Res. 2016;19(2):125–137.
- Marsh K, Caro J, Zaiser E, et al. Patient-centered decision making: lessons from multi-criteria decision analysis for quantifying patient preferences. Int J Technol Assess Health Care. 2018;34(1):105–110.
- Marsh K, Lanitis T, Neasham D, et al. Assessing the value of healthcare interventions using multi-criteria decision analysis: a review of the literature. PharmacoEconomics. 2014;32(4):345–365.
- Martelli N Introduction of innovative medical devices at French university hospitals : an overview of hospital-based health technology assessment initiatives and development of a decision support tool [PhD thesis]. Paris, France: Paris Sud University; 2015.
- Daniels N. Accountability for reasonableness. BMJ. 2000;321(7272):1300–1301.
- Le Gales C, Moatti JP. Searching for consensus through multi-criteria decision analysis. Assessment of screening strategies for hemoglobinopathies in southeastern France. Int J Technol Assess Health Care. 1990;6(3):430–449.
- Duthie T, Trueman P, Chancellor J, et al. Research into the use of health economics in decision making in the UK–Phase II. Is health economics ‘for good or evil’? Health Policy Amst Neth. 1999;46(2):143–157.
- Ormstad SS, Graff BA, Norderhaug IN Survey and Discussion of Existing Mini-HTA Systems Internationally [Internet]. Oslo, Norway: Knowledge Centre for the Health Services at The Norwegian Institute of Public Health (NIPH); 2010 [cited 2020 May 17] (NIPH Methods Resources). Available from: http://www.ncbi.nlm.nih.gov/books/NBK482057/
- Mullen PM. Quantifying priorities in healthcare: transparency or illusion? Health Serv Manage Res. 2004;17(1):47–58.
- Hofmann B. Toward a procedure for integrating moral issues in health technology assessment. Int J Technol Assess Health Care. 2005;21(3):312–318.
- Hummel JMM, Snoek GJ, van Til JA, et al. A multicriteria decision analysis of augmentative treatment of upper limbs in persons with tetraplegia. J Rehabil Res Dev. 2005;42(5):635–644.
- Wirtz V, Cribb A, Barber N. Reimbursement decisions in health policy–extending our understanding of the elements of decision-making. Health Policy Amst Neth. 2005;73(3):330–338.
- Cerezo Espinosa de Los Monteros J, Villegas Portero R. Update of the guide for the acquisition of new technologies. Seville: andalusian Agency for Health Technology Assessment (AETSA) [Internet]. [cited 2020 May 17]. Available from: https://www.crd.york.ac.uk/CRDWeb/ShowRecord.asp?ID=32007000453&ID=32007000453
- Douw K, Vondeling H, Oortwijn W. Priority setting for horizon scanning of new health technologies in Denmark: views of health care stakeholders and health economists. Health Policy Amst Neth. 2006;76(3):334–345.
- Wilson ECF, Rees J, Fordham RJ. Developing a prioritisation framework in an English Primary Care Trust. Cost Eff Resour Alloc CE. 2006;4:3.
- Cleemput I, Van Den Bruel A, Kohn L, Vlayen J, Vinck I, Thiry N, et al. Search for Evidence & Critical Appraisal: Health Technology Assessment (HTA). Brussels: Belgian Health Care Knowledge Centre (KCE); 2007. KCE Process notes (D2007/10.273/40).
- Ritchie K, Bradbury I, Eastgate J, et al. The clinical and cost effectiveness of screening for meticillin-resistant Staphylococcus aureus (MRSA). Glasgow: Quality Improvement Scotland (NHS QIS). Health Technology Assessment Report 9. [Internet]. 2007 [cited 2020 May 19]. Available from: https://www.crd.york.ac.uk/crdweb/ShowRecord.asp?ID=32006000901&ID=32006000901
- Bergh C, Alopaeus E, Jivegård L, et al. [Regional HTA work can have a good impact on health care. Good examples form Vastra Gotaland]. Lakartidningen. 2010;107(29–31):1780–1783.
- Wilson E, Sussex J, Macleod C, et al. Prioritizing health technologies in a Primary Care Trust. J Health Serv Res Policy. 2007;12(2):80–85.
- González-Zapata LI, Alvarez-Dardet C, Ortiz-Moncada R, et al. Policy options for obesity in Europe: a comparison of public health specialists with other stakeholders. Public Health Nutr. 2009;12(7):896–908.
- Saarni SI, Hofmann B, Lampe K, et al. Ethical analysis to improve decision-making on health technologies. Bull World Health Organ. 2008;86(8):617–623.
- Tannahill A. Beyond evidence–to ethics: a decision-making framework for health promotion, public health and health improvement. Health Promot Int. 2008;23(4):380–390.
- van Til JA, Renzenbrink GJ, Dolan JG, et al. The use of the analytic hierarchy process to aid decision making in acquired equinovarus deformity. Arch Phys Med Rehabil. 2008;89(3):457–462.
- Kapiriri L, Norheim OF, Martin DK. Fairness and accountability for reasonableness. Do the views of priority setting decision makers differ across health systems and levels of decision making? Soc Sci Med. 2009;68(4):766–773. 1982.
- Wild C, Simpson S, Douw K, et al. Information service on new and emerging health technologies: identification and prioritization processes for a European union-wide newsletter. Int J Technol Assess Health Care. 2009;25(Suppl 2):48–55.
- Berra S, Sánchez E, Pons JMV, et al. Setting priorities in clinical and health services research: properties of an adapted and updated method. Int J Technol Assess Health Care. 2010;26(2):217–224.
- Kaltoft M, Kirketerp G, Dowie J Literature review on the use of handhelds in acute care: towards a patient-centred Health Technology Assessment [Internet]. 2010. Available from: https://www.academia.edu/1668470/Literature_review_on_the_use_of_handhelds_in_acute_care_towards_a_patient-centred_Health_Technology_Assessment_HTA_
- Kroese M, Burton H, Whittaker J, et al. A framework for the prioritization of investment in the provision of genetic tests. Public Health Genomics. 2010;13(7–8):538–543.
- MAST project partners. MAST model info [Internet]. [cited 2016 Dec 15]. Available from: http://www.mast-model.info/
- Plüddemann A, Heneghan C, Thompson M, et al. Prioritisation criteria for the selection of new diagnostic technologies for evaluation | BMC Health Services Research | full Text. BMC Health Serv Res. [ [Internet]. 2010 [cited 2020 May 23];109(10). Available from]. [];(). : https://bmchealthservres.biomedcentral.com/articles/10.1186/1472-6963-10-109
- Grundstrom JP. Mini-HTA trends for medical devices in the Nordics. 16th Annual International Society for Pharmacoeconomics and Outcomes Research. Dublin, ireland: ISPOR; 2011.
- Agenzia Nazional per I Servizi Sanitari Regionali. Systematic review of the methodological instruments used in Health Technology Assessment [Internet]. 2011. Available from: http://www.salute.gov.it/portale/temi/documenti/dispositiviMedici/C_17_pagineAree_1202_listaFile_itemName_17_file.pdf
- Airoldi M, Morton AD, Smith J, et al. Healthcare prioritisation at the local level: a socio-technical approach [Internet]. 201AD [cited 2020 May 23]. Available from: http://www.lse.ac.uk/management/home.aspx
- A.Li.Sa. [Internet]. [cited 2020 May 23]. Available from: https://www.alisa.liguria.it/
- Tila - THL [Internet]. Terveyden ja hyvinvoinnin laitos. [cited 2020 May 23]. Available from: https://thl.fi/finohta
- Knies S, Lombardi G, Commers M, et al. Supporting decision making in cross-border regions: a health technology assessment tool for hospitals. Int J Technol Assess Health Care. 2013 Jan;29(1):71–78.
- Norwegian Institute of Public Health [Internet]. Norwegian Institute of public health. [cited 2020 May 23]. Available from: https://www.fhi.no/en/
- Tromp N, Baltussen R. Mapping of multiple criteria for priority setting of health interventions: an aid for decision makers. BMC Health Serv Res. 2012;12:454.
- Haute Autorité de Santé. Élaboration de matrices d’impacts des effets attendus de la télémédecine: applications aux chantiers prioritaires. Paris, France: HAS; 2013. Accessed on August 18th, 2020. Available at: https://www.has-sante.fr/upload/docs/application/pdf/2013-07/annexes_vf_2013-07-18_14-49-4_836.pdf.
- Pravno-informacijski sistem. Rules on inclusion of medicines on the list. Official Gazette of the Republic of Slovenia, No 35/2013 [Internet]. Business Directory of Republic of Slovenia; 2013 [cited 2017 Mar 25]. Available from: http://www.pisrs.si/Pis.web/pregledPredpisa?id=PRAV11493
- Carbonne M Réflexion sur l’utilisation du robot chirurgical: construction d’une matrice valeur/risque [Internet] [Mémoire de DES]. Nancy, France: Université de Lorraine; 2014. Available from: http://docnum.univ-lorraine.fr/public/BUPHA_T_2014_CARBONNE_MARION.pdf
- Assistance Publique des Hôpitaux de Paris (AP-HP). Historique des recommandations du CEDIT [Internet]. Cedit. 2013 [cited 2020 May 23]. Available from: http://cedit.aphp.fr/hospital-based-hta-levaluation-de-technologies-de-sante-a-lhopital/historique-des-recommendations-du-cedit/
- Dutot C Place du référencement hospiatlier dans la stratégie d’accès au marché des dispositifs médicaux innovants. [Internet] [Thèse d’exercice]. Angers, France: Université Angers; 2014. Available from: http://dune.univ-angers.fr/fichiers/20071040/2014PPHA3354/fichier/3354F.pdf
- Kriza C, Hanass-Hancock J, Odame EA, et al. A systematic review of health technology assessment tools in sub-Saharan Africa: methodological issues and implications. Health Res Policy Syst. 2014 Dec;2(12):66.
- Martelli N, van den Brink H, Denies F, et al. Evaluation des technologies de santé en milieu hospitalier : quelle organisation pour évaluer et acquérir des dispoditifs médicaux innovants? Ann Pharm Fr. 2014;72(1):3–14.
- Ølholm AM, Kidholm K, Birk-Olsen M, et al. Hospital manager’s need for information on health technology investments. Int J Technol Assess Health Care. 2015;31(6):414–425.
- Ritrovato M, Faggiano FC, Tedesco G, et al. Decision-Oriented Health Technology Assessment: one Step Forward in Supporting the Decision-Making Process in Hospitals. Value Health J Int Soc PharmacoEcon Outcomes Res. 2015 Jun;18(4):505–511.
- Roussel C, Carbonneil C, Audry A, et al. Impact organisationnel: définition et méthodes d’évaluation pour les dispositifs médicaux. Thérapie. 2016;71(1):69–82.
- Baltussen R, Jansen MPM, Bijlmakers L, et al. Value Assessment Frameworks for HTA Agencies: the Organization of Evidence-Informed Deliberative Processes. Value Health J Int Soc PharmacoEcon Outcomes Res. 2017;20(2):256–260.
- Dumont R, Toulet D, Michel C, et al. Implantation d armoires à pharmacie sécurisées au sein des services cliniques : impacts organisationnel et économique [Internet]. 65e Congrès francophonede pharmacie hospitalière; 2017 [cited 2020 May 24]; Nancy, France. Available from: https://docplayer.fr/105926786-Nancy-programme-hopipharm-mai-congres-francophone-de-pharmacie-hospitaliere-organise-par-le-synprefh.html
- Angelis A, Kanavos P. Multiple Criteria Decision Analysis (MCDA) for evaluating new medicines in Health Technology Assessment and beyond: the Advance Value Framework. Soc Sci Med. 2017;188:137–156. 1982.
- Detiček A, Janzic A, Locatelli I, et al. Decision-making criteria for medicine reimbursement in Slovenia: an expert panel discussion. BMC Health Serv Res. 2018 27;18(1):496.
- Improta G, Russo MA, Triassi M, et al. Use of the AHP methodology in system dynamics: modelling and simulation for health technology assessments to determine the correct prosthesis choice for hernia diseases. Math Biosci. 2018;299:19–27.
- Lawrenson JG, Graham-Rowe E, Lorencatto F, et al. What works to increase attendance for diabetic retinopathy screening? An evidence synthesis and economic analysis. Health Technol Assess Winch Engl. 2018;22(29):1–160.
- Pitini E, De Vito C, Marzuillo C, et al. How is genetic testing evaluated? A systematic review of the literature. Eur J Hum Genet EJHG. 2018;26(5):605–615.
- Vettoretto N, Foglia E, Ferrario L, et al. Why laparoscopists may opt for three-dimensional view: a summary of the full HTA report on 3D versus 2D laparoscopy by S.I.C.E. (Società Italiana di Chirurgia Endoscopica e Nuove Tecnologie). Surg Endosc. 2018;32(6):2986–2993.
- Uphoff ME, Kane D. Hospital-Based Technology Assessment: essential Questions and an Operational Model. Public Product Manag Rev. 1998;22(1):60–70.
- Martin DK, Pater JL, Singer PA. Priority-setting decisions for new cancer drugs: a qualitative case study. Lancet Lond Engl. 2001;358(9294):1676–1681.
- Gibson JL, Martin DK, Singer PA. Setting priorities in health care organizations: criteria, processes, and parameters of success. BMC Health Serv Res. 2004;4(1):25.
- Alberta Health Services. Alberta Health Services [Internet]. [cited 2020 May 17]. Available from: http://albertahealthservices.ca/
- Gibson J, Mitton C, Martin D, et al. Ethics and economics: does programme budgeting and marginal analysis contribute to fair priority setting? J Health Serv Res Policy. 2006;11(1):32–37.
- Mitton C, Mackenzie J, Cranston L, et al. Priority setting in the Provincial Health Services Authority: case study for the 2005/06 planning cycle. Health Policy Polit Sante. 2006;2(1):91–106.
- CIUSSS du Centre-Sud-de-l’Île-de-Montréal [Internet]. [cited 2020 May 17]. Available from: https://ciusss-centresudmtl.gouv.qc.ca/
- Kirby J, Somers E, Simpson C, et al. The public funding of expensive cancer therapies: synthesizing the ‘3Es’–evidence, economics, and ethics. Organ Ethics Healthc Bus Policy OE. 2008 Fall-Winter;4(2):97–108.
- Lehoux P, Williams-Jones B. Mapping the integration of social and ethical issues in health technology assessment. Int J Technol Assess Health Care. 2007;23(1):9–16.
- Noorani HZ, Husereau DR, Boudreau R, et al. Priority setting for health technology assessments: a systematic review of current practical approaches. Int J Technol Assess Health Care. 2007;23(3):310–315.
- Agency for Healthcare Research and Quality (AHRQ) [Internet]. [cited 2020 May 19]. Available from: https://www.ahrq.gov/
- Browman GP, Manns B, Hagen N, et al. 6-STEPPPs: A Modular Tool to Facilitate Clinician Participation in Fair Decisions for Funding New Cancer Drugs. J Oncol Pract. 2008;4(1):2–7.
- Agence d’Evaluation des Technologies et des Modes d’Intervention en Santé. Comparative analysis of Bedpan processing equipment. [Internet]. 2009. Available from: https://www.inesss.qc.ca/fileadmin/doc/AETMIS/Rapports/Sterilisation/2009-04_en.pdf
- Johnson AP, Sikich NJ, Evans G, et al. Health technology assessment: a comprehensive framework for evidence-based recommendations in Ontario. Int J Technol Assess Health Care. 2009;25(2):141–150.
- Ontario Health Technology Advisory Committee - Health Quality Ontario (HQO) [Internet]. [cited 2020 May 19]. Available from: https://www.hqontario.ca/Evidence-to-Improve-Care/Health-Technology-Assessment/Ontario-Health-Technology-Advisory-Committee
- Parker LE, Ritchie MJ, Kirchner JE, et al. Balancing health care evidence and art to meet clinical needs: policymakers’ perspectives. J Eval Clin Pract. 2009;15(6):970–975.
- Meagher T MUHC clinical activity priority setting A4R and beyond ISPHC priorities 2010 [Internet]. 2010 [cited 2020 May 23]. Available from: https://slideplayer.com/slide/6900656/
- Menon D, Stafinski T, McCabe C To fund or not to fund: A generalized decision-making model for health care resource allocation. 8th Biennial Conference of the International Society on Priorities in Health Care. Boston, Massachusetts, USA: ISHP; 2010.
- EVIDEM. EVIDEM v2.1 Decision criteria - Conceptual background, definitions, design & instructions [Internet]. 2011 [cited 2017 Aug 9]. Available from: https://www.evidem.org/docs/2012/EVIDEM-v2-1-Decision-criteria-conceptual-background-definitions-and-instructions.pdf
- Gagnon M, Desmartis M, Gagnon J, et al. Introducing the patient’s perspective in hospital health technology assessment (HTA): the views of HTA producers, hospital managers and patients. Health Expect. 2014;17(6):888–900.
- INESSS. Pertinence d’élargir le programme de dépistage néonatal sanguin au Québec [Internet]. 2013. Report No.: 7. Available from: https://inesss.qc.ca/fileadmin/doc/INESSS/Rapports/Genetique/INESSS_Depistage_neonatal_sanguin.pdf
- Diaby V, Goeree R, Hoch J, et al. Multi-criteria decision analysis for health technology assessment in Canada: insights from an expert panel discussion. Expert Rev Pharmacoecon Outcomes Res. 2015 Feb;15(1):13–19.
- Evidence in Context - Contextualization in CHRSP project [Internet]. Health research - synthesized and contextualized for use in Newfoundland and Labrador; 2015 cited 2020 May 23. Available from: https://www.nlcahr.mun.ca/CHRSP/CHRSP_CONTEXT_IN_DETAIL.pdf
- CADTH, Pan-Canadian Oncology Drug Review (pCODR). pCODR Expert Review Committee Deliberative Framework [Internet]. 2016 [cited 2020 May 23]. Available from: https://www.cadth.ca/sites/default/files/pcodr/The%20pCODR%20Expert%20Review%20Committee%20%28pERC%29/pcodr_perc_deliberative_frame.pdf
- Martin J, Polisena J, Dendukuri N, et al. Local health technology assessment in Canada : current state and next steps. Int J Technol Assess Health Care. 2016;32(3):175–180.
- Goetghebeur MM, Wagner M, Samaha D, et al. Exploring values of Health Technologies Assessment agencies using reflective multicriteria and rare disease case. Int J Technol Assess Health Care. 2017 Jan;33(4):504–520.
- Krahn M, Miller F, Bayoumi A, et al. Development of the Ontario decision framework: a values based framework for Health Technology assessment. Int J Technol Assess Health Care. 2018;34(3):290–299.
- Poder TG, Bellemare CA, Bédard SK, et al. Impact of Health Technology Assessment reports on hospital decision markers - 10 year insight from a hospital unit in Sherbrook, Canada: impact of Health Technology Assessment on hospital decisions. Int J Technol Assess Health Care. 2018;34(4):393–399.
- Tanios N, Wagner M, Tony M, et al. Which criteria are considered in healthcare decisions? Insights from an international survey of policy and clinical decision makers. Int J Technol Assess Health Care. 2013;29(4):456–465.
- Wranik WD, Skedgel C, Hu M. Drug attributes associated with the selection of drugs for reimbursement: a pilot stated preferences experiment with Canadian stakeholders. Expert Rev Pharmacoecon Outcomes Res. 2019 Feb;19(1):59–69.
- New South Wales Health. NSW Health [Internet]. [cited 2020 May 17]. Available from: https://www.health.nsw.gov.au/Pages/default.aspx
- Bowen S, Zwi AB. Pathways to ‘evidence-informed’ policy and practice: a framework for action. PLoS Med. 2005;2(7):e166.
- Monash Health. Monash Health [Internet]. [cited 2020 May 17]. Available from: https://monashhealth.org/
- Golan O, Hansen P. A new decision-support framework for prioritization of new health technologies: the ‘Value for Money’ Chart. 2010. Available from: https://www.1000minds.com/sectors/academic/research
- Irving MJ, Tong A, Rychetnik L, et al. Nephrologists’ perspectives on the effect of guidelines on clinical practice: a semistructured interview study. Am J Kidney Dis Off J Natl Kidney Found. 2010 Feb;55(2):241–249.
- Greenberg D, Pliskin JS, Peterburg Y. Decision making in acquiring medical technologies in Israeli medical centers: a preliminary study. Int J Technol Assess Health Care. 2003;19(1):194–201.
- Greenberg D, Peterburg Y, Vekstein D, et al. Decisions to adopt new technologies at the hospital level: insights from Israeli medical centers. Int J Technol Assess Health Care. 2005;21(2):219–227.
- Mathew JL. KNOW ESSENTIALS: a tool for informed decisions in the absence of formal HTA systems. Int J Technol Assess Health Care. 2011 Apr;27(2):139–150.
- Mobinizadeh M, Raeissi P, Nasiripour AA, et al. The health systems’ priority setting criteria for selecting health technologies: A systematic review of the current evidence. Med J Islam Repub Iran. 2016;30:329.
- Nobre FF, Trotta LT, Gomes LF. Multi-criteria decision making–an approach to setting priorities in health care. Stat Med. 1999;18(23):3345–3354.
- Rubinstein A, Belizán M, Discacciati V. Are economic evaluations and health technology assessments increasingly demanded in times of rationing health services? The case of the Argentine financial crisis. Int J Technol Assess Health Care. 2007;23(2):169–176.
- Vargas V, Poblete S. Health prioritization: the case of Chile. Health Affairs Project Hope. 2008;27(3):782–792.
- Reichenbach L. The politics of priority setting for reproductive health: breast and cervical cancer in Ghana. Reprod Health Matters. 2002;10(20):47–58.
- Holdsworth M, El Ati J, Bour A, et al. Developing national obesity policy in middle-income countries: a case study from North Africa. Health Policy Plan. 2013 Dec;28(8):858–870.
- Ghaffar A. Setting research priorities by applying the combined approach matrix. Indian J Med Res. 2009;129(4):368–375.