Artificial intelligence in medical device software and high-risk medical devices – a review of definitions, expert recommendations and regulatory initiatives

Figures & data

Table 1. Definitions.

Term	Definition recommended and/or used in this paper
Artificial Intelligence (AI)	Preferred simple definition = The science of making machines do things that would require intelligence if done by men. [Minsky] [6] EU definition recommended by the High Level Expert Group = See Table 2 for full quotation
Machine learning (ML)	A sub-discipline of AI, where systems can learn from data to solve problems without being explicitly programmed to do so.
Deep learning	A form of ML that uses ‘deep’ neural networks (mathematical models inspired by the architecture of the human brain).
AI model	A physical, mathematical or otherwise logical representation of a system, entity, phenomenon, process or data. [7]
Model parameter	Internal variable of a model. [7]
AI system	An engineered system that generates outputs such as content, forecasts, recommendations or decisions for a given set of human-defined objectives. [7]An AI system is composed of (1) the training, validation, and evaluation data; (2) the model structure and parameter values; and (3) the algorithms used to pre- and post-process data and to train and deploy the AI system.
Training	Process to optimize the parameters of an ML model, based on an ML algorithm, by using training data. [7]
Data	Consists of training data (data used to train the model), validation data (data used to compare the performance [and fine tune] different candidate models), and evaluation data (data used to assess the performance of a final model). [7]
Interpretable AI	An AI system with the ability to explain or to present [its output] in understandable terms to a human. [8] The availability of information about each component of an AI system.
Explainable AI	The availability of post-hoc aids or models to explain the decisions of an AI system. [9]
Transparency	[The ability of an AI system] to be open to appropriate public examination and oversight. [10]
Trustworthy AI	An AI system that is lawful (complying to all applicable laws and regulations), ethical (adherent to ethical principles and values), and robust (from both technical and sociable perspectives). [11]

Table 2. Definitions of artificial intelligence in reports and regulatory documents with relevance for medical devices.

Organization/institution	Document details (title, year)	Verbatim extract from definition
International Medical Device Regulators Forum (IMDRF)
Working Group on Software as a Medical Device (SaMD)	Software as a medical device (SaMD): Key definitions. IMDRF/SaMD WG/N10: 2013 [32]	“Software as a Medical Device” (SaMD) is defined as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device.
SaMD Working Group	Software as a medical device: clinical evaluation. IMDRF/SaMD WG/N41: 2017 [33]	SaMD can best be described as software that utilizes an algorithm (logic, set of rules, or model) that operates on data input (digitized content) to produce an output that is intended for medical purposes as defined by the SaMD manufacturer.
Working Group on Artificial Intelligence Medical Devices	Machine Learning-enabled Medical Devices – A subset of Artificial Intelligence-enabled Medical Devices: Key Terms and Definitions. IMDRF/AIMD WG (PD1)/N67. Proposed document, 16 September 2021. [34]	Machine Learning-enabled Medical Device (MLMD) = A medical device that uses machine learning, in part or in whole, to achieve its intended medical purpose.
European Union
High-Level Expert Group on Artificial Intelligence	A definition of AI: main capabilities and scientific disciplines. (2018) [35]	AI refers to systems designed by humans that, given a complex goal, act in the physical or digital world by perceiving their environment, interpreting the collected structured or unstructured data, reasoning on the knowledge derived from this data and deciding the best action(s) to take (according to pre-defined parameters) to achieve the given goal. AI systems can also be designed to learn to adapt their behaviour by analysing how the environment is affected by their previous actions.
High-Level Expert Group on Artificial Intelligence	Ethics Guidelines for Trustworthy AI (2019) [11]	AI systems are software (and possibly also hardware) systems designed by humans that, given a complex goal, act in the physical or digital dimension by perceiving their environment through data acquisition, interpreting the collected structured or unstructured data, reasoning on the knowledge, or processing the information, derived from this data and deciding the best action(s) to take to achieve the given goal. AI systems can either use symbolic rules or learn a numeric model, and they can also adapt their behavior by analysing how the environment is affected by their previous actions.
Medical Device Coordination Group	MDCG 2019–11. Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR (2019). [5]	“Software” is defined as a set of instructions that processes input data and creates output data. Medical device software (MDSW) is software that is intended to be used, alone or in combination, for a purpose as specified in the definition of a “medical device” in the Medical Devices Regulation or In Vitro Diagnostic Medical Devices Regulation.
European Commission	Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence. Article 3.1, and *Annex I (2021) [37] (also known as the AI Act)	AI system means software that is developed with one or more of the techniques and approaches listed* and [that] can, for a given set of human-defined objectives, generate outputs such as content, predictions, recommendations, or decisions influencing the environments they interact with. * (a) Machine learning approaches, including supervised, unsupervised and reinforcement learning, using a wide variety of methods including deep learning; (b) Logic- and knowledge-based approaches, including knowledge representation, inductive (logic) programming, knowledge bases, inference and deductive engines, (symbolic) reasoning and expert systems; (c) Statistical approaches, Bayesian estimation, search and optimization methods.
European Parliament. Directorate General for Parliamentary Research Services: Scientific Foresight Unit (STOA)	Artificial Intelligence in healthcare. Applications, risks, and ethical and societal impacts. (2022) [38]	Here we will first use the historical definition of AI, i.e. when a machine is able to mimic human intelligence or even surpass it to perform a given task such as prediction or reasoning [..] we will mostly focus on one subfield of AI that is dominant in the healthcare area, namely machine learning. ML is a subfield of AI and concerns the methods that learn to perform given tasks, such as prediction or classification, based on existing data.
Selected other international, European and national institutions
International Standardization Organization / International Electrotechnical Commission (ISO/IEC) Joint Technical Committee 1/SC 42	Information technology – Vocabulary. (2015) ISO/IEC 2382:2015 [39] (revision under review)	Branch of computer science devoted to developing data processing systems that perform functions normally associated with human intelligence, such as reasoning, learning, and self-improvement.
ISO/IEC JTC 1 (on information technology)/Sub-committee 42 (on AI) [as above]	Information technology – Artificial intelligence – Artificial intelligence concepts and terminology. (2022) 2022–07 ISO/IEC 22989:2022 [40]	AI is a technical and scientific field devoted to the engineered system that generates outputs such as content, forecasts, recommendations or decisions for a given set of human-defined objectives.
IEEE Engineering in Medicine and Biology Society (EMBS) Artificial Intelligence Medical Devices Working Group [41] (EMB/Stds Com/AIMDWG)	IEEE Recommended practice for the quality management of datasets for medical artificial intelligence. (2022) IEEE 2801–2022 [42]	This identifies best practices for establishing a quality management system for datasets used for AI medical device, including data collection, transfer, utilization, storage, maintenance and update; and it lists critical factors that impact the quality of datasets.
Organisation for Economic Co-operation and Development OECD	OECD Science, Technology and Industry Scoreboard 2017: The Digital Transformation. [43]	AI is a term used to describe machines performing human-like cognitive functions (e.g. learning, understanding, reasoning or interacting).
OECD Council on Artificial Intelligence	OECD, Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449. (2019) [44]	An AI system is a machine-based system that can, for a given set of human-defined objectives, make predictions, recommendations, or decisions influencing real or virtual environments. AI systems are designed to operate with varying levels of autonomy.
OECD Expert Group on Artificial Intelligence	Scoping the OECD AI principles: Deliberations of the Expert Group on Artificial Intelligence at the OECD (AIGO). (2019) OECD Digital Economy Papers No. 291. [45]	An AI system is a machine-based system that is capable of influencing the environment by producing an output (predictions, recommendations or decisions) for a given set of objectives. It uses machine and/or human-based data and inputs to (i) perceive real and/or virtual environments; (ii) abstract these perceptions into models through analysis in an automated manner (e.g. with machine learning), or manually; and (iii) use model inference to formulate options for outcomes.
OECD Committee on Digital Economy Policy	OECD Framework for the Classification of AI systems. (2022) OECD Digital Economy Papers, No. 323. [46]	AI models are actionable representations of all or part of the external context or environment of an AI system (encompassing, for example, processes, objects, ideas, people and/or interactions taking place in context). AI models use data and/or expert knowledge provided by humans and/or automated tools to represent, describe and interact with real or virtual environments.
World Health Organization	Ethics and governance of artificial intelligence for health. WHO guidance (2021) [47]	AI refers to the ability of algorithms encoded in technology to learn from data so that they can perform automated tasks without every step in the process having to be programmed explicitly by a human.
Food and Drug Administration, USA (FDA)	Proposed regulatory framework for modifications to artificial intelligence /machine learning (AI/ML)-based software as a medical device (SaMD). (2021) [48]	The IMDRF defines SaMD as software intended to be used for one or more medical purposes that perform these purposes without being part of a hardware medical device. FDA [..] considers medical purpose as those purposes that are intended to treat, diagnose, cure, mitigate, or prevent disease or other conditions.

Figure 1. Comparison of regulatory definitions of medical device software and their scope.

Sources: IMDRF: Software as a Medical Device, SaMD WG/N10, 2013 [32]. Also reviewed in [36]. EU: Medical Device Coordination Group. MDCG 2019–11: Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR [5]. FDA: Software as a Medical Device, and Software in a Medical Device: Content of Premarket Submissions for Device Software Functions – Draft Guidance for Industry and Food and Drug Administration Staff – 4 November 2021 (lines 180–7) [49].

Table 3. Factors influencing the risks of software as a medical device (SaMD).

Factors influencing the risks of software as a medical device (SaMD)
The type of disease or condition
Fragility of the patient with respect to the disease or condition
Progression of the disease or the stage of the disease/condition
Usability of the application
Designed towards a specific user type
Level of dependence or reliance by the user upon the output information
Ability of the user to detect an erroneous output information
Transparency of the inputs, outputs and methods to the user
Level of clinical evidence available and the confidence on the evidence
The type of output information and the level of influence on the clinical intervention
Complexity of the clinical model used to derive the output information
Known specificity of the output information
Maturity of clinical basis of the software and confidence in the output
Benefit of the output information vs. baseline
Technological characteristics of the platform the software are intended to operate on
Method of distribution of the software

From: IMDRF Software as a Medical Device (SaMD) Working Group. ‘Software as a Medical Device’: Possible Framework for Risk Categorization and Corresponding Considerations. IMDRF/SaMD WG/N12FINAL:2014 [4].

Table 4. Cross-specialty recommendations on medical AI, developed using a systematic expert consensus methodology.

ACRONYM	YEAR	PUBLICATION	TOPIC
Pre-clinical evaluation
TRIPOD–AI	upcoming	BMJ Open (protocol) [60]	Reporting guideline for the development and validation of predictive models
PROBAST–AI	upcoming	BMJ Open (protocol) [60]	Risk of bias assessment tool for prediction models
STARD–AI	upcoming	BMJ Open (protocol) [61]	Reporting guideline for AI-centred diagnostic accuracy studies
Clinical evaluation
DECIDE–AI	2022	● BMJ [59] [& joint publication in Nature Medicine]	Reporting guideline for the early stage, live clinical evaluation of decision support systems driven by artificial intelligence
SPIRIT–AI	2020	● BMJ [58] [& joint publication in Nature Medicine, and Lancet Digital Health]	Reporting guideline for clinical trial protocols of interventions involving artificial intelligence
CONSORT–AI	2020	● BMJ [57] [& joint publication in Nature Medicine, and Lancet Digital Health]	Reporting guidelines for clinical trial reports of interventions involving artificial intelligence

A repository of professional standards is available at: https://www.equator-network.org/

Table 5. Cross-specialty or specialty-specific recommendations on medical AI, published by medical associations or groups of investigators.

ACRONYM/FIRST AUTHOR	YEAR	PUBLICATION	SPECIALTY	TOPIC
Methodological guidelines
Park	2020	JAMIA Open [65]	Cross-specialty	Recommendations for a phased approach to medical AI evaluation
Sendak	2020	EMJ Innovation [66] European Medical Journal	Cross-specialty	Recommendations for translation of ML products into healthcare delivery
McCradden	2020	Nature Medicine [67]	Cross-specialty	Recommendation for an ethical integration of healthcare AI
Higgins	2020	Advanced Intelligent Systems [68]	Cross-specialty	Framework for AI product development in healthcare
PRIME Sengupta	2020	JACC Cardiovascular Imaging [69]	Cardiology	Requirements for cardiovascular imaging-related ML evaluation ^§
Schwendicke*	2021	Journal of Dentistry [63]	Dentistry	Planning, conducting and reporting of AI studies in dental research ^§
FUTURE–AI Lekadir	2021	Arxiv (preprint) [70]	Radiology	Recommendations for trustworthy AI in medical imaging
Reporting guidelines
MI-CLAIM Norgeot	2020	Nature Medicine [71]	Cross-specialty	Minimum information about clinical AI modeling
MINIMAR Hernandez-Boussard	2020	JAMIA [72]	Cross-specialty	MINimum Information for Medical AI Reporting
Sendak	2020	NPJ Digital Medicine [73]	Cross-specialty	ML model fact labels
CAIR Olczak	2021	Acta Orthopaedica [74]	Cross-specialty	Checklist for clinical AI research reporting
CLAIM Mongan	2020	Radiology: Artificial Intelligence [75]	Radiology	Artificial intelligence in medical imaging
Computer Vision in Surgery International Collaborative* Kitaguchi	N/A	Annals of Surgery [62]	Surgery	Proposal for assessment of surgical procedure and assistance using AI-based computer vision
Quality control
Vollmer	2020	BMJ [76]	Cross-specialty	20 critical questions on transparency, replicability, ethics, and effectiveness
Scott	2020	BMJ Health and Care Informatics [77]	Cross-specialty	Checklist for suitability of machine learning applications in healthcare
OSCAR-AI Petzold	2021	Annals of Clinical and Translational Neurology [78]	Ophthalmology & neurology	Quality control criteria for AI-enhanced OCT applications
CLEAR Derm* Daneshjou	2021	JAMA Dermatology [64]	Dermatology	Checklist for evaluation of image-based AI algorithm reports in dermatology

§ a joint methodological and reporting guideline;

* developed through a systematic consensus methodology.

Table 6. Key components of consensus-based guidelines.

Key AI components/criteria recommended for inclusion	TRIPOD-AI	PROBAST-AI	STARD-AI	DECIDE-AI [59]	SPIRIT-AI [58]	CONSORT-AI [57]	CLEAR Derm [64]	Schwendicke et al. [63]	Computer Vision in Surgery Int. Collab.
Intended use	Upcoming	Upcoming	Upcoming	X	X	X	X		Upcoming
Preexisting evidence				X	X
Characteristics of training set, including sample size and class distribution				X			X	X
Data acquisition process				X	X	X	X	X
Missing or poor quality data				X	X	X		X
Data preprocessing				X	X	X	X	X
Computational resources and technical integration						X		X
Model description, including versioning				X	X	X	X	X
Training process, including dataset split				X			X	X
Validation process				X			X	X
Characteristics of study population and external dataset, including sample sizes and inclusion/exclusion criteria				X	X	X	X	X
Characteristics of the user population				X
Ground truth and labeling							X	X
Characteristics of study settings				X	X	X	X
Performance assessment				X	X	X	X	X
Uncertainty				X				X
Out-of-distribution results							X
AI system output				X	X	X		X
Explainability								X
Benchmarking and comparison				X	X	X	X	X
Clinical outcomes				X	X	X	X
Integration in clinical pathway				X	X	X
AI system iterative improvement				X
Human factors evaluation				X	X	X
Safety and errors				X	X	X
Patient involvement				X
Ethics and biases				X	X		X	X
Data protection					X		X	X
Code and data availability				X	X	X	X	X
Funding				X	X	X		X

Figure 2. A simplified landscape of the inter-relationships of European and global bodies engaged in the development of standards for AI medical devices.

Explanation of acronyms: ASTM International, American Society for Testing and Materials; CEN, Comité Européen de Normalization; CENELEC, Comité Européen de Normalization Electrotechnique; CISPR, Comité International Spécial des Perturbations Radioélectriques; DICOM, Digital Imaging and Communications in Medicine; ETSI, European Telecommunications Standards Institute; HealthIT.gov, website of the Office of the National Coordinator for Health Information Technology (ONC), U.S. Department of Health and Human Services; HL7, Health Level Seven International; IEC, International Electrotechnical Commission; IECEE, IEC System of Conformity Assessment Schemes for Electrotechnical Equipment and Components; IHE International, Integrating the Healthcare Enterprise; ISO, International Standardization Organization; ITU-R, ITU Radiocommunication Sector; IEEE, Institute of Electrical and Electronics Engineers; ITU-T, International Telecommunication UnionTelecommunication Standardization Sector; NCCLS, National Committee for Clinical Laboratory Standards (now known as Clinical and Laboratory Standards Institute (CLSI); Details of the European National Standardization Bodies listed in the bottom left box are given at the website of CEN/CENELEC: https://standards.cencenelec.eu/dyn/www/f?p=CEN:5

Table 7. Information technology standards relating to general aspects of AI, that have already been published.

Publication Date	Reference	Title
2022–07	ISO/IEC 22989:2022	Information technology – Artificial intelligence – Artificial intelligence concepts and terminology
2022–06	ISO/IEC 23053:2022	Framework for Artificial Intelligence (AI) Systems Using Machine Learning (ML)
2022–05	IEEE 2801–2022	IEEE Recommended Practice for the Quality Management of Datasets for Medical Artificial Intelligence
2021–12	ISO/IEC TR 24372:2021	Information technology – Artificial intelligence (AI) – Overview of computational approaches for AI systems
2021–11	ISO/IEC TR 24027:2021	Information technology – Artificial intelligence (AI) – Bias in AI systems and AI aided decision making
2021–05	ISO/IEC TR 24030:2021	Information technology – Artificial intelligence (AI) – Use cases
2021–03	ISO/IEC TR 24029–1:2021	Artificial Intelligence (AI) – Assessment of the robustness of neural networks – Part 1: Overview
2020–05	ISO/IEC TR 24028:2020	Information technology – Artificial intelligence – Overview of trustworthiness in artificial intelligence

Details obtained from https://www.iso.org/ics/35.020/x/

(accessed week of 18 July 2022)

Table 8. General standards under joint development by the International Standardization Organization (ISO) and the International Electrotechnical Commission (IEC), related to artificial intelligence.

Reference	Title
ISO/IEC PRF TS 4213	Information technology – Artificial Intelligence – Assessment of machine learning classification performance
ISO/IEC AWI 5259–1	Artificial intelligence – Data quality for analytics and machine learning (ML) – Part 1: Overview, terminology, and examples
ISO/IEC AWI 5259–2	Artificial intelligence – Data quality for analytics and machine learning (ML) – Part 2: Data quality measures
ISO/IEC AWI 5259–3	Artificial intelligence – Data quality for analytics and machine learning (ML) – Part 3: Data quality management requirements and guidelines
ISO/IEC AWI 5259–4	Artificial intelligence – Data quality for analytics and machine learning (ML) – Part 4: Data quality process framework
ISO/IEC AWI 5259–5	Artificial intelligence – Data quality for analytics and machine learning (ML) – Part 5: Data quality governance
ISO/IEC CD 5338	Information technology – Artificial intelligence – AI system life cycle processes
ISO/IEC CD 5339	Information Technology – Artificial Intelligence – Guidelines for AI applications
ISO/IEC CD 5392	Information technology – Artificial intelligence – Reference architecture of knowledge engineering
ISO/IEC DTR 5469	Artificial intelligence – Functional safety and AI systems
ISO/IEC AWI TS 5471	Artificial intelligence – Quality evaluation guidelines for AI systems
ISO/IEC DIS 8183	Information technology – Artificial intelligence – Data life cycle framework
ISO/IEC AWI TS 8200	Information technology – Artificial intelligence – Controllability of automated artificial intelligence systems
ISO/IEC AWI TS 6254	Information technology – Artificial intelligence – Objectives and approaches for explainability of ML models and AI systems
ISO/IEC AWI TS 12791	Information technology – Artificial intelligence – Treatment of unwanted bias in classification and regression machine learning tasks
ISO/IEC AWI 12792	Information technology – Artificial intelligence – Transparency taxonomy of AI systems
ISO/IEC AWI TR 17903	Information technology – Artificial intelligence – Overview of machine learning computing devices
ISO/IEC DIS 23894	Information technology – Artificial intelligence – Guidance on risk management
ISO/IEC DIS 24029–2	Artificial intelligence (AI) – Assessment of the robustness of neural networks – Part 2: Methodology for the use of formal methods
ISO/IEC AWI TR 24030	Information technology – Artificial intelligence (AI) – Use cases
ISO/IEC TR 24368	Information technology – Artificial intelligence – Overview of ethical and societal concerns
ISO/IEC FDIS 24668	Information technology – Artificial intelligence – Process management framework for big data analytics
ISO/IEC DIS 25059	Software engineering – Systems and software Quality Requirements and Evaluation (SQuaRE) – Quality model for AI systems
ISO/IEC AWI TS 29119–11	Information technology – Artificial intelligence – Testing for AI systems – Part 11:
ISO/IEC CD 42001.2	Information Technology – Artificial intelligence – Management system
And in development by other TCs:
ISO/IEC CD TR 27563	Security and privacy in artificial intelligence use cases

Details obtained from https://www.iso.org/committee/6794475/x/catalogue/p/0/u/1/w/0/d/0

(accessed week of 18 July 2022)

Table 9. Selected regulatory initiatives and proposals under development.

Organization/committee	Task/project	Details/objectives
International Medical Device Regulators Forum (IMDRF)
IMDRF SaMD Working Group	“Software as a Medical Device”: Possible Framework for Risk Categorization and Corresponding Considerations. (2014) [4]	Two major factors together provide a description of the intended use of SaMD, and can categorize it: “A. Significance of the information provided by the SaMD to the healthcare decision, and B. State of the healthcare situation or condition.”
IMDRF SaMD Working Group	Software as a medical device: clinical evaluation. (2017) [33]	Pre-market: the manufacturer generates evidence of the product’s accuracy, specificity, sensitivity, reliability, limitations, and scope.
IMDRF AI Working Group	IMDRF/AIMD WG (PD1)/N67:2021. Machine Learning-enabled Medical Devices – A subset of Artificial Intelligence-enabled Medical Devices: Key Terms and Definitions. (2021) [34]	Proposed definitions to complement earlier IMDRF documents on SaMD. A Machine Learning-enabled Medical Device (MLMD) is “a medical device that uses machine learning, in part or in whole, to achieve its intended medical purpose.”
European Union
European Commission, Medical Device Coordination Group	MDCG 2019–11. Guidance on qualification and classification of software in Regulation (EU) 2017/745 MDR & Regulation (EU) 2017/746 IVDR.	Rule 11 MDR describes the classification of medical device software. The details are quoted in the text of this paper, section 3.1 (see [5]).
AI Watch [101] Joint Research Centre, European Commission, Seville	60 papers published as of September 2022.	Review of initiatives for standardization of AI in the EU. (Nativi S et al. 2021 [97]; updated in 2022)
European Medicines Agency (EMA)	Regulatory Science, Research Needs (2021) [102]	Objective H2.7.7 = Identify and define regulatory requirements to validate AI algorithms used in the context of regulated medicines.
EMA with HMA (Heads of Medicines Agencies)	Accelerating Clinical Trials in the EU (ACT EU) (2022) [103]	Priority action 8 = Develop and publish key methodologies guidance e.g. on AI/ML impacted clinical trials
European Parliament. Directorate General for Parliamentary Research Services. Scientific Foresight Unit (STOA)	A governance framework for algorithmic accountability and transparency. (2019) [104]	Policy options for regulatory oversight, legal liability, and global coordination. For transparency, understanding how a system behaves is more important than knowing how it works.
European Parliament. Directorate General for Parliamentary Research Services: Scientific Foresight Unit (STOA)	Artificial Intelligence in healthcare. Applications, risks, and ethical and societal impacts. (2022) [38]	Identifies 7 main risks of AI in healthcare including bias and error. Recommends extended regulatory frameworks and codes of practice for medical AI, to assess algorithmic fairness, clinical safety, clinical acceptance, transparency and traceability.
Other European organizations
OECD. AI Policy Observatory [105]	The OECD AI principles were adopted in 2019.	The Observatory provides a center for collecting and sharing evidence on AI.
Council of Europe Ad Hoc Committee on AI (CAHAI) [106]	Succeeded in 2022 by the Committee on Artificial Intelligence (CAI) [107]. Terms of reference not public.	The agenda of CAI includes consideration of legal instruments for risk and impact assessment.
Council of Europe [108] Steering Committee for Human Rights in the fields of Biomedicine and Health (CDBIO)	The impact of artificial intelligence on the doctor-patient relationship. Report prepared by Brent Mittelstadt, Anonymized. (2021) [109] CDBIO now has a drafting group on AI in healthcare.	The potential human rights impact of AI on the doctor-patient relationship can be categorized according to six themes: Inequality in access to high quality healthcare Transparency to health professionals and patients Risk of social bias in AI systems Dilution of the patient’s account of well-being Risk of automation bias, de-skilling, and displaced liability Impact on the right to privacy
CEN-CENELEC Joint Technical Committee 21 ‘Artificial Intelligence’ [110] (Secretariat held by DS, the Danish Standardization Body)	CEN-CENELEC Work Programme 2022 (page 30) [111]	JTC 21 will produce standardization deliverables to address European market and societal needs and to underpin legislation and policies based on the EU’s principles and values: the objective is to contribute to the development of an ethical, human-centered and trustworthy AI from which all EU citizens can benefit.
Global organizations
World Health Organization	Ethics and governance of artificial intelligence for health (2021) [47]	6 principles for the use of AI for health: Protecting human autonomy. Promoting human well-being and safety and the public interest. Ensuring transparency, explainability and intelligibility. Fostering responsibility and accountability. Ensuring inclusiveness and equity. Promoting AI that is responsive and sustainable
WHO/ITU-T Focus Group on Artificial Intelligence for Health (FG-AI4H)	Established in 2018. Goals include facilitating global dialogue for AI for health, and developing national guidance documents to ensure the safe and appropriate use of AI in health [112].	Working Groups include Regulatory considerations on AI for health, and Clinical evaluation of AI for health’, to establish a standardized assessment framework for the evaluation of AI-based methods for health, diagnosis, triage or treatment decisions [113].
International Standardization Organization ISO ISO/IEC JTC 1/SC 42 Artificial Intelligence [114]	ISO created a standardization project on AI in 2017, that has been operational since 2018.	See Zielke et al for description of project [92]. 25 standards listed in development.
International Electrotechnical Commission (IEC)	62/411/NP: Testing of Artificial Intelligence/Machine Learning-enabled Medical Devices [115]	IEC 63450 in development, publication foreseen in 2024.
IEC	PWI 62–3 ED1: Publication due in 2022.	Artificial Intelligence/Machine Learning-enabled Medical Device – Performance Evaluation Process [116]
IEEE Engineering in Medicine and Biology Society (EMBS)
IEEE P2673 Intelligence Augmentation for Medical Imaging Working Group [117]	Standard for patient digital biomedical data files with 3D topological mapping of macroanatomy and microanatomy for use in big data and augmented intelligence systems (in development)	Framework for organization and use of new patient biomedical files containing medical imaging and imaging biomarker information for use in big data cloud-based augmented intelligence systems.
EMBS Artificial Intelligence Medical Devices Working Group [118] (EMB/Stds Com/AIMDWG)	IEEE 2801–2022 [119,120]. Recommended practice for the quality management of datasets for medical artificial intelligence. (2022)	Best practices for establishing quality management system for datasets used for AI medical devices, and critical factors that impact the quality of datasets.
Artificial Intelligence Medical Devices Working Group	Standard for the performance and safety evaluation of artificial intelligence based medical device: Terminology. New IEEE standard (IEEE P2802) [121,122].	Definitions of fundamental concepts and methodology that describe the safety, effectiveness, risks and quality management of AI medical devices. It also establishes a vocabulary for the development of future standards for AI medical devices.

Figure 3. Overview of European and global institutions and organizations engaged in the development of regulations for medical AI systems.

Details of all these acronyms are given in the text.The logo in the top left corner is the symbol for the CORE-MD consortium.

Samoili S, López Cobo M, Gómez E, et al. AI watch. Defining artificial iIntelligence. Towards an operational definition and taxonomy of artificial intelligence. EUR 30117 EN. Luxembourg: Publications Office of the European Union; 2020.

 Google Scholar

ISO/IEC 22989:2022 Information technology - Artificial intelligence - Artificial intelligence concepts and terminology. Available from: https://www.iso.org/standard/74296.html

 Google Scholar

Doshi-Velez F, Kim B Towards a rigorous science of interpretable machine learning. 2017. ArXiV pre-print. Available from: https://arxiv.org/abs/1702.08608

 Google Scholar

Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead |. Nature Mach Intell. 2019;1:206–215.

 PubMedGoogle Scholar

IEEE Global Initiative on ethics of autonomous and intelligent systems. Ethically aligned design: a vision for prioritizing human well-being with autonomous and intelligent systems. First. IEEE. 2019; Available from: https://ethicsinaction.ieee.org/wp-content/uploads/ead1e.pdf

 Google Scholar

High-Level Expert Group. European Commission, Directorate-General for Communications Networks, Content and Technology. Ethics guidelines trustworthy AI, 2019. Available from: https://data.europa.eu/doi/10.2759/346720.

 Google Scholar

International Medical Device Regulators Forum, SaMD Working Group. Software as a Medical Device (SaMD): Key Definitions. IMDRF/SaMD WG/N10FINAL:2013. Available from: https://www.imdrf.org/documents/software-medical-device-samd-key-definitions

 Google Scholar

International Medical Device Regulators Forum, SaMD Working Group. Software as a medical device: clinical evaluation. IMDRF/SaMD WG/N41: 2017. Available from: https://www.imdrf.org/documents/software-medical-device-samd-clinical-evaluation.

 Google Scholar

International Medical Device Regulators Forum, Working Group on Artificial Intelligence Medical Devices. machine learning-enabled medical devices—A subset of artificial intelligence-enabled medical devices: key terms and definitions. IMDRF/AIMD WG (PD1)/N67. Proposed document, 16 Sep 2021. Available from: https://www.imdrf.org/sites/default/files/2021-10/Machine%20Learning-enabled%20Medical%20Devices%20-%20A%20subset%20of%20Artificial%20Intelligence-enabled%20Medical%20Devices%20-%20Key%20Terms%20and%20Definitions.pdf

 Google Scholar

High-Level Expert Group on Artificial Intelligence. A definition of AI: main capabilities and scientific disciplines. 2018. Available from: https://ec.europa.eu/futurium/en/system/files/ged/ai_hleg_definition_of_ai_18_december_1.pdf

 Google Scholar

Medical Device Coordination Group. MDCG 2019-11: Guidance on Qualification and Classification of Software in Regulation (EU) 2017/745 – MDR and Regulation (EU) 2017/746 – IVDR. Available from: https://ec.europa.eu/health/system/files/2020-09/md_mdcg_2019_11_guidance_qualification_classification_software_en_0.pdf.

 Google Scholar

European Commission. Proposal for a Regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligence Act) And amending certain union legislative acts. Brussels. 2021;21(4):COM(2021) 206 final 2021/0106 (COD).

 Google Scholar

Lekadir K, Quaglio G, Tselioudis Garmendia A, et al. Artificial intelligence in healthcare. Applications, risks, and ethical and societal impacts. European Parliamentary Research Service - Scientific Foresight Unit (EPRS - STOA). PE 729.512 – June 2022. Available from: https://www.europarl.europa.eu/RegData/etudes/STUD/2022/729512/EPRS_STU%282022%29729512_EN.pdf.

 Google Scholar

International Standardisation Organisation/International Electrotechnical Commission (ISO/IEC). Joint Technical Committee 1/SC 42. Information technology — vocabulary. (2015). ISO/IEC 2382: 2015.(revision under review) Available from: https://www.iso.org/obp/ui/#iso:std:iso-iec:2382:ed-1:v1:en

 Google Scholar

ISO/IEC JTC 1 (on information technology)/Sub-committee 42 (on AI). Information technology — artificial intelligence — artificial intelligence concepts and terminology. 2022. 2022–07 ISO/IEC 22989:2022 Available from: https://www.iso.org/obp/ui/#iso:std:iso-iec:22989:ed-1:v1:en

 Google Scholar

ISO/IEC JTC 1 (on information technology)/Sub-committee 42 (on AI) IEEE Engineering in Medicine and Biology Society (EMBS). Artificial Intelligence Medical Devices Working Group (EMB/Stds Com/AIMDWG). Available from: https://sagroups.ieee.org/aimdwg/

 Google Scholar

IEEE Recommended practice for the quality management of datasets for medical artificial intelligence. 2022 IEEE 2801-2022 Available from: https://standards.ieee.org/ieee/2801/7459/

 Google Scholar

Organisation for Economic Co-operation and Development. OECD Science, Technology and Industry Scoreboard 2017: The Digital Transformation. doi: 10.1787/9789264268821-en

 Google Scholar

OECD. Council on Artificial Intelligence. OECD, Recommendation of the Council on Artificial Intelligence, OECD/LEGAL/0449. 2019. Available from: https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL-0449

 Google Scholar

Expert OECD Group on Artificial Intelligence. Scoping the OECD AI principles: deliberations of the expert group on artificial intelligence at the OECD (AIGO). OECD digital economy Papers No. 291. OECD Publishing; Paris; 2019. doi: 10.1787/d62f618a-en

 Google Scholar

OECD Committee on digital economy policy. OECD Framework for the classification of AI systems. (2022). OECD Digital Economy Papers, No. 323.

 Google Scholar

World Health Organization. Ethics and governance of artificial intelligence for health. WHO guidance. Geneva (2021) https://www.who.int/publications/i/item/9789240029200.

 Google Scholar

Food and Drug Administration, USA. Proposed regulatory framework for modifications to artificial intelligence /machine learning (AI/ML)-based software as a medical device (SaMD). 2019. Available from: https://www.fda.gov/files/medical%20devices/published/US-FDA-Artificial-Intelligence-and-Machine-Learning-Discussion-Paper.pdf.

 Google Scholar

Larson DB, Harvey H, Rubin DL, et al. Regulatory frameworks for development and evaluation of artificial intelligence-based diagnostic imaging algorithms: summary and recommendations. J Am Coll Radiol. 2021;18:413–424.

 PubMed Web of Science ®Google Scholar

U.S. Food & Drug Administration. Content of premarket submissions for device software functions - draft guidance for industry and food and drug administration staff November 4, 2021. Available from: https://www.fda.gov/media/153781/download

 Google Scholar

International Medical Device Regulators Forum. Software as a Medical Device (SaMD) Working Group. “Software as a Medical Device”: possible framework for risk categorization and corresponding considerations. IMDRF/SaMD WG/N12FINAL. 2014. Available from: https://www.imdrf.org/documents/software-medical-device-possible-framework-risk-categorization-and-corresponding-considerations.

 Google Scholar

Collins GS, Dhiman P, Andaur Navarro CL, et al. Protocol for development of a reporting guideline (TRIPOD-AI) and risk of bias tool (PROBAST-AI) for diagnostic and prognostic prediction model studies based on artificial intelligence. BMJ Open. 2021;11(7):e048008.

 PubMed Web of Science ®Google Scholar

Sounderajah V, Ashrafian H, Golub RM, et al. STARD-AI steering committee. Developing a reporting guideline for artificial intelligence-centred diagnostic test accuracy studies: the STARD-AI protocol. BMJ Open. 2021;11(6):e047709.

 PubMed Web of Science ®Google Scholar

Vasey B, Nagendran M, Campbell B, et al. DECIDE-AI expert group. Reporting guideline for the early stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. BMJ. 2022;377:e070904.

 PubMedGoogle Scholar

Rivera SC, Liu X, Chan AW, et al. SPIRIT-AI and CONSORT-AI Working Group. Guidelines for clinical trial protocols for interventions involving artificial intelligence: the SPIRIT-AI Extension. BMJ. 2020;370:m3210.

 PubMedGoogle Scholar

Liu X, Rivera SC, Moher D. et al; SPIRIT-AI and CONSORT-AI Working Group. Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI Extension. BMJ. 2020;Vol. 370:m3164.

 Google Scholar

Park Y, Jackson GP, Foreman MA, et al. Evaluating artificial intelligence in medicine: phases of clinical research. JAMIA Open. 2020;3:326–331.

 PubMedGoogle Scholar

Sendak MP, D’Arcy J, Kashyap S, et al. A path for translation of machine learning products into healthcare delivery. EMJ Innov. 2020. DOI:10.33590/emjinnov/19-00172.

 Google Scholar

McCradden MD, Stephenson EA, Anderson JA. Clinical research underlies ethical integration of healthcare artificial intelligence. Nat Med. 2020;26:1325–1326.

 PubMed Web of Science ®Google Scholar

Higgins D, Madai VI. From bit to bedside: a practical framework for artificial intelligence product development in healthcare. Adv Intell Sys. 2020;2(10):2000052.

 Google Scholar

Sengupta PP, Shrestha S, Berthon B, et al. Proposed Requirements for Cardiovascular Imaging-Related Machine Learning Evaluation (PRIME): a checklist: reviewed by the American College of Cardiology Healthcare Innovation Council. JACC Cardiovasc Imaging. 2020;13:2017–2035.

 PubMed Web of Science ®Google Scholar

Schwendicke F, Singh T, Lee JH, et al. IADR E-oral health network and the ITU WHO focus group AI for Health. Artificial intelligence in dental research: checklist for authors, reviewers, readers. J Dent. 2021;107:103610.

 Web of Science ®Google Scholar

Lekadir K, Osuala R, Gallin C, et al. FUTURE-AI: guiding principles and consensus recommendations for trustworthy artificial intelligence in medical imaging. https://arxiv.org/pdf/2109.09658.pdf (accessed 2022 Aug 18).

 Google Scholar

Norgeot B, Quer G, Beaulieu-Jones BK, et al. Minimum information about clinical artificial intelligence modeling: the MI-CLAIM checklist. Nat Med. 2020;26:1320–1324.

 PubMed Web of Science ®Google Scholar

Hernandez-Boussard T, Bozkurt S, Ioannidis JPA, et al. MINIMAR (MINimum Information for Medical AI Reporting): developing reporting standards for artificial intelligence in health care. J Am Med Inform Assoc. 2020;27:2011–2015.

 PubMed Web of Science ®Google Scholar

Sendak MP, Gao M, Brajer N, et al. Presenting machine learning model information to clinical end users with model facts labels. NPJ Digit Med. 2020;3:41.

 PubMed Web of Science ®Google Scholar

Olczak J, Pavlopoulos J, Prijs J, et al. Presenting artificial intelligence, deep learning, and machine learning studies to clinicians and healthcare stakeholders: an introductory reference with a guideline and a Clinical AI Research (CAIR) checklist proposal. Acta Orthop. 2021;92:513–525.

 PubMed Web of Science ®Google Scholar

Mongan J, Moy L, Kahn CE Jr. Checklist for Artificial Intelligence in Medical Imaging (CLAIM): a Guide for Authors and Reviewers. Radiol Artif Intell. 2020;2:e200029.

 PubMedGoogle Scholar

Kitaguchi D, Watanabe Y, Madani A, et al. Computer Vision in Computer Vision in Surgery International Collaborative. Artificial intelligence for computer vision in surgery: a call for developing reporting guidelines. Ann Surg. 2022;275:e609–e611.

 PubMed Web of Science ®Google Scholar

Vollmer S, Mateen BA, Bohner G, et al. Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness. BMJ. 2020;368(l):6927.

 Google Scholar

Scott I, Carter S, Coiera E. Clinician checklist for assessing suitability of machine learning applications in healthcare. BMJ Health & Care Informatics. 2021;28:e100251.

 PubMedGoogle Scholar

Petzold A, Albrecht P, Balcer L, et al. IMSVISUAL, ERN-EYE Consortium. Artificial intelligence extension of the OSCAR-IB criteria. Ann Clin Transl Neurol. 2021;8:1528–1542.

 PubMedGoogle Scholar

Daneshjou R, Barata C, Betz-Stablein B, et al. Checklist for evaluation of image-based artificial intelligence reports in dermatology: CLEAR derm consensus guidelines from the international skin imaging collaboration artificial intelligence working group. JAMA Dermatol. 2022;158:90–96.

 PubMed Web of Science ®Google Scholar

AI Watch. Joint Research Centre, European Commission, Seville. Available from: https://ai-watch.ec.europa.eu/publications_en

 Google Scholar

Nativi S, De Nigris S. AI standardisation landscape: state of play and link to the EC proposal for an AI regulatory framework, EUR 30772 EN. Luxembourg: Publications Office of the European Union; 2021.

 Google Scholar

European Medicines Agency (EMA). Regulatory science, research needs (2021) Available from: https://www.ema.europa.eu/en/documents/regulatory-procedural-guideline/ema-regulatory-science-2025-strategic-reflection_en.pdf

 Google Scholar

EMA with HMA (Heads of Medicines Agencies). Accelerating clinical trials in the EU (ACT EU) (2022). Available from: https://www.ema.europa.eu/en/news/accelerating-clinical-trials-eu-act-eu-better-clinical-trials-address-patients-needs

 Google Scholar

Koene A, Clifton C, Hatada Y, et al. A governance framework for algorithmic accountability and transparency. European Parliament Directorate General for Parliamentary Research Services, Scientific Foresight Unit (STOA). PE 624.262 April 2019. https://www.europarl.europa.eu/RegData/etudes/STUD/2019/624262/EPRS_STU(2019)624262_EN.pdf.

 Google Scholar

OECD.AI Policy Observatory. Available from: https://oecd.ai/en/

 Google Scholar

Council of Europe Ad Hoc Committee on AI (CAHAI). Available from: https://www.coe.int/en/web/artificial-intelligence/cahai

 Google Scholar

Committee on Artificial Intelligence (CAI). Available from: https://www.coe.int/en/web/artificial-intelligence/cai

 Google Scholar

Council of Europe. Steering Committee for Human Rights in the fields of Biomedicine and Health (CDBIO). Available from: https://www.coe.int/en/web/artificial-intelligence

 Google Scholar

Mittelstadt B. The impact of artificial intelligence on the doctor-patient relationship. Report commissioned by the steering committee for human rights in the fields of biomedicine and health. Council of Europe, 2021. Available from: https://www.coe.int/en/web/bioethics/report-impact-of-ai-on-the-doctor-patient-relationship

 Google Scholar

CEN-CENELEC Joint Technical Committee 21 ‘Artificial intelligence.’ Available from: https://www.cencenelec.eu/areas-of-work/cen-cenelec-topics/artificial-intelligence/

 Google Scholar

CEN-CENELEC Work Programme 2022. Available from: https://www.cencenelec.eu/media/CEN-CENELEC/News/Publications/2022/cen-cenelec_work_programme2022.pdf

 Google Scholar

WHO/ITU-T Focus Group on Artificial Intelligence for Health (FG-AI4H). Available from: https://www.itu.int/en/ITU-T/focusgroups/ai4h/Documents/FG-AI4H-ToR.pdf

 Google Scholar

Working Groups include Regulatory considerations on AI for health, and Clinical evaluation of AI for health. Available from: https://www.itu.int/en/ITU-T/focusgroups/ai4h/Pages/default.aspx

 Google Scholar

International Standardization Organisation ISO. ISO/IEC JTC 1/SC 42 Artificial Intelligence. Available from: https://www.iso.org/committee/6794475.html

 Google Scholar

Zielke T. Is artificial intelligence ready for standardization? In: Yilmaz M, Niemann J, Clarke P, et al., editors. Systems, software and services process improvement. EuroSPI 2020. Communications in computer and information science. 2020;1251:259–274. Springer: Cham.

 Google Scholar

International Electrotechnical Commission (IEC). 62/411/NP: testing of Artificial Intelligence/Machine Learning-enabled Medical Devices. Available from: https://www.iec.ch/dyn/www/f?p=103:23:612016022541470::::FSP_ORG_ID,FSP_LANG_ID:1245,25

 Google Scholar

International Electrotechnical Commission (IEC). Artificial intelligence/machine learning-enabled medical device – performance evaluation process. Available from: https://www.iec.ch/ords/f?p=103:38:7211217005198::::FSP_ORG_ID,FSP_APEX_PAGE,FSP_PROJECT_ID:1245,23,107066

 Google Scholar

IEEE P2673 Intelligence Augmentation for Medical Imaging Working Group. Available from: https://sagroups.ieee.org/2673/

 Google Scholar

EMBS Artificial Intelligence Medical Devices Working Group. Available from: https://sagroups.ieee.org/aimdwg/

 Google Scholar

IEEE 2801-2022. Recommended practice for the quality management of datasets for medical artificial intelligence. 2022 Available from: https://standards.ieee.org/ieee/2801/7459/

 Google Scholar

IEEE 2801-2022. Recommended practice for the quality management of datasets for medical artificial intelligence. 2022. Available from: https://sagroups.ieee.org/aimdwg/wp-content/uploads/sites/23/2018/12/P2801-PAR.pdf

 Google Scholar

Artificial Intelligence Medical Devices Working Group. Standard for the performance and safety evaluation of artificial intelligence based medical device: terminology. Available from: https://sagroups.ieee.org/aimdwg/wp-content/uploads/sites/23/2018/12/P2802-PAR.pdf

 Google Scholar

New IEEE standard (IEEE P2802) Available from: https://development.standards.ieee.org/myproject-web/public/view.html#pardetail/6755

 Google Scholar