Balancing innovation and ethics in AI governance for health technology assessment

Introduction

The use of Artificial Intelligence (AI) in healthcare promises unprecedented advancements in health technology assessment (HTA). HTA is a multidisciplinary and systematic process that methodically evaluates the properties, effects, impacts, and overall value of healthcare technologies at various stages of their lifecycle. The aim of HTA is to inform policy and decision-making in health systems, ensuring equitable, efficient, and high-quality health care.^1,2 AI will unlock improvements in efficiency and accuracy in HTA, and patient care strategies,^3–6 which go beyond mere technological advancement. It includes the possibility to deepen our understanding of complex health economic issues, refine outcomes research methodologies, and elevate HTA standards.^7,8 The use of AI in healthcare comes with ethical, legal and social challenges that can be navigated through robust and adaptable governance models.^9–11 Realizing the potential of AI depends on the adoption of governance models towards ethical use of AI.

This editorial seeks to elaborate on governance models to enable AI’s effective deployment in HTA. Here we emphasize the necessity for governance models that are effective and ethically sound, aligning them with the principles of equity and innovation.^12,13 We’re guided by a vision where AI’s impact on healthcare decision-making is both transformative and ethically grounded,¹⁴ through (i) building awareness of current global governance approaches of AI in healthcare, (ii) extracting governance lessons from Big Data in healthcare, and (iii) considering the challenges and gaps in AI governance. The crucial question at the heart of this evolution is how to achieve a balance between innovation and ethics in the governance of AI for HTA.

Global governance approaches of AI in healthcare

The United Nations (UN) and the World Health Organization (WHO) have played crucial roles in shaping governance models for AI, with a focus on enhancing healthcare decision-making processes, integrating AI technologies, and addressing the needs of a rapidly evolving healthcare landscape. The UN’s recommendation for AI governance seeks to align AI development with international norms.¹⁵ It highlights five principles as preliminary recommendations, set to be finalized by August 2024 (Table 1). The recent UN General Assembly resolution A/78/L.49, adopted on 11 March 2024, operationalizes the UN’s commitment to leveraging AI for sustainable development while ensuring safety, security, and trustworthiness.¹⁶ This resolution emphasizes effective governance, and the alignment of AI systems with human rights, reinforcing the UN's guiding principles on AI governance. Similarly, the WHO presented guidance on the ethics and governance of AI in healthcare as recommendations for governments, technology companies, and healthcare providers to ensure the appropriate use of AI.¹⁷ This guidance emphasizes AI's potential to significantly improve healthcare outcomes and was recently applied in the context of Large Multi-Modal Models (LMMs).¹⁸

Table 1. The UN and WHO’s principles on AI.

UN (2023) Guiding Principles: Governing AI for Humanity	WHO (2021) Ethical Principles for use of AI for Health
AI should be governed inclusively, by and for the benefit of all	Ensure inclusiveness and equity
AI must be governed in the public interest	Promote human well-being, human safety and the public interest
AI governance should be built in step with data governance and the promotion of data commons	Ensure transparency, explainability, and intelligibility
AI governance must be universal, networked and rooted in adaptive multi-stakeholder collaboration	Foster responsibility and accountability
AI governance should be anchored in the UN Charter, International Human Rights Law, and other agreed international commitments such as the Sustainable Development Goals	Protect autonomy
	Promote AI that is responsive and sustainable

While both organizations emphasize inclusivity and public interest (Table 1), there are principles that do not share exact similarities. For instance, the UN references important international normative frameworks to guide the design of AI governance models, such as the UN Charter and International Human Rights. The WHO emphasizes a more ethical interpretation of accountability. These differences highlight potential gaps and inconsistencies but also demonstrate various models in framing guiding governance principles for AI use. A nuanced approach to governance models can help bridge these gaps.¹⁹ To this end, the governance of Big Data in healthcare may inform the deployment of AI in healthcare, particularly in HTA.

Governance lessons from Big Data in healthcare

The historical application of governance principles in healthcare and Big Data has revealed critical insights into ethical considerations, data privacy, and regulatory frameworks.^17,18 With the shift to electronic health records (EHR) and the advent of Big Data analytics, issues like patient consent, data security, and algorithmic biases have become prominent. These developments have highlighted the need for robust governance models that can adapt to technological advancements while safeguarding patient rights and ensuring equitable access to healthcare.^20,21

The potential of Big Data in the health sector is significant, offering enhancements in personal and clinical care, public health, and research through the integration and analysis of large healthcare datasets and EHR.²² However, this potential also brings forth privacy, security, and ethical challenges that necessitate ongoing ethical oversight and comprehensive governance frameworks that evolve in response to these issues.^22,23 The European Union’s General Data Protection Regulation (GDPR) exemplifies efforts to strengthen data protection and privacy, underscoring the importance of regulatory responses to the complexities introduced by Big Data in healthcare.²² Balancing privacy concerns with the benefits of Big Data in healthcare is intricate, as governance frameworks strive to protect privacy without stifling innovation.²¹ This balance is crucial in navigating the legal and ethical dilemmas associated with patient privacy.

Big Data in healthcare has emphasized the need for coordinated policy development to tackle issues of data quality, confidentiality, usage, and interoperability.^24,25 Consequently, integrative, and increasingly democratic approaches have enhanced interoperability, data quality, and governance. These initiatives lay the groundwork for addressing both shared and unique challenges associated with the use of AI in HTA.

Challenges and gaps in the governance of AI in HTA

Incorporating AI into HTA offers potential solutions, particularly in analyzing complex relationships within Big Data, such as voluminous EHR and real-world evidence (RWE).^24,25 AI can expedite decision-making processes and promote fairness,²⁶ while enhancing disease or risk detection and healthcare classification. It aids in identifying patient cohorts with shared characteristics that may not be readily discernible through conventional methods. For example, AI tools have been used to select patients for oncology studies or extract physician-reported side effects from EHR.²⁶ Furthermore, AI can accelerate time-consuming tasks such as systematic literature reviews, assist with tailoring treatment options such as personalized treatment, and predict trajectories of health outcomes.^3,26 Challenges in the application of AI exists in the form of several aspects such as the lack of longitudinal data, especially for advanced therapeutic medicinal products (ATMPs), lack of high-quality data sources, combing clinical data with RWE, and barriers to perform complex health interventions.^25,26

The fusion of healthcare and technology is primarily characterized by the integration of various types of data and advancements in technology—transforming decision-making through AI-augmented systems.^6,13,26 When AI is incorporated into HTA, it should be designed to augment, not replace, human intelligence. A human-centered, problem-driven approach to building AI systems involves considerations of design, development, evaluation, validation, scaling, diffusion, monitoring, and maintenance.⁶ This approach requires the involvement of stakeholders who understand the core principles of ethical AI use and can evaluate the effectiveness of outcomes.^3,6

Monitoring the use of AI in HTA will involve aligning with the recommendations for governance frameworks.^15,18,27 Establishing global governance guidelines for AI in HTA is crucial, covering areas such as data management, privacy, ownership, and independent decision-making.^3,13,26 Operationalizing the governance principles from the UN and WHO could be a significant step towards refining a governance framework for AI use in HTA. To achieve this goal, Table 2 provides essential questions aimed at assisting HTA professionals in exploring how the governance of AI can be ethically structured and fully realize its potential.

Table 2. Principles and essential questions on governance of AI in HTA.

Principles	Essential questions
Inclusivity	How were applicable patient cohorts, populations, or samples defined and included for model training? How was the AI model and data monitored and evaluated for bias, and how were stakeholders engaged in this process? How were the limitations of the data identified, addressed, and communicated to relevant stakeholders?
Equity	How were diverse populations considered in the development and application of the AI tool? What measures were taken to prevent the AI tool from exacerbating existing health disparities? How were capacity and infrastructure constraints in diverse settings considered?
Well-being and Safety	What specific norms and standards were integrated to ensure well-being and safety in AI-enabled HTA models? How are adverse events or errors in the AI tool monitored, addressed, and communicated?
Public Interest	How are health outcomes quantified and aligned with public interest? What mechanisms are in place for public engagement and feedback regarding the AI tool?
Promoting Autonomy	How are stakeholders’ rights to self-determination and informed decision-making protected? How were data privacy and confidentiality maintained, and what processes are available for stakeholders to challenge or appeal outcomes/decisions made by the AI tool?
Sustainable	How was AI used to reduce the burden on healthcare decision-making, and what are the environmental impacts of deploying the AI tool? What processes are in place for the continuous improvement and maintenance of the AI tool?
Responsibility and Accountability	How were stakeholders responsible for AI outcomes identified, and how is accountability maintained? What mechanisms are in place for addressing errors, biases, or unintended consequences detected post-deployment?
Adaptive Multi- stakeholder Collaboration	How is the effectiveness of the existing multi-stakeholder network and structure regularly reviewed and updated? How were stakeholders from diverse backgrounds involved in the development and adaptation processes?
Transparency	How transparent are the AI decision-making processes, and how is information about AI tool performance shared with stakeholders? What strategies are in place to document and store specific parameters and configurations of the AI model for reproducibility?
Explainability	How are the criteria for identifying relevant HTA users established, and how are explanations tailored to different user groups? What training is provided to ensure responsible and informed use of AI tools?
Intelligibility	What validation methods are used for testing model responses, and how are model outputs validated for accuracy? How are the AI tool’s outputs communicated to end-users, and what feedback mechanisms are in place to improve intelligibility?

Conclusion

Integrating AI into HTA models requires a forward-looking approach that balances innovation with ethical considerations. Insights from global governance models, past applications, and contemporary challenges inform the development of comprehensive, adaptable governance frameworks. Moreover, the essential questions on the governance of AI in HTA represent an applied approach to operationalize the principles of the UN and WHO. However, international cooperation and dialogue will be crucial in ensuring that AI acts as a catalyst for improving patient outcomes and bridging health disparities. This journey will require concerted efforts from all stakeholders toward flexible and adaptable regulatory frameworks that accommodate the unique elements of AI and enhance its utility in healthcare.

Transparency

Declaration of financial/other interests

JLC, DB, and RC are employees at Syenza and consult for pharmaceutical, medical device, and healthtech companies.

Author contributions

JLC, DB, and RC: Study concept and design, data collection, data interpretation, manuscript writing, and manuscript revisions.

Reviewer disclosures

Peer reviewers on this manuscript have no relevant financial or other relationships to disclose.

Acknowledgements

Sumona Bose, former Syenza associate, for contributing to prior drafts of the manuscript.

Additional information

Funding

No funding was received to produce this article.