Transferability of new methods for health technology assessment in the field of diabetes between early and late adopters’ countries

Abstract

This study aimed to investigate the transferability of novel artificial intelligence (AI) methods for prediction modelling of diabetes based on real-world data (RWD) between early and late adopters of emerging health technologies from the perspective of developers and health technology assessment (HTA) experts. A two-step approach was used. Developers of the new AI methods within HTx consortium completed a survey about the benefits, usability, barriers associated with implementing the new prediction models in routine HTA practices. Then, HTA experts from Central and Eastern European (CEE) countries participated in a focus group discussion. Developers generally expressed optimism regarding the transferability of the methods, while acknowledging potential disparities across CEE countries. Key benefits that were identified included enhanced understanding of diabetes, improved cost-effectiveness modelling, and refined patient stratification, all of which could contribute to clinical and reimbursement decisions across various jurisdictions. The focus group underscored the value of real-world data for diabetes prediction modelling, serving as a beneficial resource for both clinicians and HTA agencies. However, there was a recognized need to clarify the processes of integrating randomized clinical trial data with real-world data. For the other stakeholders, the advancement of the methodology will improve the diagnosis and therapy during the process of decision making. Experts from CEE countries recognized the potential of artificial intelligence-based methods employing real-world data for diabetes modelling. These methods are seen as instrumental in elucidating the heterogeneous nature of the disease, supporting clinician decision-making and holding promises for HTA purposes.

Keywords:

• Transferability
• health technology assessment
• Central and Easter European Countries
• diabetes
• artificial intelligence

Introduction

Diabetes is recognized as one of the socially important diseases in the twenty first century, largely affecting the quality and quantity of life in humans worldwide [1,2]. The latest figures from the International Diabetes Federation show that 537 million adults (20-79 years) are currently living with diabetes, with that number being expected to rise to 643 million by 2030 and 783 million by 2045 [3]. Mortality from diabetes reached 6.7 million deaths in 2021. With the introduction of novel treatment options and newer molecules, the financial burden of diabetes has documented a 316% increase in the health care cost over the last 15 years [3].

The development and use of real-world data (RWD) for diabetes has provided an opportunity for predicting the relative risk of short- and long-term complications appearance [4,5]. There are several registries and accessible databases which could be used by researchers to explore the impact of the disease progression on people’s life and to model it. Most of them are not publicly available, but they can be accessed for research purposes under certain conditions [6–8].

New methods for health technology assessment (HTA) based on RWD for the purposes of predicting long term therapeutic effect are increasingly being developed and employed by clinical researchers and agencies [9–12]. Barriers to their usage for HTA purposes by the late adopter countries of new technologies have been described and their importance has been previously evaluated [13,14].

For the HTA agencies, the issue of transferability of methods used in other jurisdictions remains a challenge [15–17]. It is especially true when the methods were developed by using real-world evidence created elsewhere due to the questionable comparability of therapeutic practices worldwide [18]. In addition to transferability across different countries, the generalizability of new HTA methods to other disease areas or patient groups needs to be evaluated to ensure their wider application [19–21].

With the advancement of technologies and statistical methods there is an opportunity to unify and compare information from different databases for diabetes progression over time and to make more reliable and generalized predictions for the progression of the diseases in individual patients by using artificial intelligence (AI) [22].

This study aimed to investigate the transferability of novel artificial intelligence (AI) methods for prediction modelling of diabetes based on RWD between early and late adopters of emerging health technologies from the perspective of developers and health technology assessment (HTA) experts.

This work is part of the H2020 HTx project, whose goal is to provide a new generation of health technology assessments (https://www.htx-h2020.eu/). In the field of diabetes, the project aims to apply technological improvements to data curation, as well as to combine evidence extracted from real-world data sources for better prediction of the course of disease.

Methodology

Case study methodology

The HTx project aims at supporting patient-centred, societally oriented, real-time decision-making on access to, and reimbursement for health technologies throughout Europe. Diabetes is one of the case studies (CS2) in HTx project, which focuses on understanding the heterogeneous care needs in diabetes using AI algorithms with RWD. For this, two different approaches have been followed: 1) Identification of patients’ subgroups characterized by specific conditions and study of treatment pathways and quality of life; and 2) creation of personalized predictive models which can help to optimize treatment recommendations, to get evidence about risks, and to support decision making processes in Health Technology Assessment and shared decision making in clinical practice. The transferability issues were debated on this specific case to answer to the question whether the new prediction methodologies could be transferred between HTA jurisdiction to improve decision making processes.

The CS2 relied on several databases. Clinical Practice Research Datalink (CPRD) is a real-world research service supporting retrospective and prospective public health and clinical studies [6]. CPRD collects anonymized patient data from a network of general practitioner (GP) practices across the United Kingdom (UK). It covers over 11.3 million patients from 674 practices. For patients with diabetes, data extracted consisted of 17,260 patients with type 1 diabetes and 249,099 patients with type 2 diabetes. The Type 1 Diabetes Exchange Registry is a publicly available dataset which includes routine clinic exams for more than 34,000 participants with Type 1 diabetes, from 83 clinic sites in the United States [23]. The Maastricht Study is a population-based cohort study focused on the etiology, pathophysiology, complications and comorbidities of T2DM [7]. BIFAP, the Spanish database for Pharmacoepidemiological Research in Primary Care is a computerized medical longitudinal population-based database of anonymized electronic medical records of primary care, which includes demographic factors, consultation visits, referrals, hospital admissions, laboratory test results, diagnostic procedures, diagnoses and prescriptions [8].

Different AI methods used are cluster analysis to identify subgroups of patients in T1D and T2D [24] and complex predictive models with Machine Learning (ML) or Deep Learning (DL) algorithms for different clinical outcomes of interest in diabetes. Primary outcomes identified are mortality and glycaemic control measured with HbA1c levels. Secondary outcomes identified include short- and long-term complications and quality of life. ML, as part of AI, is based on systems that can recognize patterns and extract knowledge automatically from data [25]. DL methods have been boosted lately to exploit the complexity of temporal data sequences such as data recorded during clinical visits. Both ML and DL methods have allowed obtaining evidence about risks associated to different treatment pathways, to predict changes in glycaemic control after the start of a specific treatment or to predict acute complications [26]. Also, different treatment trajectories have been identified in patients with T2D, and specific predictive models of trajectory membership for individual patients have been implemented [27].

CS2 partners have obtained access to different RWD sources with T1D and/or T2D patients, which have been used to identify subgroups of patients with cluster analysis or to train and validate predictive models. For each data source, longitudinal data are available with information registered along time during clinical encounters for individual patients. However, the registration time is variable, depending on each RWD source. Also, it is important to notice that although all RWD sources contain information related to demographics, laboratory tests, complications, treatment recommendations and clinical outcomes relevant for patients with T1D and T2D, the available variables are not the same in all the data sources. Transferability of the methods created in the CS2 are being analyzed for the different data sources available.

Methodology to explore transferability to Central and Eastern European Countries

A two-step approach was used to explore transferability of the new HTA methodology for diabetes prediction modelling to Central and Eastern European countries between May and November 2023. First, developers of the new AI methods within HTx consortium completed a survey about the benefits, usability, barriers associated with implementing the new prediction models, explained above in routine HTA practices. Table 1 presents the responses in the survey.

Table 1. Survey results from method developers.

N	Question	Response 1	Response 2	Response 3
1	Do you think that the methods from diabetes case study could be applied in other jurisdictions, especially in CEE countries?	I think that the methods from diabetes case study could be applied in other jurisdictions. I have doubts if they could be applied in all CEE countries, but I think it is possible in most.	Yes	The transferability of methods is the highest for diabetes case study.
2	How would you explain the main benefits of diabetes case study for HTA doers and users in CEE countries?	I believe that the methods developed could be a very useful tool for understanding the disease and the different factors that influence its control.	In the case of type-2 diabetes, we developed cost-effectiveness model with individual-level data. We have ambitions that this model could incorporate various risk prediction models that fit different CEE countries. Users (e.g. HTA agencies) or beneficiaries of the model (e.g. clinicians) could obtain more accurate information on cost-effectiveness of a drug related to type-2 diabetes.	Identifying patient groups either by risks or clusters and give them different maximum budget levels, which still makes the treatment cost-effective.
3	What are the main challenges in transferring results via your methodology to other countries across Europe?	The main challenge in transferring my methodology is the availability of the same variables in different countries.	Individual data that are used to develop a risk prediction model in a specific country can be missing.	Data availability.
4	How could these challenges be overcome?	Only the most common variables could be used.	Aggregate data might be used to create an artificial individual data set. While it has some limitations, it might overcome the challenges to some extent.	Data sharing.
5	Do you think that the HTA doers in different jurisdictions can apply the methodology that you developed?	I think that it is possible although I am not sure.	Yes. We took into account the transferability issues at the beginning. Though not perfect, we provided a solution.	Yes.
6	What kind of expertise will be needed for HTA doers to be able to apply your methodology?	They would need more technical expertise.	To use the model and to check the model quality, the HTA doers may need to use R software, which is difficult for someone who only uses Excel. The doers at least need to have knowledge regarding the disease itself, such as the information on variables (e.g. use of insulin, complications, BMI, etc.) related to the disease (i.e. diabetes), as doers might need to set these variables themselves when using the model.	Some understanding of statistical methods.
7	Do you think that the HTA users in different jurisdictions can benefit from your methodology in their decisions?	Yes, I do.	Yes, as a cost-effectiveness model that can be used in different jurisdictions contributes to reimbursement or clinical decisions regarding a treatment.	Yes.
8	What kind of expertise will be needed for HTA users to benefit from your methodology in their decisions?	We are working on a decision-making tool, then they would not need any specific kind of expertise.	Similarly, users may feel hard to understand a R model, as it needs some expertise.	Some understanding of statistical methods.

Next, HTA experts from CEE countries – including Bulgaria, Croatia, Czechia, Hungary, Kazakhstan, North Macedonia, Poland, Romania, Slovakia, Slovenia and Ukraine participated in a focus group discussion held during the conference workshop. Experts were invited through their institutions (HTA agencies) or as Intentional Society of Pharmacoeconomics and Outcomes Research (ISPOR) members. Only one country could not participate (Serbia).

The focus group started with presenting major findings and draft results from the newly developed methods. Experts were then asked to vote on the challenges in transferring the prediction models between different countries, as well as on their innovativeness for the current HTA practice. After voting, the experts held a more detailed discussion in the focus group about the transferability and generalizability of the created methods from the point of view of HTA agencies and their own experience. Qualitative analysis was used for gathering feedback on the concepts using Microsoft Office Excel 97-2003.

Results

Survey results

The survey was distributed among all 16 participating institutions in the HTx project consortia. Three of them were involved in the development of the new methodology in diabetes and their responses are summarized in Table 1.

In general, the developers of the methods are positive about their transferability, although they did point out that this could not be the case for all CEE countries. Understanding the nature of diabetes, the natural progression, cost-effectiveness modelling, and improved patient stratification were considered as major benefits for clinicians, patients and HTA agencies and those benefits can be used in different jurisdictions thus contributing to reimbursement or clinical decisions regarding treatment. All three responders consider that data availability and quality are the main challenges in transferability and that they could be overcome by data aggregating, sharing, or usage of most common variables. When taking into account the transferability issues in the beginning of model development, the HTA experts from different jurisdictions can apply the methodologies in their practice. To be applied by HTA agencies new methods require technical and statistical expertise, skills in R programming, and knowledge about the disease.

Results from voting on transferability

The first question for the participants in voting was ‘Do you agree that the diabetes prediction model applied in the HTx project is an improvement to current standard HTA methods?’. The prevailing part of experts (87%) agreed that the approach in the HTA project is a major or radical improvement to the existing standard HTA approach (Figure 1).

Figure 1. Voting results on ‘Do you agree that the diabetes prediction model applied in the HTx project is an improvement to current standard HTA methods?’.

The second question attempted to investigate the experts’ opinion about generalizability of the new method within the framework of joint European HTA. Thirteen experts (81%) considered that the prediction model for diabetes fits in the framework of joint European HTA.

In ranking the main challenges to transferring the prediction model for diabetes in CEEC, 12 out of 16 experts ranked first the differences in patient pathway for specific patient groups (Figure 2). Lack of available data for adaptation of international HTA methods is the second larger challenge (12 out of 16 experts), followed by lack of available financial resources and experts for the local adaptation of international HTA methods (9 out of 16 experts). It seems that patient characteristics, data and expertise are more important than governance and limited interest toward the acceptance of new healthcare evaluation methods.

Figure 2. Voting results on ‘Challenges in transferring prediction model for diabetes in different countries’.

HTA agencies experts’ comments during focus group discussion

The focus group discussion was concentrated and highlighted the several issues summarized below:

1. Easy to understand description of generic tools to combining data related to CS2 on diabetes.

   The following concerns and comments were raised by the experts with respect to the understanding of generic tools. There are several prediction models for HbA1c in diabetes mellitus. The incremental benefit of HTx model compared to previous models should be described in both lay language and technical reports. Both types of documents should incorporate sections dedicated to the usability of the findings and methods in real life.

   Generally, the use of the tools resulting from this case study requires an advanced level of understanding of the related sciences and should not require purchasing specific software. It needs to be described how HbA1c levels are related to endpoints such as survival or quality of life that are of importance for HTA evaluations.

   Potential use could be when evaluating new drugs, re-evaluating previous decisions, or finding sub-populations that are more cost effective than other sub-groups. Methods and results coming from CS2 should be available to the widest audience possible. In order to achieve this, a lay language summary can be suggested, in addition to the more technical reports.

   A description (e.g. report) of the outputs emanating from CS2 that is easy to understand for non-technical audiences, such as HTA representatives and patient organizations is valuable. This could focus on the typical questions of ‘why?’, ‘how?’ and ‘so what?’, so HTx representatives are better prepared to ‘sell’ the outputs as useful and meaningful for their organizations and the wider HTA community. We would also welcome a discussion on the level of uncertainty surrounding the predictions and how these should be weighed in policy decisions.

   We suggest that the above ‘informed lay’ description could be accompanied by a more technical document. This could take the form of a ‘how to’ guide, similar to the technical support documents commissioned by NICE. It would be an immediately useful addition to an HTx ‘toolkit’ of methods outputs. Detailed peer-reviewed publications may serve this purpose.

2. Expectations on generalizability:

   Experts consider that combining RWD from multiple sources can be especially valuable results of this case study, as most of the time researchers only work with RWD from a single source.

   Questions arise about the extrapolation of RCT into an RWD setting. Will the same patient level effect be obtained when treating patients in a real world setting and will the time course of effect be similar?

   The experience with building personalized prediction models is also of high importance and can be useful in other disease areas as well. This is closely related to the telemonitoring study – another concept that can and should be extended beyond diabetes.

   It would also be valuable if the case study could reflect on general considerations regarding developing models to identify patients at risk of developing adverse events, to minimize their incidence. This is different to identifying patients to maximize treatment effectiveness.

   It is important to make recommendations on how the methodology should be used for policy decisions. This concerns the initial decision-making on technologies included in the analysis; reviewing previous policy decisions related to technologies in the analysis; or to extrapolate results to support policy decisions of forthcoming technologies for the same patients.

3. Expectations on transferability to support the implementation of joint European HTA (as described in the EU legislation):

   CS2 has a good starting point, as a Central and Eastern European country. However, efforts still should be made to allow the use of the methods and results of this case study to be useable and meaningful in EU member states with limited resources (and possibly even beyond the EU).

Specific studies on transferability, for example the estimates of cost-effectiveness of the different treatment and monitoring modalities mentioned in the work plan should already be set up in various jurisdictions, across countries with different income levels (e.g. lower income countries below the average EU GDP per capita).

A list of variables which are similar across participant countries (core transferable components) and those which vary between countries (country-specific components) has to be determined. Solutions to increase collaboration and transparency on these models has to be found.

Methodology should be recommended for key influential factors such as input data in the developed evaluation methods and tools in order to determine which country-specific components cause the highest variability in the results. The problems in data quality and bias must be addressed.

Patient demographics factors as well as co-treatments differ between different countries, and it needs to be shown how this affects treatment outcome in order to be able to show how the results can be transferred into another country setting.

Discussion

In this study, we explored the opinion of HTA experts from CEE countries, which usually are considered as late technologies adopters, about the transferability of novel HTA methods for curation of information from real world data bases for diabetic patients. Data curation involves collecting, structuring, indexing and cataloguing data for users in an organization, group, or the general public. [12,28]. Data curation allows to systematise existing datasets and solve problems with missing data in different registries.

Several novel methods combining RWD extraction from different sources were created during the HTx project. The methods used an AI approach and predicted the short- and long-term diabetes outcomes on both general and individual levels. We do not consider discussing the methods as technical and mathematical approaches but only their usability from the point of view of HTA experts from CEE countries [29]. We also do not consider discussing the influence of socioeconomic status on the transferability of data between different countries. We might presume that this variable is in fact considered when each expert gave their own opinion based on the national characteristics of the health care system.

It is evident from the inquiry and focus group discussion that the RWD for diabetes prediction modelling is a useful tool both for clinicians and HTA agencies. Clinicians can use prediction modelling on an individual level to make decisions for therapy maintenance [30]. HTA agencies can combine predictions with results from RCT in the process of evaluation of new medicines, although it was pointed out that there is a need to explain how the RCT data can be combined with RWD [31]. For the other stakeholders, the advancement of the methodology will improve the diagnosis and therapy during the process of decision making.

The experts’ expectations about the new methods are high and positive but still many questions need to be solved to make them usable in a pragmatic HTA world.

The AI based methods for diseases prediction models are already a fact that is entering clinical and regulatory practice worldwide [32]. They need to be made accessible and understandable for a wide range of professionals, including the HTA authorities, to benefit the health care of society. To achieve this, access to databases is needed, technical and programming expertise, guidelines on RWD usage for HTA models, for application of methods and models in HTA practice. Transferability and generalizability of methods remain an issue worth exploring. For CEE countries additional barriers could also be the lack of expertise in the interpretation of methods as well as the limited pool of experts to draw from.

Conclusions

Experts from CEE countries recognized the potential of artificial intelligence-based methods employing real-world data for diabetes modelling. These methods are seen as instrumental in elucidating the heterogeneous nature of the disease, supporting clinician decision-making and holding promises for HTA purposes. Implementation of these methods in practice requires access to databases, technical and programming expertise, and clear guidelines on real-world data usage in HTA models. Furthermore, the issue of transferability and generalizability of these methods warrants further exploration.

Author contributions

KT, ZK, SK, BN, GP participated in design, methodology, analysis, writing and approving the manuscript, GG, FS, JT, MH participated in development methods, application of methods, analysis, writing and approving, MD, MK, ZM, ZP, TT, MP, OP, IL, AS, AlS, MM, RH, PB, MS, TD participated in interview, focus group, interpretation, final approval of manuscript. All authors read and approve the final version.

Consent form

All participants in the study agreed to be interviewed and participate as co-authors.

Disclosure statement

The authors reported no potential conflicts of interest.

Data availability

All data from the interview and focus group discussion are published in the manuscript. There are no additional data available.

Additional information

Funding

The HTx project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement no. 825162. This dissemination reflects only the authors’ view, and the commission is not responsible for any use that may be made of the information it contains. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

This work was funded by the HORIZON EUROPE European Research Council.