Digital health; what do we mean by clinical validation?

1

KEYWORDS:

• Clinical validation
• digital health
• evidence
• Healthtech
• mHealth
• technology

1. Introduction

Innovation is born from significant events and new technological advances, with the greatest arising from a combination of the two; more recently, this has been demonstrated with the COVID-19 pandemic and the digital revolution in healthcare. Telehealth has been around for some time, however due to poor existing infrastructure, adoption is usually slow to be implemented in healthcare provider services [1]. Since 2019, digital health has seen an explosion in uptake and integration – more people than ever are beginning to support telemedicine, remote/virtual care, as well as the implementation of ML/AI techniques in real time data analytics [2]. A race has begun amongst players within these existing and evolving fields of digital health – but how can we ensure this innovation is more than just a bubble waiting to burst? In the past, digital health technologies had the support of traditionally provided healthcare and was more of an adjunct until COVID-19 required them to be the sole method of healthcare delivery. The promise of revolutionary technologies, and the benefit they can bring, must be weighed against proper evaluation to ensure gold standard treatment that is safe, efficient, and effective. However, thorough evaluation should not come at the detriment to progression, a balance must be struck to ensure we promote innovation with safe and progressive care.

2. What is digital health?

Digital health is a broad term that encompasses the field of health services delivered using technologies for measurement and intervention. These include, but are not limited to, mobile health (mHealth), health information technology (IT), wearables (biosensors to smart watches), machine learning algorithms, as well as telehealth. It is important to accurately define the digital health tool in question as this is pertinent to the level of clinical validation needed, e.g. a mobile app that promotes healthy diets will require less evidence for approval than a complex machine learning algorithm used in helping diagnose specific types of cancer. This differentiation often dictates whether the digital health tool falls under the category of wellness and therefore does not require medical device classification all way through to class 3 medical devices that often need to demonstrate rigorous clinical validation before regulatory approval and CE marking. Digital health tools have increased exponentially since 2020, largely in part due to COVID-19 [3] and as a result, global governing bodies along with institutions like The Digital Medicine Society and the Digital Therapeutics Alliance are aligning on key definitions and universal terms to help the definition process. Although initiatives such as hospital at home have aimed to reduce length of stay and move some acute care to the community, the necessity to reduce exposure to COVID-19 in hospital led to the rapid drive in virtual wards and remote patient care using digital tools. Where the digital hospital once revolved around electronic notes and digital imaging, we now have complex integrated health systems that link primary and secondary service providers and are starting to venture into ML-based predictive analytics for disease and oncological identification [4].

3. Verification and validation

Where pharmaceuticals undergo intense scrutiny through several phases of clinical trial and peer review, often followed by postmarket surveillance, digital health products often adopt an ‘implement now, clinically validate later’ ethos. However, these technologies have the potential to impact and harm us on the same, if not a much larger scale. So how do we apply the same level of rigor to digital products? Understanding the safety, effectiveness, and efficacy of these products needs to become unified globally to ensure that the technologies we use reflect the patient and healthcare provider’s needs, safely.

Taking the approach of a ‘fit-for-purpose’ assessment of biometric monitoring technologies (BioMeTs), digital health product evaluation could be broken down into verification, analytical validation, and clinical validation [5]. The verification asks the question, ‘does your digital health product meet its intended purpose?.’ Validation asks ‘does your digital health product accurately and reliably generate the intended output and is this output clinically meaningful in the defined condition?.’ Usually, verification is carried out by the engineering team whilst analytical validation is a combination of both the clinical and engineering team. Understanding that these are separate to clinical validation is key to ensuring the appropriate research methodology are performed. Table 1 illustrates the definitions as take from the FDA’s guidelines on software as a medical device (SaMD).

Table 1. FDA guidance on definitions when validating SaMD [6]

Clinical validation can be defined as demonstrating the clinical safety, effectiveness, and efficacy. Safety illustrates that the product is with minimal chance for error or adverse effect to the patient. Effectiveness reflects how reproducible the product is in delivering benefit in the real world, whilst efficacy captures the degree to which the product performs its role well. Where there is a clear global consensus on how to evaluate these three areas in traditional healthcare products such as pharmaceutical compounds and implantable devices, this is less clear in novel aspects of digital health e.g. mobile health apps (mHealth).

4. Generating the evidence

Digital health technologies such as software as a medical device (SaMD) do not fit easily within the standard evaluative techniques of traditional therapeutics. As a result, most technologies manage to avoid double-blind multicenter RCT to establish safety, effectiveness, and efficacy, as this would be too time consuming and expensive to conduct, with equally good methods of clinical validation available. So how therefore, can we evaluate the use of tools such as complex integrated digital systems, remote monitoring devices, and other tools? Whilst regulatory approval and device classification processes are established, often the guidance surrounding the level and quality of clinical validation required is gray. Health apps have generally avoided SaMD classification by defining themselves as ‘wellness’ applications. As a result the FDA have issued clarification guidance as to what defines a wellness application vs SaMD [7]. Under the Medical Device Directive (MDD), many have sat in a self-certified Class 1 category in the United Kingdom. Where the Medical Device Regulation (MDR) in the EU aims to address some of the limitations, this is not without its own problems which are out of scope of this discussion.

Whilst there are some frameworks for clinically validating digital health products, only a handful have been submitted for peer review or been taken on by regulatory bodies [8]. Digital health encompasses such a vast array of technologies that a single approach to validation can be challenging – specific evaluative techniques must be laid out to validate these tools properly and pragmatically. Attempts at this can be seen by groups such as ISPOR who provide guidelines in the clinical validation of electronic patient recorded outcome measures (ePROMs) [9]. Some non-governmental organizations such as ORCHA, who work closely with regulatory bodies and implementors, help companies achieve ‘validation,’ branding them with the widely accepted ORCHA seal of approval [10]. However, the rise of organizations such as ORCHA highlights that companies are requiring help in understanding and navigating this complex and misaligned system of validation.

5. Software as a medical device (SaMD) global approaches to evaluation

Some geographies have started to address this problem directly. The National Institute of Health and Care Excellence (NICE), a UK based organization, now provides one of the most comprehensive guidelines in validation of SaMD – the evidence standard framework (ESF) [11]. NICE propose a tier-based approach (Figure 1) to help users identify what level of evidence a digital health tool requires. The criteria for approval range from clinical effectiveness, regulatory approval, clinical safety, privacy & confidentiality, security, usability & accessibility, interoperability, technical stability, and change management.

Figure 1. Tier system implemented by NICE [11].

Evidence requirements become more significant with potential clinical risk, thus tier 1 is the least rigorous and tier 3b, whose clinical risk is greatest, has a more substantial specification. Focusing on clinical validity, tiers 1 and 2 can be supported by feasibility studies or in some instances live pilots, with outcome data and end-user satisfaction being the basis of evidence required. Tier 3a demands the need for a robust observational or quasi-experimental research to demonstrate efficacy, effectiveness, and safety. These studies can be prospective or retrospective, either on existing data sets or on patients in real time. Tier 3b advises an interventional study directly comparing the intervention to a control (note this does not have to be the ‘gold standard’ double blinded randomized studies we often associate with organic compounds).

In the United States (US), the Food and Drug Agency (FDA) has identified a need for a more streamlined and efficient regulatory model and as such, has set up The Software Pre-Certification (Pre-Cert) Pilot Programme. As part of its Digital Health Innovation Action Plan, the Pre-Cert programme will inform this future regulatory model. This approach aims to look first at the manufacturers/developers of the digital health tool rather than the tool itself [9]. Their guidance on regulatory submission of software as a medical device (SaMD) helps developers understand the difference between analytical validation and clinical validation (as well as some of the ways in which collected data could demonstrate validity. However, the FDA falls short of advising on the types of study that could be used to gather this data. In an effort to improve guidance, the FDA launched their center of excellence for digital health, focusing on delivering high quality digital health products with innovative regulatory process for these bespoke solutions [12]. Other organizations such as the National Evaluation System of Health Technology (NESTcc) work closely with the FDA to develop a voluntary network of organizations aiming to create a resource bank of real-world data (RWD), to inform medical device evaluations and support regulatory decision making for medical devices including mobile health (mHealth) solutions [13]. The Medical Device Innovation Consortium (MDIC) have begun the process of creating workstreams leveraging key opinion leaders and subject matter experts to complement the FDA’s efforts to develop an innovative regulatory pathway for digital health products [14]. Countries like Germany have revolutionized their reimbursement pathways to ensure digital health technologies are readily accessible for both providers and patients [15].

6. The future of digital health evaluation

The market for digital health products is growing and many are already being integrated into care pathways [1,2]. A clear, pragmatic, and unified approach must be taken in the validation of these technologies. The spectrum of technologies that digital health encompasses means it is intrinsically challenging to categorize into niche category e.g. tiers according to the ESF. However, common themes are present and there is significant overlap in the expectation of evidence required in evaluation. Approaches such as the Digital Health Scorecard capture these themes over four domains including cost, usability, clinical, and technical to provide a global digital health score. Regarding pragmatism, there needs to be a consensus on levels of evidence required. Pilot and feasibility studies are great ways of getting a minimum viable product to market quickly and demonstrating initial clinical validity. However developers need to ensure that in their pipeline, continued robust clinical research is conducted in order to demonstrate true clinical validation. Real world evidence studies can and should be used for this. Finally, manufacturers/developers, regulators, and healthcare bodies must work together to promote a supportive environment for evaluation. Validation is not a one off study but a continuous process through the product lifecycle. Thus, open dialogue in this rapidly evolving field is integral to maintaining current and relevant standards.

Declaration of interest

S.S. Shah is an employee of GlaxoSmithKline. A. Gvozdanovic is and employee of Deloitte Touche Tohmatsu Ltd. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Reviewer disclosures

A reviewer on this manuscript has disclosed that they are an advisor for the Digital Therapeutics Alliance. Peer reviewers on this manuscript have no other relevant financial relationships or otherwise to disclose.

Additional information

Funding

This paper was not funded.