Distinguishing features in the assessment of mHealth apps

ABSTRACT

Introduction: The unparalleled surge in digital health adoption during the COVID-19 pandemic has emphasized the potential of mHealth apps. However, the quality of available evidence is generally low, and regulatory frameworks have focused on apps with medical purposes only, overlooking apps with significant interactions with patients that may require stronger oversight.

Areas covered: To support this expanded evidence generation process, we identified the reasons that distinguish mHealth apps compared to medical devices at large and that should differentially feature their assessment. mHealth apps are characterized by the iterative nature of the corresponding interventions, frequent user interactions with a non-linear relationship between technology usage, engagement and outcomes, significant organizational implications, as well as challenges associated with genericization, their broad diagnostic potential, and price setting.

Expert Opinion: The renewed reliance experienced during the pandemic and the unprecedented injection of resources through recovery instruments can further boost the development of apps. Only robust evidence of the benefits of mHealth apps will persuade health-care professionals and beneficiaries to systematically deploy them. Regulatory bodies will need to question their current approaches by adopting comprehensive evaluation processes that adequately consider the specific features of mHealth apps.

KEYWORDS:

• Assessment
• HTA
• mHealth
• mobile apps
• regulation

1. Introduction

Mobile health (mHealth) apps are defined as software applications that run on mobile platforms, such as smartphones and tablets, and are used to manage health and wellness [1]. With over 325,000 units already available on commercial platforms as of 2017 [2], mHealth today represents a mature market for apps and is projected to grow further, up to a value above 60 billion U.S. dollars by 2021, with a threefold increase from 2017 [3]. The COVID-19 pandemic has undoubtedly accelerated this development process, with a 65% growth in mHealth app downloads during 2020 globally [4].

That apps may contribute to improve patients’ health and quality of life, streamline the delivery process of health-care services and reduce the siloed approach based on service categories has been reported in the literature [5–7]. However, the quality of available evidence is generally low, and, above all, there is a lack of clarity as to when and how it should be generated and assessed by regulatory and Health Technology Assessment (HTA) bodies.

2. Overview of the regulatory landscape for mHealth apps

A significant regulatory challenge relates to the extreme diversity of apps in terms of main functions and potential risks to users. Just like the US Food and Drug Administration (FDA), the EU Medical Device Regulation (MDR) has thus far focused on the subset of mHealth apps that can be classified as medical devices, i.e. when they are either intended to be used as an accessory to a regulated medical device, or transform a mobile platform into a regulated device [1].

Consistently with risk-based classification regulatory systems [8], the FDA had initially focused its oversight on mobile medical apps that present high risks to patients, exercising enforcement discretion for products that help patients self-manage their conditions or automate simple tasks [1]. With the 2017 Digital Health Innovation Action Plan [9], the FDA reimagined its regulatory approach by developing a new program that should replace the need for a premarket submission for certain apps through the pre-certification of digital health developers rather than products [10]. The first step of this firm-based approach is a pilot program that has so far involved nine companies, including Apple and Fitbit [11].

At the European level, in view of the implementation of the new MDR and In-Vitro Diagnostics Regulation (IVDR) in May 2021, the Medical Device Coordination Group (MDCG) has issued guidance concerning evidential requirements for Medical Device Software (MDSW), focusing on software (including apps) with medical purposes only [12,13]. According to this guidance, evidence generation must be consistent with the level of risk of MDSW and must follow the same rules applicable to any other medical devices or in vitro diagnostics, as set out in the MDR and IVDR.

However, as the MDCG’s guidelines mostly refer to the work done by the International Medical Device Regulatory Forum, the mobile medical apps actually covered by the guidance, except for limited cases, are not meant to provide direct contact with patients and are instead aimed to inform or drive clinical management and the diagnostic or therapeutic process [14]. In contrast, other available apps are intended to elicit behavior change, enhance adherence with treatment, deliver disease-related education or act as standalone digital therapeutics [15], all functionalities that need a significant interaction with patients and highlight the challenges to keep up with an evolving and diversified spectrum of apps by the current regulation [16]. It is having also and especially these potentially non-medical mHealth apps in mind that an expanded review policy could be implemented by regulatory bodies [11]. To support this expanded evidence generation process, we aim at clarifying the distinguishing features of mHealth apps and the associated challenges that would be encountered in their assessment.

3. What does make apps assessment more challenging?

Drummond et al. described six reasons that mark a difference between drugs and medical devices when assessment is at stake [17]. These equally apply, and are often emphasized, in the case of mHealth apps indicating the need for a distinction between the assessment process of medical devices at large and that to be implemented for apps.

First, just as it happens for some medical devices, evidence generation through randomized controlled studies (RCTs) is extremely challenging for mHealth apps. The incremental nature of technological development of MDs makes RCTs, in some cases, unsuitable to demonstrate efficacy, and therefore alternative or additional types of studies have been suggested (e.g. real-world evidence) [8]. Experimental studies are even more widely inapt for mHealth apps, that can evolve very rapidly, incorporate continuous measurement of the intervention effects, and need to be constantly readjusted just to keep up with their intended use [18]. The adaptive nature of apps can go as far as to the possibility of tailoring the intervention components to individual end-users’ characteristics and personalizing them in real time [19]. This perpetual beta state of mobile software is not only an inherent feature but also a requirement to adapt to the changing contextual circumstances which may at any time reverse the effectiveness of the intervention. Moreover, apps typically deliver complex interventions in terms of components, outcomes, and causal pathways [20,21]. For instance, apps containing gamification elements were found to include a median of 14 different behavior change techniques (BCTs) [22], which interact among them and may evolve as a result of technology or scientific advancement. Each intervention may go as far as changing the behaviors of other family members other than the initial target only, and the impact on outcomes may be influenced by several different mediators and moderators. As such, the typical positivistic assumptions underpinning RCT study designs, based on a supposed causal relationship between the intervention and the outcomes of interest, are challenged in the context of app evaluations [23] by the host of mechanisms of change attainable through mHealth apps inherently linked to the social environment [21]. New features that are more suitable for digital technologies and possibly provide a better understanding of within-subject differences have been proposed, including model-based designs of adaptive research, the use of factorial designs to assess the effectiveness of single intervention components, optimization strategies borrowed from engineering or other data-intensive domains, and the use of predictive modeling [19,24,25], up to additional solutions to increase the efficiency of traditional RCTs [26]. Nonetheless, the overwhelming majority of the few peer-reviewed studies are still adopting the RCT as the preferred design for the evaluation of mHealth apps [27].

Second, as for some medical devices and as noted in the previous paragraph, mHealth apps’ efficacy depends not only on the product itself but also on how it is used. While for medical devices (e.g. implantable medical devices) these user characteristics mainly refer to the health-care professionals’ learning curve in administering them, in the case of apps the interventions are often directly managed by patients themselves. From an ‘embodied technology’ in the case of drugs [17], we have now moved toward a fully ‘interactive technology,’ where not only different individuals may develop different forms of interplay with the app, but the relationship between technology usage, engagement and outcomes is often non-linear [24]. This entails that effective engagement can only be defined individually and that the same level of utilization (in terms of access metrics) may generate different outcomes according to individual characteristics, target groups, and interventions. Whilst it is beyond dispute that the individual level of engagement with the app can produce decisive effects on outcomes, this makes the evidence generation process even more complex. On top of generating the right level of engagement individually, the challenge with apps is to sustain it longitudinally. Retention is to mHealth apps as adherence is to chronic pharmacological treatments, and is indeed one of the most significant issues for any eHealth intervention [28], as only small portions of users show consistent access to mHealth apps over time [29]. To foster retention and leverage in this revamped patient centricity, user-centered designs must be embraced to anticipate users’ needs in the development phase and make sure finalized interventions are responsive to stated preferences. Despite several alternative design methods of mHealth apps have been proposed [30], only positioning the patient as a constant partner in the development process of mHealth apps could offer insights that may substantially improve the health-care outcomes for patients and providers [31].

Third, app implementation has the potential to generate relevant organizational implications, possibly wider than any other medical device. The digital nature of mHealth apps can radically transform how the delivery process of health-care services is organized within health-care centers. Let us think about enhanced data sharing and the possibility for different types of health-care professionals in different patient care settings to receive, at the same time, information on patients’ symptoms and outcomes so as to enact integrated care protocols and services. The wide organizational implications of app adoption are usually context-specific, meaning that change management processes adopted in a single institution can seldom be transferred to another and all analyses should be performed considering the specific organizational and cultural backgrounds [32]. In this respect, Bluestar, the first app to receive the FDA approval as a mobile prescription therapy for diabetes self-management [33] based on preliminary studies [34,35], showed no effect on glycemic control in a later pragmatic multisite implementation, highlighting the importance of contextual factors, such as site readiness and training activities, in determining app effectiveness [36].

This leads us to the fourth additional challenge. ‘Genericization’ is theoretically attractive but practically seldom possible for apps. Regulatory bodies adopt the concept of substantial equivalence to streamline the regulatory process for medical devices. When MDs present the same characteristics of a predicate (i.e. another device with the same intended use already approved), manufacturers do not need to produce any additional evidential requirement and access to market is expedited. Resorting to substantial equivalence may seem straightforward for mHealth apps, particularly given the exceptional number of legally different products that seemingly share comparable technological features and behavior change techniques (when applicable). These class effect recommendations are nonetheless hardly applicable to mHealth apps considering that: i) their success significantly depends on the organizational adjustments that are necessarily local and grounded in the implementation process; ii) it is particularly challenging to achieve a detailed specification of any mHealth app intervention and its components [18], given the continuous evolution and the possibility to tailor several components to individual needs and preferences. As such, even apps designed for the same target population with comparable functionalities and mechanisms of action may generate differential outcomes due to apparently insignificant differences in design processes, observed usability, professional acceptability, and integration with human-led components.

Compared to medical devices at large, apps may also have diagnostic potential by gauging signs and symptoms directly from lay people (i.e. patients and their caregivers). In terms of app evaluation, this produces an additional need to assess and certify these inputs’ reliability and the methods applied to collect them, which may generate valuable insights but also significant challenges by leveraging on the growing amount of available data [37]. Further evaluation implications are symmetrical to what was previously emphasized for medical devices: the need to account for the value resulting from the treatment that follows diagnosis, and to appreciate apps’ indivisibility across multiple potential applications.

Finally, prices for medical devices are more likely to fluctuate over time compared to those for drugs. In the case of apps, since advanced jurisdictions are starting to pilot prescription mechanisms just now [38], there are no consolidated procurement mechanisms, yet. Unit prices for subscription to individual apps should be negligible (think of commercially available apps’ fees as a benchmark), but to date, vendors have instead marketed their apps as an add-on service attached to a drug or a medical device. Of course, there are differences expected for standalone digital therapeutics but, if the paradigm of apps as an enhancement of currently available therapies will persist, the merging of apps with their originating technology will continue to generate extra challenges from a clinical and economic evaluation standpoint, making it difficult to predict the future trend in terms of pricing. Furthermore, price setting is challenging for mHealth apps also for reasons associated with costs. First, like most digital interventions, apps can rapidly change scale at minimal incremental cost compared to traditional technologies. Second, the cost structure of apps is different from the ones of commonly fixed products in the pharma and device industries: while research costs may be significantly lower compared to drugs and devices, the initial outflows for development, design and their multifold iterative steps are not negligible and must be accurately captured. Third, specific costs associated with app interventions include the promotional costs to push the intervention uptake to secure that ‘effective engagement’ is reached and to sustain it longitudinally [24]. However, cost-effectiveness analyses of mHealth apps have rarely been performed and the related evidence is still scant [39]. Table 1 synthesizes the six analyzed domains and the associated challenges that distinguish the assessment of mHealth apps.

Table 1. Challenges in the assessment of mHealth apps

Domain	Associated challenges
I. Evidence generation	- Adaptive nature of apps makes experimental studies inapt
	- Complexity of app-based interventions in terms of components, outcomes and causal pathways
	- Non-applicability of positivistic assumptions
II. Interaction	- Efficacy depends on the way apps are used
	- Effective engagement is strictly individual
	- Retention rates significantly decrease longitudinally
III. Organizational implications	- Change management processes are typically context-specific
	- App-related evidence has limited transferability
	- Organizational readiness and connected human-led components may influence app effectiveness
IV. Genericization	- Substantial equivalence is hardly applicable to mHealth apps
	- App interventions cannot be locked down due to their continuous evolution
	- Despite similar features and target populations, different apps may generate differential outcomes
V. Diagnostic potential	- Potential inputs by lay people and associated need to certify their reliability
	- Value of improved diagnosis needs to also account for the associated improvement in clinical outcomes
	- Need to weigh all potential diagnostic applications of any mHealth app
VI. Pricing	- No consolidated procurement mechanisms as a guideline
	- Different cost structures compared to traditional healthcare technologies
	- Plurality of possible marketing approaches (add-on service vs standalone technology)

4. Conclusion

As it was done for the comparison between medical devices and drugs [17], we have envisaged the reasons why the assessment of mHealth apps is more challenging than that of traditional medical devices.

To date, mHealth apps have not lived up to the same evidence standards in place for drugs and devices. This is partially attributable to the inability to appreciate their distinctive features that have significant implications for clinical and economic evaluations. While early-stage evidence generation during the product development process may be enough to convince ‘early adopters’ [25], only robust evidence of benefits will persuade the majority of potential beneficiaries, as well as payors, to adopt mHealth app systematically.

Despite many have supported the case for digital exceptionalism [40], this should not turn into a generalized amendment for mHealth apps, whose future deployment cannot be justified through social enthusiasm, economic circumstances, and political expediency only but should anyway be supported by robust evidence [41].

5. Expert opinion

Europe is the jurisdiction with the highest number of mHealth apps manufacturers (34%) globally [42], with a great proportion of these apps likely to be characterized by a significant interaction with patients. As proof of this, the mobile health categories that were thought to carry the best global market potential 5 years ago were indeed remote monitoring (32%), diagnostic apps (31%), and medical condition management (30%) [43]. Furthermore, the COVID-19 pandemic has laid bare the inadequacies of analog health-care systems and showcased the benefits of a digital health.

Accordingly, ‘digitilisation’ is among the top priorities of Next Generation EU [44], the €750 billion recovery instrument agreed on at the European level to support the recovery and make Europe, among others, more digital, potentially boosting the development of mHealth and apps further. Many EU countries will thus receive an unprecedented injection of resources to invest in a better society and economy for future generations. Health will undeniably be at the heart of these changes, as COVID-19 has clearly shown that without a healthy population we cannot achieve a wealthy society.

To make the most of this potential for mHealth apps, a shared effort is necessary. Regulatory and HTA bodies would need to expand their current regulation of health apps and consider their specific features in their guidance to avoid that assessment processes and procedures developed with traditional MDs in mind are applied tout court to new, fast evolving products. If inadequately regulated, the future deployment of mHealth apps can be threatened either way: too few will access the market (e.g. if RCTs will become the rule to generate evidence) thus diluting the huge potentials to increase patients’ outcomes and (sometimes) reduce health-care costs; or too many (e.g. if blurred boundaries persist between what must and must not be regulated) will flood the market with unchecked effectiveness on patients’ health and quality of life, and poor retention rates darkening the future of this high-potential industry.

In regards to the evaluation for HTA purposes, the National Institute for Health and Care Excellence in the UK has recently adopted a more comprehensive approach and defined effectiveness and economic impact standards for all Digital Health Technologies (DHTs), by stratifying them into evidence tiers based on their main clinical function and corresponding risk levels [45]. This and other experiences can be streamlined by the European network of HTA agencies (EUnetHTA), especially in view of the new EU HTA regulation that will likely come into force this year.

In this scenario, decision makers and regulators should also be supported by non-governmental actors, that could foster emerging industry standards and rigorously review the numerous rating scales and frameworks that are the subject of a vivid scientific debate [46].

The future of mHealth apps will also depend on how, besides the regulatory and HTA bodies, health-care providers and professionals will respond to procurement and organizational implications of their utilization. Once market authorization has been granted and efficacy has been checked, mHealth apps would need to be incorporated into routine practice to deliver their full potential and this can happen only under certain conditions. First, new models of organizing the production and delivery of health-care services must be developed. mHealth has the potential to bridge the silos and connect different settings of care, substantially contributing to an integrated, cross-cutting care approach. Second, conventional procurement activities must develop into value-based, innovative practices aimed at assessing the added value of mHealth apps, especially when it is entangled in another technology. From the above, it follows that the conventional fee-for-service reimbursement schemes poorly capture the value of mHealth and other payment structures that offer greater flexibility need to be developed.

With more smartphones (8.3 billion subscriptions in 2019) than people on the planet (7.7 billion in 2019) [47], mHealth will definitely represent the main driver of the digital health revolution over the next years. What is less certain is the time needed for health-care systems to adapt to a new combination of analog and mobile services: this will mainly depend upon the degree of flexibility provided by health-care organizations and health-care professionals. No matter how we reach this new equilibrium, it is fundamental that none is left behind by the digital revolution as it happened with analog systems for several youth and adult chronic patients during the COVID-19 pandemic [48–50]. If mHealth could eliminate geographical distances, it can, at the same time, marginalize deprived social classes if left unregulated [51].

A lot is at stake in the case of mHealth apps. Decision makers and regulatory bodies should extend their review beyond apps intended for medical use only. Adequate evaluation processes that take into account the inherent specificities of this class of health-care technologies will enhance the likelihood that the full potential of mHealth apps can be realized in the foreseeable future.

Article highlights

• mHealth apps are extensively available on commercial platforms, but are often not supported by adequate clinical and economic evidence

• Just like in the US, the regulatory guidance issued at the European level has focused on apps with explicit medical purposes only, neglecting many apparently lower-risk apps that have significant interactions with patients and may cause harm to users if not properly regulated

• Expanded evidence generation is needed to support the consolidation of mHealth apps in the healthcare realm and should account for their distinguishing features compared to other healthcare technologies

• In the case of mHealth apps, the characteristics that differentiate medical devices from drugs are often emphasized and raise additional challenges for their evaluation

• Apps typically deliver complex interventions because of their adaptive nature, the coexistence of several potential mechanisms of change, the patient centricity in the development and delivery of the technology, as well as the context-specific implications that limit the transferability of observed results

Declaration of interest

All authors are involved in an RCT to evaluate a mobile supportive care app for patients with metastatic lung cancer. The authors have no other relevant affiliations or financial involvement with any organization or entity with a financial interest in or financial conflict with the subject matter or materials discussed in the manuscript apart from those disclosed.

Reviewers disclosure

Peer reviewers on this manuscript have no relevant financial relationships or otherwise to disclose.

Additional information

Funding

This project received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement #779306 (COMED—Pushing the Boundaries of Cost and Outcome Analysis of Medical Technologies). The results reflect only the authors’ views, and the EU is not responsible for any use that may be made of the information it contains.