Feasibility of Integrating a Mobile Decision-Support App into a Multicomponent CME Initiative: Developing Clinician Competence at the Point of Care

ABSTRACT

Mobile health (mHealth) technologies such as smartphone applications are increasingly being adopted in the healthcare setting to support the delivery of evidence-based care. Given the approaching ubiquity of mHealth tools in medical practice, it is incumbent on the continuing medical education (CME) community to understand how these tools can be leveraged to develop clinician knowledge and competence, and how we can assess these educational outcomes. In this report, we describe our experience developing and incorporating a mobile decision-support tool into multiple activity formats within the European Immuno-Oncology Clinic Companion CME initiative.

KEYWORDS:

• CME
• CPD
• medical education
• outcomes
• mobile health
• mHealth

Introduction

Mobile health (mHealth) technologies such as smartphone applications (apps) are increasingly being adopted to support clinical practice. In oncology, a recent review of mHealth tools found 794 oncology-specific apps, with nearly half (45%) designed for the healthcare professional audience (versus patients) with the intended functions of professional education (37%) or clinical decision support (19%)[1]. In 2019, 77% of oncology professionals reported using mHealth apps to inform or support their daily clinical practice, including tasks such as reading test results (46%), monitoring treatment side effects (37%), and managing the risk of future adverse events (29%)[2]. Given the approaching ubiquity of mHealth tools in medical practice, it is incumbent on the continuing medical education (CME) community to understand how these tools can be leveraged to develop clinician knowledge and competence, and how we can assess these educational outcomes.

In this report, we describe our experience developing and incorporating a mobile decision-support tool into multiple activity formats within the European Immuno-Oncology Clinic Companion (EIOCC) CME initiative.

Programme Rationale and Educational Design

In developing the EIOCC CME initiative, Siyemi Learning assessed knowledge and practice gaps among European oncology professionals involved in the care of patients undergoing treatment with immuno-oncology (IO) agents. One of the major barriers to integrating these novel therapies into practice, particularly in the first few years following their approval, was the lack of guidance on the management of immune-related adverse events (irAEs). Between July 2017 and February 2018, however, four leading international oncology societies published evidence-based guidelines on the management of irAEs, including the European Society for Medical Oncology (ESMO), the Society for Immunotherapy of Cancer (SITC), and the American Society of Clinical Oncology (ASCO) in collaboration with the National Comprehensive Cancer Network (NCCN) [3–6]. While the availability of multiple new guidelines presented an opportunity to elevate the standards of care around IO therapy, oncologists remained challenged to update their knowledge and competence to keep pace with this rapidly evolving field.

Straightforward clinical pathways, when used at the point of care, are effective tools to support evidence-based decision-making and improve patient-provider communication regarding complex treatment options[7]. As part of the multicomponent EIOCC CME initiative, we developed the IO Toxicity Tool to bring simplified irAE care pathways–built from detailed algorithms on toxicity monitoring and management–into the oncology practice setting, thereby supporting clinician competence.

IO Toxicity Tool Development

Under the direction of EIOCC faculty, Siyemi Learning collaborated with the ONCOassist (https://oncoassist.com) medical oncology team to develop the custom irAE management algorithms based on the ESMO, SITC, and ASCO/NCCN guidelines [3–6]. The algorithms included a range of options (e.g., starting doses for corticosteroid therapy, duration of steroid tapering) to reflect the spectrum of recommendations within the source guidelines and to support individualisation of care. In rare scenarios where guidelines offered conflicting recommendations, the tool deferred to the ESMO guidance and noted this preference.

The IO Toxicity Tool launched at the 2018 ASCO Annual Meeting, joining a suite of existing oncology decision-support tools embedded within the ONCOassist mobile app. To use the tool, clinicians select the toxicity of interest (e.g., colitis, pneumonitis) and the grade of the event, per the Common Terminology Criteria for Adverse Events (CTCAE) version 4.0, to access the grade-specific care pathway for each irAE. Additional tool features include options to save specific irAE algorithms as favourites, share algorithms via email, and access the full CTCAE version 4.0 criteria. Companion tools within the ONCOassist app, which may drive clinician engagement with the IO Toxicity Tool, include tumour staging criteria, tumour-specific prognostic scores, response criteria, and calculators for predicting the survival benefit of adjuvant therapy. ONCOassist is the only point-of-care app classified as a medical device and CE-approved for use in European Economic Area countries to aid in clinical decisions.

To maximise its utility as a clinical resource, the IO Toxicity Tool was made freely available to all ONCOassist users; clinicians were not required to register or engage with EIOCC to access the tool. As such, users were not able to claim CME credit for engagement with the tool itself. Instead, clinicians were invited to participate in CME activities hosted on the EIOCC platform (accessible via the IO Toxicity Tool info screen) to build competence and confidence in using the tool as a decision-support resource. However, because we developed the IO Toxicity tool de novo as a component of the CME initiative, Siyemi Learning sought guidance from the Accreditation Council for Continuing Medical Education (ACCME) to ensure compliance with relevant standards at each stage of the user experience. In addition, we conducted a focus group with clinician learners to confirm the ease of use and successful integration of the IO Toxicity Tool with the EIOCC platform and its online courses.

Tool Integration into the CME Initiative

We developed patient case scenarios to guide learners through complex treatment decisions during IO therapy, including irAE evaluation and management. Where appropriate, we incorporated screenshots of the relevant IO Toxicity Tool algorithms into the case discussions to support the rationale for each treatment choice (Figure 1). This unobtrusive method kept the focus on the treatment algorithms while reinforcing the tool as an available clinical resource.

Figure 1. Example IO toxicity tool irAE management algorithm within an online case-based CME module

Using this case-based approach, we integrated guideline-based irAE treatment algorithms from the IO Toxicity Tool into three separate CME activity formats:

• Online case-challenge modules hosted on the EIOCC platform

• A live satellite symposium held at the 2019 European Association of Urology annual congress in Madrid, Spain (March 2019)

• Live small-group audit-feedback discussions with oncology professionals in Milan, Italy (January 2018); London, UK (April 2019); and Barcelona, Spain (October 2019)

Results

From June 2018 through December 2019, 10,866 unique users engaged with the IO Toxicity Tool across 23,291 user sessions, totalling 74,325 screen views. Reach was truly global, with users from 153 countries accessing the irAE guideline recommendations (Figure 2). The top 10 countries of engagement, measured by total user sessions, were the USA (n = 2,577), India (n = 2,004), the UK (n = 1,994), Italy (n = 1,697), Spain (n = 1,399), Ireland (n = 901), Brazil (n = 877), Australia (n = 759), Portugal (n = 636), and Argentina (n = 557).

Figure 2. Top 10 countries of IO toxicity tool engagement by total user sessions (N = 23,291)

The 5 most frequently accessed treatment algorithms were those for immune-related colitis (n = 16,341), pneumonitis (n = 13,839), hepatitis (n = 13,786), arthritis/arthralgia (n = 8,259), and rash (n = 7,960), suggesting that these irAEs are of greatest interest to clinicians who care for patients undergoing treatment with IO therapies (Figure 3).

Figure 3. Top 5 most commonly accessed toxicity-management algorithms by total screen views (N = 74,325)

Engagement with some of the IO Toxicity Tool features was surprisingly low. Clinicians accessed the CTCAE criteria 1,043 times, accounting for <1.5% of all tool interactions. The share and favourite features were used 191 and 147 times, respectively. Clinicians accessed the info screen, which described the methods for irAE management algorithm development and linked to the EIOCC CME portal, 690 times.

Discussion

According to the ACCME analysis of CME activities developed by ACCME-accredited providers, live and enduring internet-based CME accounted for 45% of 17.3 million physician interactions in 2019, and 70% of 19.7 million interactions with other learners [8]. Examples of the overlap between online CME and mHealth tools abound, beginning with the use of mobile apps to deliver CME content and extending to CME-certified app-based case simulations. For the multicomponent EIOCC CME initiative, we developed a discrete mobile decision-support tool for the broad oncology community, supplemented by live and online CME-certified activities designed to model the appropriate use of the tool within patient case scenarios.

With nearly 75,000 screen views, the extensive use of the EIOCC IO Toxicity tool provided valuable insights into clinical practice behaviour and learner needs. For instance, a recent meta-analysis of irAEs found that pneumonitis occurred with the lowest prevalence (4.6%) of six toxicities evaluated, including colitis (14.5%) and hepatitis (10.4%)[9]. By comparison, based on the proportion of total screen views, we observed that guidelines for pneumonitis were the second-most frequently accessed (19%), following colitis (22%) and similar to hepatitis (19%); algorithms for arthritis/arthralgia (11%) and rash (11%) were much less commonly accessed. This suggests that although pneumonitis occurs at a lower prevalence relative to other irAEs, clinicians may require extra support in the real-world management of patients who develop pneumonitis during IO therapy.

Beyond these inferences, the EIOCC CME initiative was not designed to further characterise the educational impact of the IO Toxicity Tool. For future iterations of the tool, metrics that may yield additional qualitative insights into clinician competence include user behaviour by job role (i.e., physician, nurse, pharmacist, and other members of the clinical team), responses to follow-up surveys, and prompts for self-reported practice change. For a quantitative assessment of the educational effectiveness of the tool, the post-activity competence gains of learners who complete two versions of case-based CME activities–differing only by the inclusion of the IO Toxicity Tool as a guide for navigating irAE management recommendations–could be compared.

Achieving a better understanding of the strengths and limitations of decision-support tools is critical for advancing the use of these resources in practice, particularly in the setting of CME programmes. In 2019, an independent research team published a qualitative study of the adoption and implementation of the ONCOassist mobile app in clinical practice among oncology professionals in Europe[10]. Users described multiple benefits of the point-of-care decision-support tool in oncology practice:

• The tool improved efficiency and performance (e.g., decreased interruptions related to the need to refer to other source materials or conduct web searches)

• The tool “sharpened their practice” (e.g., provided a backup and a safety net for clinical decision-making based on guideline recommendations)

• The mobile app offered location flexibility (e.g., helped learners overcome computer shortages; facilitated decision-making at the point of care or in meetings; and served as a resource in areas where a computer is not allowed)

The qualitative study of the ONCOassist mobile app also identified several barriers in the adoption of point-of-care decision-support tools [10]. The top three barriers were:

• Lack of electronic medical record (EMR) integration

• Lack of information completeness (i.e., the need to refer out to the source irAE guidelines for additional detail and/or discussion of supporting evidence)

• Workplace culture that is slow to embrace new technologies

Findings from mHealth research offer important lessons for the CME/CPD community as we seek to integrate new tools and technologies into our programmes. Addressing systems-based barriers such as lack of EMR integration will be critical to realising the full potential of tools designed to support competence and performance at the point of care.

In summary, our experience with the IO Toxicity Tool demonstrates the feasibility of incorporating an mHealth resource into a multicomponent CME initiative, supporting the overarching goal of developing clinician competence in guideline-based irAE care. The analysis of user behaviour validated the appetite for point-of-care decision-support tools within the oncology community and provided real-world insights regarding unmet learner needs. Future research is needed to measure the impact of mHealth apps on gains in clinician competence and performance.

Acknowledgments

We gratefully acknowledge Eoin O’Carroll and Kevin Bambury of ONCOassist, Kerry, Ireland, for their technical expertise in the development of the IO Toxicity Tool.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by independent medical educational grants from AstraZeneca, Bristol-Myers Squibb, MSD, and Pfizer.