Adverse event monitoring in mHealth for psychosis interventions provides an important opportunity for learning

Introduction

Mobile health technology, also known as mHealth, makes potentially empowering health interventions more widely available. One area of focus across health conditions has been in relation to the remote measurement of changes in wellbeing using mobile technology (Simblett et al., 2018). Remote monitoring systems have been developed and tested for people with experiences of psychosis (Marzano et al., 2015; O’Hanlon et al., 2016), where the anticipation and prevention of relapse has long been a treatment and policy goal and, while it is too early to draw firm conclusions on effectiveness, acceptability and feasibility have been found to be consistently high (Naslund, Marsch, McHugo, & Bartels, 2015). While we have had evidence for some time that it is possible to identify early signs of psychosis, acting upon those signs in a way which is supportive and avoids the need for coercive treatment decisions is harder to achieve (Morriss, Vinjamuri, Faizal, Bolton, & McCarthy, 2013). It is also established that early signs are accompanied by a fear of relapse (Herz & Melville, 1980) and that fear of relapse is a predictor of relapse which may block help seeking (Gumley et al., 2015).

Mobile technology provides an opportunity to overcome some of the barriers to implementation of early signs work through active (e.g. completion of self-report assessments) and sometimes passive monitoring (e.g. routinely gathered location or phone usage data) in real time and in people’s normal environments. This approach, which can also be characterised as ecological momentary assessment (EMA), has the potential to reduce the type of recall bias associated with retrospectively gathering data (Shiffman, Stone, & Hufford, 2008) and is increasingly being applied in psychosis research and clinical settings (Bell, Lim, Rossell, & Thomas, 2017). Such routine monitoring also provides data, which might support shared decision making and reduce the risk of practitioners acting conservatively when faced with ambiguity and uncertainty.

Monitoring mental health is, however, not a neutral process – measuring things generally has the potential to effect the person being measured both positively and negatively (Miles et al., 2018). There is a risk that in our enthusiasm for the potential benefits of mHealth for psychosis that we are failing to properly assess, and learn from, potential adverse effects. There have been warnings against the risk of seeing digital interventions in mental health as a panacea for long standing and complex problems (Wykes & Brown, 2016). Interventions may also be causing harm through overselling their benefits (Wykes, 2019) and as a result of being insufficiently sensitive to the often complex needs of people affected by serious mental health problems (Lipczynska, 2016). An increased focus on the duality of costs and benefits in digital interventions has been commented on in this journal (Guha, 2017) and the need to more rigorously review both costs and benefits of digital interventions for mental health has been highlighted elsewhere (Armontrout, Torous, Fisher, Drogin, & Gutheil, 2016; Naeem et al., 2015; Rozental et al., 2014; Torous, Nicholas, Larsen, Firth, & Christensen, 2018). If we believe that such interventions have the potential to be efficacious then we must also be prepared for the possibility that effects may also be negative (Rozental et al., 2014). Fully informed decisions on the future development or use of mHealth for psychosis interventions should be based on having detailed information available on all relative costs and benefits so it is telling that references to safety reporting and adverse events are rare in mHealth for psychosis literature.

Adverse events and possible proxies in mHealth for psychosis literature

Bell et al. (2017) systematically reviewed nine studies of Ecological Momentary Assessment (EMA) and Interventions (Ben-Zeev et al., 2014; Ben-Zeev, Wang, et al., 2016; Depp et al., 2010; Granholm, Ben-Zeev, Link, Bradshaw, & Holden, 2012; Pijnenborg et al., 2010; Sablier et al., 2012; Španiel, Vohlídka, Hrdlička, et al., 2008; Španiel, Vohlídka, Kožený, et al., 2008; Španiel et al., 2012). All featured some form of mHealth intervention designed to enhance care for people with experiences of psychosis. None of those studies described having undertaken adverse events monitoring. We identified nine further studies of mHealth for psychosis that either did not meet the inclusion criteria for the Bell et al. (2017) review or which were published subsequent to the review (Ainsworth et al., 2013; Barnett et al., 2018; Bucci et al., 2018; Eisner, Bucci, et al., 2019; Kumar et al., 2018; Meyer et al., 2018; Niendam et al., 2018; Palmier-Claus et al., 2012; Španiel et al., 2018) Of these just one described adverse event monitoring and in that instance it was limited to the identification of serious adverse events,¹ with none noted over the twelve-month period of the Actissist trial (Bucci et al., 2018). There was no indication of whether non-serious events were monitored or whether the relatedness of events to the digital intervention was assessed.

Although there is little evidence of systematic within-study monitoring or reporting of adverse events and experiences, a number of studies do report relevant indices of acceptability and engagement. These are worthy of investigation as in some cases data may be indicative of adverse reactions to mHealth interventions. In a study of the FOCUS intervention, increased paranoia was cited as a reason for non-engagement by at least two people (Ben-Zeev, Scherer, et al., 2016). Palmier-Claus et al. (2012) included three questions to assess the safety of Clintouch monitoring, but data were only reviewed once people had stopped using the system. Participants with the most acute symptoms were described as being the most “reactive” to questions in the App although it was not possible to tell whether those reactions were positive or negative due to the wording of the questions. Ainsworth et al. (2013) compared text and App-based delivery of the Clintouch questionnaire. At the end of the study, participants were asked to rate how stressful and challenging they found the monitoring with some reporting mild negative effects. One participant also asked for text messaging to be terminated early because the questions led to unhelpful rumination. Meyer and colleagues asked 14 participants, who were using a combination of wearable and smartphone devices capturing rest and activity data, to complete a post-study usability assessment (Meyer et al., 2018). Through this, concerns were expressed about the potential for the intervention to generate false alarms, discomfort from the wearable device and frustration at the repetitive nature of the integrated Clintouch symptom tracking component. In qualitative interviews, completed with people who had used the EXPRESS App to measure early signs and basic symptoms of psychosis, two people expressed concerns about feelings of paranoia and two mentioned concerns about potentially punitive service responses to their App data (Eisner, Bucci, et al., 2019; Eisner, Drake, et al., 2019).

A number of studies described relatively high levels of non-engagement and drop out, which may be an important opportunity to investigate adverse events or effects (Rozental et al., 2014). It may be reasonable to hypothesise that in some instances negative experiences of interventions may have played a role in determining non-engagement. It is also notable that regardless of problems with engagement or wider implementation challenges, interventions are almost universally described as being acceptable and feasible. For example, in a small study of passive monitoring of behavioural indicators 20% of participants were described as being upset by the approach when acceptability was assessed at the end of the study (Ben-Zeev, Wang, et al., 2016). Despite this feedback and the absence of routine adverse events monitoring the intervention was still found to be feasible and acceptable. A recent review of engagement with Apps for people with diagnoses of depression, anxiety, schizophrenia or bipolar disorder found that no two studies, out of the 40 included, used the same means to assess engagement (Ng, Firth, Minen, & Torous, 2019). In addition, all 40 reported positive results for engagement or feasibility, including 15 studies where there was no objective measurement of engagement. Ng et al. (2019) proposed that the observed lack of consistency in describing, measuring and reporting engagement could be masking significant problems with usability and safety.

In summary, this brief review of mHealth for psychosis literature suggests that the monitoring and reporting of adverse events or effects is largely neglected. While many of the included papers described small scale pilot or feasibility studies it is notable that just one described and reported adverse event monitoring procedures, and in that instance monitoring was limited to serious adverse events with no reference to the relatedness of events to the digital intervention (Bucci et al., 2018). Although studies tend to report levels of acceptability, usability and/or engagement these are poor surrogates for the direct and contemporaneous assessment of adverse or unwanted experiences. Based on our experience, we believe that incorporating procedures to routinely identify and respond to adverse events enhances our understanding of how interventions are experienced. This improves our ability to modify interventions as required and to respond appropriately to safety issues as and when they arise.

Learning from adverse events monitoring in the EMPOWER study

One driver for improved adverse event monitoring could be regulatory. In the United States, the Food and Drugs Administration has adopted a system based largely on demonstrating equivalence to already approved medical software (Armontrout et al., 2016). In the European Union member, countries have introduced new systems to assess and regulate software as Medical Devices, based on European Commission Guidance on the Qualification and Classification of standalone software as Medical Devices (MEDDEV 2.1/6). In the United Kingdom, the competent authority is the Medicines and Healthcare products Regulatory Agency (MHRA) who registered the first App as a medical device in 2013 (McCarthy, 2013). It is our understanding that EMPOWER was the first mHealth intervention for mental health to be registered and regulated as a medical device by MHRA in 2017 (CI/2017/0039). Medical device regulation requires manufacturers and investigators to demonstrate compliance with the essential requirements of the European Directive. This includes legal requirements in relation to the assessment of performance and safety, including detailed monitoring and reporting of adverse events. New medical device regulations, which include clearer guidance on software as medical devices and its classification, will come into force in Europe in May 2020. This amendment is likely to bring increased regulatory and reporting requirements for researchers and developers. For example, it has been clarified that any software involved in decisions with a diagnostic or clinical therapeutic purpose will now be automatically categorised as at least Class 2a, which brings with it the need for assessment of conformity with regulations by third party notified bodies.

EMPOWER (Early signs Monitoring to Prevent relapse in psychosis and prOmote Wellbeing, Engagement and Recovery, ISRCTN: 99559262) is a feasibility study of an mHealth intervention to enhance detection of Early Warning Signs of psychosis and prevent relapse. In our cluster randomised controlled trial, participants receiving care from community mental health services in Scotland and Australia are randomised to receive the EMPOWER intervention or treatment as usual. The intervention involves inviting participants to self-monitor their wellbeing, including early signs of psychosis, for up to one year through a mobile phone App. Participants use the App, which integrates a degree of personalisation, for an initial twenty-eight day period to identify a personal baseline of their typical variation in wellbeing. Our stepped care approach includes the delivery of tailored messages in response to lower threshold changes in scores against that baseline and the option to check in with App users and, or, community mental health staff, where there has been a higher threshold change. The algorithm, which reviews data and determines the best response, is registered and monitored as a Class 1 medical device with MHRA. While initial medical device registration was challenging the consequent heightening and refinement of routine adverse effects monitoring and reporting has usefully allowed us to respond quickly to feedback and adapt our intervention approach in light of end user experiences.

Our adverse event procedures involve recording all untoward medical occurrences or clinical indications, their relatedness to the investigational medical device, their seriousness and intensity and whether or not the event was anticipated. We also separately monitor and report device deficiencies which are inadequacies of the medical device (the algorithm) with respect to its identity, quality, reliability, safety or performance. People using the EMPOWER App receive regular phone support from Peer Support Workers during which any negative experiences are reviewed. In addition, where a significant change in putative early signs is identified clinical staff contact participants to check in with their wellbeing and review any further actions required. Our approach is described in a detailed Standard Operating Procedure, and all staff receive training in the identification and reporting of adverse events. There are regular opportunities to discuss and reflect on positive and negative experiences of people using the App. While team members are encouraged to take all adverse events seriously and to respond appropriately, they are also trained to see any negative effects as opportunities for learning about how we might improve end user experiences and to inform the development of the intervention.

These processes have led to the identification of a number of important experiences that we have recorded as adverse events. While none have been related to the EMPOWER algorithm (i.e. the MHRA regulated medical device), a number of them are related to aspects of the digital intervention as a whole. In total, 41 people had the EMPOWER App installed between May and September 2018. Levels of usage have been variable and will be reported elsewhere in due course. In the period up to March 2019, we recorded 43 adverse events across the study as a whole. Of these, 27 were related to 17 people who were allocated to the EMPOWER arm of the study. Of this group of 17 participants, we determined that the adverse event was related to the intervention in nine instances (accounting for the excess of adverse events in the EMPOWER arm), affecting seven people using the App, which will now be described.

All but one of the nine adverse event that we deemed to be related to the EMPOWER App was categorised as non-serious. The one serious adverse event involved a hospital admission, which was described by the participant as being in part related to the installation of the EMPOWER App and an associated sense of feeling overwhelmed. The participant, who had not actually entered data into the App, then withdrew from the study after meeting with a member of the EMPOWER Team.

On two occasions, we recorded adverse events specifically related to the exposure to personalised items in the EMPOWER App. People using the App have the option to personalise a number of items to better fit with their own unique experiences. This function was included to allow for a more tailored experience and to improve the sensitivity and specificity of early signs to potential relapse. However, in these instances, we found that memories of distressing and traumatic experiences of psychosis were triggered during routine monitoring. Our response in both instances was to edit the question set to remove personalised items and consequently both participants continued in the study. Personalised warning signs may have triggered intrusive memories of distressing psychosis, which can perhaps be understood in the context of previous research showing fear of recurrence predicted traumatic memories of psychosis (White & Gumley, 2009).

Two separate adverse event reports for the same participant related to increased feelings of paranoia as a result of being prompted to respond to questions at an inconvenient time. In both instances, the participant was offered assurances by a peer support worker. This included reminding the participant of an inbuilt five-hour window to respond to questions, a feature which was included as a result of beta-testing feedback that suggested some people felt pressured by a limited response window. A heightened sense of paranoia and increased fear of recurrence were included in our protocol and medical device registration documents as anticipated risks of the intervention and heightened feelings of paranoia as a result of routine monitoring in psychosis have been reported elsewhere (Eisner, Drake, et al., 2019; Terp, Jørgensen, Laursen, Mainz, & Bjørnes, 2018).

One participant also alluded to increased fear of relapse as a result of the intervention, particularly on days when their mental health had taken a dip. Assurances were again given by a peer support worker and options for managing the App in the context of fluctuating mental health were discussed. The potential to generate fear, which is itself a strong predictor of relapse in psychosis (Gumley et al., 2015), through routine monitoring is well summed up by a participant in a recent qualitative examination of an mHealth for psychosis intervention: “Being notified of all the changes sometimes made me anxious. It made me wonder if the illness was maybe about to get out of control” (Terp et al., 2018, p. 8).

A further intervention related adverse event early in the study was associated with a participant’s experience of self-monitoring being at odds with their usual coping strategy of “burying things” and “putting on a face”. The participant was offered additional peer support and was encouraged to take a break from using the App but ultimately chose to withdraw from the study entirely. The potential for routine monitoring of signs and symptoms to encourage unhelpful rumination and pessimism for recovery has been described elsewhere (Ainsworth et al., 2013; Faurholt-Jepsen et al., 2015; Nicholas et al., 2010). This may suggest the need to assess the extent to which self-monitoring might fit or be at odds with a participant’s current means of managing their wellbeing prior to using a digital intervention. In some instances, it may be advisable to conduct preparatory work to orientate potential users to the relative costs and benefits of monitoring wellbeing. In EMPOWER, our experience is that these conversations can take place with a peer worker who can also support self-management. This need for heightened early monitoring and support is supported by the fact that in six of the nine intervention related adverse events in EMPOWER occurred within a month of the App being installed, and four of those within two weeks.

Technical issues can also lead to problems with engagement and adverse effects for mHealth intervention users (see, for example, Kumar et al., 2018). Any such intervention is contingent upon an effective combination of software (the App) and hardware (the mobile device being used) and it can be extremely hard to anticipate all potential conflicts between a piece of software and the myriad of devices it may be deployed on (even where they use the same operating system). In one instance, an unexplained conflict between the EMPOWER App and a specific handset led to notifications being sent in the night causing distress to a participant who felt compelled to respond. The participant subsequently withdrew from the study and following investigations it became clear that this problem was specific to one particular type of handset and as a result difficult to anticipate. There is also of course the possibility of misplacing research-provided handsets and one adverse event in the study related to distress and anxiety experienced as a result of a participant misplacing a study phone. They were offered reassurance and the possibility of a replacement handset.

Conclusions

A brief review of literature suggests that adverse event monitoring in mHealth for psychosis interventions is under developed with procedures being poorly reported. Heightened adverse event monitoring in the context of the EMPOWER study as a result of medical device registration has highlighted relatively frequent adverse events, when compared with existing literature in the field, which can in some way be related to an aspect of the digital intervention. We have developed procedures and practices which have facilitated the timely identification of adverse events and we also encourage a culture whereby such events are seen as an opportunity to refine and improve the intervention. Whenever possible, we adapt the intervention to better meet users’ needs.

The philosophy underpinning EMPOWER is to enable and support individuals to lean in to their experiences of the “ebb and flow” of wellbeing. We do this by tailoring messages to enhance self-management, providing peer support with regular checking in to support and empower people using the App. Similarly, the culture in the team is important and we have actively supported staff to lean in to adverse effects that may occur in the context of the trial generally and app usage specifically. We believe this has produced a culture of learning that is enabling us to optimise the EMPOWER intervention for a future trial.

One means of improving adverse event monitoring and reporting could be through an increased emphasis in reporting guidelines. It is therefore notable that the World Health Organisation checklist for reporting evidence and effectiveness in mHealth, also known as the mERA guidelines (Agarwal et al., 2016), makes no reference to either safety reporting or adverse events within their 16-item check list and that recommendations are similarly lacking from the CONSORT-EHEALTH reporting guidelines (Eysenbach & CONSORT-EHEALTH Group, 2011). Updating and expanding these, and other relevant standards could go a long way to improving reporting of mHealth for psychosis research which should ultimately improve the acceptability of interventions and minimise harm.

Adverse events in mHealth for psychosis should be anticipated and while as a research team we were obliged to introduce enhanced monitoring the effect of this has been overwhelmingly positive. With increased interest internationally in the regulation and monitoring of mHealth interventions it seems likely that the need for more mitigate the risk of detailed safety reporting will increase. However, regardless of the demands of regulators we encourage researchers to voluntarily adopt enhanced adverse event monitoring procedures, to apply them those throughout studies and to fully report their findings for the advancement of the field. While developing and implementing such procedures is time-consuming, ultimately they should improve experiences for end users and reduce inherent risks, while improving the acceptability and design of interventions. It might also unrealistically positive framing of mHealth for psychosis interventions from enthusiastic early adopters.

Disclosure statement

No potential conflict of interest was reported by the authors.

Note

Additional information

Funding

This study is funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme (13/154/04). The views expressed are those of the authors and not necessarily those of the NIHR or the Department of Health and Social Care. This study was also supported by NHS Research Scotland, through the Chief Scientist Office and the Scottish Mental Health Research Network.

Notes

1 Serious adverse events are defined by the UK Health Research Authority as including death, life threatening persistent or significant disability or incapacity, hospitalisation or prolongation of existing hospitalisation congenital anomaly or birth defects.