SERIES: eHealth in primary care. Part 5: A critical appraisal of five widely used eHealth applications for primary care – opportunities and challenges

Figures & data

Figure 1. Conceptualisation of eHealth in primary care, derived from Shaw et al. [11].

Table 1. Description of five widely used eHealth applications.

Description	Shaw domain
Thuisarts.nl
https://www.thuisarts.nl. Non-commercial public website in the Netherlands, launched by the Dutch College of General Practitioners. It provides patients with understandable and reliable medical information combined with automated (non-personal) advice. It includes the content of evidence-based guidelines on 600 topics. Each topic consists of several ‘patient situations’ (e.g. I have the flu, my child has the flu) and provides information on what to do and when to contact the GP. It includes illustrations, short videos, patient decision aids and eHealth self-management tools. It has an extremely high in-country use (24 million visitors in 2018) [37]. https://gpinfo.nl is the English version of Thuisarts.nl, covering a selection of medical topics.	Inform, monitor, track
Liva Healthcare
https://livahealthcare.com. Innovative digital health programs to improve a variety of lifestyle behaviours by creating bonds between the lifestyle coach and the patient. The digital behavioural change programs consist of real-time personal coaching and pre-recorded videos, group-based interventions, tailored health plans, goal-tracking and self-monitoring, and fixed evidence-based curriculums personalised to support people at risk of or living with chronic conditions. Developed in Denmark in 2015 and was recently implemented in Norway, UK and Australia.	Inform, monitor, track Interaction
SHUTi
https://www.somryst.com. An online intervention for the treatment of insomnia in adults. It is a fully automated, interactive and tailored web-based program that incorporates the primary tenets of face-to-face Cognitive Behavioural Therapy for Insomnia (CBT-I), including sleep restriction, stimulus control, cognitive restructuring, sleep hygiene and relapse prevention. Developed in the United States.	Inform, monitor, track Interaction
Babylon
https://www.babylonhealth.com. AI system that can receive data about a patient’s symptoms, compare the information to a database of known conditions and illnesses, and then identify a course of action and related risk factors. It contains a virtual GP appointment, digital prescriptions, digital health check, instant symptom chequer, online view of medical records, and a chatbot. It is now an integrated part of the UK-based NHS. It can be used by patients who have registered for the service. Patients register with ‘GP at Hand’ and can then consult a digital GP within minutes of registration; however, they have to switch from their current GP practice to Babylon. Certain GPs work specifically for Babylon, with whom the patient registers. Video appointments are possible 24/7 in collaboration with 7 clinics in London and 1 clinic in Birmingham.	Inform, monitor, track Interaction Data utilisation
SkinVision
https://www.skinvision.com/nl/. International skin cancer detection app allows patients to check suspicious skin spots themselves, promoting timely, appropriate care. It is available as either a freely downloadable app or a paid service, which various health insurers reimburse. A user can self-assess the skin cancer risk of a skin lesion by taking a photo with his/her smartphone, which is then processed by an algorithm. The user receives feedback on the risk presented by the skin lesion. SkinVision was developed by an official dermatology clinic regulated by the Dutch healthcare system, and can replace a normal dermatology consultation. Patients can also choose to send the photo to their healthcare professional, who will also receive the algorithm’s outcome as support in their diagnosis. The app also has a tracking function that can track skin spots over time. https://www.healthnavigator.org.nz/apps/f/firstcheck-app/. is a similar version from New Zealand	Inform, monitor, track Interaction Data utilisation

AI: Artificial Intelligence; CBT: cognitive behavioural therapy; GP: general practitioner; NHS: National Health Service; UK: United Kingdom.

Table 2. Evaluation of five widely used eHealth applications using six preconditions for high-quality eHealth.

Co-creation	Blended care	Individual and inclusive Applicable in high and low resource setting	Scientific evaluation	Ethics	Summary and recommendations
Thuisarts.nl
Team of physicians and editors developed the content End-users involved in a later phase Process of co-creation could be improved	Intended use of Thuisarts.nl: by the general public/patients; to obtain information about their symptoms or before a consultation GP; to explain medical information GP; refer a patient to the website GP; to check if their advice is up to date As patients have a better understanding of their complaints, the consultation could be more tailored.	Information is simply phrased, and supported by pictures and videos The popularity of the website has grown exponentially Several million unique page views each month Also used by an older population [29]	After launch, a decreasing trend in consultation rates, mainly caused by a decrease in short- and telephone consultations [29] No evidence available on usability or impact on patient outcomes	No known adverse or unintended effects among the patient population Further improvement could be made by adding an ethical expert to the team, and by promoting co-creation with vulnerable and low-literacy patient groups	Website that informs patients and GP’s about basic medical conditions. Simple application, without a major negative impact on daily routines. However, Thuisarts.nl has not yet systematically adapted to the wishes and requirements of the end-users, which we believe is a precondition for the development of a professional and high-quality eHealth application
Liva Healthcare
Development by: entrepreneurs behind NetDoctor.com multidisciplinary teams of healthcare professionals, software developers, scientists and patients	The healthcare professional refers a patient to Liva Healthcare No direct consequences for the routines of the referring healthcare professional An important driver of long-term weight loss was a strong relationship with a personal coach	More accessible than conventional treatment and ‘easy to use’ [30] Participants felt that having an initial one-off, face-to-face meeting with the dietician or lifestyle coach was important for their future web-based interaction	Effective in improving the lifestyle of patients with diabetes, e.g. more exercise and weight loss [32,33] Open to peer-reviewed scientific evaluation; programme will be improved after analysis by scientists [30]	No adverse or unintended effects among the patient population Further improvement could be made by adding an ethical expert to the team, and by promoting co-creation with vulnerable and low-literacy patient groups	Liva Healthcare is an easy-to-use application to improve lifestyle, with evidence-based results on clinical outcomes, high user-satisfaction and limited resistance from doctors [38]. This eHealth tool can support behavioural changes through monitoring and providing relevant feedback. Support from family and peers also matters, and long-term success depends on ability to establish positive day-to-day support [30]
SHUTi
Development by multidisciplinary team of: psychologists psychiatrist engineers software developers commercial experts	GP’s can prescribe the SHUTi program for treatment of insomnia in adults The acceptance of this online cognitive behavioural therapy is adequate, as GP’s generally recognise the value for the treatment of insomnia. No frequent referral because of consultation time constraints and limited knowledge [39] No specific education of healthcare professionals is needed, and elaborate implementation strategies are not necessary as healthcare professionals can simply prescribe the online programme [40]	Acceptance by patients was high, with a relatively high usage and adherence [34] The design of SHUTi aims to simulate the schedule and framework of an in-person treatment. It uses approachable and supportive language to help the user feel encouraged and motivated [41]	The efficacy of SHUTi has been demonstrated in multiple randomised controlled trials, which reported a significant and persistent reduction in symptoms of insomnia [34,42] and depression [42] A recent systematic review and meta-analysis confirms the general efficacy of similar online programs [43] A subsequent version of SHUTi (i.e. Somryst) recently obtained authorisation from the FDA in the USA to be used as a ‘prescription digital therapeutic’ for the treatment of insomnia in adult patients	No adverse or unintended effects among the patient population Further improvement could be made by adding an ethical expert to the team, and by promoting co-creation with vulnerable and low-literacy patient groups	Effective and practical online solution for the treatment of insomnia in general practice. Insomnia is currently not adequately treated in the primary care setting [44] due to limited knowledge of sleep treatments [39] . Hence, contrary to international clinical guidelines that recommend CBT-I as the first line treatment for insomnia, hypnotic medication remains the primary treatment modality in the majority of the patients [45]. SHUTi is considered a viable option in the treatment of insomnia [43]
Babylon
Development by: medical doctors scientists engineers working for NHS and Babylon Babylon is regulated by the Care Quality Commission in England	Using Babylon means a different way of working for GP’s, with a more coaching, supportive role Patients do not need to learn a lot; in practice, a lot of young and healthy patients use this service	The advantages are: 24/7 digital access high satisfaction by users support by the government	No analysis by an external academic partner Impossible to determine how well it would perform with a broader set of cases Babylon's own study does not offer convincing evidence that its system can outperform GPs in any realistic situation, and there is a possibility that it might perform significantly worse when diagnosing disease [46]	To convince citizens to use Babylon Health, the developers use highly developed marketing techniques Data are stored to promote development of algorithms, without transparency in the algorithm method. Further clinical evaluation is necessary to ensure confidence in patient safety, privacy and accuracy of the algorithms [46] Possible risk that it improves the health status of the so-called ‘worried well’, but not that of vulnerable, high-risk groups	AI-based platform that claims to help patients to make a diagnosis. The usability is good, but there is no peer-reviewed analysis of data and a significant risk of underperforming diagnostics
SkinVision
Development by a multidisciplinary team: dermatologists scientists engineers According to existing law SkinVision is a healthcare provider and must meet established qualitative and safety requirements. Initially, it did not comply with legal requirements and, among others, had to draw up a policy plan by October 2020 [47]	False-negative outcomes might delay the detection and treatment of skin cancer False-positive outcomes may cause unnecessary stress among users or unnecessary visits to GP’s The user may ignore the advice given in the smartphone app due to a lack of trust or unawareness GP’s need to understand the basics of AI to deal with such algorithms in the diagnostic process	The app seems easy to use. Users need to be able to take a photo with their mobile phones There are no (scientific) publications available on usability	The accuracy of algorithm-based smartphone apps for the assessment of skin cancer is debateable [35] The algorithm has a sensitivity of 95% and a specificity of 78.3% in recognising signs of skin cancer [36] SkinVision outperforms other skin cancer detection apps in terms of accuracy [35] The current accuracy of the algorithm may warrant its use by lay users or GP’s	The medical consequences of the use of SkinVision are still unclear Although studies have demonstrated that SkinVision can have beneficial effects when diagnosing skin cancer, adverse effects have also been reported [35] Patient safety is not fully guaranteed in this phase	Skin cancer detection app for patients and doctors, based on an algorithm with high sensitivity and moderate specificity to detect skin cancer; there is room for improvement in terms of specificity and in terms of qualitative and safety requirements

AI: Artificial intelligence; GP: general practitioner; QoL: quality of life.

Table

Engagement of and co-creation by all stakeholders Blended care: eHealth combined with regular care Individualised and inclusive Applicable in high- and low-resource settings Evidence-based and supported by educational guidance Being attentive to ethical considerations, privacy and patient safety

Danish Ministery of Health, Danish Minestery of Finance, Danish Regions, et al. A coherent and trustworthy health network for all – digital health strategy 2018-2022 Odense (Denmark); 2018. [cited 2021 July 12]. Available from: https://sundhedsdatastyrelsen.dk/da/diverse/download.

 Google Scholar

Springvloed L, Bos N, de Jong J, et al. Thuisarts.nl – How does thuisarts.nl contribute to transparency and the best suitable care? [thuisarts.nl – Hoe draagt thuisarts.nl bij aan transparantie en de best passende zorg?]. Utrecht: NIVEL; 2019. [cited 2021 Jul 12]. Available from: https://www.nivel.nl/sites/default/files/bestanden/6-Transparantiemonitor-2018-Thuisartsnl.pdf

 Google Scholar

Spoelman WA, Bonten TN, de Waal MW, et al. Effect of an evidence-based website on healthcare usage: an interrupted time-series study. BMJ Open. 2016;6(11):e013166.

 PubMed Web of Science ®Google Scholar

Brandt CJ, Sogaard GI, Clemensen J, et al. Determinants of successful eHealth coaching for consumer lifestyle changes: qualitative interview study among health care professionals. J Med Internet Res. 2018;20(7):e237.

 PubMed Web of Science ®Google Scholar

Haste A, Adamson AJ, McColl E, et al. Web-based weight loss intervention for men with type 2 diabetes: pilot randomized controlled trial. JMIR Diabetes. 2017;2(2):e14.

 PubMedGoogle Scholar

Komkova A, Brandt CJ, Hansen Pedersen D, et al. Electronic health lifestyle coaching among diabetes patients in a real-life municipality setting: observational study. JMIR Diabetes. 2019;4(1):e12140.

 PubMedGoogle Scholar

Johansen MY, MacDonald CS, Hansen KB, et al. Effect of an intensive lifestyle intervention on glycemic control in patients with type 2 diabetes: a randomized clinical trial. JAMA. 2017;318(7):637–646.

 PubMed Web of Science ®Google Scholar

Ogeil RP, Chakraborty SP, Young AC, et al. Clinician and patient barriers to the recognition of insomnia in family practice: a narrative summary of reported literature analysed using the theoretical domains framework. BMC Fam Pract. 2020;21(1):1.

 PubMed Web of Science ®Google Scholar

van der Zweerde T, Lancee J, Ida Luik A, et al. Internet-Delivered cognitive behavioral therapy for insomnia: tailoring cognitive behavioral therapy for insomnia for patients with chronic insomnia. Sleep Med Clin. 2019;14(3):301–315.

 PubMed Web of Science ®Google Scholar

Ritterband LM, Thorndike FP, Ingersoll KS, et al. Effect of a web-based cognitive behavior therapy for insomnia intervention with 1-year follow-up: a randomized clinical trial. JAMA Psychiatry. 2017;74(1):68–75.

 PubMed Web of Science ®Google Scholar

Archer D. SHUTi: a new insomnia treatment via the internet. 2017. [cited 2021 July 12]. Available from: https://www.psychologytoday.com/us/blog/reading-between-the-headlines/201702/shuti-new-insomnia-treatment-the-internet.

 Google Scholar

Christensen H, Batterham PJ, Gosling JA, et al. Effectiveness of an online insomnia program (SHUTi) for prevention of depressive episodes (the GoodNight study): a randomised controlled trial. Lancet Psychiatry. 2016;3(4):333–341.

 PubMed Web of Science ®Google Scholar

Zachariae R, Lyby MS, Ritterband LM, et al. Efficacy of internet-delivered cognitive-behavioral therapy for insomnia – a systematic review and meta-analysis of randomized controlled trials. Sleep Med Rev. 2016;30:1–10.

 PubMed Web of Science ®Google Scholar

Grandner MA, Chakravorty S. Insomnia in primary care: misreported, mishandled, and just plain missed. J Clin Sleep Med. 2017;13(8):937–939.

 PubMed Web of Science ®Google Scholar

Baglioni C, Altena E, Bjorvatn B, et al. The European academy for cognitive behavioural therapy for insomnia: an initiative of the European insomnia network to promote implementation and dissemination of treatment. J Sleep Res. 2020;29(2):e12967.

 PubMed Web of Science ®Google Scholar

Fraser H, Coiera E, Wong D. Safety of patient-facing digital symptom checkers. Lancet. 2018;392(10161):2263–2264.

 PubMed Web of Science ®Google Scholar

Inspectie Gezondheidszorg en Jeugd. Report of the inspection research of SkinVision in the period from January 2019 to December 2019 [Rapport van het inspectieonderzoek van SkinVision in de periode van januari 2019 tot december 2019] Utrecht: Inspectie Gezondheidszorg en Jeugd; 2020. [cited 2021 July 12]. Available from: https://www.toezichtdocumenten.igj.nl/.

 Google Scholar

Freeman K, Dinnes J, Chuchu N, et al. Algorithm based smartphone apps to assess risk of skin cancer in adults: systematic review of diagnostic accuracy studies. Br Med J. 2020;368:m127.

 PubMed Web of Science ®Google Scholar

Udrea A, Mitra GD, Costea D, et al. Accuracy of a smartphone application for triage of skin lesions based on machine learning algorithms. J Eur Acad Dermatol Venereol. 2020;34(3):648–655.

 PubMed Web of Science ®Google Scholar