Human factors evaluation of a novel digital medicine system in psychiatry

Figures & data

Figure 1 Information components and data communication.

Notes: Profit D, Rohatagi S, Zhao C, Hatch A, Docherty JP, Peters-Strickland TS. Developing a digital medicine system in psychiatry: ingestion detection rate and latency period. The Journal of Clinical Psychiatry. Volume 77(9). Pages e1095–e1100. Copyright 2016. Reprinted with permission.¹³
Abbreviations: HCP, health care provider; IEM, ingestible event marker; MDDS, Medical Device Data System.

Figure 2 Steps in (A) human factors studies overall and (B) in risk analysis.

Notes: Six formative human factors studies were conducted on the patient interface and one on the HCP and caregiver interface. One validation study was conducted on the patient interface and one on the HCP and caregiver interface. The risk analysis was conducted in accordance with FDA recommendations and FDA-recognized standards on human factors and usability engineering of medical devices.^20,21 Adapted from US Department of Health and Human Services. Draft guidance for industry and Food and Drug Administration staff: applying human factors and usability engineering to medical devices to optimize safety and effectiveness in design. Available from: https://www.fda.gov/downloads/MedicalDevices/.../UCM259760.pdf. Accessed July 11, 2017.²¹
Abbreviations: ALARP, as low as reasonably practical; DMS, digital medicine system; FDA, US Food and Drug Administration; HCP, health care provider; NAC, not acceptable; RAC, risk analysis code; TOL, tolerable; uFMEA, use failure mode and effect analysis.

Table 1 Overview of human factors studies

Study	Participants	Device/interface tested	Onboarding	Changes resulting from the study
Patient interface studies
Formative 1 December 2013	Patients N=10 (2 BP-I, 4 MDD, 4 SZ)	Wearable sensor version RP4, DMS version 1.3, preliminary labeling	No in-person assistance	New wearable sensor design implemented, mandatory in-app training video generated
Formative 2 February 2014	Patients N=7 (3 BP-I, 2 MDD, 2 SZ)	Wearable sensor version DW5, DMS version 1.5, preliminary labeling	No in-person assistance, full-length, simulated in-app video	Packaging, medication reminder, and notification tile redesigned
Formative 3 December 2014	Patients N=36 (18 SZ, 8 MDD, 10 BP-I)	Wearable sensor version DW5, DMS version 1.5.2	In-app videos during onboarding; HCP-assisted onboarding for 18 participants followed by independent use	Modifications to pill tiles and reminders, wearable sensor status tiles and patch application videos, and language and visuals of onboarding videos
Formative 4 March 2015	Patients N=36 (12 SZ, 10 BP-I, 14 MDD)	Wearable sensor version DW5, DMS version 1.5.2, draft labeling	In-app video during onboarding; HCP-assisted onboarding for 17 participants	Introduction of quick start guide, elimination of pill reminder functionality, modification to wearable sensor tiles, and language of onboarding videos
Formative 5 a, b, c July 2016–October 2016	Patients N=30 (15 SZ, 9 BP-I, 6 MDD)	Wearable sensor version DW5, DMS version 2.0, draft labeling (including draft quick start guide)	In-app videos during onboarding	Modifications to onboarding flow, daily summary tile, wearable sensor icons, and introduction of quick start guide
Formative 6 October 2016	Patients N=10 (6 SZ, 1 BP-I, 3 MDD)	Wearable sensor version DW5, DMS version 2.0, draft labeling (including draft quick start guide)	In-app videos during onboarding	Updates to onboarding videos related to wearable sensor issues and the daily summary tile; changes to increase legibility of text in the system
Patient validation 2017	Patients N=35 (11 SZ, 12 BP-I, 12 MDD)	Wearable sensor version DW5, DMS version 2.0, draft labeling	In-app videos during onboarding; HCP-assisted onboarding for 17 patients
HCP/caregiver studies
Formative March 2014	HCPs and caregivers N=11 (6 HCPs, 5 caregivers)	HCP dashboard, caregiver dashboard	No in-person assistance, full-length, simulated dashboard video for caregivers	Feedback on structuring of assisted onboarding
Validation February 2015	HCPs and caregivers N=33 (17 HCPs, 16 caregivers)	HCP dashboard, caregiver dashboard	No assistance

Abbreviations: BP-I, bipolar I disorder; DMS, digital medicine system; HCP, health care provider; MDD, major depressive disorder; SZ, schizophrenia.

Table 2 Examples of formative study findings and corresponding design modifications to the patient interface

Finding	Original setup	Design modification	Modification characteristic
General setup and account setting
Errors resulting from removal of content in an unintended order	Packaging does not lead to removing the content components in the correct order	Packaging forces ordered and sequenced interaction with the system components; introduction of quick start guide	Providing an overview and sequencing of tasks
Paper-based IFU not used	Paper-based IFU	Electronic IFU, placed within the mandatory app video	Simplifying and sequencing tasks
Difficulty inviting HCPs	Selection of an HCP from a list	A manual entry of the HCP name and email	Using concrete and explicit language
Failure to edit notification preferences	Superfluous information within the app settings	Reorganization of menu options	Avoiding information overload
Difficulty accessing time change control for the medication reminder	No specific information in the system overview video	Pill reminder functionality was removed	Simplifying tasks
Wearable sensor
Organization of timeline section for the sensor status in the app was perceived as complicated	Numerous tiles in the timeline section	Fewer tiles with specific prompts that led users straight to the action that needed to be performed	Avoiding information overload and simplifying tasks
Pairing the wearable sensor with the app was often skipped during onboarding	Optional pairing of the wearable sensor during onboarding	Mandatory pairing of the sensor during onboarding	Sequencing of tasks
Failure to remove old wearable sensor before pairing the new wearable sensor	No specific information	Pairing flow for wearable sensor was revised to include explicit instructions	Using concrete and explicit language
Failure to click start on the 0 day tile before changing wearable sensor	No instructions to press start	0 day tile redesigned to clarify messaging	Using concrete and explicit language
Medication
Poor understanding of manual registration of medication ingestion	Manual registration of medication ingestion feature	Removal of the feature and replacement with the medication adherence survey	Simplifying tasks
Misinterpretation of “other medication” reminder	Reminder to take other medication built in the app	Removal of the feature	Simplifying tasks

Abbreviations: HCP, health care provider; IFU, instructions for use.

Table 3 Performance assessment for tasks in the validation study of patient interface

Task description	Task criticality	Task outcomes, n
		S	D	CCa	F
General setup and account setting
Open kit	N	35	0	0	0
Download app	N	34	0	0	1
Enter account information	N	35	0	0	0
Enter doctor information	N	35	0	0	0
Log in to app	N	35	0	0	0
Act on weekly/monthly	C	35	0	0	0
Act on daily summary tile	C	32	0	1	2
Verify medication registered	C	35	0	0	0
Wearable sensor
Remove wearable sensor from package	N	35	0	0	0
Pair first wearable sensor	N	35	0	0	0
Peel off large tab	N	34	0	0	1
Apply first wearable sensor	N	31	2	0	2
Push data from wearable sensor	N	35	0	0	0
Remove wearable sensor	De	34	0	0	1
Remove replacement wearable sensor from package	N	35	0	0	0
Pair replacement wearable sensor	N	29	3	0	3
Peel off large tab	N	35	0	0	0
Apply replacement wearable sensor	N	33	0	0	2
Check wearable sensor status	N	34	1	0	0
Check 0 day status	N	34	0	0	1
Fix poor contact	N	35	0	0	0
Medication
Ingest first pill	N	33	0	0	2
View overview video	N	35	0	0	0

Note:

a Cells with gray shading denote tasks for which a close call was not possible because the task was not critical.

Abbreviations: C, critical; De, desirable; N, necessary; S/D/CC/F, number of performances rated as success/difficulty/close call/failure.

Figure 3 Reduction in the percentage of failures on performance tasks and knowledge assessments during human factors testing of the patient interface.

Notes: Percentages for unassisted user group were calculated as follows: number of failures on user tasks and knowledge assessments divided by the total number of opportunities for failure (ie, total number of attempts). The numbers for each study were as follows: Formative 4, 71/586; Formative 5a, 29/304; Formative 5b, 16/297; Formative 5c, 13/378; Formative 6, 4/303; and Validation 2017, 11/710.

Table S1 Examples of user tasks tested in formative studies of patient interface

Component	Task (subtask)
General setup and use	Opening package contents in ordered sequence
	Entering HCP information
	Verifying that medication is registered
	Viewing weekly/monthly activity
Wearable sensor	First-time application
	Pairing with smartphone
	Attaching to body
	Checking status
	Checking poor skin contact status
	Checking 0 day sensor status
	Fixing poor skin contact
	Initiating data transfer
	Pairing a replacement sensor
	Attaching a replacement sensor to body
Medication	Removing pill(s) from packaging
	Storing medication
	First-time ingestion
	Taking a single pill
	Discarding old medication
	Understanding that wearable sensor is not medicated
	Ingesting pill
	Confirming no alteration of medication

Abbreviation: HCP, health care provider.

Table S2 Cautionary statements with a potential for serious or higher severity harm evaluated in the validation study

Cautionary statement	Packaging and labeling	Attempts, n	Task outcome, n
			S	D	F
Do not prepare a new patch until instructed by the app	QSG	34	34	0	0
Do not take your first DMS pill until instructed by the app	QSG	36	35	1	0
Do not stop or change your medication dosage based on information provided by the DMS kit; consult your health care provider	QSG	35	34	1	0
Do not change your medication unless instructed by your prescribing physician	Pill tray	35	34	1	0
Keep DMS components out of the reach of children	QSG	35	35	0	0
Do not chew pill	QSG	34	34	0	0
Do not place pill in water	QSG	34	34	0	0
Do not use the patch if allergic to adhesive	QSG Patch box insert	34	34	0	0

Abbreviations: D, difficulty; DMS, digital medicine system; F, failure; QSG, quick start guide; S, success.

Table S3 App messaging evaluated in the validation study

Question	Task criticality	Attempts, n	Task outcome,a n
			S	D	F
Patches do not contain medication	N	35	35	0	0
Only DMS pills will work with the system	N	35	34	0	1
Pill registered	C	34	33	0	1
Pill registered (timeline)	C	34	34	0	0
Pill does not seem to be registering	C	33	32	0	1

Note:

a No close calls were observed on critical tasks.

Abbreviations: C, critical; D, difficulty; DMS, digital medicine system; F, failure; N, necessary; S, success.

ProfitDRohatagiSZhaoCHatchADochertyJPPeters-StricklandTSDeveloping a digital medicine system in psychiatry: ingestion detection rate and latency periodJ Clin Psychiatry2016779e1095e110027379966

 PubMed Web of Science ®Google Scholar

Advancing Safety in Medical TechnologyANSI/AAMI HE75: Human Factors Engineering–Design of Medical DevicesArlington, VAAssociation for the Advancement of Medical Instrumentation2009

 Google Scholar

US Department of Health and Human Services, US Food and Drug AdministrationApplying Human Factors and Usability Engineering to Medical Devices: Draft Guidance for Industry and Food and Drug Administration staff2016 Available from: https://www.fda.gov/downloads/MedicalDevices/.../UCM259760.pdfAccessed July 11, 2017

 Google Scholar