Subjects with osteoarthritis can easily use a handheld touch screen electronic device to report medication use: qualitative results from a usability study

Abstract

Objectives

Electronic data capture is increasingly used to improve collection of patient-reported outcome measures in clinical trials and care. The validation of electronic patient-reported outcome devices requires information on patient preference and ease of use. This study conducted usability testing for a General Symptom Questionnaire and Medication Module™ on a handheld device for subjects with osteoarthritis (OA) to determine whether subjects can report on their symptoms and medication use using an electronic diary.

Methods

Nine subjects with OA participating in a large US mode equivalency study were surveyed independently in this study. Subjects completed a General Symptom Questionnaire and Medication Module™ using the LogPad^® LW handheld device. Demographic and technology use information was collected, and the subjects were queried on device usability.

Results

Subjects reported that the handheld device was easy to use and that they were able to complete a General Symptom Questionnaire and Medication Module™ with little or no assistance. They did not report any issues with the screen or size of the device. Subjects were willing to travel with the device to complete electronic diaries at home or in public. Participants indicated that they would be able to use the handheld device to answer questions during a clinical trial. Subjects with OA experienced no physical discomfort during completion of either questionnaire.

Conclusion

The General Symptom Questionnaire and Medication Module™ were usable and acceptable to subjects with OA on a handheld device. This was consistent regardless of previous experience and confidence with technology, despite the potential physical restrictions for an OA cohort.

Video abstract

Point your SmartPhone at the code above. If you have a QR code reader the video abstract will appear. Or use:

http://youtu.be/qkQWjM1uSF0

Keywords:

• ePRO
• patient preference
• eDiary
• usability testing

Introduction

Osteoarthritis (OA) is a chronic joint disease and the most frequent cause of disability among adults in the United States. OA most commonly affects joints of the knee, hip, hands, and spine.¹ Data from the 2010 to 2012 National Health Interview Survey determined that >52 million adults in the United States had at least one type of clinically diagnosed OA.² By 2030, this number is estimated to reach 67 million, or 25% of Americans older than 18 years.³ This is attributed to increases in aging and obese populations.^4,5 The total cost associated with arthritis care in the United States in 2003 was $128 billion; two-thirds of this went to medical costs, while the remaining third was due to earning losses.^6,7 OA was the fourth most common cause of hospitalization in 2009.⁷

OA clinical care is largely focused on alleviating symptoms and improving health-related quality of life as assessed by patients.⁸ Selecting an appropriate outcome mode is critical to the design of clinical trials and patient care. Patient-reported outcomes (PROs) are becoming an increasingly popular and reliable method for collecting patient health and quality-of-life data.⁹ A study that examined 215 prescription drugs approved in the United States from 1997 to 2002 found that 30% of the product labels utilized PRO endpoints.¹⁰ In addition, PROs were the only outcome in 11% of these US Food and Drug Administration-approved product labels. PROs are especially valuable when the ultimate treatment goal is to alleviate pain and to improve quality of life¹¹ and physical functioning, as is the case with OA therapy. Pain and discomfort are predominant symptoms of OA, and therefore, subjective patient assessment is essential for OA diagnosis and care.¹² In addition, there are other elements to the pain in OA, which may have further treatment implications; as the disease progresses, the pain experienced may become chronic, unexpected, or even indicate concomitant disorders, such as neuropathy.^13,14 Currently, the National Health Service in England requires patients to be monitored using PROs before and after certain surgeries, including knee and hip replacement.^15,16

In 1997, the Outcome Measures in Rheumatoid Arthritis Clinical Trials (OMERACT) committee identified a core set of outcome assessments for OA clinical trials that require patients to self-report on pain, physical function, and global assessments.¹⁷ More recently, the Osteoarthritis Research Society (OARSI), together with OMERACT, proposed a simplified set of patient-reported criteria to establish international standardization across OA clinical trials and to include assessment of symptoms prior to patients entering OA clinical trials.¹⁸ This is essential as the success of OA clinical trials is determined by a measurable improvement in symptoms. To this end, it is also necessary to ensure that changes in pain level are detectable from data provided by PROs. Tubach et al set out to establish the minimal clinically important improvement reported by patients with knee or hip OA using the OMERACT–OARSI outcome measures: pain, physical function, and patient global assessment.¹³ They confirmed minimal clinically important improvement as one of the useful clinical criteria to measure OA symptom improvement based on individual patient opinion and perspective. A number of PRO instruments that meet OARSI measure guidelines have been recommended for use in OA clinical trials (reviewed in Fitzgerald et al¹⁹). In addition, feasibility testing on several PRO instruments, such as EQ-5D, SF-36, HOOS, and OHS, demonstrated that they are acceptable to and appropriate for use by OA patients.²⁰ Furthermore, the Hip/Knee Osteoarthritis Decision Quality Instrument was found feasible and acceptable to facilitate high-quality decisions for patients considering joint replacement surgery.²¹

With the success and validation of PROs for clinical trial endpoints, particularly for OA, comes an increasing use of electronic methods to capture patient data. Electronic PROs (ePROs) are recognized by the US Food and Drug Administration as a promising clinical trial outcome measure.²² ePROs offer many advantages over paper and pencil instruments, including less time to completion and decreased patient burden, elimination of missing data points and data entry mistakes, increased patient compliance, and improved accuracy of data collection.^23–25 Previous studies recorded response rates of only 72%–83.4% for paper administration of PROs to patients with OA.^20,26,27 In addition, one study reported a rate of missing response items up to 4.3%.²⁰ Implementation of ePROs for this patient population may increase compliance and eliminate missing responses.

Despite many advantages to paper administration, it is necessary to provide sufficient evidence for measurement equivalence of ePRO devices, including cognitive debriefing, equivalency testing, and usability testing.²⁸ A recent review of PRO and ePRO equivalency studies, including general and condition-specific instruments, found that 78% of studies demonstrated equivalence between paper and electronic formats.²⁹ In addition, 87% of publications that included preference data found that patients preferred electronic methods over paper administration. As PROs are especially useful for therapeutic areas, such as OA, it is essential to collect ePRO equivalency and preference data for these patients specifically. A web-based ePRO system demonstrated excellent equivalency to pencil and paper administrations of the OHS, McCarthy hip score, UCLA activity score, and howRU score on OA patients.³⁰ Furthermore, several additional instruments were tested on electronic handheld devices by subjects with rheumatoid arthritis.³¹ All seven of the surveys tested were found to be equivalent between modes of administration. Importantly, subjects found that the electronic method is easy to use and preferred this mode over paper, regardless of age or previous experience with technology.

ePRO is a promising method for capturing patient data, particularly in a chronic condition such as OA, where pain symptoms and quality of life are best assessed by the patients. Although mode equivalency and preference studies have demonstrated appropriate use of ePROs for patients with OA, there is very little information on usability testing for this group. Especially in the case of OA, usability studies should consider potential physical pain or discomfort when using devices. The objective of this study was to conduct usability testing on a General Symptom Questionnaire and a Medication Module™ using the LogPad^® LW handheld device for subjects with OA.

Methods

The usability testing described in this report was conducted in the context of a larger randomized, crossover mode equivalence study measuring the equivalence of a standardized PRO questionnaire when administered on paper and on two electronic modalities (N=400). Subjects who participated in this usability testing (n=9) did so immediately after participating in the equivalence study. Some data collected as part of the equivalence study (eg, demographics and technology use) were used in the analysis for the usability testing. The purpose of the usability testing was to determine whether subjects had any difficulty in using the LogPad^® LW device or understanding the General Symptom Questionnaire or the Medication Module™ as they were presented on the LogPad^® LW handheld electronic diary.

Ethics

This study was reviewed and approved by Copernicus Group independent review board (IRB) prior to any contact with study subjects. The following study documents were submitted to the IRB: protocol, informed consent form, demographic and condition questionnaire, and interview guide (IG). Following review, IRB approval was granted on April 28, 2014.

Questionnaires

A General Symptom Questionnaire was created that consists of three screens: 1) an instruction screen, 2) a single-select question about subject fatigue, and 3) an alert screen. The answer options on screen 2 were one-word descriptors on a four-point scale of intensity (none, mild, moderate, and severe). As the subject selected one of the answer options, a more detailed definition of that answer dynamically displayed at the bottom of the screen, and subjects could change their selection to view the full definition of each answer option. The alert screen is an example of the type of message that might be displayed to the patient if their reported symptoms are worsening over time.

The Medication Module™ is a component of the eResearch Technologies, Inc (ERT) ePRO system used to track subject’s consumption of medications. It enables entry of medication data by subjects via the ERT LogPad^® LW handheld device and allows subjects to record medications between visits and identify when any new medications have been taken that have not previously been described. The Medication Module™ consists of several sections. First, the subject indicates whether or not they took any medications since yesterday. If they select “No”, they are directed to the end of the questionnaire; if they select “Yes”, they are directed to a screen which displays commonly used drug/dose/mode combinations that are tailored to the subject. The subject selects one of these medications or “Other”, then enters the date, time, and dose amount of the medication. Subjects can then choose to add another time/dose of the same medication, add a different medication, or review medications they have already entered. Once they finish adding medications, they are shown a review screen displaying all medications they have added. Subjects can edit or delete any of these medications or continue to add new medications until the review screen accurately displays the medications they have taken that day.

Recruitment

Following IRB approval, subjects were recruited for the main PRO equivalence study. Advertisements were placed in local media in Boston, MA, and potentially eligible subjects contacted ERT via phone or email. ERT personnel spoke with subjects via phone and screened them for eligibility; eligible subjects were scheduled an appointment at the study site in Boston, MA. Subjects arrived at the site and participated in the main PRO equivalence study; after completing the study, ERT personnel approached subjects about participating in an additional 15-minute usability test and interview. Consecutive subjects were approached until the target sample size for usability testing was achieved.

Sample size

Previous research suggests that a minimum of 3 and a maximum of 20 subjects is an appropriate sample size for a qualitative usability study, depending on the complexity of the study.^32,33 The general consensus is that five participants will uncover 80% of the usability issues that should arise^34,35 and seven or more participants would be an optimal number.³⁶ Recruitment for this usability study targeted nine participants.

Eligibility criteria

Subjects recruited for the usability study had already met the eligibility criteria of the main PRO equivalence study. Eligible subjects for the equivalence study were 18 years or older, provided a signed informed consent form indicating an understanding of the study objectives and study procedures and a willingness to participate in the study, had the ability to complete surveys alone or without assistance in English, reported having been told by a doctor or other health professional that they had OA or degenerative arthritis, and reported having pain, swelling, or limitation in range of motion in a knee or hip during the past 4 weeks. Subjects were ineligible if they reported having been medically diagnosed with traumatic brain injury, dementia, schizophrenia, any psychoses with current symptoms, or alcohol or drug dependence with current symptoms. The only additional eligibility criterion for the usability study was that subjects verbally consent to participate in the study.

Interviews

Prior to conducting any interviews, the study research team created a semistructured IG to identify potentially challenging aspects of the interview and highlight the most important questions to address in order to answer the research questions. Subjects who consented to participate were interviewed by ERT personnel who have undergone National Institutes of Health Human Participant Protection training, as well as training for subject interviews. The interviews were conducted in a private room to ensure confidentiality.

The interviewer introduced him/herself and gave an overview of the objectives of the interview. The interviewer then turned on the audio recorder and secured additional verbal consent from the subject to have the interview audio recorded. The subject was then given a LogPad^® LW device with the General Symptom Questionnaire and the Medication Module™ and asked to complete each questionnaire as if they were participating in a clinical trial.

Following completion of both questionnaires, the interview began. A semistructured IG was used to obtain both structured and spontaneous feedback from subjects. The guide included open-ended questions to collect qualitative data from the subject’s perspective and was constructed to establish the usability of both the LogPad^® LW itself and the questionnaires implemented on it. After completion of the interview, each subject was compensated with a $100 gift card.

Qualitative data analysis

Audio recordings of the interviews were transcribed verbatim and anonymized by removing identifying information, such as names. Transcripts were manually coded in Microsoft Excel^® by one research analyst, using predetermined codes that aligned with questions in the IG and expected subject response categories. Subject responses that were not adequately described by a predetermined code were categorized with new codes and direct subject quotes were extracted from the interview transcript as supplementary data. Another research analyst reviewed and aggregated the coded data from all transcripts. Each transcript was considered a unit of analysis; a third researcher performed a descriptive analysis on the aggregated data.

Results

Socioeconomic demographics, comorbid conditions, and technology use

Information on socioeconomic demographics and comorbid conditions reported by each subject on the demographic and conditions questionnaire is presented in Table 1. Information on technology use and preferences reported by each subject on the demographic and conditions questionnaire is presented in Table 2. In addition to these data collected on paper, subjects were asked during the interview to describe their level of familiarity and confidence with using touch screen devices using a four-point rating scale. These responses are presented in Table 3.

Table 1 Demographic and conditions information (N=9)

	n
Sex
Male	5
Female	4
Age (years)
Mean	60.4
Range	49–80
Spanish/Hispanic/Latino
No	8
Yes	1
Race
White	4
Black/African American	5
Education
Some high school	1
High school graduate/GED	3
Some college/technical degree/AA	5
Annual household income
Less than $20,000	6
Between $20,000 and $49,999	2
Between $50,000 and $99,999	1
Occupational status
Homemaker	1
Unemployed	1
On disability	3
Employed full-time	2
Employed part-time	2
Relationship status
Never married	4
Married	1
Living with a partner (committed relationship)	1
Divorced	3
Have you ever been told by a doctor or other health professional that you have rheumatoid arthritis?
Yes	3
No	6

Abbreviations: GED, General Educational Development; AA, Associate of Arts.

Table 2 General technology use information (N=9)

	n
Do you have a smartphone?
Yes	4
No	5
How frequently do you use email?
Daily	2
Weekly	2
Monthly	0
I have used it, but do not recall when	1
I do not use email	4
Do you have internet access available at home?
Yes, wireless internet (Wi-Fi)	3
Yes, other internet access (not Wi-Fi)	2
Other internet access	0
No, I do not have internet access available at home	4
How frequently do you use the internet?
Daily	3
Weekly	1
Monthly	0
I have used it, but do not recall when	0
I do not use the internet	5
Do you have a computer (eg, laptop, desktop, tablet) at home?
Yes	5
No	4
What device do you most commonly use to browse/use the internet?
Computer (laptop, desktop)	3
Computer tablet (iPad, Kindle Fire)	0
Smartphone	0
Multiple devices	1
I do not use the internet	5

Table 3 Familiarity and confidence with touch screen devices (N=9)

	n
How familiar are you with touch screen devices?
Very familiar	0
Moderately familiar	3
A little familiar	3
Not familiar	3
How confident are you with touch screen devices?
Very confident	0
Moderately confident	3
A little confident	4
Not confident	2

Overall, subjects were middle aged or older with an annual household income of $20,000/year or less; none had a college degree. Three subjects reported rheumatoid arthritis as a comorbid condition. Current level of technology use varied widely, with about half of subjects reporting that they never used Internet or email and others using Internet and email daily. Self-reported familiarity and confidence with touch screen devices was low, with no subjects reporting that they were “very familiar” or “very confident” using these devices.

Usability testing of the General Symptom Questionnaire and ERT Medication Module™ with subjects with OA

Following completion of the General Symptom Questionnaire, subjects were asked a series of questions regarding the usability of this questionnaire on the device. Data from these questions are shown in Table 4, with selected quotes from subject interviews in italics. Subjects also completed the Medication Module™ to track their pain medication usage. After finishing the Medication Module™, participants were asked a series of questions regarding usability of this medication diary on the device. Data from these questions are summarized in Table 5. Overall, subjects did not encounter any significant issues when completing the General Symptom Questionnaire or Medication Module™.

Table 4 Usability testing of the General Symptom Questionnaire on LogPad^® LW (N=9)

	n
Would you say that after this brief training you know how to use the LogPad properly?
No	0
Yes	6
Somewhat	3
•Not really, I would have to… well yes I think I did because you pressed the button and I went ahead and did everything else so I think I would be able to learn it.
•If I had to.
•Somewhat.
Did you understand the instructions on the LogPad?
No	0
Yes	8
Somewhat	1
•Somewhat.  [Interviewer: What was not clear?] I think the question about fatigue was a little vague.  [Interviewer: How could you think of improving that?] I think the question should be more of degree of pain that I experience rather than overall tiredness.
Did you consider this brief training difficult or easy?
Difficult	0
Easy	9
Could you have completed the questionnaires on the device without any help?
No	2
•No. [Interviewer: Why not?] This is all new to me.
•No.  [Interviewer: So you don’t think you could have done it if I wasn’t here?] No.  [Interviewer: What would help you to understand if you had to fill it out on your own?] Instructions.  [Interviewer: The instructions on the screen weren’t enough for you?] Yeah they were.  [Interviewer: But you said you couldn’t do it if I wasn’t here?] In case something went wrong.
Yes	3
Probably/maybe	4
•I don’t know, maybe I could. I would say maybe. This is my first time.
•No I don’t think so. I would have to see how to turn it on and set it up, I think so, it might take me a little while but I think I might be able to do it.
•Probably.
•I need a little help.  [Interviewer: What would you need help with?] Just to get started.
Were the screens on the LogPad easy to use?
No	0
Yes	9
Was it difficult or easy to understand what to do next?
Difficult	0
Easy	9
If you were at home completing a daily diary, would you find it easy or difficult to use the LogPad?
Difficult	0
Easy	9
Would you like the ability to turn a reminder alarm on or off?
No	0
Yes	9
If educational material were available on this device, would you access it?
No	0
Yes	8
Probably	1
•Probably.

Table 5 Usability performance of the ERT Medication Module™ on LogPad^® LW (N=9)

	n
After completing this medication diary, do you feel that you would be able to record your medications on the LogPad?
No	0
Yes	9
Did you understand the instructions on the LogPad?
No	0
Mostly	1
•Mostly, yes.  [Interviewer: What do you mean, “mostly”?] I had to read one of them twice.  [Interviewer: So it wasn’t clear?] It was clear after I read it the second time.
Yes	8
Did you consider this diary difficult or easy?
Difficult	0
Easy	9
Could you have completed the diary without help?
No	0
Somewhat/Probably	2
•I am going to probably say somewhat.    [Interviewer: Why do you say that?] It is my first time but like I said it is very self explanatory so I could probably do it by myself.  •Without you, no. I don’t think so, I might need to ask certain questions. You mean like if I could have did (sic) the questions without asking you?    [Interviewer: The questions without me.] A little bit.
Yes	7
Were the screens easy to use?
No	0
Yes	9
Was it difficult or easy to understand what to do next?
Difficult	0
Had one issue	1
•Only when I got to the medication dose. When you saw the arrows there was one pointing up and one pointing down I knew I had to hit the one pointing up because I wanted to put one. I didn’t want to put zero but I imagine you had to tap it twice if you wanted to take two. I figured that out in my head so that was fine. That would have been a little confusing.
Easy	8
After completing the medication diary, do you think you could accurately answer the questions in the study?
No	0
Yes	9

Abbreviation: ERT, eResearch Technologies Inc.

Usability testing of the LogPad^® LW with subjects with OA

We asked subjects several questions about the comfort and portability of the LogPad^® LW device; data from these questions are presented in Table 6. Overall, subjects did not experience any physical discomfort during completion of either questionnaire and were satisfied with the size and portability of the device. The majority of subjects were willing to use the LogPad^® LW handheld device to complete diaries at home or in public.

Table 6 Usability testing of the LogPad^® LW (N=9)

	n
Did you experience any physical discomfort when using the LogPad?
No	9
Yes	0
If you had to carry this device with you and complete a diary outside of the home, would you be able to carry this device with you?
No	0
Yes	9
Would you be bothered by the size of this device?
No	8
Yes	0
No, but it could be lighter	1
• No. It’s cute. A little heavy though, can we make it a little lighter. Maybe the case got it heavier. It could be a little lighter.
Would you be willing to use the LogPad in public?
No	1
• I don’t think so.  [Interviewer: Why not?] I think somebody would take it from me. Security reasons.
Yes	8
After completing this training module, do you think you could accurately answer questions in a clinical trial, using the LogPad?
No	0
Yes	9

Discussion

Regulatory guidelines and scientific evidence indicate that there is improved data quality with electronic data capture. As such, there is increased use of ePROs to improve the collection and quality of PRO data for clinical trials, post-marketing studies, disease management studies, and clinical care. In this study, we tested usability of a General Symptom Questionnaire and Medication Module™ on the LogPad^® LW handheld device. The General Symptom Questionnaire was created to record patient symptoms using standard types of response scales and question/response formats, while the Medication Module™ provides patients with a way to record their pain medications using the device.

During the course of the usability testing, some subjects alluded to the fact that they were using the device for the first time and might have questions. This is a common concern among subjects, and for this reason, we suggest that when ePRO assessments are deployed in clinical trials, subjects be provided custom training and complete a practice diary on site with clinical staff present before they are required to complete any diaries at home by themselves. This provides the opportunity for subjects to ask questions and become acquainted with the functionality of the device, which can increase compliance, data quality, and patient engagement in the trial.

Conclusion

In summary, subjects did not encounter any significant issues in completing the General Symptom Questionnaire or the Medication Module™ on the handheld device. Despite reporting limited daily technology experience and limited confidence and familiarity with touch screen devices, all nine subjects found the device screens easy to use, found it easy to progress through the questionnaires, and thought that they could accurately answer items using the device. Although this usability testing was conducted using subjects with OA (three of which reported also having rheumatoid arthritis), none of the subjects reported experiencing physical discomfort when using the handheld device. These findings are consistent with previously published studies demonstrating the usability of electronic data capture of PROs in patients with OA and other rheumatologic conditions.^37–39

Limitations of this study include the possibility of selection bias. We did not document the number of subjects who declined to participate in the usability testing and are thus unable to compare their demographic characteristics with the subjects in our study to determine whether any selection bias occurred. A second limitation is around the generalizability of our results. We used a convenience sample of nine subjects, thus it is possible that the demographics, technology experience, or other characteristics of the subjects do not exactly reflect the characteristics of the average OA patient. For example, about half of all US people older than 65 years have been diagnosed with OA,² while our sample contained only two subjects (22%) older than 65 years. Furthermore, usability research using a larger, more representative sample of subjects may provide firmer empirical evidence of generalizability; however, we are not convinced that it will bring to light any additional significant usability findings.

Funding

This study was supported by ERT.

Disclosure

The authors are employees of ERT. The authors report no other conflicts of interest in this work.