Usability of Conversational Agents by Patients with Inadequate Health Literacy: Evidence from Two Clinical Trials

Abstract

Embodied Conversational Agents (ECA) are computer-animated characters that simulate face-to-face conversation with patients. These agents can be programmed with best practices in human-human health communication and used for automated health education and behavior change counseling interventions. Evidence is presented from two ongoing clinical trials demonstrating that patients at different levels of health literacy find these agents acceptable and easy to use for automated health communication interventions. Innovative computer interface systems can be used to ensure that inadequate health literacy not serve as a barrier to interventions using health information technology.

There has been an explosion of interest and creativity in the field of health information technology, driven not only by the tremendous advantages of electronic medical records, but also by the great prospect for this technology to directly support patients for self-care and health behavior change. Research in this area has accelerated over the course of the past 25 years; however, the potential health benefits of this technology have not been realized. Two of the chief reasons for this are related to accessibility and usability. If patients cannot acquire the technologies or use them correctly, there is little possibility that such advances could lead to improvement in clinical outcomes.

Indeed, it is likely that current advances in patient-facing health information technology will exacerbate health disparities, as the benefits of such technologies will disproportionately accrue to the wealthiest, most educated, and technologically advanced members of society (Norman & Skinner, 2006; Bodie & Dutta, 2008). In particular, patients with inadequate health literacy are likely to be particularly vulnerable in this regard. People with inadequate health literacy are much less likely to use computers and have difficulty processing health information (Kutner, Greenberg, Jin, & Paulsen, 2006). Addressing disparities in access and usability is thus an essential element of addressing health disparities in general.

We have developed a computer interface—called an Embodied Conversational Agent (ECA)—that is usable by people with inadequate health literacy (Bickmore, Pfeifer, & Paasche-Orlow, 2009). The interface uses the universal and familiar format of face-to-face conversation, not just as an interface metaphor, but as the actual model of interaction. This is accomplished through the use of an animated character that talks to patients using synthetic speech and synchronized conversational nonverbal behavior, such as hand gestures, head nods, and eyebrow raises (Figure 1) (Cassell, Sullivan, Prevost, & Churchill, 2000). Patients talk to the character using touch screen input.

Figure 1 Embodied conversational agent interface for walking promotion trial.

Motivation for Using Embodied Conversational Agents with Inadequate Health Literacy Patients

Evidence suggests that face-to-face encounters with a health provider—in conjunction with written instructions—remain one of the best methods for communicating health information to patients in general, but especially those with inadequate health literacy (Qualls, Harris, & Rogers, 2002; Colcher & Bass, 1972; Madden, 1973; Morris & Halperin, 1979; Clark & Brennan, 1991). Face-to-face consultation is effective because it requires that the provider focus on the most salient information to be conveyed and that the information be delivered in a simple, conversational speaking style (Qualls et al., 2002). Protocols for “grounding” in face-to-face conversation—the use of verbal and nonverbal cues such as head-nods, gaze and acknowledgment tokens (“uh-huh,” “OK”) to signal mutual understanding (Clark & Brennan, 1991)—allows providers to dynamically assess a patient's level of understanding and repeat or elaborate information as necessary. Face-to-face conversation also allows providers to make their communication more explicitly interactive by asking patients to do, write, say, or show something that demonstrates their understanding (Doak, Doak, & Root, 1996). Finally, face-to-face interaction allows providers to use verbal and nonverbal behaviors, such as empathy (Frankel, 1995) and immediacy (Argyle, 1988; Richmond & McCroskey, 1995) to elicit patient trust, enabling better communication and satisfaction.

Given the efficacy of face-to-face consultation, Embodied Conversational Agents (ECA) show particular promise for conveying health information to patients with inadequate health literacy by simulating face-to-face conversation with a provider. These systems can produce verbal and nonverbal conversational behaviors that signify understanding, mark significance, and convey information in redundant channels of information (including speech intonation, hand gesture, facial display, body posture shift, and eye gaze), to maximize message comprehension. They can use the verbal and nonverbal communicative behaviors used by health providers to establish trust and rapport with their patients in order to increase satisfaction and adherence to treatment regimens (Bickmore, Gruber, & Picard, 2005). They can adapt their messages to the particular needs of patients and to the immediate context of the conversation, since each utterance by the agent is dynamically composed (not just pre-recorded). They can emulate clinicians' extensive use of pointing gestures when explaining written materials to patients in order to clarify references and describe the structure and layout of the text (as in Figure 2) (Bickmore, Pfeifer, & Yin, 2008). Finally, they can provide health information in a consistent manner and in a low-pressure environment in which patients are free to take as much time as they need to thoroughly understand it.

Figure 2 Embodied conversational agent interface in rehospitalization trial (photo Glenn Kulbako).

ECA-Based Health Intervention Clinical Trials

We are currently using the ECA interface in two randomized clinical trials that specifically examine the role of health literacy. In one of these trials, the ECA is being used to teach patients being discharged from the hospital about their after hospital care plan. In the other it is being used to promote walking in older adults. The goal of the current analysis is to evaluate data from these ongoing trials regarding the usability of the ECA system for people with inadequate health literacy.

The two clinical trials of ECA-based health interfaces are being conducted at Boston Medical Center, a large urban safety-net hospital and ambulatory care center with an ethnically diverse patient population. Both studies use an ECA-based computer interface to communicate health information to patients, modeling best practices in health communication for patients with inadequate health literacy.

In the current analysis we are focused on measures related to satisfaction, usability and other process measures (in both studies we are blinded to health outcomes until trial completion: 30-day hospital utilization in the rehospitalization study and walking steps and fitness in the walking study).

Rehospitalization Trial

The first trial, entitled, “A RCT to Reduce Cardiopulmonary Rehospitalization” (PI: Jack, R01HL081307) is a two-armed intervention trial to improve patient education and safety in the transition between hospital and home with a primary goal of reducing 30-day hospital readmission. The system is designed to be used by patients in their hospital beds. The agent is deployed on a wheeled kiosk with a touch screen display attached to an articulated arm that can be positioned in front of patients (Figure 2). The system is designed to interact with patients once each day they are in the hospital, but the primary interaction is just before hospital discharge (75% of patients only had this final, discharge interaction, due to short hospital stays or logistical constraints). The final interaction is performed after the final list of discharge medications are approved (“medication reconciliation”), and typically just before the patient leaves the hospital. In this interaction, patients spend approximately half an hour using the system, to review the layout and contents of a personalized “After Hospital Care Plan” booklet that is produced for them and contains their post-discharge self-care instructions. The paper booklet is given to patients before their conversation with the agent, and the agent displays and reviews a digital version of the patient's booklet in the interface, so that patients can follow along with the agent's explanation in their paper booklets to review medications, exercise and diet recommendations, and follow-up appointments. The specific approach to discharge education used in this project was modeled on our prior intervention—the Re-Engineered Hospital Discharge (RED)—which was delivered by a nurse (Jack, Chetty, & Anthony, 2009).

Rehospitalization Trial Methods

Participants

Participants in the rehospitalization study were English-speaking patients, 18 years of age or older, admitted to the teaching service of Boston Medical Center between October, 2008 and August, 2009. Patients were required to have a telephone, be able to comprehend study details and the consent process in English, and have plans to be discharged to a U.S. community. Patients were not enrolled if they were admitted from a skilled nursing facility or other hospital, admitted for a planned hospitalization, on hospital precautions, on suicide watch, deaf, or blind. Of the 417 participants enrolled to date into the parent study (of a planned 750), 208 were randomized into the ECA intervention arm of the study. Of these, 143 completed all measures necessary for our analyses (there were no significant differences in demographic characteristics between those who completed all measures and those who left the hospital prior to completing the study protocol).

Measures

Health Literacy

Health literacy was assessed using the Rapid Estimate of Adult Literacy in Medicine (REALM) (Davis, Long, & Jackson, 1993). The sample was split into “adequate” and “inadequate” health literacy groups, using a REALM score of 9th grade and above, as other authors have done (Lindau, Basu, & Leitsch, 2006; Mancuso & Rincon, 2006; Sudore, Yaffe, & Satterfield, 2006; Lincoln, Paasche-Orlow, & Cheng, 2006).

Usability

Usability was assessed using single scale-measure self-report items to assess overall satisfaction with the ECA, ease of use, desire to continue working with the ECA, and preference for the ECA relative to human health providers, in addition to session duration.

Attitudes Towards the Agent—Therapeutic Alliance

Patient attitude towards the ECA was assessed using a measure of therapeutic alliance, specifically the affective bond subscale of the Working Alliance Inventory. This is a self-reported 12-item Likert scale questionnaire assessing the emotional dimension of a patient's trust and belief that they can work together with a provider to achieve desired therapeutic outcomes (Horvath & Greenburg, 1989).

Procedure

Participants were provided with brief training on how to “talk” to the ECA, in which the agent walks on the screen and greets the participant; participants are then told to “touch what you want to say on the screen” (that is typically the extent of the training). At the conclusion of their interaction with the ECA they answered questions regarding usability and attitudes toward the gent just prior to leaving the hospital. All self-report measures were verbally collected by research staff to accommodate patients with limited literacy.

Rehospitalization Trial Results

Demographics and Health Literacy

Table 1 shows demographics of the study population. Participants with inadequate health literacy in the rehospitalization study were significantly older, less educated, and more likely to be non-White compared with participants in that study with adequate health literacy. Participants with inadequate health literacy also had significantly lower levels of computer literacy compared to participants with adequate health literacy.

Table 1. Subject demographics by health literacy level

Health literacy level	Inadequate	Adequate	p value
Rehospitalization Study
N	68	75
Sex (% Male)	56.9	50.0	n.s.
Age (range 20–84)	52.7	46.6	.004
Race: % African American	63.9	48.6
Race: % White	12.5	35.1	<.05
Race: % Other	23.6	16.3
% Hispanic or Latino	13.9	10.8
Highest grade completed	11.6	12.9	.002
Computer Literacy (1 = never use one; 4 = expert)	2.01	2.73	<.001
Walking Study
N	15	18
Sex (% Male)	33.3	22.2	n.s.
Age (range 65–85)	73.0	73.5	n.s.
Race: % African American	86.7	61.1
Race: % White	6.7	22.2	n.s.
Race: % Other	6.6	16.7
% Hispanic or Latino	20.0	0.0
Highest grade completed	11.3	13.3	n.s.
Computer Literacy	1.4	1.8	.08

Usability

Participants reported very high levels of overall satisfaction and ease of use, regardless of health literacy level: 78% of all participants scored satisfaction a 7 on a 7-point Likert-type scale (with 7 = “very satisfied”), and 78% scored ease of use a 1 on a 7-point scale (with 1 = “very easy to use”). In addition, participants with inadequate health literacy showed a trend of greater satisfaction with the ECA compared to participants in that study with adequate health literacy.

None of the other usability measures were significantly different across health literacy levels.

Attitudes Toward the Agent

Participants scored well above the Likert scale midpoint on overall mean Working Alliance Bond subscale scores, regardless of health literacy level, and only 11% of participants scored below the midpoint of the composite measure. In addition, there were no significant differences between literacy groups on overall Working Alliance scores. However, differences on a few of the individual items in the scale reached significance, indicating a greater degree of personification of the agent (mutual respect, importance of relationship with the agent) by participants with inadequate health literacy.

Geriatrics Walking Promotion Trial

The second trial, entitled, “Computer Agents to Promote Walking in Older Adults with Low Health Literacy” (PI: Silliman, R01AG028668) is a two-armed intervention trial to promote walking in older adults with a primary goal of improving the number of steps per day at 12 months. Older adult ambulatory clinic patients at Boston Medical Center are given pedometers which link to tablet-PC computers. Intervention participants are given a tablet-PC to use at home for 2 months and are asked to interact with the ECA daily to set and discuss walking goals (Figure 1). In addition, participants can interact with the agent on a kiosk in the waiting room of their primary care provider.

Walking Trial Methods

Participants

Participants in the walking study were English-speaking patients, 65 years or older, who attend the geriatrics or internal medicine ambulatory care clinics at Boston Medical Center between April, 2009 and September, 2009. Patients were required to speak and read English at a level required to interact with the ECA (via a screening conversation with the agent) and to understand the study protocol, be inactive but medically able to begin a moderate intensity physical activity program, and free of cognitive impairment and significant depressive symptoms. Of the 88 participants enrolled to date into the parent study (of a planned 270), 44 were randomized into the ECA intervention arm of the study and 2-month study measures were obtained from 33 of these.

Measures

Health Literacy

Health literacy was assessed using the Test of Functional Health Literacy in Adults (TOFHLA) (Parker, Baker, Williams, & Nurss, 1995). A different measure was used relative to the rehospitialization trial due to the different patient populations and study settings. Patients with subclinical dementia can often pronounce a word correctly but not know what the word means, invalidating REALM results. As this is more likely to occur in older cohorts, we chose to avoid the REALM in the walking study, and used the TOFHLA. However, as the TOFHLA takes more time to administer, it was not the best choice for rushed hospital environments, especially those with relatively younger adult populations. Both of these measures reflect print literacy and reading ability (Berkman, Pignone, Sheridan, & Lohr, 1994) and so may not be the most accurate assessments of ability to act on health information communicated verbally.

The sample was split into adequate and inadequate health literacy groups, using a TOFHLA score of 23 or above, as other authors have done (Lindau et al., 2006; Mancuso & Rincon, 2006; Sudore et al., 2006; Lincoln et al., 2006).

Usability

Usability was assessed through actual voluntary use of the system during the first two months in which patients had the tablet computer at home, based on the tablet log files. Measures included the number of sessions completed out of 60 possible daily conversations, the average duration of each session, and the percent of sessions in which participants plugged in their pedometer (the agent asked them to plug it in every session).

Attitudes Towards the Agent—Therapeutic Alliance

Attitude towards the agent was assessed using the affective bond subscale of the Working Alliance Inventory, as in the rehospitalization trial (Horvath & Greenberg, 1989).

Procedure

Participants were provided with the same brief ECA training as in the rehospitalization study, given at time of enrollment, before being sent home with a tablet computer for two months of home-based interactions with the agent. Assessments of attitudes toward the ECA were administered at an in-person research interview immediately following these 2 months. All self-report measures were verbally collected by research staff to accommodate patients with limited literacy.

Analysis

In order to examine the trends in participant use of the system over time, we analyzed the sessions data using mixed-effect modeling. All analysis was performed using R 2.9.0 (R Development Core Team, 2008) with the “nlme” package, fitting linear mixed-effect regression models to the sessions per week and literacy category data. Best fit results were for a model with random effects for intercept but not study week (slope).

Walking Trial Results

Demographics and Health Literacy

As in the rehospitalization study, participants with inadequate health literacy had lower levels of computer literacy compared to participants with adequate health literacy, although this difference was only trending towards significance, likely due to the smaller sample size (Table 1).

Usability

Mixed effect regression indicates that participants with inadequate health literacy completed fewer home-based conversations with the ECA compared to participants with adequate health literacy (p < .05), Note that a simple t-test on total number of sessions also shows this result (approaching significance, Table 2). Regression results also indicate a significant decrease in home-based conversations with the ECA over time for all participants of −0.29 sessions/week (p < .001). There was no significant interaction between sessions per week and literacy category; participants had similar patterns of decreasing use over time, regardless of literacy category.

Table 2. Outcomes by health literacy level

Health literacy level	Inadequate	Adequate	p value
Rehospitalization Study
Satisfaction (1 = not at all; 7 = very much)	6.57	6.45	.083
Ease of Use (1 = very easy; 7 = very difficult)	1.82	1.83	n.s.
Desire to Continue with Agent (1 = not at all; 7 = very much)	5.82	5.39	n.s.
Prefer Human Provider over Agent (1 = definitely prefer doctor or nurse; 7 = definitely prefer agent)	4.50	4.12	n.s.
Average session time (minutes)	31.62	27.38	n.s.
WAI* Bond (overall composite)	5.80	5.49	n.s.
I am comfortable with the agent.	5.58	5.78	n.s.
The agent and I understand each other.	5.67	5.68	n.s.
The agent likes me.	5.50	5.29	n.s.
The agent is concerned about my welfare.	6.16	5.64	n.s.
The agent and I respect each other.	6.24	5.59	.027
The agent is honest about her feelings towards me.	4.83	5.29	n.s.
I am confident in the agent's ability to help me.	6.43	6.20	n.s.
The agent appreciates me.	5.97	5.52	n.s.
The agent and I trust one another.	5.68	5.32	n.s.
My relationship with the agent is important to me.	5.82	4.99	.012
The agent cares about me, even if I do something wrong.	5.24	4.74	n.s.
The agent will keep working with me, even if I say something wrong.	5.76	5.81	n.s.
Walking Study
Sessions completed (of 60 possible)	26.73	38.39	.078
Average time per session (minutes)	7.49	7.67	n.s.
Sessions with pedometer uploads (percent)	64.00	83.55	.058
WAI* Bond (overall composite)	5.71	5.24	n.s.
I am comfortable with the agent.	5.67	4.94	n.s.
The agent and I understand each other.	6.20	4.83	.015
The agent likes me.	5.93	5.93	n.s.
The agent is concerned about my welfare.	5.93	5.39	n.s.
The agent and I respect each other.	6.20	5.28	n.s.
The agent is honest about her feelings towards me.	5.60	5.11	n.s.
I am confident in the agent's ability to help me.	6.20	5.50	n.s.
The agent appreciates me.	5.67	5.22	n.s.
The agent and I trust one another.	5.60	5.22	n.s.
My relationship with the agent is important to me.	5.60	5.06	n.s.
The agent cares about me, even if I do something wrong.	5.93	5.11	n.s.
The agent will keep working with me, even if I say something wrong.	3.73	5.61	.011

There were no significant differences in session durations between literacy categories, but there was a trend for participants with adequate health literacy to plug in their pedometers more frequently compared to participants with inadequate health literacy (p = .058, Table 2).

Attitudes Toward the Agent

As in the rehospitalization study, participants scored well above the Likert scale midpoint on overall mean Working Alliance Bond subscale scores, regardless of health literacy level, and only 9% of participants scored below the midpoint of the composite measure. In addition, there were no significant differences between literacy groups on overall Working Alliance scores. However, differences on a few of the individual items in the scale reached significance, indicating a greater degree of personification of the agent (mutual understanding) and a lower level of understanding of the technology (thinking the agent would discontinue use if the participant said the wrong thing) by participants with inadequate health literacy.

Discussion

Overall, there were very few differences in measures of acceptance and usability between patients with adequate and inadequate health literacy, suggesting that ECAs are approachable and usable by patients regardless of health literacy level. In the few measures in which there were significant or near-significant differences on health literacy, these were mostly in favor of patients with inadequate health literacy, indicating that ECAs may be even more acceptable to this population than to patients with adequate health literacy.

In addition to the theoretical reasons why ECAs may be ideal interfaces for patients with inadequate health literacy, described earlier, patients interviewed in the pilot studies that preceded the two trials provided a better understanding of their reasons for accepting the technology (Bickmore, Pfeifer, & Jack, 2009; Bickmore, Caruso, Clough-Gorr, & Heeren, 2005). Patients in both pilots indicated that the system was very easy to use, even if they had little or no experience with computers:

	“I don't like computers but that was easy.” (rehospitalization pilot)
	“That is so easy. That is so good. Regular computers I don't do. But, that was so easy, even a baby could do that.” (walking pilot)

Patients in the rehospitalization pilot indicated that they liked being able to take as much time as they needed to understand everything, something they said that their doctors or nurses typically did not provide:

	“Sometimes doctors just talk and assume you understand what they're saying. With a computer you can go slow, go over things again and she checks that you understand.”
	“I prefer Louise [the name of the ECA character], she's better than a doctor, she explains more, and doctors are always in a hurry.” Patients in both pilots were also mostly positive about the interventions:
	“It was the best thing that happened to me, to have something that pushed me out and get me walking.” (walking pilot)
	“She's nice. She's really good. Really good. She asks you the right questions. She tells if you if you're not doing up to par, you know, and all that. And if you're doing good, she'll tell you. If you're not she'll tell you. And it's honest. And it works. It really does. I like it. I like talking to her.” (walking pilot)
	“She treated me like a real person! She's not like a computer. This is awesome work! This is really excellent.” (rehospitalization pilot)
	“I've had problems with, not this hospital, but other hospitals. I wasn't given the quality time that this lady gave me.” (rehospitalization pilot)

One area of possible concern is that patients with adequate health literacy in the walking trial completed more sessions with the ECA compared to patients with inadequate health literacy. This may indicate that, despite having similar attitudes towards and satisfaction with the agent and despite finding the system easy to use, there may be other important factors such as patient activation that dictate the amount of use. However, the relationship between intervention dose and health outcomes in behavioral studies can be complex, and it could even be that fewer sessions result in better outcomes. The ECA provides an accessible and usable communication channel for patients irrespective of health literacy, but more research is required to ascertain contexts in which dose is important and then to tailor information and counseling dialogue content to ensure that a given intervention is effective for patients with inadequate health literacy.

Patients with inadequate health literacy appear to anthropomorphize ECAs more than patients with adequate health literacy, as reflected by specific items related to mutual understanding and respect, and belief that the agent may decide to stop working with them if they say something wrong. Although this indicates a general lack of understanding of the underlying technology, it may ultimately prove beneficial for these patients if the increased personification leads to a greater sense of working alliance and increased adherence to the ECA's recommendations. Some patients may actually confuse the agent with a real person (e.g., if delirious in the hospital), which could be partially addressed by having both the humans administering the agent and the agent itself periodically remind users that it is just a computer. Another concern is that the results may indicate that patients with adequate health literacy do not like the social aspects of the interactions, feeling that they are unnecessary, slow, or even disingenuous. Future systems may allow patients to choose more conventional graphical user interfaces that let them work through the information in a session more efficiently.

Future Work

Our immediate future plans are to complete the rehospitalization and walking trials in order to demonstrate efficacy—in terms of clinically important health outcomes—regardless of health literacy level.

Now that we have established that ECAs can provide an acceptable and usable health communication channel for patients with inadequate health literacy, the opportunities for developing patient and consumer education and counseling interventions are limitless. Specific areas that we are investigating include:

	Automated explanation of written medical information to patients with varying levels of health literacy (Bickmore, Pfeifer, & Paasche-Orlow, 2009).
	Linguistically and culturally tailored health interventions, such as exercise promotion for older bilingual Latino adults (Yin, Bickmore, Byron, & Cortes, 2010).
	Longitudinal health behavior change interventions, in which alliance with the ECA is used to promote retention in the intervention as well as adherence (Bickmore, Schulman, & Yin, 2010).
	Deployment on other computer platforms, including mobile devices (Bickmore & Mauer, 2009).

A final important area of ongoing research is the automatic adaptation of the computer interface in response to patient characteristics and needs. Our finding that patients with high levels of computer literacy are less satisfied with the ECA may indicate that such patients should be given the option of using a more traditional computer interface to more efficiently access the information they need, while patients with low computer and/or health literacy would use the ECA. In addition, in some of our studies we have found that nurses provide different information to patients depending on their level of health literacy—providing more technical detail to patients with adequate health literacy, but providing more scaffolding (information about document structure) to patients with inadequate health literacy (Bickmore, Pfeifer, & Yin, 2008)—and this difference in presentation could also be emulated by an ECA that dynamically adjusts its dialogue based on patient needs.

Notes

Health literacy assessed via REALM for Rehospitalization Study and TOFHLA for Walking Study.

All t-tests except Satisfaction (Mann-Whitney due to ceiling effect).

*WAI: Working Alliance Inventory (all items Likert scale, 1 = disagree completely; 7 = agree completely).