Understanding the Internal and External Validity of Health Literacy Interventions: A Systematic Literature Review Using the RE-AIM Framework

Abstract

We conducted a systematic literature review, using the RE-AIM framework, with the goal of determining what information is available to inform research to practice translation of health promotion interventions developed to address health literacy. Thirty-one articles reflecting 25 trials published between 2000 and 2010 met inclusion criteria. Two researchers coded each article, using a validated RE-AIM (reach, effectiveness/efficacy, adoption, implementation, maintenance) data extraction tool, and group meetings were used to gain consensus on discrepancies. Across all studies (14 randomized controlled trials, 11 quasi-experimental; 24 clinic-based, 1 community-based), the mean level of reporting RE-AIM indicators varied by dimension (reach = 69%; efficacy/ effectiveness = 58%; adoption = 36%; implementation = 35%; maintenance = 11%). Among participants enrolled in the 25 interventions, approximately 38% were identified as low health literate. Only eight of the studies examined health literacy status as a moderator of intervention effectiveness. This review suggests that the current research on health promotion for participants with low health literacy provides insufficient information to conclude whether interventions for health literacy can attract the target population, achieve an effect that is sustainable, or be generalized outside of clinical settings. Recommendations for enhancing the design and reporting of these trials are provided.

There is a large body of literature that indicates that low health literacy and health are related (Berkman et al., 2004, Berkman et al., 2011). This relationship exists across health care utilization (e.g., related to decreased use of various screening procedures and increased use of emergency services), health knowledge (e.g., related to less awareness of health consequences of chronic disease), lifestyle behaviors (e.g., more likely to smoke and less likely to adhere to medication protocols), and health outcomes (e.g., lower global health status). Indeed, health literacy, defined as “the degree to which individuals have the capacity to obtain, process, and understand basic health information” is a better predictor of health status than age, income, employment, ethnicity, or education level (American Medical Association, 2004). Further, it is estimated that 87 million (36%) of adults in the United States are at basic or below basic health literacy levels and that the annual costs of low health literacy range from $106 billion to $238 billion (Vernon, Trujillo, Rosenbaum, & DeBuono, 2007).

In response to the prevalence, health outcomes, and costs associated with low health literacy, the issue has been on the national health care agenda for well over a decade and has generated a vast amount of research. A number of systematic reviews of the literature have been completed, including a seminal report published by the Agency for Healthcare Research and Quality in 2004. That review concluded that there was a strong link established between health literacy and health outcome, but that the 29 intervention studies that attempted to mitigate the effects of low health literacy on health were of only fair quality and were equivocal on intervention effectiveness. Reviews of literature that appeared shortly after the AHRQ report highlighted similarly mixed findings and concluded that the body of literature needed stronger research designs, more comprehensive interventions, and consistency in measurement and analyses of the influence of health literacy on health outcome (Berkman et al., 2004; Santo, Laizner, & Shohet, 2005).

More recent reviews of health literacy-related intervention research focused on experimental research designs and reported on the effectiveness of interventions to change knowledge, social cognitions, patient and provider satisfaction, and clinical outcomes. Although there were notably positive trials related to improving clinical outcomes (DeWalt et al., 2004; Weiss, Francis, Senf, & Heist, 2006), the primary outcomes that demonstrated improvement included health knowledge, some health behavior change, and social cognitions (Clement, Ibrahim, Crichton, Wolf, & Rowlands, 2009). Recommendations from these reviews include examining potential best practices, improving the internal validity of trials, using theoretical approaches for intervention development, and comparing simple versus more complex interventions (Berkman et al., 2011; Santo et al., 2005; Schaefer, 2008).

Each of these previous reviews has primarily focused on the internal validity of these studies. Specifically, they have attempted to provide sufficient evidence of cause and effect between the independent and dependent variable while ruling out plausible alternative explanations (Shadish, Cook, & Campbell, 2002). However, these review articles have also commented on the potential lack of generalizability of health literacy interventions when considering different populations, settings, or even nations. However, to date no review of health literacy literature has systematically addressed nor reported on issues related to external validity. External validity is defined as the generalizability of results to or across target populations, settings, times, and so forth (Shadish et al., 2002). In other behavioral intervention domains (e.g., physical activity, nutrition, tobacco use, weight management), there have been a number of reviews of literature using the RE-AIM framework (Glasgow, Vogt, & Boles, 1999)—a set of criteria to assess the reporting on outcomes related to both internal and external validity—to demonstrate the degree to which researchers were reporting on external validity issues. (Akers, Estabrooks, & Davy, 2010; Bull, Gillette, Glasgow, & Estabrooks, 2003; Dzewaltowski, Estabrooks, Klesges, Bull, & Glasgow, 2004; Estabrooks, Dzewaltowski, Glasgow, & Klesges, 2002; Glasgow, 2007; Glasgow, Klesges, Dzewaltowski, Bull, & Estabrooks, 2004; Glasgow, Nelson, Strycker, & King, 2006; Klesges, Dzewaltowski, & Glasgow, 2008; White, McAuley, Estabrooks, & Courneya, 2009). The RE-AIM framework includes the assessment of:

• R each (i.e., the number, proportion, and representativeness of study participants),

• E ffectiveness (i.e., changes in primary study outcomes, quality of life, and potential negative effects),

• A doption (i.e., the number, proportion, and representativeness of settings and staff who agree to deliver an intervention),

• I mplementation (i.e., the degree to which an intervention was delivered as intended and the cost of implementation), and

• M aintenance (i.e., extent to which an intervention and its effects are sustained over time).

The purpose of this paper is to present the findings of a RE-AIM review in order to determine the degree to which health literacy interventions targeting disease self-management and health promotion reported on internal and external validity indicators that could inform research to practice translation and to identify methodological gaps related to research design, evaluation, and reporting.

Methods

Selection of Studies for Review

Articles that were published between 2000 and February 2010 and that met the inclusion criteria illustrated in Table 1 were identified. This review focused on health literacy and behavioral interventions targeting disease self-management and health promotion. Therefore, as illustrated in Table 1, we focused on experimental and quasi-experimental designs with the key criteria being some comparator, whether pre-post or a comparison group. Cross-sectional, descriptive, and purely methodological articles were excluded, as well as single-session studies that focused on comprehension and knowledge gains assessed in a same day follow-up. Finally, because the potential for health interventions to initiate behavior change is a key focus of this review, articles that reported on brief interventions with a follow-up one week or sooner after the intervention were excluded.

Table 1. Inclusion criteria for articles

Data type	Inclusion criteria
Participants	• >17 years of age
Study design	• Experimental and quasi-experimental
Control condition	• Any comparator including active control, inactive control, or pre- and postmeasure
Intervention	• Mention of literacy, health literacy, or numeracy in description of the intervention• Targets disease self-management and/or health promotion
Measurement	• Assesses literacy, health literacy, or numeracy directly among participants• Follow-up period one week or greater after the intervention was delivered
Setting	• Any
Primary outcome(s) (at least one of these outcomes)	• Clinical outcomes• Health literacy or numeracy• Health behavior• Proposed theoretical mediators• Self-reported health status or quality of life• Utilization of health care• Satisfaction with interventions• Health professional behavior or communication skills

The literature search was conducted from September 2009–February 2010. Three primary search strategies were used to identify potentially relevant articles including: (1) queries with search engines, (2) reverse citation of Rapid Estimate Assessment of Literacy in Medicine (REALM) and Test of Functional Health Literacy Assessment (TOFHLA) validation articles (Davis et al., 1991; Parker, Baker, Williams, & Nurss, 1995) and (3) hand searches from the reference lists of five health literacy reviews (Berkman et al., 2004; Berkman et al., 2011; Clement et al., 2009; Santo et al., 2005; Schafer, 2008). The PubMed and Web of Science search engines that were used included the databases Science Citation Index Expanded, Social Sciences Citation Index, Arts & Humanities Citation Index, and MEDLINE. The search terms used were health literacy, numeracy, and intervention(s). Since REALM and TOFHLA are two of the earliest developed and most widely used metrics to assess health literacy, reverse citation of these validation articles was an important element of the search strategies (Davis et al., 1991; Parker et al., 1995). Finally, to thoroughly evaluate the degree of RE-AIM reporting across a series of related papers reporting on a single trial, for any article meeting inclusion criteria we also extracted data from companion papers describing different aspects of the same study (e.g., methods paper, cost-effectiveness paper).

The initial search yielded 587 articles that potentially met inclusion criteria (see Figure 1). The titles and abstracts of the articles were screened by two researchers (JZ and KA) and 519 were excluded for not meeting inclusion criteria (e.g., cross-sectional design; no measure of health literacy among participants). The full-texts of 72 papers were independently assessed against the inclusion criteria and an additional 41 papers were excluded for reasons detailed in Figure 1. Of note, we considered all validated and direct measures of health literacy and found that studies included in our review used REALM (Bosworth et al., 2005, 2008; Bosworth, Olsen, Dudley, et al., 2009; Bosworth, Olsen, Grubber, et al., 2009; Brock & Smith, 2007; Cavanaugh et al., 2009; Coleman et al., 2003; Davis et al., 2008; DeWalt et al., 2004; Ferreira et al., 2005; Gazmararian, Jacobson, Pan, Schmotzer, & Kripalani, 2010; Rothman, DeWalt, et al., 2004; Rothman, Malone, et al., 2004; Rothman et al., 2005), modified REALM (van Servellen et al., 2003; van Servellen et al., 2005), TOFHLA (Cordasco et al., 2009; Yin et al., 2008), S-TOFHLA (DeWalt et al., 2004; 2006; 2009; Gerber et al., 2005; Handley, Shumway, & Schillinger, 2008; Kim, Love, Quistberg, & Shea, 2004; Murray et al., 2007; Ntiri & Stewart, 2009; Paasche-Orlow et al., 2005; Schillinger et al., 2008; Schillinger, Handley, Wang, & Hammer, 2009; Seligman et al., 2005; Wallace et al., 2009), and A-TOFHLA (Rudd et al., 2009). In total, 31 articles representing 25 trials were included in this systematic review.

Figure 1 Selection of health literacy intervention studies for systematic review. (¹Contact lead author for complete citations of 41 reviewed, but excluded articles.)

RE-AIM Criteria

A 21-item validated data extraction tool based on the RE-AIM Framework was used to code eligible articles on the degree to which indicators of internal and external validity were reported (Akers et al., 2010; Dzewaltowski et al., 2004; Estabrooks et al., 2002; Glasgow et al., 2004). In addition, as described below, we added 10 exploratory indicators across the reach, effectiveness, adoption, and implementation dimensions.

Reach

Articles were coded for information on the method used to identify the target population, inclusion criteria, exclusion criteria, participation rate (i.e., sample size/the number of eligible participants exposed to recruitment activities), and representativeness (i.e., comparison of participants and eligible nonparticipants on demographics and key outcomes). In addition to the items included in the original validation tool, we also coded articles for including information on presence or absence of a health literacy-based inclusion criteria and the proportion of participants with low health literacy.

Efficacy/Effectiveness

Effectiveness is defined as a test of whether a “program does more good than harm delivered under real-world conditions” while efficacy is defined as a test of whether a “program does more good than harm delivered under optimum conditions” in a controlled setting (Flay, 1986). Indicators for this dimension included the assessment of changes in the primary outcome, the completion of analyses on those present at follow-up versus an intention-to-treat approach, the presence of a quality-of-life measure, and the reporting of percentage attrition. Including quality-of-life measures can provide a critical participant-centered check on the impact of delivery practices and an opportunity to better understand standardized effects across a diverse range of studies. Again, we included additional criteria related to the purpose of this review, such as whether the researchers reported trial type (i.e., efficacy versus effectiveness; single or multileveled). Finally, we also coded whether an article assessed changes in health literacy scores or completed any mediation or moderation analyses that would help identify the robustness and mechanisms of intervention effects. As defined by Baron and Kenny (1986) a moderator variable is one that alters the strength of the casual relationship, whereas a mediator variable is on the causal pathway and alters the effect of the independent variable on the dependent variable. Using health literacy as an example, moderation examines the robustness of effects across different levels of health literacy (does the intervention work in the same way for high and low health literate clients). Mediation implies a causal sequence where an intervention may change health literacy, and this change in health literacy results in improved health outcomes.

Adoption

Indicators for the reporting of adoption were assessed at both the setting and staff levels. Specifically, articles were reviewed to identify the extent to which they provided a description of intervention location, the staff who delivered intervention, the methods used to identify and target the staff who delivered the intervention (delivery agent), and the level of expertise of those who delivered the intervention. Further, the degree to which an article provided information on setting or staff inclusion and exclusion criteria as well as rate of delivery agent or setting adoption was coded. We also added an indicator to the coding tool that reflected whether or not the authors reported on the cost of adoption (i.e., start-up costs) of the intervention.

Implementation

Articles were assessed to determine if they reported on the duration and frequency of the intervention, the extent the protocol was delivered as intended, and the cost of delivery. To determine the degree to which other important implementation factors were reported across articles, we added indicators for reporting the theoretical basis of the intervention and the methods used to address low literacy. Further, we coded whether or not authors reported on the degree to which the participants received intervention components.

Maintenance

Reporting of individual-level maintenance was coded as present if the study included assessments of intervention outcomes at six or more months after the completion of individual participation in the interventions. Six months is widely accepted as the demonstrated sustainability time frame in the health behavior change field (Glasgow et al., 1999; Prochaska & DiClemente, 1983; Prochaska & Velicer, 1997). At the organizational level, studies were coded on whether or not they reported that the intervention was sustained after research activities were completed and on any costs associated with maintenance.

Coding Protocol and Scoring

All studies that met inclusion criteria were abstracted and coded independently by two members of the research team. Specifically, each reviewer coded a “yes” or “no,” reflecting the presence or absence of the RE-AIM indicators outlined above. In addition, appropriate data was extracted from the articles when a given indicator was reported. Following the individual coding of each article, the research team met to discuss articles, resolve uncertainty, and gain consensus in coding. Analyses included providing count and percentage data across RE-AIM indicators. To determine the overall quality of RE-AIM reporting, we also examined the degree to which articles reported across the 21 items from the validated data extraction tool (see Table 2). Specifically, quality was assessed based on tertile cut-points of reporting 15–21, 8–14, or 0–7 dimensions reflecting high, moderate, and low quality, respectively.

Table 2. Proportion of health literacy interventions reporting RE-AIM dimensions and components (n = 25 trials)

RE-AIM dimensions and components	Proportion reporting*
Reach
Method to identify target population	100.0
Inclusion criteria	96.0
Exclusion criteria	76.0
Participation rate	44.0
Representativeness	28.0
Average across Reach Components	68.8
Efficacy/effectiveness
Measures/results for at least one follow-up	100.0
Intent-to-treat analysis utilized	32.0
Quality-of-life or potential negative outcomes	22.0
Percent attrition	76.0
Average across Efficacy/Effectiveness Components	57.5
Adoption
Description of intervention location	100.0
Description of staff who delivered intervention	12.0
Method to identify staff who delivered intervention (target delivery agent)	4.0
Level of expertise of delivery agent	72.0
Inclusion/exclusion criteria of delivery agent or setting	18.0
Adoption rate of delivery agent or Setting	8.0
Average across Adoption Components	36.0
Implementation
Intervention duration and frequency	72.0
Extent protocol delivered as intended (%)	20.0
Measures of cost of implementation	12.0
Average across Implementation Components	35.0
Maintenance
Assessed outcomes ≥6 months post intervention	20.0
Indicators of program-level maintenance	8.0
Measures of cost of maintenance	4.0
Average across Maintenance Components	11.0

Results

Our search yielded 31 articles, from which merging papers yielded 25 unique studies. Of those, 14 were randomized controlled trials (Bosworth et al., 2005, 2008; Bosworth, Olsen, Dudley, et al., 2009; Bosworth, Olsen, Grubber, et al., 2009; Cavanaugh et al., 2009; Cordasco et al., 2009; DeWalt et al., 2006; Ferreira et al., 2005; Gerber et al., 2005; Handley, Shumway, & Schillinger, 2008; Murray et al., 2007; Rothman, DeWalt, et al., 2004; Rothman et al., 2005; Rudd et al., 2009; Schillinger et al., 2008; Schillinger et al., 2009; Seligman et al., 2005; van Servellen et al., 2003; van Servellen et al., 2005; Yin et al., 2008) and 11 were quasi-experimental studies (Brock & Smith, 2007; Coleman et al., 2003; Davis et al., 2008; DeWalt et al., 2004; 2009; Gazmararian, Jacobson, Pan, Schmotzer, & Kripalani, 2010; Kim et al., 2004; Ntiri & Stewart, 2009; Paasche-Orlow et al., 2005; Rothman, Malone, et al., 2004; Wallace et al., 2009). The average number of RE-AIM indicators reported was 9.0 (±1.8) out of 21, with a range of 4 to 17 indicators reported. One trial was rated as high quality (Handley et al., 2008; Schillinger et al., 2008; Schillinger et al., 2009) while 18 and six, respectively, were rated as moderate and low quality. The majority of the studies were conducted in a single site, and most interventions were delivered in clinical settings, with the exception of one study delivered in a community setting (Ntiri & Stewart, 2009).

Reach

The overall proportion of reported indicators of RE-AIM dimensions was highest for reach at 68.8% (Table 2). Typical information relative to the internal validity of the data, such as reporting of specific inclusion and exclusion criteria, was quite high, as was the proportion that reported on the methods used to identify the target population. However, the description of the methods varied greatly across studies from single-sentence descriptors to thorough reporting of the protocol used. All studies provided a sample size and these ranged from 20 to over 600 participants with a median of 217. When considering components that align with external validity and impact generalizabilty, there was a substantial drop in the proportion of articles reporting participation rate and representativeness. Providing a range and median for participation rate is problematic as the operational definitions used to determine a denominator varied widely (e.g., those who responded to recruitment activities versus those who were exposed to recruitment). The indicators added specifically for this review indicated that only two of the 25 studies applied an inclusion criterion to ensure that participants had a low health literacy status (DeWalt et al., 2004; Seligman et al., 2005). Furthermore, estimates revealed that only about 38% of participants across these 25 interventions had low health literacy.

Efficacy/Effectiveness

The overall proportion reporting for efficacy/effectiveness was 57.5% (Table 2). Similar to the reach dimension, articles included strong reporting of factors associated with internal validity, such as changes in the primary outcome and percent attrition. The average attrition rate was around 14% and ranged from 2% to 39%. Of the studies that reported participants with high attrition (25% or higher), only one out of five reported using an intent-to-treat analysis. Seventeen (68%) trials reported on present-at-follow-up data, three reported on intent-to-treat analysis, and five studies reported on intent-to-treat and present-at-follow-up data. Of the studies that reported attrition rates, eight compared characteristics of dropouts with those who remained in the study. From those eight studies, five studies reported significantly different characteristics.

Coding showed that only about one in five studies reported whether the focus was on efficacy or effectiveness. Of trials that reported, two indicated that they were efficacy trials (Davis et al., 2008; R. L. Rothman et al., 2004; R. L. Rothman et al., 2005) and four indicated that they were effectiveness trials (DeWalt et al., 2004; Handley et al., 2008; Kim et al., 2004; Schillinger et al., 2008; Schillinger et al., 2009; van Servellen et al., 2003; van Servellen et al., 2005). Only two studies reported on multileveled interventions that included strategies for both the health professional and patient (Davis et al., 2008; Gazmararian et al., 2010), one focused solely on the health professional (Seligman et al., 2005), and the remainder were individual patient-level interventions. The types of control conditions included 16 trials that reported using usual care (Bosworth et al., 2005, 2008; Bosworth, Olsen, Dudley, et al., 2009; Bosworth, Olsen, Grubber, et al., 2009; Cavanaugh et al., 2009; Cordasco et al., 2009; DeWalt et al., 2006; Ferreira et al., 2005; Gazmararian et al., 2010; Handley, et al., 2008; Murray et al., 2007; Rothman, DeWalt, et al., 2004, Rothman et al., 2005; Rudd et al., 2009; Schillinger et al., 2008; Schillinger et al., 2009; Seligman et al., 2005; van Servellen et al., 2003; van Servellen et al., 2005; Yin et al., 2008) and three used a minimal intervention control (Bosworth, Olsen, Grubber et al., 2009; Gerber et al., 2005; Rudd et al., 2009).

Only two trials examined changes in health literacy as a result of their interventions, and both were successful in improving health literacy (Nitiri & Stewart, 2009; van Servellen et al., 2003; van Servellen et al., 2005). Only one study reported analyses to determine mediators of intervention effects (Paasche-Orlow et al., 2005), while eight studies reported the influence of health literacy as moderator on outcomes (DeWalt et al., 2006; Ferreira et al., 2005, Gerber, 2005; Kim et al., 2004; Paasche-Orlow et al., 2005; Rothman, DeWalt, et al., 2004; Rothman, Malone, et al., 2004; Rothman et al., 2005; Wallace et al., 2009). Results were mixed with regard to the degree to which health literacy status influenced intervention effectiveness.

Adoption

The average percentage of reporting for the adoption elements across all studies was 36%. All studies provided a description of the intervention location. The next most commonly reported adoption element was the level of expertise of the delivery agent (72%). In contrast, few studies reported on the description of intervention staff (12%), method to identify target delivery agent (4%), inclusion/exclusion criteria of delivery agent (18%), adoption rate of delivery agent or setting (8%), and information on intervention start-up costs (4%).

Implementation

The mean level of reporting across implementation components was 35%. The most commonly reported implementation element was intervention duration and frequency (72%). Only five of 27 studies reported on the extent protocol delivered as intended, and even fewer studies (n = 3) reported on measures of implementation costs. Approximately a third of the studies reported the theoretical basis of the intervention and the majority of these used social cognitive theory. All of the studies reported on the methods used to address health literacy in their intervention. The most frequent methods cited were an emphasis on verbal communication, the use of concrete, simplified explanations, pictorial depictions, and the teach-back method. Finally, more than half of the studies included a description of the degree to which participants received the intervention. Methods to assess participant receipt typically included monitoring class attendance or the completion of intervention telephone calls.

Maintenance

Among the RE-AIM dimensions, maintenance indicators were reported the least (11%). Only one study reported on measures related to cost of maintenance, and rarely did studies report on outcomes assessed at longer than or equal to 6 months following the intervention (n = 5) or indicators of program-level maintenance (n = 2).

Discussion

The overarching goal of this paper was to provide an initial understanding of the state of research on health literacy interventions targeting disease self-management and health promotion from both an internal and external validity perspective. Within the context of the RE-AIM framework, the health literacy intervention studies included in this review share a number of similarities with, and a few distinctions from, the intervention literature that targets smoking, physical activity, nutrition, and weight-loss maintenance (Akers et al., 2010; Glasgow et al., 2004; White et al., 2009). When considered within the context of health literacy-related intervention research, our review also filled a number of gaps identified in previous systematic reviews by identifying the degree to which information on patient, provider, and setting characteristics were generalizable to different populations and settings (Santo et al., 2005; Schaefer, 2008).

Our systematic review supports the consistent finding that the current body of literature on health promotion interventions regularly reports on internal validity indicators of reach, such as the method to identify the target population and inclusion/exclusion criteria (Akers et al., 2010; White et al., 2009). However, reporting of participation rate (44%) was considerably lower in health literacy intervention studies when compared with the studies targeting changes in physical activity, nutrition, and smoking behaviors (e.g., 76% reporting; Glasgow et al., 2004). Still, our systematic review suggests that health literacy intervention researchers report representativeness (30%) to a greater degree than in other behavioral intervention domains (range 0–14%; Akers et al., 2010; Glasgow et al., 2002; White et al., 2009). Although this finding might be promising in light of other reviews of literature, we identified that few studies specifically targeted participants who could benefit most—those with low health literacy.

Further, as the proportion of participants with low health literacy was modest across studies (i.e., 38%), and few studies examined health literacy as a moderator, it is unclear what conclusions can truly be made about the applicability of the body of literature in typical community or clinical settings. Of the eight studies included in our review that performed a moderation analysis by health literacy category, four studies found no moderating effects across all outcomes measured (DeWalt et al., 2006; Paasche-Orlow et al., 2005; Rothman, Malone, et al., 2004; Wallace et al., 2009), three found mixed effects across outcomes (Gerber et al., 2005; Kim et al., 2004; Rothman, DeWalt, et al., 2004; Rothman et al., 2005) (RCT, N = 217), and only one found that the intervention was more effective for low health literate participants compared with their higher health literate counterparts (Ferreira et al., 2005). Assessing changes in health literacy status might be viewed as controversial, because many scholars view health literacy and the associated metrics as static. For this reason, very few studies assess longitudinal changes in health literacy status. This divisive assumption creates a dilemma because there is little available evidence to suggest that health literacy is indeed static. Advancing research related to the sensitivity and specificity of health literacy metrics, as well as understanding changes in health literacy, could help advance the health literacy field. Furthermore, those studies that did examine the potential of interventions to change health literacy demonstrated that improvements in health literacy were achievable (Nitiri & Stewart, 2009; van Servellen et al., 2003; van Servellen et al., 2005).

The reporting of other factors that could facilitate, or impede, the translation of health literacy interventions into practice varied in a similar way to reach. For example, our review indicated that studies on health literacy were more likely to provide a description of the location where the intervention was delivered (100%) when compared with previous reviews (range 48–60%; Akers et al., 2010; White et al., 2009). Unfortunately, it appears that information on the methods used to engage those who would deliver the intervention (or a description of their characteristics), intervention costs, and individual and program-level maintenance were all rarely reported (<12% of the time). Again, this is consistent with other RE-AIM reviews and underscores the need to improve reporting on these important factors associated with external validity and generalizability (Santo et al., 2005; Schaefer, 2008).

It appears that there is still a paucity of research examining the potential best-practices research as identified by previous systematic reviews (Santo et al., 2005; Schaefer, 2008). The studies we reviewed provided some description of their efforts to develop the intervention specifically for participants with low health literacy, yet the depth of the information varied widely among articles. Although several health literacy experts have challenged researchers to provide clarity in the conceptualization of health literacy interventions and causal pathways to achieving desired health outcomes, including theory-based approaches and the acknowledgement of sociocultural and psychological factors (Paasche-Orlow & Wolf, 2007; von Wagner, Steptoe, Wolf, & Wardle, 2009), only about one-third of the studies in this review described theoretical underpinnings for the health literacy intervention. Unfortunately, there remains a lag in the reporting of broad underlying theoretical approaches to intervention development, and of those that do report using a theoretical basis, nearly all focus only on individual-level behavioral theories. It is increasingly clear that when addressing complex health behaviors, it is insufficient to rely solely on approaches directed at individuals (Sallis & Owen, 2002; Stokols, 1996). This is reflected in the continued lack of multileveled interventions based on more ecological theories that provide direction for intervention development that focuses on the individual's participation, his/her environment, and the interaction between the two. For example, health literacy has been recognized as a complex interaction among an individual's health literacy skills, the communication skills of the health care provider, and the health care environment. Therefore, research approaches that attend to health literacy should consider social ecological frameworks that attend to various contexts in which behavior occurs and the many levels of external influences on individual behavior.

Finally, evidenced by the finding that only one of 25 included studies recruited participants through a community setting (Ntiri & Stewart, 2009), there is a clear need to develop, implement, and evaluate health literacy intervention within community settings. Previous research has established that access to and utilization of health care is compromised among low literate individuals (American Medical Association, 2005; Institute of Medicine, 2004), further strengthening the argument to diversify health literacy interventions beyond primary care and clinical settings.

Recommendations

Although the body of literature testing the effectiveness of health literacy interventions reflects a relatively new area of research and contains significant gaps in reporting external validity issues, there were a number of individual articles that provided excellent methods for addressing different components of the RE-AIM framework. Table 3 provides specific recommendations for future health literacy intervention research and provides exemplars when available.

Table 3. Recommendations for future studies and examples of exemplary reporting across RE-AIM components in the current body of health literacy literature

RE-AIM component	Recommendations for reporting on future health literacy studies
Reach	• Report on the methods to identify the participant population• Define a denominator for participation rate that is based on the projected number of eligible participants who are exposed to recruitment activities• Provide a comparison of those who agree to participate and those who decline on basic demographic and health informationExample articles addressing reach:• Murray et al., 2007; Rudd et al., 2009; Schillinger et al., 2008; each used a medical record system to identify a denominator of eligible participants, stated the inclusion/exclusion criteria explicitly, and provided information necessary to calculate an accurate participation rate• Schillinger and colleagues (2008) compared age, sex, language, race/ethnicity, insurance, and HgbA1c levels of participants and nonparticipants
Effectiveness	• Explicitly identify the trial type (e.g., efficacy, effectiveness, hybrid)• Use intention to treat methods• Use appropriately designed and adequately powered studies that execute moderation analysis by health literacy levels to better understand the robustness of intervention effects across different levels of health literacy• Include measures of quality of life or potential negative consequences of the interventionExample articles addressing effectiveness:• Murray et al., 2007; Schillinger et al., 2009; both reported across all the dimensions of efficacy/effectiveness
Adoption	• Continue to report on intervention location• Provide a description of the characteristics of the settings and individuals that delivered the intervention• Report methods to identify settings and individuals• Provide an indication of participation rate amongst delivery agents and settings• Continue to include, but to also move beyond, clinical settings to apply strategies more broadly in local communitiesExample articles addressing adoption:• Ferreira et al., 2005; Schillinger et al., 2008; both reported on four of six adoption dimensions• None of the reviewed studies reported on all six adoption dimensions
Implementation	• Report on intervention content, duration, and frequency• Provide information on implementation costs• Report the degree to which the intervention was delivered and received as intended• Indicate the theoretical approach• Increased focus on testing multileveled interventions based on social ecological modelsExample articles addressing implementation:• Davis et al., 2008; and Schillinger et al., 2009; report on duration and frequency and extent protocol delivered as intended• Bosworth et al., 2009; Murray et al., 2007; reported measure of cost and included fixed and variable costs association with intervention implementation and compared these costs among their study groups• None of the reviewed studies reported on all three implementation dimensions
Maintenance	• Add follow-up assessments that occur at least 6 months after the intervention is completed to understand individual level maintenance• Report the degree to which the intervention is sustained in the community or clinical setting• Report on costs to maintain interventionsExample articles addressing maintenance:• Bosworth et al., 2009; Dewalt et al., 2006; and Rudd et al., 2009; each report at least a 6-month follow-up assessment• None of the reviewed studies reported on all three maintenance dimensions

Limitations

This review was not without limitations. First, because we explicitly focus on health literacy and behavioral interventions targeting disease self-management and health promotion with at least a one week follow-up, we excluded numerous cross-sectional and brief intervention studies that had been included in previous health literacy reviews. Second, we focused on quality of reporting across the RE-AIM dimensions, which is notably different from a typical efficacy-based review where quality ratings are based on factors such as adequacy of study design, sample sizes, validated metrics, and statistical methods. It is feasible to suggest that articles rated high in an efficacy-based review might rate low on an RE-AIM review, and vice versa. Despite these potential limitations, our approach was based on the essential elements of an evidence-based systematic review including a rigorous search strategy, well-defined inclusion and exclusion criteria, and validated data extraction tool.

Conclusion

Our systematic RE-AIM review provides important information regarding the body of health literacy literature that is lacking from more traditional reviews focused primarily on internal validity and intervention effectiveness to improve health outcomes. In addition to effectiveness, our study demonstrates that addressing reach, adoption, implementation, and maintenance are critical factors to advance the field of health literacy research and to fully understand the potential public health impact of intervention efforts. Our findings were consistent with others in the need for stronger reporting of external validity factors, theory-based approaches, and best-practices approaches. Recommendations were provided to improve the methodology of future health literacy and behavioral intervention research.

Acknowledgments

This research was funded in part by a grant from the National Institutes of Health-National Cancer Institute (1R03CA136457-01A2).

Notes

*Based on denominator of 25 intervention trials, reported across 31 articles.