Measurement Variation Across Health Literacy Assessments: Implications for Assessment Selection in Research and Practice

Abstract

National priorities and recent federal initiatives have brought health literacy to the forefront in providing safe accessible care. Having valid and reliable health literacy measures is a critical factor in meeting patients' health literacy needs. In this study, the authors examined variation across three brief health literacy instruments in categorizing health literacy levels and identifying associated factors. The authors screened 378 veterans using the short form of the Test of Functional Health Literacy in Adults; the Rapid Estimate of Adult Literacy in Medicine; and a 4-Item Brief Health Literacy Screening Tool (known as the BRIEF). They analyzed data using prevalence estimates, Pearson product moment correlations, and logistic regression. When categorizing individuals' health literacy, agreement among instruments was present for 37% of the sample. There were consistencies; however, categorization and estimated risk factors varied by instrument. Depending on instrument, increased age, low education, minority status, and self-reported poor reading level were associated with low health literacy. Findings suggest that these instruments measure health literacy differently and are likely conceptually different. As the use of health literacy screening gains momentum, alignment between instrument and intended purpose is essential; in some cases, multiple instruments may be appropriate. When selecting an instrument, one should consider style of administration, purpose for measure, and availability of time and resources.

Health literacy is described as the capacity to obtain, process, and understand the basic health information and access services needed to make appropriate health decisions (Nielsen-Bohlman, Panzer, & Kindig, 2004). Health literacy skills include reading comprehension, listening, analyzing, and decision making. In addition, health literacy is a critical skill set that affects patients' ability to communicate with health care providers, adhere to recommended treatments, and access and navigate services to manage health conditions. The Institute of Medicine (Nielsen-Bohlman et al., 2004) and Healthy People 2010 (U.S. Department of Health and Human Services, 2000) have identified health literacy as a national priority and a key factor in reducing disease and health disparities. Recent federal mandates and initiatives, including the Affordable Care Act of 2010, the Department of Health and Human Services' National Action Plan to Improve Health Literacy, and the Plain Writing Act of 2010, will bring health literacy to the forefront in initiating safe accessible care to patients with health literacy needs (Koh et al., 2012).

As these mandates are implemented, the importance of accurately assessing and measuring individual health literacy levels will continue to be a critical issue for appropriately responding to the health information needs of patients. While individuals with adequate health literacy typically have the skills to access, exchange, and use health information, individuals with low health literacy skills have difficulty managing such tasks and are at risk for poor outcomes (Nielsen-Bohlman et al., 2004). Any patient may need assistance accessing, exchanging, and using health information. However, on the basis of previous research, individuals at the highest risk for low health literacy include those who are older in age (Baker et al., 2002; Gazmararian et al., 1999; Howard, Gazmararian, & Parker, 2005; Sudore, Mehta, et al., 2006; Sudore, Yaffe, et al., 2006), of minority (Bennett et al., 1998; Cooper-Patrick et al., 1999; Cooper & Roter, 2003; Nielsen-Bohlman et al., 2004) and low socioeconomic status (Nielsen-Bohlman et al., 2004; Sudore, Mehta, et al., 2006), and less educational attainment (Arozullah et al., 2006; Nielsen-Bohlman et al., 2004; Sudore, Mehta, et al., 2006). Identifying instruments that efficiently and accurately measure health literacy skills, and appropriately identify associated sociodemographic risk factors is critical for supporting national priorities to improve health care and outcomes. Identification of instruments with these qualities is also important for advancing this field of science.

Developing and validating brief health literacy instruments is an ongoing focus of inquiry (Baker, 2006; Baker, Williams, Parker, Gazmararian, & Nurss, 1999; Cheney & Nelson, 1988; Chew, Bradley, & Boyko, 2004; Chew et al., 2008; Davis, Crouch, et al., 1991; Davis, Long, et al., 1993; Nath, Sylvester, Yasek, & Gunel, 2001; Osborn et al., 2007; Parker, Baker, Williams, & Nurss, 1995; Powers, Trinh, & Bosworth, 2010; Rawson et al., 2010; Shea et al., 2004; Wallace, Rogers, Roskos, Holiday, & Weiss, 2006). Most instruments are designed to measure general health literacy (Chew et al., 2004; Chew et al., 2008; Davis, Crouch, et al., 1991; Davis, Long, et al., 1993; Nurss, Parker, Williams, & Baker, 2001; Osborn et al., 2007; Parker et al., 1995; Powers et al., 2010; Rawson et al., 2010; Wallace et al., 2006). However, other research has emphasized the development of field-specific instruments for use in areas such as dentistry (Lee, Rozier, Lee, Bender, & Ruiz, 2007; Richman et al., 2007), diabetes (Nath et al., 2001), and genetics (Erby, Roter, Larson, & Cho, 2008). The specificity of these instruments allows assessment of condition-specific knowledge and skills. Although condition-specific instruments are advantageous, they do not negate the need for a valid and reliable measure of general health literacy.

In the past, the short form of the Test of Functional Health Literacy in Adults (S-TOFHLA; Baker et al., 1999; Nurss et al., 2001; Parker et al., 1995) and the Rapid Estimate of Adult Literacy in Medicine (REALM; Davis, Long, et al., 1993; Davis et al., 2006) have been popular health literacy instruments. Despite noted limitations in the literature for the S-TOFHLA and REALM, widespread use of these instruments resulted in their recognition in the literature as reference standards for measuring health literacy (Baker, 2006; see Table 1).

Table 1. Comparison of characteristics of the S-TOFHLA, REALM, and BRIEF

Variable	S-TOFHLA*	REALM*	BRIEF*
Description	Reading comprehension	Medical word recognition and pronunciation test	Self-report of health literacy skills
Time to administer	7 minutes	2–7 minutes	1–2 minutes
Score range ^†	0–36	0–66	4–20
Scoring	0–16 = inadequate 17–22 = marginal 23–36 = adequate	0–44 = limited (0–6th-grade level) 45–60 = marginal (7th–8th-grade level) 61–66 = adequate (9th-grade and above)	4–12 = inadequate literacy 13–16 = marginal literacy 17–20 = adequate literacy
Advantages	Available in Spanish	Can be quick, uses medical terminology	Quick and easy to administer
Limitations	• Has to be administered and timed • Requires materials	• Has to be administered • Requires materials	Self-report, not performance based
Correlations ^‡ (r value)	• REALM 0.61 • BRIEF 0.42	• S-TOFHLA 0.61 • BRIEF 0.40	• S-TOFHLA 0.42 • REALM 0.40

In addition to the S-TOFHLA and REALM other researchers identified a set of self-report items for assessing health literacy (Chew et al., 2004; Chew et al., 2008; Powers et al., 2010; Wallace et al., 2006). In 2004, Chew and colleagues tested the ability of 16 health literacy items to effectively screen for health literacy, and findings indicated predictive ability for 3 of the 16 items:

1.	How often do you have someone help you read hospital materials?
2.	How confident are you filling out medical forms by yourself?
3.	How often do you have problems learning about your medical condition because of difficulty understanding written information?

Subsequent research tested item generalizability using more diverse populations and settings (Chew et al., 2008; Wallace et al., 2006). Recently, a literature review of brief health literacy instruments published in JAMA described the items developed by Chew and colleagues as effective and efficient for identifying patient health literacy levels (Powers et al., 2010).

Findings published by Griffin and colleagues (2010) suggest variation among the S-TOFHLA and REALM. This variation suggests greater accuracy of one instrument over the other and/or variable parameters across populations/settings and warrants further investigation. Literature determining the appropriate use of these new instruments in diverse settings is sparse, and, to date, these instruments are typically used interchangeably in research and clinical practice. While the advantages inherent in assessing individual levels of health literacy gain increasing attention, identification of available instruments, as well as understanding their unique associated characteristics increases in importance. Addressing variation across health literacy instruments is a critical issue since differences in construct measurement may affect measures of association between health literacy and other related variables. For example, variation in health literacy measure produces variation among categorization and nonstandard associated factors. Screening instrument characteristics, including operational measure and categorization of health literacy, need to be examined to inform proper use in research and practice.

In this study, we examined the variation of health literacy categorization and risk factors associated with low health literacy across three health literacy instruments, the S-TOFHLA, REALM, and 4-Item Brief Health Literacy Screening Tool (BRIEF). Findings from this study inform the proper use of these instruments in research and practice.

Method

Design and Sample

During 2006, we administered a written survey to a cross section of veterans attending ambulatory clinics within eight rural and nonrural VA medical facilities within a VA regional health care system in the Southeast United States (Haun, Noland Dodd, Graham-Pole, Rienzo, & Donaldson, 2009). We selected sites based on availability of volunteer data collectors within the VA health care region. Inclusion criteria specified English-speaking veterans, at least 18 years of age, and ability to provide written participation consent. We did not assess vision, hearing, and/or cognitive disorders, such as dementia and Alzheimer's. Participation in the study was voluntary and without compensation.

Twenty-one trained volunteers collected data from patients during routine health care visits in an ambulatory care setting within the VA facilities. Volunteer data collectors consisted of the principal investigator, a nutritionist, dental technician, nurse educator, and 17 nurses. The co–principal investigator, a registered nurse employed by the VA Health Care System, generated an interoffice e-mail to recruit data collectors. Before data collection, research team members provided a 30-minute on-site training session for data collectors. We provided training for data collectors, including review of data packets, administration instructions and practice opportunities.

We used published results, using the S-TOFHLA as the reference measure, to inform the power analyses for this study (Chew et al., 2004). Using a binary unconditional logistic regression modeling approach to predict adequate health literacy with primary fixed effects for age (per 10 years), female, non-Caucasian, and educational (high school or less) status; recruitment of 350 participants provided at least 85% power to detect a moderate effect size (OR = 2.0) assuming a Type I error rate of 5%, two-tailed and a 33% low health literacy rate (Nielsen-Bohlman et al., 2004).

To ensure adequate sample size, we recruited a convenience sample of 378 English-speaking veterans attending ambulatory care clinics. The identified population consisted mostly of older adults, a group identified by the literature as at high risk for inadequate health literacy (Baker et al., 2002; Gazmararian et al., 1999; Howard, Gazmararian, & Parker, 2005; Nielsen-Bohlman, Panzer, et al., 2004; Sudore, Yaffe, et al., 2006; Williams et al., 1995).

Trained data collectors recruited participants during routine clinical care encounters. Following consent, a data collector administered the REALM orally using a face-to-face interview format. Participants completed the S-TOFHLA (administration time 7 minutes), four BRIEF items, and a demographic information survey in written format. Each interview was approximately 20 minutes in length.

Measures

Each participant completed three health literacy screening instruments (REALM, S-TOFHLA, BRIEF) and 19 standard sociodemographic and health related items written at the seventh-grade reading level. Seven items assessed demographic characteristics: age, gender, race/ethnicity, education level, language (English as first or second language); home ownership (income proxy), employment/retirement/functional status. Additional items assessed self-reported reading ability, reported on a 5-point Likert-type scale ranging from 1 (excellent) to 5 (poor); self-reported health status (based on three dichotomous indicators—diabetes, high blood pressure, stroke); and eight items to assess respondents' self-reported ability to define health literacy, readiness to seek and access health information resources, and confidence in their ability to do so. These eight items were reported on a 5-point Likert-type scale ranging from 1 (strongly agree) to 5 (strongly disagree). Findings for the eight items assessing participants' ability to define health literacy, seek information and perceived self-confidence seeking health information are not reported in this article.

We used Paasche-Orlow and Wolf's (2007) causal pathways model, associating individual levels of health literacy with sociodemographic factors and health status, to guide our selection of the 12 independent variables included in the analysis [sociodemographic (i.e., age, gender, race/ethnicity, education level, perceived reading level, home ownership (i.e., income proxy), employment status, retirement status, functional status and health status (i.e., high blood pressure, stroke, diabetes)]. Because 97% of the sample reported speaking English as their first language, we excluded this variable from the analysis. On the basis of the literature's support for use of the REALM and S-TOFHLA, and increasing interest in the single health literacy items (known as the BRIEF in this study), we designated the REALM, S-TOFHLA, BRIEF scores as the dependent variables.

The S-TOFHLA (Parker et al., 1995) consists of two prose passages with 36 fill-in-the-blank response items worth one point each. Possible scores range from 0 to 36 and administration time is 7 minutes. Using the S-TOFHLA score individual health literacy skills are divided into three criterion levels: (a) inadequate (0–16); (b) marginal (17–22); and (c) adequate (23–36; Baker et al., 1999).

Administration of the REALM requires respondents to verbally articulate three columns of 22 health-related terms. The words in each column appear in ascending order of difficulty. The REALM score is a summed value based on the number of correctly pronounced words in each column. REALM scores can range from 0 to 66 and classify literacy at three levels, limited (0–44); marginal (45–60); and adequate (61–66; Davis, Long, et al., 1993).

The BRIEF instrument measures health literacy using four items:

1.	How often do you have someone help you read hospital materials?
2.	How confident are you filling out medical forms by yourself?
3.	How often do you have problems learning about your medical condition because of difficulty understanding written information?
4.	How often do you have a problem understanding what is told to you about your medical condition?

The first three items are identified in the literature as effective for identifying individuals with inadequate/marginal health literacy skills (Chew et al., 2004; Chew et al., 2008; Wallace et al., 2006). The fourth item addresses spoken information, an identified gap in health literacy measurement in clinical practice. On the basis of results of a principal component analysis indicating the four items accounted for 60% of score variance by measuring one distinct construct, the fourth item was added to constitute the BRIEF health literacy screening instrument (Haun et al., 2009). The four-item BRIEF resulted in a 0.77 Cronbach's alpha. Further analysis using the receiving operator characteristic curve indicated the combined four-item BRIEF had greater sensitivity when measuring inadequate health literacy than any of the four individual BRIEF health literacy screening items (Haun et al., 2009).

The BRIEF response options offer 5-point Likert-type scales for each item [items 1, 3, and 4 (1 = always to 5 = never); and item 2 (1 = not at all to 5 = extremely)]. The BRIEF score is based on the sum of the four nonweighted items and can range from 4 to 20. BRIEF levels are categorized as follows: (a) inadequate (4–12); (b) marginal (13–16); and adequate (17–20). The BRIEF levels are based on the scale used throughout the ongoing development of these items (Chew et al., 2004; Chew et al., 2008; Wallace et al., 2006). This study reports on an extended analysis of these data.

Data Analysis

We calculated descriptive statistics including means, frequencies and proportions to provide preliminary statistical information, and Pearson product moment correlation coefficients to determine comparative validity between screening instruments in the study sample. We also calculated agreement between categorization of the respondents as having adequate, inadequate, or marginal health literacy. We dichotomized (adequate vs. inadequate/marginal) scores for each instrument (S-TOFHLA, REALM, BRIEF) and modeled separate binary logistic regression models to evaluate the relation between potential predictors and probability of inadequate/marginal health literacy. We initially categorized scores for each instrument into one of three levels. Since scores in the inadequate and marginal categories indicate risk for associated health outcomes, we combined scores for these categories resulting in a dichotomous variable to compare to the referent group (adequate). Because of small sample cell sizes, we collapsed the independent variables education, self-reported reading level, and ethnicity prior to analysis. We assessed outliers and collinearity using visual inspection. We also assessed collinearity using the variance inflation factor, with the variance inflation factor defined as 1/tolerance. We assumed a tolerance of ≤ 0.1 or equivalently a variance inflation factor ≥of 10, to be a cause for concern. Variance inflation factors ranged from 2 to 4; moderate collinearity was evident among variables age and retirement. However in this sample, as in the veteran population as a whole, retirement age has large variation. Given the sample size and strategy for separate modeling, we included equal parameters within all three final models.

For the initial data analysis, we used a univariate approach for determining the unique contribution of each variable (e.g., statistical significance of p < .2). As previously stated, we chose the independent variables based on a review of the health literacy literature. To establish continuity across data analyses, we included parameters in subsequent models only if they contributed to at least one of the instruments. Therefore identical parameters comprise each of the final reported models. We conducted all analyses with SAS (version 9.2, Cary, NC).

Results

The response rate was high (90%); we did not assess reasons for refusal. Overall, the sample of 378 veterans were predominantly white (73.5%; n = 278) and male (94.2%; n = 356) with a mean age of 61.5 years (SD = 11.9 years; see Table 2). Most veterans reported having at least a high school diploma (80.7%; n = 305), speaking English as their first language (97%; n = 366), and owning their home (77%; n = 290).

Table 2. Demographic distribution for descriptive variables

Demographic variable	Value
Age, years
Range	23–89
Average (SD)	61.5 (11.9)
Gender, no. (%)
Male	356 (94.2)
Female	19 (5.0)
Not reported	3 (0.8)
Ethnicity, no. (%)
Caucasian/White (Nonminority)	278 (73.5)
African American/Black	69 (18.3)
Hispanic/Latino	12 (3.2)
Native American/Alaskan Native	12 (3.2)
Hawaiian, Pacific Islander	0 (0.0)
Asian American	1 (0.3)
Other	3 (0.8)
Not reported	3 (0.8)
Education, no. (%)
Less than high school	71 (18.9)
High school diploma/GED	98 (25.9)
Some college	126 (33.3)
College degree	80 (21.2)
Trade school	1 (0.3)
Not reported	2 (0.5)
English first language, no. (%)
Yes	366 (97.0)
No	10 (2.6)
Not reported	2 (0.5)
Self-reported reading level, no. (%)
Excellent	95 (25.1)
Very good	113 (29.9)
Good	113 (29.9)
Fair	42 (11.1)
Poor	13 (3.4)
Not reported	2 (0.5)
Home ownership (Income Proxy), no. (%)
Rent	80 (21.2)
Own	290 (76.7)
Not reported	8 (2.1)
Employment status, no. (%)
Employed	366 (96.8)
Unemployed	10 (2.6)
Not reported	2 (0.5)
Retirement status, no. (%)
Retired	147 (38.9)
Not retired	229 (60.6)
Not reported	2 (0.5)
Functional status, no. (%)
Disability	91 (24.0)
No disability	285 (75.4)
Not reported	2 (0.5)
Diabetes, no. (%)
Yes	132 (34.9)
No	244 (64.6)
Not reported/don't know	2 (0.5)
High blood pressure, no. (%)
Yes	242 (64.0)
No	133 (35.2)
Not reported/don't know	3 (0.8)
Stroke, no. (%)
Yes	41 (10.8)
No	335 (88.6)
Not reported/don't know	2 (0.5)

Mean scores for the three screening instruments were as follows: S-TOFHLA = 29.92 (SD = 7.95; possible range = 0–36); REALM = 59.46 (SD = 9.00; possible range = 0–66); and BRIEF = 15.41 (SD = 3.63; possible range = 4–20), respectively (see Figure 1).

Figure 1 Participants' health literacy level as indicated by the S-TOFHLA, REALM, and BRIEF.

Pearson correlation results were as follow: r(378) = .40, p < .01, between the BRIEF and REALM; r (378) = .42, p < . 01, for the BRIEF and S-TOFHLA; and r(378) = .61, p < . 01, for the REALM and S-TOFHLA (see Table 1).

To provide a framework to compare results on the three screening instruments, we classified scores as being inadequate, marginal, or adequate. Agreement among the three screening instruments was present among 37.3% (n = 141) of the sample; categorization across the three instruments was most consistent when respondents' scores were adequate (n = 129, 34.1%). For the other 62.7% of respondents, score categorization was not consistent across instruments, however nearly 53% did agree on at least two instruments, of which the majority (27.0%) of agreement occurred between the REALM and the S-TOFHLA.

The S-TOFHLA classified 83% (n = 315) of the respondents as having adequate health literacy, 8% with marginal (n = 29), and 9% (n = 34) with inadequate health literacy, compared to the REALM which placed 63% of respondents in the adequate health literacy category (n = 240, 63%). Approximately one third (n = 113) of the sample placed in the marginal category and about 7% (n = 25) in the limited category. The BRIEF was least likely to categorize respondents' health literacy levels as adequate (n = 164, 43%) and more likely to place respondents in the marginal (n = 138, 37%) and inadequate (n = 76, 20%) categories.

Univariate analyses detected an association of all the independent variables—i.e., age, gender, ethnicity, education, self-reported reading level, home ownership, employment, retirement, functional status, and health status (high blood pressure, diabetes, stroke)—with at least one of the health literacy outcome variables (p < .2), although the pattern of association varied across the instruments. These findings appear in Table 3.

Table 3. Univariate results for demographical variables associated with health literacy level instrument by the S-TOFHLA, REALM, and BRIEF

	S-TOFHLA (n = 378)			REALM (n = 378)			BRIEF (n = 378)
Variable	Inadequate/marginal (n = 63)	Adequate (n = 315)	p	Limited/marginal (n = 138)	Adequate (n = 240)	p	Inadequate/marginal (n = 214)	Adequate (n = 164)	p
Age (years)*
59 or younger	12 (7.6)	145 (92.4)	.001	53 (33.8)	104 (66.2)	.848	82 (52.2)	75 (47.8)	.478
60–69	16 (13.0)	107 (87.0)		46 (37.4)	77 (62.6)		73 (59.3)	50 (40.7)
70–79	25 (35.7)	45 (64.3)		27 (38.6)	43 (61.4)		42 (60.0)	28 (40.4)
80 or older	14 (56.0)	11 (44.0)		10 (40.0)	15 (60.0)		16 (64.0)	9 (36.0)
Gender
Male	64 (18.0)	292 (82.0)	.154	133 (37.4)	223 (62.6)	.057	206 (57.9)	150 (42.1)	.024
Female	1 (5.3)	18 (94.7)		3 (15.8)	16 (84.2)		6 (31.6)	13 (68.4)
Ethnicity
Minority	23 (23.5)	75 (76.5)	.073	49 (50.0)	49 (50.0)	.001	56 (57.1)	42 (42.9)	.958
Nonminority	43 (15.5)	235 (84.5)		87 (31.3)	191 (68.7)		158 (56.8)	120 (43.2)
Education
High school or more	22 (10.7)	184 (89.3)	.001	93 (54.7)	77 (45.3)	.001	117 (68.8)	53 (31.2)	.001
Less than high school	45 (26.5)	125 (73.5)		43 (20.9)	163 (79.1)		96 (46.6)	110 (53.4)
Self-reported reading level
Fair/poor	25 (45.5)	30 (54.5)	.001	41 (74.5)	14 (25.5)	.001	51 (92.7)	4 (7.3)	.001
Good/excellent	42 (13.0)	281 (87.0)		97 (30.0)	226 (70.0)		163 (50.5)	160 (49.5)
Home ownership
Rent home	11 (13.8)	69 (86.2)	.317	31 (38.8)	49 (61.2)	.620	47 (58.8)	33 (41.2)	.623
Own home	54 (18.6)	237 (81.4)		104 (35.7)	187 (64.3)		162 (55.7)	129 (44.3)
Employment
Unemployed	0 (0.0)	10 (100.0)	.136	5 (50.0)	5 (50.0)	.377	7 (70.0)	3 (30.0)	.388
Employed	67 (18.3)	299 (81.7)		133 (36.3)	233 (63.7)		206 (56.3)	160 (43.7)
Retirement
Retired	37 (25.2)	110 (74.8)	.003	50 (43.0)	97 (66.0)	.386	78 (53.1)	69 (46.9)	.261
Not Retired	30 (13.1)	199 (86.9)		88 (38.4)	141 (61.6)		135 (59.0)	94 (41.0)
Functional status
Disability	71 (78.0)	20 (22.0)	.234	40 (44.0)	51 (56.0)	.099	63 (69.2)	28 (30.8)	.005
No disability	47 (16.5)	238 (83.5)		98 (34.4)	187 (65.6)		150 (52.6)	135 (47.4)
Health status
Diabetes	20 (15.2)	112 (84.8)	.368	44 (33.3)	88 (66.7)	.358	82 (62.1)	50 (37.9)	.115
No diabetes	46 (18.9)	198 (81.1)		93 (38.1)	151 (61.9)		131 (53.7)	113 (46.3)
High blood pressure	50 (20.7)	192 (79.3)	.036	95 (39.3)	147 (60.7)	.183	149 (61.6)	93 (38.4)	.008
No high blood pressure	16 (12.0)	117 (88.0)		43 (32.3)	90 (67.7)		63 (47.4)	70 (52.6)
Stroke	12 (29.3)	29 (70.7)	.037	16 (39.0)	25 (61.0)	.715	28 (68.3)	13 (31.7)	.111
No stroke	54 (16.1)	281 (83.9)		121 (36.1)	214 (63.9)		185 (55.2)	150 (44.8)

Logistic Regression Models

Preliminary assessments to identify outliers and test for co-linearity suggested neither outlying data nor evidence of multicollinearity between the independent variables included in the regression analyses. Independent variables of interest were age (per 10 years), female, minority status and education (high school or less). Models were adjusted for self-reported reading level, retirement, and health status variables (diabetes, high blood pressure, stroke). Overall, inadequate health literacy was more prevalent among older respondents, of minority status with low educational attainment and fair/poor self-reported reading level.

Interscreening Instrument Consistency

Perceived reading level was consistently associated with health literacy levels across all three instruments. S-TOFHLA scores for respondents reporting fair/poor reading levels were more than four times more likely of being classified for low health literacy levels than those reporting good/excellent reading levels (95% CI [2.00, 9.54]). Respondents reporting fair/poor reading levels were five times more likely to receive REALM scores in the low health literacy category than were those reporting good/excellent reading levels (95% CI [2.53, 11.40]). Participant scores on the BRIEF indicated a probability of low health literacy 11 times greater than those participants reporting a reading level of good/excellent (95% CI [3.40, 38.39]). Association among the independent variables: gender, retirement status, functional status (disability) or the health status indicators (diabetes, high blood pressure, stroke) was consistently absent.

Interscreening Instrument Variation

Table 3 (univariate data) presents age as a categorical variable. However, in the regression model age is calculated as a continuous variable. Reported odds ratios were associated with 10-year increases in age. Independent variables assessed using the S-TOFHLA suggest the association of each 10-year increase in age with an increase in probability of low health literacy 1.12 times (95% CI [1.07, 1.16]), after adjustment for potential confounders (e.g., health status, functional status, etc.). Consistent with the S-TOFHLA, evaluation of risk factors associated with the BRIEF finds that with each 10-year increase in age the probability for a low health literacy score increases 1.02 times (95% CI [1.00, 1.05]). Analyses of data based from the REALM produced no statistically significant association between age and health literacy.

Minority respondents' health literacy scores were lower than nonminority respondents on the S-TOFHLA (OR = 2.5; 95% CI [1.17, 5.24]) and on the REALM (OR = 2.7; 95% CI [1.51, 4.70]). Minority status lacked statistical significance with health literacy on the BRIEF.

Those participants with less than a high school education had a 1.8 times greater likelihood of having low health literacy (95% CI [1.15, 2.92]) on the BRIEF and 4 times greater probability for low health literacy on the REALM (95% CI [2.38, 6.43]), after adjustment for potential confounders (e.g., health status, functional status). Education was not significantly associated with health literacy on the S-TOFHLA. Odds ratios and 95% confidence intervals are presented in Table 4.

Table 4. Adjusted logistic regression of sociodemographic and health status variables associated with low health literacy

	S-TOFHLA		REALM		BRIEF
	Odds ratio	Confidence interval	Odds ratio	Confidence interval	Odds ratio	Confidence interval
Age, per 10 years	1.12	(1.07, 1.16)*	1.01	(0.99, 1.04)	1.02	(1.00, 1.05)*
Male	0.92	(0.10, 8.30)	1.80	(0.49, 6.67)	2.22	(0.75, 6.53)
Minority, non-White	2.47	(1.17, 5.24)*	2.66	(1.51, 4.70)*	0.87	(0.51, 1.50)
High school education or less	1.83	(0.94, 3.56)	3.91	(2.38, 6.43)*	1.83	(1.15, 2.92)*
Self-reported reading level, poor	4.37	(2.00, 9.54)*	5.37	(2.53, 11.40)*	11.43	(3.40, 38.39)*
Retired, yes	1.28	(0.47, 3.47)	0.76	(0.37, 1.53)	0.54	(0.28, 1.04)
Disability, yes	2.56	(0.96, 6.85)	0.98	(0.50, 1.91)	1.24	(0.66, 2.34)
Diabetes, yes	0.57	(0.28, 1.16)	0.60	(0.35, 1.02)	1.15	(0.70, 1.87)
High blood pressure, yes	1.10	(0.52, 2.36)	1.16	(0.67, 2.00)	1.28	(0.78, 2.11)
Stroke, yes	1.68	(0.71, 3.98)	0.80	(0.36, 1.80)	1.38	(0.63, 3.00)

Discussion

Nearly a decade ago the Institute of Medicine (Nielsen-Bohlman et al., 2004) identified health literacy as a national priority. However, current federal initiatives (Koh et al., 2012), have accentuated the need for increased and timely efforts aimed at improving the reliability and validity of health literacy instruments, especially those measuring prevalence estimates across populations. Improved understanding of current health literacy instruments is critical if health systems are to be responsive to related federal initiatives. In this study, we examined the association between scores on three health literacy screening instruments (S-TOFHLA, REALM, and BRIEF) as well as the patterns of associations among common predictors of health literacy across the instruments. While correlation among the three health literacy screening instruments was positive, categories of health literacy and associated factors (e.g., gender, race) varied depending on the instrument used to assess health literacy.

In this sample, data from the REALM and BRIEF categorized the mean scores as having marginal health literacy skills. However, within the same sample health literacy levels as measured by the S-TOFHLA, placed the mean scores in the adequate category. Estimates using the BRIEF items were consistent with previous research in the veteran population using composite scores for three of the BRIEF items (Chew et al., 2004; Chew et al., 2008). However, in other studies with demographically similar populations, health literacy level estimates were lower as measured by the S-TOFHLA in a Medicare population (Gazmararian, Baker, et al., 1999); and in an urban hospital setting using the TOFHLA (Williams et al., 1995).

The three instruments concurred most often when categorizing respondents with adequate health literacy skills and were most susceptible to variation among those in the marginal and inadequate categories. Scores for all three instruments correlated, however the strongest association was between the S-TOFHLA and the REALM. These findings point to the complexity inherent in measuring health literacy as a construct.

Consistent with previous work, in logistic regression analyses, increased age (Baker et al., 2002; Gazmararian et al., 1999; Howard et al., 2005; Sudore, Mehta, et al., 2006; Sudore, Yaffe, et al., 2006), less education (Arozullah et al., 2006; Nielsen-Bohlman et al., 2004; Sudore, Mehta, et al., 2006), and minority status (Bennett et al., 1998; Cooper-Patrick et al., 1999; Cooper & Roter, 2003; Nielsen-Bohlman et al., 2004) were associated with low health literacy (Paasche-Orlow, Parker, Gazmararian, Nielsen-Bohlman, & Rudd, 2005; Shea et al., 2004). While predictors for health literacy were present among the three instruments, variation emerged when assessing statistical significance of associated factors across instruments. Self-reported reading level was associated with all three of the screening instruments while age, minority status and education were associated with at least two of three of the measures. Specifically, minority status was associated with low health literacy on the REALM and S-TOFHLA, but not the BRIEF; and age was associated with low health literacy on the S-TOFHLA and BRIEF, but not on the REALM. These findings suggest populations identified at risk for low health literacy differ by instrument.

While some overlap was present among the instruments examined in this study, findings indicate the need for more research exploring variation among health literacy instruments and the validity issues implied by these inconsistencies across instruments. Though these finding may be a function of sample size, it is consistent with variations previously reported in the literature by Griffin and colleagues (2010). Categorizing data, as done in this analysis, may compromise the ability to evaluate the nuance of variance estimates. However, this approach is consistent with previous literature, and provides meaningful categories for evaluating and interpreting health literacy scores.

When choosing the most appropriate health literacy instrument, it is important for researchers and clinicians to recognize the differences present in the way these three instruments measure or weigh the components of health literacy, as well as possible conceptual differences. Griffin and colleagues (Griffin et al., 2010) suggested several reasons for instrument variation including greater accuracy of one instrument over another, instability of parameters across demographically diverse groups, or measurement of different health literacy domains (e.g., vocabulary, comprehension). These data provide insight for advancing the science of health literacy measurement. However further research is needed to determine why variation is present. Identifying the health literacy domains most associated with the demonstration of functional health literacy and the most effective instruments for measuring these domains is critical. Using multiple instruments is warranted in future research and practice until the characteristics of health literacy instruments are better understood.

For purposes of determining their appropriate use in research and practice characteristics and operationalization of health literacy instruments should be compared and examined. Improved understanding of each predictor's contributory weight to health literacy will improve the ability to interpret findings regardless of the instrument used. Careful consideration of instrument operationalization, as well as characteristics of the individuals being measured is recommended. While findings from this study provide an indication of each screening instrument's appropriate use in research and clinical practice, continued work along this line of inquiry is essential.

Findings are generalizable to veteran patients in the southeast and may not apply to other populations or geographical regions of the United States. Further, using a convenience sample may impact results, as some individuals may self-exclude from participation to avoid disclosing their literacy level (Paasche-Orlow & Wolf, 2008). We did not collect data from individuals who declined participation in the study, posing a potential study limitation particularly if refusal was for reasons such as shame associated with one's lack of health literacy. In addition, we did not test participants for visual, hearing, or cognitive impairment, which further limits interpretation of these findings. Future studies will benefit by collecting these data.

As indicated, the categorization of individual health literacy scores may reduce ability to identify variance estimates of individual health literacy levels and present a potential study limitation. Though this analysis used meaningful categorizations for each instrument, future research could evaluate these instruments as continuous scales based on score range.

This study sample is small; the small sample sizes based on ethnic orientation and gender may have affected findings. Future work will benefit from a larger more diverse nationally representative population. Socioeconomic status was controlled by the study's residential status item, as an income proxy; however, this item resulted in a nonsensitive control variable. Previous research indicates health literacy skill should be affected by socioeconomic status. Additional measures of socioeconomic status other than home ownership may be more informative. We recommend that future research collect data in units of annual income, or total assets, to collect a precise measure of socioeconomic status.

Despite these limitations, this research is timely and relevant to advancing the science of health literacy measurement. Previous work in the area of reviewing health literacy instruments (Powers et al., 2010) lacks studies using primary data to examine variation among instruments and has focused on the REALM and the S-TOFHLA (Griffin et al., 2010). In contrast, this study used primary data to compare the REALM and the S-TOFHLA, in addition to the four BRIEF screening items.

Implications for Research and Practice

When selecting a health literacy instrument, we encourage clinicians and researchers to consider the conceptual match between how the health literacy screening instrument operationalizes the construct, the task in question and the purpose for measure, while also giving consideration to the style of administration, and availability of time and resources. Choosing a health literacy instrument closely aligned with the topic or task under consideration is critical for accurate measures of the domains being assessed. Features to consider when selecting a health literacy tool are presented in the Appendix.

When time and resources are limited use of the BRIEF instrument may be the most efficient screening instrument; particularly in the clinical setting where results can be quickly integrated into the patient medical record to inform clinical decision-making (Haun et al., 2009). The BRIEF items cast a broad net by assessing reading comprehension, verbal health information and confidence filling out medical forms; these items can be evaluated individually to assess individual skills. The BRIEF can also be administered by phone and utilized to support tele-health and/or phone survey based research. This instrument avoids performance pressure and embarrassment, as it is self-report and allows the respondent to determine the course of the screening. Because of its high sensitivity (Haun et al., 2009), the BRIEF items can be used to screen patients, and the S-TOFHLA or REALM can serve as confirmatory measures.

If a performance-based task is necessary, a more appropriate appraisal of health literacy is offered by the S-TOFHLA or REALM. The S-TOFHLA measures domains such as reading comprehension and numeracy, while the REALM assesses medical vocabulary using word recognition and pronunciation. For example, the S-TOFHLA provides context for tasks related to medical instruction comprehension and completion of medical forms, while the REALM may be more appropriate when assessing verbal health communication skills. The S-TOFHLA was the most sensitive instrument when evaluating associated sociodemographic predictors in this sample. However, the REALM may be more useful when time and personnel are limited.

In summary, we suggest considering these differences when choosing a health literacy instrument. Measurement of health literacy skills is not a perfected science. Until conclusive research is available, we recommended use of multiple health literacy instruments when possible. As research advances the science of health literacy measurement, researchers and clinicians will benefit from knowing not only of the instruments available for measuring health literacy, but also of measurement variation and conceptual differences across instruments.

Acknowledgments

This article not subject to US copyright law.

The authors thank Mary Elizabeth Bowen, PhD, and Scott Barnett, PhD, for their careful review and constructive feedback regarding this manuscript during its development.

This research was supported by the Tampa James A. Haley VA Medical Center, HSR&D & RR&D Research Center of Excellence and does not represent the views of Veterans Affairs.

Notes

*S-TOFHLA = Short Test of Functional Health Literacy in Adults; REALM = Rapid Estimate of Adult Literacy in Medicine; BRIEF = 4-item BRIEF Health Literacy Screening Tool.

^†Higher score signifies higher health literacy level.

^‡Correlations reported in Haun et al. (2009).

*Sample size varied because veterans elected to not disclose personal data.

*p < .05.

^†Models are adjusted for all other terms in table.

Appendix: Features to Consider When Comparing the S-TOFHLA, REALM, and BRIEF

Table

S-TOFHLA	REALM	BRIEF
Assesses reading and comprehension	Assesses medical vocabulary using word recognition and pronunciation	Assesses reading and verbal comprehension, need for assistance, and confidence filling out medical forms
7 minutes to administer	2–7 minutes to administer	1–2 minutes to administer
Performance-based measure	Performance-based measure	Not performance-based, self-report bias potential
Identified as reference standard in the literature	Identified as reference standard in the literature	Individual items assess different skill components and can be evaluated individually
Context related to medical instruction comprehension and completing medical forms	Appropriate when assessing verbal health communication skills	Self-report avoids performance pressure and embarrassment
Requires materials, timing, and formal administration	Requires materials and formal administration	Does not require materials or formal administration
	Condition specific versions available	Can be administered by phone