Measuring Health Literacy: A Pilot Study of a New Skills-Based Instrument

Abstract

Although a number of instruments have been used to measure health literacy, a key limitation of the leading instruments is that they only measure reading ability or print literacy and, to a limited extent, numeracy. Consequently, the present study aimed to develop a new instrument to measure an individual's health literacy using a more comprehensive and skills-based approach. First, we identified a set of skills to demonstrate and tasks to perform. Next, we selected real-world health-related stimuli to enable measurement of these skills, and then we developed survey items. After a series of cognitive interviews, the survey items were revised, developed into a 38-item instrument, and pilot tested using a Web-based panel. Based on the psychometric properties, we removed items that did not perform as well, resulting in a 25-item instrument named the Health Literacy Skills Instrument. Based on confirmatory factor analysis, the items were grouped into five subscales representing prose, document, quantitative, oral, and Internet-based information seeking skills. Construct validity was supported by correlations with the short form of the Test of Functional Health Literacy in Adults and self-reported skills. The overall instrument demonstrated good internal consistency, with a Cronbach's alpha of 0.86. Additional analyses are planned, with the goal of creating a short form of the instrument.

A number of instruments have been used to measure health literacy. Two of the most commonly used instruments include the Rapid Estimate of Adult Literacy in Medicine (REALM) (Davis et al., 1991) and the Test of Functional Health Literacy in Adults (TOFHLA) (Parker, Baker, Williams, & Nurss, 1995). However, an important limitation of these instruments is that they measure reading ability or print literacy, and in the case of the TOFHLA, numeracy; they do not reflect a comprehensive assessment of health literacy (Berkman et al., 2004; IOM, 2009). Instruments also exist that attempt to screen patient health-literacy level in clinical settings (e.g., the Newest Vital Sign; Weiss et al., 2005), measure provider-level facilitation of health literacy (e.g., Consumer Assessment of Health Providers and Systems [CAHPS] Item Set for Addressing Health Literacy; Agency for Healthcare Research and Quality, n.d.), and assess health literacy using sociodemographic and geographic data elements (Paasche-Orlow, Schillinger, Greene, & Wagner, 2006).

The Department of Education's 2003 National Assessment of Adult Literacy (NAAL) Survey is the only national assessment of literacy that includes some health literacy tasks (Kutner, Greenberg, Jin, & Paulsen, 2006). Of the 28 health literacy tasks on the NAAL, 3 represented a clinical domain, 14 represented a prevention domain, and 11 items represented navigation of the health care system. The NAAL yields estimates of the distribution of levels of health literacy for various population groups. Though it overcomes some of the limitations of other measures, including a focus on assessing skills other than reading, the NAAL has been criticized for its lack of availability, lack of transparency, and challenges in using it (Weiss, 2009).

In a landmark report, Health Literacy: A Prescription to End Confusion, the Institute of Medicine (IOM) recommended the development of new health literacy measures (Nielson-Bohlman, Panzer, & Kindig, 2004). The IOM's Recommendation 2–2 calls for the development, testing, and use of culturally appropriate measures of health literacy and to field them as part of large ongoing population surveys. Parker & Kindig (2006) echoed the call for new measures by stating that “new measures of health literacy must be developed and evaluated” (p. 891), and the authors concluded that “while progress [in the area of health literacy] is being made, the scope is not broad enough and the pace is not fast enough” (p. 891). Baker (2006) also supported the need for more comprehensive measures.

This study aimed to advance the field by developing a more comprehensive measure of health literacy that is publicly available. Similar to other measures, this instrument, titled the Health Literacy Skills Instrument (HLSI), measures print literacy. However, it is innovative in that it also uses non-print stimuli and examines oral and Internet-based information seeking skills. In terms of content domains, it reflects health related issues across the life course for health promotion and disease prevention, health care maintenance and treatment, and health system navigation. The HLSI can be self-administered via a computer, which can reduce data collection costs and minimize potential discomfort or embarrassment among participants. The HLSI is designed to be used in intervention research studies as well as for large scale surveillance.

This article describes the development of this novel instrument to measure individual level health literacy using a skills-based approach. We explain the instrument development and testing process and provide the results from a pilot test of the instrument, including how individual items performed as well as validation data. We also discuss policy implications, make recommendations for next steps, and address measurement considerations.

Instrument Development

Our core instrument development team included individuals with backgrounds in health communication, health services research, psychometrics, literacy, health literacy, plain language, cultural competency, and clinical medicine. We used Ratzan and Parker's (2000) definition of health literacy, with minor modifications to the wording: The degree to which individuals can obtain, process, understand, and communicate about health-related information needed to make informed health decisions. A group of eight health literacy expert panel members provided oversight and guidance, especially in the early developmental phases.

To develop the instrument, we adopted a hierarchical process. First, we identified skills to demonstrate and tasks to perform (see Table 1). Tasks were categorized into three skills set areas: print, oral, and Internet-based information seeking. We classified tasks as follows: identifying and understanding health-related text; interpreting information and/or data in the form of tables, charts, pictures, symbols, maps, and videos; completing computations; making inferences based on information presented or applying given information to a specific scenario; and information seeking and interactions on Internet websites.

Table 1. Health literacy skill area by task and by health domain (25-item instrument)

	Skills and stimuli format
Task	Print (prose, document, or quantitative)	Oral (listening or audiovisual)*	Internet-based information seeking
1) Identifying and understanding health-related text	Prose 1) Which of the following is probably not a second-degree burn? (b) 2) Which of the following is a symptom of lactose intolerance? (b) 3) Which of the following is not a sign of a stroke? (b)
2) Interpreting information and/or data in the form of tables, charts, pictures, symbols, maps, and videos	Prose 4) Which set of low density lipoprotein (LDL) and high density lipoprotein (HDL) levels is best? (a)Document 5) Which of the following problems could be caused by this medicine? (b) 6) Which of the following entrances is closest to the elevator? (c) 7) In the example listed in the first row of the table, when should the medicine be taken? (b)Document 8) How much will the insurance company pay for the physical therapy received on 7/22/09? (c) 9) How much does the patient have to pay for the laboratory services received on 7/15/09? (c)	18) What do the muscles in the throat typically do when a person is sleeping? (b)19) What part of the body do lunge exercises work? (a)
3) Completing computations	Quantitative10) More men die from prostate cancer than from other causes. Would you say this is true, false, or are you not sure? (a)11) Who is more likely to die of prostate cancer? (a)12) How many grams of fiber are in two servings? (a)13) If a person is on a 2,500 calorie diet, what percent of the daily value of saturated fat would he get from one serving? (a)
4) Making inferences based on the information presented or applying information to a specific scenario	Prose14) If a person is at high risk for heart disease, which of the following levels of low density lipoprotein (LDL) cholesterol is best? (a)Document15) If Joe was visiting someone in Room 130 and wanted to go to the cafeteria, which of these places would he pass if he took the shortest route? (c)16) A person is making a salad and wants to add one serving of chopped, uncooked carrots. How much should she use? (a)17) A person is cooking dinner for himself and wants to include one serving from the meat and beans group. What should he choose? (a)	20) If a person was worried about his cough, what number should he press? (c)21) If a person wanted to check on the date and time of an appointment she already made, what number should she press? (c)
5) Utilizing the Internet/computer to obtain health information			22) John weighs 200 pounds and he walked at a medium pace on a firm surface for 30 minutes. How many calories did he burn? (a)23) Kate weighs 150 pounds. Which activity would burn the most calories? (a)24) What does this (heart attack risk) tool do? (a)25) John is 39 years old and smokes. His blood pressure is 130/90 and he's on blood pressure medicine. His HDL cholesterol is 50 and his total cholesterol is 230. What is his estimated 10-year risk of heart attack? (a)

Second, we selected health-related stimuli to enable an assessment of those skills and tasks. We sought diversity in the stimuli across three major health domains reflecting various points in the life course, during periods of health and illness: health promotion and disease prevention, heath care maintenance and treatment, and health system navigation (see Table 1). To the extent possible, we used the following inclusion criteria for stimuli selection: (a) relevant to the health of a large segment of the public (i.e., gender neutral; not specific to subgroups); (b) culturally sensitive; (c) clinically relevant and less controversial health topics; and (d) both public and private-sector materials in a variety of formats/channels, including print documents (e.g., brochure, newspaper article, fact sheet), telephone recording, video clip, and Web site. We chose stimuli that would take most participants about 1 minute to view or read. We used real-world stimuli, with a limited number of plain-language materials. Copyright approval or user permission was obtained for private-sector materials.

Third, we developed survey items. We attempted to map each survey item to a skill set area as well as a task. In some cases, a survey item could arguably fit into more than one skill and/or task. The initial version of the instrument included 38 items, with 1 to 2 survey items for each stimulus (example stimuli and survey items are shown in the Appendix). The survey items did not require prior or outside knowledge, that is, each question could be answered based only on the information in the stimulus. We sought to include items with varying levels of difficulty that were not contingent upon each other. To score an instrument, it is necessary to have a single correct response option and multiple incorrect response options or plausible “distracters.” We included three to four response options for most questions, with a “don't know” option.

Finally, we chose the mode of survey administration. To effectively view all of the stimuli and demonstrate the skills, we determined that the survey needed to be administered using a computer through either Web-based or in-person data collection.

Nine in-person cognitive interviews were conducted using a laptop computer and with an interviewer present. Five males and four females participated ranging in age from 21 to 78 years (mean age is 42 years). Five participants had college degrees and four participants had less than this. Respondents completed the survey using the keyboard and mouse after viewing each stimulus on the screen. Despite an introduction at the beginning of each question instructing respondents to base their answers only on the information in the stimulus, some respondents tried to use prior knowledge or logic to answer questions. For example, respondents who were familiar with a given health topic sometimes answered questions based on their own experiences. In other cases, respondents used logic or common sense to answer a question. Respondents tended to have more difficulty answering a question that required making an inference, and when the wording in the survey question was not exactly the same as the wording in the stimulus. We revised the 38 items based on the cognitive testing results for use in the next phase of the study, which included pilot data collection.

Data Collection Methods

Sample and Setting

We pilot tested the instrument using KnowledgePanel®, created by Knowledge Networks, an online Non-Volunteer Access Panel. Potential panel members are chosen via a statistically valid sampling method and using known published sampling frames that cover 99% of the U.S. population. Address-Based Sampling (ABS), which is based on the U.S. Postal Service Delivery Sequence File, was used to select a probability sample of all U.S. households. This sample also comprises cell-phone households as well as non-Internet households. ABS is one of the most innovative means of obtaining nationally representative samples at minimum cost. Sampled non-Internet households are provided a laptop computer and free Internet service. KnowledgePanel consists of about 50,000 U.S. residents, aged 18 years or older, including persons of Hispanic origin that were selected probabilistically (for more information about the panel, see http://www.knowledgenetworks.com/knpanel/index.html). Between October 7, 2009, and November 19, 2009, a total of 2,212 Knowledge Network panelists aged 18 or over were invited to participate in the survey. Respondents received $15 for completing the survey.

Other Measures

In addition to the 38 health-literacy items, we also administered the short-form of the Test of Functional Health Literacy in Adults (S-TOFHLA) (Wallace, 2006). The TOFHLA and S-TOFHLA are timed reading comprehension tests that use the modified Cloze procedure, in which every 5th to 7th word in a passage is omitted and replaced with a blank space. The patient must select a word to fit into the blank spaces from the four multiple-choice options provided for each space. Baker, Williams, Parker, Gazmararian, and Nurss (1999) reported Cronbach's alpha of 0.68 for numeracy and 0.97 for the reading comprehension items of the S-TOFHLA. The overall correlation between the S-TOFHLA and the REALM was 0.80.

For validation analyses, we also asked participants to self-report their performance on the kind of skills being assessed in the survey. Specifically, we asked how easy or difficult it is to remember information they read versus hear; how easy or difficult it is to understand information they read versus hear; and how easy or difficult it is to explain a health issue to their doctor, find health information they need, and locate health information on the Web. Responses for each of the seven items included very difficult/difficult/somewhat easy/very easy. Socio-demographic characteristics and selected health-related data on respondents were available from Knowledge Networks.

Hypotheses

We hypothesized that participants with higher education levels and those who reported less difficulty with skills related to health literacy would have higher scores on the instrument. Given the different focus of the S-TOFLHA and this new measure (e.g., reading comprehension versus skills-based), we hypothesized that the two instruments would be moderately correlated, with the highest correlations for the print-based skill sets (print-prose and print-document), which involve more reading.

Statistical Analysis

We assessed several psychometric properties of the health-literacy items, including the percentage of correct responses and the correlation between each item and the total score, excluding the item of interest. We also estimated two-parameter logistic item response theory (IRT) models, using the Multilog software program (Thissen, Chen, & Bock, 2003). These models estimate a discrimination and threshold parameter for each item. The IRT discrimination parameter indicates how well the item distinguishes between persons with high versus low health literacy, as estimated based on the other items included on the scale. Ideally, items should have discrimination parameters of 1.0 or higher. The threshold indicates the difficulty of the item; items with larger thresholds are more difficult. We conducted higher order confirmatory factor analyses using the Mplus software program (Muthén & Muthén, 1998–2007) to assess whether the items clustered into five first-order factors based on skill set areas (print-prose, print-document, print-quantitative, oral, and Internet-based information seeking), as well as a single overarching second-order factor representing health literacy. We examined several fit indices, including the comparative fit index (CFI), Tucker-Lewis fit index (TLI), and root mean square error of approximation (RMSEA), to determine the most appropriate factor model. Sampling strata and weights were incorporated into the analyses to account for the survey design.

Based on the psychometric properties, we reduced the item pool by removing items that performed less well. Specifically, items were candidates for removal if they had low item-total correlations, factor loadings, or IRT discrimination parameters and/or very high percentages correct, suggesting poor discrimination. Additionally, we reviewed the content of items with similar threshold parameters to determine if they were redundant and should be removed from the scale. Balancing the statistical results with content validity considerations, we developed the final 25-item scale and computed scale scores as the percentage of items answered correctly.

To assess the construct validity of the scale, we conducted a series of analysis of variance (ANOVA) procedures to compare health-literacy scores by socio-demographic characteristics and participants' reports of their self-reported difficulty with various health-literacy-related skills. The latter items included a 4-point response scale ranging from very easy to very difficult, which was collapsed into easy versus difficult due to skewness of the response distribution. Further, we calculated the correlations between participants' scores on each of the four skill-set areas and the S-TOFHLA. Because no instruments have been validated for computer-based administration, few options were available from which to choose.

Finally, we investigated possible cut-points for classifying participants into three categories based on their health literacy levels: proficient, basic, and below basic. We conducted a series of receiver operating characteristic (ROC) analyses to determine which cut-points optimally distinguish participants based on their self-reported difficulty with understanding information they read and their highest level of educational achievement. For example, to determine the cut-point to distinguish proficient versus basic literacy, we compared persons who reported that understanding information they read is very easy with persons who reported that they find this task is not very easy (i.e., persons who respond somewhat easy, somewhat difficult, or very difficult). For each analysis, the cut-point was selected based on the point that maximizes the sum of sensitivity and specificity for distinguishing the two groups.

Results

Data Collection Results

Of the 2212 individuals sampled, 1559 panelists responded to the survey, for a completion rate of 71%. Among the respondents, 889 (57%) were able to see the test video and consented to complete the main survey. Because not all panelists agreed to complete the survey or to see the test video, the potential exists that survey estimates could be biased due to non-response and non-coverage, if not corrected. Therefore, the design-based sampling weights of study participants were adjusted upward to compensate for persons who failed to respond at both response levels, using a response propensity model-based approach. It took participants 45 minutes, on average, to complete the entire survey, which included the 38-item health literacy items, seven questions to self-report on one's health literacy skills, and the approximate 7-minute S-TOFHLA.

Participant Characteristics

The respondents were equally distributed across four age categories and about half were female (see Table 2). Respondents were categorized into three education-level groups: more than a high school education (n = 316); a high school education (n = 295); and less than a high school education (n = 278). Among the 278 respondents in the lowest education-level group, one-fourth (n = 77) had an 8th grade or lower education level. About two thirds of respondents were White, 13% were Black, 17% were Hispanic, and 6% classified themselves in to the “other” race/ethnicity category.

Table 2. Mean health literacy scores by participant characteristics and self-reported skills

			Health literacy data
Participant characteristic	N	Weighted %	Mean	SE	p
Gender
Male	458	48	70.3	1.5	.469
Female	431	52	68.8	1.5	REF
Age (years)
18–29	180	22	68.7	2.1	.617
30–44	205	25	70.7	2.5	.271
45–59	255	27	71.3	1.8	.143
60+	249	26	67.2	1.6	REF
Education
Less than high school	278	31	58.8	1.9	<.001
High school graduate	295	33	68.1	1.8	<.001
More than high school	316	36	80.3	1.2	REF
Race/ethnicity
Non-Hispanic White	664	64	73.8	1.2	.038
Non-Hispanic Black	83	13	55.9	3.0	.095
Hispanic	80	17	65.1	2.9	.951
Other	62	6	64.8	3.8	REF
Marital status
Married	489	46	73.5	1.3	<.001
Not married	400	54	66.1	1.7	REF
Employment status
Employed	470	51	73.8	1.5	.048
Retired	161	15	67.8	1.8	.845
Disabled	81	11	54.5	2.4	.002
Unemployed	177	23	68.4	2.1	REF
Geographic region
Northeast	161	18	66.9	2.5	.263
Midwest	206	22	69.5	2.6	.742
South	338	38	70.1	1.7	.838
West	184	22	70.7	2.2	REF
Self-reported skills
I am good at math
Agree	429	46	76.6	1.4	<.001
Disagree	450	54	64.0	1.5	REF
Remembering information I read
Easy	565	65	73.3	1.3	<.001
Difficult	317	35	63.4	1.7	REF
Remembering information I hear
Easy	584	66	72.5	1.2	.001
Difficult	295	34	64.5	1.8	REF
Understanding information I read
Easy	665	73	75.1	1.0	<.001
Difficult	211	27	56.7	2.2	REF
Understanding information I hear
Easy	691	77	73.3	1.1	<.001
Difficult	189	23	58.4	2.3	REF
Explaining a health issue to my doctor
Easy	682	74	72.1	1.2	<.001
Difficult	196	26	63.3	2.3	REF
Locating health information on the Internet
Easy	654	72	73.6	1.1	<.001
Difficult	222	28	60.6	2.3	REF
Finding health information I need
Easy	645	72	73.3	1.1	<.001
Difficult	231	28	61.7	2.4	REF

Reliability and Validity of the Instrument

The health-literacy scale demonstrated good internal consistency, with a Cronbach's alpha of 0.86. The psychometric properties of the individual items are shown in Table 3. The percentage of correct responses ranged from 24% to 91%. The higher order confirmatory factor analysis model fit well (CFI = 0.95, TLI = 0.98, and RMSEA = 0.03). The factor loadings for the four skill-set area factors are shown in Table 2; the loadings of these four factors on the overall health literacy factor are as follows: print-prose (0.98), print-document (0.98), print-quantitative (0.95), oral (0.85), and Internet (0.81). Almost all items had factor loadings and item-total correlations of 0.40 or higher and IRT discrimination parameters of 1.00 or higher, indicating good discrimination. One exception is item 13, which differs somewhat from the other items in that it requires respondents to perform a mathematical calculation. It is also by far the most difficult item, with only 24% of respondents answering it correctly. This item was retained to ensure content validity of the scale by measuring quantitative skills, which are a component of health literacy.

Table 3. Psychometric properties of health literacy items

Content area/stimulus	Item	% correct	Item-total correlation	Factor loading	IRT slope	IRT threshold
Print-prose
Cholesterol
If a person is at high risk for heart disease, which of the following levels of low density lipoprotein (LDL) cholesterol is best?		76	0.44	0.67	1.51	−1.15
Which set of low density lipoprotein (LDL) and high density lipoprotein (HDL) levels is best?		66	0.49	0.61	1.32	−0.78
Burns
Which of the following is probably not a second-degree burn?		67	0.41	0.59	1.20	−0.73
Lactose intolerance
Which of the following is a symptom of lactose intolerance?		91	0.36	0.65	1.44	−2.17
Stroke
Which of the following is not a sign of a stroke?		90	0.44	0.77	1.92	−1.79
Print-document
Drug side effects label
Which of the following problems could be caused by this medicine?		82	0.27	0.48	0.96	−1.86
Hospital map
If Joe was visiting someone in room 130 and wanted to go to the cafeteria, which of these places would he pass if he took the shortest route?		81	0.44	0.68	1.51	−1.50
Which of the following entrance is closest to the elevator?		80	0.40	0.59	1.22	−1.42
Medicine record
In the example listed in the first row of the table, when should the medicine be taken?		59	0.37	0.53	1.00	−0.47
Portion control
A person is making a salad and wants to add one serving of chopped, uncooked carrots. How much should she use?		64	0.38	0.43	0.73	−1.10
A person is cooking dinner for himself and he wants to include one serving from the meat and beans group. What should he choose?		75	0.39	0.65	1.48	−1.07
Insurance form
How much will the insurance company pay for the physical therapy received on 7/22/09?		53	0.39	0.56	0.98	−0.34
How much does the patient have to pay for the laboratory services received on 7/15/09?		80	0.44	0.57	1.06	−1.73
Print-quantitative
Nutrition label
How many grams of fiber are in two servings?		78	0.45	0.72	1.68	−1.22
If a person is on a 2,500 calorie diet, what percent of the daily value of saturated fat would he get from one serving?		24	0.13	0.34	0.66	2.15
Prostate cancer graph
More men die from prostate cancer than from other causes. Would you say this is true, false, or are you not sure?		80	0.46	0.72	1.64	−1.35
Who is more likely to die of prostate cancer?		82	0.42	0.73	1.55	−1.50
Oral
Telephone recording
If a person was worried about his cough, what number should he press?		58	0.37	0.60	0.96	−0.37
If a person wanted to check on the date and time of an appointment she already made, what number should she press?		58	0.38	0.57	0.89	−0.48
Sleep apnea
What do the muscles in the throat typically do when a person is sleeping?		86	0.40	0.68	1.26	−1.89
Lunges
What part of the body do lunge exercises work?		89	0.46	0.94	2.26	−1.76
Internet
Calories
John weighs 200 pounds and he walked at a medium pace on a firm surface for 30 minutes. How many calories did he burn?		59	0.52	0.72	1.73	−0.35
Kate weighs 150 pounds. Which activity would burn the most calories?		54	0.52	0.71	1.75	−0.16
Heart attack risk
What does this tool do?		68	0.57	0.87	2.23	−0.71
John is 39 years old and smokes. His blood pressure is 130/90 and he's on blood pressure medicine. His HDL cholesterol is 50 and his total cholesterol is 230. What is his estimated 10 year risk of a heart attack?		55	0.59	0.90	2.53	−0.25

On average, respondents answered 70% of the items correctly. The comparisons by sociodemographic characteristics indicated that respondents with higher education levels, who were non-Hispanic White, who were married and employed had higher health-literacy scores relative to their counterparts; whereas respondents who were disabled had lower scores than respondents who were unemployed. Across all of the self-reported skill items, respondents who reported less difficulty with the skill had higher scores on the health-literacy scale. Respondents tended to perform less well on tasks that required higher level skills, including some mathematical manipulations and the application of information to a specific scenario. As hypothesized, correlations between the health literacy domains and S-TOFHLA were highest for the print-prose, print-document, and print-quantitative skill areas with correlations of 0.47, 0.45, and 0.41, respectively (Table 4). The correlations were much lower for the Internet and oral literacy domains, which require fewer reading skills (r = 0.31 and r = 0.27, respectively; Table 4). Together, the results of the mean comparisons and correlations support the construct validity of the instrument.

Table 4. Correlation of health literacy domains and S-TOFHLA

Scale	Overall	Print-prose	Print-document	Print-quantitative	Internet	Oral
Health Literacy Domains
Overall	1.00
Print-Prose	0.77	1.00
Print-Document	0.87	0.61	1.00
Print-Quantitative	0.73	0.49	0.57	1.00
Internet	0.79	0.51	0.55	0.48	1.00
Oral	0.66	0.36	0.50	0.37	0.42	1.00
S-TOFHLA	0.47	0.45	0.41	0.34	0.31	0.24

Finally, the ROC analyses comparing the health-literacy scores to the categories for respondents' reporting difficulty understanding the information they read suggested a cut-point of 82 for distinguishing those who find the task very easy versus all others (sensitivity = 0.63, specificity = 0.67) and a cut-point of 70 for distinguishing among those who find the task easy versus difficult (i.e., very/somewhat easy vs. very/somewhat difficult) (sensitivity = 0.71, specificity = 0.65). Comparisons by education level indicated a cut-point of 82 for differentiating respondents with some college education versus no college education (sensitivity = 0.61, specificity = 0.72) and a cut-point of 74 for differentiating respondents with at least a high school education (i.e., high school or college) versus respondents with less than a high school education (sensitivity = 0.66, specificity = 0.68). Based on these analyses, we classified participants into three groups: proficient literacy (score ≥82), basic literacy (score of 70–81), and below basic literacy (score <70). In our sample, 40% of participants have proficient literacy, 22% have basic literacy, and 38% have below basic literacy.

Discussion

This new health-literacy instrument fills an existing gap in an important area of measurement, demonstrates robust psychometric properties, and is moderately correlated with an existing measure of literacy. Like the NAAL, it reflects a range of tasks and skills that adults are likely to face in their daily lives in the context of the U.S. health care system. Unlike the NAAL, however, this instrument measures the ability to obtain and use health information from print as well as non-print sources, which is more consistent with how people more typically receive their health information today. The mean score across the 25-item instrument was 70% correct. This is comparative with the NAAL, which aimed for a 67% probability of doing a task correctly (Kutner et al., 2006).

We advocate for measuring health literacy using a skills-based approach. With this as a priority for measurement, we used a computer-based data collection approach. This instrument is well suited for self-administered data collection via the Web and/or in-person data collection. Web-based data collection offers the advantages of cost and time efficiencies. The generalizability of Web-based data collection is increasing. An interviewer could be present with in-person data collection, and if an individual could not complete the survey on the computer, an alternate format could be administered, similar to an approach taken with the NAAL. Other modes of data collection are possible, but only for administering a subset of the items.

We identified a range of skills that individuals often need to monitor and manage their health in periods of health and illness. We included Internet information-seeking skills in the assessment, and given that a computer is required to complete the survey, some basic computer skills are needed unless an individual has assistance. Thus, one could argue that a person's health-literacy skills are contingent on or at least related to their computer skills. Being able to navigate search engines and websites has become increasingly important, as 75% of Americans use the Internet and, in 2008, 75% of these Internet users looked for health or medical information online (Pew Internet & American Life Project, 2009). The need for these skills will only increase as health-information technologies infiltrate the modern health care system, including the use of personal health records (PHRs).

Other methodological and conceptual issues are also worthy of consideration. The instrument uses real-world stimuli, as opposed to solely plain-language-approved stimuli. Thus, survey items needed to use the terminology consistent with these stimuli, which drove up the reading level somewhat. We plan to revisit selected survey items following this pilot study and adjust the reading level if possible. This raises questions about the influence of materials in the current health care system on health literacy and its measurement. If a respondent answers a question incorrectly, does that mean his or her health-literacy skill is deficient, or is the stimulus deficient, or is it a combination of the two? We are currently examining scoring of the items as it relates to the complexity of the stimuli and the difficulty of the survey items. We also are exploring the use of fictitious diseases or conditions to better control for prior knowledge. Also, there is merit to considering whether or not the stimuli could be changed or updated over time.

There is growing recognition of the need to account for the demands of the public health and health care systems when measuring health literacy at the individual level (Baker, 2006). In addition to increases in system-level demands, the expectations about consumers' roles and responsibilities as active participants in their health care are also expanding (McCormack, Treiman, Peinado, & Alexander, 2009; Olsen, Aisner, & McGinnis, 2007). These expectations include taking proactive steps, such as obtaining recommended preventive health services, eating a healthy diet and getting regular physical activity, recognizing signs of illness and disease, self-managing chronic illnesses, and navigating the health insurance system (Hibbard, 2009). These real-world expectations assume that consumers will use valid and often complex information to support these behaviors. This expanded role for consumers may raise parameters for what constitutes health literacy.

The overall aim of this study was to develop a publicly available skills-based instrument to measure health literacy. The HLSI can be used in intervention research studies that seek to improve health-literacy levels and for large-scale surveillance. We will continue to analyze the data, looking at the relationships between health literacy and health outcomes, and develop a short form of the instrument.

The authors wish to thank our expert panel members for their contribution to this project. We would also like to thank Rebecca Moultrie and Tania Fitzgerald for their assistance with the project and Shelton Jones for statistical support. This study was supported by National Cancer Institute grant R01 CA115861-01A2. The views expressed herein are solely those of the authors and do not necessarily represent the views of the National Cancer Institute.

Notes

*We were not able to measure speaking skills in this instrument.

The 2003 version of the NAAL put health tasks into prose, document, or quantitative scales based on the following definitions, which we attempted to adhere to: The prose literacy scale measured the knowledge and skills needed to search, comprehend, and use information from texts that were organized in sentences or paragraphs. The document literacy scale measured the knowledge and skills needed to search, comprehend, and use information from non-continuous texts in various formats. The quantitative scale measured the knowledge and skills needed to identify and perform computations using numbers embedded in printed materials (Kutner et al., 2006).

Key: (a) = Health promotion/disease prevention. (b) = Health care maintenance and treatment. (c) = Health care system navigation.

Note: REF = reference category.

Note: Factor loadings based on higher-order confirmatory factor analysis with four first-order factors and one second-order factor (CFI = 0.95, TLI = 0.98, RMSEA = 0.03).

Source: Used with permission from Mayo Foundation for Medical Education and Research. “Mayo,” “Mayo Clinic,” “MayoClinic.com,” “Mayo Clinic Health Information,” and the triple-shield Mayo logo are trademarks of MFMER. All Rights Reserved.

Appendix: Example Stimuli and Survey Items

Expanding Portions

Are you eating a variety of healthy foods, exercising, and still struggling with your weight? Some people may need to pay closer attention to portion control (managing the amount of food that they eat) as their total calorie intake determines their weight.

A serving isn't what they happen to put on their plate. It's a specific amount of food defined by common measurements, such as cups, ounces, or pieces. The serving sizes represented here are part of the Mayo Clinic Healthy Weight Pyramid—a food pyramid designed to promote weight loss and long-term health. Use these serving sizes in conjunction with a diet based on a variety of healthy foods. Add the right amount of regular physical activity, and a person will be well on their way to enjoying good nutrition and controlling their weight.

Vegetables

Until they're comfortable judging serving sizes, you may need to use measuring cups and spoons. A half a cup of cooked carrots, for example, equals one serving. Here are the recommended serving sizes for other vegetables:

Table

Food	Serving size
Raw leafy vegetables	= 2 cups
Raw vegetables, chopped	= 1 cup
Chopped, cooked or canned vegetables	= 1/2 cup

Meat and Beans

Familiar objects can help a person picture proper portions for meat, poultry, fish, and beans. For example, a 3-ounce serving of fish is about the size of a deck of cards. Here are the serving sizes for meat and meat substitutes:

Table

Food	Serving size
Cooked skinless poultry or fish	= 3 ounces
Cooked lean meat	= 1 1/2 ounces
Cooked legumes or dried beans	= 1/2 cup or about the size of an ice cream scoop
Egg	= 1 medium

A person is making a salad and wants to add one serving of chopped, uncooked carrots. How much should she use?

Table

Select one answer only

Explanation of Benefits

Table

ABC Insurance Company Plan
Member: John Doe
Patient: Jane Doe
Dates of service	Type of service	Submitted	Not covered	Covered	Co-pay	Plan liability	Patient responsibility	Note
7/22/09	Physical therapy	140.00	0.00	140.00	140.00	0.00	140.00	A
7/15/09	Laboratory	170.00	66.00	104.00	30.00	74.00	30.00	B
Total		310.00	66.00	244.00	170.00	74.00	170.00

How much will the insurance company pay for the physical therapy received on 7/22/09?

Table

Select one answer only

How much does the patient have to pay for the laboratory services received on 7/15/09?

Table

Select one answer only