Health Literacy INDEX: Development, Reliability, and Validity of a New Tool for Evaluating the Health Literacy Demands of Health Information Materials

Abstract

There is no consensus on how best to assess the health literacy demands of health information materials. Comprehensive, reliable, and valid assessment tools are needed. The authors report on the development, refinement, and testing of Health Literacy INDEX, a new tool reflecting empirical evidence and best practices. INDEX is comprised of 63 indicators organized into 10 criteria: plain language, clear purpose, supporting graphics, user involvement, skill-based learning, audience appropriateness, user instruction, development details, evaluation methods, and strength of evidence. In a sample of 100 materials, intercoder agreement was high: 90% or better for 52% of indicators, and above 80% for nearly all others. Overall scores generated by INDEX were highly correlated with average ratings from 12 health literacy experts (r = 0.89, p < .0001). Additional research is warranted to examine the association between evaluation ratings generated by INDEX and individual understanding, behaviors, and improved health. Health Literacy INDEX is a comprehensive tool with evidence for reliability and validity that can be used to evaluate the health literacy demands of health information materials. Although improvement in health information materials is just one aspect of mitigating the effects of limited health literacy on health outcomes, it is an essential step toward a more health literate public.

[Supplementary materials are available for this article. Go to the publisher's online edition of Journal of Health Communication for the following free supplemental resource: Health Literacy INDEX]

Health literacy, or the degree to which individuals have the capacity to obtain, communicate, process, and understand health information and services needed to make appropriate health decisions (U.S. Department of Health and Human Services, 2011), is an important predictor of health outcomes and health care utilization. The European Health Literacy Project has defined four dimensions of health literacy: accessing, understanding, appraising, and applying information to make everyday health decisions over the life span (Doyle, Cafferkey, & Fullam, 2012). The effects of limited health literacy on health and health care costs (Baker et al., 1996; Berkman, Pignone, DeWalt, & Sheridan, 2004; Gazmararian, Williams, Peel, & Baker, 2003; Nielsen-Bohlman, Panzer, & Kindig, 2004; Parker, Baker, Williams, & Nurss, 1995; Weiss, Hart, McGee, & D'Estelle, 1992; Weiss et al., 2005; Weiss & Palmer, 2004) have been conceptualized as being based on interactions between individuals' skill levels and the demands of health care and social systems (Nielsen-Bohlman et al., 2004). It is therefore critical to develop instruments that organizations can use to assess the health literacy demands of health materials in order to better understand the effects of health literacy on these important outcomes.

In past inquiries, investigating the health literacy demands of health information materials has often been narrowly defined as evaluating the reading level required to understand print materials. More than 300 articles have been published in public health and medical journals reporting on assessments of various types of health information materials, and these findings have been critical in drawing attention to the gap between materials' demands and the skills of the intended target audience (Nielsen-Bohlman et al., 2004; Rudd, Colton, & Schacht, 2000; Rudd, Comings, & Hyde, 2003). Because no consensus has been reached in the field on the best instrument with which to evaluate health information materials, various approaches have been used across prior studies. The most common approach has been to use readability formulas, which provide grade-level estimates of the reading difficulty of a piece of prose (i.e., words organized into sentence and paragraph format; Nielsen-Bohlman et al., 2004; Rudd et al., 2000; Rudd et al., 2003). Many different readability formulas exist, measuring somewhat different variables, but they are all generally based on aspects of writing style such as sentence and word length (Flesch, 1948; Fry, 1977; McLaughlin, 1969; Nielsen-Bohlman et al., 2004). However, these formulas capture only some of the cognitive demands that difficult health information materials place on users (Doak, Doak, & Root, 1996; Nielsen-Bohlman et al., 2004).

Therefore, other instruments have been developed in order to conduct more comprehensive evaluations of the health literacy demands of health information materials. The checklist developed by the National Cancer Institute, which describes key principles of effective low-literacy print materials (National Cancer Institute, 2003), and the Suitability Assessment of Materials instrument (Doak et al., 1996) capture multiple factors in addition to writing style that affect the demands of health information materials. The National Cancer Institute checklist assesses content, layout, and graphics; the Suitability Assessment of Materials instrument also includes the domains of learning motivation and cultural appropriateness. The Suitability Assessment of Materials instrument has provided useful assessments of health information materials, as described in the published literature (e.g., Kaphingst, Rudd, DeJong, & Daltroy, 2004; Kaphingst, Zanfini, & Emmons, 2006). However, these instruments do not have proven intercoder reliability, which can introduce subjective judgments into assessments of materials depending upon the characteristics of coders, and the validity of these existing instruments has not been systematically evaluated. In addition, neither the National Cancer Institute checklist nor the Suitability Assessment of Materials instrument include the domains of accessing and appraising health information materials. Although other instruments have been developed to evaluate health information materials, such as the EQIP tool (Moult, Franck, & Brady, 2004), and the Tool to Evaluate Materials Used in Patient Education (Clayton, 2009), these have not been widely adopted and do not comprehensively address all four domains of health literacy discussed earlier. Other instruments described in the literature are specific to certain types of health-related content, such as the DISCERN index (Charnock, Shepperd, Needham, & Gann, 1999; Khazaal et al., 2009) and the eHealth Engagement Scale (Lefebvre, Tada, Hilfiker, & Baur, 2010) or to single dimensions of health literacy, such as the Cultural Sensitivity Assessment Tool (Guidry & Walker, 1999) or the PMOSE/IKIRSCH instrument (Mosenthal & Kirsch, 1998), limiting their usefulness for other content or domains of health literacy.

Therefore, there is a need for a comprehensive, reliable, and valid instrument that can be used to evaluate the health literacy demands of health information materials that relate to accessing, understanding, appraising, and applying health information. Such an instrument could not only be critical in health literacy research but could also assist organizations and practitioners in choosing appropriate materials according to their objectives and audience. The present article describes the development of the Health Literacy INDEX, a comprehensive tool that can be used to evaluate the health literacy demands of health information materials. Our goals in creating the tool were twofold: (a) to develop a tool to be used in research examining the effects of health literacy demands of health information materials on health outcomes and (b) to help potential users of health information materials (i.e., organizations and practitioners) make informed decisions about which materials to use. By administering a standardized assessment of objective and observable criteria derived from health literacy science and practice, direct comparisons can be made between different materials. This article describes the development and testing of the tool, possible uses of the tool, and next steps to advance this area of the health literacy literature.

Method

Tool Development

Health Literacy INDEX was developed through an iterative process of literature review, operational definition, expert feedback, and reliability testing.

Literature Review

We conducted a literature review of the construct of health literacy to identify key concepts related to health literacy demands that might be included in the tool. In this literature review, we examined how health literacy demands of health information materials related to accessing (i.e., audience appropriateness, material development details), understanding (i.e., plain language, clearly stated purpose, use of graphics), appraising (i.e., evaluation findings), and applying (i.e., skill-based learning, user involvement, provision of instructions) health information. We searched the Medline, ERIC, PsycINFO, Social Work Abstracts, and Communication and Mass Media Complete databases using combinations of the following key words: health literacy, knowledge, comprehension, listening, speaking, writing, numeracy, reading, verbal communication skills, health information, risk information, access to health information, media literacy, health information seeking, discrimination in healthcare, cultural competency, language, and client–provider communication. We also reviewed existing assessment tools and guidelines developed for print and web-based materials (Bechtel & Davidhizar, 2000; Brach & Fraser, 2000; Center for Plain Language; Centers for Disease Control and Prevention, 1999, 2000; Davis, Williams, Marin, Parker, & Glass, 2002; Doak et al., 1996; DuBay, 2004; Gray & Leary, 1972; Grier & Bryant, 2005; Harvard School of Public Health; Klare, 1984; Kreuter, Lukwago, Bucholtz, Clark, & Sanders Thompson, 2003; Meade & Byrd, 1989; Meade, Diekmann, & Thornhill, 1992; Meyer, 1982; Michielutte, Bahnson, & Beal, 1990; Michielutte, Bahnson, Dignan, & Schroeder, 1992; Moriarty, 1995; National Cancer Institute, 2003; Office of Cancer Communications, 1989; Parks & Kreuter, 2007; Petty & Cacioppo, 1981; Pfizer Clear Communication Initiative, 2004; Plain Language Action and Information Network; Resnicow, Baranowski, Ahluwalia, & Braithwaite, 1999; Rudd, Kaphingst, Colton, Gregoire, & Hyde, 2004; Sabogal, Otero-Sabogal, Pasick, Jenkins, & Perez-Stable, 1996; Southern Institute on Children and Families, 2005; Stableford & Mettger, 2007; U.S. Securities and Exchange Commission, 1998; Wilson, 1995). We then developed a matrix of guidelines and practices. Although we did not quantify the strength of evidence for each guideline and/or practice because of wide variability in the evidence base, we considered whether research evidence existed for each guideline or practice and the number of studies or reviews that described and evaluated the guideline or practice. Best practices were defined as those practices consistently described in multiple resources or studies, or in reviews of the literature, as improving the health literacy demands of health information materials. We considered these best practices as possible indicators of good health literacy practice. This process led to an initial set of 76 indicators.

Operational Definitions

After the initial set of indicators was identified, we developed operational definitions for each that would allow coders to observe the indicator in health information materials. These operational definitions were refined during the reliability process, which is subsequently described. The first versions of these definitions were complex, using an ordinal response scale with different definitions for each level, and for materials delivered via different media. Because reliability was difficult to achieve using this approach, we adopted a simpler system in the third and fourth rounds of reliability testing. Each indicator had a single operational definition, stated in the positive (i.e., indicating good health literacy practice), and was coded as present or absent.

Reliability Testing

We conducted four rounds of reliability testing. For each round, we identified a convenience sample of 50 (rounds 1 and 2) or 100 (rounds 3 and 4) print, website, and audiovisual health information materials. The materials ranged widely on health topic (e.g., cancer, heart disease, sexually transmitted disease, mental health); format (e.g., brochures, toolkits); source/developer (e.g., government, non-profit organizations); target audience (e.g., public, patients); and outcome of interest (e.g., behavior change, awareness, education). All materials were independently coded by at least two master's-level coders who had been trained in using the INDEX tool. We calculated percent agreement and kappa value for each indicator to assess intercoder agreement (Cohen, 1960; Everitt & Hay, 1992).

The number of indicators used in each round of reliability testing and the characteristics of the samples are summarized in Table 1. After each round of reliability testing, the research team reviewed the results. Indicators with intercoder reliability falling below a priori thresholds of percent agreement (70%) and kappa (0.6) were reviewed and revised. Specifically, we identified items discordant between coders and reviewed the indicators and operational definitions as a team to identify and address possible sources of confusion. After conducting the first round of reliability testing, we split broad concepts into multiple, more specific indicators. After each subsequent round of reliability testing, indicator wording was then revised in order to make indicators and operational definitions less subjective and redundant indicators were deleted. The proportion of indicators that met the reliability threshold increased with each iteration of the tool and round of testing.

Table 1. Summary of reliability testing for Health Literacy INDEX tool

Reliability testing	Round 1 (n = 50)	Round 2 (n = 50)	Round 3 (n = 100)	Round 4 (n = 100)
Number of indicators	76	110	93	63
	n (%)	n (%)	n (%)	n (%)
Print materials	24 (48)	25 (50)	50 (50)	62 (62)
Websites	19 (38)	18 (36)	35 (35)	36 (36)
Audiovisual materials	7 (14)	7 (14)	15 (15)	2 (2)
National source or developer	29 (58)	30 (60)	60 (60)	65 (65)
Local source or developer	21 (42)	20 (40)	40 (40)	35 (35)

Expert Feedback

At five times during the development process, we sought feedback on the latest iteration of the tool from community and academic professionals directly involved in health literacy to refine the tool. A group (n = 12) of health literacy grantees and affiliated program staff from the Missouri Foundation for Health provided feedback twice, after the first and third rounds of reliability testing. A second group (n = 25) of community advisory network members from community-based agencies and organizations actively involved in health literacy provided feedback three times, before reliability testing and after the first and third rounds of reliability testing. These professionals critiqued the tool's criteria and indicators, suggested additions and deletions, and described how they might use the tool.

Examining Variability

After conducting the third round of intercoder reliability testing, we examined the level of variability across the sample on each indicator (i.e., what proportion of health information materials were meeting the standard set by the indicator). A major goal for the tool was to help users (e.g., organizations, practitioners) distinguish between health information materials with better and less favorable health literacy profiles, and indicators that were met by all resources would not help in making distinctions between materials. After calculating percent agreement and kappa for this round of reliability testing as described above, the two coders completed a consensus process to resolve all discrepancies in coding. Based on these consensus codes for 100 materials, we determined the proportion that met each indicator. We dropped indicators (n = 22) that were met by > 95% of materials from the tool to focus INDEX on criteria that varied across materials, thereby providing information that would inform the selection of health information materials.

Validity Testing

When we reached acceptable intercoder reliability and the final version of INDEX was established, we examined its validity. INDEX generates an overall score and criterion-specific scores for each material reviewed, as subsequently described. Using the overall score, we rank ordered 94 materials from highest (i.e., most favorable) to lowest (i.e., least favorable) score. We then selected eight materials from the top quartile and eight from the bottom quartile and created four distinct groups of materials. Each group contained two top quartile materials and two bottom quartile materials. We then recruited 12 academic researchers and practitioners currently working in health literacy to rate the materials, blind to the scores generated by INDEX. Each participant in the validity study rated one group of four materials on the basis of their own judgment, without using the INDEX tool. They were asked to sort the four materials into sets: two that they felt best reflected good health literacy practices and two that they felt did not reflect these practices. Each group of materials was evaluated by three different participants. We compared the expert ratings to ratings generated from the INDEX tool. The validity test activities were approved by the institutional review board at Washington University in St. Louis.

Analysis

We conducted all analyses using SAS 9.2 for Windows (Cary, NC). We calculated percent agreement and kappa values during reliability testing (Cohen, 1960; Everitt & Hay, 1992). In validity testing, we examined the association between INDEX scores and expert ratings using a Spearman correlation coefficient. After conducting reliability and validity testing, we examined descriptive statistics for a sample of materials that were evaluated using the final version of the INDEX tool.

Results

INDEX Tool

The final version of Health Literacy INDEX includes 63 indicators organized into 10 criteria: plain language, clear purpose, supporting graphics, user involvement, skill-based learning, audience appropriateness, user instruction, development details, evaluation methods, and strength of evidence (see Table 2). These criteria address features of text (e.g., plain language, clear purpose), features of graphics (e.g., supporting graphics), and content characteristics (e.g., user involvement, skill-based learning, audience appropriateness, user instruction, developmental details, evaluation methods, and strength of evidence). The definitions for each criterion and indicator were created by the team on the basis of the literature review. When a health information material is evaluated using INDEX, it receives a “yes” or “no” score for each indicator. Some indicators are contingent upon whether a previous indicator was met (e.g., evaluation indicators are only rated for materials for which an evaluation was conducted). An overall INDEX score is computed by taking the proportion of “yes” answers out of the number of applicable indicators. This score can therefore range from 0% to 100%. In the final reliability test sample, overall INDEX scores ranged from 18% to 69%. The same methods are used to compute criterion-specific scores. The final criteria and indicators are subsequently described.

Table 2. Criteria addressed by the INDEX material evaluation tool

Criteria	Indicators/criterion (n)	Topics addressed
Plain language	8	• Writing style• Jargon• Numbers and data
Clear purpose	8	• Summaries and reviews• Number of key points• Purpose of material• Visual elements
Supporting graphics	6	• Graphics• Explanatory labels
User involvement	7	• Interactive strategies• Audience response
Skill-based learning	3	• Focus on behavioral skills
Audience appropriateness	6	• Audience identified• Match between audience and material
Instructions	6	• Technical instructions• User instructions
Development details	3	• Recency of information• Contact information
Evaluation methods	13	• Formative evaluation• Process evaluation• Outcome evaluation
Strength of evidence	3	• Evaluation findings

Plain Language

Eight indicators assessed reading grade level, writing style (i.e., use of active voice, second person, sentence length), use of jargon, and use and understandability of data. Intercoder agreement was high (range = 73–99%) and kappa was generally acceptable (above the 0.6 threshold for five indicators; see Table 3). Among these indicators, materials were most likely to meet the indicators for presenting data (92–100%, when data were present) and least likely to meet the indicators for reading grade level (21%) and sentence length (61%).

Table 3. Reliability of INDEX material evaluation tool

Criteria Indicators	n ^a	Coder agreement (%)	Kappa	Materials meeting indicator (%)
Plain language
Active voice used	94	97	0.00	97
Second person used	94	73	0.45	71
Jargon is limited and defined when used	93	88	0.14	87
Most sentences have 15 words or less	94	86	0.72	61
Reading grade level is eighth-grade or lower	94	93	0.78	21
Numbers used in key points	48	99	0.98	100
Data can be understood without math	48	98	0.96	100
Data phrasing is easy to understand	48	95	0.90	92
Clear purpose
Section topics introduced at beginning of material	94	81	0.10	94
Section topics are reviewed at end of material	94	67	0.32	45
Sections have five or fewer key points	94	76	0.51	52
Purpose can be identified at a glance	94	97	–0.01	99
Purpose can be identified after use	94	98	–0.01	99
Apparent purpose and stated purpose match	94	98	0.00	100
Visual elements communicate purpose	94	97	–0.01	99
Attention drawn to key elements	94	99	0.00	100
Supporting graphics
Graphics illustrate content	85	96	0.88	89
Graphics demonstrate behavior or record an event	83	77	0.60	57
Data in graphics are related to content	85	92	0.70	6
Graphics help readers understand abstract concepts	85	81	0.65	35
Most visual displays have explanatory labels	5	98	0.79	60
Design of visual displays is simple	5	100	1.00	100
Criteria Indicators	n ^a	Coder agreement (%)	Kappa	Materials meeting indicator (%)
User involvement
Use of advice column format	94	92	0.51	11
Use of frequently asked questions format	94	100	–––	100
Use of quiz or game format	94	96	0.78	12
Use of story or narrative format	94	87	0.62	22
Use of teasers	93	87	0.78	6
Audience asked to do something	94	87	0.64	21
Place given for audience response	94	86	0.61	22
Skill-based learning
Explicitly addresses health behavior	94	82	0.64	48
Provides specific action steps	35	81	0.62	98
Action steps or behaviors illustrated with examples	35	79	0.60	96
Audience appropriateness
Audience explicitly stated in title	94	89	0.70	23
Audience explicitly stated in text	94	77	0.54	42
Uses elements or visual cues related to audience	35	94	0.84	77
Uses data showing how audience affected by topic	44	65	0.14	80
Uses language appropriate for target audience	94	95	0.42	4
Mentions values, beliefs, experiences shared by audience	94	81	0.39	23
User instructions
Technical instructions provided	20	86	0.80	50
Technical instructions easy to find	10	86	0.80	50
Technical instructions comprehensive	10	76	0.63	30
User instructions provided	94	87	0.74	43
User instructions easy to find	40	81	0.62	44
User instructions comprehensive	40	83	0.66	53
Development details
Provides date for development or most recent review	94	90	0.78	77
Reviewed or developed within last two years	67	95	0.92	66
Contact information provided	92	84	0.38	84
Evaluation methods
Formative evaluation conducted	20	88	0.58	59
Process evaluation conducted	20	95	0.64	27
Outcome evaluation conducted	20	99	1.00	13
Formative evaluation collected background data	9	86	0.51	85
Formative evaluation included comparable participants	9	87	0.56	85
Process evaluation monitored frequency of use	4	94	0.56	50
Process evaluation monitored reach	4	94	0.56	100
Process evaluation monitored reactions	4	93	0.47	67
Process evaluation monitored implementation	4	95	0.64	100
Outcome evaluation assessed whether objectives met	3	99	1.00	100
Outcome evaluation used pre and post-test	3	99	1.00	100
Outcome evaluation had a comparison group	3	99	1.00	100
Outcome evaluation used random assignment	3	98	0.83	100
Strength of evidence
Material reflects formative evaluation findings	8	87	0.51	46
Implementation reflects process evaluation	1	92	0.19	60
Outcome evaluation found significant effect of material	3	98	0.85	100

Clear Purpose

Eight indicators assessed clarity of purpose, use of visual elements to communicate purpose, review of section topics, and number of key points. Intercoder agreement was high (range = 67–99%), although kappa values varied greatly. Most materials met the clear purpose indicators. However, less than half of materials met the indicator of reviewing major topics at the end of the material (45%), and only half (52%) limited the number of key points to five or fewer per section.

Supporting Graphics

Six indicators assessed whether graphics illustrated content, demonstrated behaviors, or helped readers understand abstract concepts, used data related to content, had explanatory labels, and had a simple design. Intercoder agreement was high for these indicators (range = 77–100%) as were kappa values (range = 0.6–1.0). Among these indicators, materials with graphics generally met the indicator of using graphics to illustrate content (89%). However, few materials with graphics included data as part of the graphics that were related to content (6%).

User Involvement

Seven indicators assessed the use of common user involvement strategies (i.e., advice column, frequently asked questions, quiz or game, story or narrative format or teasers), whether the audience was asked to do something in the material, and whether space was provided for audience response. Intercoder agreement was high (range = 86–100%) and kappa was generally acceptable (range = 0.51–0.78). All materials used a frequently asked questions format at least once. Materials were much less likely to meet the indicators of asking the audience to do an activity (21%) or providing space for audience response (22%).

Skill-Based Learning

Three indicators assessed whether materials explicitly addressed health behaviors, provided specific action steps, and illustrated these with examples. Intercoder agreement ranged from 79% to 82% and kappa values from 0.60 to 0.64. Although only about half (48%) met the indicator of explicitly addressing a health behavior, of those, almost all provided specific action steps and illustrated steps with examples.

Audience Appropriateness

Six indicators assessed whether the target audience was explicitly identified and whether the material used peripheral (visual cues), evidential (data), linguistic (appropriate language), and/or sociocultural (values, beliefs, experiences) strategies for cultural appropriateness. Intercoder agreement ranged from 65% to 95%, although kappa values varied from 0.14 to 0.70. Among materials that included graphics, most (77%) met the indicator of use of peripheral cues related to the target audience, and among materials that included data, most (80%) used evidential strategies. However, materials were less likely to meet indicators for stating the audience in the title (23%) or using sociocultural strategies on the basis of values, beliefs, and experiences of the audience (23%).

User Instructions

Six indicators assessed whether technical and user instructions were provided, easy to find, and comprehensive. Intercoder agreement was high (range = 76–86%) and kappa values were acceptable (0.62–0.80). Among those with instructions, materials were most likely to meet the indicator for providing comprehensive user instructions (53%), but were least likely to meet the indicators for providing comprehensive technical instructions (30%).

Development Details

Three indicators assessed whether a development date was provided, the recency of last review or development, and provision of contact information to reach the developer. Intercoder agreement ranged from 84 to 95%, with kappa values from 0.38 to 0.92. Materials were most likely to meet the indicator for providing contact information (84%) and least likely to meet the indicator for having reviewed or developed the materials within the last two years, of those that provided dates (66%).

Evaluation Methods

Thirteen indicators assessed whether a formative, process, or outcome evaluation was conducted and, if so, the evaluation methods used. Twenty materials had any type of evaluation; nine had a formative evaluation, four a process evaluation, and three an outcome evaluation. Of the few materials to which these criteria applied, intercoder reliability was high (range = 86–99%) and kappa values ranged from 0.47 to 1.0. Among materials with any type of evaluation information available, few met the indicators for conducting a process (27%) or outcome (13%) evaluation. Among materials subjected to an outcome evaluation, all met the four indicators for methods used.

Strength of Evidence

Three indicators assessed whether the materials incorporated findings from formative, process or outcome evaluations. Intercoder reliability ranged from 87% to 98% and kappa values ranged from 0.19 to 0.85. Of the few materials with outcome evaluation findings, all met the indicator of reporting a significant effect of the material. Materials with formative evaluation findings were less likely to meet the indicator that the material explicitly reflected these findings (46%).

Indicators Not Included in Final Version of INDEX

In the penultimate round of reliability testing, we eliminated 22 indicators of good health literacy practice because of limited variability (see Table 4). Many of these indicators are widely recognized best practices in health literacy, and their exclusion from the final version of the tool in no way reflects doubts about their importance, but rather an effort to create a more parsimonious tool.

Table 4. Indicators removed during refinement of INDEX

Deleted indicator	Materials meeting indicator (%)
Content
Clearly defined blocks of content (chunking) used	100
Sections are about a single topic	100
Content presented using a funnel approach	95
Content of sections and main purpose of material clearly linked	99
Section topics are well supported by main messages	96
Layout and typography
Use of consistent layout and sequence of information	100
Visual cuing devices used to highlight content	100
Adequate white space used	100
Use of color supports key points without distracting	99
High contrast used between text and surface	100
Headings
Headings used in material	96
Headings used before every section	95
Content and design of headings are consistent	95
Headings are easy to understand	99
Most headings represent section content clearly	99
Design and production
Design optimizes usability	98
Content is easy to use	100
Material is durable	96
Material has high production quality	99
Visual identity
Visual identity present throughout material	100
Visual identity helps identify titles and sections of material	100
Grammar and punctuation
Few grammatical or punctuation mistakes	100

Validity

We found that overall scores generated by INDEX were highly correlated with average ratings from 12 health literacy experts (r = 0.89, p < .0001).

Discussion

This article describes the development and testing of INDEX, a materials evaluation tool comprising 63 indicators organized into 10 criteria. INDEX is a comprehensive material evaluation tool for assessing the health literacy demands of health information materials. The tool assesses a number of major domains, including characteristics of text, characteristics of graphics, and content characteristics. The tool builds upon accrued knowledge from several decades of health literacy research and practice, but brings strengths in intercoder reliability, validity testing, and comprehensiveness to address gaps in the literature. Because INDEX can be reliably administered by trained coders and has demonstrated validity against expert ratings, it should have value to researchers and practitioners in health literacy.

Although it was not the primary goal of this project, our findings also highlight places for improvement in health information materials. The INDEX results showed that most materials required a reading level of greater than eighth grade, a finding that is consistent with many other assessments of health information materials (Nielsen-Bohlman et al., 2004; Rudd et al., 2000). These findings therefore suggest that the language used in materials could not be understood by many U.S. adults (Doak et al., 1996), and that there is still a need for improvement in the use of plain language and techniques such as definitions of technical jargon. Some aspects of graphics could be improved, particularly making use of graphics to illustrate health behaviors or show abstract concepts (e.g., Doak et al., 1996). Strategies to actively engage users were rare in these materials, and users could benefit from the development of materials that ask them to actively participate and respond to information.

The findings suggest that many materials may not be appropriate for their intended audience. The target audience was identified in less than half of the materials, which may make it more difficult for organizations to determine how well the material would meet their needs. Although peripheral strategies for audience appropriateness (e.g., showing pictures of the target audience) were sometimes used, deeper sociocultural strategies that reflect the audience's values, beliefs and experiences were rarely used. This is consistent with findings from previous reviews (Parks & Kreuter, 2007). Application of sociocultural strategies based on clearly defined target audiences could improve the effectiveness of health information materials (Kreuter et al., 2003; Kreuter & McClure, 2004; Resnicow et al., 1999). Greater use of easy-to-find and comprehensive instructions could also assist material users.

Last, few materials reported having conducted any type of evaluation activities, and reporting on outcome evaluations was particularly rare. There is a strong need to evaluate health information materials in order to ensure that they meet the needs of the target audience, are implemented well and reach the target audience, and have the desired outcomes (Office of Cancer Communications, 1989; Valente, 2002). We acknowledge the possibility that materials might have been evaluated without the information being made public. However, this would not allow practitioners selecting materials to draw upon such evidence; we encourage materials developers to make this information public. In nearly all areas of public health and medicine, the use of evidence-based approaches and programs is increasingly important (Brownson, Baker, Leet, Gilliespie, & True, 2010; Sackett, Rosenberg, Gray, Haynes, & Richardson, 1996; Victora, Habicht, & Bryce, 2004), and this emphasis will likely be of increasing importance in health literacy as well. The scores generated by INDEX do not constitute evidence of effectiveness in affecting health-related outcomes but are rather indicators of adherence to practices shown or believed to affect health literacy demands. In the absence of evidence about specific materials, these kinds of indicators may be useful in guiding material development and selection in practice.

During the process of developing INDEX, we found that the health information materials evaluated were consistently rated as having good organization of content (e.g., chunking, organization, consistency); effective use of some layout and typography features (e.g., consistent layout, visual cuing devices, adequate white space, color, and high contrast); and consistent use of clear headers. These results suggest that the extensive public and private efforts to develop and disseminate plain language and health literacy trainings and materials may be having an impact in improving health information materials. We have included the removed criteria in a supplement to INDEX so that they can be incorporated into document development processes. We consider these criteria to be “minimal qualifying criteria” for evaluation by INDEX. In other words, materials that do not meet these basic criteria are not suitable for evaluation by the tool and would not be considered to be useful in practice. This then focuses the INDEX criteria on a more parsimonious set of indicators found to differentiate between health information materials in terms of health literacy demands.

The INDEX tool builds on the existing literature in health literacy and provides a way to evaluate a fairly comprehensive set of indicators to evaluate the health literacy demands of health information materials, but limitations of the study should be considered in interpreting our results. Because the indicators are scored as present or absent, variations across a single material are not captured. Given the convenience nature of the samples of materials reviewed, findings about the extent to which different indicators were met in these samples should not be generalized to all health information materials. Almost all of the health information materials were developed in the United States, and the reliability and validity of the tool in other social contexts needs to be evaluated. In addition, some of the discarded indicators may still be applicable in other countries with less health literacy experience. Creating a supplement to INDEX that includes these discarded items and testing the usefulness of this information to materials developers is an important next step. Most of the materials reviewed were print materials, and the reliability and validity of INDEX for social and interactive games needs to be determined. In addition, while many of the indicators could be reliably measured after a number of iterations of testing and development, some indicators still had lower kappa values. Kappa values for some indicators were low primarily due to the distribution of the underlying phenomenon or small sample sizes for which the indicator was applicable; this may explain why kappa values were sometimes near zero despite high percentage agreement between coders. In the validity testing, we asked experts to sort materials from the top and bottom quartiles of health information materials into two groups. Further validity testing could explore whether INDEX scores are related to ranked or more finely graded expert ratings in order to explore the validity of the instrument for materials scoring in the middle of the scale. Future research could explore whether a subset of criteria or indicators are related to usefulness for organizational users who are focused on improving understanding rather than access to information.

Additional research is needed to examine the association between INDEX scores and individuals' experience with health information materials and health-related outcomes. While many of the indicators in the tool are based on best practices and existing literature, the research base needs to be strengthened. Prior studies have shown that while improving readability may increase comprehension (Nielsen-Bohlman et al., 2004), this is not always be the case (Davis et al., 1998) and it is not clear what factors affect the differences in this association. One existing study showed that a large-scale plain language effort did not show significant improvements in prescription drug utilization (Shrank et al., 2009), but the research base for this type of large-scale initiative is limited. More studies are needed to address how and when changing the health literacy demands of health information materials affects downstream outcomes. Examining the associations between overall and criterion-specific INDEX scores and understanding, health behaviors, and health outcomes as part of intervention research studies can help us to understand the role of health literacy demands of health information materials in affecting individual outcomes. Important practice questions about implementation must also be answered, for example, understanding how providers and organizations would use the tool and how to make INDEX-based information publically available and easily accessible.

Conclusion

Health Literacy INDEX is a comprehensive tool that can be used to evaluate the health literacy demands of health information materials, and has demonstrated intercoder reliability and validity against ratings of 12 experts. In addition to possible use as a tool for organizations selecting health information materials, INDEX can also be used as a checklist for developers of materials. The criteria-level scores generated by the tool can guide developers in refining drafts of materials. As with any coding instrument, we found that training and practice in applying the INDEX tool was critical for coding reliability. Developing a training and certification process for coders is an important next step to making INDEX practical for widespread use while assuring the integrity of its scores. This type of training could be web-based with some didactic review of INDEX, followed by practice coding of sample materials and comparison of coders' ratings to standard ratings. We expect that this training would require 3–4 hours total.

We welcome feedback and dialogue on the indicators and criteria in the INDEX tool, and look forward to further refining the tool and filling in gaps. Multiple efforts to develop material evaluation tools like INDEX are already underway and being led by experienced leaders in health literacy. The field will benefit from these efforts and from continued collaboration to generate an essential set of criteria on the basis of best practices, evidence, and consensus. While improvement in health information materials is just one aspect of mitigating the effects of limited health literacy on health outcomes, it is an essential step toward patient activation, a better equipped professional field, and a more health literate public.

Acknowledgments

This research was supported by a contract from the Missouri Foundation for Health and by funding from the Barnes-Jewish Hospital Foundation. The authors thank the health literacy grantees and affiliated program staff from the Missouri Foundation for Health and the Community Advisory Network members for providing feedback during the tool development process. The authors also thank those who agreed to participate in our validity testing process for their invaluable contribution.

Notes

^a Number of health information materials to which criterion applied.