Health Literacy INDEX: Development, Reliability, and Validity of a New Tool for Evaluating the Health Literacy Demands of Health Information Materials

Figures & data

Table 1. Summary of reliability testing for Health Literacy INDEX tool

Reliability testing	Round 1 (n = 50)	Round 2 (n = 50)	Round 3 (n = 100)	Round 4 (n = 100)
Number of indicators	76	110	93	63
	n (%)	n (%)	n (%)	n (%)
Print materials	24 (48)	25 (50)	50 (50)	62 (62)
Websites	19 (38)	18 (36)	35 (35)	36 (36)
Audiovisual materials	7 (14)	7 (14)	15 (15)	2 (2)
National source or developer	29 (58)	30 (60)	60 (60)	65 (65)
Local source or developer	21 (42)	20 (40)	40 (40)	35 (35)

Table 2. Criteria addressed by the INDEX material evaluation tool

Criteria	Indicators/criterion (n)	Topics addressed
Plain language	8	• Writing style• Jargon• Numbers and data
Clear purpose	8	• Summaries and reviews• Number of key points• Purpose of material• Visual elements
Supporting graphics	6	• Graphics• Explanatory labels
User involvement	7	• Interactive strategies• Audience response
Skill-based learning	3	• Focus on behavioral skills
Audience appropriateness	6	• Audience identified• Match between audience and material
Instructions	6	• Technical instructions• User instructions
Development details	3	• Recency of information• Contact information
Evaluation methods	13	• Formative evaluation• Process evaluation• Outcome evaluation
Strength of evidence	3	• Evaluation findings

Table 3. Reliability of INDEX material evaluation tool

Criteria Indicators	n ^a	Coder agreement (%)	Kappa	Materials meeting indicator (%)
Plain language
Active voice used	94	97	0.00	97
Second person used	94	73	0.45	71
Jargon is limited and defined when used	93	88	0.14	87
Most sentences have 15 words or less	94	86	0.72	61
Reading grade level is eighth-grade or lower	94	93	0.78	21
Numbers used in key points	48	99	0.98	100
Data can be understood without math	48	98	0.96	100
Data phrasing is easy to understand	48	95	0.90	92
Clear purpose
Section topics introduced at beginning of material	94	81	0.10	94
Section topics are reviewed at end of material	94	67	0.32	45
Sections have five or fewer key points	94	76	0.51	52
Purpose can be identified at a glance	94	97	–0.01	99
Purpose can be identified after use	94	98	–0.01	99
Apparent purpose and stated purpose match	94	98	0.00	100
Visual elements communicate purpose	94	97	–0.01	99
Attention drawn to key elements	94	99	0.00	100
Supporting graphics
Graphics illustrate content	85	96	0.88	89
Graphics demonstrate behavior or record an event	83	77	0.60	57
Data in graphics are related to content	85	92	0.70	6
Graphics help readers understand abstract concepts	85	81	0.65	35
Most visual displays have explanatory labels	5	98	0.79	60
Design of visual displays is simple	5	100	1.00	100
Criteria Indicators	n ^a	Coder agreement (%)	Kappa	Materials meeting indicator (%)
User involvement
Use of advice column format	94	92	0.51	11
Use of frequently asked questions format	94	100	–––	100
Use of quiz or game format	94	96	0.78	12
Use of story or narrative format	94	87	0.62	22
Use of teasers	93	87	0.78	6
Audience asked to do something	94	87	0.64	21
Place given for audience response	94	86	0.61	22
Skill-based learning
Explicitly addresses health behavior	94	82	0.64	48
Provides specific action steps	35	81	0.62	98
Action steps or behaviors illustrated with examples	35	79	0.60	96
Audience appropriateness
Audience explicitly stated in title	94	89	0.70	23
Audience explicitly stated in text	94	77	0.54	42
Uses elements or visual cues related to audience	35	94	0.84	77
Uses data showing how audience affected by topic	44	65	0.14	80
Uses language appropriate for target audience	94	95	0.42	4
Mentions values, beliefs, experiences shared by audience	94	81	0.39	23
User instructions
Technical instructions provided	20	86	0.80	50
Technical instructions easy to find	10	86	0.80	50
Technical instructions comprehensive	10	76	0.63	30
User instructions provided	94	87	0.74	43
User instructions easy to find	40	81	0.62	44
User instructions comprehensive	40	83	0.66	53
Development details
Provides date for development or most recent review	94	90	0.78	77
Reviewed or developed within last two years	67	95	0.92	66
Contact information provided	92	84	0.38	84
Evaluation methods
Formative evaluation conducted	20	88	0.58	59
Process evaluation conducted	20	95	0.64	27
Outcome evaluation conducted	20	99	1.00	13
Formative evaluation collected background data	9	86	0.51	85
Formative evaluation included comparable participants	9	87	0.56	85
Process evaluation monitored frequency of use	4	94	0.56	50
Process evaluation monitored reach	4	94	0.56	100
Process evaluation monitored reactions	4	93	0.47	67
Process evaluation monitored implementation	4	95	0.64	100
Outcome evaluation assessed whether objectives met	3	99	1.00	100
Outcome evaluation used pre and post-test	3	99	1.00	100
Outcome evaluation had a comparison group	3	99	1.00	100
Outcome evaluation used random assignment	3	98	0.83	100
Strength of evidence
Material reflects formative evaluation findings	8	87	0.51	46
Implementation reflects process evaluation	1	92	0.19	60
Outcome evaluation found significant effect of material	3	98	0.85	100

Table 4. Indicators removed during refinement of INDEX

Deleted indicator	Materials meeting indicator (%)
Content
Clearly defined blocks of content (chunking) used	100
Sections are about a single topic	100
Content presented using a funnel approach	95
Content of sections and main purpose of material clearly linked	99
Section topics are well supported by main messages	96
Layout and typography
Use of consistent layout and sequence of information	100
Visual cuing devices used to highlight content	100
Adequate white space used	100
Use of color supports key points without distracting	99
High contrast used between text and surface	100
Headings
Headings used in material	96
Headings used before every section	95
Content and design of headings are consistent	95
Headings are easy to understand	99
Most headings represent section content clearly	99
Design and production
Design optimizes usability	98
Content is easy to use	100
Material is durable	96
Material has high production quality	99
Visual identity
Visual identity present throughout material	100
Visual identity helps identify titles and sections of material	100
Grammar and punctuation
Few grammatical or punctuation mistakes	100