The Reliability of Informal Reading Inventories: What Has Changed?

Abstract

Over time, criticisms related to the technical rigor of informal reading inventories (IRIs) have led many to question using these assessment instruments for high- or low-stakes purposes. In this article, I examine reliability evidence reported in 11 new and updated IRIs and make comparisons with Spector's earlier analysis that revealed fewer than half of the IRIs reported reliability. Based on the findings, I discuss ways in which IRIs have changed and use criteria recommended by Nitko, Salvia and Ysseldyke, and Spector to consider various educational uses of IRI data.

Notes

Note. Definitions are adapted from Salvia and Ysseldyke (2004) and Gall et al. (2007).

Note. Minimum levels recommended refer to reliability coefficients.

^a Recommended by Nitko (2001), Salvia & Ysseldyke (2004), and Spector (2005).

^b Recommended by Salvia and Ysseldyke (2004) and Spector (2005).

^c Recommended by Nitko (2001) and Spector (2005), as long as other information also will be taken into consideration in making classroom decisions.

Note. Dashes indicate that no data were reported. IRI = informal reading inventory.

Note. Dashes indicate that no data were reported. IRI = informal reading inventory; Y = yes (reliability evidence is acceptable for using IRI data to inform routine classroom instruction); N = no (reliability evidence is not acceptable); Tentative = conclusions drawn from the data must be viewed with caution because of limitations noted.

^a Refers to the grade levels of participants in studies conducted to establish the reliability levels of measures.

^b Based on .70 as the minimum criterion for acceptability of reliability coefficients derived from correlational approaches (Nitko, 2001; Spector, 2005) or 80% as the minimum criterion for acceptability of percentage of agreement when test data will be used for informing routine classroom instruction (Spector, 2005).

^c Unclear which scores are correlated (e.g., word recognition, comprehension, or both).

^d Reliability study data are missing (e.g., edition used for the reliability study, year in which data were collected, participant characteristics, the number of students by age or grade level, whether passage readings were done orally or silently) and/or reliability levels are reported across rather than within grades.

^e Unclear which subtests (e.g., silent reading comprehension, oral reading, listening comprehension, or all three) apply to the reliability estimates reported.

^f Data are missing because of insufficient sample size.

^g Acceptability is limited to the fourth-grade passages from Forms A and B in the second edition of the manual, read orally (Helgren-Lempesis & Mangrum, 1986).

^h Reports most coefficients exceeded .80 (individual coefficients are not provided) based on comprehension scores, and 75% were ≥.90.

Note. Dashes indicate that no data were reported. IRI = informal reading inventory; Y = yes (reliability evidence is acceptable for using IRI data to inform routine classroom instruction); N = no (reliability evidence is not acceptable); Tentative = conclusions drawn from the data must be viewed with caution because of limitations noted.

^a Refers to the grade levels of participants in studies conducted to establish the reliability of measures.

^b Based on .70 as the minimum criterion for acceptability of reliability coefficients derived from correlational approaches (Nitko, 2001; Spector, 2005) or 80% as the minimum criterion for acceptability of percentage of agreement when test data will be used for informing routine classroom instruction (Spector, 2005).

^c Reliability is reported across rather than within grades, an approach that tends to conceal significant between-grade differences and can inflate reliability estimates for all grades (Hammill et al., 1992; Salvia & Ysseldyke, 2004; Spector, 2005).

^d Scoring agreement method and reliability estimates are not reported; however, the test manual indicates that experts' and novices' scorings were identical for 16 of 30 instructional-level passage readings, varied by 1 point on 12 passages scored, and varied by 2 points on 2 passages scored.

^e Percentages of agreement or reliability coefficients are not reported.

^f Scoring agreement method and reliability estimates are not reported; however, the test manual indicates that four reading specialist experts' determinations of oral and silent instructional levels for one reader at each level “agreed” (Bader & Pearce, 2009, p. 166).

^g Statistical approaches and reliability estimates are not reported; however, the test manual indicates that 10 undergraduates' determinations of the oral instructional level for one elementary reader “agreed” (Bader & Pearce, 2009, p. 166) and that 9 of the 10 “agreed” (p. 166) on the silent reading level.

^h Average agreement rate reported rather than agreement for each reading level scored; reliability study data, such as edition and passages scored, are not reported in the IRI manual (Johns & L'Allier, 2003).

ⁱ Reliability study data, such as edition used and passages scored, are not reported in the IRI manual (Johns & L'Allier, 2004, 2007).

^j Reliability study data (e.g., specific passages scored) are missing, and/or reliability levels are reported across rather than within grades.

Note. Y = yes; N = no.

^a Refers to the grade levels of participants in studies conducted to establish the reliability levels of measures.

^b Based on the criteria recommended by Nitko (2001), Salvia and Ysseldyke (2004), and Spector (2005), as outlined in Table 2; accordingly, measures with reliability levels less than .70 are problematic and are not recommended for use.

^c Because details are lacking with regard to which specific informal reading inventory components the reliability coefficients apply to, conclusions drawn must be tentative and data limited to informing routine classroom instruction only.