A Study of Reliability Across Multiple Raters When Using the NAEP and MDFS Rubrics to Measure Oral Reading Fluency

References

• Andrade, H. L., Du, Y., & Wang, X. (2008). Putting rubrics to the test: The effect of a model, criteria generation, and rubric-referenced self-assessment on elementary school students’ writing. Educational Measurement: Issues and Practices, 27, 3–13.
   Google Scholar
• Ashby, K. (2006). Prosody in skilled silent reading: Evidence from eye movements. Journal of Research in Reading, 29, 318–333.
   Web of Science ®Google Scholar
• Bader, M. (1998). Prosodic influences of reading syntactically ambiguous sentences. In J. Fodor & F. Ferreira (Eds.), Reanalysis in sentence processing (pp. 1–46). Dordrecht: Kluwer.
   Google Scholar
• Beckman, M. E. (1996). The parsing of prosody. Language and Cognitive Processes, 11, 17–67. doi:10.1080/016909696387213
   Web of Science ®Google Scholar
• Breland, H. M. (1983). The direct assessment of writing skill: A measurement review. New York: College Entrance Examination Board. doi:10.1002/j.2330-8516.1983.tb00032.x
   Google Scholar
• Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
   Google Scholar
• Carlson, K. (2009). How prosody influences sentence level comprehension. Language and Linguistics Compass, 3, 1188–1200. doi:10.1111/j.1749-818x.2009.00150.x
   Google Scholar
• Cortina, J. M. (1993). What is coefficient alpha? An examination of theory and applications. Journal of Applied Psychology, 78, 98–104. doi:10.1037/0021-9010.78.1.98
   Web of Science ®Google Scholar
• Council of Chief State School Officer (CCSSO) and National Governor’s Association (NGA). (2010). Common core standards for English language arts and literacy in history/social studies, science, and technical subjects. Retrieved from http://www.corestandards.org/assets/CCSSL_ELA%20Standards.pdf
   Google Scholar
• Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. Chicago, IL: Holt, Rinehart and Winston Inc.
   Google Scholar
• Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.
   PubMed Web of Science ®Google Scholar
• Daane, M. C., Campbell, J. R., Grigg, W. S., Goodman, M. J., & Oranje, A. (2005). Fourth-grade students reading aloud: NAEP 2002 Special Study of Oral Reading. Washington, DC: U.S. Department of Education, Institute of Education Sciences.
   Google Scholar
• Dowhower, S. L. (1991). Speaking of prosody: Fluency’s unattended bedfellow. Theory into Practice, 30, 165–175.
   Google Scholar
• Ericsson, A., & Pool, R. (2017). Peak: Secrets from the new science of expertise. Boston, MA: Mariner Books.
   Google Scholar
• Field, A. (2009). Discovering statistics using SPSS (3rd ed.). London: Sage.
   Google Scholar
• Fodor, J. D. (2002). Prosodic disambiguation in silent reading. In M. Hirotani (Ed.), Proceedings of the North East Linguistics Society (Vol. 32. pp. 112–132). Amherst, MA: GSLA.
   Google Scholar
• Frazier, L., Carlson, K., & Cliftonjr, C. (2006). Prosodic phrasing is central to language comprehension. Trends in Cognitive Sciences, 10, 244–249.
   PubMed Web of Science ®Google Scholar
• Goldman, S. R., Meyerson, P. M., & Cote, N. (2006). Poetry as a mnemonic prompt in children’s stories. Reading Psychology, 27, 345–376.
   Google Scholar
• Graham, M., Milanowski, A., & Miller, J. (2012). Measuring and promoting inter-rater agreement of teacher and principal performance ratings. Center for Education Compensation Reform: Author. Retrieved from files.eric.ed.gov/fulltext/ED532068.pdf
   Google Scholar
• Henson, R. (2001). Methods, plainly speaking - understanding internal consistency reliability estimates: A conceptual primer on coefficient alpha. Measurement and Evaluation in Counseling and Development, 34(3), 177.
   Web of Science ®Google Scholar
• Himmelmann, N. P., & Ladd, D. R. (2008). Prosodic description: An introduction for field workers. Language Documentation & Conservation, 2, 244–274.
   Google Scholar
• Jonsson, A., & Svingby, G. (2007). The use of scoring rubrics: Reliability, validity, and educational consequences. Educational Research Review, 2, 130–144.
   Google Scholar
• Klauda, S. L., & Guthrie, J. T. (2008). Relationships of three components of reading fluency to reading comprehension. Journal of Educational Psychology, 100, 310–321.
   Web of Science ®Google Scholar
• Knoch, U., Read, J., & von Randow, J. (2007). Re-training writers on-line: How does it compare with face-to-face training? Assessing Writing, 12, 26–43.
   Google Scholar
• Koriat, A., Greenberg, S. N., & Kreiner, H. (2002). The extraction of structure during reading: Evidence from reading prosody. Memory & Cognition, 30, 270–280.
   PubMed Web of Science ®Google Scholar
• Likert, R. (1932). A technique for the measurement of attitudes. Archives of Psychology, 140, 1–5.
   Google Scholar
• Loo, R. (2001). Motivational orientations toward work: An evaluation of the work preference inventory (Student form). Measurement and Evaluation in Counseling and Development, 33, 222–233.
   Web of Science ®Google Scholar
• Miller, J. M., & Schwanenflugel, P. J. (2006). Prosody of syntactically complex sentences in the oral reading of young children. Journal of Educational Psychology, 98, 839–853.
   PubMed Web of Science ®Google Scholar
• Miller, M. D., & Linn, R. L. (2000). Validation of performance-based assessments. Applied Psychological Measurement, 24, 367–378.
   Web of Science ®Google Scholar
• Moser, G. P., Sudweeks, R. R., Morrison, T. G., & Wilcox, B. (2014). Reliability of ratings of children’s expressive reading. Reading Psychology, 35, 58–79. doi:10.1080/02702711.2012.675417.
   Google Scholar
• Moskal, B. M., & Laydens, J. A. (2000). Scoring rubric development: Validity and reliability. Practical Assessment, Research & Evaluation, 7, Retrieved from http://pareonline.net/htm/v7n10.htm
   Google Scholar
• Myerberg, N. J. (1996). Inter-rater reliability on various types of assessments scored by school district staff. Paper presented at the Annual Meeting of the National Association of Research in Science Teaching, New York, NY. Retrieved from files.eric.ed.gov/fulltext/ED400291/pdf
   Google Scholar
• Nunnally, J. C. (1967). Psychometric theory. New York, NY: McGraw-Hill.
   Google Scholar
• Paige, D. D., Rasinski, T. V., Magpuri-Lavell, T., & Smith, G. S. (2014). Interpreting the relationships among prosody, automaticity, accuracy, and silent reading comprehension in secondary students. Journal of Literacy Research, 46, 123–156.
   Web of Science ®Google Scholar
• Paige, D. D., Rupley, W. H., Smith, G. S., Nichols, W., & Magpuri-Lavell, T. (2017). Is prosodic reading a strategy for comprehension? Journal of Educational Research Online, 9, 243–273. http://www.j-e-r-o.com/index.php/jero/article/view/774
   Web of Science ®Google Scholar
• Perfetti, C. A. (1985). Reading ability. New York: Oxford Press.
   Google Scholar
• Perfetti, C. A. (1988). Verbal efficiency in reading ability. In M. Daneman, G. E. MacKinnon, & T. G. Waller (Eds.) Reading research: Advances in theory and practice (pp. 109–143). New York: Academic Press. doi:10.1086/ahr/88.3.760
   Google Scholar
• Pinnell, G. S., Pikulski, J. J., Wixon, K. K., Campbell, J. R., Gough, P. B., & Beatty, A. S. (1995). Listening to children read aloud. Washington, DC: Office of Educational Research and Improvement, U.S. Department of Education.
   Google Scholar
• Rasinski, T. V. (1985). A study of factors involved in reader-text interactions that contribute to fluency in reading. (Unpublished doctoral dissertation). The Ohio State University, Columbus.
   Google Scholar
• Rasinski, T. V., Reutzel, D. R., Chard, D., Linan-Thompson, S., (2011). Reading fluency. In M. L. Kamil, P. D. Pearson, E. B., Moje, & P. Afferbach (Eds.), Handbook of reading research (Vol. 4, pp. 286–319). Mahwah, NJ: Erlbaum.
   Google Scholar
• Rasinski, T. V., Rikli, A., & Johnston, S. (2009). Reading fluency: More than automaticity? More than a concern for the primary grades? Literacy Research and Instruction, 48, 350–361.
   Google Scholar
• Reddy, Y. M., & Andrade, H. (2010). A review of rubric use in higher education. Assessment & Evaluation in Higher Education, 35, 435–448. DOI: 10.1080/02602930902862859.
   Web of Science ®Google Scholar
• Rezaei, A. E., & Lovorn, M. (2010). Reliability and validity of rubrics for assessment through writing. Assessing Writing, 15, 18–19.
   Google Scholar
• Ross-Fisher, R. L. (2005). Developing effective rubrics. Kappa Delta Pi, 41, 131–135. doi:10.1080/00228958.2005.10518823
   Google Scholar
• Sandermann, A. A., & Collier, R. (1997). Prosodic phrasing and comprehension. Language and Speech, 40, 391–409.
   Web of Science ®Google Scholar
• Schreiber, P. A. (1980). On the acquisition of reading fluency. Journal of Reading Behavior, 12, 177–186.
   Google Scholar
• Schreiber, P. A. (1987). Prosody and structure in children’s syntactic processing. In R. Horowitz & S. J. Samuels (Eds.), Comprehending oral and written language (pp. 243–270). New York: Academic Press.
   Google Scholar
• Schreiber, P. A. (1991). Understanding prosody’s role in reading acquisition. Theory into Practice, 30, 158–164.
   Google Scholar
• Schreiber, P. A., & Read, C. (1980). Children’s use of phonetic cues in spelling, parsing, and – maybe – reading. Bulletin of the Orton Society, 30, 209–224. doi:10.1007/BF02653719
   Google Scholar
• Schwanenflugel, P. J., Hamilton, A. M., Kuhn, M. R., Wisenbaker, J. M., & Stahl, S. A. (2004). Becoming a fluent reader: Reading skill and prosodic features in the oral reading of young readers. Journal of Educational Psychology, 96, 119–129. doi:10.1037/0022-0663.96.1.119
   PubMed Web of Science ®Google Scholar
• Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Thousand Oaks, CA: Sage Publications.
   Google Scholar
• Slowiaczek, M. L., & Clifton, C. Jr. (1980). Subvocalization and reading for meaning. Journal of Verbal Learning and Verbal Behavior, 19, 573–582.
   Google Scholar
• Snow, D. (1994). Phrase-final syllable lengthening and intonation in early child speech. Journal of Speech and Hearing Research, 37, 831–840.
   PubMedGoogle Scholar
• Speer, S. R., & Ito, K. (2009). Prosody in first language acquisition: Acquiring intonation as a tool to organize information in conversation. Language and Linguistics Compass, 3, 90–110. doi:10.1111/j.1749-818x.2008.00103.x.
   Google Scholar
• Streiner, D. L. (2003). Starting at the beginning: An introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80, 99–103.
   PubMed Web of Science ®Google Scholar
• Swets, B., Desmet, T., Hambrick, D. Z., & Ferreira, F. (2007). The role of working memory in syntactic ambiguity resolution: A psychometric approach. Journal of Experimental Psychology, General, 136, 64–81. DOI: 10.1037/0096-3455.136.1.64.
   PubMed Web of Science ®Google Scholar
• Tinsley, H. E. A., & Weiss, D. J. (1975). Interrater reliability and agreement of subjective judgments. Journal of Counseling Psychology, 22, 358–374. doi:10.1037/h0076640
   Web of Science ®Google Scholar
• Trainin, G., Hayden, H. E., Wilson, K., & Erickson, J. (2016). Examining the impact of QuickReads’ technology and print formats on fluency, comprehension, and vocabulary development for elementary students. Journal of Research on Educational Effectiveness, 9, 93–116. DOI: 10.1080/19345747.2016.1164778.
   Web of Science ®Google Scholar
• U. S. Department of Education (2002). National assessment of educational progress (NAEP) 2002 oral reading fluency study. Washington, DC: Institute of Education Sciences, National Center for Education Statistics. Retrieved from nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2006469
   Google Scholar
• Valencia, S. W., Smith, A. T., Reece, A. M., Li, M., Wixson, K. K., & Newman, H. (2010). Oral reading fluency assessment: Issues of construct, criterion, and consequential validity. Reading Research Quarterly, 45, 270–291. doi:10.1598/RRQ.45.3.1
   Web of Science ®Google Scholar
• Veenendaal, N. J., Goren, M. A., & Verhoeven, L. (2016). Bidirectional relations between text reading prosody and reading comprehension in the upper primary school grades: A longitudinal perspective. Scientific Studies of Reading, 2Q, 189–202. doi:10.1080/10888438.2015.1128939.
   Google Scholar
• Whalley, K., & Hansen, J. (2006). The role of prosodic sensitivity in children’s reading development. Journal of Research in Reading, 29, 288–303.
   Web of Science ®Google Scholar
• Wiederholt, J. L., & Bryant, B. R. (2012). Gray Oral Reading Test (5th ed.). Austin, TX: Pro-Ed.
   Google Scholar
• Wilson, M., & Case, H. (2000). An examination of variation in rater severity over time: A study in rater drift. In M. Wilson & G. Engelhard (Eds.), Objective measurement: Theory into practice (pp. 113–133). Stamford, CT: Ablex.
   Google Scholar
• Zutell, J., & Rasinski, T. V. (1991). Training teachers to attend to their students’ oral reading Fluency. Theory into Practice, 30, 211–217.
   Google Scholar