Validating Arguments for Observational Instruments: Attending to Multiple Sources of Variation

REFERENCES

• Allen , J. P. , Pianta , R. C. , Gregory , A. , Mikami , A. and Lun , J. 2011 . An interaction-based approach to enhancing secondary school instruction and student achievement . Science , 333 : 1034 – 1037 .
   PubMed Web of Science ®Google Scholar
• American Educational Research Association/American Psychological Association/National Council on Measurement in Education . 1999 . Standards for educational and psychological testing Washington , DC : AERA. .
   Google Scholar
• Ball , D. L. , Thames , M. H. and Phelps , G. 2008 . Content knowledge for teaching: What makes it special? . Journal of Teacher Education , 59 : 389 – 407 .
   Web of Science ®Google Scholar
• Bangert , A. W. 2009 . Building a validity argument for the community of inquiry survey instrument . The Internet and Higher Education , 12 : 104 – 111 .
   Web of Science ®Google Scholar
• Bell , C. , Gitomer , D. , McCaffrey , D. , Hamre , B. , Pianta , R. and Qi , Y. 2012/this issue . An argument approach to observation protocol validity . Educational Assessment , 17 : 62 – 87 .
   Google Scholar
• Brennan , R. L. 2001 . Generalizability theory New York , NY : Springer-Verlag. .
   Google Scholar
• Chapelle , C. A. , Enright , M. K. and Jamieson , J. M. , eds. 2007 . Building a validity argument for the Test of English as a Foreign Language™ New York , NY : Routledge. .
   Google Scholar
• Cohen , D. K. , Raudenbush , S. and Ball , D. L. 2003 . Resources, instruction and research . Educational Evaluation and Policy Analysis , 25 : 119 – 142 .
   Web of Science ®Google Scholar
• Cronbach , L. J. 1988 . “ Five perspectives on validity argument. ” . In Test validity Edited by: Wainer , H. and Braun , H. 3 – 17 . Hillsdale , NJ : Erlbaum. .
   Google Scholar
• Fisher , R. A. 1921 . On the “probable error” of a coefficient of correlation deduced from a small sample . Metron , 1 ( 4 ) : 3 – 32 .
   Google Scholar
• Grossman , P. , Loeb , S. , Cohen , J. , Hammerness , K. , Wyckoff , J. , Boyd , D. and Lankford , H. May 2010 . Measure for measure: The relationship between measures of instructional practice in middle school English Language Arts and teachers' value-added scores. May , Cambridge , MA : National Bureau of Economic Research. . (NBER Working Paper 16015)
   Google Scholar
• Hawkins , R. E. , Margolis , M. J. , Dunning , S. J. and Norcini , J. J. 2010 . Constructing a validity argument for the mini-clinical evaluation exercise: A review of the research . Academic Medicine , 85 : 1453 – 1461 .
   PubMed Web of Science ®Google Scholar
• Hill , H. C. , Blunk , M. , Charalambous , C. , Lewis , J. , Phelps , G. C. , Sleep , L. and Ball , D. L. 2008 . Mathematical knowledge for teaching and the mathematical quality of instruction: An exploratory study . Cognition and Instruction , 26 : 430 – 511 .
   Web of Science ®Google Scholar
• Hill , H. C. , Charalambous , C. Y. and Kraft , M. 2012 . When rater reliability is not enough: Observational systems and a case for the G-study . Educational Researcher , 41 ( 2 ) : 56 – 64 .
   Web of Science ®Google Scholar
• Hill , H. C. and Herlihy , C. 2011 . “ Prioritizing teaching quality in a new system of teacher evaluation. ” . In Education Outlook Retrieved from http://www.aei.org/outlook/101089
   Google Scholar
• Hill , H. C. , Kapitula , L. R. and Umland , K. L. 2011 . A validity argument approach to evaluating value-added scores . American Educational Research Journal , 48 : 794 – 831 .
   Web of Science ®Google Scholar
• Hill , H. C. , Umland , K. U. , Litke , E. and Kapitula , L. 2012 . Teacher quality and quality teaching: Examining the relationship of a teacher assessment to practice . American Journal of Education , 118 : 489 – 519 .
   Web of Science ®Google Scholar
• Kane , M. T. 2001 . Current concerns in validity theory . Journal of Educational Measurement , 38 : 319 – 342 .
   Web of Science ®Google Scholar
• Kane , M. T. 2004 . Certification testing as an illustration of argument-based validation . Measurement: Interdisciplinary Research and Perspectives , 2 : 135 – 170 .
   Google Scholar
• Kane , M. T. 2006 . “ Validation. ” . In Educational measurement, , 4th ed. Edited by: Brennan , R. L. 17 – 64 . Westport , CT : American Council on Education/Praeger. .
   Google Scholar
• Kane , T. J. and Staiger , D. O. 2012 . Gathering feedback for teaching: Combining high-quality observations with student surveys and achievement gains. Seattle , WA : Bill & Melinda Gates Foundation. . Retrieved from http://www.metproject.org/reports.php
   Google Scholar
• Kane , T. J. , Taylor , E. S. , Tyler , J. and Wooten , A. March 2010 . Identifying effective classroom practices using student achievement data March , Cambridge , MA : National Bureau of Economic Research. . (NBER Working Paper 15803)
   Google Scholar
• Kennedy , M. M. 2010 . Attribution error and the quest for teaching quality . Educational Researcher , 39 : 591 – 598 .
   Web of Science ®Google Scholar
• Koretz , D. , Stecher , B. , Klein , S. and McCaffrey , D. 1994 . The Vermont Portfolio Assessment Program: Findings and implications . Educational Measurement: Issues and Practice , 13 ( 3 ) : 5 – 16 .
   Google Scholar
• Lane , S. , Liu , M. , Ankenmann , R. D. and Stone , C. A. 1996 . Generalizability and validity of a mathematics performance assessment . Journal of Educational Measurement , 3 : 71 – 92 .
   Google Scholar
• Llosa , L. 2008 . Building and supporting a validity argument for a standards-based classroom assessment of English proficiency based on teacher judgments . Educational Measurement: Issues and Practice , 27 ( 3 ) : 32 – 42 .
   Google Scholar
• Matsumura , L. C. , Garnier , H. E. , Slater , S. C. and Boston , M. D. 2008 . Toward measuring instructional interactions “at-scale.” . Educational Assessment , 13 : 267 – 300 .
   Google Scholar
• McCollum , J. A. , Hemmeter , M. L. and Hsieh , W. Coaching teachers for emergent literacy instruction using performance-based feedback . Topics in Early Childhood Education. , (in press)
   Google Scholar
• McGaghie , W. C. , Cohen , E. R. and Wayne , D. B. 2011 . Are United States medical licensing exam step 1 and 2 scores valid measures for postgraduate medical residency selection decisions? . Academic Medicine , 86 ( 1 ) : 48 – 52 .
   PubMed Web of Science ®Google Scholar
• Newton , X. 2010 . Developing indicators of classroom practice to evaluate the impact of a district mathematics reform initiative: A generalizability analysis . Studies in Educational Evaluation , 36 : 1 – 13 .
   Google Scholar
• Rosenbaum , P. R. 2002 . Observational studies, , 2nd ed. New York , NY : Springer-Verlag. .
   Google Scholar
• Rowland , T. , Turner , F. , Thwaites , A. and Huckstep , P. 2009 . Developing primary mathematics teaching: Reflecting on practice with the Knowledge Quartet London , , UK : Sage. .
   Google Scholar
• Sawchuck , S. 2009 . New teacher-evaluation systems face obstacles: Stimulus funds require districts to revamp teacher yardsticks . Education Week , Retrieved from http://www.edweek.org/ew/articles/2009/12/11/15evaluate.h29.html
   Google Scholar
• Schilling , S. G. and Hill , H. C. 2007 . Assessing measures of Mathematical Knowledge for Teaching: A validity argument approach . Measurement: Interdisciplinary Research and Perspectives , 5 : 70 – 80 .
   Google Scholar
• Shavelson , R. J. and Webb , N. M. 1991 . Generalizability theory: A primer Newbury Park , CA : Sage. .
   Google Scholar
• Taylor , E. S. and Tyler , J. H. March 2011 . The effect of evaluation on performance: Evidence from longitudinal student achievement data of mid-career teachers March , Cambridge , MA : National Bureau of Economic Research. . (NBER Working Paper 16877)
   Google Scholar