Updating the Duplex Design for Test-Based Accountability in the Twenty-First Century

REFERENCES

• Argyris , C. 1997 . Learning and Teaching: A Theory of Action Perspective . Journal of Management Education , 21 ( 1 ) : 9 – 26 .
   Google Scholar
• Atkinson , R. C. and Geiser , S. 2009 . Reflections on a century of college admissions tests . Educational Researcher , 38 ( 9 ) : 665 – 676 .
   Web of Science ®Google Scholar
• Attali , Y. and Powers , D. 2008 . Effect of immediate feedback and revision on psychometric properties of open-ended GRE® subject test items (RR-08-21) , Princeton, NJ : Educational Testing Service .
   Google Scholar
• Baker , E. L. and Linn , R. L. 2004 . “ Validity issues for accountability systems ” . In Redesigning Accountability , Edited by: Fuhrman , S. and Elmore , R. 47 – 72 . New York : Teachers College Press .
   Google Scholar
• Bejar , I. I. , Braun , H. and Tannenbaum , R. 2007 . “ A prospective, predictive and progressive approach to standard setting ” . In Assessing and modeling cognitive development in school: Intellectual growth and standard setting , Edited by: Lissitz , R. W. 1 – 30 . Maple Grove, MN : JAM Press .
   Google Scholar
• Bejar , I. I. , Graf , E. A. , Oranje , A. and Veldkamp , B. P. Application of optimization methods to assessment design . Paper presented in April at the National Council of Measurement in Education . New York, NY.
   Google Scholar
• Bejar, I. I., Lawless, R. R., Morley, M. E., Wagner, M. E., Bennett, R. E., & Revuelta, J. (2003). A feasibility study of on-the-fly item generation in adaptive testing. Journal of Technology, Learning, and Assessment, 2(3) http://www.jtla.org (http://www.jtla.org) (Accessed: 1 March 2010 ).
   Google Scholar
• Bennett , R. E. Cognitively based assessment of, for, and as learning (CBAL): A preliminary theory of action for summative and for mative assessment . Measurement: Interdisciplinary Research and Perspectives . Vol. 8(2–3) , pp. 70 – 91 .
   Google Scholar
• Bennett , R. E. and Gitomer , D. H. 2009 . “ Transforming K–12 assessment: Integrating accountability testing, formative assessment and professional support ” . In Educational assessment in the 21st century , Edited by: Wyatt-Smith , C. and Cumming , J. 43 – 61 . New York : Springer .
   Google Scholar
• Betebenner, D. (2005). The importance of performance standards in measures of student growth. Paper presented at the CCSSO http://216.250.255.51/content/pdfs/Betebenner2005NCLSA.pdf (http://216.250.255.51/content/pdfs/Betebenner2005NCLSA.pdf) (Accessed: 1 March 2010 ).
   Google Scholar
• Betebenner , D. 2009 . Norm- and criterion-referenced student growth . Educational Measurement: Issues and Practice , 28 ( 4 ) : 42 – 51 .
   Google Scholar
• Betebenner , D. W. , Shang , Y. , Xiang , Y. , Zhao , Y. and Yue , X. 2008 . The impact of performance level misclassification on the accuracy and precision of percent at performance level measures . Journal of Educational Measurement , 45 ( 2 ) : 119 – 137 .
   Web of Science ®Google Scholar
• Black , P. and Wiliam , D. 1998 . Inside the black box: Raising standards through classroom assessment . Phi Delta Kappan , 80 ( 2 ) : 139 – 148 .
   Web of Science ®Google Scholar
• Bock , R. D. and Mislevy , R. J. 1981 . An item response curve model for matrix-sampling data: The California grade-three assessment . New Directions for Testing and Measurement , 10 : 65 – 90 .
   Google Scholar
• Bock , R. D. and Mislevy , R. J. 1988 . Comprehensive educational assessment for the states: The Duplex Design . Educational Evaluation and Policy Analysis , 10 ( 2 ) : 89 – 105 .
   Google Scholar
• Bransford , J. D. , Brown , A. I. and Cocking , R. R. 1999 . How people learn , Washington, DC : National Academy Press .
   Google Scholar
• Braun , H. 2009 . Discussion: With choices come consequences . Educational Measurement: Issues and Practice , 28 ( 4 ) : 52 – 55 .
   Google Scholar
• Brennan , R. L. , Ping , Y. and Kane , M. T. 2003 . Methodology for examining the reliability of group mean difference scores . Journal of Educational Measurement , 40 ( 3 ) : 207 – 230 .
   Google Scholar
• Briggs , D. C. and Weeks , J. P. 2009 . The impact of vertical scaling decisions on growth interpretations . Educational Measurement: Issues and Practice , 28 ( 4 ) : 3 – 14 .
   Google Scholar
• Carlson, D. (undated). Focusing state educational accountability systems: Four methods of judging school quality and progress [Electronic version]. Council of Chief State School Officers. http://www.ccsso.org/content/pdfs/Dale020402.doc (http://www.ccsso.org/content/pdfs/Dale020402.doc) (Accessed: 28 August 2007 ).
   Google Scholar
• Clauser , B. E. , Kane , M. T. and Swanson , D. B. 2002 . Validity issues for performance-based tests scored with computer-automated scoring systems . Applied Measurement in Education , 15 ( 4 ) : 413 – 432 .
   Web of Science ®Google Scholar
• Cronbach , L. J. , Linn , R. L. , Brennan , R. L. and Haertel , E. 1995 . Generalizability analysis for educational assessments . Educational and Psychological Measurement , 57 ( 3 ) : 373 – 399 .
   Google Scholar
• Darling-Hammond, L. (2010). Performance counts: Assessment systems that support high-quality learning. Washington, DC, Council of Chief State School Officers. http://www.ccsso.org/publications/details.cfm?PublicationID=381 (http://www.ccsso.org/publications/details.cfm?PublicationID=381) (Accessed: 1 March 2010 ).
   Google Scholar
• Darling-Hammond , L. and McCloskey , L. 2008 . Assessment for learning around the world: What would It mean to be internationally competitive? . Phi Delta Kappan , 90 ( 4 ) : 263 – 272 .
   Web of Science ®Google Scholar
• Deane , P. 2010 . Rethinking K–12 writing assessment , Unpublished manuscript .
   Google Scholar
• Dorans , N. , Holland , P. and Pomerich , M. , eds. 2007 . Linking and aligning scores and scales , New York : Springer-Verlag .
   Google Scholar
• Drasgow , F. , Luecht , R. and Bennett , R. E. 2006 . “ Technology and testing ” . In Educational measurement , 4th , Edited by: Brennan , R. L. 471 – 515 . Westport, CT : Praeger .
   Google Scholar
• Dunn , J. L. and Allen , J. 2009 . Holding schools accountable for the growth of nonproficient students: Coordinating measurement and accountability . Educational Measurement: Issues and Practice , 28 ( 4 ) : 27 – 41 .
   Google Scholar
• Embretson , S. 1993 . “ Psychometric models for learning and cognitive processes ” . In Test theory for a new generation of tests , Edited by: Frederiksen , N. , Mislevy , R. J. and Bejar , I. I. 125 – 150 . Hillsdale, NJ : Erlbaum .
   Google Scholar
• Embretson , S. E. 1983 . Construct validity: Construct representation versus nomothetic span . Psychological bulletin , 93 : 179 – 197 .
   Web of Science ®Google Scholar
• Ferrara , S. , Phillips , G. W. , Williams , P. L. , Leinwand , S. , Mahoney , S. and Ahadi , S. 2007 . “ Vertically articulated performance standards: An exploratory of inferences about achievement and growth ” . In Assessing and modeling cognitive development in school: Intellectual growth and standard setting , Edited by: Lissitz , R. W. Maple Grove, MN : Jam Press .
   Google Scholar
• Feuer , M. J. 2008 . “ Future directions for educational accountability: Notes for a political economy of measurement ” . In The future of test-based educational accountability , Edited by: Ryan , K. E. and Shepard , L. A. 115 – 137 . New York : Routledge .
   Google Scholar
• Feuer , M. J . Externalities of testing: Lessons from the Blizzard of 2010 . Measurement: Interdisciplinary Research and Perspectives . Vol. 8 , pp. 59 – 69 .
   Google Scholar
• Forsyth , R. A. NAEP frameworks and achievement levels . Proceedings of achievement levels workshop: Boulder, CO: National Assessment Governing Board (Available from ERIC, Document ED458220) . Edited by: Bourque , M. L.
   Google Scholar
• Gambell , T. and Hunter , D. 2004 . Teacher scoring of large-scale assessment: Professional development or debilitation? . Journal of Curriculum Studies , 36 ( 6 ) : 697 – 724 .
   Web of Science ®Google Scholar
• Goldberg , G. L. and Roswell , B. S. 2000 . From perception to practice: The impact of teachers' scoring experience on performance-based instruction and classroom assessment . Educational Assessment , 6 ( 4 ) : 257 – 290 .
   Google Scholar
• Haberman , S. 2008 . When can subscores have value? . Journal of Educational and Behavioral Statistics , 33 ( 2 ) : 204
   Google Scholar
• Haertel , E. H. 1999 . Validity arguments for high-stakes testing: In search of the evidence . Educational Measurement: Issues and Practice , 18 ( 4 ) : 5 – 9 .
   Google Scholar
• Haertel , E. H. and l , Herman, J. 2005 . A historical perspective on validity arguments for accountability testing . Yearbook of the National Society for the Study of Education , 104 ( 2 ) : 1 – 34 .
   Google Scholar
• Hambleton , R. K. , Brennan , R. L. , Brown , W. , Dodd , B. , Forsyth , R. A. Mehrens , W. A. 2000 . A response to “Setting Reasonable and Useful Performance Standards” in the National Academy of Science' Grading the Nations Report Card . Educational Measurement: Issues and Practice , 19 ( 2 ) : 5 – 14 .
   Google Scholar
• Harris , D. J. 2007 . “ Practical issues in vertical scaling ” . In Linking and aligning scores and scales , Edited by: Dorans , N. J. , Pommerich , M. and Holland , P. W. 231 – 252 . New York : Springer-Verlag .
   Google Scholar
• Hill, R. (2001). Issues related to the reliability of school accountability scores. The National Center for the Improvement of Educational Assessment. http://www.nciea.org/publication/RILS2000Paper_Hill01.pdf (http://www.nciea.org/publication/RILS2000Paper_Hill01.pdf) (Accessed: 1 March 2010 ).
   Google Scholar
• Kane , M. T. 2006 . “ Validation ” . In Educational measurement , 4th , Edited by: Brennan , R. L. 17 – 64 . Westport, CT : Praeger .
   Google Scholar
• Kingsbury, G. G. (2007). CAT in the K–12 schools: 50 million CATs and counting. In D. J. Weiss (Ed.), Proceedings of the 2007 GMAC Conference on Computerized Adaptive Testing. http://www.psych.umn.edu/psylabs/catcentral/cat07schedule.htm (http://www.psych.umn.edu/psylabs/catcentral/cat07schedule.htm)
   Google Scholar
• Kiplinger , V. L. 2008 . “ Reliability of large-scale assessment and accountability systems ” . In Reliability of large-scale assessment and accountability systems , Edited by: Ryan , K. E. and Shepard , L. A. 93 – 114 . New York : Routledge .
   Google Scholar
• Kiplinger , V. L. and Hamilton , L. S. 2008 . “ Equating and linking of educational assessments in high stakes accountability systems ” . In The future of test-based educational accountability , Edited by: Ryan , K. E. and Shepard , L. A. 115 – 137 . New York : Routledge .
   Google Scholar
• Kirsch, I. S., Jungeblut, A., Jenkins, L., & Kolstad, A. (1993). Adult literacy in America: A first look at the findings of the National Adult Literacy Survey. Washington, DC: National Center for Education Statistics. http://nces.ed.gov/pubs93/93275.pdf (http://nces.ed.gov/pubs93/93275.pdf) (Accessed: 1 March 2010 ).
   Google Scholar
• Koretz , D. 2005 . “ Alignment, high stakes, and the inflation of test scores ” . In Uses and misuses of data in accountability testing. Yearbook of the National Society for the Study of Education, vol. 104, Part 2 , Edited by: Herman , J. and Haertel , E. 99 – 118 . Malden, MA : Blackwell .
   Google Scholar
• Koretz , D. 2006 . “ Testing for accountability in K–12 ” . In Educational measurement , 4th , Edited by: Brennan , R. 531 – 578 . Westport, CT : Praeger .
   Google Scholar
• Koretz , D. 2008 . “ Further steps toward the development of an accountability-oriented science of measurement ” . In The future of test-based educational accountability , Edited by: Ryan , K. E. and Shepard , L. A. 71 – 91 . New York : Routledge .
   Google Scholar
• Koretz , D. M. and Béguin , A. Self-monitoring assessments for educational accountability systems . Measurement: Interdisciplinary Research and Perspectives . Vol. 8(2–3) , pp. 92 – 109 .
   Google Scholar
• Lane , S. and Stone , C. A. 2002 . Strategies for examining the consequences of assessment and accountability programs . Educational Measurement: Issues and Practice , 21 ( 1 ) : 23 – 30 .
   Google Scholar
• Leighton , J. P. and Gierl , M. J. , eds. 2007 . Cognitive diagnostic assessment in education: Theory and applications , New York : Cambridge University Press .
   Google Scholar
• Levin , H. M. 1974 . A conceptual framework for accountability in education . School Review , 82 ( 3 ) : 363 – 391 .
   Web of Science ®Google Scholar
• Linn , R. L. 2004 . “ Accountability models ” . In Redesigning accountability , Edited by: Fuhrman , S. and Elmore , R. 73 – 93 . New York : Teachers College Press .
   Google Scholar
• Linn R. L. Conflicting demands of No Child Left Behind and state systems: Mixed messages about school performance Education Policy Analysis Archives 2005a 13 33 http://epaa.asu.edu/ojs/article/view/2138 (http://epaa.asu.edu/ojs/article/view/2138) (Accessed: 16 March 2010 ).
   Google Scholar
• Linn, R. L. (2005b). Fixing the NCLB accountability system (CRESST Policy Brief 8) Los Angeles: CA: CRESST. http://www.cse.ucla.edu/products/policy/cresst_policy8.pdf (http://www.cse.ucla.edu/products/policy/cresst_policy8.pdf) (Accessed: 15 December 2007 ).
   Google Scholar
• Linn , R. L. 2008 . “ Educational accountability systems ” . In The future of test-based educational accountability , Edited by: Ryan , K. E. and Shepard , L. A. 3 – 24 . New York : Routledge .
   Google Scholar
• Linn , R. L. and Burton , E. 1994 . Performance-based assessment: Implications of task specificity . Educational measurement: Issues and practice , 13 ( 1 ) : 5 – 8 .
   Google Scholar
• Lissitz, R. W., & Huynh, H. (2003). Vertical equating for state assessments: Issues and solutions in determining adequate yearly progress and school accountability. Practical Assessment, Research, & Evaluation, 8(10). http://PAREonline.net.getvn.asp?v=8&n=10 (http://PAREonline.net.getvn.asp?v=8&n=10) (Accessed: 1 March 2010 ).
   Google Scholar
• Lord , F. M. 1980 . Applications of item-response theory to practical testing problems , Hillsdale, NJ : Erlbaum .
   Google Scholar
• Messick , S. 1994 . The interplay of evidence and consequences in the validation of performance assessments . Educational Researcher , 23 ( 2 ) : 13 – 23 .
   Google Scholar
• Mislevy , R. J. 2007 . Validity by design . Educational Researcher , 36 ( 8 ) : 463 – 469 .
   Google Scholar
• Mislevy , R. J. and Haertel , G. D. 2006 . Implications of evidence-centered design for educational testing . Educational Measurement: Issues and Practice , 25 ( 4 ) : 6 – 20 .
   Google Scholar
• Mislevy , R. J. , Steinberg , L. S. and Almond , R. G. 2003 . On the structure of educational assessments . Measurement: Interdisciplinary Research and Perspectives , 1 ( 1 ) : 3 – 62 .
   Google Scholar
• Myers , M. 2003 . “ What can computers and AES contribute to a K–12 writing program? ” . In Automated essay scoring: A cross-disciplinary perspective , Edited by: Shermis , M. D. and Burstein , J. Mahwah, NJ : Lawrence Erlbaum .
   Google Scholar
• National Council of Teachers of Mathematics. (2006). Curriculum focal points for prekindergarten through grade 8 mathematics. http://www.nctm.org/focalpoints/downloads.asp (http://www.nctm.org/focalpoints/downloads.asp) (Accessed: 12 September 2006 ).
   Google Scholar
• Oregon Department of Education. (2008). 2007–2008 Technical Report: Oregon Statewide Assessment System. http://www.ode.state.or.us/search/page/?=1305 (http://www.ode.state.or.us/search/page/?=1305) (Accessed: 26 March 2010 ).
   Google Scholar
• Patz , R. J. and Yao , L. 2007 . “ Methods and models of vertical scaling ” . In Linking and aligning scores and scales , Edited by: Dorans , N. J. , Pommerich , M. and Holland , P. W. 253 – 272 . New York : Springer-Verlag .
   Google Scholar
• Pellegrino , J. W. 2000 . A response to ACT's technical advisers on NAEP standard setting . Educational Measurement: Issues and Practice , 19 ( 2 ) : 14 – 15 .
   Google Scholar
• Pellegrino , J. W. , Chudowsky , N. and Glaser , R. 2001 . Knowing what students know: The science and design of educational assessment , Washington, DC : National Academy Press .
   Google Scholar
• Perie , M. 2008 . A guide to understanding and developing performance-level descriptors . Educational Measurement: Issues and Practice , 27 ( 4 ) : 15 – 29 .
   Google Scholar
• Plake , B. S. and Huff , K. Evidence-centered assessment design as a foundation for achievement level descriptor development and for standard setting . Paper presented at the National Council of Measurement in Education . April . San Diego, CA
   Google Scholar
• Puhan , G. 2009 . Detecting and correcting scale drift in test equating: An illustration from a large scale testing program . Applied Measurement in Education , 22 ( 1 ) April : 79 – 103 .
   Google Scholar
• Rawls , A. , Yu , L. and Liu , Y. An overview of online assessments in statewide tests . Paper presented at the National Council of measurement in Education . San Diego, CA.
   Google Scholar
• Ryan , K. E. and Shepard , L. A. 2008 . The future of test-based educational accountability , New York : Routledge .
   Google Scholar
• Scalise, K., & Gifford, B. (2006). Computer-based assessment in e-learning: A framework for constructing “intermediate constraint” questions and tasks for technology platforms. Journal of Technology, Learning, and Assessment, 4(6), http://www.jtla.org (http://www.jtla.org) (Accessed: 1 March 2010 ).
   Google Scholar
• Sheingold , K. , Heller , J. I. and Paulukonis , S. T. 1994 . Actively seeking evidence: Teacher change through assessment development (Center for Performance Assessment Report No. 2 MS # 94–04) , Princeton, NJ : Educational Testing Service .
   Google Scholar
• Shepard , L. 2000 . The role of assessment in a learning culture . Educational Researcher , 29 ( 7 ) : 4 – 14 .
   Google Scholar
• Shepard , L. A. 2006 . “ Classroom assessment ” . In Educational Measurement , 4th , Edited by: Brennan , R. L. 623 – 646 . Westport, CT : Praeger .
   Google Scholar
• Shepard , L. A. 2008 . “ A brief history of accountability testing ” . In The future of test-based educational accountability , Edited by: Ryan , K. E. and Shepard , L. A. 1965 – 2007 . 25 – 46 . New York : Routledge .
   Google Scholar
• Stecher , B. M. and Hanser , L. M. 1992 . Local accountability in vocational education: A theoretical model and its limitations in practice , Palo Alto, CA : Rand Corporation .
   Google Scholar
• Tate , R. L. and King , F. J. 1994 . Factors which influence precision of school-level IRT ability estimates . Journal of Educational Measurement , 31 ( 1 ) : 1 – 15 .
   Google Scholar
• United States Congress . 2001 . No Child Left Behind Act of 2001: Conference report to accompany H.R. 1 (Report 107–334) , Washington, DC : Government Printing Office .
   Google Scholar
• V-model. (2004, July 22). Wikipedia, the free encyclopedia. http://en.wikipedia.org/wiki/Sudoku (http://en.wikipedia.org/wiki/Sudoku) (Accessed: 6 April 2009 ).
   Google Scholar
• Wainer , H. , Hambleton , R. K. and Meara , K. 1999 . Alternative displays for communicating NAEP results: A redesign and validity study . Journal of Educational Measurement , 36 ( 4 ) : 301 – 335 .
   Web of Science ®Google Scholar
• Wang , S. and Jiao , H. 2009 . Construct equivalence across grades in a vertical scale for a K–12 large-scale reading assessment . Educational and Psychological Measurement , 69 ( 5 ) : 760 – 777 .
   Google Scholar
• Weiss , D. J. , ed. 1983 . New horizons in testing: Latent trait test theory and computerized adaptive testing , New York : Academic Press .
   Google Scholar
• Wholey , J. S. 1979 . Evaluation-promise and performance , Washington : Urban Institute .
   Google Scholar
• Williamson , D. M. , Mislevy , R. J. and Bejar , I. I. 2006 . Automated scoring of complex tasks in computer-based testing , Mahwah, NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Wilson , M. 2009 . Measuring progressions: Assessment structures underlying a learning progression . Journal of Research in Science Teaching , 46 ( 6 ) : 716 – 730 .
   Web of Science ®Google Scholar
• Wilson , M. , ed. 2004 . Towards coherence between classroom assessment and accountability , Chicago : Chicago University Press .
   Google Scholar
• Yen , W. M. 2007 . “ Vertical scaling and No Child Left Behind ” . In Linking and aligning scores and scales , Edited by: Dorans , N. J. , Pommerich , M. and Holland , P. W. 273 – 284 . New York : Springer-Verlag .
   Google Scholar