Comparing Panelists' Understanding of Standard Setting Across Multiple Levels of an Alternate Science Assessment

REFERENCES

• American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME) . 1999 . Standards for educational and psychological testing , Washington , DC : American Psychological Association .
   Google Scholar
• Angoff , W. H. 1971 . “ Scales, norms and equivalent scores ” . In Educational measurement: Theories and applications , 2nd , Edited by: Thorndike , R. L. 508 – 600 . Washington , DC : American Council on Education .
   Google Scholar
• Cizek , G. J. , Bunch , M. B. and Koons , H. 2004 . Setting performance standards: Contemporary methods . Educational Measurement: Issues and Practice , 23 ( 4 ) : 31 – 50 . doi: 10.1111/j.1745-3992.2004.tb00166.x
   Google Scholar
• Crawford , L. , Tindal , G. and Carpenter , D. M. 2006 . Exploring the validity of the Oregon Extended Writing Assessment . Journal of Special Education , 40 ( 1 ) : 16 – 27 .
   Web of Science ®Google Scholar
• Dawber , T. and Lewis , D. M. 2002 . The cognitive experience of bookmark standard setting participants , New Orleans , LA : Paper presented at the Annual Meeting of the American Educational Research Association .
   Google Scholar
• Decker , D. M. and Bolt , S. E. 2008 . Challenges and opportunities for promoting student achievement through large-scale assessment results: Research, reflections, and future directions . Assessment for Effective Intervention , 34 ( 1 ) : 43 – 51 . doi: 10.1177/1534508408314173
   Google Scholar
• Egan , K. L. , Ferrara , S. , Schneider , M. C. and Barton , K. E. 2009 . Writing performance level descriptors and setting performance standards for assessments of modified achievement standards: The role of innovation and importance of following conventional practice . Peabody Journal of Education , 84 ( 4 ) : 552 – 577 . doi: 10.1080/01619560903241028
   Google Scholar
• Elliott , S. N. , Compton , E. and Roach , A. T. 2007 . Building validity evidence for scores on a state-wide alternate assessment: A contrasting groups, multimethod approach . Educational Measurement: Issues and Practice , 26 ( 2 ) : 30 – 43 . doi: 10.1111/j.1745-3992.2007.00092.x
   Google Scholar
• Elliott , S. N. and Roach , R. T. 2007 . Alternate assessments of students with significant disabilities: Alternative approaches, common technical challenges . Applied Measurement in Education , 20 ( 3 ) : 301 – 333 . doi: 10.1080/08957340701431385
   Web of Science ®Google Scholar
• Ferdous , A. A. and Plake , B. S. 2005 . Understanding the factors that influence decisions of panelists in a standard-setting study . Applied Measurement in Education , 18 ( 3 ) : 257 – 267 . doi: 10.1207/s15324818ame1803_4
   Web of Science ®Google Scholar
• Geisinger , K. F. and McCormick , C. M. 2010 . Adopting cut scores: Post-standard setting panel considerations for decision makers . Educational Measurement: Issues and Practice , 29 ( 1 ) : 38 – 44 . doi: 10.1111/j.1745-3992.2009.00168.x
   Google Scholar
• Giraud , G. , Impara , J. C. and Plake , B. S. 2005 . Teachers' conceptions of the target examinee in Angoff standard setting . Applied Measurement in Education , 18 ( 3 ) : 223 – 232 . doi: 10.1207/s15324818ame1803_2
   Web of Science ®Google Scholar
• Gong, B., & Marion, S. (2006). Dealing with flexibility in assessments for students with significant cognitive disabilities (Synthesis Report 60). National Center on Educational Outcomes. Retrieved from /Synthesis60.pdf http://www.cehd.umn.edu/NCEO/onlinepubs (http://www.cehd.umn.edu/NCEO/onlinepubs)
   Google Scholar
• Haertel , E. H. 2002 . Standard setting as a participatory process: Implications for validation of standards-based accountability programs . Educational Measurement: Issues and Practice , 21 ( 1 ) : 16 – 22 .
   Google Scholar
• Hambleton , R. K. and Pitoniak , M. J. 2006 . “ Setting performance standards ” . In Educational measurement , 4th , Edited by: Brennan , R. L. 433 – 470 . Washington , DC : American Council on Education .
   Google Scholar
• Hambleton , R. K. and Plake , B. S. 1995 . Using an extended Angoff procedure to set standards on complex performance assessments . Applied Measurement in Education , 8 ( 1 ) : 41 – 55 .
   Web of Science ®Google Scholar
• Hansen , M. A. , Lyon , S. , Heh , P. and Zigmond , N. April 2010 . Development and validation of alternate science performance level descriptors , April , Denver , CO : Paper presented at the Annual Meeting of the American Educational Research Association .
   Google Scholar
• Hein , S. F. and Skaggs , G. E. 2009 . A qualitative investigation of panelists' experiences of standard setting using two variations of the bookmark method . Applied Measurement in Education , 22 ( 3 ) : 207 – 228 . doi: 10.1080/08957340902983978
   Web of Science ®Google Scholar
• Ho , A. D. , Lewis , D. M. and MacGregor Farris , J. L. 2009 . The dependence of growth-model results on proficiency cut scores . Educational Measurement: Issues and Practice , 28 ( 4 ) : 15 – 26 . doi: 10.1111/j.1745-3992.2009.00159.x
   Google Scholar
• Hurtz , G. M. and Hertz , N. R. 1999 . How many raters should be used for establishing cutoff scores with the Angoff method? A generalizability theory study . Educational and Psychological Measurement , 59 ( 6 ) : 885 – 897 . doi: 10.1177/00131649921970233
   Web of Science ®Google Scholar
• Impara , J. C. and Plake , B. S. 1997 . Standard setting: An alternative approach . Journal of Educational Measurement , 34 ( 4 ) : 355 – 368 .
   Web of Science ®Google Scholar
• Individuals with Disabilities Education Act (IDEA) . 1997 . 20 U.S.C. § 1400
   Google Scholar
• Kane , M. 1994 . Validating the performance standards associated with passing scores . Review of Educational Research , 64 ( 3 ) : 425 – 461 .
   Web of Science ®Google Scholar
• Kane , M. T. 2001 . Current concerns in validity theory . Journal of Educational Measurement , 38 ( 4 ) : 319 – 342 . doi: 10.1111/j.1745-3984.2001.tb01130.x
   Web of Science ®Google Scholar
• Lane , S. , Parke , C. S. and Stone , C. A. 1998 . A framework for evaluating consequences of assessment programs . Educational Measurement: Issues and Practice , 17 ( 2 ) : 24 – 28 . doi: 10.1111/j.1745-3992.1998.tb00830.x
   Google Scholar
• Lane , S. and Stone , C. A. 2002 . Strategies for examining consequences of assessment and accountability programs . Educational Measurement: Issues and Practice , 21 ( 1 ) : 23 – 30 . doi: 10.1111/j.1745-3992.2002.tb00082.x
   Google Scholar
• Loomis , S. C. and Bourque , M. L. 2001 . “ From tradition to innovation: Standard setting on the National Assessment of Educational Progress ” . In Setting performance standards: Concepts, methods, and perspectives , Edited by: Cizek , G. J. Mahwah , NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Malmgren , K. W. , McLaughlin , M. J. and Nolet , V. 2005 . Accounting for the performance of students with disabilities on statewide assessments . Journal of Special Education , 39 ( 2 ) : 86 – 96 .
   Web of Science ®Google Scholar
• Marion , S. F. and Pellegrino , J. W. 2006 . A validity framework for evaluating the technical quality of alternate assessments . Educational Measurement: Issues and Practices , 25 ( 4 ) : 47 – 57 . doi: 10.1111/j.1745-3992.2006.00078.x
   Google Scholar
• McGinty , D. 2005 . Illuminating the “black box” of standard setting: An exploratory qualitative study . Applied Measurement in Education , 18 ( 3 ) : 269 – 287 . doi: 10.1207/s15324818ame1803_5
   Web of Science ®Google Scholar
• McKinley , D. W. , Boulet , J. R. and Hambleton , R. K. 2005 . A work-centered approach for setting passing scores on performance-based assessments . Evaluation and the Health Professions , 28 ( 3 ) : 349 – 369 . doi: 10.1177/0163278705278282
   PubMed Web of Science ®Google Scholar
• No Child Left Behind Act of 2001 (NCLB), 20 U.S.C. § 6319. (2008)
   Google Scholar
• Olson, B., Mead, R., & Payne, D. (2002). A report of a standard setting method for alternate assessments for students with significant disabilities (Synthesis Report 47). National Center on Education Outcomes. Retrieved from /NCEO/OnlinePubs/Synthesis47.html http://education.umn.edu (http://education.umn.edu)
   Google Scholar
• Pant , H. A. , Rupp , A. A. , Tiffin-Richards , S. P. and Koller , O. 2009 . Validity issues in standard-setting studies . Studies in Educational Evaluation , 35 : 95 – 101 .
   Google Scholar
• Perie, M. (2007). A guide to understanding and developing performance level descriptors. Dover, NH: National Center for the Improvement of Educational Assessment. http://www.nciea.org/cgi-bin/pubspage.cgi?pgRequested=2&curPage=1&sortby=pub_date&itemsToDisplay=10 (http://www.nciea.org/cgi-bin/pubspage.cgi?pgRequested=2&curPage=1&sortby=pub_date&itemsToDisplay=10)
   Google Scholar
• Perie , M. 2008 . A guide to understanding and developing performance-level descriptors . Educational Measurement , 27 ( 4 ) : 15 – 29 . doi: 10.1111/j.1745-3992.2008.00135.x
   Google Scholar
• Perie , M. , Hess , K. and Gong , B. March 2008 . Writing performance level descriptors: Applying lessons learned from the general assessment to alternate assessments based on alternate and modified achievement standards , March , New York , NY : Paper presented at the annual conference of the National Council of Measurement in Education .
   Google Scholar
• Plake , B. S. 2002 . Evaluating the technical quality of educational tests used for high-stakes decisions . Measurement and Evaluation in Counseling and Development , 35 ( 3 ) : 144 – 152 .
   Web of Science ®Google Scholar
• Plake , B. S. and Impara , J. C. 2001 . Ability of panelists to estimate item performance for a target group of candidates: An issue in judgmental standard setting . Educational Assessment , 7 ( 2 ) : 87 – 97 .
   Google Scholar
• Roach , A. T. , Elliott , S. N. and Webb , N. L. 2005 . Alignment of an alternate assessment with state academic standards: Evidence for the content validity of the Wisconsin Alternate Assessment . Journal of Special Education , 38 ( 4 ) : 218 – 231 .
   Web of Science ®Google Scholar
• Sireci , S. G. , Hauger , J. B. , Wells , C. S. , Shea , C. and Zenisky , A. I. 2009 . Evaluation of the standard setting on the 2005 Grade 12 National Assessment of Educational Progress mathematics test . Applied Measurement in Education , 22 ( 4 ) : 339 – 358 . doi: 10.1080/08957340903221659
   Web of Science ®Google Scholar
• Skorupski , W. P. and Hambleton , R. K. 2005 . What are panelists thinking when they participate in standard-setting studies? . Applied Measurement in Education , 18 ( 3 ) : 233 – 256 . doi: 10.1207/s15324818ame1803_3
   Web of Science ®Google Scholar
• Thurlow , M. L. 2002 . Positive educational results for all students . Remedial and Special Education , 23 ( 4 ) : 195 – 202 .
   Web of Science ®Google Scholar
• Thurlow , M. L. and Johnson , D. R. 2000 . High stakes testing of students with disabilities . Journal of Teacher Education , 51 ( 4 ) : 305 – 314 .
   Web of Science ®Google Scholar
• Thurlow , M. L. , Lazarus , S. S. , Thompson , S. J. and Morse , A. B. 2005 . State policies on assessment participation and accommodations for students with disabilities . Journal of Special Education , 38 ( 4 ) : 218 – 231 . doi: 10.1177/00224669050380040401
   Web of Science ®Google Scholar
• Towles-Reeves , E. , Kearns , J. , Klienert , H. and Klienert , J. 2009a . An analysis of the learning characteristics of students taking alternate assessments based on alternate achievement standards . Journal of Special Education , 42 ( 4 ) : 241 – 254 . doi: 10.1177/0022466907313451
   Web of Science ®Google Scholar
• Towles-Reeves , E. , Kleinert , H. and Muhomba , M. 2009b . Alternate assessment: Have we learned anything new? . Exceptional Children , 75 ( 2 ) : 233 – 252 .
   Web of Science ®Google Scholar
• Wiener, D. (2002). Massachusetts: One state's approach to setting performance levels on the alternate assessment (Synthesis Report 48). National Center on Educational Outcomes. http://education.umn.edu/NCEO/OnlinePubs/Synthesis48.html (http://education.umn.edu/NCEO/OnlinePubs/Synthesis48.html)
   Google Scholar