Advanced search
Publication Cover
International Multilingual Research Journal Volume 13, 2019 - Issue 4
Submit an article Journal homepage
238
Views
4
CrossRef citations to date
0
Altmetric
Articles

Design and use of pop-up illustration glossaries as accessibility resources for second language learners in computer-administered tests in a large-scale assessment system

Guillermo Solano-FloresGraduate School of Education, Stanford University, CACorrespondence[email protected]
,
Magda ChiaUnderstanding Language, Stanford University, CA
&
Rachel KachchafSupports for Students and Content, Smarter Balanced Assessment Consortium
Pages 277-293 | Published online: 17 May 2019

References

  • Abedi, J. (2006). Language issues in item development. In T. Haladyna & S. Downing (Eds.), Handbook of test development (pp. 337–398). Mahwah, NJ: Lawrence Earlbaum Associate Publishers.
  • Abedi, J. (2008). Classification system for English language learners: Issues and recommendations. Educational Measurement: Issues and Practice, 27(3), 17–31. doi:10.1111/j.1745-3992.2008.00125.x
  • Abedi, J. (2009). Computer testing as a form of accommodation for English language learners. Educational Assessment, 14, 195–211. doi:10.1080/10627190903448851
  • Abedi, J. (2014). The use of computer technology in designing appropriate test accommodations for English language learners. Applied Measurement in Education, 27, 261–272. doi:10.1080/08957347.2014.944310
  • Abedi, J., & Ewers, N. (2013). Accommodations for English Language Learners and Students with Disabilities: A research-based decision algorithm. Smarter Balanced Assessment Consortium. Retrieved from https://portal.smarterbalanced.org/library/en/accommodations-for-english-language-learners-and-students-with-disabilities-a-research-based-decision-algorithm.pdf
  • Abedi, J., Hofstetter, C., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1–28. doi:10.3102/00346543074001001
  • American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for educational and psychological testing. Washington, DC: Joint Committee on Standards for Educational and Psychological Testing (U.S.).
  • Ayres, P., & Sweller, J. (2005). The split-attention principle in multimedia learning. In The Cambridge handbook of multimedia learning (pp. 135–146). New York, NY: Cambridge University Press.
  • Batalova, J., & McHugh, M. (2010). Top languages spoken by English language learners nationally and by state. Washington, D.C.: Migration Policy Institute. Retrieved from http://www.migrationpolicy.org/sites/default/files/publications/FactSheet%20ELL3%20-%20FINAL.pdf
  • Benson, P. J. (1997). Problems in picturing text: A study of visual/verbal problem solving. Technical Communication Quarterly, 6(2), 141–160. doi:10.1207/s15427625tcq0602_2
  • Bétrancourt, M., & Bisseret, A. A. (1998). Integrating textual and pictorial information via pop-up windows: An experimental study. Behaviour & Information Technology, 17(5), 263–273. doi:10.1080/014492998119337
  • Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer Verlag.
  • Cawthon, S., Leppo, R., Carr, T., & Kopriva, R. (2013). Toward accessible assessments: The promises and Limitations of test item adaptations for students with disabilities and English language learners. Educational Assessment, 18, 73–98. doi:10.1080/10627197.2013.789294
  • Chia, M., & Kachchaf, R. (2018). Designing, developing, and implementing an accessible computer-based national assessment system. In S. Elliott, R. J. Kettler, P. Beddow, & A. Kurtz (Eds.), Handbook of Accessible Instruction and Testing Practices: Issues, Innovations, and Applications (pp. 75–91). New York, NY: Springer.
  • Chia, M., Kachchaf, R., Solano-Flores, & Shyyan, (2018). Designing and evaluating illustrations for a national next generation mathematics assessment: Results from a pilot study. Paper presented at the invited symposium, Boundary-Pushing Innovations in the Assessment of ELs in K-12 Contexts. Annual Meeting of the National Council of Measurement in Education.
  • Commission, I. T. (2017). Guidelines for the large-scale assessment of linguistically diverse populations. Retrieved, March 7, 2017, from https://www.researchgate.net/publication/315761630_ITC_Guidelines_for_the_Large-Scale_Assessment_of_Linguistically_Diverse_Populations
  • Cronbach, L. J., Gleser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements. Wiley: New York, NY.
  • Ercikan, K., & Allalouf, A. (2016). Assessment design for accuracy of scores, meaningfulness of interpretations, and fairness of decision-making in high stakes educational testing. In F. T. L. Leong, D. Bartram, F. M. Cheung, K. F. Geisinger, & D. Iliescu (Eds.), The ITC International Handbook of Testing and Assessment (pp. 449–460). New York, NY: Oxford University Press.
  • Ercikan, K., & Pellegrino, J. W. (2017). Validation of score meaning for the next generation of assessments: The use of response processes. New York, NY: Routledge.
  • Ercikan, K., Roth, W.-M., Simon, M., Sandilands, D., & Lyons-Thomas, J. (2014). Inconsistencies in DIF detection for sub-groups in heterogeneous language groups. Applied Measurement in Education, 27, 275–285. doi:10.1080/08957347.2014.944306
  • Erhel, S., & Jamet, E. (2006). Using pop-up windows to improve multimedia learning. Journal of Computer Assisted Learning 22, 22, 137–147. doi:10.1111/j.1365-2729.2006.00165.x
  • Faulkner-Bond, M., & Sireci, S. G. (2015). Validity issues in assessing linguistic minorities. International Journal of Testing, 15(2), 114–135. doi:10.1080/15305058.2014.974763
  • Fichtner, D., Peitzman, F., & Sasser, L. (1994). What‘s fair? Assessing subject matter knowledge of LEP students in sheltered classrooms. In F. Peitzman & G. Gadda (Eds.), With different eyes: Insights into teaching language minority students across the disciplines (pp. 143–159). Los Angeles, CA: UCLA Publishing Center for Academic Interinstitutional Programs.
  • García, O., Kleifgen, A., & Falchi, L. (2008). From English language learners to emergent bilinguals. Equity Matters: Research Review No. 1. New York, NY: Teachers College, Columbia University.
  • Grosjean, F. (2001). The bilingual’s language modes. In J. Nicol (Ed.), One Mind, Two Languages: Bilingual Language Processing (pp. 1–22). Oxford, UK: Blackwell Publishers Ltd.
  • Guzman-Orth, D., & Wolf, M. (2017). Illustration glossaries for English learners: Findings from cognitive labs. Paper presented at the Annual Conference of the American Educational Research Association. San Antonio, TX.
  • International Test Commission (2005). International guidelines on computer-based and internet delivered testing. Retrieved November 7, 2005, from http://www.intestcom.org
  • Kachchaf, R. (2011). Exploring problem solving strategies on multiple-choice science items: comparing native Spanish-speaking English language learners and mainstream monolinguals. Unpublished doctoral dissertation. University of Colorado Boulder.
  • Kachchaf, R., & Chia, M. (2017). Illustratability of mathematics in a large-scale mathematics assessment. Paper presented at the symposium, Designing illustrations glossaries for special populations in a national large-scale mathematics assessment, organized by Rachel Kachchaf. Annual meeting of the American Educational Research Association. San Antonio, TX
  • Kachchaf, R., & Solano-Flores, G. (2012). Rater language background as a source of measurement error in the testing of English language learners. Applied Measurement in Education, 25, 167–172. doi:10.1080/08957347.2012.660366
  • King, T., & Laitusis, C. (2017). Examination of illustration glossaries for deaf and hard-of-hearing students. Paper presented at the Annual Conference of the American Educational Research Association. San Antonio, TX..
  • Kopriva, R. J. (Ed.). (2008). Improving testing for English language learners. New York, NY: Routledge.
  • Kopriva, R. J., Emick, J. E., Hipolito-Delgado, C. P., & Cameron, C. A. (2007). Do proper accommodation assignments make a difference? Examining the impact of improved decision making on scores for English language learners. Educational Measurement: Issues and Practice, 26(3), 11–20. doi:10.1111/j.1745-3992.2007.00097.x
  • Kress, G., & van Leeuwen, T. (2001). Multimodal Discourse: The Modes and Media of Contemporary Communication. New York, NY: Oxford University Press.
  • Linquanti, R., & Cook, H. G. (2013). Toward a “common definition of English language learner.”. Washington, DC: Council of Chief State School Officers, February 1.
  • MacSwan, J., & Rolstad, K. (2003). Linguistic diversity, schooling, and social class: Rethinking our conception of language proficiency in language minority education. In C. B. Paulston & G. R. Tucker (Eds.), Sociolinguistics: The essential readings (pp. 329–340). Malden, MA: Blackwell Publishing, Ltd.
  • Martiniello, M. (2009). Linguistic complexity, schematic representations, and differential item functioning for English language learners in math tests. Educational Assessment, 14, 160–179. doi:10.1080/10627190903422906
  • Mayer, R., & Moreno, R. (2003). Nine ways to reduce cognitive complexity in multimedia learning. Educational Psychology, 38(1), 43–52. doi:10.1207/S15326985EP3801_6
  • Mayer, R. E. (2005). Cognitive theory of multimedia learning. Introduction to multimedia learning. In R. E. Mayer (Ed.), The Cambridge handbook of multimedia learning (pp. 1–16). Cambridge, NY: Cambridge University Press.
  • National Academies of Sciences, Engineering, and Medicine. (2018). English learners in STEM subjects: Transforming classrooms, schools, and lives. Washington, DC: The National Academies Press. doi:10.17226/25182
  • National Center for Education Statistics (NCES) (2017). The Condition of education 2017 (2017-144). Available at: https://nces.ed.gov/programs/coe/indicator_cgf.asp
  • Noble, T., Rosebery, A., Suarez, C., Warren, B., & O’Connor, C. (2014). Science assessments and English language learners: Validity evidence based on response processes. Applied Measurement in Education, 27(4), 248–260. doi:10.1080/08957347.2014.944309
  • Norman, D. (2013). The design of everyday things. New York, NY. Basic Books, Perseus Books Group.
  • O’Malley, J. M., & Pierce, L. V. (1996). Authentic assessment for English language learners: Practical approaches for teachers. Reading, MA: Addison-Wesley Publishing Company.
  • Oakland, T. (2016). Testing and assessment of immigrants and second-language learners. In F. T. L. Leong, D. Bartram, F. M. Cheung, K. F. Geisinger, & D. Iliescu (Eds.), The ITC International Handbook of Testing and Assessment (pp. 318–332). New York, NY: Oxford University Press.
  • Oliveri, M. E., Ercikan, K., & Zumbo, B. D. (2014). Simulating the effect of linguistic heterogeneity on the accuracy of detection of differentially functioning items between linguistic groups. Applied Measurement in Education, 27(4), 286–300. doi:10.1080/08957347.2014.944305
  • Penock-Roman, M., & Rivera, C. (2011). Mean effects of test accommodations for ELLs and non-ELLs: A meta-analysis of experimental studies. Educational Measurement: Issues and Practice, 30(3), 10–28. doi:10.1111/j.1745-3992.2011.00207.x
  • Rivera, C., & Collum, E. (Eds.). (2006). State assessment policy and practice for English language learners: A national perspective. Mahwah, NJ: Lawrence Erlbaum Associates.
  • Rivera, C., Willner, L. S., & Acosta, B. (2008). Guide for refining state assessment policies for accommodating English language learners. Washington, D.C: Center for Equity and Excellence in Education. George Washington University.
  • Ruiz-Primo, M. A., & Li, M. (2015). The relationship between item context characteristics and student performance: The case of the 2006 and 2009 PISA science ítems. Teachers College Record, 117(1), 010306.
  • Russel, M. (2018). Recent advances in the accessibility of digitally delivered educational assessments. In S. Elliott, R. J. Kettler, P. Beddow, & A. Kurtz (Eds.), Handbook of Accessible Instruction and Testing Practices: Issues, Innovations, and Applications (pp. 247–262). New York:, NY: Springer.
  • Saffer, D. (2014). Microinteractions: Designing with details. Sebastopol, CA: O’Reilly.
  • Sale, R. (2006). International guidelines on computer-based and internet-delivered testing: A practitioner’s perspective. International Journal of Testing, 6(2), 181–188. doi:10.1207/s15327574ijt0602_6
  • Schwabe, F., von Davier, A. A., & Chalhoub-Deville, M. (2016). Language and culture in testing. In F. T. L. Leong, D. Bartram, F. M. Cheung, K. F. Geisinger, & D. Iliescu (Eds.), The ITC International Handbook of Testing and Assessment (pp. 300–317). New York, NY: Oxford University Press.
  • Sharrocks-Taylor, D., & Hargreaves, M. (1999). Making it clear: A review of language issues in testing with special reference to the National Curriculum Mathematics Tests at Key Stage 2. Educational Research, 41(2), 123–136. doi:10.1080/0013188990410201
  • Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
  • Sireci, S. G., Li, S., & Scarpati, S. (2003). The effects of test accommodation on test performance: A review of the literature (Research Report 485). Amherst, MA: Center for Educational Assessment.
  • Smarter Balanced Assessment Consortium (2014). Accessibility and Accommodations Framework. Retrieved from https://www.smarterbalanced.org/wp-content/uploads/2015/09/Accessibility-and-Accommodations-Framework.pdf
  • Smarter Balanced Assessment Consortium (2015). Practice and training tests. http://www.smarterbalanced.org/assessments/practice-and-training-tests/
  • Smarter Balanced Assessment Consortium (2017). Sample of illustrated items. https://tds.smarterbalanced.org/student/Pages/TestShell.aspx. Retrieved January 12, 2017.
  • Solano-Flores, G., & Trumbull, E. (2008). In what language should English language learners be tested? In R. In J. Kopriva (Ed.), Improving testing for English language learners (pp. 169–200). New York, NY: Routledge.
  • Solano-Flores, G. (2010a). The use of pictorial supports as an accommodation for increasing access to test items for students with limited proficiency in the language of testing. Paper presented at the 7th Conference of the International Test Commission. Hong Kong, China, July 19-21, 2010.
  • Solano-Flores, G. (2010b). Vignette illustrations as a form of testing accommodation for English language learners: A design methodology for use in large-scale science assessment. Paper presented at the Annual Conference of the National Council of Measurement in Education, Denver, Colorado.
  • Solano-Flores, G. (2011). Assessing the cultural validity of assessment practices: An introduction. Basterra. In M. R. Trumbull, E. & G. Solano-Flores (Eds.), Cultural validity in assessment: Addressing linguistic and cultural diversity (pp. 3–21). New York, NY: Routledge.
  • Solano-Flores, G. (2012). Translation accommodations framework for testing English language learners in mathematics. Developed for the Smarter Balanced Assessment Consortium (SBAC). http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/09/Translation-Accommodations-Framework-for-Testing-ELL-Math.pdf.
  • Solano-Flores, G. (2014). Probabilistic approaches to examining linguistic features of test items and their effect on the performance of English language learners. Applied Measurement in Education., 27(4), 236–247. doi:10.1080/08957347.2014.944308
  • Solano-Flores, G. (2016). Assessing English language learners: Theory and practice. New York, NY: Routledge.
  • Solano-Flores, G., & Chia, M. (2017a). Conceptual foundations of test illustration. Paper presented at the symposium, Designing illustrations glossaries for special populations in a national large-scale mathematics assessment, organized by Rachel Kachchaf. Annual meeting of the American Educational Research Association: San Antonio, TX
  • Solano-Flores, G., & Chía, M. (2017b). Multiple language versions of tests. In K. Ercikan & J. Pellegrino (Eds.), Validation of score meaning in the next generation of assessments: The use of response processes (pp. 127-137). New York, NY: Routledge.
  • Solano-Flores, G., & Hakuta, K. (2017). Assessing Students in Their Home Language. Retrieved from Stanford University, Understanding Language website: https://stanford.app.box.com/s/uvwlgjbmeeuokts6c2wnibucms4up9c2
  • Solano-Flores, G., & Li, M. (2008). Examining the dependability of academic achievement measures for English-Language Learners. Assessment for Effective Intervention, 33(3), 135–144. doi:10.1177/1534508407313238
  • Solano-Flores, G., & Li, M. (2009). Language variation and score variation in the testing of English language learners, native Spanish speakers. Educational Assessment, 14, 1–15).
  • Solano-Flores, G., & Li, M. (2013). Generalizability theory and the fair and valid assessment of linguistic minorities. Educational Research and Evaluation, 19(2–3), 245–263. doi:10.1080/13803611.2013.767632
  • Solano-Flores, G., Shade, C., & Chrzanowski, A. (2014). Item accessibility and language variation conceptual framework. Submitted to the Smarter Balanced Assessment Consortium. October 10. https://portal.smarterbalanced.org/library/en/item-accessibility-and-language-variation-conceptual-framework.pdf
  • Solano-Flores, G., & Wang, C. (2015). Complexity of illustrations in PISA-2009 science items and its relationship to the performance of students from Shanghai-China, the United States, and Mexico. Teachers College Record, 117(1), 1–18. ID Number: 17725: http://www.tcrecord.org
  • Solano-Flores, G., Wang, C., Kachchaf, R., Soltero-Gonzalez, L., & Nguyen-Le, K. (2014). Developing testing accommodations for English language learners: Illustrations as visual supports for item accessibility. Educational Assessment, 19, 267–283. doi:10.1080/10627197.2014.964116
  • Solano-Flores, G., Wang, C., & Shade, C. (2016). International semiotics: Item difficulty and the complexity of science item illustrations in the PISA-2009 international test comparison. International Journal of Testing, 16(3), 205–219. doi:10.1080/15305058.2015.1099534
  • U.S. Department of Education. (2000). A guide to the individualized education plan. Retrieved from https://www2.ed.gov/parents/needs/speced/iepguide/index.html
  • Valdés, G., & Figueroa, R. A. (1994). Bilingualism and testing: A special case of bias. Norwood, NJ: Ablex.
  • Vertovec, S. (2007). Super-diversity and its implications. Ethnic and Racial Studies, 30(6), 1024–1054. doi:10.1080/01419870701599465
  • Wang, C. (2012). The use of illustrations in large-scale science assessment: A comparative study. Doctoral dissertation. University of Colorado Boulder.
  • Wang, C., Chia, M., Kachchaf, R., & Solano-Flores, G. (2012). Item illustration complexity and the performance of English language learners in a science test. Paper presented at the Annual Conference of the American Educational Research Association, Vancouver, Canada, April 13–17, 2012.
  • Wolf, M. K., Kao, J. C., Rivera, N. M., & Chang, S. M. (2012). Accommodation practices for English language learners in states‘ mathematics assessments. In Teachers College Record, 114(3), 1–26.

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Order Reprints Request Corporate Permissions

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

Request Academic Permissions

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.