Probabilistic Approaches to Examining Linguistic Features of Test Items and Their Effect on the Performance of English Language Learners

REFERENCES

• Abedi, J. (2006). Language issues in item development. In. S. M. Downing & T. M. Haladyna ( Eds.), Handbook of test development ( pp. 377–398). Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Abedi, J., Hofstetter, C. H., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1–28.
   Web of Science ®Google Scholar
• Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14(3), 219–234.
   Web of Science ®Google Scholar
• Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2001). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16–26.
   Google Scholar
• Allalouf, A. (2003). Revising translated differential item functioning items as a tool for improving cross-lingual assessment. Applied Measurement in Education, 16(1), 55–73.
   Web of Science ®Google Scholar
• Allalouf, A., Hambleton, R., & Sireci, S. (1999). Identifying the causes of DIF on verbal items. Journal of Educational Measurement, 36, 18–198.
   Web of Science ®Google Scholar
• American Educational Research Association, American Psychological Association, & National Council of Measurement in Education. (1999). Standards for educational and psychological testing. Washington, DC: AERA.
   Google Scholar
• Bachman, L. F., Lynch, B. K., & Mason, M. (1995). Investigating variability in tasks and rater judgments in a performance test of foreign language speaking. Language Testing, 12(2), 238–257.
   Google Scholar
• Bailey, A. L., & Butler, F. A. (2003). An evidentiary framework for operationalizing academic language for broad application to K–12 education: A design document. (CSE Tech. Rep. No. 611). Los Angeles, CA: University of California, National Center for Research on Evaluation, Standards, and Student Testing.
   Google Scholar
• Basterra, M. R. (2012). Cognition, culture, language, and assessment: How to select culturally valid assessments in the classroom. In M. R. Basterra, E. Trumbull, & G. Solano-Flores ( Eds.). Cultural validity in assessment: Addressing linguistic and cultural diversity ( pp. 72–95). New York, NY: Routledge.
   Google Scholar
• Bialystok, E. (2001). Bilingualism in development: Language, literacy, and cognition. Cambridge, UK: Cambridge University Press.
   Google Scholar
• Bod, R., Hay, J., & Jannedy, S. ( Eds.) (2003). Probabilistic linguistics. Cambridge, MA: MIT.
   Google Scholar
• Brennan, R. L. (2001). Generalizability theory. New York, NY: Springer-Verlag.
   Google Scholar
• Cronbach, L. J., Glaser, G. C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements. New York, NY: Wiley.
   Google Scholar
• De Corte, E., & Verschaffel, L. (1991). Some factors influencing the solution of addition and subtraction word problems. In.K. Durkin & B. Shire ( Eds.), Language in mathematical education: Research and practice ( pp. 17–30), Buckingham, UK: Open University Press.
   Google Scholar
• Ercikan, K. (2002). Disentangling sources of differential item functioning in multi-language assessments. International Journal of Testing, 2, 199–215.
   Google Scholar
• Ercikan, K., Arim, R. G., Law, D. M., Lacroix, S., Gagnon, F., & Domene, J. F. (2010). Application of think-aloud protocols in examining sources of differential item functioning. Educational Measurement: Issues and Practice, 29, 24–35.
   Google Scholar
• Ercikan, K., Roth, W.-M., Simon, M., Sandilands, D., & Lyons-Thomas, J. (2014). Inconsistencies in DIF Detection for Sub-Groups in Heterogeneous Language Groups. Applied Measurement in Education, 27, 273–285.
   Google Scholar
• General Accounting Office. (2006). No Child Left Behind Act: Assistance from Education could help states better measure progress of students with limited English proficiency (GAO Report to Congressional Requesters, Report No. GAO-355-06-815). Retrieved from http://www.gao.gov/assets/260/250866.pdf
   Google Scholar
• Gibson, J. J. (1977). The theory of affordances. In R. E. Shaw & J. Bransford ( Eds.), Perceiving, acting, and knowing: Toward an ecological psychology (pp. 67–82). Hillsdale, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Gierl, M. J. (2000). Construct equivalence on translated achievement tests. Canadian Journal of Education/Revue canadienne de l’education, 25(4), 280–296.
   Google Scholar
• Gierl, M. J., & Khaliq, S. N. (2001). Identifying sources of differential item and bundle functioning on translated achievement tests: A confirmatory analysis. Journal of Educational Measurement, 38, 16–187.
   Web of Science ®Google Scholar
• Gries, S. T. (2013). Sources of variability relevant to the cognitive sociolinguist, and corpus- as well as psycholinguistic methods and notions to handle them. Journal of Pragmatics, 52, 5–16.
   Web of Science ®Google Scholar
• Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language and meaning. London, UK: Hodder Arnold.
   Google Scholar
• Halliday, M. A. K. (2005). Computational and quantitative studies. New York, NY: Continuum.
   Google Scholar
• Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger ( Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3–38). Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Hambleton, R. K., & Jones, R. W. (1992). Comparison of empirical and judgmental methods for detecting differential item functioning. Paper presented at the annual meeting of the National Council on Measurement in Education, San Francisco, CA.
   Google Scholar
• Hudicourt-Barnes, J., Noble, T., O’Connor, M. C., Rosebery, A. S., Suarez, C., Warren, B., & Wright, C. G. (2008). Making sense of children’s performance on achievement tests: The case of the 5th grade science MCAS. Symposium paper presented at the annual meeting of the American Educational Research Association. New York, NY.
   Google Scholar
• Kachchaf, R. R. (2011). Exploring problem solving strategies on multiple-choice science items comparing native Spanish-speaking English language learners and mainstream monolinguals. Doctoral dissertation. University of Colorado at Boulder, Department of Education.
   Google Scholar
• Kachchaf, R. R., & Solano-Flores, G. (2012). Rater language background as a source of measurement error in the testing of English language learners. Applied Measurement in Education, 25, 167–172.
   Web of Science ®Google Scholar
• Kopriva, R. J. (2000). Ensuring accuracy in testing for English language learners: A practical guide for assessment development. Washington, DC: Council of Chief State School Officers.
   Google Scholar
• Kopriva, R. J. (2008). Improving testing for English language learners: A comprehensive approach to designing, building, implementing and interpreting better academic assessments. New York, NY: Routledge.
   Google Scholar
• Kopriva, R. J., Emick, J. E., Hipólito-Delgado, C. P., & Cameron, C. A. (2007). Do proper accommodation assignments make a difference? Examining the impact of improved decision making on scores for English language learners. Educational Measurement: Issues and Practice, 26(3), 11–20.
   Google Scholar
• Kopriva, R., J., & Koran, J. (2008). Proper assignment of accommodations to individual students. In R. J. Kopriva ( Ed.), Improving testing for English language learners: A comprehensive approach to designing, building, implementing and interpreting better academic assessments ( pp. 217–254). New York, NY: Routledge.
   Google Scholar
• Martiniello, M. (2009). Linguistic complexity, schematic representations, and differential item functioning for English language learners in math tests. Educational Assessment, 14(3), 160–179.
   Google Scholar
• Menezes, V. (2011). Affordances for language learning beyond the classroom. In P. Benson & H. Renders ( Eds.), Beyond the language classroom ( pp. 59–71). Basingstoke, UK: Palgrave Macmillan.
   Google Scholar
• Messick, S. (1989). Validity. In R. L. Linn ( Ed.), Educational measurement (3rd ed.) (pp. 147–200). New York, NY: Macmillan Publishing.
   Google Scholar
• Mislevy, R. J., Wilson, M. R., Ercikan, K., & Chudowsky, N. (2003). Psychometric principles in student assessment. In T. Kellaghan & D. Stufflebeam ( Eds.), International handbook of educational evaluation ( pp. 489–531). Dordrecht, the Netherlands: Kluwer Academic Press. ( Eds.).
   Google Scholar
• Nguyen-Le, K. (2010). Personal and formal backgrounds as factors which influence linguistic and cultural competency in the teaching of mathematics. Doctoral dissertation, Educational Equity and Cultural Diversity Program, University of Colorado at Boulder, 2010.
   Google Scholar
• Noble, T., Suarez, C., Rosebery, A., O’Connor, M. C., Warren, B., & Hudicourt-Barnes, J. (2012). “I never thought of it as freezing”: How students answer questions on large-scale science tests and what they know about science. Journal of Research in Science Teaching, 49(6), 778–803.
   Web of Science ®Google Scholar
• Norman, D. A. (1990). The design of everyday things. New York, NY: Doubleday.
   Google Scholar
• O’Connor, M. C. (1998). Language socialization in the mathematics classroom: Discourse practices and mathematical thinking. In M. Lampert & M. Blunk ( Eds.), Talking mathematics: Studies of teaching and learning in school ( pp. 17–55). New York, NY: Cambridge University Press.
   Google Scholar
• Penock-Roman, M., & Rivera, C. (2011). Mean effects of test accommodations for ELLs and non-ELLs: A meta-analysis of experimental studies. Educational Measurement: Issues and Practice, 30(3), 10–28.
   Google Scholar
• Prosser, R. R., & Solano-Flores, G. (2010). Including English language learners in the process of test development: A study on instrument linguistic adaptation for cognitive validity. Paper presented at the annual conference of the National Council of Measurement in Education, Denver, CO.
   Google Scholar
• Rivera, C., & Stansfield, C. W. (2004). The effect of linguistic simplification of science test items on score comparability. Educational Assessment, 9(3&4), 79–105.
   Google Scholar
• Sato, E., Rabinowitz, S., Gallagher, C., & Huang, C.-W. (2010). Accommodations for English language learner students: The effect of linguistic modification of math test item sets (NCEE 2009-4079). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
   Google Scholar
• Scarcella, R. C. (2003). Academic English: A conceptual framework (Rep. No. 2003-1). Santa Barbara, CA: University of California, Linguistic Minority Research Institute.
   Google Scholar
• Shaftel, J., Belton-Kocher, E., Glasnapp, D., & Poggio, G. (2006). The impact of language characteristics in mathematics test items on the performance of English language learners and students with disabilities. Educational Assessment, 11(2), 105–126.
   Google Scholar
• Shavelson, R. J., & Webb, N. M. (1991). Generalizability theory: A primer. Newbury Park, CA: Sage.
   Google Scholar
• Sireci, S. G., Li, S., & Scarpati, S. (2003). The effects of test accommodation on test performance: A review of the literature. ( Center for Educational Assessment, Research Report No. 485). Amherst, MA: University of Massachusetts at Amherst.
   Google Scholar
• Solano-Flores, G. (2006). Language, dialect, and register: Sociolinguistics and the estimation of measurement error in the testing of English-language learners. Teachers College Record. 108(11), 2354–2379.
   Web of Science ®Google Scholar
• Solano-Flores, G. (2008). Who is given tests in what language by whom, when, and where? The need for probabilistic views of language in the testing of English language learners. Educational Researcher, 37(4), 189–199.
   Web of Science ®Google Scholar
• Solano-Flores, G. (2009). The testing of English language learners as a stochastic process: Population misspecification, measurement error, and overgeneralization. In K. Ercikan & W. M. Roth ( Eds.), Generalizing from educational research ( pp. 33–48). New York, NY: Routledge.
   Google Scholar
• Solano-Flores, G. (2010). Function and form in research on language and mathematics education. In Moschkovich, J. ( Ed.), Language and mathematics in education: Multiple perspectives and directions for research ( pp. 113–149). Charlotte, NC: Information Age Publishing.
   Google Scholar
• Solano-Flores, G. (2012). Translation accommodations framework for testing English language learners in mathematics. (Developed for Smarter Balanced Assessment Consortium [SBAC]). Retrieved from http://www.smarterbalanced.org/wordpress/wp-content/uploads/2012/09/Translation-Accommodations-Framework-for-Testing-ELL-Math.pdf
   Google Scholar
• Solano-Flores, G., Backhoff, E., & Contreras-Niño, L. A. (2009). Theory of test translation error. International Journal of Testing, 9, 78–91.
   Google Scholar
• Solano-Flores, G., Contreras-Niño, L. A., & Backhoff, E. (2012). The measurement of translation error in PISA-2006 items: An application of the theory of test translation error. In M. Prenzel, M. Kobarg, K. Schöps, & S. Rönnebeck ( Eds.), Research in the context of the programme for international student assessment (pp. 71–85). Germany: Springer Verlag.
   Google Scholar
• Solano-Flores, G., & Gustafson, M. (2012). Assessment of English language learners: A critical, probabilistic, systemic view. In M. Simon, K. Ercikan, & M. Rousseau ( Eds.), Improving large scale assessment in education: Theory, issues, and practice. ( pp. 87–109). New York, NY: Routledge.
   Google Scholar
• Solano-Flores, G., & Li, M. (2006). The use of generalizability (G) theory in the testing of linguistic minorities. Educational Measurement: Issues and Practice 25(1), 13–22.
   Google Scholar
• Solano-Flores, G., & Li, M. (2008). Examining the dependability of academic achievement measures for English-language learners. Assessment for Effective Intervention, 33(3), 135–144.
   Google Scholar
• Solano-Flores, G., & Li, M. (2009). Language variation and score variation in the testing of English language learners, native Spanish speakers. Educational Assessment, 14, 1–15.
   Google Scholar
• Solano-Flores, G., & Li, M. (2013). Generalizability theory and the fair and valid assessment of linguistic minorities. Educational Research and Evaluation, 19(2–3), 245–263.
   Google Scholar
• Solano-Flores, G., Li, M., Speroni, C., Rodriguez, J., Basterra, M., & Dovholuk, G. (2007). Comparing the properties of teacher-adapted and linguistically-simplified test items for English language learners. Paper presented at the annual meeting of the American Educational Research Association, Chicago, IL.
   Google Scholar
• Solano-Flores, G., & Trumbull, E. (2003). Examining language in context: The need for new research and practice paradigms in the testing of English-language learners. Educational Researcher, 32(2), 3–13.
   Google Scholar
• Solano-Flores, G., & Trumbull, E. (2008). In what language should English language learners be tested? In R. J. Kopriva ( Ed.), Improving testing for English language learners ( pp. 169–200). New York, NY: Routledge.
   Google Scholar
• Stansfield, C. W. (2003). Test translation and adaptation in public education in the USA. Language Testing, 20(2), 189–207.
   Google Scholar
• Stansfield, C. W., & Bowles, M. (2006). Study 2: Test translation and state assessment policies for English language learners. In C. Rivera & E. Collum ( Eds.), State assessment policy and practice for English language learners: A national perspective. ( pp. 1–173). Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• Steyvers, M., & Griffiths, T. (2007). Probabilistic topic models. In T. K. Landauer, D. S. McNamara, S. Dennis, & W. Kintsch ( Eds.), Handbook of latent semantic analysis ( pp. 427–448). Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar
• van Lier, L. (2004). The ecology and semiotics of language learning: A sociocultural perspective. Dordrecht, Netherlands: Kluwer Academic Publishers.
   Google Scholar
• Wellington, J., & Osborne, J. (2001). Language and literacy in science education. Buckingham, UK: Open University Press.
   Google Scholar
• Winter, P. C., Kopriva, R. J., Chen, C.-S., & Emick, J. E. (2006). Exploring individual and item factors that affect assessment validity for diverse learners: Results from a large-scale cognitive lab. Learning and Individual Differences, 16(4), 267–276.
   Web of Science ®Google Scholar
• Wolf, M. K., Kao, J., Herman, J., Bachman, L. F., Bailey, A., Bachman, P. L., … Chang. S. M. (2008). Issues in assessing English language learners: English language proficiency measures and accommodation uses: Literature review (Part 1 of 3). (CRESST Report 731). Los Angeles, CA: National Center for Research on Evaluation, Standards, and Student Testing and Graduate School of Education & Information Studies. University of California.
   Google Scholar
• Wolf, M. K., & Leon, S. (2009). An investigation of the language demands in content assessments for English language learners. Educational Assessment, 14(3), 139–159.
   Google Scholar
• Wolfram, W., Adger, C. T., & Christian, D. (1999). Dialects in schools and communities. Mahwah, NJ: Lawrence Erlbaum Associates.
   Google Scholar