The Effect of Linguistic Factors on Assessment of English Language Learners’ Mathematical Ability: A Differential Item Functioning Analysis

References

• Abedi, J. (2006). Psychometric issues in the ELL assessment and special education eligibility. Teachers College Record, 108(11), 2282–2303. doi:10.1111/j.1467-9620.2006.00782.x
   Web of Science ®Google Scholar
• Abedi, J., Bailey, A., Butler, F., Castellon-Wellington, M., Leon, S., & Mirocha, J. (2005). The validity of administering large-scale content assessments to english language learners: An investigation from three perspectives. CSE Report 663. Retrieved from https://eric.ed.gov/?id=ED492891
   Google Scholar
• Abedi, J., Courtney, M., & Leon, S. (2003). Effectiveness and validity of accommodations for English language learners in large-scale assessments: CSE Report 608. https://cresst.org/wp-content/uploads/R608.pdf
   Google Scholar
• Abedi, J., & Gándara, P. (2006). Performance of English language learners as a subgroup in large-scale assessment: Interaction of research and policy. Educational Measurement: Issues and Practice, 25(4), 36–46. doi:10.1111/j.1745-3992.2006.00077.x
   Google Scholar
• Abedi, J., Hofstetter, C., Baker, E., & Lord, C. (2001). NAEP math performance and test accommodations: Interactions with student language background: CSE technical report 536. Retrieved from http://cresst.org/wp-content/uploads/newTR536.pdf
   Google Scholar
• Abedi, J., & Lord, C. (2001). The language factor in mathematics tests. Applied Measurement in Education, 14(3), 219–234. doi:10.1207/S15324818AME1403_2
   Web of Science ®Google Scholar
• Abedi, J., Lord, C., Hofstetter, C., & Baker, E. (2000). Impact of accommodation strategies on English language learners’ test performance. Educational Measurement: Issues and Practice, 19(3), 16–26. doi:10.1111/j.1745-3992.2000.tb00034.x
   Google Scholar
• Allalouf, A., Hambleton, R. K., & Sireci, S. G. (1999). Identifying the causes of DIF in translated verbal items. Journal of Educational Measurement, 36(3), 185–198. doi:10.1111/j.1745-3984.1999.tb00553.x
   Web of Science ®Google Scholar
• Ambrose, R., & Molina, M. (2010). First-grade Latino English-language learners’ performance on story problems in English versus Spanish. Canadian Journal of Science, Mathematics and Technology Education, 10(4), 356–369. doi:10.1080/14926156.2010.524968
   Google Scholar
• American Educational Research Association (AERA), American Psychological Association (APA), & National Council on Measurement in Education (NCME). (2014). Standards for educational and psychological testing. Washington, DC: AERA.
   Google Scholar
• Avalos, M. A., Medina, E., & Secada, W. G. (2018). Reading mathematics problems: Exploring how language counts for middle school students with varying mathematics proficiency. In A. Bailey, C. Maher, & L. Wilkinson (Eds.), Language, literacy, and learning in the STEM disciplines: Language counts for english learners (pp. 55–78). New York, NY: Taylor Francis.
   Google Scholar
• Avenia-Tapper, B., & Llosa, L. (2015). Construct relevant or irrelevant? The role of linguistic complexity in the assessment of English language learners’ science knowledge. Educational Assessment, 20(2), 95–111. doi:10.1080/10627197.2015.1028622
   Web of Science ®Google Scholar
• Baddeley, A. (1992). Working memory. Science, 255(5044), 556–559. doi:10.1126/science.1736359
   PubMed Web of Science ®Google Scholar
• Bailey, A. (2001/2005). Language analysis of standardized achievement tests: Considerations in the assessment of English lanaguage learners. In J. Abedi, A. Bailey, F. Butler, M. Castellon-Wellington, S. Leon, & J. Mirocha Eds., The validity of administering large-scale content assessments to english language learners: An investigation from three perspectives. National Center for Research on Evaluation, Standards, and Student Testing (CRESST). Retrieved from https://files.eric.ed.gov/fulltext/ED492891.pdf
   Google Scholar
• Bailey, A., & Butler, F. (2003). An evidentiary framework for operationalizing academic language for broad application to K-12 education: A design document (CSE Tech. Rep. No. 611). Retrieved from https://cresst.org/wp-content/uploads/R611.pdf
   Google Scholar
• Bailey, A. L., & Carroll, P. E. (2015). Assessment of English language learners in the era of new academic content standards. Review of Research in Education, 39(1), 253–294. doi:10.3102/0091732X14556074
   Web of Science ®Google Scholar
• Banks, K., Jeddeeni, A., & Walker, C. M. (2016). Assessing the effect of language demand in bundles of math word problems. International Journal of Testing, 16(4), 269–287. doi:10.1080/15305058.2015.1113972
   Web of Science ®Google Scholar
• Barwell, R. (2018). From language as a resource to sources of meaning in multilingual mathematics classrooms. The Journal of Mathematical Behavior, 50, 155–168. doi:10.1016/j.jmathb.2018.02.007
   Web of Science ®Google Scholar
• Butler, F. A., & Stevens, R. (2001). Standardized assessment of the content knowledge of English language learners K-12: Current trends and old dilemmas. Language Testing, 18(4), 409–427. doi:10.1177/026553220101800406
   Google Scholar
• Byrnes, J. P. (2008). Cognitive development and learning in instructional contexts (3rd ed.). New York, NY: Pearson Education.
   Google Scholar
• Camili, G., & Shepard, L. (1994). Methods for identifying biased test items. Newbury Park, CA: Sage.
   Google Scholar
• Castelli, F., Glaser, D. E., & Butterworth, B. (2006). Discrete and analogue quantity processing in the parietal lobe: A functional MRI study. Proceedings of the National Academy of Sciences of the United States of America, 103(12), 4693–4698. doi:10.1073/pnas.0600444103
   PubMed Web of Science ®Google Scholar
• Castellon-Wellington, M. (2000). The impact of preference for accommodations: The performance of English language learners on large-scale academic achievement Tests (CSE Technical Report 524). Retrieved from http://cresst.org/wp-content/uploads/TECH524.pdf
   Google Scholar
• Chalmers, R. P. (2010). mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1–29. doi:10.18637/jss.v048.i06
   Web of Science ®Google Scholar
• Cheuk, T., Daro, P., & Daro, V. (2018). The language of mathematics and summative assessment: Interactions that matter for English learners. In A. Bailey, C. Maher, & L. Wilkinson (Eds.), Language, literacy, and learning in the STEM disciplines(pp. 187–205). New York, NY: Routledge.
   Google Scholar
• De Corte, E., Verschaffel, L., & de Win, L. (1985). Influence of rewording verbal problems on children’s problem representations and solutions. Journal of Educational Psychology, 77(4), 460–470. doi:10.1037/0022–0663.77.4.460
   Web of Science ®Google Scholar
• Dehaene, S., Molko, N., Cohen, L., & Wilson, A. J. (2004). Arithmetic and the brain. Current Opinion in Neurobiology, 14(2), 218–224. doi:10.1016/j.conb.2004.03.008
   PubMed Web of Science ®Google Scholar
• Dehaene, S., Spelke, E. S., Pinel, P., Stanescu, R., & Tsivkin, S. (1999). Sources of mathematical thinking: Behavioral and brain-imaging evidence. Science, 284(5416), 970–974. doi:10.1126/science.284.5416.970
   PubMed Web of Science ®Google Scholar
• Dorans, N. J., & Holland, P. W. (1993). DIF detection and description: Mantel-Haenszel and Standardization. In P. W. Holland & H. Wainer (Eds.), Differential item functioning (pp. pp. 35–66). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Durán, R. P. (1989). Assessment and instruction of at-risk Hispanic students. Exceptional Children, 56(2), 154–158. doi:10.1177/001440298905600207
   PubMed Web of Science ®Google Scholar
• Education Quality and Accountability Office. (2016). EQAO’s Technical Report for the 2015–2016 Assessments. Retrieved from https://www.eqao.com/en/assessments/DMA-docs/technical-report-2015-2016.pdf
   Google Scholar
• Fuchs, L. S., Compton, D. L., Fuchs, D., Paulsen, K., Bryant, J. D., & Hamlett, C. L. (2005). The prevention, identification, and cognitive determinants of math difficulty. Journal of Educational Psychology, 97(3), 493–513. doi:10.1037/0022-0663.97.3.493
   Web of Science ®Google Scholar
• García, G. E. (1991). Factors influencing the English reading test performance of Spanish-speaking Hispanic children. Reading Research Quarterly, 26(4), 371–392. doi:10.2307/747894
   Web of Science ®Google Scholar
• Gelman, R., & Butterworth, B. (2005, January). Number and language: How are they related? Trends in Cognitive Sciences, 9(1), 6–10. doi:10.1016/j.tics.2004.11.004
   PubMed Web of Science ®Google Scholar
• Gierl, M. J. (2005). Using dimensionality-based DIF analyses to identify and interpret constructs that elicit group differences. Educational Measurement: Issues and Practice, 24(1), 3–14. doi:10.1111/j.1745-3992.2005.00002.x
   Google Scholar
• Gierl, M. J., Gotzmann, A., & Boughton, K. A. (2004). Performance of SIBTEST when the percentage of DIF items is large. Applied Measurement in Education, 17(3), 241–264. doi:10.1207/s15324818ame1703_2
   Web of Science ®Google Scholar
• Hecht, S. A., Torgesen, J. K., Wagner, R. K., & Rashotte, C. A. (2001). The relations between phonological processing abilities and emerging individual differences in mathematical computation skills: A longitudinal study from second to fifth grades. Journal of Experimental Child Psychology, 79(2), 192–227. doi:10.1006/jecp.2000.2586
   PubMed Web of Science ®Google Scholar
• Hickendorff, M. (2013). The language factor in elementary mathematics assessments: Computational skills and applied problem solving in a multidimensional IRT framework. Applied Measurement in Education, 26(4), 253–278. doi:10.1080/08957347.2013.824451
   Web of Science ®Google Scholar
• Jang, E. E., & Roussos, L. (2009). Integrative analytic approach to detecting and interpreting L2 vocabulary DIF. International Journal of Testing,9(3), 238–259. doi:10.1080/15305050903107022
   Google Scholar
• Jordan, N. C., Kaplan, D., Nabors Oláh, L., & Locuniak, M. N. (2006). Number sense growth in kindergarten: A longitudinal investigation of children at risk for mathematics difficulties. Child Development, 77(1), 153–175. doi:10.1111/j.1467-8624.2006.00862.x
   PubMed Web of Science ®Google Scholar
• Kieffer, M. J., Lesaux, N. K., Rivera, M., & Francis, D. J. (2009). Accommodations for English language learners taking large-scale assessments: A meta-analysis on effectiveness and validity. Review of Educational Research, 79(3), 1168–1201. doi:10.3102/0034654309332490
   Web of Science ®Google Scholar
• Koedinger, K. R., & Nathan, M. J. (2004). The real story behind story problems: Effects of representations on quantitative reasoning. Journal of the Learning Sciences, 13(2), 129–164. doi:10.1207/s15327809jls1302_1
   Web of Science ®Google Scholar
• Krajewski, K., & Schneider, W. (2009). Early development of quantity to number-word linkage as a precursor of mathematical school achievement and mathematical difficulties: Findings from a four-year longitudinal study. Learning and Instruction, 19(6), 513–526. doi:10.1016/j.learninstruc.2008.10.002
   Web of Science ®Google Scholar
• Lager, C. A. (2006). Types of mathematics-language reading interactions that unnecessarily hinder algebra learning and assessment. Reading Psychology, 27(2–3), 165–204. doi:10.1080/02702710600642475
   Google Scholar
• Lan, X., Legare, C. H., Ponitz, C. C., Li, S., & Morrison, F. J. (2011). Investigating the links between the subcomponents of executive function and academic achievement: A cross-cultural analysis of Chinese and American preschoolers. Journal of Experimental Child Psychology, 108(3), 677–692. doi:10.1016/j.jecp.2010.11.001
   PubMed Web of Science ®Google Scholar
• Landerl, K., Bevan, A., & Butterworth, B. (2004). Developmental dyscalculia and basic numerical capacities: A study of 8–9-year-old students. Cognition, 93(2), 99–125. doi:10.1016/j.cognition.2003.11.004
   PubMed Web of Science ®Google Scholar
• Lee, K., Ng, E. L., & Ng, S. F. (2009). The contributions of working memory and executive functioning to problem representation and solution generation in algebraic word problems. Journal of Educational Psychology, 101(2), 373–387. doi:10.1037/a0013843
   Web of Science ®Google Scholar
• LeFevre, J.-A., Fast, L., Skwarchuk, S.-L., Smith-Chant, B. L., Bisanz, J., Kamawar, D., & Penner-Wilger, M. (2010). Pathways to mathematics: Longitudinal predictors of performance. Child Development, 81(6), 1753–1767. doi:10.1111/j.1467-8624.2010.01508.x
   PubMed Web of Science ®Google Scholar
• Mahoney, K. (2008). Linguistic influences on differential item functioning for second language learners on the national assessment of educational progress. International Journal of Testing, 8(1), 14–33. doi:10.1080/15305050701808615
   Google Scholar
• Martiniello, M. (2008). Language and the performance of English-language learners in math word problems. Harvard Educational Review, 78(2), 333–368. doi:10.17763/haer.78.2.70783570r1111t32
   Web of Science ®Google Scholar
• Martiniello, M. (2009). Linguistic complexity, schematic representations, and differential item functioning for English language learners in math tests. Educational Assessment, 14(3–4), 160–179. doi:10.1080/10627190903422906
   Google Scholar
• Mestre, J. P. (1988). The role of language comprehension in mathematics and problem solving. In R. R. Cocking & J. P. Mestre (Eds.), Linguistic and cultural influences on learning mathematics (pp. 201–220). Hillsdale, NJ: Erlbaum.
   Google Scholar
• Moschkovich, J. (2018). Talking to learn mathematics with understanding: Supporting academic literacy in mathematics for English learners. In A. Bailey, C. Maher, & L. Wilkinson (Eds.), Language, literacy, and learning in the STEM disciplines(pp. 13–34). New York, NY: Routledge.
   Google Scholar
• Nathan, M. J., Kintsch, W., & Young, E. (1992). A theory of algebra-word-problem comprehension and Its implications for the design of learning environments. Cognition and Instruction, 9(4), 329–389. doi:10.1207/s1532690xci0904_2
   Web of Science ®Google Scholar
• National Academies of Sciences Engineering & Medicine. (2018). English learners in STEM subjects: Transforming classrooms, schools, and lives. National Academies Press.
   Google Scholar
• Ockey, G. J. (2007). Investigating the validity of math word problems for English language learners with DIF. Language Assessment Quarterly, 4(2), 149–164. doi:10.1080/15434300701375717
   Google Scholar
• OECD. (2012). Untapped skills: Realising the potential of immigrant students. Author. doi:10.1787/9789264172470-en
   Google Scholar
• OECD. (2016). PISA 2015 results (Volume I): Excellence and equity in education. Author. doi:10.1787/9789264266490-en
   Google Scholar
• Okamoto, Y., & Case, R. (1996). Exploring the microstructure of children’s conceptual structures in the domain of number. Monographs of the Society for Research in Child Development, 61(1–2), 27–58. doi:10.1111/j.1540-5834.1996.tb00536.x
   PubMed Web of Science ®Google Scholar
• Paas, F., Renkl, A., & Sweller, J. (2003). Cognitive load theory and instructional design: Recent developments. Educational Psychologist, 38(1), 1–4. doi:10.1207/S15326985EP3801_1
   Web of Science ®Google Scholar
• Pitoniak, M. J., Young, J. W., Martiniello, M., King, T. C., Buteux, A., & Ginsburgh, M. (2009). Guidelines for the assessment of English language learners. Princeton, NJ: Educational Testing Service.
   Google Scholar
• Powell, S. R. (2011). Solving word problems using schemas: A review of the literature. Learning Disabilities Research & Practice, 26(2), 94–108. doi:10.1111/j.1540-5826.2011.00329.x
   PubMed Web of Science ®Google Scholar
• Purpura, D. J., Logan, J. A. R., Hassinger-Dasm, B., & Napoli, A. R. (2017). Why do early mathematics skills predict later reading? The role of mathematical language. Developmental Psychology, 53(9), 1633–1642. doi:10.1037/dev0000375
   PubMed Web of Science ®Google Scholar
• Purpura, D. J., & Reid, E. E. (2016). Mathematics and language: Individual and group differences in mathematical language skills in young children. Early Childhood Research Quarterly, 36(Complete), 259–268. doi:10.1016/j.ecresq.2015.12.020
   Google Scholar
• Purpura, D. J., Schmitt, S. A., & Ganley, C. M. (2017). Foundations of mathematics and literacy: The role of executive functioning components. Journal of Experimental Child Psychology, 153(Complete), 15–34. doi:10.1016/j.jecp.2016.08.010
   PubMedGoogle Scholar
• Resnick, L. B. (1989). Developing mathematical knowledge. American Psychologist, 44(2), 162–169. doi:10.1037//0003-066X.44.2.162
   Web of Science ®Google Scholar
• Roussos, L., & Stout, W. (1996a). A multidimensionality-based DIF analysis paradigm. Applied Psychological Measurement, 20(4), 355–371. doi:10.1177/014662169602000404
   Web of Science ®Google Scholar
• Roussos, L., & Stout, W. (1996b). Simulation studies of the effects of small sample size and studied item parameters on SIBTEST and Mantel-Haenszel Type I error performance. Journal of Educational Measurement, 33(2), 215–230. doi:10.1111/j.1745-3984.1996.tb00490.x
   Web of Science ®Google Scholar
• Shealy, R., & Stout, W. (1993). A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF. Psychometrika, 58(2), 159–194. doi:10.1007/BF02294572
   Web of Science ®Google Scholar
• Shepard, L. A. (2000). The role of assessment in a learning culture. Educational Researcher, 29(7), 4–14. doi:10.3102/0013189X029007004
   Google Scholar
• Snetzler, S., & Qualls, A. (2000). Examination of differential item functioning on a standardized achievement battery with limited English proficient students. Educational and Psychological Measurement, 60(4), 564–577. doi:10.1177/00131640021970727
   Web of Science ®Google Scholar
• Solórzano, R. W. (2008). High stakes testing: Issues, implications, and remedies for English language learners. Review of Educational Research, 78(2), 260–329. doi:10.3102/0034654308317845
   Web of Science ®Google Scholar
• Spelke, E. S., & Tsivkin, S. (2001). Language and number: A bilingual training study. Cognition, 78(1), 45–88. doi:10.1016/S0010-0277(00)00108-6
   PubMed Web of Science ®Google Scholar
• Stern, E., & Lehrndorfer, A. (1992). The role of situational context in solving word problems. Cognitive Development, 7(2), 259–268. doi:10.1016/0885-2014(92)90014-I
   Web of Science ®Google Scholar
• Stevens, R., Butler, F. A., & Castellon-Wellington, M. (2000). Academic language and content assessment: Measuring the progress of ELLs. Final Deliverable to OERI, Contract No. R305B60002. Retrieved from https://cresst.org/wp-content/uploads/TR552.pdf
   Google Scholar
• Stout, W., & Roussos, L. (1995). SIBTEST users manual [Computer program manual] (2nd ed.). Urbana-Champaign: University of Illinois, Department of Statistics.
   Google Scholar
• Thomas, S., & Oldfather, P. (1997). Intrinsic motivations, literacy, and assessment practices: “That’s my grade. That’s me.”. Educational Psychologist, 32(2), 107–123. doi:10.1207/s15326985ep3202_5
   Web of Science ®Google Scholar
• Ulibarri, D. M., Spencer, M. L., & Rivas, G. A. (1981). Language proficiency and academic achievement: A study of language proficiency tests and their relationship to school ratings as predictors of academic achievement. NABE Journal, 5(3), 47–80. doi:10.1080/08855072.1981.10668410
   Google Scholar
• Van Rinsveld, A., Schiltz, C., Brunner, M., Landerl, K., & Ugen, S. (2016). Solving arithmetic problems in first and second language: Does the language context matter? Learning and Instruction, 42, 72–82. doi:10.1016/j.learninstruc.2016.01.003
   Web of Science ®Google Scholar
• Vicente, S., Orrantia, J., & Verschaffel, L. (2007). Influence of situational and conceptual rewording on word problem solving. British Journal of Educational Psychology, 77(4), 829–848. doi:10.1348/000709907X178200
   PubMed Web of Science ®Google Scholar
• Vukovic, R. K., & Lesaux, N. K. (2013a). The language of mathematics: Investigating the ways language counts for children’s mathematical development. Journal of Experimental Child Psychology, 115(2), 227–244. doi:10.1016/j.jecp.2013.02.002
   PubMed Web of Science ®Google Scholar
• Vukovic, R. K., & Lesaux, N. K. (2013b). The relationship between linguistic skills and arithmetic knowledge. Learning and Individual Differences, 23, 87–91. doi:10.1016/j.lindif.2012.10.007
   Web of Science ®Google Scholar
• Walker, C., Zhang, B., Banks, K., & Cappaert, K. (2012). Establishing effect size guidelines for interpreting the results of differential bundle functioning analyses using SIBTEST. Educational and Psychological Measurement, 72(3), 415–434. doi:10.1177/0013164411422250
   Web of Science ®Google Scholar
• Wolf, M. K., Kao, J., Griffin, N., Herman, J. L., Bachman, P. L., Chang, S. M., & Farnsworth, T. (2008). Issues in assessing English language learners: English language proficiency measures and accommodation uses. Practice Review (Part 2 of 3). CRESST Report 732. Retrieved from https://cresst.org/wp-content/uploads/R731.pdf
   Google Scholar
• Wolf, M. K., Kao, J. C., Rivera, N. M., & Chang, S. M. (2012). Accommodation practices for english language learners in states’ mathematics assessments. Teachers College Record, 114(3), n3.
   Web of Science ®Google Scholar
• Wolf, M. K., & Leon, S. (2009). An investigation of the language demands in content assessments for English language learners. Educational Assessment, 14(3–4), 139–159. doi:10.1080/10627190903425883
   Google Scholar
• Young, J. W. (2009). A framework for test validity research on content assessments taken by English language learners. Educational Assessment, 14(3–4), 122–138. doi:10.1080/10627190903422856
   Google Scholar
• Young, J. W., Cho, Y., Ling, G., Cline, F., Steinberg, J., & Stone, E. (2008). Validity and fairness of state standards-based assessments for English language learners. Educational Assessment, 13(2–3), 170–192. doi:10.1080/10627190802394388
   Google Scholar
• Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland & H. Wainer (Eds.), Differential Item Functioning(pp. 337–348). Hillsdale, NJ: Lawrence Erlbaum Associates Inc.
   Google Scholar
• Zwick, R., & Ercikan, K. (1989). Analysis of differential item functioning in the NAEP history assessment. Journal of Educational Measurement, 26(1), 55–66. doi:10.1111/j.1745-3984.1989.tb00318.x
   Web of Science ®Google Scholar