1,735
Views
23
CrossRef citations to date
0
Altmetric
Articles

Language effects in international testing: the case of PISA 2006 science items

, &
Pages 427-455 | Received 15 Feb 2016, Accepted 24 Jul 2016, Published online: 25 Aug 2016

References

  • Ahmed, A., & Pollitt, A. (1999). Curriculum demands and question difficulty. Paper presented at the International Association for Educational Assessment, Bled, Slovenia.
  • Anderson, C. W. (1999). Preface: Inscriptions and science learning. Journal of Research in Science Teaching, 36, 973–974.10.1002/(SICI)1098-2736(199911)36:9<>1.0.CO;2-N
  • Andrich, D. (2014). A structure of index and causal variables. Rasch Measurement Transactions, 28, 1475–1477.
  • Andrich, D., & El Masri, Y. H. (n.d.). PISA and other large-scale assessments: The trade-off between model fit, invariance and validity. Unpublished manuscript.
  • Asil, M., & Brown, G. T. L. (2016). Comparing OECD PISA reading in English to other languages: Identifying potential sources of non-invariance. International Journal of Testing, 16, 71–93.10.1080/15305058.2015.1064431
  • Bailey, A. (2007). The language demands of school: Putting academic English to the test. New Haven, CT: Yale University Press.
  • Baird, J., & Black, P. (2013). Test theories, educational priorities and reliability of public examinations in England. Research Papers in Education, 28, 5–21.10.1080/02671522.2012.754224
  • Baumert, J., Lüdtke, O., Trautwein, U., & Brunner, M. (2009). Large-scale student assessment studies measure the results of processes of knowledge acquisition: Evidence in support of the distinction between intelligence and student achievement. Educational Research Review, 4, 165–176. doi:10.1016/j.edurev.2009.04.002
  • Bramley, T., Hughes, S., Fisher-Hoch, H., & Pollitt, A. (1998). Sources of difficulty in examination questions: Science. UCLES Internal Report.
  • Bybee, R. (1997). Achieving scientific literacy: From purposes to practices. Portsmouth, NH: Heinemann.
  • Bybee, R., & McCrae, B. (2011). Scientific literacy and student attitudes: Perspectives from PISA 2006 science. International Journal of Science Education, 33, 7–26.10.1080/09500693.2010.518644
  • Bybee, R., McCrae, B., & Laurie, R. (2009). PISA 2006: An assessment of scientific literacy. Journal of Research in Science Teaching, 46, 865–883. doi:10.1002/tea.20333
  • Carpenter, P. A., & Shah, P. (1998). A model of the perceptual and conceptual processes in graph comprehension. Journal of Experimental Psychology: Applied, 4, 75–100.
  • Cole, N. S., & Zieky, M. J. (2001). The new faces of fairness. Journal of Educational Measurement, 38, 369–382.10.1111/jedm.2001.38.issue-4
  • Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52, 281–302.10.1037/h0040957
  • Cummins, J. (1979). Cognitive/academic language proficiency, linguistic interdependence, the optimum age question and some other matters. Working Papers on Bilingualism, 19, 121–129.
  • Cummins, J. (1981). The role of primary language development in promoting educational success for language minority students. In California State Department of Education (Ed.), Schooling and language minority students: A theoretical framework (pp. 3–49). Los Angeles, CA: Evaluation, Dissemination and Assessment Center California State University.
  • Cummins, J. (2008). BICS and CALP: Empirical and theoretical status of the distinction. In B. Street & N. H. Hornberger (Eds.), Encyclopedia of language and education (2nd ed., pp. 71–83). New York, NY: Springer Science + Business Media LLC.
  • DeBoer, G. E. (2000). Scientific literacy: Another look at its historical and contemporary meanings and its relationship to science education reform. Journal of Research in Science Teaching, 37, 582–601. doi:10.1002/1098-2736(200008)37:6<582:AID-TEA5>3.0.CO;2-L
  • Dorans, N. J. (1989). Two new approaches to assessing differential item functioning: Standardization and the Mantel-Haenszel method. Applied Measurement in Education, 2, 217–233.10.1207/s15324818ame0203_3
  • Eason, S. H., Goldberg, L. F., Young, K. M., Geist, M. C., & Cutting, L. E. (2012). Reader–text interactions: How differential text and question types influence cognitive skills needed for reading comprehension. Journal of Educational Psychology, 104, 515–528.10.1037/a0027182
  • Eivers, E. (2010). PISA: Issues in implementation and interpretation. The Irish Journal of Education/Iris Eireannach an Oideachais, 38, 94–118. Retrieved from http://www.jstor.org/stable/20789130
  • El Masri, Y. H. (2015). Comparability of science assessment across cultures: The case of PISA science 2006. Oxford: University of Oxford.
  • Ercikan, K., & Koh, K. (2005). Examining the construct comparability of the English and French versions of TIMSS. International Journal of Testing, 5, 23–35.10.1207/s15327574ijt0501_3
  • Evans, N., & Levinson, S. C. (2009). The myth of language universals: Language diversity and its importance for cognitive science. Behavioral and Brain Sciences, 32, 329–492.
  • Fisher-Hoch, H., Hughes, S., & Bramley, T. (1997). What makes GCSE examination questions difficult? Outcomes of manipulating difficulty of GCSE questions. Paper presented at the British Educational Research Association Annual Conference, University of York.
  • Frantz, R. S., Bailey, A. L., Starr, L., & Perea, L. (2014). Measuring academic language proficiency in school-age English language proficiency assessments under new college and career readiness standards in the United States. Language Assessment Quarterly, 11, 432–457.10.1080/15434303.2014.959123
  • Friel, S. N., Curcio, F. R., & Bright, G. M. (2001). Making sense of graphs: Critical factors influencing comprehension and instructional implications. Journal for Research in Mathematics Education, 32, 124–158.10.2307/749671
  • Gaskins, I. W., Guthrie, J. T., Satlow, E., Ostertag, J., Six, L., Byrne, J., & Connor, B. (1994). Integrating instruction of science, reading, and writing: Goals, teacher development, and assessment. Journal of Research in Science Teaching, 31, 1039–1056.10.1002/(ISSN)1098-2736
  • Gee, J. P. (2000). Discourse and sociocultural studies in reading. In M. L. Kamil, P. Mosenthal, P. D. Pearson, & R. Barr (Eds.), Handbook of reading research (Vol. 3, pp. 195–207). Mahweh, NJ: Erlbaum.
  • Goldstein, H. (2004). International comparisons of student attainment: Some issues arising from the PISA study. Assessment in Education: Principles, Policy and Practice, 11, 319–330.10.1080/0969594042000304618
  • Goldstein, H., Bonnet, G., & Rocher, T. (2007). Multilevel structural equation models for the analysis of comparative data on educational performance. Journal of Educational and Behavioral Statistics, 32, 252–286.10.3102/1076998606298042
  • Graesser, A. C., Bommareddy, S., Swamer, S., & Golding, J. M. (1996). Integrating questionnaire design with a cognitive computational model of human question answering. In N. Schwartz & S. Sudman (Eds.), Answering questions: Methodology for determining cognitive and communicative processes in survey research (pp. 143–174). San Francisco, CA: Jossey-Bass.
  • Graesser, A. C., Cai, Z., Louwerse, M. M., & Daniel, F. (2006). Question Understanding Aid (QUAID): A web facility that tests question comprehensibility. Public Opinion Quarterly, 70, 3–22. doi:10.1093/poq/nfj012
  • Graesser, A. C., Jeon, M., Yang, Y., & Cai, Z. (2007). Discourse cohesion in text and tutorial dialogue. Information Design Journal, 15, 199–213.
  • Graesser, A. C., & McNamara, D. S. (2011). Computational analyses of multilevel discourse comprehension. Topics in Cognitive Science, 3, 371–398. doi:10.1111/j.1756-8765.2010.01081.x
  • Graesser, A. C., McNamara, D. S., Cai, Z., Conley, M., Li, H., & Pennebaker, J. (2014). Coh-Metrix measures text characteristics at multiple levels of language and discourse. The Elementary School Journal, 115, 210–229.10.1086/678293
  • Graesser, A. C., McNamara, D. S., & Kulikowich, J. M. (2011). Coh-Metrix: Providing multilevel analyses of text characteristics. Educational Researcher, 40, 223–234. doi:10.3102/0013189X11413260
  • Grisay, A., de Jong, J. H. A. L., Gebhardt, E., Berezner, A., & Halleux-Monseur, B. (2007). Translation equivalence across PISA countries. Journal of Applied Measurement, 8, 249–266.
  • Grisay, A., & Monseur, C. (2007). Measuring the equivalence of item difficulty in the various versions of an international test. Studies in Educational Evaluation, 33, 69–86. doi:10.1016/j.stueduc.2007.01.006
  • Halliday, M. A. K. (1978). Language as social semiotic: The social interpretation of language meaning. London: Edward Arnold.
  • Halliday, M. A. K. (1985). Spoken and written language. Waurn Ponds: Deakin University.
  • Halliday, M. A. K. (1994). A language development approach to education. In N. Bird (Ed.), Language and learning (pp. 5–17). Hong Kong: Institute of Language in Education.
  • Halliday, M. A. K., & Martin, J. R. (1993). Writing science: Literacy and discursive power. Pittsburgh, PA: University of Pittsburgh.
  • Hambleton, R. K. (1993). Translating achievement tests for use in cross-national studies. European Journal of Psychological Assessment, 9, 54–65.
  • Hambleton, R. K. (2001). The next generation of the ITC test translation and adaptation guidelines. European Journal of Psychological Assessment, 17, 164–172.10.1027//1015-5759.17.3.164
  • Hambleton, R. K. (2005). Issues, designs, and technical guidelines for adapting tests into multiple languages and cultures. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3–38). Mahwah, NJ: Lawrence Erlbaum Associates.
  • Hambleton, R. K., Merenda, P. F., & Spielberger, C. D. (2005). Adapting educational and psychological tests for cross-cultural assessment. Mahwah, NJ: Lawrence Erlbaum Associates.
  • Haneda, M. (2014). From academic language to academic communication: Building on English learners’ resources. Linguistics and Education, 26, 126–135.10.1016/j.linged.2014.01.004
  • Hasan, R., & Martin, J. R. (1989). Language development: Learning language, learning culture. Norwood, NJ: Ablex.
  • Holbrook, J., & Rannikmae, M. (2007). The nature of science education for enhancing scientific literacy. International Journal of Science Education, 29, 1347–1362. doi:10.1080/09500690601007549
  • Holland, P. W., & Thayer, D. T. (1988). Differential item performance and the Mantel-Haenszel procedure. In H. Wainer & H. Braun (Eds.), Test validity (pp. 129–145). Hillsdale, NJ: Lawrence Erlbaum Associates.
  • Holland, P. W., & Wainer, H. (1993). Differential item functioning. Hillsdale, NJ: Lawrence Erlbaum Associates.
  • Huang, X., Wilson, M., & Wang, L. (2016). Exploring plausible causes of differential item functioning in the PISA science assessment: Language, curriculum or culture. Educational Psychology, 36, 378–390. doi:10.1080/01443410.2014.946890
  • International Test Commission. (2005). International guidelines on test adaptation. Retrieved from www.intestcom.org
  • Jones, L. R., Wheeler, G., & Centurino, V. A. S. (2013). TIMSS 2015 science framework. In I. V. S. Mullis & M. O. Martin (Eds.), TIMSS 2015 assessment frameworks (pp. 29–59). Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
  • Jurafsky, D., & Martin, J. H. (2009). Speech and language processing: An introduction to natural language processing, computational linguistics, and speech recognition. Upper Saddle River, NJ: Pearson Prentice Hall.
  • Kaufman, A. S., & Kaufman, N. L. (1990). Kaufman brief intelligence test. Circle Pines, MN: American Guidance Service.
  • Klare, G. R. (1974). Assessing readability. Reading Research Quarterly, 10, 62–102.10.2307/747086
  • Kreiner, S., & Christensen, K. B. (2014). Analyses of model fit and robustness. A new look at the PISA scaling model underlying ranking of countries according to reading literacy. Psychometrika, 79, 210–231.10.1007/s11336-013-9347-z
  • Landauer, T. K. (2012). Improving text complexity measurement through the reading maturity metric. Pearson. Retrieved from http://images.pearsonassessments.com/images/tmrs/Word_Maturity_and_Text_Complexity_NCME.pdf
  • Landauer, T. K., Kireyev, K., & Panaccione, C. (2011). Word maturity: A new metric for word knowledge. Scientific Studies of Reading, 15, 92–108. doi:10.1080/10888438.2011.536130
  • Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch, W. (Eds.). (2007). Handbook of latent semantic analysis. Mahwah, NJ: Lawrence Erlbaum.
  • Lederman, N. G. (1992). Students’ and teachers’ conceptions of the nature of science: A review of the research. Journal of Research in Science Teaching, 29, 331–359. doi:10.1002/tea.3660290404
  • Lenzner, T., Kaczmirek, L., & Lenzner, A. (2010). Cognitive burden of survey questions and response times: A psycholinguistic experiment. Applied Cognitive Psychology, 24, 1003–1020. doi:10.1002/acp.1602
  • Li, H., Graesser, A. C., Conley, M., Cai, Z., Pavlik, P., & Pennebaker, J. W. (2016). A new measure of text formality: An analysis of discourse of Mao Zedong. Discourse Processes, 53, 205–232. doi:10.1080/0163853X.2015.1010191
  • Lupyan, G., & Dale, R. (2010). Language structure is partly determined by social structure. PLoS One, 5, e8559.10.1371/journal.pone.0008559
  • Mahboob, A. (in press). Understanding language variation: Implications of the NNEST lens for TESOL teacher education programs. In J. de Dios & A. Mart’nez (Eds.), Native and non-native speakers in English language teaching: Implications and challenges for teacher education. Boston, MA: De Gruyter Mouton.
  • Maul, A. (2013a). Method effects and the meaning of measurement. Frontiers in Psychology, 4, 1–13. doi:10.3389/fpsyg.2013.00169
  • Maul, A. (2013b). On the ontology of psychological attributes. Theory and Psychology, 23, 752–769. doi:10.1177/0959354313506273
  • McNamara, D. S., Graesser, A. C., McCarthy, P. M., & Cai, Z. (2014). Automated evaluation of text and discourse with Coh-Metrix. Cambridge, MA: Cambridge University Press.10.1017/CBO9780511894664
  • Millar, R., & Osborne, J. F. (1998). Beyond 2000: Science education for the future. London: King’s College London.
  • National Research Council. (1996). National science education standards. Washington, DC: National Academy of Sciences.
  • Nelson, J., Perfetti, C., Liben, D., & Liben, M. (2011). Measures of text difficulty: Testing their predictive value for grade levels and student performance. New York, NY: Student Achievement Partners.
  • Norman, O. (1998). Marginalized discourses and scientific literacy. Journal of Research in Science Teaching, 35, 365–374.10.1002/(ISSN)1098-2736
  • Norris, S. P., & Phillips, L. M. (2003). How literacy in its fundamental sense is central to scientific literacy. Science Education, 87, 224–240.10.1002/(ISSN)1098-237X
  • NSES. (1996). National science education standards. Washington, DC: National Academy Press.
  • Oliveri, M. E., & Ercikan, K. (2011). Do different approaches to examining construct comparability in multilanguage assesments lead to similar conclusions? Applied Measurement in Education, 24, 349–366. doi:10.1080/08957347.2011.607063
  • Organisation for Economic Cooperation and Development. (1999). Measuring student knowlede and skills: A new framework for assessment. Paris: Author.
  • Organisation for Economic Cooperation and Development. (2006). Assessing scientific, reading and mathematics literacy: A framework for PISA 2006. Paris: Author.
  • Organisation of Economic Co-operation and Development. (2009a). PISA 2006 technical report. Paris: Author.
  • Organisation for Economic Co-operation and Development. (2009b). PISA data analysis manual (2nd ed.). Paris: Author.
  • Organisation for Economic Cooperation and Development. (2016). PISA 2015 science framework. In PISA 2015 assessment and analytical framework: Science, reading, mathematic and financial literacy (pp. 17–46). Paris: OECD Publishing. doi:http://dx.doi.org/10.1787/9789264255425-3-en
  • Osborne, J. (2007). Science education for the twenty first century. Eurasia Journal of Mathematics, Science & Technology Education, 3, 173–184.
  • Osborne, J. (2013). The 21st century challenge for science education: Assessing scientific reasoning. Thinking Skills and Creativity, 10, 265–279. doi:10.1016/j.tsc.2013.07.006
  • Pellegrino, J. W., & Hilton, M. L. (Eds.). (2015). Education for life and work: Developing transferable knowledge and skills in the 21st century. Washington, DC: The National Academies Press.
  • Pollitt, A., & Ahmed, A. (1999). A new model of the question answering process. Paper Presented at the International Association for Educational Assessment, Bled.
  • Pollitt, A., Ahmed, A., & Crisp, V. (2007). The demands on examination syllabuses and question papers. In P. Newton, J.-A. Baird, H. Goldstein, H. Patrick, & P. Tymms (Eds.), Techniques for monitoring the comparability of examination standards (pp. 166–206). London: Qualifications and Curriculum Authority.
  • Pollitt, A., Entwistle, N. J., Hutchinson, C. J., & De Luca, C. (1985). What makes exam questions difficult? Edinburgh: Scottish Academic Press.
  • Poortinga, Y. H. (1995). Use of tests across cultures. In T. Oakland & R. K. Hambleton (Eds.), International perspectives on academic assessment (pp. 187–206). Boston, MA: Kluwer Academic Publishers.10.1007/978-94-011-0639-9
  • Prais, S. J. (2003). Cautions on OECD’S recent educational survey (PISA). Oxford Review of Education, 29, 139–163.10.1080/0305498032000080657
  • Roth, W.-M. (2002). Reading graphs: Contributions to an integrative concept of literacy. Journal of Curriculum Studies, 34(1), 1–24.10.1080/00220270110068885
  • Sandilands, D., Oliveri, M. E., Zumbo, B. D., & Ercikan, K. (2013). Investigating sources of differential item functioning in international large-scale assessments using a confirmatory approach. International Journal of Testing, 13, 152–174. doi:10.1080/15305058.2012.690140
  • Scarcella, R. (2003). Academic English: A conceptual framework (Technical Report NO. 2003–1). Irvine, CA: The University of California Linguistic Minority Research Institute.
  • Schleppegrell, M. J., & Colombi, M. C. (Eds.). (2002). Developing advanced literacy in first and second languages: Meaning with power. Mahwah, NJ: Lawrence Erlbaum Associates.
  • Shala, L., Rus, V., & Graesser, A. C. (2010). Automated speech act classification in Arabic. Subjetividad Y Procesos Cognitivos, 14, 284–292.
  • Shala, L., Vasile, R., & Graesser, A. C. (2014). A bilingual analysis of cohesion in a corpus of leader speeches. In E. William & C. Boonthum-Denecke (Eds.), Twenty-seventh international Florida artificial intelligence research society conference (pp. 225–230). Palo Alto, CA: The AAAI Press.
  • Sheehan, K. M., Kostin, I., Napolitano, D., & Flor, M. (2014). The TextEvaluator tool. The Elementary School Journal, 115, 184–209.10.1086/678294
  • Sireci, S. G. (1997). Technical issues in linking tests across languages. Educational Measurement: Issues and Practice, 16, 12–19.
  • Snow, C. E. (2010). Academic language and the challenge of reading for learning about science. Science, 328, 450–452.10.1126/science.1182597
  • Snow, C. E., & Uccelli, P. (2009). The challenge of academic language. In D. R. Olson & N. Torrance (Eds.), The Cambridge handbook of literacy (pp. 112–133). New York, NY: Cambridge University Press.10.1017/CBO9780511609664
  • Spielberger, C. D., Moscoso, M. S., & Brunner, T. M. (2005). Cross-cultural assessment of emotional states and personality traits. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 343–368). Mahwah, NJ: Lawrence Erlbaum Associates.
  • Stroud, C., & Kerfoot, C. (2013). Towards rethinking multilingualism and language policy for academic literacies. Linguistics and Education, 24, 396–405.10.1016/j.linged.2013.09.003
  • Thomas, G., & Durant, J. (1987). Why should we promote the public understanding of science? Scientific Literacy Papers, 1, 1–14.
  • Trudgill, P. (2011). Sociolinguistics typology: Social determinants of linguistic structure and complexity. Oxford: Oxford University Press.
  • van de Vijver, F. J. R., & Poortinga, Y. H. (2005). Conceptual and methodological issues in adapting tests. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 39–63). Mahwah, NJ: Lawrence Erlbaum Associates.
  • Wellington, J., & Osborne, J. (2001). Language and literacy in science education. Philadelphia, PA: Open University Press.
  • Wiliam, D. (2008). International comparisons and sensitivity to instruction. Assessment in Education: Principles, Policy & Practice, 15, 253–257. doi:10.1080/09695940802417426
  • Wiliam, D. (2010). What counts as evidence of educational achievement? The role of constructs in the pursuit of equity in assessment. Review of Research in Education, 34, 254–284.10.3102/0091732X09351544
  • Wu, A. D., & Ercikan, K. (2006). Using multiple-variable matching to identify cultural sources of differential item functioning. International Journal of Testing, 6, 287–300. doi:10.1207/s15327574ijt0603_5
  • Yore, L. D., & Treagust, D. F. (2006). Current realities and future possibilities: Language and science literacy – empowering research and informing instruction. International Journal of Science Education, 28, 291–314. doi:10.1080/09500690500336973
  • Yu, G., Rea-Dickins, P., & Kiely, R. (2012). The cognitive processes of taking IELTs academic writing task 1. In J. Osborne (Ed.), IELTS Research Reports Volume 11 (pp. 373–449). IDP: IELTS Australia and British Council.
  • Yu, A., & Ebbs, D. (2012). Translation and translation verification. In M. O. Martin & I. V. S. Mullis (Eds.), Methods and procedures in TIMSS 2011 and PIRLS 2011. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College. Retrieved from http://timssandpirls.bc.edu/methods/t-translation.html

Reprints and Corporate Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

To request a reprint or corporate permissions for this article, please click on the relevant link below:

Academic Permissions

Please note: Selecting permissions does not provide access to the full text of the article, please see our help page How do I view content?

Obtain permissions instantly via Rightslink by clicking on the button below:

If you are unable to obtain permissions via Rightslink, please complete and submit this Permissions form. For more information, please visit our Permissions help page.