Assessing scientific reasoning: a comprehensive evaluation of item features that affect item difficulty

References

• AAAS (American Association for the Advancement of Science). 1993. Benchmarks for Science Literacy. New York: Oxford University Press.
   Google Scholar
• AAAS (American Association for the Advancement of Science). 2009. Benchmarks for Science Literacy. New York: Oxford University Press.
   Google Scholar
• AERA (American Educational Research Association), APA (American Psychological Association), and NCME (National Council on Measurement in Education). 2014. Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
   Google Scholar
• Bates, D., M. Maechler, B. Bolker, and S. Walker. 2015. “Fitting Linear Mixed-effects Models Using lme4.” Journal of Statistical Software 67 (1): 1–48. doi:10.18637/jss.v067.i01.
   Web of Science ®Google Scholar
• Baxter, G. P., and R. Glaser. 1998. “Investigating the Cognitive Complexity of Science Assessments.” Educational Measurement: Issues and Practice 17 (3): 37–45. doi:10.1111/j.1745-3992.1998.tb00627.x.
   Google Scholar
• Bramley, T., S. Hughes, H. Fisher-Hoch, and A. Pollitt. 1998. Sources of Difficulty in Examination Questions: Science. Cambridge: UCLES Internal Report.
   Google Scholar
• Bybee, R. W. 2010. “What Is STEM Education?” Science 329: 996.10.1126/science.1194998
   PubMed Web of Science ®Google Scholar
• Crawford, B., and M. Cullin. 2005. “Dynamic Assessments of Preservice Teachers’ Knowledge of Models and Modelling.” In Research and the Quality of Science Education, edited by K. Boersma, M. Goedhart, O. de Jong, and H. Eijkelhof, 309–323. Dordrecht: Springer.10.1007/1-4020-3673-6
   Google Scholar
• Day, A. 2013. The Structure of Scientific Examination Questions. Dordrecht: Springer.
   Google Scholar
• De Boeck, P., M. Bakker, R. Zwitser, M. Nivard, A. Hofman, F. Tuerlinckx, and I. Partchev. 2011. “The Estimation of Item Response Models with the lmer Function from the lme4 Package in R.” Journal of Statistical Software 39 (12): 1–28. doi:10.18637/jss.v039.i12.
   PubMed Web of Science ®Google Scholar
• Dhillon, D. 2003. Predictive Models of Question Difficulty: A Critical Review of the Literature. Manchester: AQA. https://cerp.aqa.org.uk/sites/default/files/pdf_upload/CERP-RP-DD-01022003_0.pdf.
   Google Scholar
• Duggan, S., P. Johnson, and R. Gott. 1996. “A Critical Point in Investigative Work: Defining Variables.” Journal of Research in Science Teaching 33 (5): 461–474.10.1002/(ISSN)1098-2736
   Web of Science ®Google Scholar
• Fischer, F., I. Kollar, S. Ufer, B. Sodian, H. Hussmann, R. Pekrun, B. Neuhaus, et al. 2014. “Scientific Reasoning and Argumentation: Advancing an Interdisciplinary Research Agenda in Education.” Frontline Learning Research 2 (3): 28–45.
   Google Scholar
• Giere, R. N., J. Bickle, and R. F. Mauldin. 2006. Understanding Scientific Reasoning. 5th ed. Belmont, CA: Thomson/Wadsworth.
   Google Scholar
• Glynn, S. M. 2012. “International Assessment: A Rasch Model and Teachers’ Evaluation of TIMSS Science Achievement Items.” Journal of Research in Science Teaching 49 (10): 1321–1344. doi:10.1002/Tea.21059.
   Web of Science ®Google Scholar
• Hartig, J., A. Frey, G. Nold, and E. Klieme. 2012. “An Application of Explanatory Item Response Modeling for Model-based Proficiency Scaling.” Educational and Psychological Measurement 72: 665–686.10.1177/0013164411430707
   Web of Science ®Google Scholar
• Hartmann, S. 2013. “Die Rolle von Leseverständnis und Lesegeschwindigkeit beim Zustandekommen der Leistungen in schriftlichen Tests zur Erfassung naturwissenschaftlicher Kompetenz.” [The Role of Reading Comprehension and Reading Speed in Text-based Assessments of Scientific Inquiry Skills]. Doctoral diss., University of Duisburg-Essen, Germany.
   Google Scholar
• Hartmann, S., S. Mathesius, J. Stiller, P. Straube, D. Krüger, and A. Upmeier zu Belzen. 2015a. “Kompetenzen der naturwissenschaftlichen Erkenntnisgewinnung als Teil des Professionswissens zukünftiger Lehrkräfte: Das Projekt Ko-WADiS.” [Competencies in the Field of Scientific Inquiry as Part of Professional Knowledge of Prospective Teachers]. In Kompetenzerwerb an Hochschulen: Modellierung und Messung. Zur Professionalisierung angehender Lehrerinnen und Lehrer sowie frühpädagogischer Fachkräfte, edited by B. Koch-Priewe, A. Köker, J. Seifried, and E. Wuttke, 39–58. Bad Heilbrunn: Klinkhardt.
   Google Scholar
• Hartmann, S., and A. Upmeier zu Belzen, D. Krüger, and H. A. Pant. 2015b. “Scientific Reasoning in Higher Education.” Zeitschrift für Psychologie 223: 47–53. doi:10.1027/2151-2604/a000199.
   Web of Science ®Google Scholar
• Hoang, Viet-Ngu, Lynette A. May, and T. Tang. 2012. The Effects of Linguistic Factors on Student Performance on Economics Multiple Choice Questions. Brisbane: Social Science Research Network. doi:10.2139/ssrn.2121816.
   Google Scholar
• KMK (The Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in the Federal Republic of Germany), ed. 2005a. Bildungsstandards im Fach Biologie für den Mittleren Schulabschluss (Jahrgangsstufe 10) [Standards for Secondary Education in Biology]. Beschlüsse der Kultusministerkonferenz. München: Wolter Kluwer.
   Google Scholar
• KMK (The Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in the Federal Republic of Germany), ed. 2005b. Bildungsstandards im Fach Chemie für den Mittleren Schulabschluss (Jahrgangsstufe 10) [Standards for Secondary Education in Chemistry]. Beschlüsse der Kultusministerkonferenz. München: Wolter Kluwer.
   Google Scholar
• KMK (The Standing Conference of the Ministers of Education and Cultural Affairs of the Länder in the Federal Republic of Germany), ed. 2005c. Bildungsstandards im Fach Physik für den Mittleren Schulabschluss (Jahrgangsstufe 10) [Standards for Secondary Education in Physics]. Beschlüsse der Kultusministerkonferenz. München: Wolter Kluwer.
   Google Scholar
• Koeppen, K., J. Hartig, E. Klieme, and D. Leutner. 2008. “Current Issues in Competence Modeling and Assessment.” Zeitschrift für Psychologie/Journal of Psychology 216: 61–73.10.1027/0044-3409.216.2.61
   Web of Science ®Google Scholar
• Krell, M., and A. Upmeier zu Belzen, and D. Krüger. 2014. “Students’ Levels of Understanding Models and Modelling in Biology: Global or Aspect-dependent?” Research in Science Education 44: 109–132.10.1007/s11165-013-9365-y
   Web of Science ®Google Scholar
• Le Hebel, F., A. Tiberghien, and P. Montpied. 2013. “Sources of difficulties in Pisa science items.” Proceedings of the European Science Education Research Association Conference 2013. Nicosia: ESERA.
   Google Scholar
• Leong, S. C. 2006. “OnVarying the Difficulty of Test Items.” Paper presented at the 32nd Annual Conference of the International Association for Educational Assessment, Singapore, May 21–26.
   Google Scholar
• Lincoln, D., and M. L. Kearney. 2015. “Competence Assessment in Higher Education.” Studies in Higher Education 40 (3): 391–392. doi:10.1080/03075079.2015.1005334.
   Web of Science ®Google Scholar
• Martiniello, M. 2009. “Linguistic Complexity, Schematic Representations, and Differential Item Functioning for English Language Learners in Math Tests.” Educational Assessment 14: 160–179. doi:10.1080/10627190903422906.
   Google Scholar
• Mathesius, S., A. Upmeier zu Belzen, and D. Krüger. 2014. “Kompetenzen von Biologiestudierenden im Bereich der naturwissenschaftlichen Erkenntnisgewinnung. Entwicklung eines Testinstruments.” [Biology Students’ Scientific Inquiry Competencies: Developing a Measurement Instrument] Erkenntnisweg Biologiedidaktik 13: 73–88.
   Google Scholar
• Mayer, R. E. 2005. “Principles for Reducing Extraneous Processing in Multimedia Learning: Coherence, Signaling, Redundancy, Spatial Contiguity, and Temporal Contiguity Principles.” In The Cambridge Handbook of Multimedia Learning, edited by R. E. Mayer, 183–200. Cambridge: Cambridge University Press.10.1017/CBO9780511816819
   Google Scholar
• Mayer, J. 2007. “Erkenntnisgewinnung als wissenschaftliches Problemlösen.” [Scientific Inquiry as Scientific Problem Solving] In Theorien in der biologiedidaktischen Forschung. Ein Handbuch für Lehramtsstudenten und Doktorande, edited by D. Krüger and H. Vogt, 177–184. Berlin: Springer.
   Google Scholar
• Miller, G. A. 1999. “On Knowing a Word.” Annual Review of Psychology 50: 1–19.10.1146/annurev.psych.50.1.1
   PubMed Web of Science ®Google Scholar
• Mitzel, H. C., D. M. Lewis, R. J. Patz, and D. R. Green. 2001. “The Bookmark Procedure: Psychological Perspectives.” In Setting Performance Standards: Concepts, Methods and Perspectives, edited by G. Cizek, 249–281. Mahwah, NJ: Erlbaum.
   Google Scholar
• Mullis, I. V. S., M. O. Martin, and P. Foy. 2013. “The Impact of Reading Ability on TIMSS Mathematics and Science Achievement at the Fourth Grade: An Analysis by Item Reading Demands.” In TIMSS and PIRLS 2011: Relationships among Reading, Mathematics, and Science Achievement at the Fourth Grade – Implications For Early Learning, edited by M. O. Martin and I. V. S. Mullis, 67–110. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Lynch School of Education, Boston College.
   Google Scholar
• NGSS (Next Generation Science Standards) Lead States. 2013. Next Generation Science Standards: For States, By States. Washington, DC: The National Academies Press.
   Google Scholar
• Nissan, S., F. DeVincenzi, and K. L. Tang. 1996. An Analysis of Factors Affecting the Difficulty of Dialogue Items in TOEFL Listening Comprehension. Princeton, NJ: ETS.
   Google Scholar
• Nitz, S., H. Prechtl, and C. Nerdel. 2014. “Survey of Classroom Use of Representations: Development, Field Test and Multilevel Analysis.” Learning Environments Research 17 (3): 401–422.10.1007/s10984-014-9166-x
   Google Scholar
• Nowak, K., A. Nehring, R. Tiemann, and A. Upmeier zu Belzen. 2013. “Assessing Students′ Abilities in Processes of Scientific Inquiry in Biology Using a Paper–Pencil-test.” Journal of Biological Education 47 (3): 182–188.10.1080/00219266.2013.822747
   Web of Science ®Google Scholar
• NRC (National Research Council). 1996. National Science Education Standards. Washington, DC: National Academic Press.
   Google Scholar
• Oh, Phil S., and Sung J. Oh. 2011. “What Teachers of Science Need to Know about Models: An Overview.” International Journal of Science Education 33 (8): 1109–1130. doi:10.1080/09500693.2010.502191.
   Web of Science ®Google Scholar
• Pollitt, A., N. Entwistle, C. Hutchinson, and C. de Luca. 1985. What Makes Exam Questions Difficult? Edinburgh: Scottish Academic Press.
   Google Scholar
• Prenzel, M., P. Häußler, J. Rost, and M. Senkbeil. 2002. “Der PISA-Naturwissenschaftstest: Lassen sich die Aufgabenschwierigkeiten vorhersagen?” [The PISA Science Test: Can Item Difficulties Be Predicted?]. Unterrichtswissenschaft 30: 120–135.
   Google Scholar
• QCA (Qualifications and Curriculum Authority). 2007. Science: Programme of Study for Key Stage 3 and Attainment Targets. London: QCA.
   Google Scholar
• R Core Team. 2015. R: A Language and Environment for Statistical Computing. Vienna, Austria: The R Foundation for Statistical Computing. https://www.r-project.org/
   Google Scholar
• Schnotz, W. 2005. “An Integrated Model of Text and Picture Comprehension.” In The Cambridge Handbook of Multimedia Learning, edited by R. E. Mayer, 49–70. Cambridge: Cambridge University Press.10.1017/CBO9780511816819
   Google Scholar
• Schwartz, R. S., N. G. Lederman, and B. A. Crawford. 2004. “Developing Views of Nature of Science in an Authentic Context: An Explicit Approach to Bridging the Gap Between Nature of Science and Scientific Inquiry.” Science Education 88: 610–645. doi:10.1002/sce.10128.
   Web of Science ®Google Scholar
• Spearritt, D. 1980. The Improvement of Measurement in Education and Psychology: Contributions of Latent Trait Theories. Melbourne: The Australian Council for Educational Research.
   Google Scholar
• Stiller, J., P. Straube, S. Hartmann, V. Nordmeier, and R. Tiemann. 2015. “Erkenntnisgewinnungskompetenz Chemie- und Physik-Lehramtsstudierender. Untersuchungen zu Domänenspezifität.” [Pre-service Chemistry and Physics Teachers’ Scientific Inquiry Competencies : Investigating Domain Specificity]. In Berlin-Brandenburger Beiträge zur Bildungsforschung 2015. Herausforderungen, Befunde und Perspektiven interdisziplinärer Bildungsforschung, edited by Jurik Stiller and Christin Laschke, 179–202. Frankfurt am Main: Peter Lang.10.3726/978-3-653-04961-9
   Google Scholar
• Taskin, V., S. Bernholt, and I. Parchmann. 2015. “An Inventory for Measuring Student Teachers’ Knowledge of Chemical Representations: Design, Validation, and Psychometric Analysis.” Chemistry Education Research and Practice 16: 460–477. doi: 10.1039/c4rp00214.
   Web of Science ®Google Scholar
• Upmeier zu Belzen, A., and D. Krüger. 2010. “Modellkompetenz im Biologieunterricht.” [Model Competence in Biology Teaching]. Zeitschrift für Didaktik der Naturwissenschaften 16: 59–75.
   Google Scholar
• Wellnitz, N., S. Hartmann, and J. Mayer. 2010. “Developing a Paper-and-Pencil-test to Assess Students’ Skills in Scientific Inquiry.” In Contemporary Science Education Research: Learning and Assessment, edited by G. Çakmaki and F. Taar, 289–294. Ankara: ESERA.
   Google Scholar
• Wong, T. F. 2003. “A Case Study of Test Design in an Engineering Programme Using Item Aanalysis.” INTI Journal 1 (3): 223–232.
   Web of Science ®Google Scholar
• Zlatkin-Troitschanskaia, O., R. J. Shavelson, and C. Kuhn. 2015. “The International State of Research on Measurement of Competency in Higher Education.” Studies in Higher Education 40 (3): 393–411. doi:10.1080/03075079.2015.1004241.
   Web of Science ®Google Scholar