A Psychometric Framework for the Evaluation of Instructional Sensitivity

References

• Adams, R. J., Wilson, M., & Wu, M. (1997). Multilevel item response models: An approach to errors in variables regression. Journal of Educational and Behavioral Statistics, 22, 47–76.
   Web of Science ®Google Scholar
• Baker, E. L. (1994). Making performance assessment work: The road ahead. Educational Leadership, 51, 58–62.
   Web of Science ®Google Scholar
• Burstein, L. (1989, March). Conceptual considerations in instructionally sensitive assessment. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA.
   Google Scholar
• Clauser, B. E., Nungester, R. J., & Swaminathan, H. (1996). Improving the matching for DIF analysis by conditioning on both test score and an educational background variable. Journal of Educational Measurement, 33, 453–464.
   Web of Science ®Google Scholar
• Cox, R. C., & Vargas, J. S. (1966, February). A comparison of item-selection techniques for norm referenced and criterion referenced tests. Paper presented at the Annual Conference of the National Council on Measurement in Education, Chicago, IL.
   Google Scholar
• Creemers, B. P. M., & Kyriakides, L. (2008). The dynamics of educational effectiveness: A contribution to policy, practice and theory in contemporary schools. London, England: Routledge.
   Google Scholar
• D'Agostino, J. V., Welsh, M. E., & Corson, N. M. (2007). Instructional sensitivity of a state's standards-based assessment. Educational Assessment, 12, 1–22.
   Google Scholar
• De Boeck, P., & Wilson, M. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York, NY: Springer.
   Google Scholar
• Fox, J.-P. (2010). Bayesian item response modeling: Theory and applications. New York, NY: Springer.
   Google Scholar
• Gelman, A. B., & Hill, J. (2006). Data analysis using regression and multilevel/hierarchical models. Cambridge, England: Cambridge University Press.
   Google Scholar
• Glaser, R. (1963). Instructional technology and the measurement of learning outcomes: Some questions. American Psychologist, 18, 519–521.
   Web of Science ®Google Scholar
• Haladyna, T. M. (2004). Developing and validating multiple-choice test items. Mahwah, NJ: Erlbaum.
   Google Scholar
• Hardy, I., Hertel, S., Kunter, M., Klieme, E., Warwas, J., Büttner, G., & Lühken, A. (2011). Adaptive Lerngelegenheiten in der Grundschule: Merkmale, methodisch-didaktische Schwerpunktsetzungen und erforderliche Lehrerkompetenzen [Adaptive learning environments in primary school]. Zeitschrift für Pädagogik, 57, 819–833.
   Google Scholar
• Hardy, I., Kleickmann, T., Koerber, S., Mayer, D., Möller, K., Pollmeier, J., … Schwippert, K. (2010). Die Modellierung naturwissenschaftlicher Kompetenz im Grundschulalter [Modeling science competence in primary school]. In E. Klieme, D. Leutner, & M. Kenk (Eds.), Kompetenzmodellierung. Zwischenbilanz des DFG-Schwerpunktprogramms und Perspektiven des Forschungsansatzes. 56. Beiheft der Zeitschrift für Pädagogik (pp. 115–124). Weinheim, Germany: Beltz.
   Google Scholar
• Hartig, J., Klieme, E., & Leutner, D. (Eds.). (2008). Assessment of competencies in educational contexts. Göttingen, Germany: Hogrefe & Huber.
   Google Scholar
• Hartig, J., & Kühnbach, O. (2006). Schätzung von Veränderung mit Plausible Values in mehrdimensionalen Rasch-Modellen [Estimating change using plausible values in multidimensional Rasch models]. In A. Ittel & H. Merkens (Eds.), Veränderungsmessung und Längsschnittstudien in der Erziehungswissenschaft (pp. 27–44). Wiesbaden, Germany: Verlag für Sozialwissenschaften.
   Google Scholar
• Holland, P. W., & Wainer, H. (Eds.). (1993). Differential item functioning: Theory and practice. Hillsdale, NJ: Erlbaum.
   Google Scholar
• Ing, M. (2008). Using instructional sensitivity and instructional opportunities to interpret students' mathematics performance. Journal of Educational Research & Policy Studies, 8, 23–43.
   Google Scholar
• Kamata, A. (2001). Item analysis by the hierarchical generalized linear model. Journal of Educational Measurement, 38, 79–93.
   Web of Science ®Google Scholar
• Kane, M. T. (2013). Validating the interpretations and uses of test scores. Journal of Educational Measurement, 50, 1–73.
   Web of Science ®Google Scholar
• Kleickmann, T., Hardy, I., Möller, K., Pollmeier, J., Tröbst, S., & Beinbrech, C. (2010). Die Modellierung naturwissenschaftlicher Kompetenz im Grundschulalter: Theoretische Konzeption und Testkonstruktion [Modeling science competence of primary school children]. Zeitschrift für Didaktik der Naturwissenschaften, 16, 263–281.
   Google Scholar
• Klieme, E., & Baumert, J. (2001). Identifying national cultures of mathematics education: Analysis of cognitive demands and differential item functioning in TIMSS. European Journal of Psychology of Education, 16, 383–400.
   Google Scholar
• Klieme, E., Pauli, C., & Reusser, K. (2009). The Pythagoras study: Investigating effects of teaching and learning in Swiss and German mathematics classrooms. In T. Janík & T. Seidel (Eds.), The power of video studies in investigating teaching and learning in the classroom (pp. 137–160). Münster, Germany: Waxmann.
   Google Scholar
• Lee, G., Brennan, R. L., & Frisbie, D. A. (2000). Incorporating the testlet concept in test score analyses. Educational Measurement: Issues and Practice, 19, 9–15.
   Google Scholar
• Li, M., Ruiz-Primo, M. A., & Wills, K. (2012, April). Comparing methods to evaluate the instructional sensitivity of items. Paper presented at the Annual Conference of the National Council on Measurement in Education, Vancouver, Canada.
   Google Scholar
• Linn, R. L., & Harnisch, D. L. (1981). Interactions between item content and group membership on achievement test items. Journal of Educational Measurement, 18, 109–118.
   Web of Science ®Google Scholar
• Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel modeling. Methodology: European Journal of Research Methods for the Behavioral and Social Sciences, 1, 86–92.
   Google Scholar
• Martin, M. O., Mullis, I. V. S., & Foy, P.  (with Olson, J.F., Erberber, E., Preuschoff, C., & Galia, J.). (2008). TIMSS 2007 International Science Report: Findings from IEA's Trends in International Mathematics and Science Study at the Fourth and Eighth Grades. Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.
   Google Scholar
• Meulders, M., & Xie, Y. (2004). Person-by-item predictors. In P. De Boeck & M. Wilson (Eds.), Explanatory item response models: A generalized linear and nonlinear approach (pp. 213–240). New York, NY: Springer.
   Google Scholar
• Muthén, B. O. (1989). Using item-specific instructional information in achievement modeling. Psychometrika, 54, 385–396.
   Web of Science ®Google Scholar
• Muthén, B. O., Kao, C.-F., & Burstein, L. (1991). Instructionally sensitive psychometrics: Application of a new IRT-based detection technique to mathematics achievement test items. Journal of Educational Measurement, 28, 1–22.
   Web of Science ®Google Scholar
• Naumann, A., Hochweber, J., & Hartig, J. (2014). Modeling instructional sensitivity using a longitudinal multilevel differential item functioning approach. Journal of Educational Measurement, 51, 381–399.
   Web of Science ®Google Scholar
• Niemi, D., Wang, J., Steinberg, D. H., Baker, E. L., & Wang, H. (2007). Instructional sensitivity of a complex language arts performance assessment. Educational Assessment, 12, 215–237.
   Google Scholar
• Pellegrino, J. W. (2002). Knowing what students know. Issues in Science & Technology, 19, 48–52.
   Web of Science ®Google Scholar
• Pianta, R. C., & Hamre, B. K. (2009). Conceptualization, measurement, and improvement of classroom processes: Standardized observation can leverage capacity. Educational Researcher, 38, 109–119.
   Web of Science ®Google Scholar
• Polikoff, M. S. (2010). Instructional sensitivity as a psychometric property of assessments. Educational Measurement: Issues and Practice, 29(4), 3–14.
   Google Scholar
• Popham, J. W. (2007). Instructional insensitivity of tests: Accountability's dire drawback. Phi Delta Kappan, 89, 146–155.
   Web of Science ®Google Scholar
• Raudenbush, S. W. (2008). The Brown legacy and the O'Connor challenge: Transforming schools in the images of children's potential. Educational Researcher, 38, 169–180.
   Web of Science ®Google Scholar
• Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods. Thousand Oaks, CA: Sage.
   Google Scholar
• Reckase, M. (2009). Multidimensional item response theory. New York, NY: Springer.
   Google Scholar
• Robitzsch, A. (2009). Methodische Herausforderungen bei der Kalibrierung von Leistungstests [Methodical challenges in the calibration of performance tests]. In D. Granzer, O. Köller, & A. Bremerich-Vos (Eds.), Bildungsstandards Deutsch und Mathematik (pp. 42–106). Weinheim, Germany: Beltz.
   Google Scholar
• Rowan, B., Correnti, C., & Miller, R. J. (2002). What large-scale, survey research tells us about teacher effects on student achievement: Insights from the Prospects study of elementary schools. Teachers College Record, 104, 1525–1567.
   Web of Science ®Google Scholar
• Ruiz-Primo, M. A., Shavelson, R. J., Hamilton, L., & Klein, S. (2002). On the evaluation of systemic science education reform: Searching for instructional sensitivity. Journal of Research in Science Teaching, 39, 369–393.
   Web of Science ®Google Scholar
• Schmidt, W. H., & Maier, A. (2009). Opportunity to learn. In G. Sykes, B. Schneider, & D. N. Plank (Eds.), Handbook of education policy research (pp. 541–559). New York, NY: Routledge.
   Google Scholar
• van den Noortgate, W., De Boeck, P., & Meulders, M. (2003). Cross-classification multilevel logistic models in psychometrics. Journal of Educational and Behavioral Statistics, 28, 369–386.
   Web of Science ®Google Scholar
• van der Linden, W. J. (1981). A latent trait look at pretest–posttest validation of criterion-referenced test items. Review of Educational Research, 51, 379–402.
   Web of Science ®Google Scholar
• Vieluf, S., Kaplan, D., Klieme, E., & Bayer, S. (2012). Teaching practices and pedagogical innovation: Evidence from TALIS. Paris, France: OECD Publishing.
   Google Scholar
• Vosniadou, S. (2007). The cognitive-situative divide and the problem of conceptual change. Educational Psychologist, 42, 55–66.
   Web of Science ®Google Scholar
• Zwick, R., & Ercikan, K. (1989). Analysis of differential item functioning in the NAEP history assessment. Journal of Educational Measurement, 26, 55–66.
   Web of Science ®Google Scholar