Development and Application of a Novel Rasch-based Methodology for Evaluating Multi-Tiered Assessment Instruments: Validation and utilization of an undergraduate diagnostic test of the water cycle

References

• Aguado, E., & Burt, J. E. (2004). Understanding weather and climate. Upper Saddle River, NJ: Pearson Education.
   Google Scholar
• American Association for the Advancement of Science (AAAS). (1990). Science for all Americans. New York, NY: Oxford University Press.
   Google Scholar
• American Meteorological Society. (2001). The global water cycle: An introduction. Boston, MA: Author.
   Google Scholar
• Arhens, C. D. (2009). Meteorology today: An introduction to weather, climate, and the environment. Belmont, CA: Brooks/Cole.
   Google Scholar
• Arslan, H. O., Cigdemoglu, C., & Moseley, C. (2012). A three-tier diagnostic test to assess pre-service teachers’ misconceptions about global warming, greenhouse effect, ozone layer depletion, and acid rain. International Journal of Science Education, 1, 1–20.
   Google Scholar
• Ausubel, D. P. (1968). Educational psychology: A cognitive view. New York: Holt, Rinehart, and Winston.
   Google Scholar
• Bar, V. (1989). Children's views about the water cycle. Science Education, 73, 481–500. doi: 10.1002/sce.3730730409
   Web of Science ®Google Scholar
• Bar, V., & Galili, I. (1994). Stages of children's views about evaporation. International Journal of Science Education, 16(2), 157–174. doi: 10.1080/0950069940160205
   Web of Science ®Google Scholar
• Bar, V., & Travis, A. (1991). Children's views concerning phase changes. Journal of Research in Science Teaching, 28, 363–382. doi: 10.1002/tea.3660280409
   Web of Science ®Google Scholar
• Bell, B. (1995). Interviewing: A technique for assessing science knowledge. In S. M. Glynn & R. Duit (Eds.), Learning science in the schools: Research reforming practice (pp. 347–364). Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• Ben-zvi-Assarf, O., & Orion, N. (2005). A study of junior high students’ perceptions of the water cycle. Journal of Geoscience Education, 53, 366–373.
   Google Scholar
• Bond, T. G., & Fox, C. M. (2007). Fundamental measurement in the human sciences (2nd ed.). Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• Boone, W. J., & Scantlebury, K. (2006). The role of Rasch analysis when conducting science education research utilizing multiple choice tests. Science Education, 90(2), 253–269. doi: 10.1002/sce.20106
   Web of Science ®Google Scholar
• Brody, M. J. (1993, April). Student understanding of water and water resources: A review of literature. Paper presented at the annual meeting of the American Educational Research Association, Atlanta, GA.
   Google Scholar
• Caleon, I. S., & Subramaniam, R. (2010a). Development and application of a three-tier diagnostic test to assess secondary students’ understanding of waves. International Journal of Science Education, 32, 939–961. doi:10.1080/09500690902890130
   Web of Science ®Google Scholar
• Caleon, I. S., & Subramaniam, R. (2010b). Do students know what they know and what they don't know? Using a four-tier diagnostic test to access the nature of students’ alternative conceptions. Research in Science Education, 40, 313–337. doi:10.1007/s11165-009-9122-4
   Web of Science ®Google Scholar
• Cardak, O. (2009). Science students’ misconceptions of the water cycle according to their drawings. Journal of Applied Sciences, 9, 865–873. doi: 10.3923/jas.2009.865.873
   Google Scholar
• Carr, M. (1996). Interviews about instances and interviews about events. In D. F. Treagust, R. Duit, & B. J. Fraser (Eds.), Improving teaching and learning in science and mathematics (pp. 44–53). New York: Teachers College Press.
   Google Scholar
• Cetin-Dindar, A., & Geban, O. (2011). Development of a three-tier test to assess high school students’ understanding of acids and bases. Procedia—Social and Behavioral Sciences, 15, 600–604. doi: 10.1016/j.sbspro.2011.03.147
   Google Scholar
• Chandrasegaran, A. L., Treagust, D. F., & Mocerino, M. (2007). The development of a two-tier multiple-choice diagnostic instrument for evaluating secondary school students’ ability to describe and explain chemical reactions using multiple levels of representation. Chemistry Education Research and Practice, 8, 293–307. doi: 10.1039/B7RP90006F
   Web of Science ®Google Scholar
• Covitt, B. A., Gunckel, K. L., & Anderson, C. W. (2009). Students’ developing understanding of water in environmental systems. The Journal of Environmental Education, 40(3), 37–51. doi: 10.3200/JOEE.40.3.37-51
   Web of Science ®Google Scholar
• D'Avanzo, C. (2003). Research on learning: Potential for improving college ecology teaching. Frontiers in Ecology and the Environment, 1(10), 533–540. doi: 10.2307/3868164
   Web of Science ®Google Scholar
• Driver, R., & Easley, J. (1978). Pupils and paradigms: A review of literature related to concept development in science. Studies in Science Education, 13, 105–122. doi: 10.1080/03057268608559933
   Google Scholar
• Driver, R., & Erickson, G. (1983). Theories-in-action: Some theoretical and empirical issues in the study of students’ conceptual frameworks in science. Studies in Science Education, 10, 37–60. doi: 10.1080/03057268308559904
   Google Scholar
• Duit, R. (2009). Bibliography—STCSE: Students’ and teachers’ conceptions and science education. Retrieved from: ReindersDuit [email protected]
   Google Scholar
• Galli, S., Chiesi, F., & Primi, C. (2008). The construction of a scale to measure mathematical ability in psychology students: An application of the Rasch Model. TPM (Testing Psicometria Metodologia), 15(1), 1–16.
   Google Scholar
• Gunckel, K. L., Covitt, B. A., Salinas, I., & Anderson, C. W. (2012). A learning progression for water in socio-ecological systems. Journal of Research in Science Teaching, 49(7), 843–868. doi: 10.1002/tea.21024
   Web of Science ®Google Scholar
• Hasan, S., Bagayoko, D., & Kelley, E. L. (1999). Misconceptions and the certainty of response index (CRI). Physics Education, 34(5), 294–299. doi: 10.1088/0031-9120/34/5/304
   Google Scholar
• Henriques, L. (2000, April). Children's misconceptions about weather: A review of the literature. Paper presented at the annual meeting for the National Association for Research in Science Teaching, New Orleans, LA.
   Google Scholar
• Lin, S. W. (2004). Development and application of a two-tier diagnostic test for high school students’ understanding of flowering plant growth and development. International Journal of Science and Mathematics Education, 2, 175–199. doi: 10.1007/s10763-004-6484-y
   Google Scholar
• Linacre, J. M. (1999). Relating Cronbach and Rasch reliabilities. Rasch Measurement Transactions, 13, 696.
   Google Scholar
• Linacre, J. M. (2002). What do infit and outfit, mean-square and standardized mean? Rasch Measurement Transactions, 16(2), 878.
   Google Scholar
• Linacre, J. M. (2009). Local independence and residual covariance: A study of Olympic figure skating ratings. Journal of Applied Measurement, 11, 157–169.
   Google Scholar
• Linacre, J. M. (2010). Winsteps (Version 3.70.0). Retrieved from www.winsteps.com
   Google Scholar
• Linacre, J. M., & Tennant, A. (2009). More about critical eigenvalue sizes (variances) in standardized-residual principal components analysis (PCA). Rasch Measurement Transactions, 23(3), 1228.
   Google Scholar
• Lord, F. M., & Novick, M. R. (1968). Statistical theories of mental test scores. Reading, MA: Addison-Wesley.
   Google Scholar
• Lutgens, F. K., & Tarbuck, E. J. (2010). The atmosphere: An introduction to meteorology. Upper Saddle River, NJ: Pearson Education.
   Google Scholar
• Masters, G. N. (1988). Item discrimination: When more is worse. Journal of Educational Measurement, 25(1), 15–29. doi: 10.1111/j.1745-3984.1988.tb00288.x
   Web of Science ®Google Scholar
• Modell, H., Michael, J., & Wenderoth, M. B. (2005). Helping the learner to learn: The role of uncovering misconceptions. The American Biology Teacher, 67(1), 20–26. doi: 10.1662/0002-7685(2005)067[0020:HTLTLT]2.0.CO;2
   Web of Science ®Google Scholar
• Morrell, P. D., & Schepia, A. (2009, January). Exploring preservice elementary teachers’ conceptualization of the water cycle. Paper presented at the annual conference of the Association for Science Teacher Education, Hartford, Connecticut.
   Google Scholar
• Munson, B. H. (1994). Ecological misconceptions. Journal of Environmental Education, 25(4), 30–35. doi: 10.1080/00958964.1994.9941962
   Google Scholar
• NGSS Lead States. (2013). Next generation science standards: For states, by states. Achieve, Inc. on behalf of the twenty-six states and partners that collaborated on the NGSS. Washington, DC: National Academy Press.
   Google Scholar
• Novak, J. D. (1995). Concept mapping: A strategy for organizing knowledge. In S. M. Glynn& R. Duit (Eds.), Learning science in the schools: Research reforming practice (pp. 229–245). Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• Novak, J. D. (2002). Concept mapping: A tool for improving science teaching and learning. In D. F. Treagust, R. Duit, & B. J. Fraser (Eds.), Improving teaching and learning in science and mathematics (pp. 32–43). New York: Teachers College Press.
   Google Scholar
• Odom, A. L. (1992). The development and validation of a two-tier diagnostic test measuring college biology students’ understanding of diffusion and osmosis (Unpublished doctoral dissertation). University of Missouri, Columbia, MO.
   Google Scholar
• Odom, A. L., & Barrow, L. H. (1995). Development and application of a two-tier diagnostic test measuring college biology students’ understanding of diffusion and osmosis after a course of instruction. Journal of Research in Science Teaching, 32, 45–61. doi: 10.1002/tea.3660320106
   Web of Science ®Google Scholar
• Odom, A. L., & Barrow, L. H. (2007). High school biology students’ knowledge and certainty about diffusion and osmosis concepts. School Science and Mathematics, 107(3), 94–101. doi: 10.1111/j.1949-8594.2007.tb17775.x
   Google Scholar
• Pesman, H., & Eryilmaz, A. (2010). Development of a three-tier test to assess misconceptions about simple electric circuits. Journal of Educational Research, 103, 208–222. doi: 10.1080/00220670903383002
   Web of Science ®Google Scholar
• Phillips, W. C. (1991). Earth science misconceptions. Science Teacher, 58(2), 21–23.
   Google Scholar
• Popper, K. (1987). Science: Conjectures and refutations. In J. A. Kourany (Ed.), Scientific knowledge: Basic issues in the philosophy of science (pp. 139–157). Belmont, CA: Wadsworth.
   Google Scholar
• Posner, G. J., & Gertzog, W. A. (1982). The clinical interview and the measurement of conceptual change. Science  Education, 66, 195–209. doi: 10.1002/sce.3730660206
   Google Scholar
• Project WET and Council for Environmental Education. (1995). The incredible journey (Project WET Curriculum and Activity Guide, pp. 161–165), Bozeman, MT: Water Conversation Council for Environmental Education. Retrieved from www.projectwet.org
   Google Scholar
• Raiche, G. (2005). Critical eigenvalue sizes in standardized residual principle components analysis. Rasch Measurement Transactions, 19(1), 1012.
   Google Scholar
• Romine, W. L., Barrow, L. H., & Folk, W. R. (2013). Exploring secondary students’ knowledge and misconceptions about influenza: Development, validation, and implementation of a multiple-choice influenza knowledge scale. International Journal of Science Education, 35(11), 1874–1901. doi: 10.1080/09500693.2013.778439
   Web of Science ®Google Scholar
• Ruiz-Primo, M. A. (2000). On the use of concept maps as an assessment tool in science: What we have learned so far. Revista Electronica de Investigacion Educativa, 2(1), 29–53.
   Google Scholar
• Russell, T., Harlen, W., & Watt, D. (1989). Children's ideas about evaporation. International Journal of Science Education, 11, 566–576. doi:10.1080/0950069890110508
   Web of Science ®Google Scholar
• Schaffer, D. L. (2013). The development and validation of a three-tier diagnostic test measuring pre- service elementary education and secondary science teachers’ understanding of the water cycle (Doctoral dissertation), University of Missouri—Columbia. Retrieved from https://mospace.umsystem.edy/xmlui/bitsteam/handle/10355/37809/research.pdf?sequence=2
   Google Scholar
• Schaffer, D. L., & Barrow, L. H. (2015). Alternative conceptions or lack of knowledge: Assessing pre-service teachers’ understanding of the water cycle. Journal of Science Teacher Education. Manuscript submitted for publication.
   Google Scholar
• Schaffer, D. L., Romine, W. L., & Barrow, L. H. (2015, April). Misconception or lack of knowledge: Using confidence to enhance measurement validity in a multi-tiered assessment. Paper to be presented at the 2015 annual meeting for the National Association for Research in Science Teaching, Chicago, IL.
   Google Scholar
• Shepardson, D. P., Wee, B., Priddy, M., Schelleberger, L., & Harbor, J. (2009). Water transformation and storage in the mountains and at the coast: Midwest students’ discon­nected conceptions of the water cycle. International Journal of Science Education, 31, 1447–1471. doi:10.1080/09500690802061709
   Web of Science ®Google Scholar
• Smith, III, J. P., di Sessa, A. A., & Roschelle, J. (1993). Misconceptions reconceived: A constructivist analysis of knowledge in transition. The Journal of the Learning Sciences, 3(2), 115–163. doi: 10.1207/s15327809jls0302_1
   Google Scholar
• Stamp, N., & Armstrong, M. (2005). Using “The Power of Story” to overcome ecological misconceptions and build sophisticated understanding. Bulletin of the Ecological Society of America, 86(3), 177–183. doi: 10.1890/0012-9623(2005)86[177:UTPOST]2.0.CO;2
   Google Scholar
• Taiwo, A. A. (1999). Perceptions of the water cycle among primary school children in Botswana. International Journal of Science Education, 21, 413–429. doi:10.1080/095006999290633
   Web of Science ®Google Scholar
• Treagust, D. (1986). Evaluating students’ misconceptions by means of diagnostic multiple choice items. Research in Science Education, 16, 199–207. doi: 10.1007/BF02356835
   Google Scholar
• Treagust, D. F. (1988). The development and use of diagnostic instruments to evaluate students’ misconceptions in science. International Journal of Science Education, 10, 159–169. doi: 10.1080/0950069880100204
   Web of Science ®Google Scholar
• Treagust, D. F. (1995). Diagnostic assessment of students’ science concepts. In S. Glynn & R. Duit (Eds.), Learning science in the schools: Research reforming practice (pp. 327–346). Mahwah, NJ: Lawrence Erlbaum.
   Google Scholar
• Tsai, C. C., & Chou, C. (2002). Diagnosing students’ alternative conceptions in science. Journal of Computer Assisted Learning, 18, 157–165. doi: 10.1046/j.0266-4909.2002.00223.x
   Web of Science ®Google Scholar
• Wang, J. R. (2004). Development and validation of a two-tier instrument to examine understanding of internal transport in plants and the human circulatory system. International Journal of Science and Mathematics Education, 2, 131–157. doi: 10.1007/s10763-004-9323-2
   Google Scholar
• Wandersee, J. H., Mintzes, J. J., & Novak, J. D. (1994). Research on alternative conceptions in science. In D. L. Gabel (Ed.), Handbook of research on science teaching and learning (pp. 177–210). New York: Macmillan.
   Google Scholar
• What Works Clearinghouse. (2014). WWC procedures and standards handbook (version 3.0). Princeton, NJ: US Department of Education, Institute of Education Sciences.
   Google Scholar
• Wright, B. D., & Linacre, J. M. (1994). Reasonable mean-square fit values. Rasch Measurement Transactions, 8(3), 370.
   Google Scholar
• Wright, B. D., & Stone, M. A. (1979). Best test design. Chicago, IL: Mesa Press.
   Google Scholar
• Zimmerman, D. W., & Williams, R. H. (2003). A new look at the influence of guessing on the reliability of multiple-choice tests. Applied Psychological Measurement, 27(5), 357–371. doi: 10.1177/0146621603254799
   Web of Science ®Google Scholar