From Log Files to Assessment Metrics: Measuring Students' Science Inquiry Skills Using Educational Data Mining

REFERENCES

• Anderson , J. 1993 . Rules of the mind , Hillsdale , NJ : Erlbaum .
   Google Scholar
• Anderson , J. R. , Boyle , C. F. , Corbett , A. T. and Lewis , M. W. 1990 . Cognitive modeling and intelligent tutoring . Artificial Intelligence , 42 : 7 – 49 .
   Web of Science ®Google Scholar
• Baker , R. S. , Corbett , A. T. , Roll , I. and Koedinger , K. R. 2008 . Developing a generalizable detector of when students game the system . User Modeling and User-Adapted Interaction , 18 ( 3 ) : 287 – 314 .
   Web of Science ®Google Scholar
• Baker , R. , Corbett , A. and Wagner , A. 2006 . “ Human classification of low-fidelity replays of student actions ” . In C. Heiner, R. Baker, & K. Yacef (Eds.), Proceedings of the Educational Data Mining Workshop at the 8th International Conference on Intelligent Tutoring Systems , 29 – 36 . Jhongli , , Taiwan : Springer .
   Google Scholar
• Baker , R. and de Carvalho , A. 2008 . “ Labeling student behavior faster and more precisely with text replays ” . In Proceedings of the 1st International Conference on Educational Data Mining Edited by: Baker , R. S. , Barnes , T. and Beck , J. E. 38 – 47 . Montreal , Canada.
   Google Scholar
• Baker , R. S. , Mitrovic , A. and Mathews , M. 2010 . “ Detecting gaming the system in constraint-based tutors ” . In Proceedings of the 18th Annual Conference on User Modeling, Adaptation, and Personalization, UMAP 2010. LNCS 6075 , Edited by: De Bra , P. , Kobsa , P. and Chin , D. 267 – 278 . New York , NY : Springer-Verlag .
   Google Scholar
• Baker , R. and Yacef , K. 2009 . The state of educational data mining in 2009: A review and future visions . Journal of Educational Data Mining , 1 ( 1 ) : 3 – 17 .
   Google Scholar
• Beck , J. E. and Mostow , J. 2008 . “ How who should practice: Using learning decomposition to evaluate the efficacy of different types of practice for different types of students ” . In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the 9th International Conference on Intelligent Tutoring Systems , 353 – 362 . Berlin , , Germany : Springer .
   Google Scholar
• Behrens, J. T. (2013, April). Harnessing the currents of the digital ocean. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
   Google Scholar
• Ben-David , A. 2008 . About the relationship between ROC curves and Cohen's kappa . Engineering Applications of Artificial Intelligence , 21 : 874 – 882 .
   Web of Science ®Google Scholar
• Black , P. 1999 . Testing: Friend or foe? Theory and practice of assessment and testing , New York , NY : Falmer Press .
   Google Scholar
• Brusilovsky , P. and Peylo , C. 2003 . Adaptive and intelligent Web-based educational systems . International Journal of Artificial Intelligence in Education , 13 ( 2–4 ) : 159 – 172 .
   Google Scholar
• Buckley , B. C. , Gobert , J. D. and Horwitz , P. 2006 . “ Using log files to track students' model-based inquiry ” . In Proceedings of the 7th International Conference on Learning Sciences, ICLS , 57 – 63 . Mahwah , NJ : Erlbaum .
   Google Scholar
• Buckley , B. , Gobert , J. , Horwitz , P. and O'Dwyer , L. 2010 . Looking inside the black box: Assessments and decision-making in biologica . International Journal of Learning Technology , 5 ( 2 ) : 166 – 190 .
   Google Scholar
• Bybee , R. , McCrae , B. and Laurie , R. 2009 . PISA 2006: An assessment of scientific literacy . Journal of Research in Science Teaching , 46 : 865 – 883 .
   Web of Science ®Google Scholar
• Cen, H., Koedinger, K. R., & Junker, B. (2008). Comparing two IRT models for conjunctive skills. In B. Woolf, E. Aimeur, R. Nkambou, & S. Lajoie (Eds.), Proceedings of the 9th International Conference of Intelligent Tutoring Systems (pp. 796–798). Berlin, Germany: Springer.
   Google Scholar
• Cetintas , S. , Si , L. , Xin , Y. and Hord , C. 2010 . Automatic detection of off-task behaviors in intelligent tutoring systems with machine learning techniques . IEEE Transactions on Learning Technologies , 3 ( 3 ) : 228 – 236 .
   Web of Science ®Google Scholar
• Chen , Z. and Klahr , D. 1999 . All other things being equal: Acquisition and transfer of the control of variables strategy . Child Development , 70 : 1098 – 1120 .
   PubMed Web of Science ®Google Scholar
• Chi , M. , VanLehn , K. and Litman , D. 2010 . “ Do micro-level tutorial decisions matter: Applying reinforcement learning to induce pedagogical tutorial tactics ” . In Intelligent tutoring systems: 10th International Conference, ITS 2010 , Edited by: Aleven , V. , Kay , J. and Mostow , J. 184 – 193 . Heidelberg , Germany : Springer .
   Google Scholar
• Clarke-Midura , J. , Code , J. , Zap , N. and Dede , C. 2012 . “ Assessing science inquiry in the classroom: A case study of the virtual assessment project ” . In Cases on inquiry through instructional technology in math and science: Systemic approaches (pp. 138–164) Edited by: Lennex , L. and Nettleton , K. New York , NY: IGI.
   Google Scholar
• Clarke-Midura , J. , Dede , C. and Norton , J. 2011 . The road ahead for state assessments , Cambridge , MA : Rennie Center for Education Research & Policy .
   Google Scholar
• Corbett, A., & Anderson, J. (1995). Knowledge-tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4, 253–278.
   Google Scholar
• Corbett, A., Kaufmann, L., MacLaren, B., Wagner, A., & Jones, E. (2010). A cognitive tutor for genetics problem solving: Learning gains and student modeling. Journal of Educational Computing Research, 42, 219–239.
   Google Scholar
• DeBoer , G. 1991 . A history of ideas in science education: Implications for practice , New York , NY : Teachers College Press .
   Google Scholar
• DeBoer , G. , Abell , C. , Gogos , A. , Michiels , A. , Regan , T. , Wilson , P. , Douglas , , R. and Stearns , C. 2008 . “ Assessment linked to science learning goals: Probing student thinking through assessment. Project 2061. American Association for the Advancement of Science ” . In Assessing science learning: Perspectives from research and practice , Edited by: Coffey , J. 231 – 252 . Arlington , VA : NSTA Press .
   Google Scholar
• de Jong , T. and van Joolingen , W. R. 1998 . Scientific discovery learning with computer simulations of conceptual domains . Review of Educational Research , 68 ( 2 ) : 179 – 201 .
   Web of Science ®Google Scholar
• Desmarais , M. C. , Meshkinfam , P. and Gagnon , M. 2006 . Learned student models with item to item knowledge structures . User Modeling and User-Adapted Interaction , 16 : 403 – 434 .
   Web of Science ®Google Scholar
• Dewey , J. 1910 . January 28). Science as subject-matter and as method . Science , 31 : 121 – 127 .
   PubMedGoogle Scholar
• Dewey, J. (1916). Democracy and education: An introduction to the philosophy of education. New York, NY: MacMillan.
   Google Scholar
• DiCerbo , K. E. and Behrens , J. T. 2012 . “ Implications of the digital ocean on current and future assessment ” . In Computers and their impact on state assessment: Recent history and predictions for the future (pp. 273–306) , Edited by: Lissitz , R. Charlotte , NC : Information Age .
   Google Scholar
• Duschl , R. A. 2003 . “ Assessment of inquiry ” . In Everyday assessment in the science classroom (pp. 41–59) , Edited by: Atkin , J. M. and Coffey , J. E. Arlington , VA : NSTA Press .
   Google Scholar
• Efron , B. and Gong , G. 1983 . A leisurely look at the bootstrap, the jackknife, and cross-validation . The American Statistician , 37 ( 1 ) : 36 – 48 .
   Web of Science ®Google Scholar
• Ericsson , K. A. and Simon , H. 1980 . Verbal reports as data . Psychological Review , 87 : 215 – 251 .
   Web of Science ®Google Scholar
• Fadel , C. , Honey , M. and Pasnick , S. 2007 . Assessment in the age of innovation . Education Week , 26 ( 38 ) : 34 – 40 .
   Google Scholar
• Fan , J. , Upadhye , S. and Worster , A. 2006 . Understanding receiver operating characteristic (ROC) curves . Canadian Journal of Emergency Medicine , 8 ( 1 ) : 19 – 20 .
   Google Scholar
• Ford , M. J. 2012 . A dialogic account of sense-making in scientific argumentation and reasoning . Cognition and Instruction , 30 ( 3 ) : 207 – 245 .
   Web of Science ®Google Scholar
• Fu , A. C. , Raizen , S. and Shavelson , R. J. 2009 . “ December 18). The nation's report card: A vision of large-scale science assessment ” . In Science Vol. 326 , 1637 – 1638 .
   Google Scholar
• Gertner, A. S., & VanLehn, K. (2000). Andes: A coached problem solving environment for physics. In G. Gauthier, C. Frasson, & K. VanLehn (Eds.), Proceedings of the 5th International Conference on Intelligent Tutoring Systems, LNCS 1839 (pp. 133–142). Montreal, Canada: Springer-Verlag.
   Google Scholar
• Glaser , R. , Schauble , L. , Raghavan , K. and Zeitz , C. 1992 . “ Scientific reasoning across different domains ” . In Computer-based learning environments and problem-solving , Edited by: DeCorte , E. , Linn , M. , Mandl , H. and Verschaffel , L. 345 – 371 . Heidelberg , Germany : Springer-Verlag .
   Google Scholar
• Gobert , J. 2005 . “ Leveraging technology and cognitive theory on visualization to promote students' science learning and literacy ” . In Visualization in science education , Edited by: Gilbert , J. 73 – 90 . Dordrecht : The Netherlands: Springer-Verlag .
   Google Scholar
• Gobert , J. in press . “ Microworlds ” . In Encyclopedia of science education , Edited by: Gunstone , R. Heidelberg , Germany : Springer-Verlag .
   Google Scholar
• Gobert , J. and Koedinger , K. September 2011 . “ Presentation at the Society for Research Effectiveness ” . In Using model-tracing to conduct performance assessment of students' inquiry skills within a microworld September , Washington , DC.
   Google Scholar
• Gobert , J. , Sao Pedro , M. , Baker , R. S. , Toto , E. and Montalvo , O. 2012 . Leveraging educational data mining for real time performance assessment of scientific inquiry skills within microworlds . Journal of Educational Data Mining , 4 ( 1 ) : 153 – 185 .
   Google Scholar
• Gobert, J., Sao Pedro, M., Toto, E., Montalvo, O., & Baker, R. (2011, April). Science ASSISTments: Assessing and scaffolding students' inquiry skills in real time. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA.
   Google Scholar
• Gobert, J. D., & Buckley, B. C. (2000). Introduction to model-based teaching and learning in science education. International Journal of Science Education, 22, 891–894.
   Google Scholar
• Haertel , G. , Lash , A. , Javitz , H. and Quellmalz , E. April 2006 . “ Paper presented at the annual meeting of the American Educational Research Association ” . In An instructional sensitivity study of science inquiry items from three large-scale science examinations April , San Francisco , CA.
   Google Scholar
• Hambleton, R. K., & Jones, R. W. (1993). Comparison of classical test theory and item response theory and their applications to test development. Educational Measurement: Issues and Practice, 12(3), 38–47.
   Google Scholar
• Hanley , J. and McNeil , B. 1982 . The meaning and use of the area under a receiver operating characteristic (ROC) curve . Radiology , 143 : 29 – 36 .
   PubMed Web of Science ®Google Scholar
• Harrison , A. M. and Schunn , C. D. 2004 . “ The transfer of logically general scientific reasoning skills ” . In Proceedings of the 26th Annual Conference of the Cognitive Science Society , Edited by: Forbus , K. , Gentner , D. and Regier , T. 541 – 546 . Mahwah , NJ : Erlbaum .
   Google Scholar
• Jacobson , M. J. and Wilensky , U. 2006 . Complex systems in education: Scientific and educational importance and implications for the learning sciences . Journal of the Learning Sciences , 15 ( 1 ) : 11 – 34 .
   Web of Science ®Google Scholar
• Kasurinen, J., & Nikula, U. (2009). Estimating programming knowledge with Bayesian knowledge tracing. In Proceedings of the 14th Annual ACM SIGCSE Conference on Innovation and Technology in Computer Science Education (pp. 313–317). New York, NY: ACM Press.
   Google Scholar
• Kent , J. T. 1983 . Information gain and a general measure of correlation . Biometrika , 70 ( 1 ) : 163 – 173 .
   Web of Science ®Google Scholar
• Ketelhut , D. J. and Dede , C. April 2006 . “ Paper presented at the National Association for Research in Science Teaching ” . In Assessing inquiry learning April , San Francisco , CA.
   Google Scholar
• Ketelhut , D. , Nelson , B. , Sil , A. and Yates , A. April/May 2013 . “ Presentation at the annual meeting of the American Educational Research Association ” . In Discovering what students know through data mining their problem-solving actions within the immersive virtual environment, SAVE Science April/May , San Francisco , CA.
   Google Scholar
• Klahr , D. and Nigam , M. 2004 . The equivalence of learning paths in early science instruction: Effects of direct instruction and discovery learning . Psychological Science , 15 : 661 – 667 .
   PubMed Web of Science ®Google Scholar
• Koedinger, K., & Corbett, A. (2006). Cognitive tutors: Technology bringing learning sciences to the classroom. In R. Sawyer (Ed.), The Cambridge handbook of the learning sciences (pp. 61–77). New York, NY: Cambridge University Press.
   Google Scholar
• Koedinger , K. R. and MacLaren , B. A. 2002 . Developing a pedagogical domain theory of early algebra problem solving , CMU-HCII Tech. Rep. No. 02-100 . Pittsburgh, PA: Carnegie Mellon University.
   Google Scholar
• Koedinger , K. R. , Suthers , D. D. and Forbus , K. D. 1999 . Component-based construction of a science learning space . International Journal of Artificial Intelligence in Education , 10 : 292 – 313 .
   Google Scholar
• Koomen , M. April 2006 . “ Paper presented at the annual meeting of the American Educational Research Association ” . In The development and implementation of a computer-based assessment of science literacy in PISA 2006 April , San Francisco , CA.
   Google Scholar
• Kuhn , D. 2005a . Education for thinking , Cambridge , MA : Harvard University Press .
   Google Scholar
• Kuhn , D. 2005b . What needs to be mastered in mastery of scientific method? . Psychological Science , 16 : 873 – 874 .
   PubMed Web of Science ®Google Scholar
• Kuhn , D. , Black , J. , Keselman , A. and Kaplan , D. 2000 . The development of cognitive skills to support inquiry learning . Cognition and Instruction , 18 : 495 – 523 .
   Web of Science ®Google Scholar
• Kuhn , D. and Pease , M. 2008 . What needs to develop in the development of inquiry skills? . Cognition and Instruction , 46 : 512 – 559 .
   Google Scholar
• Kuhn , D. , Schauble , L. and Garcia-Mila , M. 1992 . Cross-domain development of scientific reasoning . Cognition and Instruction , 9 : 285 – 327 .
   Web of Science ®Google Scholar
• Lee , D. M. C. , Rodrigo , M. M. T. , Baker , R. S. J. d. , Sugay , J. O. and Coronel , A. 2011 . “ Exploring the relationship between novice programmer confusion and achievement ” . In Proceedings of the 4th Bi-Annual International Conference on Affective Computing and Intelligent Interaction, Part 1 (pp. 175–184). Berlin, Germany: Springer In S. K. D'Mello, A. Graesser, B. Schuller, & J. Martin (Eds.),
   Google Scholar
• Leeman-Munk, S., Wiebe, E., & Lester, J. (2013, April). Mining student science argumentation text to inform an intelligent tutoring system. Paper presented at the annual meeting of the American Educational Research Association, San Francisco, CA.
   Google Scholar
• Linn, R. L. (1994). Performance assessment: Policy promises and technical measurement standards. Educational Researcher, 23(9), 4–14.
   Google Scholar
• Linn, R. L. (2000). Assessments and accountability. Educational Researcher, 29(2), 4–16.
   Google Scholar
• Linn, R. L., Baker, E. L., & Dunbar, S. B. (1991). Complex, performance-based assessment: Expectations and validation criteria. Educational Researcher, 20(8), 15–21.
   Google Scholar
• Lynch , C. , Ashley , K. , Pinkwart , N. and Aleven , V. 2009 . Concepts, structures, and goals: Redefining ill-definedness . International Journal of Artificial Intelligence in Education , 19 ( 3 ) : 253 – 266 .
   Google Scholar
• Massachusetts Department of Education. 2006 . “ science and technology/engineering curriculum framework ” . In Massachusetts Malden , MA Author
   Google Scholar
• McElhaney , K. and Linn , M. 2008 . “ Impacts of students' experimentation using a dynamic visualization on their understanding of motion ” . In Proceedings of the 8th International Conference of the Learning Sciences , Vol. 2 , 51 – 58 . Utrecht, The Netherlands: International Society of the Learning Sciences . In P. A. Kirschner, J. J. G. van Merriënboer, & T. de Jong (Eds.)Vol.pp.
   Google Scholar
• McElhaney , K. and Linn , M. 2010 . “ Helping students make controlled experiments more informative ” . In Learning in the disciplines: Proceedings of the 9th International Conference of the Learning Sciences (ICLS 2010): Vol. 1. Full papers , Edited by: Gomez , K. , Lyons , L. and Radinsky , J. 786 – 793 . Chicago , IL : International Society of the Learning Sciences .
   Google Scholar
• Mierswa , I. , Wurst , M. , Klinkenberg , R. , Scholz , M. and Euler , T. 2006 . “ YALE: Rapid prototyping for complex data mining tasks ” . In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 935 – 940 . Philadelphia , PA : ACM Press .
   Google Scholar
• Mislevy , R. J. , Behrens , J. T. , DiCerbo , K. and Levy , R. 2012 . Design and discovery in educational assessment: Evidence centered design, psychometrics, and data mining . Journal of Educational Data Mining , 4 : 11 – 48 .
   Google Scholar
• Mislevy, R., Hamel, L., Fried, R., G., Gaffney, T., Haertel, G., Hafter, A., … Wenk, A. (2003). Design patterns for assessing science inquiry (PADI Technical Report 1). Menlo Park, CA: SRI International.
   Google Scholar
• Montalvo , O. , Baker , R. S. , Sao Pedro , M. A. , Nakama , A. and Gobert , J. D. 2010 . “ Identifying students' inquiry planning using machine learning ” . In Proceedings of the 3rd International Conference on Educational Data Mining 141 – 150 . In R. Baker, A. Merceron, & P. Pavlik (Eds.)pp.Pittsburgh, PA: EDM.
   Google Scholar
• National Assessment Governing Board. (2011). Science framework for the 2011 National Assessment of Educational Progress. Washington, DC: U.S. Department of Education.
   Google Scholar
• National Research Council. 1996 . National science education standards , Washington , DC : National Academies Press .
   Google Scholar
• National Research Council. 2000 . Inquiry and the national science education standards: A guide for teaching and learning , Washington , DC : National Academies Press .
   Google Scholar
• National Research Council. 2011 . A framework for K-12 science education: Practices, crosscutting concepts, and core ideas , Washington , DC : National Academies Press .
   Google Scholar
• Njoo , M. and de Jong , T. 1993 . Exploratory learning with a computer simulations for control theory: Learning processes and instructional support . Journal of Research in Science Teaching , 30 : 821 – 844 .
   Web of Science ®Google Scholar
• Organisation for Economic Co-operation and Development. (2010). PISA computer-based assessment of student skills in science. Retrieved from http://browse.oecdbookshop.org/oecd/pdfs/free/9810041E.pdf
   Google Scholar
• Papert, S. (1980). Computer-based microworlds as incubators for powerful ideas. In R. Taylor (Ed.), The computer in the school: Tutor, tool, tutee (pp. 203–201). New York, NY: Teachers College Press.
   Google Scholar
• Pardos , Z. , Baker , R. , Gowda , S. and Heffernan , N. 2011 . The sum is greater than the parts: Ensembling models of student knowledge in educational software . SIGKDD Explorations , 13 ( 2 ) : 37 – 44 .
   Google Scholar
• Pavlik , P. , Cen , H. and Koedinger , J. 2009 . “ Performance factors analysis—A new alternative to knowledge tracing ” . In Proceedings of the 14th International Conference on Artificial Intelligence in Education , 531 – 540 . Brighton , England : IOS Press . In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.),pp.
   Google Scholar
• Pellegrino , J. , Chudowsky , N. and Glaser , R. 2001 . Knowing what students know: The science and design of educational assessment , Washington , DC : National Academies Press .
   Google Scholar
• Quellmalz , E. , DeBarger , A. , Haertel , G. and Kreikemeier , P. 2005 . Validities of science inquiry assessments: Final report , Menlo Park , CA : SRI International .
   Google Scholar
• Quellmalz , E. S. , DeBarger , A. H. , Haertel , G. , Schank , P. , Buckley , B. , Gobert , J. and Ayala , C. 2008 . “ Exploring the role of technology-based simulations in science assessment: The Calipers Project ” . In Science assessment: Research and practical approaches , Washington , DC : NSTA . …In
   Google Scholar
• Quellmalz , E. and Haertel , G. 2004 . Technology supports for state science assessment systems , Washington , DC : National Research Council .
   Google Scholar
• Quellmalz , E. , Kreikemeier , P. , DeBarger , A. H. and Haertel , G. April 2007 . “ Paper presented at the annual meeting of the American Educational Research Association ” . In A study of the alignment of the NAEP, TIMSS, and new standards science assessments with the inquiry abilities in the national science education standards April , Chicago , IL.
   Google Scholar
• Quellmalz , E. S. and Pellegrino , J. W. 2009 . January 2). Technology and testing . Science , 323 : 75 – 79 .
   PubMed Web of Science ®Google Scholar
• Quellmalz, E. S., Silberglitt, M. D., & Timms, M. J. (2011). How can simulations be components of balanced state science assessment systems (Policy Brief). San Francisco, CA: WestEd. Retrieved from http://www.wested.org/cs/we/view/rs/1198
   Google Scholar
• Quellmalz , E. S. , Timms , M. J. , Buckley , B. C. , Davenport , J. , Loveland , M. and Silberglitt , M. D. 2012 . “ 21st century dynamic assessment ” . In Technology-based assessments for 21st century skills: Theoretical and practical implications from modern research , Edited by: Mayrath , M. , Clarke-Midura , J. and Robinson , D. H. Charlotte , NC : Information Age .
   Google Scholar
• Quellmalz , E. , Timms , M. and Schneider , S. 2009 . Assessment of student learning in science simulations and games , Washington , DC : National Research Council .
   Google Scholar
• Quellmalz , E. S. , Timms , M. J. , Silberglitt , M. D. and Buckley , B. C. 2012 . Science assessments for all: Integrating science simulations into balanced state science assessment systems . Journal of Research in Science Teaching , 49 : 363 – 393 .
   Web of Science ®Google Scholar
• Quinlan , J. R. 1993 . C4.5: Programs for machine learning , San Francisco , CA : Morgan Kaufmann .
   Google Scholar
• Razzaq, L., Feng, M., Nuzzo-Jones, G., Heffernan, N., Koedinger, K. R., Junker, B., … Rasmussen, K. (2005). The Assistment Project: Blending assessment and assisting. In G. M. C. K. Looi (Ed.), Proceedings of the 12th International Conference on Artificial Intelligence in Education (pp. 555–562). Amsterdam, The Netherlands: IOS Press.
   Google Scholar
• Reimann , P. 1991 . Detecting functional relations in a computerized discovery environment . Learning and Instruction , 1 ( 1 ) : 45 – 65 .
   Google Scholar
• Romero , C. and Ventura , S. 2010 . Educational data mining: A review of the state-of-the-art . IEEE Transactions on Systems, Man, and Cybernetics-Part C: Applications and Reviews , 40 : 601 – 618 .
   Web of Science ®Google Scholar
• Rowe, J., & Lester, J. (2010). Modeling user knowledge with dynamic Bayesian networks in interactive narrative environments. In C. G. Youngblood & V. Bulitko (Eds.), Proceedings of the 6th Annual AI and Interactive Digital Entertainment Conference (pp. 57–62). Palo Alto, CA: AAAI Press.
   Google Scholar
• Rupp, A. A., Gushta, M., Mislevy, R. J.Shaffer, D. W.2010Evidence-centered design of epistemic games: Measurement principles for complex learning environments. Journal of Technology, Learning, and Assessment, 8(4). Retrieved from http://escholarship.bc.edu/jtla/vol8/4 (http://escholarship.bc.edu/jtla/vol8/4)
   Google Scholar
• Rutherford, F. J., & Ahlgren, A. (1989). Science for all Americans: A Project 2061 report on literacy goals in science, mathematics, and technology. Washington, DC: American Association for the Advancement of Science.
   Google Scholar
• Sao Pedro , M. A. 2013 . Real-time assessment, prediction, and scaffolding of middle school students' data collection skills within physical science simulations , Worcester : MA . Unpublished doctoral dissertationWorcester Polytechnic Institute
   Google Scholar
• Sao Pedro, M., Baker, R., & Gobert, J. (2013a). Incorporating scaffolding and tutor context into Bayesian knowledge tracing to predict inquiry skill acquisition. In S. K. D'Mello, R. A. Calvo, & A. Olney (Eds.), Proceedings of the 6th International Conference on Educational Data Mining (pp. 185–192). Memphis, TN.
   Google Scholar
• Sao Pedro , M. , Baker , R. and Gobert , J. 2013b . “ What different kinds of stratification can reveal about the generalizability of data-mined skill assessment models ” . In Proceedings of the 3rd Conference on Learning Analytics and Knowledge (pp. 190–194) Leuven, Belgium: ACM Press.
   Google Scholar
• Sao Pedro , M. , Baker , R. , Gobert , J. , Montalvo , O. and Nakama , A. 2013 . Leveraging machine-learned detectors of systematic inquiry behavior to estimate and predict transfer of inquiry skill . User Modeling and User-Adapted Interaction , 23 : 1 – 39 .
   Web of Science ®Google Scholar
• Sao Pedro , M. A. , Baker , R. S. , Montalvo , O. , Nakama , A. and Gobert , J. D. 2010 . “ (pp. 181–190) ” . In Proceedings of the 3rd International Conference on Educational Data Mining Edited by: Baker , R. , Merceron , A. and Pavlik , P. Pittsburgh , PA. Using text replay tagging to produce detectors of systematic experimentation behavior patterns
   Google Scholar
• Sao Pedro , M. , Gobert , J. and Baker , R. April 2012 . “ Paper presented at the annual meeting of the American Educational Research Association ” . In Assessing the learning and transfer of data collection inquiry skills using educational data mining on students' log files April , Vancouver, British Columbia, Canada.
   Google Scholar
• Sao Pedro , M. , Gobert , J. , Heffernan , N. and Beck , J. 2009 . “ Comparing pedagogical approaches for teaching the control of variables strategy ” . In Proceedings of the 31st Annual Meeting of the Cognitive Science Society , Eds , Edited by: Taatgen , N. A. and vanRijn , H. 1294 – 1299 . Austin , TX : Cognitive Science Society .
   Google Scholar
• Sao Pedro , M. A. , Gobert , J. D. and Raziuddin , J. 2010 . “ Comparing pedagogical approaches for the acquisition and long-term robustness of the control of variables strategy ” . In Learning in the disciplines: Proceedings of the 9th International Conference of the Learning Sciences, ICLS 2010: Vol. 1. Full papers , Edited by: Gomez , K. , Lyons , L. and Radinsky , J. 1024 – 1031 . Chicago , IL : International Society of the Learning Sciences .
   Google Scholar
• Scalise , K. , Timms , M. , Clark , L. and Moorjani , A. April 2009 . Student learning in science simulations: What makes a difference? , April , San Diego : CA . Paper presented at the annual meeting of the American Educational Research Association
   Google Scholar
• Schauble , L. , Glaser , R. , Duschl , R. , Schulze , S. and John , J. 1995 . Students' understanding of the objectives and procedures of experimentation in the science classroom . Journal of the Learning Sciences , 4 ( 2 ) : 131 – 166 .
   Web of Science ®Google Scholar
• Schauble , L. , Glaser , R. , Raghavan , K. and Reiner , M. 1991 . Causal models and experimentation strategies in scientific reasoning . Journal of the Learning Sciences , 1 ( 2 ) : 201 – 238 .
   Google Scholar
• Schauble , L. , Klopfer , L. E. and Raghavan , K. 1991 . Students' transition from an engineering model to a science model of experimentation . Journal of Research in Science Teaching , 28 : 859 – 882 .
   Web of Science ®Google Scholar
• Schunn , C. D. and Anderson , J. R. 1998 . “ In J. R. Anderson, The atomic components of thought ” . In Scientific discovery , 385 – 428 . Mahwah , NJ : Erlbaum .
   Google Scholar
• Schunn , C. D. and Anderson , J. R. 1999 . The generality/specificity of expertise in scientific reasoning . Cognitive Science , 23 ( 3 ) : 337 – 370 .
   Web of Science ®Google Scholar
• Shavelson , R. , Wiley , E. W. and Ruiz-Primo , M. 1999 . Note on sources of sampling variability in science performance assessments . Journal of Educational Measurement , 36 ( 1 ) : 61 – 71 .
   Web of Science ®Google Scholar
• Shute , V. and Glaser , R. 1990 . A large-scale evaluation of an intelligent discovery world: Smithtown . Interactive Learning Environments , 1 : 55 – 71 .
   Google Scholar
• Siler , S. , Klahr , D. , Magaro , C. , Willows , K. and Mowery , D. 2010 . “ Predictors of transfer of experimental design skills in elementary and middle school children ” . In Proceedings of the 10th International Conference on Intelligent Tutoring Systems, Part II, LNCS 6095 Edited by: Aleven , V. , Kay , J. and Mostow , J. 198 – 208 . Pittsburgh , PA: Springer
   Google Scholar
• Slotta, J. D., & Linn, M. C. (2009). WISE science: Web-based inquiry in the classroom. New York, NY: Teachers College Press.
   Google Scholar
• Stecher, B. M., & Klein, S. P. (1997). The cost of science performance assessments in large-scale testing programs. Educational Evaluation and Policy Analysis, 19(1), 1–14.
   Google Scholar
• Strand-Cary , M. and Klahr , D. 2008 . Developing elementary science skills: Instructional effectiveness and path independence . Cognitive Development , 23 : 488 – 511 .
   Web of Science ®Google Scholar
• Timms, M., Clements, D. H., Gobert, J., Ketelhut, D. J., Lester, J., Reese, D. D.Wiebe, E.2012New measurement paradigms. Retrieved from http://works.bepress.com/michael_timms/36 (http://works.bepress.com/michael_timms/36)
   Google Scholar
• Tsirgi , J. 1980 . Sensible reasoning: A hypothesis about hypotheses . Child Development , 51 : 1 – 10 .
   Web of Science ®Google Scholar
• van Joolingen , W. and de Jong , T. 1991 . Supporting hypothesis generation by learners exploring an interactive computer simulation . Instructional Science , 20 : 389 – 404 .
   Web of Science ®Google Scholar
• van Joolingen, W., & de Jong, T. (1993). Exploring a domain through a computer simulation: Traversing variable and relation space with the help of a hypothesis scratchpad. In D. Towne, T. de Jong, & H. Spada (Eds.), Simulation-based experiential learning (pp. 191–206). Berlin, Germany: Springer-Verlag.
   Google Scholar
• Vapnik V. (1995). The nature of statistical learning theory. New York, NY: Springer-Verlag.
   Google Scholar
• Voss , J. F. 2006 . “ Toulmin's model and the solving of ill-structured problems ” . In Arguing on the Toulmin model: New essays in argument analysis and evaluation , Edited by: Hitchcock , D. and Verheij , B. 303 – 311 . Berlin , Germany : Springer .
   Google Scholar
• Walonoski , J. and Heffernan , N. 2006 . “ Detection and analysis of off-task gaming behavior in intelligent tutoring systems ” . In Proceedings of the 8th International Conference on Intelligent Tutoring Systems, ITS 2006. LNCS 4053 , Edited by: Ikeda , M. , Ashlay , K. and Chan , T.-W. 382 – 391 . Johngli , Taiwan : Springer-Verlag .
   Google Scholar
• Williamson , D. , Mislevy , R. and Bejar , I. 2006 . Automated scoring of complex tasks in computer-based testing , Mahwah , NJ : Erlbaum .
   Google Scholar
• Wilson, M., & Bertenthal, M. (Eds.). (2006). Systems for state science assessment. Washington, DC: National Academies Press.
   Google Scholar
• Witten , I. H. and Frank , E. 2005 . Data mining: Practical machine learning tools and techniques with Java implementations , San Francisco , CA : Morgan Kaufmann .
   Google Scholar
• Yanamandra , K. , Grudin , M. A. , Casaite , V. , Meskys , R. , Forsgren , L. and Morozova-Roche , L. 2011 . a-Synuclein reactive antibodies as diagnostic markers in blood sera of Parkinson's disease patients . PlosONE , 6(4), e18513. doi:10.1371/journal.pone.0018513
   Google Scholar
• Yoon , S. 2008 . Using memes and memetic processes to explain social and conceptual influences on student understanding about complex socio-scientific issues . Journal of Research in Science Teaching , 45 : 900 – 921 .
   Web of Science ®Google Scholar
• Zimmerman, C. (2007). The development of scientific thinking skills in elementary and middle school. Developmental Review, 27, 172–223.
   Google Scholar
• Zohar , A. and David , B. A. 2008 . Explicit teaching of meta-strategic knowledge in authentic classroom situations . Metacognition and Learning , 3 ( 1 ) : 59 – 82 .
   Web of Science ®Google Scholar