Performance-based assessment of graduate student research skills: timing, trajectory, and potential thresholds

References

• Caicedo , J. M. , Pierce , C. E. , Flora , J. , Timmerman , B. , Nichols , A. P. , Graf , W. and Ray , T. Forthcoming . “ Instructional environment to stimulate critical thought of freshmen civil engineering students ” . In Advances in Engineering Education
   Google Scholar
• Carnegie Initiative on the Doctorate . 2001 . Overview of doctoral educational studies and reports: 1990–present , Stanford , CA : Carnegie Foundation for the Advancement of Teaching .
   Google Scholar
• Davis , G. and Fiske , P. 2001 . The 2000 national doctoral program survey University of Missouri-Columbia: National Association of Graduate-Professional Students
   Google Scholar
• Dehaan , R. L. 2005 . The impending revolution in undergraduate science education . Journal of Science Education and Technology , 14 ( 2 ) : 253 – 70 . (doi:10.1007/s10956-005-4425-3)
   Google Scholar
• Dunbar , K. 2000 . How scientists think in the real world: Implications for science education . Journal of Applied Developmental Psychology , 21 ( 1 ) : 49 – 58 . (doi:10.1016/S0193-3973(99)00050-7)
   Web of Science ®Google Scholar
• Ericsson , K. A. and Charness , N. 1994 . Expert performance: Its structure and acquisition . American Psychologist , 49 ( 8 ) : 725 – 47 . (doi:10.1037/0003-066X.49.8.725)
   Web of Science ®Google Scholar
• Feldon , D. 2007 . The implications of research on expertise for curriculum and pedagogy . Educational Psychology Review , 19 ( 2 ) : 91 – 110 . (doi:10.1007/s10648-006-9009-0)
   Web of Science ®Google Scholar
• Feldon , D. 2010 . Do psychology researchers tell it like it is? A microgenetic analysis of research strategies and self-report accuracy along a continuum of expertise . Instructional Science , 38 ( 4 ) : 395 – 415 . (doi:10.1007/s11251-008-9085-2)
   Web of Science ®Google Scholar
• Feldon , D. F. , Maher , M. M. and Timmerman , B. E. 2010 . Performance-based data in the study of STEM PhD education . Science , 329 ( 5989 ) : 282 – 3 . (doi:10.1126/science.1191269)
   PubMed Web of Science ®Google Scholar
• Feldon , D. F. , Timmerman , B. E. , Showman , R. and Stowe , K. A. 2010 . Translating expertise into effective instruction: The impacts of cognitive task analysis (CTA) on laboratory report quality and student retention in the biological sciences . Journal of Research in Science Teaching , 47 ( 10 ) : 1165 – 85 . (doi:10.1002/tea.20382)
   Web of Science ®Google Scholar
• Golde , C. M. 2001 . Findings of the survey of doctoral education and career preparation: A report to the preparing future faculty program , Madison : University of Wisconsin .
   Google Scholar
• Hackett , E. and Rhoten , D. 2009 . The snowbird charrette: Integrative interdisciplinary collaboration in environmental research design . Minerva , 47 ( 4 ) : 407 – 40 . (doi:10.1007/s11024-009-9136-0)
   Web of Science ®Google Scholar
• Halonen , J. S. , Bosack , T. , Clay , S. , McCarthy , M. , Dunn , D. S. , Hill , G. W. , McEntarffer , R. , Mehrotra , C. , Nesmith , R. , Weaver , K. A. and Whitlock , K. 2003 . A rubric for learning, teaching, and assessing scientific inquiry in psychology . Teaching of Psychology , 30 ( 3 ) : 196 – 208 . (doi:10.1207/S15328023TOP3003_01)
   Web of Science ®Google Scholar
• Hinds , P. J. 1999 . The curse of expertise: The effects of expertise and debiasing methods on predictions of novice performance . Journal of Experimental Psychology: Applied , 5 ( 2 ) : 205 – 21 . (doi:10.1037/1076-898X.5.2.205)
   Web of Science ®Google Scholar
• Hofstein , A. and Lunetta , V. N. 2004 . The laboratory in science education: Foundations for the twenty-first century . Science Education , 88 ( 1 ) : 28 – 54 . (doi:10.1002/sce.10106)
   Web of Science ®Google Scholar
• Johnson , R. L. , Penny , J. and Gordon , B. 2000 . The relation between score resolution methods and interrater reliability: An empirical study of an analytic scoring rubric . Applied Measurement in Education , 13 ( 2 ) : 121 – 38 . (doi:10.1207/S15324818AME1302_1)
   Web of Science ®Google Scholar
• Johnson , R. L. , Penny , J. , Gordon , B. , Shumate , S. R. and Fisher , S. P. 2005 . Resolving score differences in the rating of writing samples: Does discussion improve the accuracy of scores? . Language Assessment Quarterly: An International Journal , 2 ( 2 ) : 117 – 46 . (doi:10.1207/s15434311laq0202_2)
   Google Scholar
• Keys , C. W. 1994 . The development of scientific reasoning skills in conjunction with collaborative writing assignments: An interpretive study of six ninth-grade students . Journal of Research in Science Teaching , 31 ( 9 ) : 1003 – 22 . (doi:10.1002/tea.3660310912)
   Web of Science ®Google Scholar
• Kiley , M. 2008 . Quality in postgraduate education , Canberra : Australian National University . Centre for Educational Development and Academic Methods 2008. http://www.qpr.edu.au/overview.html
   Google Scholar
• Kiley , M. 2009 . Identifying threshold concepts and proposing strategies to support doctoral candidates . Innovations in Education and Teaching International , 46 ( 3 ) : 293 – 304 . (doi:10.1080/14703290903069001)
   Web of Science ®Google Scholar
• Kiley , M. and Wisker , G. 2009 . Threshold concepts in research education and evidence of threshold crossing . Higher Education Research & Development , 28 ( 4 ) : 431 – 41 . (doi:10.1080/07294360903067930)
   Web of Science ®Google Scholar
• Land , R. , Meyer , J. H.F. and Smith , J. 2008 . Threshold concepts within the disciplines , Edited by: Land , R. , Meyer , J. H.F. and Smith , J. Rotterdam : Sense .
   Google Scholar
• Lawson , A. E. 2008 . Biology: An inquiry approach , 2 , Dubuque , IA : Kendall/Hunt Publishing Company .
   Google Scholar
• Marx , R. W. , Blumenfeld , P. C. , Krajcik , J. S. , Fishman , B. , Soloway , E. , Geier , R. and Tal , R. T. 2004 . Inquiry-based science in the middle grades: Assessment of learning in urban systemic reform . Journal of Research in Science Teaching , 41 ( 10 ) : 1063 – 80 . (doi:10.1002/tea.20039)
   Web of Science ®Google Scholar
• Mervis , J. 2000 . Graduate educators struggle to grade themselves . Science , 287 ( 5453 ) : 568 – 70 . (doi:10.1126/science.287.5453.568)
   Web of Science ®Google Scholar
• Meyer , J. H.F. and Land , R. 2006 . Overcoming barriers to student understanding: Threshold concepts and troublesome knowledge , Edited by: Meyer , J. H.F. and Land , R. Abingdon , UK : Routledge .
   Google Scholar
• National Research Council . 1996 . National science education standards , Washington , DC : National Academy Press .
   Google Scholar
• Onwuegbuzie , A. J. 2003 . Modeling statistics achievement among graduate students . Educational and Psychological Measurement , 63 ( 6 ) : 1020 – 38 . (doi:10.1177/0013164402250989)
   Web of Science ®Google Scholar
• Ostriker , J. P. and Kuh , C. V. 2003 . Assessing research-doctorate programs: A methodology study , Edited by: Ostriker , J. P. and Kuh , C. V. Washington , DC : National Academies Press .
   Google Scholar
• Popper , K. 1959 . The logic of scientific discovery , New York : Routledge .
   Google Scholar
• Roberts , G. 2002 . Set for success: The supply of people with science, technology, engineering and mathematical skills HM Treasury, April 2002. http://www.employment-studies.co.uk/pubs/report.php?id=1440robert
   Google Scholar
• Sandoval , W. A. and Reiser , B. J. 2004 . Explanation-driven inquiry: Integrating conceptual and epistemic scaffolds for scientific inquiry . Science Education , 88 ( 3 ) : 345 – 72 . (doi:10.1002/sce.10130)
   Web of Science ®Google Scholar
• Schroeder , C. M. , Scott , T. P. , Tolson , H. , Huang , T.-Y. and Lee , Y.-H. 2007 . A meta-analysis of national research: Effects of teaching strategies on student achievement in science in the United States . Journal of Research in Science Teaching , 44 ( 10 ) : 1436 – 60 . (doi:10.1002/tea.20212)
   Web of Science ®Google Scholar
• Schunn , C. D. and Anderson , J. R. 1999 . The generality/specificity of expertise in scientific reasoning . Cogitive Science , 23 ( 3 ) : 337 – 70 . (doi:10.1207/s15516709cog2303_3)
   Web of Science ®Google Scholar
• Sevian , H. and Gonsalves , L. 2008 . Analysing how scientists explain their research: A rubric for measuring the effectiveness of scientific explanations . International Journal of Science Education , 30 ( 11 ) : 1441 – 67 . (doi:10.1080/09500690802267579)
   Web of Science ®Google Scholar
• Seymour , E. 2001 . Tracking the processes of change in US undergraduate education in science, mathematics, engineering and technology . Science Education , 86 ( 1 ) : 79 – 105 . (doi:10.1002/sce.1044)
   Web of Science ®Google Scholar
• Timmerman , B. E. , Strickland , D. C. , Johnson , R. L. and Payne , J. 2011 . Development of a ‘universal’ rubric for assessing undergraduates' science inquiry and reasoning skills using scientific writing across multiple courses . Assessment & Evaluation in Higher Education , 36 ( 5 ) : 509 – 47 . (doi:10.1080/02602930903540991)
   Web of Science ®Google Scholar
• Willison , J. and O'Regan , K. 2007 . Commonly known, commonly not known, totally unknown: A framework for students becoming researchers . Higher Education Research & Development , 26 ( 4 ) : 393 – 409 . (doi:10.1080/07294360701658609)
   Google Scholar
• Zimmerman , C. 2000 . The development of scientific reasoning skills . Developmental Review , 20 ( 1 ) : 99 – 149 . (doi:10.1006/drev.1999.0497)
   Web of Science ®Google Scholar