Assessment as Judgment-in-Context: Analysing how teachers evaluate students' writingFootnote¹

1. The research reported in this article was partially funded by a research grant from the Australian Research Council (#A79906109) to the three authors.

References

• Bransford , J. D. and Schwartz , D. L. 1999 . “ Rethinking transfer: A simple proposal with multiple implications ” . In Review of research in education , Edited by: Iran-Nejad , A. and Pearson , P. D. Vol. 24 , 61 – 100 . Washington, DC : American Educational Research Association .
   Google Scholar
• Brehmer , A. 1989 . “ Grading as a quasi-rational judgment process ” . In Teacher thinking and professional action , Edited by: Lowyck , J. , Clark , C. and Halkes , R. Leuven, , Belgium : Leuven University Press .
   Google Scholar
• Brehmer B. Joyce C. R. B. 1988 Human judgment: The SJT view Amsterdam North-Holland Elsevier
   Google Scholar
• Brunswik , E. 1952 . The conceptual framework of psychology , Chicago : University of Chicago Press .
   Google Scholar
• Brunswik , E. 1955 . Representative design and probabilistic theory in a functional psychology . Psychological Review , 62 : 193 – 217 .
   PubMed Web of Science ®Google Scholar
• Brunswik , E. 1956 . Perception and the representative design of psychological experiments , 2nd ed. , Berkeley, CA : University of California Press .
   Google Scholar
• Cohen , J. , Cohen , P. , West , S. G. and Aiken , L. S. 2003 . Applied multiple regression/correlation analysis for the behavioral sciences , 3rd ed. , Mahwah, NJ : Lawrence Erlbaum Associates .
   Google Scholar
• Cooksey , R. W. 1988 . “ Social judgment theory in education: Current and potential applications ” . In Human judgment: The SJT view , Edited by: Brehmer , B. and Joyce , C. R. B. 273 – 315 . Amsterdam : North-Holland Elsevier .
   Google Scholar
• Cooksey , R. W. 1996 . Judgment analysis: Theory, methods and applications , San Diego, CA : Academic Press .
   Google Scholar
• Cooksey , R. W. 2001 . What is Complexity Science? A contextually-grounded tapestry of systemic dynamism, paradigm diversity, theoretical eclecticism, and organizational learning . Emergence , 3 : 77 – 103 .
   Google Scholar
• Cooksey , R. W. , Freebody , P. and Davidson , G. R. 1986 . Teacher's predictions of children's early reading achievement: An application of social judgment theory . American Educational Research Journal , 23 : 41 – 64 .
   Web of Science ®Google Scholar
• Curriculum Corporation . 1998 . Literacy standards: Years 3 & 5 writing, spelling and reading , Carlton South, Victoria, , Australia : The Craftsman Press .
   Google Scholar
• Doherty , M. and Kurz , E. 1996 . Social judgement theory . Thinking & Reasoning , 2 : 109 – 140 .
   Google Scholar
• Einhorn , H. J. , Kleinmuntz , D. N. and Kleinmuntz , B. 1979 . Linear regression and process-tracing models of judgment . Psychological Review , 86 : 465 – 485 .
   Web of Science ®Google Scholar
• Einhorn , H. J. and Koelb , C. T. 1982 . A psychometric study of literary-critical judgment . Modern Language Studies , 12 : 59 – 82 .
   Google Scholar
• Elander , J. and Hardman , D. 2002 . An application of judgment analysis to examination marking in psychology . British Journal of Psychology , 93 ( 3 ) : 303 – 326 .
   PubMed Web of Science ®Google Scholar
• Ericsson , K. A. 2003 . “ Valid and non-reactive verbalization of thoughts during performance of tasks: Towards a solution to the central problems of introspection as a source of scientific data ” . In Trusting the subject: The use of introspective evidence in cognitive science , Edited by: Jack , A. and Roepstorff , A. Vol. 1 , 1 – 18 . Exeter, , UK : Imprint Academic .
   Google Scholar
• Ericsson , K. A. and Simon , H. A. 1993 . Protocol analysis: Verbal reports as data , Rev. ed. , Cambridge, MA : The MIT Press .
   Google Scholar
• Forster , M. and Masters , G. 2004 . “ Bridging the conceptual gap between classroom assessment and system accountability ” . In Towards coherence between classroom assessment and accountability , Edited by: Wilson , M. 51 – 73 . Chicago : National Society for the Study of Education . Part II
   Google Scholar
• Freebody , P. , Cumming , J. and Falk , I. 1993 . A trialling and evaluation of the Adult Literacy and Numeracy (ALAN) Scales , Vol. 1 , Canberra, ACT, , Australia : Commonwealth Department of Employment, Education and Training .
   Google Scholar
• Freebody , P. and Wyatt-Smith , C. 2004 . The assessment of literacy: Working the zone between “system” and “site” validity . Journal of Educational Enquiry , 5 ( 2 ) : 30 – 49 .
   Google Scholar
• Gamire E. Pearson P. 2006 Tech tally: Approaches to assessing technological literacy Washington, DC National Research Council of the National Academies
   Google Scholar
• Goldstein , W. M. 2004 . “ Social judgment theory: Applying and extending Brunswik's probabilistic functionalism ” . In Blackwell handbook of judgment and decision making , Edited by: Koehler , D. and Harvey , N. 10 – 11 . Oxford : Blackwell .
   Google Scholar
• Hamm , R. 1988 . Moment-by-moment variation in experts' analytic and intuitive cognitive activity . IEEE Transactions on Systems, Man, and Cybernetics , 18 : 757 – 776 .
   Web of Science ®Google Scholar
• Hammond , K. R. 1980 . “ Introduction to Brunswikian theory and methods ” . In Realizations of Brunswik's representative design , Edited by: Hammond , K. R. and Wascoe , N. E. 1 – 12 . San Francisco : Jossey-Bass .
   Google Scholar
• Hammond , K. R. 1996 . Human judgment and social policy: irreducible uncertainty, inevitable error, unavoidable injustice , New York : Oxford University Press .
   Google Scholar
• Hammond , K. R. and Stewart , T. R. 2001 . The essential Brunswik: Beginnings, explications, applications , New York : Oxford University Press .
   Google Scholar
• Hammond , K. , Stewart , T. , Brehmer , B. and Steinmann , D. 1975 . “ Social judgment theory ” . In Human judgment and decision processes , Edited by: Kaplan , M. and Schwartz , S. 271 – 312 . New York : Academic Press .
   Google Scholar
• Harlen , W. 2005 . Teachers' summative practices and assessment for learning – tensions and synergies . The Curriculum Journal , 16 ( 2 ) : 207 – 223 .
   Google Scholar
• Hursch , C. J. , Hammond , K. R. and Hursch , J. L. 1964 . Some methodological considerations in multiple-cue probability studies . Psychological Review , 71 : 42 – 60 .
   PubMed Web of Science ®Google Scholar
• Linn , R. L. 1999 . “ Validity standards and principles on equity in educational testing and assessment ” . In Measuring up: Challenges minorities face in educational assessment , Edited by: Nettles , A. L. and Nettles , M. T. 13 – 31 . Boston : Kluwer Academic Publishers .
   Google Scholar
• Luke , A. , Freebody , P. , Lau , S. and Gopinathan , S. 2005 . Towards research-based innovation and reform: Singapore schools in transition . Asia Pacific Journal of Education , 25 ( 1 ) : 5 – 28 .
   Google Scholar
• Masters , G. N. and Forster , M. 1997 . Mapping literacy achievement: Results of the 1996 National School English Literacy Survey , Canberra, ACT, , Australia : Commonwealth Department of Employment, Education and Training .
   Google Scholar
• Organisation for Economic Co-operation and Development . 2002a . Manual for the PISA 2000 Database , Paris : Author .
   Google Scholar
• Organisation for Economic Co-operation and Development . 2002b . PISA 2000 Technical Report , Paris : Author .
   Google Scholar
• Richardson , K. A. , Cilliers , P. and Lissack , M. 2001 . Complexity science: A “gray” science for the “stuff in between” . Emergence , 3 ( 2 ) : 6 – 18 .
   Google Scholar
• Scanlon , P. A. and Ford , M. P. 1998 . Grading student performance in real-world settings . New Directions for Teaching and Learning , 74 ( Summer ) : 97 – 105 .
   Google Scholar
• Shulman , L. S. and Elstein , A. 1975 . “ Studies of problem solving, judgment, and decision making: Implications for educational research ” . In Review of research in education , Edited by: Kerlinger , F. N. Vol. 3 , 3 – 42 . Itasca, NY : Peacock .
   Google Scholar
• Slomp , D. H. and Fuite , J. 2004 . Following Phaedrus: Alternate choices in surmounting the reliability/validity dilemma . Assessing Writing , 3 : 190 – 207 .
   Google Scholar
• Snow , R. E. 1968 . Brunswikian approaches to research on teaching . American Educational Research Journal , 5 : 475 – 489 .
   Web of Science ®Google Scholar
• Snow , R. E. 1974 . Representative and quasi-representative designs for research on teaching . Review of Educational Research , 44 ( 3 ) : 265 – 291 .
   Web of Science ®Google Scholar
• Speck , B. W. 1998 . Unveiling some of the mystery of professional judgment in classroom assessment . New Directions for Teaching and Learning , 74 ( Summer ) : 17 – 31 .
   Google Scholar
• Stewart , T. R. 1988 . “ Judgment analysis: Procedures ” . In Human judgment: The SJT view , Edited by: Brehmer , B. and Joyce , C. R. B. 41 – 74 . Amsterdam : North-Holland .
   Google Scholar
• Suter , L. 2004 . “ Tools for two masters: Classroom assessment and school system assessment ” . In Towards coherence between classroom assessment and accountability , Edited by: Wilson , M. 169 – 182 . Chicago : National Society for the Study of Education . Part II
   Google Scholar
• Thompson , B. 2000 . “ Q-technique factor analysis: One variation on the two-mode factor analysis of variables ” . In Reading and understanding more multivariate statistics , Edited by: Grimm , L. G. and Yarnold , P. R. 207 – 226 . Washington, DC : American Psychological Association .
   Google Scholar
• Tucker , L. R. 1964 . A suggested alternative formulation in the developments by Hursch, Hammond, & Hursch and by Hammond, Hursch, & Todd . Psychological Review , 71 : 528 – 530 .
   PubMed Web of Science ®Google Scholar
• Willett , L. Marking writing tasks for tests of literacy in years 3, 5 and 7: Questions and challenges . Paper presented at the 9th Annual Roundtable Assessing Assessment . Sydney, NSW, Australia. November .
   Google Scholar
• Wilson , M. 2004 . “ Assessment, accountability and the classroom: A community of judgment ” . In Towards coherence between classroom assessment and accountability , Edited by: Wilson , M. 1 – 19 . Chicago : National Society for the Study of Education . Part II
   Google Scholar
• Wyatt-Smith , C. , Castleton , G. , Freebody , P. and Cooksey , R. W. 2003 . The nature of teacher's qualitative judgements: A matter of context and salience. Part One: “In-context” judgements . Australian Journal of Language and Literacy , 26 ( 2 ) : 11 – 32 .
   Google Scholar
• Wyatt-Smith , C. , Castleton , G. , Cooksey , R. W. and Freebody , P. 2003 . The nature of teacher's qualitative judgements: A matter of context and salience. Part Two: Out-of-context judgements . Australian Journal of Language and Literacy , 26 ( 2 ) : 33 – 42 .
   Google Scholar
• Van Someren , M. , Barnard , Y. and Sandberg , J. 1994 . The think-aloud method: A practical guide to modelling cognitive processes , London : Academic Press .
   Google Scholar
• Yen , N. S. 2005 . The effects of task complexity, task uncertainty and time pressure on judgment in multiple cue probability learning task . Journal of Education and Psychology , 28 ( 1 ) : 101 – 126 .
   Google Scholar