Why are some GCSE examination questions harder to mark accurately than others? Using Kelly's Repertory Grid technique to identify relevant question features

References

• Ahmed , A. and Pollitt , A. 2007 . Improving the quality of contextualized questions: An experimental investigation of focus . Assessment in Education , 14 ( 2 ) : 201 – 32 .
   Google Scholar
• Baird , J.‐A. , Greatorex , J. and Bell , J.F. 2004 . What makes marking reliable? Experiments with UK examinations . Assessment in Education , 11 ( 3 ) : 333 – 47 .
   Google Scholar
• Bramley , T. 2007 . Quantifying marker agreement: Terminology, statistics and issues . Research Matters: A Cambridge Assessment Publication , 4 : 22 – 8 .
   Google Scholar
• Branthwaite , A. , Trueman , M. and Berrisford , T. 1981 . Unreliability of marking: Further evidence and a possible explanation . Education Review , 33 ( 1 ) : 41 – 6 .
   Web of Science ®Google Scholar
• Butterworth , J. and Thwaites , G. 2005 . Preparing for the BMAT: The official guide to the BioMedical admissions test , Oxford : Heinemann .
   Google Scholar
• Crisp , V. 2007 . Researching the judgement processes involved in A‐level marking . Research Matters: A Cambridge Assessment Publication , 4 : 13 – 18 .
   Google Scholar
• Crisp , V. and Johnson , M. 2007 . The use of annotations in examination marking: Opening a window into markers' minds . British Educational Research Journal , 33 ( 6 ) : 943 – 61 .
   Web of Science ®Google Scholar
• Crisp , V. and Sweiry , E. 2006 . Can a picture ruin a thousand words? The effects of visual resources and layout in exam questions . Educational Research , 48 ( 2 ) : 139 – 54 .
   Web of Science ®Google Scholar
• Cumming , A. 1990 . Expertise in evaluating second language compositions . Language Testing , 7 : 31 – 51 .
   Google Scholar
• Fisher‐Hoch , H. , Hughes , S. and Bramley , T. . What makes GCSE examination questions difficult? Outcomes of manipulating difficulty of GCSE questions . Paper presented at the annual conference of the British Educational Research Association . York, UK.
   Google Scholar
• Greatorex , J. 2001 . Making the grade – how question choice and type affect the development of grade descriptors . Educational Studies , 27 ( 4 ) : 451 – 64 .
   Web of Science ®Google Scholar
• Greatorex , J. and Bell , J.F. 2004 . Does the gender of examiners influence their marking? . Research in Education , 7 : 25 – 36 .
   Google Scholar
• Greatorex , J. , Nádas , R. , Suto , W.M.I. and Bell , J.F. . Exploring how the cognitive strategies used to mark examination questions relate to the efficacy of examiner training . Paper presented at the annual conference of the European Conference on Educational Research . September 19–22 , Ghent, Belgium.
   Google Scholar
• Greatorex , J. and Suto , W.M.I. . What goes through a marker's mind? Gaining theoretical insights into the A‐level and GCSE marking process . Paper presented at the annual conference of Association for Educational Assessment (AEA)‐Europe . November 3–5 , Dublin, Ireland.
   Google Scholar
• Greatorex , J. and Suto , W.M.I. . An empirical exploration of human judgement in the marking of school examinations . Paper presented at the 32nd annual conference of the International Association for Educational Research . May 21–26 , Singapore.
   Google Scholar
• Hamilton , F. 2007 . New “dumbed‐down” GCSE will be as easy as A, B, C . The Times , http://www.timesonline.co.uk/tol/life
   Google Scholar
• Herman , R.L. 1996 . The repertory grid technique: Path to teacher description or teacher potential? . Instructional Science , 24 : 439 – 59 .
   Google Scholar
• Hoskonen , K. 1999 . A good pupil's beliefs about mathematics learning assessed by repertory grid methodology . Psychology of Mathematics Education Conference , 3 : 97 – 104 .
   Google Scholar
• Hughes , S. , Pollitt , A. and Ahmed , A. . The development of a tool for gauging the demands of GCSE and A level exam questions . Paper presented at the annual conference of the British Educational Research Association . August 27–30 , Belfast, Ireland.
   Google Scholar
• Jankowicz , D. 2004 . The easy guide to repertory grids , Chichester : John Wiley and Sons .
   Google Scholar
• Kahneman , D. and Frederick , S. 2002 . “ Representativeness revisited: Attribute substitution in intuitive judgement ” . In Heuristics and biases: The psychology of intuitive judgement , Edited by: Gilovich , Thomas , Griffin , Dale and Kahneman , Daniel . 49 – 81 . Cambridge : Cambridge University Press .
   Google Scholar
• Kelly , G.A. 1955 . The psychology of personal constructs , New York : Norton .
   Google Scholar
• Laming , D. 1990 . The reliability of a certain university examination compared with the precision of absolute judgements . Quarterly Journal of Experimental Psychology , 42A : 239 – 54 .
   Web of Science ®Google Scholar
• Laming , D. 2004 . Human judgement: The eye of the beholder , London : Thomson .
   Google Scholar
• Lengnink , K. and Prediger , S. 2003 . Development of personal constructs about mathematical tasks – a qualitative study using repertory grid methodology . Psychology of Mathematics Education Conference , 4 : 39 – 46 .
   Google Scholar
• McCloughlin , T. 2002 . Repertory grid analysis as a form of concept mapping in science education research . Irish Educational Studies , 21 ( 2 ) : 25 – 32 .
   Google Scholar
• Meadows , M. . The effect of marker background and training on the quality of marking in GCSE English . Paper presented at the annual conference of the International Association for Educational Assessment . September 17–21 , Baku, Azerbaijan.
   Google Scholar
• Meadows , M. and Billington , L. . Can we predict who will be a reliable marker? . Paper presented at the Institute of Assessors National Conference . June , London.
   Google Scholar
• Mikulas , C. and Kern , K. . A comparison of accuracy of automated essay scoring using prompt‐specific and prompt‐independent training . Paper presented at the annual conference of the American Educational Research Association . April 7–11 , San Francisco, USA.
   Google Scholar
• Milanovic , M. , Saville , N. and Shuhong , S. 1996 . “ A study of the decision‐making behaviour of composition markers ” . In Studies in language testing 3: Performance testing, cognition and assessment , Edited by: Milanovic , Michael and Saville , Nick . 92 – 114 . Cambridge : Cambridge University Press .
   Google Scholar
• Murphy , R.J.L. 1978 . Reliability of marking in eight GCE examinations . British Journal of Educational Psychology , 48 : 196 – 200 .
   Web of Science ®Google Scholar
• Murphy , R.J.L. 1982 . A further report of investigations into the reliability of marking GCE examinations . British Journal of Educational Psychology , 52 : 58 – 63 .
   Web of Science ®Google Scholar
• Newton , P. 1996 . The reliability of marking General Certificate of Secondary Education scripts: Mathematics and English . British Educational Research Journal , 22 : 404 – 20 .
   Google Scholar
• Pinot de Moira , A. , Massey , C. , Baird , J. and Morrissey , M. 2002 . Marking consistency over time . Research in Education , 67 : 79 – 87 .
   Google Scholar
• Powers , D. and Kubota , M. 1998 . “ Qualifying essay readers for an online scoring network ” . In Educational Testing Service Report , Princeton, NJ : Educational Testing Service .
   Google Scholar
• Qualifications and Curriculum Authority (QCA) . 2007 . GCSE, GCE, VCE, GNVQ and AEA Code of Practice 2006/7 , London : Qualifications and Curriculum Authority .
   Google Scholar
• Raikes , N. and Massey , A.J. 2007 . Item‐level examiner agreement . Research Matters: A Cambridge Assessment Publication , 4 : 34 – 37 .
   Google Scholar
• Rayment , T. 2000 . Art teachers' views of National Curriculum art: A repertory grid analysis . Educational Studies , 26 ( 2 ) : 165 – 76 .
   Web of Science ®Google Scholar
• Royal‐Dawson , L. . Is teaching experience a necessary condition for markers of Key Stage 3 English? . Assessment and Qualifications Alliance report, commissioned by the Qualification and Curriculum Authority .
   Google Scholar
• Shohamy , E. , Gordon , C.M. and Kraemer , R. 1992 . The effects of raters' background and training on the reliability of direct writing tests . The Modern Language Journal , 76 : 27 – 33 .
   Web of Science ®Google Scholar
• Siraj‐Blatchford , J. . Kelly's repertory grid: a technique for developing evaluation in design and technology . Paper presented at the Design and Technology Educational Research and Curriculum Development conference, Loughborough University . Loughborough, UK.
   Google Scholar
• Stahl , J.A. and Lunz , M.E. 1996 . “ Judge performance reports: Media and message ” . In Objective measurement. Theory into practice , Edited by: Engelhard , J.R. and Wilson , M. 113 – 26 . Norwood, , NJ : Ablex Publishing .
   Google Scholar
• Stanovich , K.E. and West , R.F. 2002 . “ Individual differences in reasoning ” . In Heuristics and biases: The psychology of intuitive judgement , Edited by: Gilovich , Thomas , Griffin , Dale and Kahneman , Daniel . 421 – 40 . Cambridge : Cambridge University Press .
   Google Scholar
• Suto , W.M.I. and Greatorex , J. 2006 . A cognitive psychological exploration of the GCSE marking process . Research Matters: A Cambridge Assessment Publication , 2 : 7 – 11 .
   Google Scholar
• Suto , W.M.I. and Greatorex , J. 2007 . What goes through an examiner's mind? Using verbal protocols to gain insights into the GCSE marking process . British Educational Research Journal , 34 ( 2 ) : 213 – 33 .
   Google Scholar
• Suto , W.M.I. and Greatorex , J. 2008 . A quantitative analysis of cognitive strategy usage in the marking of two GCSE examinations . Assessment in Education. Principles, policy and practice , 15 ( 1 ) : 73 – 89 .
   Google Scholar
• Suto , W.M.I. and Nádas , R. 2007 . Research matters . A Cambridge Assessment Publication , 4 : 2 – 5 .
   Google Scholar
• Suto , W.M.I. and Nádas , R. in press . What determines GCSE marking accuracy? An exploration of expertise among maths and physics markers . Research Papers in Education ,
   Google Scholar
• Valentine , C.M. 1932 . The reliability of examinations: An enquiry with special reference to the entrance examinations to secondary schools, the school certificate examinations, and the award of scholarships at universities , London : University of London Press .
   Google Scholar
• Vaughn , C. 1992 . “ Holistic assessment: What goes on in the rater's mind? ” . In Assessing second language writing in academic contexts , Edited by: Hamp‐Lyons , L. Norwood, NJ : Ablex Publishing Corporation .
   Google Scholar
• Wigglesworth , G. 1993 . Exploring bias analysis as a tool for improving rater consistency in assessing oral interaction . Language Testing , 10 ( 3 ) : 305 – 35 .
   Google Scholar
• Wood , A. and Pollitt , A. . The development of the Preferred Alternative Construct Elicitation (PACE) approach to eliciting the construct of demand from student reviewers .
   Google Scholar