An investigation of construct relevant and irrelevant features of mathematics problem-solving questions using comparative judgement and Kelly’s Repertory Grid

References

• ACME. (2011). Mathematical needs: Mathematics in the workplace and in higher education. London: Advisory Committee on Mathematics Education.
   Google Scholar
• ACT. (2006). Ready for college and ready for work: Same or different? Iowa: American College Tests, Inc.
   Google Scholar
• Bisson, M., Gilmore, C., Inglis, M., & Jones, I. (2016). Measuring conceptual understanding using comparative judgement. International Journal of Research in Undergraduate Mathematics Education, 2, 141–164. doi: 10.1007/s40753-016-0024-3
   Google Scholar
• Bradley, R. A., & Terry, M. E. (1952). Rank analysis of incomplete block designs I: The method of paired comparisons. Biometrika, 39, 324–345. doi: 10.1093/biomet/39.3-4.324
   Web of Science ®Google Scholar
• Bramley, T. (2007). Paired comparison methods. In P. Newton, J. A. Baird, H. Goldstein, H. Patrick, & P. Tymms (Eds.), Techniques for monitoring the comparability of examination standards (pp. 166–206). London: QCA.
   Google Scholar
• CBI. (2006). Working with the three Rs: Employers’ priorities for functional skills in mathematics and English. London: DfES.
   Google Scholar
• DfE. (2010). The importance of teaching: The schools white paper 2010. London: Department for Education.
   Google Scholar
• DfE. (2013). Gcse mathematics: Subject content and assessment objectives ( DFE-00233-2013). London: Department for Education.
   Google Scholar
• Haladyna, T., & Downing, S. (2004). Construct-irrelevant variance in high-stakes testing. Educational Measurement: Issues and Practice, 23, 17–27. doi: 10.1111/j.1745-3992.2004.tb00149.x
   Google Scholar
• Heldsinger, S., & Humphry, S. (2010). Using the method of pairwise comparison to obtain reliable teacher assessments. The Australian Educational Researcher, 37, 1–19. doi: 10.1007/BF03216919
   Web of Science ®Google Scholar
• Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6, 65–70.
   Web of Science ®Google Scholar
• Hughes, S., Pollitt, A., & Ahmed, A. (1998). The development of a tool for gauging the demands of GCSE and A level exam items. Talk at British Educational Research Association conference, Queens University Belfast, August. Retrieved February 15, 2016, from http://www.cambridgeassessment.org.uk/Images/109649-the-development-of-a-tool-for-gauging-the-demands-of-gcse-and-a-level-exam-questions.pdf
   Google Scholar
• Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39, 1774–1787. doi: 10.1080/03075079.2013.821974
   Web of Science ®Google Scholar
• Jones, I., & Inglis, M. (2015). The problem of assessing problem solving: Can comparative judgement help? Educational Studies in Mathematics, 89, 337–355. doi: 10.1007/s10649-015-9607-1
   Web of Science ®Google Scholar
• Jones, I., Swan, M., & Pollitt, A. (2014). Assessing mathematical problem solving using comparative judgement. International Journal of Science and Mathematics Education, 13, 151–177. doi: 10.1007/s10763-013-9497-6
   Web of Science ®Google Scholar
• Kelly, G. A. (1955). The psychology of personal constructs: Vols 1 and 2. New York: WW Norton.
   Google Scholar
• Kimbell, R. (2012). Evolving project e-scape for national assessment. International Journal of Technology and Design Education, 22, 135–155. doi: 10.1007/s10798-011-9190-4
   Web of Science ®Google Scholar
• Laming, D. (1984). The relativity of ‘absolute’ judgements. British Journal of Mathematical and Statistical Psychology, 37, 152–183. doi: 10.1111/j.2044-8317.1984.tb00798.x
   Web of Science ®Google Scholar
• Laming, D. (1990). The reliability of a certain university exam compared with the precision of absolute judgements. The Quarterly Journal of Experimental Psychology Section A: Human Experimental Psychology, 42, 239–254. doi: 10.1080/14640749008401220
   Web of Science ®Google Scholar
• Lane, S., & Iwatani, E. (2016). Design of performance assessments in education. In S. Lane, M. Raymond, & T. Haladyna (Eds.), Handbook of test development (2nd ed., pp. 274–293). New York: Routledge.
   Google Scholar
• Leighton, J. P., & Gokiert, R. J. (2005). The cognitive effects of test item features: Informing item generation by identifying construct irrelevant variance. Paper presented at the Annual Meeting of the National Council on Measurement in Education (NCME), Montreal, Quebec, Canada.
   Google Scholar
• Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed., pp. 13–104). New York: American Council on Education and Macmillan.
   Google Scholar
• Newton, P., Baird, J., Goldstein, H., Patrick, H., & Tymms, P. (Eds.). (2007). Techniques for monitoring the comparability of examination standards. London: Qualifications and Curriculum Authority.
   Google Scholar
• OECD. (2013). Pisa 2012 assessment and analytical framework: Mathematics, reading, science, problem solving and financial literacy. Paris: OECD.
   Google Scholar
• OECD. (2014). Pisa 2012 results: What students know and can do – Student performance in mathematics, reading and science (volume 1: Revised edition, February 2014). Paris: OECD.
   Google Scholar
• Ofqual. (2011). Review of standards in art and design (GCSE 1999 and 2009, GCE 1998/9 and 2099), Ofqual/11/4848. Coventry: The Office of Qualifications and Examinations Regulation.
   Google Scholar
• Ofqual. (2012). Review of standards in GCE A level critical thinking: 2010, Ofqual/12/5158. Coventry: The Office of Qualifications and Examinations Regulation.
   Google Scholar
• Ofqual. (2014). Setting GCSE, AS and A level grade standards in summer 2014 and 2015, Ofqual/15/5759. Coventry: The Office of Qualifications and Examinations Regulation.
   Google Scholar
• Ofqual. (2015a). A comparison of expected difficulty, actual difficulty and assessment of problem solving across GCSE maths sample assessment materials, Ofqual/15/5679. Coventry: The Office of Qualifications and Examinations Regulation.
   Google Scholar
• Ofqual. (2015b). Gcse (9 to 1) subject level guidance for mathematics, Ofqual/15/5722. Coventry: The Office of Qualifications and Examinations Regulation.
   Google Scholar
• Ofqual. (2015c). Inter-subject comparability: A review of the technical literature: ISC working paper 2, Ofqual/15/5794. Coventry: The Office of Qualifications and Examinations Regulation.
   Google Scholar
• Ofsted. (2012). Mathematics: Made to measure. London: The Office for Standards in Education.
   Google Scholar
• Parsons, J. M., Graham, N., & Honess, T. (1983). A teacher’s implicit model of how children learn. British Educational Research Journal, 9, 91–101. doi: 10.1080/0141192830090110
   Google Scholar
• Pollitt, A. (2012). The method of adaptive comparative judgement. Assessment in Education: Principles, Policy and Practice, 19, 281–300. doi: 10.1080/0969594X.2012.665354
   Google Scholar
• Polya, G. (1971). How to solve it. Princeton, NJ: Princeton University Press. (Originally copyrighted in 1945).
   Google Scholar
• Raikes, N., Scorey, S., & Shiell, H. (2008). Grading examinations using expert judgements from a diverse pool of judges. Paper presented to the 34th annual conference of the International Association for Educational Assessment, Cambridge, 2008. Retrieved on 15/02/16 from www.cambridgeassessment.org.uk/Images/109766-grading-examinations-using-expert-judgements-from-a-diverse-pool-of-judges.pdf.
   Google Scholar
• Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests (reprint, with foreword and afterword by B. D. Wright, Chicago: University of Chicago Press, 1980). Copenhagen, Denmark: Danmarks Paedogogiske Institut.
   Google Scholar
• Schoenfeld, A. H. (1987). Cognitive science and mathematics education. Hillsdale, NJ: Lawrence Erlbaum.
   Google Scholar
• Searle, J., & Barmby, P. (2012). Evaluation report on the realistic mathematics evaluation pilot project. Durham: Centre for Evaluation and Monitoring, Durham University. Retrieved February 15, 2016, from www.mei.org.uk/files/pdf/RME_Evaluation_final_report.pdf.
   Google Scholar
• Suto, W. M. I., & Nádas, R. (2009). Why are some GCSE examination questions harder to mark accurately than others? Using Kelly’s Repertory Grid technique to identify relevant question features. Research Papers in Education, 24, 335–377. doi: 10.1080/02671520801945925
   Web of Science ®Google Scholar
• Swan, M., & Burkhardt, H. (2012). Designing assessment of performance in mathematics. Educational Designer, 2.
   Google Scholar
• Thurstone, L. L. (1927). A law of comparative judgement. Psychological Review, 34, 273–286. doi: 10.1037/h0070288
   Google Scholar
• Toner, P. (2011). Workforce skills and innovation (OECD education working papers). Paris: Organisation for Economic Co-operation and Development.
   Google Scholar
• Vordermann, C., Porkess, R., Budd, C., Dunne, R., & Rahman-Hart, P. (2011). A world-class mathematics education for all our young people. London: The Conservative Party.
   Google Scholar
• Webb, N. (1997). Criteria for alignment of expectations and assessments in mathematics and science education. Washington, DC: Research Monograph No. 8, Council of Chief State School Officers.
   Google Scholar
• Wright, B. D., & Masters, G. N. (1982). Rating scale analysis: Rasch measurement. Chicago, IL: MESA Press.
   Google Scholar