The effect of adaptivity on the reliability coefficient in adaptive comparative judgement

References

• Andrich, D. (1978). Relationships between the Thurstone and Rasch approaches to item scaling. Applied Psychological Measurement, 2, 449–460.
   Google Scholar
• Andrich, D. (1982). An index of person separation in latent trait theory, the traditional KR-20 index, and the Guttman scale response pattern. Education Research & Perspectives, 9(1), 95–104.
   Google Scholar
• Bradley, R. A., & Terry, M. E. (1952). The rank analysis of incomplete block designs: I. The method of paired comparisons. Biometrica, 39, 324–345.
   Web of Science ®Google Scholar
• Bramley, T. (2005). A rank-ordering method for equating tests by expert judgement. Journal of Applied Measurement, 6(2), 202–223.
   PubMedGoogle Scholar
• Bramley, T. (2007). Paired comparison methods. In P. E. Newton, J. Baird, H. Goldstein, H. Patrick, & P. Tymms (Eds.), Techniques for monitoring the comparability of examination standards (pp. 246–294). London: Qualifications and Curriculum Authority.
   Google Scholar
• Bramley, T. (2015). Investigating the reliability of adaptive comparative judgement. Cambridge Assessment Research Report. Cambridge: Cambridge Assessment.
   Google Scholar
• Glickman, M. E., & Jensen, S. T. (2005). Adaptive paired comparison design. Journal of Statistical Planning and Inference, 127(1–2), 279–293.10.1016/j.jspi.2003.09.022
   Web of Science ®Google Scholar
• Heldsinger, S., & Humphry, S. (2010). Using the method of pairwise comparison to obtain reliable teacher assessments. The Australian Educational Researcher, 37(2), 1–19.10.1007/BF03216919
   Web of Science ®Google Scholar
• Joint Council for Qualifications. (2016). Adjustments for candidates with disabilities and learning difficulties: Access arrangements and reasonable adjustments. London: Author.
   Google Scholar
• Jones, I., & Alcock, L. (2014). Peer assessment without assessment criteria. Studies in Higher Education, 39(10), 1774–1787.10.1080/03075079.2013.821974
   Web of Science ®Google Scholar
• Jones, I., Swan, M., & Pollitt, A. (2015). Assessing mathematical problem solving using comparative judgement. International Journal of Science and Mathematics Education, 13(1), 151–177.10.1007/s10763-013-9497-6
   Web of Science ®Google Scholar
• Jones, I., Wheadon, C., Humphries, S., & Inglis, M. (2016). Fifty years of A-level mathematics: Have standards changed? British Educational Research Journal, 42(4), 543–560.10.1002/berj.3224
   Web of Science ®Google Scholar
• Kimbell, R., Wheeler, T., Stables, K., Shepard, T., Martin, F., Davies, D., … Whitehouse, G. (2009). E-scape portfolio assessment phase 3 report. London: Goldsmiths, University of London.
   Google Scholar
• Linacre, J. M. (1987). FACETS (Version 3.71.4). Retrieved from www.winsteps.com
   Google Scholar
• Linacre, J. M. (2004). Rasch model estimation: Further topics. Journal of Applied Measurement, 5(1), 95–110.
   PubMedGoogle Scholar
• Linacre, J. M. (2014). A user’s guide to FACETS Rasch-Model computer programs program manual 3.71.4. Retrieved from www.winsteps.com
   Google Scholar
• McMahon, S., & Jones, I. (2015). A comparative judgement approach to teacher assessment. Assessment in Education: Principles, Policy & Practice, 22(3), 368–389.10.1080/0969594X.2014.978839
   Web of Science ®Google Scholar
• Newhouse, C. P. (2014). Using digital representations of practical production work for summative assessment. Assessment in Education: Principles, Policy & Practice, 21(2), 205–220.10.1080/0969594X.2013.868341
   Google Scholar
• Pollitt, A. (2004). Let’s stop marking exams. Paper presented at the annual conference of the International Association for Educational Assessment (IAEA), Philadelphia, PA. Retrieved November 28, 2016 from http://www.cambridgeassessment.org.uk/Images/109719-let-s-stop-marking-exams.pdf
   Google Scholar
• Pollitt, A. (2012). The method of Adaptive Comparative Judgement. Assessment in Education: Principles, Policy & Practice, 19(3), 281–300.10.1080/0969594X.2012.665354
   Google Scholar
• Pollitt, A. (2015). On ‘reliability’ bias in ACJ: Valid simulation of adaptive comparative judgement. Cambridge: Cambridge Exam Research.
   Google Scholar
• Pollitt, A., & Murray, N. L. (1993). What raters really pay attention to. Paper presented at the Language Testing Research Colloquium, Cambridge.
   Google Scholar
• Revuelta, J., & Ponsoda, V. (1998). A comparison of item exposure control methods in computerized adaptive testing. Journal of Educational Measurement, 35, 311–327.10.1111/jedm.1998.35.issue-4
   Web of Science ®Google Scholar
• Steedle, J. T., & Ferrara, S. (2016). Evaluating comparative judgement as an approach to essay scoring. Applied Measurement in Education, 29(3), 211–223.
   Web of Science ®Google Scholar
• Thurstone, L. L. (1927). A law of comparative judgment. Psychological Review, 34, 273–286.10.1037/h0070288
   Google Scholar
• Thurstone, L. L. (1931). Rank order as a psycho-physical method. Journal of Experimental Psychology, 14, 187–201.10.1037/h0070025
   Google Scholar
• van Daal, T., Lesterhuis, M., Coertjens, L., Donche, V., & De Maeyer, S. (2016). Validity of comparative judgement to assess academic writing: Examining implications of its holistic character and building on a shared consensus. Assessment in Education: Principles, Policy & Practice, 1–16.
   Google Scholar
• Wheadon, C., & Christodoulou, D. (2016, November 3–5). Improving moderation of teacher-assessed work. Paper presented at the annual conference of the Association for Educational Assessment – Europe (AEA-Europe), Limassol, Cyprus.
   Google Scholar
• Whitehouse, C., & Pollitt, A. (2012). Using adaptive comparative judgement to obtain a highly reliable rank order in summative assessment. Manchester: AQA Centre for Education Research and Policy.
   Google Scholar