Predicting item difficulty of science national curriculum tests: the case of key stage 2 assessments

ABSTRACT

Predicting item difficulty is highly important in education for both teachers and item writers. Despite identifying a large number of explanatory variables, predicting item difficulty remains a challenge in educational assessment with empirical attempts rarely exceeding 25% of variance explained.

This paper analyses 216 science items of key stage 2 tests which are national sampling assessments administered to 11 year olds in England. Potential predictors (topic, subtopic, concept, question type, nature of stimulus, depth of knowledge and linguistic variables) were considered in the analysis. Coding frameworks employed in similar studies were adapted and employed by two coders to independently rate items. Linguistic demands were gauged using a computational linguistic facility. The stepwise regression models predicted 23% of the variance with extended constructed questions and photos being the main predictors of item difficulty.

While a substantial part of unexplained variance could be attributed to the unpredictable interaction of variables, we argue that progress in this area requires improvement in the theories and the methods employed. Future research needs to be centred on improving coding frameworks as well as developing systematic training protocols for coders. These technical advances would pave the way to improved task design and reduced development costs of assessments.

KEYWORDS:

• Item difficulty
• item demands
• key stage 2
• science assessments
• coding framework
• Coh-Metrix

Acknowledgments

The authors are grateful for Dr Barbara Donahue and Mrs Louise Benson at the STA for providing item information and parameters as well as their invaluable feedback on a previous draft of the manuscript. The authors would also like to thank Miss Nia Dowell (PhD student at the Institute of Intelligent Systems, University of Memphis) for her assistance with Coh-Metrix software and Mrs Diana Ng (DPhil candidate at the Oxford University Centre for Educational Assessment) for participating in the coding of items.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes

1. http://www.cambridgeassessment.org.uk/insights/using-the-cras-framework/

2. Since 2014, a new matrix sampling design has been adopted with KS2 science assessments only administered biennially.

3. https://www.gov.uk/government/news/new-video-published-on-changes-to-2016-tests-and-assessments

4. http://www.sats-papers.co.uk/sats-papers-ks2.php?route=mustregister

5. https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/328904/2014_KS2_science_sample_sample_materials.pdf

6. The POS for KS2 can be found on http://www.sats-papers.co.uk/sats-papers-ks2.php?route=mustregister

Additional information

Funding

This project has been funded by Pearson, Inc.

Notes on contributors

Yasmine H. El Masri

Yasmine H. El Masri is a research fellow at Oxford University Centre for Educational Assessment (OUCEA). She completed her DPhil at the University of Oxford in 2015. Her research interests include language in assessment, test translation and adaptation, models of task difficulty and task demands, science assessments, differential item functioning, Rasch modelling, international large-scale assessments.

Steve Ferrara

Steve Ferrara is a senior advisor for Measurement Solutions at Measured Progress in the US. He has held similar positions in other testing companies and was the State of Maryland Director of Student Assessment. Steve started his career as a preschool teacher and high school special education teacher. His research, design, and development interests include English language speaking formative assessment and learning systems; performance assessment approaches for accountability and formative assessment purposes; and content, cognitive, and linguistic demands of items and their relationship to item development and validity of score interpretations.

Peter W. Foltz

Peter W. Foltz is a vice president for research at Pearson and professor adjoint at the University of Colorado's Institute of Cognitive Science. Prior to that, he was a professor of psychology at New Mexico State University and worked as a member of technical staff at Bell Communications Research. His research focuses on learning, language comprehension, artificial intelligence and technologies for educational assessment. He has developed patented educational technology that is in use by millions of students annually. He has served as a lead for the development of frameworks for OECD's PISA assessment for 2015 and 2018. Dr Foltz holds a PhD in psychology from the University of Colorado.

Jo-Anne Baird

Jo-Anne Baird is a director of the Department of Education. Before her appointment to this position Jo-Anne was a director of the Oxford University Centre for Educational Assessment (OUCEA). Prior to that, she held the position of professor of Education and coordinator of the Centre for Assessment and Learning Studies at the University of Bristol. Jo-Anne previously held the position of head of research at the Assessment and Qualifications Alliance, where she managed the research programme and was responsible for the standard-setting systems for public examinations. She was also a lecturer at the Institute of Education in London. Her first degree and doctorate were in psychology and she has an MBA. Jo-Anne is a visiting professor at Queen's University, Belfast.