Assessing the ease of use in the environment and markers' acceptance of on screen marking: a Rasch measurement perspective

Abstract

This study extends investigations into 2 areas: ease of use in the environment and markers' acceptance of on screen marking (OSM) in the Hong Kong public examination context. In contrast to previous studies where there was a single focus, this study comprises a heterogeneous approach. The sample contains scripts from three subjects (English, Chinese, and Liberal Studies). Scripts comprised essays and short-answer questions, as well as scripts written in either English or Chinese. Two scales assessing the ease of use and markers' acceptance of OSM were investigated from a Rasch measurement perspective; with both scales showing good psychometric properties. The findings revealed that markers generally had a high level of perceived ease of use in the environment and the overall acceptance of OSM was positive. Differences of person measures across language, question type, and subject were compared, and the implications of the two scales for future research are briefly discussed.

Keywords:

• on screen marking
• Rasch measurement
• ease of use in the OSM environment
• acceptance of OSM
• differential item functioning

Acknowledgements

We would like to thank the Hong Kong Examinations and Assessment Authority – and in particular Christina Lee, the General Manager for Assessment Development – for support on the project regarding access to markers and for data collection.

Notes on contributors

Zi Yan is an Assistant Professor in the Department of Curriculum and Instruction at The Hong Kong Institute of Education. His main research interests are in Rasch measurement, scale development, and large-scale assessment.

David Coniam is a Chair Professor in the Department of Curriculum and Instruction at The Hong Kong Institute of Education, where he is a teacher educator, working with teachers in Hong Kong secondary schools. His main publication and research interests are in language assessment, language teaching methodology, and corpus linguistics.

Notes

*Item 11 was retained since it appeared to be DIF free when Items 14, 15, and 16 were removed from the scale.

1“Person” (as in “person measure”, “person reliability”, etc.) is a commonly used term in Rasch analysis for “subjects”, “test takers”, “raters”, and so forth. In the current article, it refers to “markers”.

2The page numbers for the Coniam 2013 reference were not assigned at the time of writing.