‘Time to figure out what to do’: understanding the nature of Irish post-primary students’ interactions with computer-based exams (CBEs) that use multimedia stimuli

ABSTRACT

In line with the widespread proliferation of digital technology in everyday life, many countries are now beginning to use computer-based exams (CBEs) in their post-primary education systems. To ensure that these CBEs are delivered in a manner that preserves their fairness, validity, utility and credibility, several factors pertaining to their design and development will need to be considered. This research study investigated the extent to which the design of different types of test items (e.g. inclusion of multimedia stimuli) in a CBE can affect test-takers’ engagement and behaviour. Qualitative data from a cued-Retrospective Think Aloud (c-RTA) protocol were gathered from 12 participants who had participated in a previous eye-tracking study. Participants watched a replay of their eye movements and were asked to state out loud what they were thinking at different points of the replay. Thematic analysis of the responses from these cognitive interviews captured the nature of students’ interactions with online testing environments under three main themes: Familiarisation, Sense-making and Making Decisions. Students also provided their opinions of and recommendations for the future of Irish online assessments. These findings can offer guidelines to all stakeholders considering the use of CBEs in post-primary contexts.

KEYWORDS:

• Examinations
• post-primary
• computer-based exams
• assessment

Disclosure statement

No potential conflict of interest was reported by the author(s).

Notes

1 That some participants completed items with static multimedia objects, whilst others completed items with dynamic objects reflects the fact that the larger piece of research from which this study is taken from was focused on the comparison of test-taker performance and attentional behaviour across these two different types of multimedia stimuli.

Additional information

Funding

This work was supported by the Irish Research Council [Grant Number GOIPG/2019/1959].

Notes on contributors

Paula Lehane

Paula Lehane is an Assistant Professor in the School of Inclusive and Special Education in the Institute of Education at Dublin City University (DCU). She previously worked as a primary school teacher and was the Special Educational Needs (SEN) coordinator of a large urban primary school. She recently completed her doctoral research on the design of digital assessments for post-primary students. Her research interests encompass assessment, inclusion and technology as they relate to the education systems.

Darina Scully

Dr Darina Scully is an Assistant Professor of Child & Adolescent Learning and Development at DCU’s Institute of Education. She holds a PhD in Psychology from Trinity College, Dublin, and she is currently lecturing in quantitative research methods and social, personal & health education. Her research interests span various assessment, teaching and learning issues in primary, post-primary and higher education contexts.

Michael O'Leary

Professor Michael O’Leary holds the Prometric Chair in Assessment at DCU and is Director of the Centre for Assessment Research Policy and Practice (CARPE) at the Institute of Education there. He leads a programme of research at CARPE focused on assessment in education and in the workplace.