Educational assessment in digital environments: insights from different assessment contexts

We are living in times where many things in our everyday life are dependent on or assisted by digital tools. This will probably become even more prominent in the future, and we can only imagine the possibilities but perhaps also the challenges that this will bring. In educational contexts today, we can see many advantages with digital tools, and how technology can support teaching, learning and assessment. However, there are also risks involved, for instance when it comes to schools or students having different familiarity with or access to the latest technology, which can lead to increasing inequality and social gaps in its use. There are also, of course, other considerations having to do with how a new mode affects the quality of education or assessment, and whether changes are for the better. For such reasons, it is important to investigate the consequences of moving educational practices to a digital environment, and to make decisions guided by research when we introduce changes with potentially serious impact on students’ learning and future choices.

Today, an increasing number of tests, from small-scale ones used in the classroom to large-scale ones such as national and international tests used in global surveys, are moving from paper to digital formats. At the same time, there are many knowledge gaps when it comes to how this will affect test practice, test takers, and the area of educational assessment more generally. Furthermore, there are considerable financial stakes, where software and test developers can be reluctant to publish information about problems or challenges with the use of tests, and relevant information for guiding decisions can be difficult to access. Additional to practical issues of infrastructure and cost, there are a number of validity aspects that need to be considered, that have to do with fairness for the test takers, and the degree to which relevant assessment information can be achieved.

Based on such concerns, a conference was organised in the University of Umeå in 2016, with a focus on “Tests and test environments of the future”. The conference participants shared experiences from different perspectives and systems, and addressed issues related to ongoing and established digitalised systems of assessment. The present issue of Education Inquiry is one of the conference products, aiming to continue the work of research and ideas dissemination, with the ambition to reach out to policy, research and practice.

The papers in this issue discuss digitalization from different perspectives; at a system level, and with a focus on validity issues in relation to tests, with examples and empirical data from different countries. Separately and together, the papers provide new and valuable knowledge for test developers and test publishers, researchers and policy makers.

The paper by Brown, “Technologies and infrastructure: Costs and obstacles in developing large scale computer-based testing”, tells an interesting story from the inside about the implementation of a large-scale computer-based testing system in New Zealand (asTTle), which is one of the first and maybe also one of the more innovative national test systems in a digital format internationally. Brown describes how the asTTle was developed, and illuminates a number of challenges that should be considered when implementing large-scale systems for digital assessment. He also comes back to the main question that should be central to all research and development, namely, the need to have a clear purpose for the generation and application of digital assessment systems. The paper also addresses challenges of working within the dynamic and changing world of technology, in combination with a national education and assessment system where many considerations have to be made, from policy issues, efficiency and cost for schools and educational systems, to validity and fairness for the test takers. This survey of technological issues makes it clear that moving to online or onscreen testing may hit many snags that need substantial thought prior to engaging in the development process.

The paper by Wise, “The potential for computer-based tests to control construct-irrelevant factors”, discusses potential benefits and disadvantages that come with the change from paper-based to computer-based test formats. Drawing on extensive experience in the field of computer-based testing, in line with Brown, Wise shows that although moving to computer-based tests is not without costs (in terms of technological and financial resources as well as logistic and psychometric challenges), there may also be several positive outcomes, for example in terms of efficiency, convenience and validity. In particular, it is argued that one of the advantages of computer-based tests is the increased possibilities to control construct-irrelevant factors (motivation, anxiety, cheating) that can threaten test score validity.

The issue of validity, test mode effects and possible consequences of transitioning from paper- to computer-based tests is further elaborated on in the empirical paper by Zehner and colleagues, “Unattended consequences: How text responses alter alongside PISA’s mode change from 2012 to 2015”. Here, the authors utilise data from PISA 2012, when the test was administered on paper, and PISA 2015, when the test was administered on computer for the first time. Using a large sample of German students, student text responses in the respective PISA administrations were analysed and compared. The results show that there may be a substantial change in text responses parallel to the change in test mode, raising questions about whether we really are measuring the same thing when moving from paper-based to computer-based formats and whether valid comparisons can be made between administrations, in the case of PISA, over time.

Together, these papers show that although technology has the potential to make education and assessment better, by adding new features for teaching, learning and assessment, and making it more accessible and effective to administrate, there are also many challenges involved.

From an assessment point of view, fairness is an overarching concern, and it is important to ensure that all test takers are given equal prerequisites to show what they know and can do. When developing tests, or changing the mode or administration of tests, potential threats to validity have to be taken into consideration, especially when there are high stakes attached for the test takers and other stakeholders. Here, it should be noted that the papers in this special issue have been studying fairly low-stakes tests. In high-stakes testing new types of mechanisms will be in place, testing the vulnerability of the system and putting pressure on the test takers.

While practical matters regarding infrastructure and cost have to be solved at policy level, research is important to guide decisions when it comes to the implementation and implications of new formats and how the outcomes may be interpreted and used. The overall challenge is to ensure that all test takers are given the same possibilities to show what they know and what they can do, and there is a strong need for high-quality research in this area. It is our hope that this special issue will make an interesting and useful contribution to this field.