Validity and participation: implications for school comparison of Australia’s National Assessment Program

Abstract

The National Assessment Program – Literacy and Numeracy (NAPLAN) in Australia is a series of literacy and numeracy tests that are used for purposes of school comparison. This paper argues that a key question for this use lies in whether or not this is a reasonable, or valid, use of the test data. Using Kane’s argumentative approach to validity, this paper argues that the comparisons of the quality of student achievement made available on the My School Website have low validity due to the lack of regard to rates of participation in schools. In bringing together the literature that addresses the ‘new governance’ of education through testing and an approach to validity that addresses the technical aspects of test score interpretation, with the ethics of how test scores are used and applied, this study identifies validity as an important consideration in comparative analyses of student achievement data. The identification of the need to consider participation in such comparisons through the application of the argumentative approach to validity highlights the contribution of this article not only to the testing field but also to critical policy literature.

Keywords:

• NAPLAN
• validity
• participation
• testing

Notes

1. An example of this is the 2016 (Vol. 23) special issue of Assessment in Education: Principles, Policy and Practice where a number of theorists, psychometricians and assessment experts debated issues of validity in educational assessment.

2. This may be due to the effect of the Online Literacy and Numeracy Assessment (OLNA) (http://wace1516.scsa.wa.edu.au/assessment/olna) conducted by the WA Education Department. Students must sit OLNA in Year 10 (and Year 11 and Year 12 if they do not reach the minimum standard) to be eligible for secondary graduation. These tests are benchmarked to Year 9 NAPLAN, students who perform at a high level on Year 9 NAPLAN are exempt from sitting OLNA.

3. ACARA does not provide details of the population in the Final Reports, rather it provides the number of students who sat each domain and the participation percentage this represents. Total population was calculated by dividing the number of students sitting the tests by the participation percentage.

4. Each ICSEA value “corresponds to the average level of educational advantage of the school’s student population relative to those of other schools” (http://www.acara.edu.au/verve/_resources/About_icsea_2014.pdf). It is a numeric scale representing educational advantage incorporating, amongst other things, parents educational backgrounds and occupations in a school community. Every Australian school has an ICSEA value, and these are shown on the MySchool website.