How curriculum and assessment policies affect the role of reading in an assessment culture: a New Zealand case study

ABSTRACT

Reading assessment is a critical feature of education systems. The aim of this study is to explore reading assessment policy in the New Zealand (NZ) assessment culture. We reviewed research publications, national policy documents and test protocols on reading for year levels 6–9 (ages 9–13). We found that assessment data are broadly conceptualised and used for multiple purposes, that the representation of reading assessment data is located in an improvement-oriented environment and that the use of reading assessment for improvement purposes involves multiple actors and organisation levels. We discuss to what extent reading assessment policy supports student learning and teacher professional learning.

KEYWORDS:

• Assessment culture
• New Zealand
• reading assessment

Introduction

Reading assessment is a critical feature of education systems serving a variety of roles, from informing classroom practice to large-scale policy evaluation. There have been multiple waves of research on reading across the decades, suggesting how reading should be taught and assessed (Tierney & Pearson, 2021). In some countries, discussions of reading assessment policy has extended beyond scholarly debates (Stanovich, 2003), leading to the misrepresentation and politicisation of reading research and instruction as part of the reading wars (Alexander, 2020; Allington & Woodside Jiron, 1999). Reading assessment is therefore a contested issue in many policy contexts.

Recent policy changes in Aotearoa New Zealand (NZ) echo this debate. Media reports have covered the trial of new literacy and numeracy tests that will be made compulsory in 2024, with a significant number of students failing or performing more poorly than expected (Gerritsen, 2022). In response, a report from Education Hub, an NZ non-governmental organisation, criticises the state of literacy instruction, literacy teacher preparation, and the lack of reliable data on literacy achievement in NZ (Hill, 2022; Hughson & Hood, 2022), and advocates for national assessment reforms. Pushing back against this narrative, some NZ literacy researchers have challenged deficit views of learners and communities (Jesson, 2021) and emphasised the need for reading teachers to base their teaching on close monitoring of learners’ diverse needs and not on clinically developed intervention designs (Jesson, 2022).

Partisan animosity in any public debate can be destructive (Hartman et al., 2022). Polarised positions on reading are unfortunate because this is not just a question of learning gains but also one of social justice: the acquisition of subject-matter content such as mathematical or scientific literacy allows students to question existing insights and produce new knowledge in ways which may be powerful for marginalised youth (Moje, 2007). The contested nature of reading therefore warrants particular attention in studies of assessment policy.

The aim of this study is to explore how curriculum and assessment policies affect the role of reading in NZ. Drawing on research on assessment cultures, we first present our theoretical framework. We then provide a brief history and description of the NZ school system and assessment culture, to contextualise our study. Finally, we narrow our focus to consider the role of reading within the assessment culture of NZ.

Theoretical framework

Assessment cultures

The notion of assessment culture is defined in various ways in the literature and has evolved from reports on the technical aspects of testing to a broader, critical discussion of assessment policy and practice in social contexts (Broadfoot & Black, 2004). Some researchers privilege the study of assessment practices in contexts. For example, Alarcón and Lawn (2018) argue that the concept refers to a set of assessment practices in a specific setting of time and space, stakeholders’ meaning making within this setting, and the role of boundaries between countries (e.g. the US and Sweden), or between spheres of influence (e.g. Confucian heritage systems and the diffusion of a global testing culture). This means that studies of assessment cultures can take place across various scales, such as exploring single schools (Adie et al., 2021), the assessment system of a particular country (e.g. Broadfoot & Black, 2004; T. J. Crooks, 2002) or the role of transnational organisations such as the Organisation for Economic Co-operation and Development (OECD) (Addey, 2018).

Other researchers focus on opportunities for learning within an assessment culture. Birenbaum (2016) argues that an assessment culture exists at the intersection between a classroom learning system and a teacher professional learning system. Such studies often identify tensions between different components of the systems. Examples include studies of tensions between formatively oriented policies and compliance-oriented practices (Nieminen & Atjonen, 2022), between pre-school teachers’ views of children’s development and narrowly oriented assessment policies (Karlsudd, 2021), and between large-scale accountability policies and professional development for teachers (Klinger et al., 2012).

Yet others emphasise the need to balance close analysis of empirical evidence with critical discussion of the larger system. Fuller and Skidmore (2014) suggest that research into assessment cultures consists of (a) studies of the institutional practices suggestive of an assessment culture (e.g. goals, terms used, professional development efforts and student learning outcomes); and (b) conjectural elements hypothesised as fundamental to an assessment practice (e.g. assumptions about the integration of learning, assessment and instruction; the degree of shared responsibility between students and teachers; and the degree to which assessment is nested in authentic contexts).

What unifies these approaches to the study of assessment cultures is the combination of empirically founded analyses with conjectures about broader policy implications.While studies might take place on different scales and with differing levels of conjecture or critical discussion, the connection between empirical analysis and critical discussion remains central to analysing assessment cultures.

This study is based on the notion that curriculum and assessment policies frame teachers’ practice and professional learning, and that researchers should identify tensions between different system components. We investigate how curriculum and assessment frame classroom assessment practice and teacher learning within the boundaries of one national assessment system. We analyse policy documents, research literature, assessment manuals and related documents (e.g. technical reports) to produce an in-depth description and critique of reading assessment policy in a single cultural context. We focus on how data are conceptualised and represented and how policy initiatives balance improvement and accountability purposes. Our analysis provides insight into how curriculum and assessment policies can affect classroom assessment practice and opportunities for teacher learning in an assessment culture.The research question guiding the current study is: How is the use of reading assessment data framed in the New Zealand assessment culture?

The NZ assessment culture

Historically, the primary purpose of assessment in the NZ education system has been to support student learning; public trust in teachers’ professionalism has remained high (T. Crooks, 2011). While policies value and promote evidence-informed practice, there are few hierarchical accountability systems, and teachers exercise professional judgement (Lai & Sinnema, 2022). International organisations, such as the OECD, have considered the system to be of high quality (Nusche et al., 2012), and NZ authorities emphasise its flexibility, authenticity and closeness to practice (New Zealand Qualifications Authority, n.d.). However, there have been concerns about the accountability mechanisms inherent in the NZ assessment culture. For example, T. Crooks (2011) notes that several accountability pressures may affect NZ schools (e.g. reporting on achievement and progress to students and parents, schools inspections and legislation requiring annual reporting in schools). Nevertheless, compared with those of other countries, the accountability system of NZ has relatively low degrees of social regulation and bureaucratic control (Lai & Sinnema, 2022).

Three traits underpin the NZ assessment culture: (a) the notion of self-governing schools; (b) the relation between the national and local curricula; and (c) considerable professional discretion for teachers in assessment practice.

First, since 1989, NZ schools have been self-governing. This means that state schools are government funded and operated but governed by elected boards of trustees comprising parents, the principal, staff and student representatives (above year 9), and co-opted trustees. Schools are monitored by a national external evaluation agency, the Educational Review Office (ERO), which is separate from the MOE. The ERO is tasked with evaluating and reporting on the education and care of learners (Education Review Office, n.d.), and it adopts a negotiated and developmental approach to school evaluation (Lai & Kushner, 2013). The New Zealand education authorities promote the use of various data types, emphasising that a single data source is insufficient evidence of student learning (Ministry of Education, n.d.-b). Teachers in primary schools report on student achievement to the MOE and their respective boards of trustees. Following new legislation introduced in 2020, school boards are responsible for three-year strategic and annual implementation plans (Ministry of Education, 2023).

Second, the national curriculum framework is conceived as support for schools to develop their own local curricula, bringing together elements of the national framework within their own school contexts to provide a pathway that supports teachers in being responsive to all learners; since the rationale for a local curriculum acknowledges that the needs of each context are different, teaching strategies and local curricula must be responsive to them (Ministry of Education, 2007). As such, while the MOE coordinates nationally through curriculum and governance documents, schools (i.e. the principal, the staff and the board of trustees) are responsible for setting their own curricula in response to community needs and circumstances (Vannier, 2012). Schools thus enjoy considerable local autonomy.

Third, NZ assessment policy allows for considerable professional discretion for the teaching profession in deciding how and when to combine informal assessment methods, teacher-created summative assessments and teacher-administered standardised assessment tools (Irving et al., 2011). Teachers exercise professional discretion in selecting content and assessments, depending on the particular needs, interests and talents of students in their classes (Wilson et al., 2016). The combination of local curricula, teachers’ professional discretion and use of data is exemplified in the teaching as inquiry (TAI) approach (Ministry of Education, 2007). This process requires teachers to inquire into the impact of their professional practice by using assessment data in order to improve student learning and outcomes (H. Timperley et al., 2014). Although TAI has been criticised for lacking a clear definition and for requiring teachers to balance professional autonomy with accountability measures (Dyson, 2020), TAI still positions teachers as agentic decision-makers at the individual or collective level (Sinnema et al., 2017).

These cultural traits suggest that the NZ school system and assessment culture have sought a balance between governance through regulation and professional autonomy. In 2007, for example, the NZ authorities addressed issues of curriculum overcrowding by reducing prescription and increasing local freedom (Sinnema, 2016). However, NZ still struggles with inequities in educational achievement, particularly for Indigenous and minority students, as well as those with a lower socio-economic status (SES, OECD, 2010; Wilson et al., 2017), with some ethnic groupings performing well below the OECD average in science, reading and mathematics (Poskitt, 2018). The overrepresentation of particular student groups at the underachieving end of the distribution has led to the characterisation of the country as having a high-performance, low-equity profile (OECD Ed., 2014; Parr & Jesson, 2020). Furthermore, the COVID-19 pandemic exacerbated existing inequities (Mutch, 2021). There is also evidence of conflicting policy messages, such as when assessment for learning policies run contrary to qualification purposes (Hume & Coll, 2009). Empirical studies have shown that Indigenous and minority students (Māori and Pasifika) were significantly less likely to participate in challenging disciplinary reading instruction or engage with the longer and more sophisticated texts used in end-of-year assessments; Wilson et al. (2016) interpret this fact as an unintended consequence of the flexibility in NZ’s curriculum and assessment system. Consequently, literacy instruction in NZ makes the interesting case of an assessment system attempting to support teachers’ professionalism while fulfilling the different purposes of assessment and confronting inherent inequities.

The dynamics of NZ assessment policies

New Zealand assessment policies abandoned norm referencing in favour of criterion referencing in the mid-1980s, adopted the distinction between formative and summative assessment, mandated the use of assessment to improve learning in 1990, and made use of standardised tests primarily for diagnostic purposes; however, standardised tests in literacy, such as the Progressive Achievement Tests (PATs) and the electronic version of the Assessment for Teaching and Learning tool (e-asTTle), include norm referencing (e.g. monitoring student development in literacy and mathematics), and are used in many schools (T. J. Crooks, 2002; Smaill, 2020). In contrast to other English-speaking countries (e.g. the US and the UK), or the Asia-Pacific neighbours of NZ, the explicit adoption of assessment for learning and teachers’ professional judgement in NZ contributed to a low-stakes assessment culture (Irving et al., 2011). The early adoption of criterion referencing, formative assessment, and diagnostic tests therefore suggests that the NZ assessment culture has been less susceptible to sudden changes in policy. For example, NZ students’ mean performance in reading, mathematics and science during the 2000–2010 period was well above the OECD average in the Programme for International Student Assessment (Avvisati & Ilizaliturri, 2023), and did not lead to substantial changes in existing policies in the early 2000s (Breakspear, 2012; Dobbins, 2010). Rather, the efforts to balance accountability with improvement provided teachers with ample room for professional discretion in the selection of syllabuses, professional development, materials and assessments (Herman & Baker, 2009).

The introduction of the National Standards in 2010 required teachers to gather data, make qualitative judgements and report on the performance of students aged 5 to 12 (Smaill, 2020). Schools and authorities were expected to provide performance expectations for reading, writing and numeracy, assess student progress and compare with other students across the country, and report twice per year on the progress towards standards to parents and authorities using a four-point scale (‘above’, ‘at’, ‘below’ and ‘well below’ standard) (Lee & Lee, 2015). Some argued that this move shifted the balance towards greater standardisation and high-stakes summative assessments (T. Crooks, 2011). Following sustained criticism, the National Standards were removed in 2017. Since then, teacher judgement has become a primary source of information about students in the current NZ system.

Although the emphasis on teacher judgement in the NZ assessment culture aligns with the emphasis on professional discretion, a range of equity issues is inherent in the system. The self-managing school policy provided greater freedom and locally situated accountability, but longstanding equity issues remain (Robinson et al., 2011). For example, the number of Māori and Pasifika (Pacific island countries) students with university entrance is significantly lower than for other ethnic groups (Salahshour, 2021). Students from traditionally marginalised groups (i.e. males, Māori and Pasifika, students with special needs and students with English as a second language) receive systematically lower teacher judgements compared to standardised achievement test results (Meissel et al., 2017).

Multiple explanations have been proposed for equity issues, including the overlap between ethnicity and SES. One explanation specific to reading is that instructional approaches that are considered effective in raising literacy achievement (e.g. opportunities to read longer and more complex subject area texts) are less common in such classrooms (Wilson et al., 2017). Similarly, schools provide fewer opportunities for Māori and Pasifika students in low-SES schools to learn about the disciplinary-specific aspects of literacy in the English subject (e.g. becoming lifelong readers or interpreting the meaning, effectiveness and aesthetic value of complex literary texts) (Wilson et al., 2016). Researchers have called for a more responsive system capable of adapting to the culture, identity and literacy needs of students, especially Māori and students with special educational needs (Barker & Wood, 2019). Efforts have therefore been made to include new forms of digital literacy, adopting a life course perspective of literacy development as taking place through multiple channels, including family practices, and promoting and protecting cultural identity (McNaughton et al., 2021).

In March 2022, the release of a new strategy for literacy and communication, as well as for mathematics Ministry of Education, 2022), signalled that literacy skills are continued priorities in NZ schools. In this regard, Mutch’s (2012) characterisation of NZ assessment policies as striving for a balance between improvement and accountability purposes still holds true. Nevertheless, the response to improving literacy in recent years has followed the broader polarised debate on reading. Some (e.g. Hughson & Hood, 2022) have focused on evidence-based interventions and more reliable assessments, while others (e.g. Jesson, 2021, 2022) have argued that literacy educators need to understand how children interact with their social worlds in order to learn, thereby echoing the polarised debate on reading seen in other countries. We situate our analysis within this context to explore how reading assessment comprises a range of issues, from technical aspects of test development via teachers’ use of assessment information in daily contexts to wider issues of equity.

Materials and methods

We searched NZ government websites (e.g. https://assessment.tki.org.nz/Using-evidence-for-learning/Working-with-datasearch) and search engines, such as Eric and Google Scholar, for research publications, national policy documents and reading assessments for year levels 6–9 in NZ. This age span was selected to explore the transition from generic reading literacy (e.g. comprehension strategies, common word meanings and basic fluency) to disciplinary literacy (special skills for one subject area) (Shanahan & Shanahan, 2008). Although the NZ assessment system contains both English and Māori medium schools, we focused our search on English medium schooling only (information about assessment for Māori medium teachers is available at (Ministry of Education, n.d.).

We initially delimited our search to the period after 2007, the period when the recent NZ curriculum was introduced, and up to June 2021. However, some reading assessments have been in use for decades. For example, the Burt Word Reading Test was standardised for use in NZ in 1981. We therefore included assessments referenced in the documents retrieved, even if they pre-dated 2007. Furthermore, much of the information required for this review was not published as peer-reviewed research but was solely included in policy documents or technical documents outlining the development and use of assessment tools, some of which were not available online. We therefore consulted an external expert in NZ assessment, who provided further information (e.g. e-asTTle manuals unavailable online) on the technical aspects of the reading assessments used to collect data about student learning. The search yielded 19 reading assessments (see Table 1) and 65 documents, consisting of NZ policy documents (e.g. NZ curriculum and NZ literacy progressions), test-related materials (e.g. exemplars, rubrics and cut-score tables) and research publications related to reading assessment in the NZ context (see Table 2).

Table 1. Overview of data material: Tests and assessment tools as described by the New Zealand ministry of education.

	Year level	Standardised?	What does it assess?	Purpose	What measures does it give?
Assessment Resource Bank (ARB) – Reading	3–10	No, but trialled with approximately 200 New Zealand students to establish difficulty levels and reliability.	Activities assess the making of meaning through text. They include some NEMP tasks. Activities are most useful in supporting learning at an identified level and can be used to diagnose individual problems.	The majority of the resources are designed specifically for formative purposes. Some include self and peer assessment.	The main emphasis is on establishing what the next learning step might be, but students and teachers can also compare achievement against the established difficulty levels.
Burt Word Reading Test	2–9	Yes, the test was revised and standardised for use in New Zealand in 1981	Word recognition and decoding skills	A quick and easy assessment of word recognition skills to enable a broad estimate of a child’s reading achievement. It should be used in conjunction with other reading assessments.	Equivalent age bands
e-asTTle Reading	5–10	Yes	Reading comprehension aligned to the New Zealand Curriculum 2007	e-asTTle is a web-based assessment tool developed from successive versions of asTTle on CD-ROM. e-asTTle allows teachers and school leaders to electronically set reading comprehension tests that are aligned to the curriculum, when they want and at the level they want; to analyse results; and to measure student progress over time. Tests can be tailored to the specific needs of the classroom. It gives teachers a rich picture of how well a student, class, or school is doing compared with national average performance and the curriculum requirements (including curriculum levels). It allows comparisons with other groupings such as gender, ethnicity, English as a second language, or ‘schools like mine’.	The tool provides e-asTTle reading scores and curriculum level and sub level equivalents. It provides rich interpretations and specific feedback that relate to student performance (rather than simply providing a score). It identifies areas of student weakness and strength that may otherwise go unnoticed.
Learning Progression Frameworks – Reading	0–10	No, although the progressions of learning and their link to curriculum levels have been established using comprehensive student data	The learning progression frameworks illustrate the significant steps that learners take as they develop their expertise in reading, writing, and mathematics from years 1 to 10.	Teachers can use the frameworks to identify students’ reading, writing, and mathematics knowledge, skills, and attitudes.	Not designed as a formal assessment tool, but instead as an illustration of progressions of learning in reading, writing and mathematics up to year 10. To be used formatively by students and teachers.
Literacy Learning Progressions – Reading	0–10	No. The Literacy Learning Progressions alert teachers to what students need to know, and be able to do, at specific points in their schooling, if they are to engage with the texts and tasks of the curriculum and make the expected progress.	The learning progressions describe the reading knowledge and skills that students need to have developed at specific points in their schooling if they are to engage with the texts and tasks of the curriculum and make the expected progress.	To provide teachers with a professional tool that shows them what reading knowledge and skills their students need to meet the reading demands of the New Zealand Curriculum.	Reading progressions are shown for the early years at school entry, after six months at school, then one, two and three years at school. For the later years, progressions show literacy expectations by the end of year 4, year 6, year 8 and year 10.
Martin & Pratt Non-word reading test	2–10	Yes, Australian normed, 2001	Assesses phonological awareness and skills. Non-words are used because they are equally unfamiliar to all students.	To assess the key word reading skill of phonological recoding to identify students who are low on these skills and will require additional assistance.	It gives a detailed diagnostic picture of each student’s errors, as well as age-standardised score, percentile, and fluency score.
National Standards – Reading	0–8	No. The Reading Standards alert teachers to what students need to know, and be able to do, for each year in their schooling, if they are to engage with the texts and tasks of the curriculum and make the expected progress.	The Reading Standards alert teachers to what students need to know, and be able to do, for each year in their schooling, if they are to engage with the texts and tasks of the curriculum and make the expected progress.	To provide teachers with professional tools that show them what reading knowledge and skills their students need for each year level in order to meet the reading demands of the New Zealand Curriculum.	Reading progressions are described and illustrated for after one, two and three years at school, then for the end of Year 4 through to the end of Year 8.
Neale Reading Analysis, 3rd edition	2–9	Yes, Australian normed, 1999	Text reading fluency, accuracy and comprehension	To measure the accuracy, comprehension and rate of reading, monitor reading progress and obtain diagnostic observations of reading behaviour.	Provides standardised scores, reading ages and national profile levels for reading accuracy and comprehension. This is useful diagnostic information when a student decodes and comprehends at markedly different levels.
NEMP (National Education Monitoring Project) – Reading	4, 8	Yes	A range of assessment tasks in Reading and Speaking is provided from the rounds of testing between 1996 and 2010.	The tasks allow comparison of the performance of a school’s students with national results on some aspects of the curriculum; provide models of good assessment practice; can be modified to suit a school’s own purposes; are highly valid and useful for assessing student learning.	Results can be measured against the results for representative samples of New Zealand school students at successive points in time
PAT (Progressive Achievement Test) – Reading comprehension: Revised 2008	4–10	Yes, in 2007, with standardised administration.	Reading comprehension	The tests reflect current developments in the teaching and learning of reading comprehension and are primarily designed to help classroom teachers determine achievement levels in reading comprehension, and to indicate possible gaps in learning.	Test scores convert to scale scores. They can be compared against national norms and converted into stanines. Progress of individuals and groups can be tracked longitudinally. National norms come from testing done at the beginning of the school year, so if students sit PATs towards the end of the year, it is best to compare their achievement against the national reference group for the next year level.
PAT (Progressive Achievement Test) – Reading vocabulary: Revised 2008	4–10	Yes, in 2007, with standardised administration.	Reading vocabulary	The tests are designed to assess students’ ability to understand the words they read.	Test scores convert to scale scores. They can be compared against national norms and converted into stanines. Progress of individuals and groups can be tracked longitudinally. National norms come from testing done at the beginning of the school year, so if students sit PATs towards the end of the year, it is best to compare their achievement against the national reference group for the next year level.
PAT (Progressive Achievement Tests) – Grammar and punctuation: 2013	4–10	Yes, assessments have undergone a comprehensive standardisation trial with a representative sample of NZ students.	Assesses the students’ ability to recognise and use the grammatical conventions of standard NZ English, including punctuation, in context.	The questions assess the skills students need to make meaning as they read, to write meaningful texts, and to understand the patterns of standard New Zealand English. The questions are aligned with The Literacy Learning Progressions.	Test scores convert to scale scores. They can be compared against national norms and converted into stanines. Progress of individuals and groups can be tracked longitudinally. National norms come from testing done at the beginning of the school year, so if students sit PATs towards the end of the year, it is best to compare their achievement against the national reference group for the next year level.
PM Benchmark Literacy Assessment 1	0–8	No, but texts have been levelled through extensive trialling.	It measures students’ instructional or independent reading levels.	Captures students’ progress in reading fluency, retelling abilities and comprehension within and beyond the text.	Pre level 1: Six assessments that focus on Oral Literacy, Alphabetic Knowledge, Phonological Awareness, Phonics and Concepts about Print. Consists of two cards and one assessment sheet. Instructional or independent reading levels to level 14, reading and reading-age levels thereafter to level 30 using the Fry Readability Formula. Post-Level 30: Six cards for independent readers – reading ages from 12 to 14 years.
PM Benchmarks	1–8	No, but texts have been levelled through extensive trialling	Silent reading for the purpose of oral retelling, oral reading for miscue analysis and comprehension to ascertain the student’s level of understanding	To assess student reading accuracy, fluency, behaviour, and comprehension using unseen texts. Developed from the Price Milburn readers as an assessment tool.	Instructional or independent reading levels to level 14, reading and reading-age levels thereafter to level 30 using the Fry Readability Formula
PROBE 2 (Prose Reading Observation, Behaviour and Evaluation)	3–10	No, although texts have been levelled through trialling	Reading accuracy, reading behaviour, and reading comprehension. It may also be used as a measure of silent reading comprehension and listening comprehension.	Works in determining a reading comprehension level and as a controlled in-depth interview to achieve a greater understanding as to how readers engage with text and what specific teaching is required.	Decoding and comprehension combine to calculate reading age
Progress and Consistency Tool (PaCT) – Reading	0–8	No, although the progressions of learning within the tool and their link to curriculum levels have been established using comprehensive student data.	PaCT is an ‘engine’ that captures teacher ‘best fit’ judgements on aspects of mathematics, reading and writing and recommends an overall judgement that a teacher confirms or reviews.	The PaCT is designed to support teachers to make dependable judgements about their students’ achievement and progress in relation to standards.	The PaCT captures teachers’ decisions and uses psychometric calibrations to recommend an overall judgement in relation to standards.
Running records	1–8	No. Running Records are scored and a conversion rate is used to calculate a percentage accuracy score. According to the number of errors and self-corrections, a mathematical formula is used to determine the level of text difficulty.	A Running Record provides a score of word reading accuracy, an analysis of a reader’s errors and self-corrections, and an analysis of the reading strategies used.	With a running record a teacher can quickly and accurately record what the reader says as they read a text or section of a text aloud. After the reading, the teacher completes an analysis to ascertain specific teaching required to meet the student’s learning needs.	Based on material used for testing a score of word reading accuracy an analysis of a reader’s errors and self-corrections, the reading strategies used helps to calculate the student’s reading age and learning needs.
STAR (Supplementary Test of Achievement in Reading), 2nd edition	3–9	Yes, norms established in Term One, 2011. Standardised administration.	The aspects of reading – learning the code of written language, making meaning of texts (for Years 3 to 9), and thinking critically about texts (for Years 7 to 9). Each test assesses a range of reading skills that correspond closely to the main components of reading skill outlined in The Literacy Learning Progressions (Ministry of Education, 2010).	To supplement the assessments that teachers make about their students’ progress and achievement in reading.	Test scores convert to scale scores. They can be compared against national norms and converted into stanines. Progress of individuals and groups can be tracked longitudinally. There are 10 tests, each designed for a particular year level and time of year. National norms are set from students tested in Term 1. Consequently, if a student is tested in Term 4, then stanines for the following year reference group should be used.
Tests of Reading Comprehension, 3rd edition (TORCH–3)	3–10	Yes, normed in Australia in 2013. Tests are standardised at each year level to allow for continuity of measurement of student progress over time.	Reading comprehension	Identifies reading comprehension levels and can be used to measure progress and help identify those skills requiring further instruction. TORCH Plus is an extension for this edition which was developed in response to requests for more challenging material for upper primary and secondary schools.	Levels of the Victorian (Australia) Learning Standards

Note. Descriptions in this table are quoted verbatim from the TKI website (https://assessment.tki.org.nz/).

Table 2. Overview of data material: Policy documents, peer-reviewed research publications, and technical documents.

Document	Description
(1)2010 Cut-Score Tables. (2010).	Technical document
(2)Benade (2015). Teaching as Inquiry: Well Intentioned, but Fundamentally Flawed. New Zealand Journal of Educational Studies, 50(1), 107–120. https://doi.org/10.1007/s40841-015-0005-0	Research publication
(3)Borderfields Consulting (2009). Literacy learning progressions: Report on an analysis of feedback on the draft. https://www.educationcounts.govt.nz/__data/assets/pdf_file/0005/45338/340178-LLPs-report-on-feedback.pdf	Technical document
(4)Brown, G. T. L (Brown, 2014). AsTTle – A national testing system for formative assessment: How the national testing policy ended up helping schools and teachers. In A Developmental and Negotiated Approach to School Self-Evaluation (Vol. 14, pp. 39–56). Emerald Group Publishing Limited.https://doi.org/10.1108/S1474–7863(2013)0000014003	Research publication
(5)Brown, G. T. L., Irving, S. E., & Keegan, P. J (Brown, 2014). System assessment policies. In An introduction to educational assessment, measurement and evaluation: Improving the quality of teacher-based assessment (Third edition). Dunmore Publishing	Research publication
(6)Describe a moment in time. Writing prompts with specific exemplars (n.d.). https://e-asttle.tki.org.nz/content/download/1567/6301/file/Describe%20a%20moment%20in%20time%20-%20specific%20exemplars.pdf	Technical document
(7)Describe a process. Writing prompts with specific exemplars (n.d.). https://e-asttle.tki.org.nz/content/download/1561/6277/file/Describe%20a%20process%20-%20specific%20exemplars.pdf	Technical document
(8)Douglas, S., New Zealand, Ministry of Education, New Zealand, & Learning Media. (2009). Reading and writing standards for years 1–8. Ministry of Education.	Policy document
(9)Dymoke, S. (2012). Poetry is an Unfamiliar Text: Locating Poetry in Secondary English Classrooms in New Zealand and England during a Period of Curriculum Change. Changing English, 19(4), 395–410. https://doi.org/10.1080/1358684X.2012.736741	Research publication
(10)East, M (East, 2011). Teaching as inquiry: Is investigation into practice of any benefit? New Zealand Language Teacher, 37, 10–11.	Research publication
(11)New Zealand Ministry of Education, (n.d.-b). e-asTTle and assessment for learning. https://e-asttle.tki.org.nz/content/download/1296/5136/file/e-asTTle%20and%20assessment%20for%20learning.pdf	Technical document
(12) e-asTTle Glossary (n.d.). https://e-asttle.tki.org.nz/content/download/1439/5835/file/e-asTTle%20writing%20–%20Glossary%20and%20definitions.pdf	Technical document
(13)e-asTTle Glossary (n.d.). https://e-asttle.tki.org.nz/content/download/2247/13763/file/e-asTTle+Glossary.pdf	Technical document
(14)e-asTTle mathematics (n.d.)	Technical document
(15)e-asTTle Norms (n.d.). https://e-asttle.tki.org.nz/content/download/1229/4816/file/e-asTTle%20norm%20tables%20Sept%202010.doc	Technical document
(16)e-asTTle Norms and Curriculum Expectation by Quarter: Writing (July 2013). (n.d.). https://e-asttle.tki.org.nz/content/download/2034/8595/file/Writing%20Norms%20and%20%20curriculum%20expectations%20(July%202013).pdf	Technical document
(17)e-asTTle Project Team, Visible Learning Lab, Faculty of Education, and The University of Auckland (2008). Generation 2: e-asTTle Year Three External Coordinator Manual. Auckland Uniservices Limited.	Technical document
(18)e-asTTle writing marking rubric (E-asTTle Glossary (writing), n.d). https://docs.google.com/viewer?a=v&pid=sites&srcid=ZGVmYXVsdGRvbWFpbnx0dXRhZXBvcm9wb3JvfGd4OjYxOGE3MzM2ODMzY2NhMTA	Technical document
(19)e-asTTle writing	Technical document
(20)e-asTTle writing: The technical report (E-asTTle Glossary (writing), n.d). https://e-asttle.tki.org.nz/content/download/1574/6323/file/e-asTTle%20writing%20(revised)%20Manual%202012%20(3).pdf	Technical document
(21)e‐asTTle Cut‐Score Tables. Reading, Writing and Mathematics	Technical document
(22)Education Review Office (2009). Reading and Writing in Years 1 and 2. https://ero.govt.nz/sites/default/files/2021–05/readingwriting-y12-dec09.pdf	Technical document
(23)Education Review Office. (2007). The Collection and Use of Assessment Information.	Policy document
(24)Edwards, F., & Ogle, D (Edwards & Ogle, 2021). Data informed leadership: The work of primary mathematics lead teachers in New Zealand. Teacher Development, 25(1), 18–36.https://doi.org/10.1080/13664530.2020.1837217	Research publication
(25)Explain writing prompt with specific exemplars (n.d.). https://e-asttle.tki.org.nz/content/download/1565/6295/file/Explain%20-%20specific%20exemplars.pdf	Technical document
(26)Featonby, A. (2012). The Use of the ‘Teaching as Inquiry Model’ to Develop Students’ Self-efficacy in Literature Response Essay Writing. Kairaranga, 13(1), 24–35.	Research publication
(27)Fowler, M. (2012). Leading inquiry at a teacher level: It’s all about mentorship. Set: Research Information for Teachers, 3, Article 3. https://doi.org/10.18296/set.0391	Research publication
(28)Fowler, M. (2012). Leading inquiry at a teacher level: It’s all about mentorship. Set: Research Information for Teachers, 3, 2–7. https://doi.org/10.18296/set.0391	Research publication
(29)Hattie, J.A.C., Brown, G.T.L., Keegan, P.J., MacKay, A.J., Irving, S.E., Patel, P., Sussex, K., McCall, S., Sutherland, T., Yu, J., Cutforth, S., Mooyman, D., Leeson, H.V., & Campbell, A.R.T (Broadfoot & Black, 2004). Assessment Tools for Teaching and Learning (asTTle) Version 4, 2005: Manual. University of Auckland/Ministry of Education/Learning Media.	Technical document
(30)Irving, S. E., Harris, L. R., & Peterson, E. R (Irving et al., 2011). ‘One assessment doesn’t serve all the purposes’ or does it? New Zealand teachers describe assessment and feedback. Asia Pacific Education Review, 12(3), 413–426. https://doi.org/10.1007/s12564-011-9145-1	Research publication
(31)Ministry of Education (n.d.). Modernisation of the e-asTTle assessment tool. Ministry of Education	Policy document
(32)Ministry of Education. (2011, August). The New Zealand Curriculum Update. 12. https://nzcurriculum.tki.org.nz/content/download/8955/121786/file/NZC-update-12.pdf	Policy document
(33)Ministry of Education. (2013). Literacy and Language Learning. https://nzcurriculum.tki.org.nz/System-of-support-incl.-PLD/School-initiated-supports/Professional-learning-and-development/Literacy	Policy document
(34)Ministry of Education. (2015). Professional learning and development (English medium setting). https://nzcurriculum.tki.org.nz/System-of-support-incl.-PLD/School-initiated-supports/Professional-learning-and-development	Policy document
(35)Ministry of Education. (2020). Teaching as inquiry. https://nzcurriculum.tki.org.nz/Teaching-as-inquiry	Policy document
(36)Ministry of Education. (n.d.-b). Making an overall teacher judgement. Retrieved February 28, 2023, from https://assessment.tki.org.nz/Overall-teacher-judgment/Making-an-overall-teacher-judgment	Policy document
(37)Ministry of Education. (n.d.). Professional learning. Retrieved February 28, 2023, from https://assessment.tki.org.nz/Assessment-tools-resources/Professional-learning	Policy document
(38)Narrate writing prompts with specific exemplars (n.d.). https://e-asttle.tki.org.nz/content/download/1639/6596/file/Narrate%20-%20specific%20exemplars.pdf	Technical document
(39)New Zealand & Ministry of Education (2007). Achievement objectives by learning area: Set of 8 charts.Published for the Ministry of Education by Learning Media.	Policy document
(40)New Zealand & Ministry of Education. (2010). Literacy learning progressions: Meeting the reading and writing demands of the curriculum. http://www.literacyprogressions.tki.org.nz/content/download/143/701/version/1/file/The%20Literacy%20Learning%20Progressions.pdf	Policy document
(41)New Zealand Council for Educational Research (2012). e-asTTle writing (revised) manual. Ministry of Education. https://e-asttle.tki.org.nz/content/download/1574/6323/file/e-asTTle%20writing%20(revised)%20Manual%202012%20(3).pdf	Technical document
(42)NZCER (n.d.). PAT: Reading comprehension and vocabulary. https://www.nzcer.org.nz/system/files/PAT%20Reading_One%20Pager_2017_digital_0.pdf	Technical document
(43)O’Neill, H. J. (2006). Once preferred, now peripheral: Poetry and the national assessment for Year 11 students in New Zealand post-primary schools. English Teaching: Practice and Critique, 5(3), 93–126.	Research publication
(44)Parr, J. M., & Jesson, R (Parr & Jesson, 2020). Relations Between Literacy Research and Practice in New Zealand. The Reading Teacher, 73(6), Article 6. https://doi.org/10.1002/trtr.1868	Research publication
(45)Parr, J. M., & Jesson, R (Parr & Jesson, 2020). Relations Between Literacy Research and Practice in New Zealand. The Reading Teacher, 73(6), 691–696. https://doi.org/10.1002/trtr.1868	Research publication
(46)Persuade writing prompts with specific exemplars (n.d.). https://e-asttle.tki.org.nz/content/download/1566/6298/file/Persuade%20-%20specific%20exemplars.pdf	Technical document
(47)Poskitt, J., & Mitchell, K (Poskitt & Mitchell, 2012). New Zealand teachers’ overall teacher judgements (OTJs): Equivocal or unequivocal? Assessment Matters, 4, 53–75. https://doi.org/10.3316/informit.991841301813335	Research publication
(48)Recount writing prompt with specific exemplars (n.d.). https://e-asttle.tki.org.nz/content/download/1564/6292/file/Recount%20-%20specific%20exemplars.pdf	Technical document
(49)Research and Work Programme Summary Attitude Domains in e-asTTle	Technical document
(50)Sinnema, C., & Aitken, G (Sinnema, 2016). Teaching as inquiry. In D. Fraser & M. Hill (Eds.), The professional practice of teaching in New Zealand(5th edition, pp. 79–97). Cengage Learning.	Research publication
(51)Stringer, P., & Jhagroo, J (Stringer & Jhagroo, 2019). Towards a More Complete Understanding of ‘Teaching as Inquiry’: Perspectives of Beginning Teachers Explored. New Zealand Journal of Educational Studies, 54(1), 39–52. https://doi.org/10.1007/s40841-019-00131-6	Research publication
(52)Structure and Language notes to support the generic rubric (n.d.). https://e-asttle.tki.org.nz/Teacher-resources/Marking-resources-for-e-asTTle-writing#Structure	Technical document
(53)The e-asTTle writing score conversion table – revised April 2015 (n.d.). https://e-asttle.tki.org.nz/content/download/2150/9177/file/Conversion%20table%20April%202015%20–%20rubric%20scores%20to%20scale%20scores.pdf	Technical document
(54)The revised e-asTTle writing tool (n.d.). https://e-asttle.tki.org.nz/content/download/1586/6371/file/e-asTTle%20writing%20tutorial.pptx	Technical document
(55)Timperley, H. S., & Parr, J. M (H. S. Timperley & Parr, 2009). Chain of influence from policy to practice in the New Zealand literacy strategy. Research Papers in Education, 24(2), 135–154. https://doi.org/10.1080/02671520902867077	Research publication
(56)Timperley, H. S., Parr, J. M., & Bertanees, C (H. S. Timperley & Parr, 2009). Promoting professional inquiry for improved outcomes for students in New Zealand. Professional Development in Education, 35(2), 227–245. https://doi.org/10.1080/13674580802550094	Research publication
(57)Tunmer, W. E., & Chapman, J. W. (Eds.). (2015). Excellence and Equity in Literacy Education. Palgrave Macmillan UK. https://doi.org/10.1057/9781137415578	Research publication
(58)Visible Learning Lab, Faculty of Education, & The University of Auckland (2009). Generation 2: e-asTTle Year Three Educator Manual. Auckland Uniservices Limited.	Technical document
(59)Visible Learning Lab, Faculty of Education, & The University of Auckland (2010). External Co-ordinator Manual. e-asTTle Fitness for National Standards Stage Gate 1. Auckland Uniservices Limited.	Technical document
(60)Visible Learning Lab, Faculty of Education, & The University of Auckland (2010). Student Manual e-asTTle Fitness For National Standards Stage Gate 1 23 July 2010. Auckland Uniservices Limited.	Technical document
(61)Visible Learning Lab, Faculty of Education, & The University of Auckland (2010). Educator Manual e-asTTle Fitness For National Standards 20 September 2010. Auckland Uniservices Limited.	Technical document
(62)Wilson, A., & Lai, M. K. (2019). Leading Literacy Change Through Collaborative Analysis of Practice. Journal of Adolescent & Adult Literacy, 62(4), 455–458. https://doi.org/10.1002/jaal.916	Research publication
(63)Wilson, A., Madjar, I., & McNaughton, S (Wilson et al., 2016). Opportunity to learn about disciplinary literacy in senior secondary English classrooms in New Zealand. The Curriculum Journal, 27(2), 204–228.https://doi.org/10.1080/09585176.2015.1134339	Research publication
(64)Writing prompts for all purposes (n.d.). https://e-asttle.tki.org.nz/content/download/2356/14352/file/The%20Writing%20Prompts%20for%20all%20purposes%20Aug%2021.pdf	Technical document
(65)Years and Curriculum Levels (n.d.). https://nzcurriculum.tki.org.nz/content/download/1110/11995/file/Charts1.pdf	Policy document

In qualitative research, multiple data analysis strategies are available, such as the use of a priori codebooks and analyst training, or emergent processes in which coders develop a stable perspective or an interpretive consensus (Levitt et al., 2018). We reasoned that the contextual nature of our study, as well as the assessment culture perspective, required an abductive approach to reasoning and analysis (Tavory & Timmermans, 2014). Abduction typically draws on existing knowledge to generate hypotheses or assumptions about new or surprising phenomena (Nubiola, 2005; Råholm, 2010). Abductive approaches in qualitative research are typically generative in nature, seek to form tentative inferences about phenomena without establishing generalisable rules, and lend themselves well to forming conjectures about highly dynamic environments in which beliefs are formed and justified (Lipscomb, 2012). An abductive stance therefore aligns well with our goal of connecting empirical analyses and critical discussion of the NZ assessment culture.

We conducted a qualitative content analysis of the sources. We developed specified categories in an interactive process, emphasising public justification through quotations, rather than quantification and seeking inter-rater reliability (Drisko & Maschi, 2016, p. 6). Throughout the coding process, we discussed alternative codes and coding strategies and resolved disagreements. This interpretative mode of inquiry involved constant reading and re-reading of sources and descriptive memos, besides requiring considerable contextual insight. In our case, all authors have spent time visiting NZ schools and research institutions to familiarise themselves with the issues present in the NZ assessment culture. Two of the three authors have a non-NZ background (the first author is Norwegian, and the third is a Chinese international student) but live or have lived in NZ for longer or shorter periods of time. The second author is a naturalised NZ citizen. We take particular interest in the use of data in assessment cultures.

Following the overall abductive logic in our analysis, our coding approach was relatively open, so that our portrayal of the phenomenon could be informed by the sources reviewed and our theoretical framing. The coding process consisted of three phases: an initial stage of reading and categorising documents; a second phase of writing content analytical memos contextualising the documents in relation to the NZ system and to our theoretical framework; and a third phase in which three main categories were developed. Our abductive approach allowed us to explore the data in light of a range of theoretical concepts related to assessment cultures, such as the validity of reading assessments, the impact of accountability mechanisms or teachers’ professional learning.

In the initial coding stage, we read the documents and divided them into four main categories: reading assessments (see Table 1), research publications, policy documents (typically aimed towards a broader audience) and technical documents (aimed towards a narrower audience, such as test developers or practitioners) (see Table 2). We then wrote extensive descriptive memos to create what Saldaña (2015) terms an ‘organized and categorized narrative portrait of the environment for further analysis’ (p. 104). For example, we considered the relationship between the technical nature of test development as represented in the technical documents (e.g. discussing cut scores, validity issues and answer formats), and the policy messages advising teachers on the collection and use of reading assessment data as represented on government websites (e.g. discussing the range of data used, such as teacher judgement, national assessment results and student voice; the duration and format of professional learning programmes; and training information describing assessment administration and the use of data to support teaching and learning). We also contacted two professors specialising in assessment and reading instruction (one in NZ and one in Norway) to resolve our questions about details regarding the assessment system or about reading assessment at various stages.

The content analysis led to the development of three main categories describing the features of the NZ assessment culture: (1) how reading assessment data are conceptualised and used; (2) how the representation and use of reading assessment data are interrelated; and (3) how policy initiatives seek to emphasise improvement over accountability purposes. We elaborate on the categories in the findings section and justify our analysis through direct quotations from the source material. Drawing on standards for reporting humanities-oriented educational research (American Educational Research Association, 2009), we substantiate our claims by providing critical and contextualised descriptions and interpretations.

Results

Reading assessment data are broadly conceptualised and used for multiple purposes

New Zealand policy conceptualises reading assessment data broadly as a source of information for schools in the wider sense. Teachers are required to collect a variety of evidence of student learning to ‘compile a comprehensive picture of the areas of progress, areas requiring attention, and what a student’s progress looks like’ (Ministry of Education, n.d.-c, p. 1). National authorities provide resources illustrating how reports generated by specific assessment tools, such as e-asTTle, the PATs and the Supplementary Test of Achievement in Reading (STAR), are interpreted; and how knowledge of key statistical concepts (e.g. percentage, mean and disaggregation) required to make sense of numerical data are disseminated (Ministry of Education, n.d.-d). Data are intended to be used by teachers for multiple purposes involving actors in the school community, as stated by the ERO (2007, p. 3):

When teachers have rich information about what their students know, can do and need to do next, they are able to involve students as active participants in their learning and assessment of their own learning. They are also in a position to consult parents and the school’s communities about students’ progress.

The ERO provides numerous examples of assessment information that can be collected by teachers, including knowledge gained from parents, knowledge drawn from teachers’ day-to-day interactions with students, results from teacher-designed classroom and school-wide tests, assessment at school entry or transition points, and results from national standardised assessment tools (ERO, 2007, p. 4). Consequently, the broad conceptualisation of assessment data and their multiple purposes are framed as tools that exist primarily to support teachers’ professional practice and discretion.

To aid practitioners in reading and interpreting data, the NZ MOE has developed Te Kete Ipurangi (TKI, ‘the online knowledge basket’ https://www.tki.org.nz/About-this-site/About-Te-Kete-Ipurangi), the key public sector search tool for resources for schools and communities. The website provides a list of assessments available for practitioners. Some assessments are designed specifically for national use (e.g. e-asTTle, the Burt Word Reading Test, STAR and PATs) and use a representative sample of NZ students for norming (e.g. e-asTTLe), while others have been developed in the Australian context (e.g. the Martin and Pratt Non-word Reading Test, the Neale Analysis of Reading Ability and the Tests of Reading Comprehension). Some instruments have undergone trialling to establish difficulty levels and reliability (e.g. reading resources in the Assessment Resource Bank [ARB], an online portal containing assessment resources and support materials for teachers). Others are intended for classroom use. For example, running records assessment, such as the Prose Reading Observation, Behaviour and Evaluation assessment, is used to score students’ reading accuracy and reading comprehension. This provides teachers with information to which they can adapt their instructional strategies. A resource developed by the MOE, Using running records (Ministry of Education, 2000), remains the standard for how to administer the practice (although the resource is no longer produced, it is still available from libraries in NZ). Although classroom resources are not always standardised, some have been tested for reliability and difficulty levels (e.g. the ARB). Consistent with the ERO’s conceptualisation of data and their use, the TKI website contains instructive resources explaining how teachers can use different kinds of quantitative data displays.

The comprehensive picture of students’ learning and progress is aggregated into an overall teacher judgment (OTJ). A factsheet describing the process of gathering, interpreting and using assessment information for the purpose of assigning OTJs opens with a key message on validity: ‘No single source of information can accurately summarize a student’s achievement or progress’ (Ministry of Education, n.d.-b, p. 1). Three ways of gathering evidence are outlined in the factsheet: conversing with the student, observing the process the student uses, and gathering results from formal assessments, including standardised tools. The factsheet further discusses assessment purpose, difficulty and dependability (i.e. validity and defensibility); the role of student participation; and differentiating instruction based on assessment results. The OTJ system thus requires teachers to draw on a complex array of data (formal and informal, quantitative and qualitative) and to navigate formally defined aspects of assessment (e.g. standards, pedagogical content knowledge, curriculum and assessments) with socially interpreted aspects (e.g. professional conversations and moderation processes) (Poskitt & Mitchell, 2012). Furthermore, to ensure the consistency of teacher judgements in OTJs, social moderation and support, such as shared assessment criteria and exemplars, may be required (Smaill, 2020).

The collection of data is related to the TAI approach in the NZ curriculum, which requires teachers to use diverse assessment data. The system affords many ways of collecting data through both quantitative (e.g. test scores, attendance data and student engagement data) and qualitative (e.g. observations, video recordings, and students’ and parents’ perceptions) means. Accountability is achieved through regular school inspections from the ERO. In sum, the TAI approach of NZ positions data ‘at the heart of this process, as all decisions and actions that are taken must be linked to evidence’ (Edwards & Ogle, 2021, p. 4). This runs counter to accountability systems in other national contexts, in which the focus is on collecting information for control purposes and pressuring schools to improve performance data (Lingard & Lewis, 2016).

The TKI website presents a four-stage cycle for ‘gathering and using evidence for learning’ (Ministry of Education, n.d.-d), which emphasises the use of a wide selection of data types. For example, the Working with data subpage describes the key concepts and tools for using data: cleaning, sorting and merging data; disaggregating data; reliability and validity; and mean, median and standard deviation. The Types of data subpage describes three categories of data: nominal, ordinal and interval. Interestingly, there is no explicit mention of qualitative data, such as observations, memos and photographs. Nevertheless, the website also instructs readers that ‘Test scores should not be used in isolation to calculate the achievement level of a student’, that ‘[a]ny test is a snapshot’ and that results should be considered together with a range of evidence (Ministry of Education, n.d.-d).

The national curriculum explicitly emphasises contextual variability as a professional challenge for teachers, and not as a system issue to be managed through accountability measures: ‘Since any teaching strategy works differently in different contexts for different students, effective pedagogy requires that teachers inquire into the impact of their teaching on their students’ (Ministry of Education, 2007, p. 35). Similarly, researchers argue that evidence-based instruction is not a matter of implementing a research-informed solution; rather, instruction is seen as contextualised and ‘a response to the students in front of a teacher as well as to wider curriculum, pedagogical, and assessment influences and constraints’ (Parr & Jesson, 2020, p. 692). In sum, the broad conceptualisation and multiple purposes of data suggest that the NZ assessment culture supports both classroom learning and teachers’ professional learning.

The representation of reading assessment data is located in an improvement-oriented environment

In the NZ assessment culture, reading assessment data are primarily represented in ways that support teachers in identifying students’ strengths and weaknesses and in tailoring their instruction to these strengths and needs. This includes the ways in which scores are represented, the support provided for interpreting test results, and the encouragement to combine quantitative information with classroom observations and other kinds of narrative data. For example, test developers remind teachers that the process of collecting data consists of ‘triangulating information from various sources’ (Visible Learning Lab, 2010, p. 12). Similarly, the ARB Teacher View provides curriculum connections, advice on how to work with students and links to related resources (including printable documents and options for saving online resources to personal folders). This is in contrast to other countries that have focused more on high-stakes assessments and top-down accountability systems (Irving et al., 2011). New Zealand assessment is characterised as ‘voluntary, low-stakes and in the control of the teacher and school leader’ (Brown, 2014, p. 43). Thus, we conjecture that the environment suggests that the improvement of teaching and learning is the overall goal.

An in-depth example of how representation is improvement oriented can be seen in the e-asTTle reading assessment. The assessment generates web-based tests within parameters chosen by the teacher, including curriculum level and area. Test results can be displayed as box-and-whisker diagrams showing comparisons with national achievement norms, and can be configured to identify the curriculum outcomes that groups of students have or have not achieved. Progress reports allow teachers to track students’ learning over time, and the system allows teachers to generate a What Next Profile, which shows an average group curriculum level within each curriculum function. Information is also provided on students’ strengths and gaps in learning. Furthermore, e-asTTle reports allow for multiple report formats, supporting the communication of information to a variety of audiences, including principals, parents and students, for a variety of purposes (see Brown et al., 2018; Hattie et al., 2006). All these representations of test data provide teachers with an expectation of use; in planning their next instructional steps, they should use assessment data to understand how well students perform relative to other students and to curriculum expectations.

At the micro level of classrooms and schools, the emphasis on improvement is evident from how teachers are explicitly expected to consider individual and collective student learning outcomes and to evaluate the effects of their teaching on that learning (Parr & Jesson, 2020). The TAI approach requires teachers to consider a) their priorities for their students’ learning; b) the strategies which would be most likely to help students learn; and c) what happened as a result of the teaching and the implications for future teaching (Sinnema & Aitken, 2016). The role of improvement is also noticeable in the framing of reading assessment tools. This involves collecting baseline data on students’ ability to interact using language, enabling teachers to explore whether the students gained proficiency, confidence or competence and identifying the meaning of this evidence for future teaching practice (East, 2011). For example, the Progress and Consistency Tool (PaCT) captures teachers’ best fit judgements on aspects of reading, writing and mathematics and recommends an overall judgement that a teacher confirms or reviews (Ministry of Education, 2019). Similarly, the PAT reading test converts test scores to scale scores that can be compared against national norms and converted into stanines, so that both individuals and groups can be tracked longitudinally.

At the macro level, national policies offer substantial support with reading, interpreting and communicating data from multiple sources for improvement purposes. Policy documents provide suggestions for whole-school in-depth work, co-constructing tailored programmes for professional learning and development (e.g. creating a literacy strategic plan, conducting school-wide literacy assessments, using the TAI process, using student achievement data, implementing processes for moderation, and conducting activities, such as learning conversations with leaders, teachers and students, workshops and cluster meetings) (Ministry of Education, n.d.-b). Literacy support programmes are designed as school-wide inquiry cycles to be adapted to the identified needs of schools as determined by the analysis of data, information, teaching practices and annual plan targets (Ministry of Education, n.d.). The term ‘literacy policy’ could therefore be considered a misnomer in the NZ context, as the level of prescription is relatively low; in reality, policymaking occurs at all levels of the system, as national intentions are re-translated by actors at each layer (H. S. Timperley & Parr, 2009).

However, some aspects of teachers’ improvementwork are less well emphasised in the NZ system Questions remain regarding discrepancies between different kinds of assessment data, how teachers can engage with inquiry and how certain curriculum areas are represented. First, there is an emphasis on quantitative data in the tools and guidelines provided in the NZ assessment culture. Although there are numerous mentions of the importance of qualitative data (e.g. Ministry of Education, n.d.), few instructions on how to represent and interpret such information are available.

Second, the TAI model has been criticised for lacking methodological consistency (i.e. how TAI compares to approaches such as practitioner research or action research) (Benade, 2015) and for being difficult to use for new teachers (Stringer & Jhagroo, 2019). In conjunction with the lack of information on how to analyse qualitative data, the TAI approach could lead to variability in the ways in which new teachers (or teachers arriving from outside NZ to teach) engage with data as part of their instructional decision making.

Third, although NZ policies offer substantial support in reading, interpreting and communicating data, we noted that the emphasis is primarily on generic aspects of literacy (e.g. basic vocabulary, phonological awareness or reading comprehension) and less on the disciplinary-specific aspects of literacy (the specialised ways of reading associated with different curriculum areas). Most of the assessments listed in Table 1 focus on generic reading. For example, STAR is administered as a series of sub-tests assessing different aspects of reading ability (word recognition, sentence comprehension, paragraph comprehension and vocabulary range. The years 7–9 range does cover aspects of reading that are slightly more related to disciplinary issues. The Tests of Reading Comprehension (TORCH-3) comprise 16 reading passages in a variety of text types, including both fiction and non-fiction, and E-asTTle similarly provides data on reading comprehension across curriculum strands and objectives. Nevertheless, disciplinary aspects of literacy were not particularly present in the material we reviewed. Expectations of students' disciplinary literacy are high in secondary school, but students in low-SES classrooms have relatively few opportunities to engage with complex authentic texts, extended discussions or activities intended to develop critical literacy (Wilson et al., 2017). Therefore, there is a need to address the role of disciplinary literacy, given the high-performance, low-equity profile (OECD Ed., 2014; Parr & Jesson, 2020) and the well-documented achievement gaps between NZ student groups (Wilson et al., 2016).

The use of reading assessment for improvement purposes involves multiple actors and levels of organisation

New Zealand policy focuses on supporting multiple actors (teachers, school leaders and parents) at all layers of the interpretation and implementation of policies and initiatives (H. S. Timperley & Parr, 2009). Attempting to understand how reading assessment is positioned within the larger assessment culture therefore requires a contextualised view of the actors at different levels of the schooling system. Here, we outline the roles of these actors and the interactions between different levels of organisation, and also present the support they receive to use assessment for improvement purposes.

Teachers wield considerable discretion through the self-governing policy and development of a local curriculum. According to the documents reviewed, the MOE offers substantial resources for teachers relating to educational assessment, including opportunities to participate in professional learning activities. These opportunities are intended to support continuous improvement and include online resources, such as teacher self-review, and planning and reporting resources; how to share information with parents; and resources designed to target data use specifically (including how to use evidence for learning, taking participants through the processes of data gathering, manipulation and analysis, and explaining how to transform results into targeted teaching (New Zealand Ministry of Education, n.d.-a). The national emphasis on improvement is therefore given weight by providing opportunities for professional development and not through more prescriptive policy implementation means.

School leaders take part in improvement-oriented activities as part of their ongoing support for teachers. A curriculum update document explains how school leaders must align performance management systems with supporting teachers in improving their practice; effective school leaders are envisioned as responsible for a school culture promoting trust and responsibility, and they allow for fallibility and encourage persistence (Ministry of Education, 2011). School leaders are also supported in using assessment data for school improvement purposes, with assessment and reporting guides providing guidance for schools that are seeking to review their curriculum review systems and processes, and school evaluation indicators (New Zealand Ministry of Education, n.d.-a).

School boards are less visible in the materials reviewed in this study, possibly due to their more strategic role in long-term planning (Ministry of Education, 2023). However, school boards have responsibilities in setting local curricula appropriate for local community needs (Vannier, 2012). Given the equity issues related to literacy in NZ, we conjecture that in many cases this may include reading assessments.

At the highest level of policymaking, the MOE is responsible for the overarching frameworks for reading assessment, while formal qualifications assessments are framed by the New Zealand Qualifications Authority. However, the role of the Ministry is framed by the three traits in the assessment culture of NZ described in the introduction section (i.e. self-governing schools, the relation between the national and local curricula, and teachers’ professional discretion). Furthermore, the task of evaluating and reporting on education and care for students is performed by the ERO, which is separate from the Ministry (Education Review Office, n.d.). Although the ERO produces documents evaluating teachers’ ability to use reading assessment data for improving their classroom instruction and environments, such documents recommend that teachers, school leaders, school boards and the Ministry use these reports for support and improvement purposes (Education Review Office, 2009, p. 3). Thus, although the Ministry is responsible for governance and coordination at the national level, many of the core responsibilities remain the tasks of schools (Vannier, 2012).

The particular features of the NZ assessment culture mean that multiple layers of the system are required to interact. For example, the methods for delivering professional development outline a range of interaction efforts (e.g. co-construction of literacy strategic plans; school-wide assessment in literacy; teaching as an inquiry process; use of student achievement data; implementing processes for moderation; observations and learning conversations with leaders, teachers and students; workshops; and cluster meetings) (Ministry of Education, n.d.-a). Interaction is also visible in middle-level leaders’ responsibilities for mentoring teachers and facilitating the reciprocal sharing of inquiries using data among teachers (Fowler, 2012). The TAI approach, with its focus on using assessment data for school improvement and teacher professional learning purposes, further reinforces this aspect.

The documents we reviewed indicate that the interaction is traceable through formal policy documents, research reports and evaluations, and technical documents for assessment specialists. There is evidence of interaction as part of deliberative processes in which academics and practitioners provide insights to policymakers on the development of literacy learning progressions (Borderfields Consulting last, 2009). There is also considerable output of technical documents describing the interaction between levels of the system for how reading assessment is conducted. Technical manuals describe roles and responsibilities for actors at various levels of the system, for example, how assessment data are accessed and used in the interaction between schools and professional development providers, school cluster coordinators or researchers (e-asTTle Project Team, Visible Learning Lab, Faculty of Education, & The University of Auckland, 2008). Other documents describe the relation between assessment tools, such as e-asTTle, and assessment approaches, such as assessment for learning (New Zealand Ministry of Education, n.d.-a). This includes glossaries aimed at ensuring consistency in terminology (E-asTTle Glossary, n.d.; E-asTTle Glossary writing, n.d.), as well as research literature discussing assessment at the system level (e.g. Brown, 2014; Brown et al., 2014).

For reading specifically, the emphasis on the interaction between multiple organisational layers for improvement purposes is visible in the provision of professional learning activities for language and literacy teachers. School leaders are responsible for directing resources for professional development services for literacy instruction and for providing guidance for literacy learning (Ministry of Education, n.d.-a). Guidelines outline how literacy and language teachers can accelerate student achievement for underprivileged groups (Ministry of Education, n.d.-a). The description of the public system of support (Ministry of Education, n.d.-a) includes the following description of professional learning and development in literacy and English language learning:

• ‘the development of teacher knowledge about the value and importance of linking learning at home with learning at school

• a focus on ensuring that all parents and whānau are supported to understand where their children are currently at in their learning, and what their next learning steps will be

• a focus on the importance of culturally responsive learning and teaching that values the cultures, identities, and languages of all learners and their parents, families, whānau, and communities’. (Ministry of Education, n.d.-a)

Furthermore, teachers are expected to interact with English language assistants. Separate guidelines describe professional development initiatives for English language assistants and regional facilitators supporting English language learning in more than 300 NZ schools (Ministry of Education, n.d.-a).

Finally, the TAI approach emphasises the need to connect with students’ families and cultural belonging as part of teachers’ professional development, for example, by engaging with whānau, a Māori-language word often translated as ‘family’, although its meaning is more complex and includes the physical, emotional and spiritual dimensions (Walker, n.d..). These characteristics undergird the role of improvement in the broader assessment culture. In conclusion, there is a strong focus on equity issues, such as tackling variability across student groups and connecting with students’ lives outside the school context, and on issues relating to cultural values and identities.

Conclusion

The aim of this study was to explore the role of reading assessment in the NZ assessment culture. The findings suggest that NZ is an example of an assessment culture that supports students’ learning and teachers’ professional learning. New Zealand policies communicate a broad conceptualisation of reading assessment data. Assessment resources which align with broader features, such as the local curriculum policy, the TAI approach, and policy suggestions for how to use data for improvement purposes, are available to NZ teachers. These findings echo Birenbaum’s (2016) suggestion that teachers in improvement-oriented assessment cultures believe that assessment drives teaching and learning for both students and teachers. They also suggest that the improvement-oriented assessment culture of NZ, with its emphasis on local curricula, an inquiry mindset, and considerable interaction between multiple organisational layers, should be relatively resistant to policy changes based on polarised views of reading.

However, the findings also suggest that there is a slight mismatch between the curriculum aims in years 6–9 and the assessments provided for teachers. The emphasis on disciplinary literacy in the curriculum is not sufficiently covered by the assessments currently available. This suggests that teachers do not have access to assessment information about students’ disciplinary literacies, effectively hindering TAI for both student and teacher learning purposes. Having such information is important because there is evidence that some teachers are unwilling to engage with evidence without support and resources (Lai & Sinnema, 2022). In other words, although policymakers seek to support an improvement-oriented culture in schools, the reading assessments used by teachers may not lead to improvement in reading. Awareness of tensions between these components can uncover unintended and potentially nonconstructive policy levers.

Studying assessment cultures requires researchers to remain sensitive to how curricula and assessment policies frame teachers’ practice and professional learning, and to explore how system components such as improvement efforts and accountability mechanisms interact. We acknowledge that studying an assessment culture through policy documents and research publications has limitations, as this methodology does not capture the tacit knowledge or sense-making involved in the processes of designing, understanding or implementing such documents, nor does it adequately capture potentially contested interpretations. We further acknowledge that reading is only one part of the curriculum and that its development is also connected with other literacy dimensions beyond the scope of this study, such as writing, vocabulary, and speaking and listening skills. Moreover, this research has only focused on the English medium aspect of the NZ education system, excluding the Māori medium aspects. Nonetheless, the findings presented here offer insights into how macro-level policies governing reading assessment can shape meso-level programmes and micro-level practice in schools, and can provide valuable input for future policymaking.

Currently, the NZ curriculum is being revised. Literacy and numeracy standards in the New Zealand Qualifications Authority are changing and the ERO is adopting new methodologies. A new strategy proposed in 2022 distinguishes clearly between the generic and content-specific aspects of literacy, for example by underlining how critical reading in English contrasts with other subjects (MOE, 2022). A more prescriptive literacy curriculum paired with professional learning initiatives for teachers could ensure that some of the equity problems for low-SES or minority students are mitigated. However, a prescriptive approach might also undermine teachers’ professional autonomy and local freedom, traits which remain strong in the NZ assessment culture. We encourage policymakers to consider the balance of prescription and local freedom in ongoing reforms so that the different qualities of NZ’s assessment culture can be well aligned, and so that issues related to equity of opportunities and outcomes can be managed within the existing cultural context.

Acknowledgments

The authors wish to thank Gavin T. L. Brown, Lisbeth M. Brevik and the anonymous reviewers for their support.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the University of Auckland’s Faculty Researcher Development Fund.

Notes on contributors

Henning Fjørtoft

Henning Fjørtoft is a professor of Norwegian didactics at the Department of Teacher Education of the Norwegian University of Science and Technology. His research interests include literacy instruction, assessment and professional development in education, with a particular focus on secondary education.

Mei Kuin Lai

Mei Lai is the director of TePūtahi | Woolf Fisher Research Centre and the director of Learning Schools, Tui Tuia | Learning Circle, Faculty of Education and Social Work at the University of Auckland, New Zealand. She is an associate professor at the same faculty. Her research focuses on research–practice partnerships to improve student learning outcomes, with a focus on how teachers analyse and use data in professional learning communities.

Mengnan Li

Mengnan Li is a researcher at the New Zealand Council for Educational Research. As a former teacher, Mengnan explores educational issues from a social-psychological perspective. She has experience in providing research and analysis to support the development of innovative teaching and learning frameworks that drive equitable outcomes for all learners.