‘What works’ depends: teacher accountability policy and sociocultural context in international large-scale surveys

ABSTRACT

Despite a growing emphasis in education policy on ‘what works for whom and in what circumstances’, there is still considerable attention to decontextualised ‘best practices’ that emerge from cross-country comparisons of student achievement. Also, while operational and even political aspects of context are increasingly incorporated into policy research, there is relatively little attention to the relationship between sociocultural context and education policy. In this paper, I explore the extent to which national sociocultural context influences the relationship between one aspect of policy – teacher accountability – and student outcomes. I do so by using multilevel modelling to analyse international survey data on education (from PISA 2012, PISA 2015, and TIMSS 2015) matched at the country level with survey data on culture (from the World Values Survey and Hofstede’s IBM study). I find that one of the sociocultural constructs significantly and consistently moderates the relationship between teacher accountability and student outcomes, suggesting that some teacher accountability approaches may be beneficial in certain sociocultural contexts but detrimental in others. This finding implies a need for caution in generating universal policy prescriptions from international assessments such as PISA and TIMSS. It also strengthens the case for viewing teacher accountability as a socioculturally embedded process.

KEYWORDS:

• Teacher accountability
• sociocultural context
• international large-scale student assessments
• PISA
• education policy

Acknowledgments

I am grateful to Panayiotis Antoniou, Ricardo Sabates, Geoff Hayward, William C. Smith, Jason Silberstein, and three anonymous reviewers, as well as participants at the EARLI SIG 18 & 23 conference in 2018 and CIES in 2019, for valuable feedback on different iterations of this analysis.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Supplementary material

Supplemental data for this article can be accessed here.

Notes

1. There is also a related question of the degree to which these reorientations of teacher practice improve student outcomes, but my working assumption for the purposes of this paper is that the most salient levels of context shaping the pathway from teacher practice to student outcomes are the classroom, community, and household, rather than national sociocultural context.

2. There did not appear to be systematic differences between the cases that had data on at least some accountability items – and, hence, were included in the analysis – and the 5.3% of cases that were excluded because they did not have data on any accountability items.

3. I also ran models with each sociocultural construct entered singly for each cut of the PISA 2015 and 2012 data, which are not shown in the table. Results were broadly consistent with those shown in (Table 1), with four exceptions, one of which related to civic norms: for the PISA 2012 full sample, the interaction between Accountability_c and civic norms was insignificant, unlike in the model with all six sociocultural constructs. However, it was similarly negative in direction.

4. An additional set of graphs showing predicted scores across all combinations of the 10th, 50th, and 90th percentiles each for civic norms, ESCS, and GDP is available in Figure S1 in the supplemental online material.

5. A related concept is legal scholar Lynn Stout’s (2010) ‘unselfish prosocial behavior’ (p. 99). Stout argues that unselfish prosocial behaviour can be eroded by an overemphasis on material reward and punishment because this sends the signal that it is appropriate to make decisions based on material and other selfish factors, thus crowding out altruistic justifications. This aligns with the association suggested in (Figure 2), i.e. that more managerial accountability instruments may backfire in contexts with strong civic norms.

Additional information

Funding

The research in this paper was conducted as part of a PhD funded by the Gates Cambridge Trust (BMGF OPP1144).

Notes on contributors

Yue-Yi Hwa

Yue-Yi Hwa is a research fellow at the Research on Improving Systems of Education (RISE) Programme at the Blavatnik School of Government, University of Oxford.