Quality Assurance and Evaluation through Brazilian lenses: an exploration into the validity of umbrella concepts

ABSTRACT

In our present research we address the question of whether it is valid to apply the Quality Assurance and Evaluation (QAE) umbrella concept, which was formulated to explain new phenomena in European educational governance, to similar developments in Brazilian basic education. This led us to reflect on the possible pitfalls and potential strengths of using umbrella concepts as analytical tools. This article presents this exploration and its operationalisation. We confronted in-built assumptions in QAE with the contested, consensual and creative use of the notion of quality in Brazilian basic education, and looked for relationships. Our analysis shows that the Brazilian developments reiterate the relationships concerning global interconnectivity, and challenges those pertaining to conformity. We argue that the main risks of using umbrella concepts seem to concern the re-production of understandings, which frequently leads to the disregarding of deviation.

KEYWORDS:

• Brazil
• quality
• basic education
• umbrella concepts
• evaluation
• measuring
• large-scale assessments
• World Bank
• SAEB
• CAQI

Acknowledgements

We would like to thank Eneida Shiroma, Susanne Ress, Nadine Bernhard and Cristina Alarcón for their comments on an earlier draft of this article.

Disclosure statement

No potential conflict of interest was reported by the authors.

Notes on contributors

Vera Gorodski Centeno is a Post-doctoral Researcher at the University of Tampere.

Jaakko Kauko is Associate Professor at the University of Tampere.

Helena H. D. Candido is a research fellow at the University of Tampere and a Ph.D. student at the University of Helsinki.

Notes

1. This article contributes to the analysis of educational developments in Brazil within the “Transnational Dynamics in Quality Assurance and Evaluation Politics of Basic Education in Brazil, China and Russia” project (Academy of Finland grant 273871). The main research objective is to explore how the intertwinement of local, sub-national, national, regional and global scales constructs the local dynamics in QAE politics, and thus shapes local learning environments in the case countries.

2. The Fabricating Quality in European Education project was funded as part of the ‘Eurocores’ programme of the European Science Foundation. It comprised linked national projects in Denmark, Finland, the UK (England and Scotland) and Sweden, and was co-ordinated by the Centre for Educational Sociology, University of Edinburgh (see the project website for further information: http://www.ces.ed.ac.uk/research/FabQ/index.htm).

3. KNOWandPOL was a project funded by the European Union. As of 20 July 2009, the reports and further information are available on the project website (http://www.knowandpol.eu).

4. We are adopting the UNESCO International Standard Classification of Education, ISCED 2011 (UNESCO, 2012). Basic education corresponds to primary education (ISCED level 1, first stage of basic education), and in some countries also lower secondary education (ISCED level 2, second stage of basic education). This differentiates elementary from pre-primary (early childhood) education, and from (upper) secondary and higher education. However, basic education in Brazil officially includes pre-primary ('educação infantil’, ISCED level 020) and upper secondary (‘ensino médio’, ISCED level 3) education, thereby corresponding to compulsory education (4 to 17 years old). Alternatives to regular schooling are also included in basic education. Given the vast scope of basic education in Brazil, we have narrowed down our focus to regular elementary education (‘ensino fundamental’).

5. From 1990 until 2005, the assessment suffered several methodological changes. The most significant concerned the adoption of the Item Response Theory in the design and analysis of the tests in 1995. In 2005 the SAEB started to combine two large-scale assessments. (i) The National Evaluation of Basic Education (ANEB) was the first assessment implemented. It aims ‘to evaluate the quality, equity and efficiency of Brazilian education’ (INEP n.d., para. 2). It assesses a sample of pupils from the 5th, 9th and 12th grades, aiming at providing information to educational management. (ii) The National Evaluation of School Performance (ANRESC, ‘Prova Brasil’) has the goal of ‘evaluating the quality of education in public schools’ (INEP n.d., para. 3). It is applied to all fifth- and ninth-grade pupils in public schools and focuses on each school and municipality. Since 2013 the SAEB includes a third large-scale assessment. (iii) The National Evaluation of Literacy (ANA) targets the ‘assessment of the literacy in the Portuguese language, mathematical literacy and the conditions of the Literacy Cycle offered in public networks’ (INEP n.d., para. 4). The ANEB and the ANRESC are combined with the data from the School Census for the calculation of the Development Index of Basic Education (IDEB). The IDEB was created along with the 2007 Education Development Plan, and has a double aim of measuring the educational quality on all levels (from school to national), and of monitoring the national average educational achievement.

6. The totality of the research data was collected during 2014–2016, and it included document analysis and literature reviews, as well as interviews and observations. The semi-structured interviews (N = 90) were conducted from mid-2015 to beginning 2016, on the international (n = 7), national (n = 23), state (n = 21), municipal (n = 12), school (n = 27) levels. We privileged individual interviews, but at the request of the participants a few group interviews were also conducted (N = 101 interviewees). We conducted interviews with representatives from the major IOs active in Brazil, politicians and officials from government bodies at the three levels of policy-making (national, state and municipal levels) important national and state expert agencies and individuals, relevant national and state third-sector organisations, private organisations, school staff, parents and pupils. The computer-assisted (ATLAS.TI software) qualitative content analysis followed a mix of deductive (concept-driven) and inductive (data-driven) logics (Schreier 2014). For this article we use the preliminary analysis of almost all national interviews.

7. In this account, we also adopt these commonly accepted historical antagonistic stances for the purpose of clarification.

8. Beisiegel was one of the few scholars writing about quality in the 1970s. The educational community came back to his ideas and texts in the 2000s (e.g. Gusmão 2010; Cabral and Di Giorgi 2002; Oliveira and Araujo 2005). Beisiegel re-published his 1975 article, together with others, in a 2006 compendium, noting that the ideas were still very relevant (Beisiegel 2006).

9. It is interesting that Brazil was an early adopter not only of the application of managerial practices to education (TQM), but also of the institutionalisation of systematic performance-assessment schemes. This could be attributable to the long historical involvement of IOs in Brazilian education (see Centeno 2010; Kauko et al. 2016).

10. In their view, the historical problems reflect how the educational community understood education quality over time. For instance, the problems of school access in the 1940s and of school dropouts in the 1980s indicate that education quality meant, respectively, the expansion of school access and the regularisation of student flow in those historical periods. Policy-makers and civil-society actors (interview data) also embrace this renewed meta-narrative.

11. For the purposes of this paper we focus only on CAQi, despite the existence of CAQ and other pupil-cost instruments. See the proposal website for a brief clarification of the differences between the two mechanisms, http://www.custoalunoqualidade.org.br.

12. The Campanha (Brazilian Campaign for the Right to Education, official translation, see Campanha 2010) was born in a small room in the headquarters of another civil-society actor, the Ação Educativa (interview data), the aim being to gather ‘different political forces, to prioritize mobilization actions, political pressure and communication’ (http://www.campanhaeducacao.org.br). Created in 1999, it is now a ‘plural network of more than 200 teachers unions, social movements and civil-society organizations that aim to grant every citizen its right to a quality public education’ (Campanha 2010, 4).

13. Interview data show that Campanha was in fact closely involved in the elaboration of the SINAEB. As an interviewee states: “ … [For the] Campanha … the bigger movement … [it has] taken [it] a long time to construct that vision of the SINAEB … the National Evaluation System of Basic Education, that was a painful process [for the Campanha] … with a big effort … ” (BR-NNGO).

Additional information

Funding

This work was supported by the Academy of Finland under [grant number 273871].