Using Contextual Data for Education Quality Assessment

Abstract

The article describes an approach to assessment of the performance of educational institutions with regard to their social context. To develop this approach, the authors were guided by (1) the results of numerous studies corroborating correlations between student performance and contextual factors (both in Russia and abroad); (2) the experience of foreign colleagues working with similar problems; and (3) The ideas of availability of a baseline minimum of information required to make these kinds of assessments possible in Russia. The idea behind the proposed assessment model is that with the necessary data consistent correlations between student performance and contextual factors can be empirically identified. Researchers usually establish these correlations using a multiple regression analysis. The results of the analysis, established empirical correlations, may be used to contextualize formal progress (i.e., to adjust expectations for institutions working in more or less favorable conditions). Two possible applications of this information are discussed: one based on a formula and the other on an index specially developed by the authors for measuring school's social prosperity. Finally, possible constraints associated with these models and the contextual factors for assessing the quality of education in Russia are analyzed.

Acknowledgements

We express our gratitude to the organizers and participants of the Russian Training Center webinar at the Institute of Education Management RAO and the seminar “Current Research and Development in the Field of Education” at the Institute of Education HSE. The organizers and participants were instrumental to formulating many of the ideas expressed in this article.

When preparing the article, we tried to take into account all the comments and criticisms relating to the practical application of the education management models presented below.

Notes

English translation © 2015 Taylor & Francis Group, LLC, from the Russian text © 2014 “Voprosy obrazovaniia.” “Ispol'zovanie kontekstnykh dannykh v sisteme otsenki kachestva obrazovaniia: opyt razrabotki i aprobatsiia instrumentariia,” Voprosy obrazovaniia, 2014, no. 4, pp. 58–95. Gordey Yastrebov (candidate of sciences, sociology) is a senior research fellow at the Laboratory for Comparative Analysis of Post-Socialist Development, National Research University, Higher School of Economics, and a doctoral student at the European University Institute (Florence, Italy). Email: [email protected]. Marina Pinskaya (candidate of sciences, pedagogy) is a leading research fellow at the Center of Social and Economic School Development, Institute of Education, National Research University, Higher School of Economics. Email: [email protected]. Sergey Kosaretsky (candidate of sciences, psychology) is the director of the Center of Social and Economic School Development, Institute of Education, National Research University, Higher School of Economics. Email: [email protected] by Peter Golub.

^1. For instance, the public hearings, “Equal Opportunity for Student Education,” held by the Public Chamber of the Russian Federation for the Development of Education (June, 2012) (Froumin et al. 2012).

^2. Specifically, the Ninth International Scientific-Practical Conference: “Trends in Education: Different, but Equal. How to Bridge the Gap in Access to Quality Education” (February 17–18, 2012).

^3. Resolution of the Government of the Russian Federation (August 5, 2013. No 662), “On the Implementation of Education System Monitoring.” June 10, 2015. < www.rg.ru/2013/08/19/monitoring-site-dok.html>.

^4. ibid.

^5. In 2007, a draft of the National Education Quality Assessment System (see: osoko.ru) was prepared, but has not yet been fully implemented.

^6. All information relating to schools and assessment procedures are in the public domain and can be found at the UK Department of Education: < www.education.gov.uk/schools/performance/>.

^7. In particular, data are collected on parental occupation and education, primary family language, indigenous relation, and school location.

^8. A poverty index is used to calculate the funding school meals plans.

^9. For additional information about the contextual information used in education statistics in other countries see our previous publication (Iastrebov et al. 2013, pp. 194–95) and the thematic data of the Organization for Economic Co-operation and Development (OECD 2008, pp. 135–37).

^10. The website of the project is www.kpmo.ru. Until the fall of 2014, the monitoring data of almost every school of the Russian Federation were in the public domain. Currently, access is only possible for registered users, or indirectly through a map of educational organizations in Russia (www.educationmap.ru/). The reliability of the data within the system and the adequacy of the methodology of calculating the indicators are the subject of expert discussions.

^11. It represents indicators characterizing the financial and economic activities, infrastructure, and staffing of educational institutions, as well as some other information, such as class size, class number, and ratio of teachers to students.

^12. A similar procedure can be adapted to identify potentially high-efficiency and low-efficiency schools using other criteria of academic achievement, as it is clear that a comprehensive assessment of a school's quality will require the use of an expanded list of criteria. The extent of possible criteria can be formed by using the following parameters, which can be calculated for individual sets of students: (1) indicators of differentiation of national testing results—USE; (2) promoting power (Balfanz and 2004), which is calculated as a percentage of students who successfully transition from the 9th to 10th grade; (3) percentage of prizes from Olympiad competition and other contests (other indicators of educational and extracurricular achievements); (4) percentage of students who matriculate to higher education institutions.

^13. The ideal case for developing better models is, of course, a national sample of schools, or at least its representation on the basis of a random or some other representative sampling.

^14. Since the model demonstrated here describes the difference in USE indicators for one group of students (information on USE results are taken at the school one year prior to graduation) it is significantly simplified. In this situation, the sociodemographic composition of the students should be judged not by the entire student body of school, but only according to the relevant issues; however, the data contained in the social passport of the school make this impossible. This problem is easily resolved if all the characteristics are counted individually. Also in this case it is advisable to take into account the gender composition of the student body, which can also affect the average result due to a higher average of achievement among girls (Voyer and Voyer 2014).

^15. Since not all schools fill in the information on the social passport correctly, a smaller number of explanatory variables in the model allows us to include a larger number of educational institutions. If these variables are present in the model and if the connection between them is strong enough, the evaluation of the relevant factors may be biased and as a result, it becomes impossible to accurately determine the contribution of these variables in the characteristic oscillations of the explanatory trait.

^16. One of the main problems in modeling relationships using multiple regression analysis is the potential statistical link (correlation) between the explanatory variables.

^17. Direct information about the financing of educational institutions was not used in this case, but it can be taken into account in the construction of similar models.

^18. What, unfortunately, cannot be ruled out in the light of the results of the well-known 2013 USE results.

^19. See the main results of PISA 2012 ( < www.centeroko.ru/public.htm>).

^20. For a detailed discussion of different international education programs and projects implemented at the national and regional level, see the thematic collection of articles edited by Froumin et al (2012).