Plausible values and their use in efficiency analyses with educational data

ABSTRACT

There is extensive literature focused on the evaluation of efficiency in the education sector, both at the micro level, analyzing the performance of students or schools, and at the macro level, exploring the behavior of regions or countries. This type of studieshas been driven by exploiting data available in international large-scale assessments, where output measures are usually represented by the so-called plausible values, understood as a representation of the range of the abilities of each student. In this study, we analyze the different options available to incorporate these plausible values in applied studies focused on measuring efficiency and how the results obtained can be affected according to the selected criterion. To do this, we assess the efficiency of Spanish schools participating in PISA using the two most common methodologies in this field: data envelopment analysis and stochastic frontier analysis and considering three different proxies for the educational output: (i) a single plausible value; (ii) an aggregate measure calculated from the ten plausible values available; (iii) an average of the estimates made with the ten plausible values separately. The main conclusion derived from our results is that there are hardly any differences between in the estimates made with different strategies.

KEYWORDS:

• Efficiency
• educational evaluation
• plausible values
• PISA
• data envelopment analysis
• stochastic frontiers

Acknowledgements

J. Aparicio and L. Ortiz thank the grant PID2019-105952GB-I00 funded by Ministerio de Ciencia e Innovación/Agencia Estatal de Investigación/10.13039/501100011033. Jose M. Cordero also acknowledges the support from the Spanish Ministry of Science and Innovation and the State Research Agency through grant ECO2017-83759-P/AEI/10.13039/501100011033. He would also like to express his gratitude to the “European Regional Development Fund (FEDER). A way to make Europe” and to Junta de Extremadura through grant GR18106.

Disclosure statement

No potential conflict of interest was reportedby the authors.

Notes

¹ See Cordero, Cristobal, and Santín (2018) for a review of recent works that use data from the three assessments or Hopfenbeck et al. (2018) for the specific case of PISA.

² Reading proficiency is assessed in PISA and PIRLS, while mathematics and science are assessed in TIMSS and in PISA. In addition, it should be noted that, in its latest waves, PISA has added other additional subjects in its assessments, such as financial literacy and problem-solving skills (since 2012) or the so-called global competence (since 2018).

³ See De Witte and López-Torres (2017) for a review of this literature.

⁴ This limitation on testing time is based on considerations with respect to reducing student burden, minimizing interruptions in the school schedule and other financial and/or time constraints.

⁵ This model corresponds to the version that assumes variable returns to scale in production (Banker, Charnes, and Cooper 1984) and output orientation, that is, one is interested in maximizing the results obtained from the available inputs.:

⁶ For a detailed explanation of this procedure, see Bogetoft and Otto (2011).:

⁷ In fact, 980 Spanish schools participated in the 2015 edition, since all the Autonomous Regions decided to take part with a larger sample so that their results were internationally comparable. However, in our study we have preferred to use only the subsample of 201 schools that represents the country as a whole.

⁸ The same indicators (or very similar ones) are also used as inputs in school efficiency assessments conducted by Thieme, Prior, and Tortosa-Ausina (2013), Agasisti (2014), Aparicio, Cordero, and Pastor (2017) or Agasisti and Zoido (Agasisti and Zoido 2018, 2019).

⁹ The ESCS indicator is an index created by PISA analysts based on the information provided by students about the educational level and occupation of their parents and the educational resources and cultural possessions available at home.

¹⁰ The values of the ESCS and SCHRES variables had to be rescaled (adding the minimum value of each variable) to ensure that all units presented positive values.

¹¹ The estimates of the efficiency levels have been obtained through the Benchmarking package (Bogetoft and Otto 2015) developed for R software (R Core Team 2020) and RStudio (R Studio Team 2020).

¹² This procedure requires normalizing all the outputs by means of one of them. In our case, we have used science as a reference, so that the adjusted parametric model would be the following: $x_0$

¹³ To calculate the statistic, the npdeneqtest function of the np library (Hayfield and Racine 2008) developed for the R software (R Core Team 2020) and RStudio (R Studio Team 2020) has been used.

Additional information

Funding

The work was supported by the Government of Extremadura: GAEDAF Research Group [GR18106]; Spanish Ministry of Science and Innovation and the State Research Agency [ECO2017-83759-P/AEI/10.13039/501100011033,PID2019-105952GB-I00/ AEI / 10.13039/501100011033].