Can we assess teaching quality on the basis of student outcomes? A stochastic frontier application

ABSTRACT

This paper proposes a new application of Stochastic Frontier Analysis (SFA) for estimating the student performance gap and how this can be used to assess changes of teaching quality at the individual unit-of-study level (module-level). Although there have been other examples in the literature that assess ‘efficiency’ in student outcomes, this is the first study that proposes the use of SFA specifically at the module level and with the goal of creating an aggregate measure of ‘quality’, thus avoiding the known issue of the statistical inconsistency of unit-specific SFA estimates. A case study is presented on how the approach can be applied in practice, with discussion on potential implementation issues. This paper is targeted to academics and policy makers that are interested in the quantitative assessment of student outcomes and specifically to those who want to assess how changes in module structure and/or delivery have affected said student outcomes.

KEYWORDS:

• Educational production function
• stochastic frontier analysis
• student attainment gap
• teaching quality
• higher education

Disclosure statement

No potential conflict of interest was reported by the author.

Notes

1 A module is defined as an individual unit of study that lasts a certain amount of time and has specific and clearly stated learning aims, a specified delivery method that sets out how the learning material will be delivered and a specified assessment that is constructively aligned with the learning aims. Each student is awarded a grade at the end of the module based on the assessment and the individual module grades are aggregated at the end of the programme of study to derive a student’s overall degree grade.

2 The prevalence of these distributions in practical analysis is mainly their ease of use; both are one-parameter distributions so if one moment of the distribution is known, all the other moments of interest can be derived analytically. For a comprehensive discussion on the possible distributions that can be used for the decomposition, see Greene (2008).

3 Non-parametric approaches can accommodate discrete variables in the analysis by creating different model for each discrete group. However, modules/courses have usually relatively small sample sizes, which could result in some groups being too small. Additional complications arise when there are multiple discrete variables and the analysis wants to test how they interact.

4 Statistically consistent estimates of ui are possible in the panel setting, but only under the assumption that ui remains unchanged for the duration of the analysis, which is very restrictive in general and inappropriate in the setting of this study.

5 Note that Equation (1) can be formulated so that: $u_i=\left(\mathrm{r}\mathrm{e}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{s}\mathrm{e}\mathrm{d}\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{e}{\mathrm{s}}_i/\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{a}\mathrm{l}\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{e}{\mathrm{s}}_i\right)$. In this case, the educational production function should be expressed using a logarithmic functional form, such as the Cobb-Douglas or the translog and $u_i$ is expressed as a multiplier that converts realised outcomes to ideal outcomes of vice versa.

6 In the UK, students typically study for A-levels, a subject-based qualification, after they completed their secondary education and before applying for a university place. All universities require certain A-levels achieved or equivalent qualifications.

7 A-levels are typical split into two parts, each assessed separately. Students that qualify for the first part only are awarded an AS-level qualification.

8 Other indicators of academic performance could also be appropriate here, such as graduation mark. This study adopts the AvMark indicator to preserve the immediacy of the analysis (i.e. the analysis can be undertaken immediately after the module has concluded and assignments are marked). It is also arguable that a single-year average is more relevant in this case, as it more starkly captures transient effects on student performance specific to the year in question that the model does not directly measure (due to lack of data on individual student circumstances).

9 General-to-specific iterates on the general model by removing a single insignificant variable at each stage, re-evaluating the model and repeating the variable removal step until all remaining variables are statistically significant (moving from a general model specification to a specific one that includes only statistically significant variables). If there are multiple insignificant variables in a given step, the process selects the variable with the highest p-value for removal.

10 Most EU students are taught Mathematics to up their last year of secondary education, while UK students that don’t study a quantitative subject for their A-levels end their engagement with the subject when they finish their GCSE’s, which is usually at 16 years of age. The hypothesis that the quality of Mathematics education in UK at secondary level is somewhat lagging behind its EU counterparts is unlikely to be valid, as the PISA results for 15 year olds finds that UK student performance in Mathematics is very close to the average of the OECD countries (OECD 2013).

11 Models 4-I and 4-ii include the exact same variables as those used the more general model 3 and adopt the same functional form specification (linear). In the case of MoM, the model coefficients are also identical by construction. For MLE, there are some very small differences in the coefficients between Model 3 and Model 4, due to the change from OLS estimation to Maximum Likelihood estimation. However, the same set of variables that were statistically significant in under Model 3 are also statistically significant under Model 4.