Bringing underprivileged middle-school students to the opera: cultural mobility or cultural compliance?

Abstract

This article assesses the impact of a two-year long project-based learning program conducted by the National Opera of Paris in a large number of middle schools located in underprivileged areas, aiming at preventing school dropout and tackling educational inequalities by providing disadvantaged students with the opportunity to discover the world of opera. Taking a counterfactual approach (propensity score matching), we measure the impact of participation in the program on final exam and continuous assessment grades. The analysis displays mixed results: a significant and positive impact for the students who participate in the program for its whole duration (two years), at least for continuous assessment scores, but a negative impact for those who leave the program after only one year. The contrast between the effects of full and partial participation in the program suggests that these may be primarily due to a selection effect in favor of the most culturally and socially compliant students, in line with Bourdieu’s and Passeron’s reproduction theory (1997 [1970]) rather than a mobility effect (DiMaggio, 1982) resulting from the transfer of cultural capital to disadvantaged students.

Keywords:

• Project-based learning
• middle school
• statistical matching
• mixed method
• cultural capital

Acknowledgements

This work originates in a research agreement between the National Opera of Paris (NOP) and the French National Foundation for Political Sciences (FNSP). The authors would like to thank Pierre-Nicolas Morin and Yannick Savina for their contribution to the formatting of the data used in the analyses presented in this article, Héloïse Fradkine for her coordination of the fieldwork, and Sylvie Le Laidier for her help in accessing the FAERE database managed by the French Ministry of Education. We would also like to thank our contacts at the Paris Opera, the middle-school students and their teachers we met during the fieldwork, as well as those students from the Master of Sociology program at Sciences Po who were involved in conducting interviews and making observations.

Disclosure statement

No potential conflict of interest was reported by the authors.

A2. A brief presentation of multi-valued treatment methods used in our study

Evaluation methods are generally based on the comparison of a group of persons (here, students) who benefit from a program or an intervention (hereafter called the treatment group), and a group of comparable persons or students who do not benefit from this program (hereafter called the control group). We denote $Y_1$ the result on a test score for students belonging to the treatment group and $Y_0$ the result on the same test for students in the control group. It is not possible to directly measure the difference $Y_1-Y_0$ (called the causal effect of the treatment) between the two results for the same student, since, for a treated student, we can only observe the score $Y_1$, while for a control student, we can only observe the score $Y_0$. Moreover, the score of a treated student could be different from what her score would be in the absence of the treatment (the so-called counterfactual), and vice versa for an untreated student. This difficulty stems from the fact that students who benefit from the treatment may be selected based on characteristics which differ from those of students who are not beneficiaries. The simple difference between the average test score of students in the treatment group and that of students in the control group is therefore potentially subject to a selection bias. A least squares regression of the outcome on the treatment indicator and observable explanatory variables is also potentially biased since this procedure ignores the selection issue. In general, least squares estimates do not provide consistent estimates of causal parameters like the average treatment effect – ATE -, or the average treatment effect on the treated – ATET -.¹³

The statistical analysis reported in Section VI is based on a multi-valued treatment evaluation model with four treatment states. We recall that, in our study, treatment states are denoted as follows:

• $T_0=1$ if the student does not participate at all in the program, which corresponds to the sequence (0,0), $T_0=0$ otherwise,

• $T_1=1$ if the student participates in the program in the first year but not in the second, which corresponds to the sequence (1,0), $T_1=0$ otherwise,

• $T_2=1$ if the student does not participate in the program in the first year but participates in the second year, which corresponds to the sequence (0,1), $T_2=0$ otherwise,

• $T_3=1$ if the student participates in the program for two years, which corresponds to the sequence (1,1), $T_3=0$ otherwise.

The multi-valued treatment framework generalizes Rubin’s model (1974, 1977) introduced for the case of a unique treatment. In our case, there are 4 treatments, denoted (0,0), (1,0), (0,1) and (1,1), respectively. The assignment to treatment $j\in \left[\left(0,0\right), \left(1,0\right), \left(0,1\right),(1,1), \right]$ is indicated by the binary variable $T_j$ (j = 0, 1, 2, 3; see above). Consequently, there are 4 potential outputs (scores), denoted $Y_0,Y_1,Y_2, \text{and}\mathrm{ }{Y}_3$ both for the final exam and the continuous assessment in 9th grade.

The validity of the method is based on a crucial assumption known as the ignorability assumption: this assumption states that the distributions of potential outcomes are independent of the treatment variables¹⁴ $T_j, j=0,1,\dots ,3,$ conditional on confounding variables X (i.e. covariates), namely: $${(Y}_0,Y_1,Y_2,\mathrm{ }{Y}_3)\coprod {(T}_0,T_1,T_2,\mathrm{ }{T}_3)⌊X$$

Under this assumption, it can be shown that, for any student $i$ with potential outcomes $\left(Y_{0,i},Y_{1,i},Y_{2,i},Y_{3,i}\right)$ either at the final exam or at the continuous assessment in 9th grade:${\Pi }_j\left(X_i\right)=Pr\left(T_{j,i}=1X_i\right)$

$$\left(Y_{j,i},Y_{j^{\mathrm{\text{'}}},i}\right)\coprod {(T}_{j,i},T_{j^{\mathrm{\text{'}}},i})⌊\left({\Pi }_j\left(X_i\right),{\Pi }_{j\mathrm{\text{'}}}\left(X_i\right)\right)\text{for any }j\mathrm{\text{'}}\ne j,\mathrm{ }j\mathrm{ }\text{and}\mathrm{ }j\mathrm{\text{'}}\in \left[\left(0,0\right),\mathrm{ }\left(1,0\right),\mathrm{ }\left(0,1\right),(1,1),\mathrm{ }\right]$$with ${\Pi }_j\left(X_i\right)=Pr\left(T_{j,i}=1X_i\right)$ and ${\Pi }_{j\prime }\left(X_i\right)=Pr\left(T_{j^{\prime },i}=1X_i\right)$. This last result helps to estimate the average treatment effects on the treated (ATETs) defined as $\mathrm{E}\left[Y_j-Y_{j\prime }|T_j=1\right]$ for any $j\prime \ne j$ (in particular, $j^{\prime }=0)$, especially when using IPW (inverse probability weighting) estimates (see below).

The three types of multi-valued treatment estimators that we use (see Section IV) are defined as follows. The first is the regression adjustment (RA) estimator. To understand how this estimator is calculated, consider two ordinary least squares (OLS) regressions $Y_{j,i}=X_i{b}_j+u_{j,i}$ for $T_{j,i}=1$, and $Y_{j^{\prime },l}=X_l{b}_{j\prime }+u_{j\prime ,l}$ for $T_{j^{\prime },l}=1$. We denote ${\widehat{b}}_j$ and ${\widehat{b}}_{j\prime }$ the OLS estimates of $b_j$ and $b_{j\prime }$ and the corresponding predicted outcomes ${\widehat{Y}}_{j,i}=X_i{\widehat{b}}_j$ and ${\widehat{Y}}_{j\prime ,i}=X_i{\widehat{b}}_{j\prime }$.The RA estimate of the ATET, defined as $\mathrm{E}\left[Y_j-Y_{j\prime }|T=j\right]$, is simply:

$$\frac{{\sum }_1^{n}1\left[T_{j,i}=1\right]Y_i}{{\sum }_1^{n}1\left[T_{j,i}=1\right]}-\frac{{\sum }_1^{n}1\left[T_{j^{\mathrm{\text{'}}},i}=1\right]{\mathrm{ }w}_i\left[j,j\mathrm{\text{'}}\right]\mathrm{ }{Y}_i}{{\sum }_1^{n}1\left[T_{j^{\mathrm{\text{'}}},i}=1\right]{\mathrm{ }w}_i\left[j,j\mathrm{\text{'}}\right]}$$

$$\frac{1}{n_j}{\sum }_{i=1}^{n_j}{X}_i\left({\widehat{b}}_j-{\widehat{b}}_{j\mathrm{\text{'}}}\right)\mathrm{with}\mathrm{ }{n}_j={\sum }_1^{n}1\left[T_{j,i}=1\right]$$

The second estimator is the inverse probability weighting (IPW). In this case, under the ignorability assumption, the ATET may be estimated by: $$\frac{{\sum }_1^{n}1\left[T_{j,i}=1\right]Y_i}{{\sum }_1^{n}1\left[T_{j,i}=1\right]}-\frac{{\sum }_1^{n}1\left[T_{j^{\mathrm{\text{'}}},i}=1\right]{\mathrm{ }w}_i\left[j,j\mathrm{\text{'}}\right]\mathrm{ }{Y}_i}{{\sum }_1^{n}1\left[T_{j^{\mathrm{\text{'}}},i}=1\right]{\mathrm{ }w}_i\left[j,j\mathrm{\text{'}}\right]}$$ where $w_i\left[j,j\prime \right]=\widehat{\mathrm{Pr}}\left(T_{j,i}=1X_i\right)/\widehat{\mathrm{Pr}}\left(T_{j^{\prime },i}=1X_i\right)$. Each control unit is weighted by the reciprocal of the probability of receiving the treatment $j^{\prime }$ it receives relative to the probability of receiving the target treatment $j$ Units in control group $j^{\prime }$ with a relatively high probability to be in the treated group $j$ get larger weights since they are most representative of the target treatment group $j$. And reciprocally.

With the third estimator, namely the IPWRA (inverse probability weighting with regression adjustment) estimator, the ATET is estimated as: $$\frac{{\sum }_1^{n}1\left[T_{j,i}=1\right]X_i{\widehat{b}}_j}{{\sum }_1^{n}1\left[T_{j,i}=1\right]}-\frac{{\sum }_1^{n}1\left[T_{j^{\mathrm{\text{'}}},i}=1\right]{\mathrm{ }w}_i\left[j,j\mathrm{\text{'}}\right]X_i{\widehat{b}}_{j\mathrm{\text{'}}}}{{\sum }_1^{n}1\left[T_{j^{\mathrm{\text{'}}},i}=1\right]{\mathrm{ }w}_i\left[j,j\mathrm{\text{'}}\right]}$$ where $$w_i\left[j,j\mathrm{\text{'}}\right]=\widehat{\text{Pr}}\left(T_{j,i}=1X_i\right)/\widehat{\text{Pr}}\left(T_{j^{\mathrm{\text{'}}},i}=1X_i\right).$$

A3. IPWRA estimates of the average treatment effect on the treated (ATET).

Table A1. Sample description.

Notes

1 The counterfactual analysis of the impact of a given intervention (an educational program, a set of measures, etc). on a given outcome implies a statistical analysis based on counterfactual thinking. The ‘counterfactual’ measures what would have happened to beneficiaries in the absence of the intervention, and impact is estimated by comparing counterfactual outcomes to those observed under the intervention.

2 Students may be enrolled in sixième and cinquième (i.e. 6th and 7th grades, or years 7 and 8), cinquième and quatrième (7th and 8th grades, or years 8 and 9) or quatrième and troisième (8th and 9th grades, or years 9 and 10). Practically, the second modality (cinquième and quatrième) is the most frequent. Almost half of the 49 schools under consideration in our analysis (23 schools) enrolled their students in the program in cinquième and quatrième), 16 in sixième and cinquième, and 10 in quatrième and troisième

3 The agreement with the National Education Ministry allowed us to make an indirect matching with the FAERE dataset, identifying the classes and schools involved in the program each year.

4 In practice, it proved quite difficult to locate the students some ten years later, and when this was possible, we were confronted with an obvious problem of selection bias. It is highly realistic to imagine that those we were able to locate and who were open to sharing their views and memories about the program were not at all representative of the whole student body formerly involved in the program. They were probably more likely to be those who still had positive feelings about the program, or at least those who were most influenced by the experience.

5 Unfortunately, the dataset we use does not contain information about dropout rates, or other outcomes mentioned under arts education and problem-based learning.

6 For a detailed distribution by schools, see Appendix A1.

7 These methods are, for instance, exposed in detail by Imbens and Rubin (2015), or in a more synthetic way by Fougère and Jacquemet (2021).

8 For this purpose, we used the Stata package teffects.

9 In the absence of any reliable information on racial belonging and/or origin of the students and their families in our dataset, we will not test for any hypothesis related to this dimension of student identity and background.

10 The unit of measure is the standard deviation of the score distribution.

11 In France, foreign language learning begins in 6th grade, and students begin learning a second foreign language in 8th grade. Most pupils choose English as their first language. But traditionally, good pupils from privileged backgrounds tend to choose German as their first language and English as their second language. Not choosing English as the first language in 6th grade can therefore be considered as an indirect indicator of academic and/or social selection.

12 Of course, the observations made in 2017 and 2018 shed light on the effects identified about the implementation of the same program 10 years ago, but obviously do not apply directly to what occurred at this time.

13 The ATE is defined as the difference $Y_1-Y_0$ of potential outcomes averaged over the entire sample. The ATET is defined as the difference $Y_1-Y_0$ of potential outcomes averaged over the subsample of treated students.

14 A confounding variable is a variable that influences both the outcome (i.e. the effect) and the treatment (i.e. the cause).

15 The socio-economic background of the parents is measured by the occupation of the household head, as reported in the FAERE database. It results in a rather rough four category classification: small self-employed encompasses craftspeople and shopkeepers’ occupations; middle class encompasses managerial and professional positions, as well as teachers, lawyers or doctors together with supervisor, technicians, nurses, etc. i.e. all non-working class salaried workers; the quite large working class category includes all manual and non-manual semi or non-qualified jobs; finally, the residual inactive or unemployed category includes all household head with no occupation, be they retirees, constantly or temporarily unemployed, as well as house persons.

16 For the sake of simplification, the working-class category is aggregated with the unemployed and inactive category, as the latter excludes retirees and concerns those who have never had any employment. Retirees and the formerly employed and unemployed are classified according to their former occupation.