Is it worth attending higher education? Lessons from a systematic review on institutional contribution to learning outcomes

Abstract

There is a growing interest in understanding more precisely the contribution of HE to student learning outcomes. We present a systematic review of the literature published between 2000 and 2023 in the WoS and Scopus databases. We identified 49 research articles that analyse individual and institutional variables related to student’s academic and employment outcomes. Most focus on establishing non-causal relationships between students’ entry characteristics and outcomes. In parallel, work estimating institutional contribution has evolved. Until 2015, there was an interest in comparing different methods and outcome variables. Since then, there has been more interest in using causal techniques to identify factors related to institutional contribution. Methodologically, VA research shows variations in results depending on the type of outcome used and the inclusion of institutional selectivity in the estimates. In addition, the results show gains in student learning; however, academic gaps related to gender, SES, or ethnicity remain or even increase after HE completion. In parallel, some value-added research has identified institutional variables that may be related to academic outcomes, such as expenditure per student, type of teachers’ contract, and teaching practices such as discussions outside the classroom and participation in research projects. However, the predominance of descriptive techniques in the literature does not allow conclusive results.

Keywords:

• Higher education
• learning outcomes
• value-added
• institutional contribution

REVIEWING EDITOR:

• Sammy King Fai Hui, The Education University of Hong Kong, Hong Kong

SUBJECTS:

• Education Policy & Politics
• Higher Education
• Research Methods in Education

Introduction

Higher education (HE) has attained great importance since the mid-20th century, which has been reflected in a sustained increase in enrolment and public and private spending (OECD, 2021). In OECD countries, enrolment among 20–25 year-olds reached 41% in 2021, representing an increase of 4% from 2005 to date. The average public and private expenditure on HE in OECD countries is 17,100 USD per student or 37% of GDP per capita (OECD, 2021).

The economic literature has studied the impact of education on various labour outcomes for decades (Ehrenberg, 2004). Findings show that more years of education could increase the probability of finding a job and access to higher salaries (Becker, 1994). However, economic research from the outset noted that outcomes were not the same for everybody. Factors like students’ socioeconomic characteristics, prior academic performance (e.g. Chetty et al., 2020), the type and quality of higher education institutions (e.g. Kane & Rouse, 1995), or selection bias (e.g. Card, 2001; Black & Smith, 2004), are associated with graduate outcomes.

Despite progress in economic research, its focus has been mainly on analysing the relationship between education and economic returns (e.g. access to the labour market, wages). So, there is limited evidence regarding the effectiveness of the educational process, that is, the actual improvement of students’ learning associated to higher education (Koretz, 2019).

School Effectiveness Research studies has focused on analysing the effect of educational institutions on student learning outcomes (David et al., 2000). However, the significant progress observed in understanding primary and secondary education has not been observed in higher education (Koretz, 2019).

Characteristics of HE, such as the curricular diversity of study programs or the unfrequent availability of large-scale test results (Koretz, 2019), have hindered the advancement of value-added and school effectiveness research at the tertiary level.

Despite these difficulties, this area of research has gained momentum in the last decade due to increased concern about whether students are achieving the learning outcomes expected in areas such as analytical abilities and communication skills (Coates & Zlatkin-Troitschanskaia, 2019; Kinzie, 2019). An example of this has been the increased availability of tests such as SaberPRO in Colombia, the CLA in the United States, or the OECD's AHELO project.

This paper presents the results of a systematic literature review of the research that addressed the effect of higher education institutions on the learning outcomes of graduates using quantitative methodology. In our review, we aim to answer two questions: What is the contribution of higher education institutions to student learning outcomes? (Q1) And how was this contribution achieved? (Q2).

We analysed 654 articles in the WoS and Scopus databases published between 2000 and 2023. This article is organised into a theoretical framework section, methodological strategy section, and results section, followed by a discussion and a conclusion section, in which we outline possible lines of future research.

What is the institutional contribution?

The concept of contribution arose from economics and referred to the difference between the value of production and the cost of raw materials (Braun et al., 2010). This analysis framework was introduced early in the 1970s in the wake of the human capital theory and the concept of production function (Hanushek, 1979).

In education, this analysis framework seeks to understand better which school inputs (teacher salaries, teaching experience, training, number of students per classroom) are crucial for generating learning outcomes and economic returns. However, the transposition from economics to education has been somewhat inexact, as the contribution is less tangible when measuring changes in learning outcomes than when measuring changes in the value that result from the production process (Braun et al., 2010).

In education, the institutional contribution refers to the relative contribution that teachers, institutions, or specific programs make to various educational goals, considering other factors that are not controlled by the institutions, like socioeconomic level, race, or gender (Kim & Lalancette, 2013). Typically, this contribution has been estimated by comparing students’ academic performance on standardised tests from two or more points in time, using value-added models that consider the influence of these other factors in the gain estimate (OECD, 2008). From an equity perspective, these considerations are relevant, as they make comparisons fairer. However, some counter-arguments suggest that to understand more precisely the contribution of institutions to learning outcomes, it is crucial to understand the factors within HEIs that generate inequitable outcomes (Carter & Reardon, 2014). It requires more than just statistically controlling for factors associated with unequal educational outcomes (e.g. socioeconomic level, race, or gender).

Independent of criticism, studying the contribution of institutions to student outcomes has advanced our understanding of the influence of education on people’s lives. Today, we understand that more years of schooling are related to productivity and that specific institutional characteristics add variability to this relationship (Hox et al., 2017); however, this knowledge comes from the school system.

Understanding how much of the difference in students’ learning is attributable to the higher educational institution and not to other factors is a significant methodological challenge (OECD, 2008). In addition, learning is naturally cumulative (Braun et al., 2010), and it has a social component (Engeström, 1987) as it results from previous teachers, classmates, educational administrators’ decisions, and the students’ extra-educational experience (Braun et al., 2010). Thus researchers disagree on how pertinent and appropriate it is to mathematically model such relationships with the available data and without clear theoretical bases (Hox et al., 2017).

Additionally, there is no consensus on the most appropriate instrument to measure the HE contribution and on how to draw a representative sample of students and their HE experience. Studies in primary and secondary education illustrates attempts to measure a representative sample of students’ learning. However, determining which sample in HE represents all programs is more complicated due to curricular diversity (Kinzie, 2019). For example, mathematics in STEM programs differs from mathematics in education or psychology programs. In addition, there is the possibility that the learning measured in such exams does not fully reflect the contribution of HE to students. Practical and other workplace skills, such as teamwork, self-criticism, or oral expression, are not generally included in measures examining value-added (Koretz, 2019).

Despite these challenges, the increased interest on the effectiveness of educational processes has prompted the study of the contribution of HEIs by testing different methodological alternatives to address the challenges described above.

Methodology

We searched for articles published between 2000 and 2023 in the Web of Science (WoS) and Scopus databases. Within WoS, we searched in the Core Collection, specifically in Science Citation Index Expanded, Social Sciences Citation Index, and Arts & Humanities Citation Index. In the case of Scopus, we searched in the following indices: Social Sciences; Economics, Econometrics, and Finance, Psychology, and Arts and Humanities.

A total of 41 keywords were used, organised into six groups. Keyword groups aimed to capture synonyms within categories used frequently in higher education value-added research. Table 1 shows the keywords for each keyword group created.

Table 1. Keywords used.

Keyword Group	Keywords
Educational level	Higher education; Tertiary education; Postsecondary; University
Outcomes	Student outcomes; Learning outcomes; Outcomes assessment; Standardized tests
Methodology	Value-added; Administrative data; Regressions; Matching; Multilevel; Causal inference; Instrumental variables; Propensity scores; Treatment effect; Causal
Inequity	Inequality; Equity; Income inequality; Equality of education
Teacher training	Pre-service teacher; Teacher training; Initial teacher education; Teacher preparation; Teacher candidates; Teacher education programme
NOT	Survey; Attitudes; Self-efficacy; Believe; Meanings; Case study; Self-report; Student satisfaction; Perceptions; Flipped; Online; Health; Patient

We searched for articles combining three or four keyword groups at the time, according to the four combinations shown in Table 2. The boolean term ‘OR’ was used within the keyword group, and the boolean ‘AND’ was used to link the different keyword groups. Keywords could appear in any of the three dataset filter fields: title, topic, or abstract. The keyword group NOT referred to all study types that did not meet the studýs inclusion criteria and that needed to be excluded. Finally, we combined the code from the four search sets depicted in Table 2 using the ‘OR’ boolean. This allowed us to get a final set of results without duplicated manuscripts for each dataset (WoS, Scopus). Results from both datasets were manually combined excluding duplicates.

Table 2. Combination of Keyword Groups used.

Combinations	Keyword Groups
1	Educational level + Outcomes + Methodology + NOT
2	Educational level + Outcomes + Inequity + NOT
3	Outcomes + Methodology + Teacher training + NOT
4	Educational level + Methodology + NOT

The search strategy resulted in 430 articles indexed in WoS and 315 in Scopus. There were 91 duplicates, so the total number of articles was 654.

The publications were selected based on the titles and abstracts of the 654 articles. The inclusion criteria were: (i) quantitative empirical studies, (ii) focused on higher education, (iii) learning outcomes or access to the labour market as a dependent variable (e.g. scores on tests, grades, probability of graduation, salary, time to find a first job), (iv) academic or socio-economic characteristics of students or institutions as independent variables (eg previous academic performance, parents’ educational level, parents’ salary) and (v) outcomes were measured at the end of HE. The use of these criteria resulted in a final sample of 49 articles. The 49 selected articles were reviewed in depth with three focus: first, the objectives stated by the authors; second, the methodology used, with emphasis on the study design, the independent and dependent variables used, and the statistical technique used; third, the results achieved. The review focused on the study results and not on the new hypotheses they may include.

Results

The results are presented in two sections. The first summarises the consensus and disagreements on the methodologies used to estimate the institutional contribution and the associated factors. The second section presents the main findings regarding the contribution of HEIs to the outcomes of their graduates (Q1), as well as how this contribution is achieved (Q2).

On the methods

Before delving into the research questions, we present a brief overview of the statistical techniques used in the 49 articles to better comprehend the scope of their results. As for the type of statistical techniques used, we found causal and descriptive studies, with a predominance of the latter (78%). Among the most used descriptive techniques, we found the Ordinary Least Squares regressions (OLS) with and without fixed effects at the institutional level (N = 18), followed by Hierarchical Linear Modeling regressions (HLM). Causal techniques were used only in 22% (N = 11) of the studies, from 2015 onwards. In decreasing order, the causal techniques most frequently found included Propensity Score Matching (PSM), Regression Discontinuity Design (RDD), and Instrumental Variables (IV). Finally, we found three studies using Difference in Differences (DID), Bayesian Network, and Entropy Balancing. Table 3 presents the detailed information.

Table 3. Papers description.

Dimensions		Sub- dimensions	numbers of papers	Year(s) papers
Country		Germany	1	2019
		China	2	2019, 2023
		Colombia	9	2013-2020, 2022
		United States	26	2001-2023
		Georgia	1	2019
		Ghana	1	2018
		Italy	2	2008,2017
		United Kingdom	2	2002, 2005
		Norway	1	2022
		Switzerland	1	2021
		Portugal	1	2023
		Saudi Arabia	1	2021
		Ethiopia	1	2021
Database		Web of Science (WOS)	40	2001–2023
		Scopus	9	2005, 2018, 2019, 2021–2023
Techniques	Causal techniques	Difference in differences	1	2020
		Instrumental variables	2	2016, 2019
		Propensity Score Matching	4	2015, 2016, 2019, 2022
		Bayesian network	1	2016
		Regression discontinuity design	2	2022
		Entropy Balancing	1	2021
	Descriptive techniques	HLM regression	10	2011-2016, 2022, 2023
		Quantil regression	1	2017, 2022
		Seemingly unrelated regression	1	2021
		Multilevel Structural Equation Modeling	1	2023
		Logistic regression with institutional fixed effects	3	2002, 2012, 2021
		Logistic regression without institutional fixed effects	3	2005, 2018, 2019
		OLS regression with institutional fixed effects	11	2001-2019, 2022, 2023
		OLS regression without institutional fixed effects	7	2011-2019, 2021, 2023
		Partial Least Squares regression	1	2022
Dependent variable		Mark like concept	3	2002, 2005, 2019
		Mark-GPA	8	2017, 2021, 2022, 2023
		Exam pass rate	1	2005
		Probability/graduation rate	9	2012–2019, 2022, 2023
		Standardized test score	16	2011–2021, 2022
		Non-standardized test score	6	2001, 2005, 2011, 2019, 2022, 2023
		Approved credits	2	2022, 2023
		Graduation time	1	2023
		Highest Educational Level	1	2021
		Intergenerational Mobility	1	2023
		Labour market variables	6	2008, 2016, 2022
Number of papers per period		2001-2005	5	–
		2006-2010	1	–
		2011-2015	9	–
		2016-2020	17	–
		2021-2023	17	–

Our review revealed a growing interest in seeking more precise estimates of the institutional contribution since the early attempts to transfer school effectiveness studies to the higher education field (Rodgers, 2005; Yunker, 2005). We found discussions on (a) the relevance of using cross-sectional designs; (b) the comparison of different statistical techniques, (c) how to best measure the institutional contribution, and (d) the problems generated by the lack of randomisation in the access to HE. Each of these discussions is presented below more detail.

Regarding the type of designs (a), at least two studies concluded that the use of cross-sectional design, as opposed to the longitudinal design, tends to have an upward bias on the results. This is due to difficulties controlling for differences between first-year and final-year students, even when using causal statistical techniques such as matching. Students differ in academic, social, or motivational characteristics, among others (Liu et al., 2016; Steedle, 2012). We also note that cross-sectional designs seem to have been used due to the limited access to longitudinal databases, despite the awareness of researchers regarding their methodological weaknesses. For example, Liu et al. (2016) combined cross-sectional descriptive analyses (OLS regression) with Propensity Scoring Methodology.

We observed a growing interest during the last decade in comparing different statistical techniques to increase accuracy (b). Results confirm that considering the nested data structure using multilevel techniques such as HLM models provided more stable and consistent results over time (Steedle, 2012; Shavelson et al., 2016). However, studies such as those by Bogoya et al. (2017) and Horn and Lee (2019) reported that these models are less accurate in the distribution’s tails, so they recommended exploring techniques that reduce this imprecision, such as quantile regression.

In keeping with the search for precision, we observed an interest in methodologically broadening the concept of institutional contribution (c). Several publications analysed changes in the estimates resulting from the use of different types and numbers of dependent variables. The estimates can vary significantly depending on the type of variable used (e.g. Cunha & Miller, 2014; Milla et al., 2016; Rodríguez-Revilla & Vallejo-Molina, 2022; Shavelson et al., 2016). In that respect, a HEI could, for example, generate a high contribution to reading comprehension but a lower contribution in mathematics or vice versa. Typically, the literature (e.g. Broatch & Lohr, 2012; Espinel et al., 2019; Owusu-Agyeman & Larbi-Siaw, 2018) analysed the institutional contribution based on only one outcome variable (e.g. score on a test, grade point average, probability of graduation); however, Milla et al. (2016) included five test results (Spanish, English, Mathematics, Social Sciences, and Science) in the dependent variable of the regression model. Their results confirmed the existence of changes in the estimates of the institutional contribution depending on the use of one or several outcome variables. In that sense, their proposal is consistent with the multidimensional view of contribution, as discussed in the works of Cunha and Miller (2014) and Shavelson et al. (2016). Rodríguez-Revilla and Vallejo-Molina (2022) also contribute to this discussion. Although they did not include several outcomes in one dependent variable, as Milla et al. (2016) did, they did compare VA results considering outcomes related to mathematics and reading skills. Their results show that if VA is estimated considering mathematics skills, there are statistically significant differences between types of institutions (eg official and private), which is not the case when considering critical reading outcomes. Unfortunately, the authors do not explore this difference further. Finally, the results of Bagues et al. (2008) raise a caveat regarding using student’s grades as outcomes. Specifically, they note that students who graduate from institutions with higher final grades do not perform better in the labour market and vocational examinations. In this sense, differences in grade point averages would be more related to the different grading standards among institutions than to the institution's contribution to student learning.

The lack of randomness in the distribution of students among HEIs (d) is another issue addressed in different publications over the last decade (e.g. Falch et al., 2022; Gelbgiser, 2018; Cugnata et al., 2016; Shamsuddin, 2016). Most frequently this is done analysing institutional selectivity. This has predominantly been incorporated into HLM models as a covariate at the individual and institutional level (e.g. Cunha & Miller, 2014; Liu, 2011a) to model peers' influence on students’ academic outcomes. These studies showed that including this variable reduced the over- or underestimation of institutions’ contributions to socially and academically diverse students.

Selectivity was modelled using two types of variables. The first involved school variables measured several months or years before entering HE. Examples included courses taken in secondary school, remedial programs in which students participated, or the probability of graduating from secondary school (eg Cunha & Miller, 2014). University admission exam results was the second type of variable most widely used (49% of all papers). This variable was measured at the individual level but, in the analyses, was aggregated at the institutional level. For example, Tian et al. (2019), Kugelmass and Ready (2011), Bratti (2002), Shavelson et al. (2016), and Rodríguez-Revilla and Vallejo-Molina (2022) used the scores of an HE admission exam twice, first at the individual level and then at the institutional level as an average. In contrast, papers such as Wai and Tran (2022), Simpfenderfer (2023), Naven and Whalen (2022), and Rodríguez-Revilla and Vallejo-Molina (2022) aggregated this variable only at the institutional level (e.g. average SAT score of the institution, average parental income, institutional average SABER-11).

Main findings on HEIs contribution to learning

This section focuses on the main findings of the research included in the review. Based on our research questions, results are organized into four groups based on the differences identified in research goals, the statistical techniques, methodological designs, and the independent variables considered in the estimations. All 49 articles analysed had an academic (e.g. test results at graduation, time to graduation, academic honours) or labour outcome (e.g. access to the labour market, salary, place of work) as the dependent variable, as this was one of our inclusion criteria. We also distinguished between those actions that the institution deliberately implemented to promote student learning outcomes (e.g. curriculum, or academic support programmes); versus those variables that, while relevant for student learning outcomes, are not directly controlled by the institution (e.g. selectivity, gender, race, SES). Considering the methodological designs and statistical techniques employed, the initial three groups offer varying levels of depth in addressing our Q1: What is the contribution of higher education institutions to student learning outcomes? The first group presents the research that analyses the relationship between student characteristics (prior academic performance, SES) and student outcomes; the second group includes research looking at the relationship between institutional factors (e.g. curricular updates, student intervention programmes) and student outcomes; the third group refers to research that estimates the value added by HEIs. The fourth group includes research that estimates value-added, but also analyses institutional characteristics that help to understand differences in VA. The latter group contributes to answering our Q2: how was this contribution achieved?

Group 1: How are students' entry characteristics related to their outcomes?

Articles in the first group analyse student outcomes without explicitly considering the institution as an element in the estimation. In that sense, this first group of studies contributes to answering our Q1 but only at an initial and limited level, as it only allows us to examine the role of students' variables and their relationship to students’ results.

Using descriptive and causal techniques, such as logit models, ordinary least square regressions, weighted regressions, instrumental variables, Bayesian models, entropy balancing, hierarchical linear modelling, structural equations, and propensity score matching, studies showed a high association of SES and prior academic performance to university outcomes.

First, regarding the association with SES, students with a low SES are less likely to obtain a bachelor’s degree and get lower grades than their peers with a higher SES at the same type of institution (Ferrão, 2023; Gelbgiser, 2018; Shamsuddin, 2016). Also, they had lower levels of employability and wages despite having comparable academic performance before university entry (Bagues et al., 2008; Cugnata et al., 2016). Combet and Oesch (2021), using a causal technique (entropy balancing), found more than 20% points difference in university completion rates between high and low SES students. This relationship between socioeconomic origin and graduate outcomes would be mediated by institutional selectivity; however, it is unclear how selectivity operates within institutions to achieve differences in outcomes. This issue is further addressed in the results of group 2.

A second area of research examined the influence of prior academic performance on student outcomes. Papers such as Wariyo and Asgedom (2021), Ferrão (2023) and Garcia and Maza (2018) agree that this is the factor that most predict academic outcomes. However, they also recognise that such models explain approximately one-third of the variance in university success, so an important set of factors still needs to be investigated.

Of all the research reviewed, only one study found no significant relationship between prior academic performance and performance at graduation (El-Moussa et al., 2021). However, this article describes specific research that follows the trajectory of Saudi Arabian students who move to the USA to enrol in HE. Such students may have differentiating characteristics compared to students who do not migrate to study, which may affect the results.

In summary, this first group of studies shows how socioeconomic status and prior academic performance are highly related to student’s academic and employment outcomes. However, the designs and statistical techniques prevent us from interpreting findings as causal, nor do they allow us to understand the mechanisms that operate within institutions with greater or lesser selectivity.

Group 2: What institutional characteristics are related to student outcomes?

Studies in the second group do not use value-added models but contribute to answering our Q1 by identifying institutional characteristics that are associated with student outcomes. We organized the results from these studies along two themes: the mediating influence of institutional selectivity and the influence of changes at the curricular or teaching level.

Regarding selectivity, the studies showed that more selective universities increased students’ graduation probabilities or access to the labour market (Gelbgiser, 2018; Shamsuddin, 2016). However, this seems more related to their ability to attract students with higher academic achievement in high school and higher SES rather than to the quality of the education offered (Bagues et al., 2008; Delahoz-Dominguez et al., 2022). The research reviewed agrees that students from lower SES or prior academic performance were less likely to have access to more selective universities (Gelbgiser, 2018; Wai & Tran, 2022), either because of economic constraints or because of the influence of family upbringing on their application decision (Cugnata et al., 2016). In this sense, research does not identify the mechanisms operating within selective institutions to achieve higher outcomes. Papers such as Combet and Oesch (2021) note that students who enter more selective universities develop more ambitious educational goals, which could affect their labour market outcomes; however, the authors do not empirically support such a claim. On the other hand, more recent work questions the effectiveness of selective institutions over specific outcomes. Simpfenderfer (2023), for example, shows that less selective universities improved intergenerational mobility even for students with academic difficulties during their university education.

Regarding the curricular or teaching level variables, we identified a growing interest in reporting the outcomes of various learning experiences in students’ academic or work-related outcomes over the last three years. We present results, differentiating them by the statistical techniques used. First, we describe the results from descriptive studies, to move then to those that use quasi-experimental techniques.

Descriptive studies show that actions such as allocating a female advisor to low-performing students in secondary school (Kato & Song, 2022), having academic (workshops, courses) and vocational (attending lectures, research presentations) support to encourage enrolment in STEM programs (Draganov et al., 2023), using community based training for future doctors (Huang et al., 2023), implementing a different course sequence in the curricula (Lim et al., 2021), has a positive relationship with academic outcomes. In contrast, Bicak et al. (2023) showed that enrolling in additional mathematics courses is unrelated to better academic outcomes (GPA and graduation). Regardless of the results, the design and the statistical techniques used do not allow for the claim of a causal relationship between these actions and student outcomes.

Studies that use quasi-experimental designs and causal statistical techniques, on the other hand, come to a common conclusion: changes at the teaching or curricular level that emphasize academic support through student-centered and hands-on activities show positive results in various student outcomes. For example, teaching medical students in situations closer to the community (Latessa et al., 2015), offering instruction that engages students in learning mathematics (Wang et al., 2022), including mathematics reinforcement courses (Meiselman & Schudde, 2022), or promoting enrolment in academic enhancement courses (Turk, 2019), have a significant association with various post-graduation outcomes (e.g. retention, transfer from community college to four-year universities, graduation GPA, test scores at graduation, access to the labour market). Also, using causal techniques and quasi-experimental designs, Hahm and Kluve (2019) conclude that the Bologna reform led to a significant increase in the probability of graduating within the expected instructional time by placing the student as the teaching focus and promoting a curriculum that combines theory and practice.

Although the quasi-experimental studies find positive and significant relationships between different innovations implemented at the curricular level and student outcomes (final GPA, salary, continuity of studies), their results should be considered with caution. The studies were often conducted using data from one location (e.g. one university, except for Hahm & Kluve, 2019), and therefore, results are not generalisable to other institutions.

Group 3: What did higher education institutions contribute to the learning outcomes of their students?

We identified a third group of research that focuses on estimating the value institutions add to students’ academic outcomes (Q1). These articles used pre and post-intervention designs and statistical techniques such as OLS regressions with and without institutional fixed effects, multilevel regressions (HLM), quantile regression, PSM, and difference-in-differences.

These papers generally reported progress in student learning during HE (e.g. Liu et al., 2016; Klein et al., 2005; Pedersen & White, 2011). However, (a) the low variation in students' academic performance, (b) the influence of institutional selectivity, and (c) the persistence or deepening of prior academic gaps make it difficult to distinguish clearly whether the increase in learning is due to the experience in HE, maturation of the students, or some combination of these and other factors.

In regard to (a), several studies reported a high correlation between the scores students obtained in exams at the beginning and at the end of their education. This could be interpreted as a low value-added by the HEIs; in other words, those who had high performance in the school system are likely to continue exhibiting it in HE (e.g. Abramishvili & Tsirekidze, 2019; Bogoya & Bogoya, 2013; Delahoz-Dominguez et al., 2022; Shavelson et al., 2016).

On (b), we observe high influence of institutional selectivity on student learning outcomes associated with the low variance in the scores. Studies as those by Klein et al. (2005) and Liu (2011b) reported that scores on admission tests (e.g. SAT) explained more than 70% of the variance in university academic performance. Including selectivity in estimates of institutional contributions to outcomes decreased the diversity in HEIs' contribution (Cunha & Miller, 2014; Kugelmass & Ready, 2011; Liu, 2011a; Shavelson et al., 2016; Steedle, 2012). These results suggest that the institutional contribution may be more related to the selection made by HEIs of students with high academic performance rather to the academic contribution associated with their training.

In regard to (c), we found that the knowledge gaps observed before students enrol in HE are maintained or even increased at the end of HE. Using different techniques (e.g. simple linear regressions, difference-in-differences, PSM, HLM) and dependent variables (e.g. standardised mathematics and reading tests), several papers indicated that factors outside the control of individuals, such as gender, race, parents’ educational level, or place of origin, were more associated with learning outcomes than university education itself (Espinel et al., 2019; Gómez et al., 2020; Kugelmass & Ready, 2011). For example, Gómez et al. (2020) demonstrated that the gender-related prior achievement gap in mathematics and reading increased after university, especially women. Similarly, women’s initial advantage in reading is reversed after the university, in favour of men. In addition, Kugelmass and Ready (2011) also showed that the initial academic gaps between African-American students and white peers widened during college.

Group 4: What institutional factors influence the contribution of HEIs to student learning outcomes?

This last research group allowed us to identify more explicitly the institutional characteristics associated to students' learning outcomes (Q2). We identified three types of results. The first group examined whether differences observed in contribution estimates among institutions could be associated with various measures of institutional quality (Bratti, 2002; Falch et al., 2022; Kugelmass & Ready, 2011). To do this, they regressed the institutional effects estimated in the first stage of the analysis to institutional covariates associated with institutional quality or expenditure. Bratti (2002) and Falch et al. (2022) analysed staff quality, research quality, expenditure per student, and peer effect (average score of A-level students), while Kugelmass and Ready (2011) included in the estimates institutional selectivity and spending by HEIs on faculty development, teaching and learning centres, and academic support staff.

The studies reported expenditure and selectivity (or the peer effect) as the only institutional quality measures significantly associated with student outcomes. On this latter point, although the authors observed that studying in institutions with higher-academic-performance students contributed to increased learning outcomes, it was unclear whether this increase could be attributed solely to peer effect or whether there were institutional mechanisms not considered in the estimate.

A second factor identified is teaching quality, although the evidence is not decisive. This second set includes studies that descriptively examine teachers’ characteristics, while others describe the teaching process more in-depth. Tian et al. (2019) found that part-time teachers have a more significant effect on students’ academic performance than full-time teachers. Falch et al. (2022) conclude that full-time teachers influence institutional VA outcomes the most. On the other hand, Broatch and Lohr (2012) did not observe a relationship between teachers' characteristics (type of contract, gender, ethnicity, age, academic degree, area of work) and academic performance. Looking further into the teaching process, Anaya (2001) concludes that in institutions that add more value, there is an improvement in the learning of students who participated in out-of-class discussions with their teachers, and those who worked on their lecturers’ research projects. However, it was unclear whether the variables examined by Anaya (2001) can be attributed to the institution or to teachers’ initiatives.

We identify a third set of studies examining the contribution of HEIs to students’ academic and employment outcomes. Using HLM and regression discontinuity, Naven and Whalen (2022) found that law schools ranked in the top 14 places (T14) in the United States did not significantly impact students’ bar exam results. However, they did make a difference in the type of jobs they got. The lowest VA-ranked institution, within the top 14 law schools, increased by 30 percentage points the likelihood of its graduates being employed in important law firms. As a complement, through HLM, Rodríguez-Revilla and Vallejo-Molina (2022) analyse the academic outcomes of graduates over five years. Their results show that lower-quality institutions (non-accredited or private) added more value.

In summary, research suggests that institutional value added is not necessarily related to being ranked as a higher prestige or quality institution (e.g. Top 14 or accredited).

Discussion

The results showed a consensus on increased student learning upon completion of HE. However, it was not clear whether this increase is due to the institution, the students’ previous characteristics, or a combined effect of both. We identified three factors that could lie behind the uncertain role of institutional contribution.

First, from the methodological point of view, we find a preponderance of descriptive techniques (eg OLS and HLM regressions), which do not enable us to establish causal relationships between the institutional effect and graduate outcomes. Although most of the descriptive studies sought to control biases in their results by using designs with pre and post-measures or by including covariates to control for differences related to prior academic performance, SES, or gender, these were not enough to establish a causal link (e.g. Gerber & Green, 2012). The lack of randomisation in the distribution of students into the HEIs, methodological problems in attributing student outcomes to HEIs alone (exclusion restriction in the causal inference literature), or difficulties in controlling for peer influence on learning (SUTVA) were some of the additional problems identified (Gerber & Green, 2012).

Our review revealed a growing interest in solving these problems. Over the last five years, there have been efforts to reduce biases in the study of value added. These have been addressed by combining descriptive techniques such as HLM or OLS regressions with causal techniques such as PSM, Differences-in-Differences, or Bayesian networks (e.g. Cugnata et al., 2016; Gómez et al., 2020; Liu et al., 2016; Naven & Whalen, 2022). Despite such efforts, the studies still acknowledge pending problems that cannot be remedied by statistical analysis alone (eg Liu et al., 2016; Owusu-Agyeman & Larbi-Siaw, 2018).

Second, the studies reviewed to answer the first research question (Q1: What is the contribution of higher education institutions to student learning outcomes) reach a consensus on the fact that institutional contribution estimates varied depending on the type and number of outcome variables used and the inclusion or omission of academic institutional selectivity in the analysis. Regarding the number and type of variables, there seems to be a consensus that using only one outcome variable resulted in a limited perspective of institutional contribution. Some of the studies in the review addressed this discussion by including more than one dependent variable and estimating different models for each one of them (Cunha & Miller, 2014; Rodríguez-Revilla & Vallejo-Molina, 2022; Shavelson et al., 2016); or by including several outcome variables in the same estimate of institutional contribution (Milla et al., 2016). By implementing these alternatives, studies attempted to consider the discussion about the multidimensional contribution of HEIs, broadening the concept beyond results on a standardised test (Koretz, 2019). However, the limited development of standardised tests measuring graduate learning outcomes and the limited availability of data on other types of contributions to higher education (e.g. teamwork, leadership, innovation) made it difficult to broaden the concept of institutional contribution.

The research showed that institutional selectivity was a relevant variable in estimating institutional contribution. Students with higher SES and academic performance are concentrated in institutions with a longer tradition or prestige. Furthermore, while there is a body of literature that identifies that the most selective institutions achieved the best outcomes (Bogoya & Bogoya, 2013; Klein et al., 2005; Liu, 2011b; Owusu-Agyeman & Larbi-Siaw, 2018), it was unclear whether these outcomes were a product of what HEIs did to make students learn or whether it is a response to students’ prior academic skills and the influence these skills may have on their peers. Along this line, several studies (Bagues et al., 2008; Naven & Whalen, 2022; Simpfenderfer, 2023) suggest that the value added of selective institutions is lower than that of less selective institutions. Not considering the selectivity effect would go against an important strand of educational research that views learning as a social and cumulative phenomenon over time (Braun et al., 2010; Engeström, 1987).

We also observe an academic discussion in the modelling of selectivity. Studies included selectivity as an institutional variable in the estimates, although the variable was usually built using individual-level data (average score on HE admission exams obtained by enrolled students). In the opinion of authors such as Hox et al. (2017), such a combination would not be theoretically supported and would be inappropriate for estimating the effect of the institution. Combining the levels would not consider the contribution from high-achieving students to their peers and to the institution itself.

The analyses conducted to answer our second research question (Q2: How is the institutional contribution achieved?), showed that only 16% (n = 8) of the papers included institutional variables in their estimates. Furthermore, the variables considered in the models referred only to administrative institutional characteristics (e.g. public or private HEIs, type of teaching staff contracts, accreditation), and did not capture teaching and learning characteristics. We observe a disconnect between the literature on teaching and learning and the literature on the institutional contribution of HE. In the literature on teaching and learning, there is consensus on the important role that active learning strategies (e.g. Astin, 1993), participation in extracurricular activities (eg Pascarella & Terenzini, 2005), or institutional involvement in student learning (e.g. Kuh et al., 2011) play in fostering deeper and higher quality learning (Biggs & Tang, 2011). However, most of the VA literature included in this review (groups 3 and 4) does not address this discussion. Most of the studies that do address this issue are included in group 2 and do not use a dependent variable measuring the HEIs contribution. Through descriptive techniques, those studies report positive relationships between interventions at the curricular level and students’ academic outcomes. Although these papers do not use value-added models, they provide a path to follow. It would be interesting for the VA literature to consider variables of this type to advance our understanding of the factors related to HEIs contribution.

Despite the difficulties described, we observe areas of progress in the research on institutional contribution. At the beginning of the 2010s, only a few analyses of institutional contribution were available based on the literature review conducted by Kim and Lalancette (2013), and cross-sectional designs were frequently used. This review showed that longitudinal models have become more frequent and that there are important efforts to combine descriptive (OLS and HLM) and causal techniques (e.g. PSM) to estimate the institutional contribution starting in 2015. From our perspective, the latest developments show a growing concern for dealing with the lack of randomness in the distribution of students and the exploration of more accurate estimates.

Conclusions

The analysis of the institutional contribution of HE leaves several lessons for future research. First, developing models that more accurately estimate the contribution of institutions to high- and low-achieving students could facilitate a deeper understanding of institutional effects on a more diverse student body. An interesting line to explore is using quantile regression models (e.g. Bogoya et al., 2017; Page et al., 2017).

In addition, it is necessary to include causal techniques in the estimation of institutional contribution since these methods provide a more accurate estimation of the institutional effect on student outcomes. However, these efforts require further theoretical and methodological discussion to meet the assumptions recommended by the specialised literature on causal inference (Gerber & Green, 2012).

In parallel, moving towards a multidimensional perspective of institutional contribution is important. The evidence shows that estimates change depending on the type and number of outcome variables.

Finally, the literature on institutional contribution could be even more helpful to the scientific community and decision-makers if, in addition to identifying which institutions generate greater or lesser contributions, it would enable us to determine for which group of students the contribution occurs and to which institutional factors it associates. In this regard, future research should include institutional variables closer to the issues studied in the teaching and learning literature.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Funding

This work was supported by the Millennium Nucleus ‘Student Experience in Higher Education in Chile: Expectations and Realities’, Doctorate Scholarship Programme of the National Research and Development Agency [2018-21181511], and project Fondecyt 1211883.

Notes on contributors

Gonzalo Cifuentes Gomez

Gonzalo Cifuentes Gomez holds a master’s degree in learning assessment and is a doctoral candidate at the Pontificia Universidad Católica de Chile. His research interests include initial teacher training, learning assessment in higher education, and value-added models in higher education.

Maria Veronica Santelices

Maria Veronica Santelices, PhD, is an associate professor at Pontificia Universidad Católica de Chile, Department of Education. Her research interests include educational measurement and educational policy. In higher education, she has researched admissions to selective institutions both in Chile and in the United States, and college-going decision-making, transition to higher education, persistence and the impact of financial aid on educational outcomes.