Academic trajectories analysis with a life-course approach: A case study in medical students

Abstract

Graduation rates and students’ academic trajectories can be used to determine universities’ performance and their response to curricular change. Longitudinal follow-up of students’ academic performance offers “natural experiments” to explore the impact of modified curricula. The goal of this study was to propose a method for analyzing students’ academic trajectories and identify changes associated with a curricular reform. We analyzed students’ trajectories throughout the program in two curricula (1994 and 2010), at the National Autonomous University of Mexico (UNAM) Faculty of Medicine. Advancement of each student was calculated using the accumulated credits in a defined time period, and the percentage of students that completed the credits for each stage of the curriculum. The theoretical framework was the “life course” approach, applying concepts of trajectory, transition and state. R software and TraMineR algorithm were used for data analysis. Five student cohorts of the older curriculum were studied (1994, 1995, 1996, 2004 and 2005 classes), and two cohorts of the new curriculum (2011, 2012), a total population of 6829 students. Students in the newer curriculum had a faster, more timely and efficient advancement in academic pathways, than cohorts in the older one. There was a higher percentage of students with “regular” trajectory (without failed courses) in the newer curriculum. Regularity is a straightforward metric that allows identification of curricular associated changes. This method allows the identification of inflection points throughout students’ academic trajectories, and contrasts pre-post curricular change data.

Keywords:

• academic pathways
• curriculum change
• program evaluation
• medical education
• life course approach
• student performance

PUBLIC INTEREST STATEMENT

Rates of graduation and students’ academic trajectories during their transit in the university can provide relevant information about the effects of implementing new programs. This study used an innovative method for analyzing students’ academic trajectories, and identified changes associated with a curricular reform in a large public medical school in Latin America. The advancement of each student was calculated through time for each stage of the medical curriculum. A total population of 6829 students were analyzed, composed of five student generations that had an older curriculum, and two generations with a new curriculum. Students in the newer program had a faster, more timely and efficient academic advancement, than students in the older one. There was a larger percentage of students with “regular” trajectory (without failed courses) in the newer curriculum. Regularity is a straightforward measure that can allow identification of curricular associated changes. This method allows identification of important changes throughout students’ academic lives, and helps to compare curricular change data.

1. Introduction

Terminal efficiency in higher education has been one of the main traditional indicators for universities’ accountability and is a frequently used measure for program evaluation and accreditation (Ball & Halwachi, 1987; Labrandero et al., 2021; National Research Council, 2012; Rodríguez, 1989; Saavedra & Espinoza, 2006; Sneyers & De Witte, 2017). The measurement of terminal efficiency can be done cross-sectionally or longitudinally, the first is the arithmetic ratio of the number of graduates divided by the number of students that are admitted in a specific period; the second measures terminal efficiency through life paths (Ball & Halwachi, 1987; Rodríguez, 1989; Rodríguez Ayán & Ruiz Díaz, 2011; Saavedra & Espinoza, 2006; Sneyers & De Witte, 2017). This parameter is an indicator used by external organizations for accreditation of higher education programs in many countries (Rodríguez Ayán & Ruiz Díaz, 2011; Sneyers & De Witte, 2017). Curriculum changes, on the other hand, are driven by institutional policies and bylaws, and are influenced by multidimensional factors (Bland et al., 2000; Guthrie, 1986). Some universities define terminal efficiency as “the relationship that exists between the number of students that are admitted to the university in a class/generation, and the number that graduate after they finish all the curriculum credits in a stipulated time” (Valle Gómez-Tagle et al., 2001), this form of estimation has been labeled “real cohorts” (Labrandero et al., 2021).

Unfortunately, terminal efficiency can hide the dynamics of students’ academic advancement during their transit through the curriculum. It makes identification of students that follow atypical paths difficult, since there are students that carry larger or smaller academic loads and their time to completion varies. Longitudinal studies, as opposed to cross-sectional estimations, offer evidence of what happens between two extreme points in a real cohort (Labrandero et al., 2021). Under this perspective, follow-up of students is considered a method of longitudinal analysis of academic trajectories, based on the reconstruction of the path followed by student cohorts from their entry to the end of the cycle (Rodríguez, 1989).

2. Curricular changes

Higher education globally faces several complex challenges, including the implementation and follow-up of curricular changes (Guthrie, 1986; National Research Council, 2012). Even though universities spend a great number of resources planning and implementing educational programs, an unresolved issue is the intricate problem of evaluating the impact of curricula in students’ academic performance (Hecker & Violato, 2008, 2009; Lockyer et al., 2009). It is not straightforward to identify the individual components of a major curricular change and their effects on outcome variables, teacher performance and student learning, since there are many interlinked variables that can influence them (Alves et al., 2013; Hecker & Violato, 2008).

There are few studies that explore students’ performance during and after long-term formal educational interventions in higher education, probably due to the following: curricular changes are influenced by contextual factors that create a difficult scenario for experimental research designs; university authorities may be cautious to display the results of curricular changes, particularly when those results are not positive; program modifications are the result of lengthy political processes, and when change has been implemented it is difficult to return to the previous curricular state; and finally, curricular longitudinal analyses or pre-post studies have validity threats that limit causal inferences (Albanese, 2000; Cook, 2010; Hecker & Violato, 2008, 2009; Seifert et al., 2010).

In the past century, there have been several major proposals for radically changing medical schools’ curricula, which have had limited success globally (Anderson, 2000; Christakis, 1995; Putnam, 2006). Curricular change is one of the most difficult academic, political and educational challenges of a dean or institutional governing body, since it implies a huge effort by all involved, leadership expenditure of political capital, and changes are difficult to sustain and measure (Albanese, 2000; Norman, 2003). Even though all universities undergo curricular changes continuously, publication of these experiences are usually case reports, frequently focused on descriptions of an educational model (competency based, integrated curriculum), faculty development strategies, organizational change and leadership challenges, particular aspects of teaching a specific discipline (biomedical informatics), or an individual competency (e.g., communication skills; Albanese, 2000; Bland et al., 2000; Sánchez-Mendiola et al., 2013).

3. Conceptual framework

The quantitative method for construction of academic trajectories and its calculation presented in this paper, is placed in the meso-level theoretical perspective (Haas & Hadjar, 2020). These authors, based on a thorough review of the literature, formulated a framework with macro, meso and micro levels, to conceptualize and understand the behavior of academic trajectories. The meso-level integrates actions that allow decision-making about program duration, curricular load and courses’ sequences, among other components of curricular plans. In this meso-level, the innovative category of curricular analytics can be included (Hilliger et al., 2019). Haas and Hadjar (2020) also classify these research studies in three design types, the method used in this study is a type I research design, which corresponds to description and reconstruction of trajectories.

In the academic trajectories research field, there are several theoretical and methodological perspectives, as consequence of the increased availability of high-quality longitudinal data, related to the pathways followed by higher education students. The United States of America is the country with the largest number of studies in this field, and a smaller proportion from Australia, Canada, Italy, New Zealand, South Africa and Spain (Haas & Hadjar, 2020). Latin America has lived important student movements that have influenced the course of higher education in the area, for example, the University Reform of Córdoba, Argentina, and in Mexico the strikes at the National Autonomous University of Mexico (UNAM) as a response to foreign educational policies (Ordorika, 2021). These settings create situations where it is difficult to obtain educational trajectory data from students, which complicates follow-up of students that changed their site of residency, country or place of work.

“Life course” theory was used as the conceptual framework to analyze students’ academic trajectories in this paper. According to Elder, this theory aims to reveal the manner in which broader social forces mold the development of individual and collective life courses (Elder et al., 2003; G. H. Elder, 1999). The driving forces at the university are operationalized in changes of curricular structures and programs, and it is expected that these variations promote changes in the temporal behavior of students’ trajectories. The concepts of “trajectory”, “transition” and “state” were applied in this study. “Trajectory” refers to a life or career line that can vary and change in direction, degree or proportion (Blanco, 2011; Han et al., 2017; J. Z. G. G. H. Elder, 1998). “Transition” relates to the changes in state, position or situation that can be foreseeable or probabilistic (Elder et al., 2003; J. Z. G. G. H. Elder, 1998). “State” is the concept that articulates trajectories and transitions. A “state” is a situation in which the unit of analysis finds itself, in a determined moment, throughout time (Abbott, 2001; Abbott & Forrest, 1986; J. Z. G. G. H. Elder, 1998). In this study, the basic unit of analysis is a student throughout time, and each state is defined according to the number of credits that a student achieves after a curricular year, until the end of that educational program. The concatenation of states where a student finds herself at the end of each educational cycle determines her academic path/trajectory.

4. Academic trajectories

Academic trajectories allow calculation and analysis of indicators that explain students’ individual and group academic behavior. Traditional indicators to measure academic trajectories are academic advancement, academic performance, terminal efficiency, academic delay and dropout, regularity and failure (Sneyers & De Witte, 2017). Academic trajectories’ studies have increased in sophistication, evolving from a descriptive function of performance indicators (Caballero et al., 2011), to relational studies (Oloriz & Fernández, 2013; Villalobos & Gutiérrez, 2013) and finally to play a predictive role with the incorporation of new models (Burillo et al., 2011; Gonzalez, 2017; Haas & Hadjar, 2020; Musso et al., 2020; Trucchi et al., 2017). Academic trajectory studies have attempted to explain the effect of individual, academic, institutional, economic and cultural variables that intervene in academic behavior (Haas & Hadjar, 2020; Mellado et al., 2017). These studies contribute to more rational decision-making, to improve formative processes and academic administration (Opazo & Villalobos, 2012; Villegas et al., 2017). Academic trajectory studies have achieved relevance in higher education institutions and are an important input in the design of institutional policy planning (Opazo & Villalobos, 2012; Villegas et al., 2017). Another relevant aspect is the fact that students’ situations are dynamic or can occur in different degrees, which creates an analytical challenge (Castro et al., 2016).

In the last decade, several innovative techniques to explore students’ academic trajectories during training have been published, that add a broader perspective to program evaluation and the impact of curricular changes (Chan et al., 2018; Ellaway et al., 2019). Curricular analytics is particularly relevant, since educational programs are by definition complex adaptive systems that occur in a fixed normative framework and identified time periods (Jorm & Roberts, 2018; Komenda et al., 2015; Mennin, 2010; Popoola et al., 2018). The main goal of this study is to present a method to measure differential students’ academic pathways in several cohorts, before and after a major curricular reform.

5. Methods

5.1. Research design

Longitudinal quantitative research design, with quasi-experimental pre-post analysis comparing real cohorts of students’ academic trajectories, before and after a major curricular change.

5.2. Overview of the process for developing students’ academic trajectories

Figure 1 shows an overview of the method used to identify and measure students’ academic trajectories, considering the curriculum and the corresponding cohorts. The identification and selection of variables that contribute to the unit of analysis help the decisions of which curricula, cohorts, context and temporality will be studied. This step allows the selection of curricular programs and student cohorts, related to its temporal proximity to the curricular change. Calculation of trajectories is grounded in the idea of Markov chains, a discrete stochastic process that recognizes the location of the unit of analysis, each student in a given state, which corresponds to the number of credits covered in a time period. The concatenation of states through which the student moves allows the construction of the academic trajectories or pathways. The recognition and contrast of these patterns with the ideal trajectory (regularity), allow the visualization of multiple trajectories in comparison with each other and the ideal. Common trajectories and atypical ones should be analyzed from the perspectives of the stakeholders involved in curricular design and evaluation, where an interdisciplinary dialogue can help an appropriate interpretation of the trajectories’ behaviors.

Figure 1. Steps for the process of constructing academic trajectories.

5.3. Setting

The National Autonomous University of Mexico (UNAM) Faculty of Medicine in Mexico City is the largest medical school in the country and one of the largest in Latin America, with about 9,000 undergraduate students and more than 11,000 medical residents. It is a public institution and the largest producer of basic and clinical medical research in Mexico, through its affiliations with major academic medical centers (http://www.facmed.unam.mx). Mexico has more than 170 medical schools, the curricula are heterogeneous although most are six-year programs (two years of basic sciences, two years of clinical rotations, one year of clinical internship and one year of mandatory social service; Blancas & Bárcena, 2018; Sánchez-Mendiola et al., 2011). Students’ average age when entering medical school is 18 years.

5.4. Curricular change

The medical school underwent a major curricular reform in the year 2010 (C-2010). The previous curriculum (C-1994) had a traditional focus based on the Flexnerian model and was in place for 16 years (1994 to 2010). Some characteristics of the C-2010 program are: definition of intermediate and outcome generic competencies; three areas of knowledge (biomedical, clinical, sociomedical/humanistic); four training phases with competency profiles; new courses (biomedical informatics, evidence-based medicine, among others); integration of the Simulation Center in the curriculum with problem-based learning methodology; core curriculum for each course (Sánchez-Mendiola et al., 2011). The curricular maps are in the Appendix A (Figure A1 = 1994 MD Program curricular map; Figure A2 = 2010 MD curricular map), and a comparative table of their differences in the Appendix B (Table B1, comparison of curricula).

5.5. Sample

5.5.1. Student cohorts

We analyzed seven cohorts/generations of the MD program. The first five cohorts underwent the C-1994 curriculum (1994, 1995, 1996, 2004, 2005), the last two cohorts for which data were available had the C-2010 program (2011, 2012). The students’ distribution by year is shown in Table 1. The total population including all cohorts were 6,829 students: 4,629 underwent the 1994 program and 2,200 the 2010 curriculum.

Table 1. Number of students in the MD program at UNAM Faculty of Medicine in Mexico City, by curricular cohort (n = 6,829)

Cohort	Curriculum	Number of students
1994	C-1994	927
1995	C-1994	891
1996	C-1994	920
2004	C-1994	925
2005	C-1994	966
2011	C-2010	1,045
2012	C-2010	1,155

5.5.2. Selection criteria

In order to have representative cohorts, we included the first three generations of the 1994 curriculum. The initial years of a new curriculum implementation are usually accompanied by a strong organizational effort and intense faculty development and supervision activities, and in parallel, there is frequently a “performance dip” due to the organization learning curve (Bland et al., 2000). In 1999, there was a one-year strike at the university, which had profound effects on curriculum delivery, so we included two generations after this major event (2004 and 2005), which was an uncontrolled disrupting variable that likely affected students’ academic pathways. On the other hand, 2004 and 2005 cohorts studied the 1994 curriculum a decade after its implementation, which provided information generated in a more mature curricular environment. The newer curriculum (C-2010) samples were the first two generations that completed the program of studies in the established official time, and were the ones for which complete data were available.

5.6. Measurement Methods and Statistical Analyses

Data for construction of medical students’ trajectories were obtained from academic histories, the university official data source of academic advancement in credits. These were linked to the corresponding curriculum and program of studies. A credit is the value unit of an academic course or module; a curriculum or plan of studies is the combination of courses or modules (theoretical courses, laboratories, workshops, practices), assessment results and other institutional requirements which, when finished satisfactorily, provide evidence that students finished the MD program (Universidad Nacional Autónoma de México [UNAM], 2018).

The program of studies and credits are two variables used to establish the students’ situation after an academic year. The behavior of the academic trajectories is identified through two indicators: Advancement/Progress and Regularity/Consistency. Advancement is the percentage of accumulated credits during the academic period, in relation to the requirements established in the formal curriculum. Regularity is the percentage of students that concluded satisfactorily the number of credits established in the curriculum for that year.

Eight different states were built to identify all the possible trajectories through which a student can transit throughout his/her academic career, following the same logic of the state we call regularity. Each state corresponds to a feasible possibility where a student can be situated after an academic year, according to the number of covered credits that are associated with each passed course during that period. For example, ideally, a student should cover in the first year all the credits that are established in the curriculum. However, there is a possibility that the student does not pass all the required courses, or that he doesn’t pass any of the courses required for that year. Under this assumption, quintiles of academic advancement were defined, plus an “absorbent state” (C) that corresponds to total credit completion.

As seen in Table 2, a student that completed 100% of the required credits for an academic cycle is labeled $A_{\hspace{0.17em}}^{100}$, a student that finished between 80% and 99% of credits is labeled $A_{80}^{99}$, and so forth until the student that did not complete any credits. The label “A” associated with each state, represents the advancement with relation to the number of credits that each student should cover in an academic cycle. The subscript in each A level means the percentage of covered credits, e.g., state $A_{80}^{99}$ means that the student passed at least 80% of the required credits, but less than 100%. This indicator represents a measure of student progress in her academic trajectory (Rodríguez Ayán & Ruiz Díaz, 2011). States have two functions: they allow the quantitative recognition of each student’s situation after each academic cycle during their academic career; and they allow comparison of academic progress of each student with the corresponding curriculum.

Table 2. Interval labels of students’ transition states during their academic journey

States	State intervals
$A_{\hspace{0.17em}}^{100}$	Academic progress in agreement with the curriculum
$A_{80}^{99}$	80% ≤ Academic progress < 100%
$A_{60}^{79}$	60% ≤ Academic progress < 80%
$A_{40}^{59}$	40% ≤ Academic progress < 60%
$A_{20}^{39}$	20% ≤ Academic progress < 40%
$A_{10}^{19}$	10% ≤ Academic progress < 20%
$A_0^9$	0 ≤ Academic progress < 10%
C	Completion (absorbent state)
NR	Not registered in the academic cycle

5.7. Operationalization of states in the curricula

Course credits and the time used by students to complete them allow estimation of academic progress, equal to the ratio of accumulated credits divided by the time in which credits were completed. The unit of measure of a student’s academic progress is the course credit, and passing a course is equivalent to an established number of credits in the curriculum. Accumulated credits are the sum of credits equivalent to the number of courses that a student should complete in a specified time period.

Credit accumulation begins once a student is registered for the first time in the program. This condition allows comparability of accumulated progress among students registered for different curricula, since students’ starting point were homologated. Students can be accepted to the MD program but not register for any course. The idealized transition state for each academic cycle is $A_{\hspace{0.17em}}^{100}$, as established in the curriculum. The extreme states are the completion state (C) and the not registered (NR) state. A student is in the C state when she accumulates the total number of credits established in the curriculum, and a student is in the NR state when he does not register for any course that year.

Table 3 shows an example of students’ representative accumulative academic progress. At the end of each academic year, the student is classified in one of the possible states. For example, in the first row the student passed all the courses and accumulated the total of year one credits, the same thing occurred in second (1995) and third (1996) years, so the student completed the credits in 1997.

Table 3. Examples of state by academic year, each row is a student of the C-1994 program

Student	Cycle
	1994	1995	1996	1997	1998	1999	2000
1	$A^{100}$	$A^{100}$	$A^{100}$	$C$	$C$	$C$	$C$
2	$A_{20}^{39}$	$A_{20}^{39}$	$A_{40}^{59}$	$A_{40}^{59}$	$A_{60}^{79}$	$A_{80}^{99}$	$A_{80}^{99}$
3	$A_{80}^{99}$	$A_{60}^{79}$	$A_{40}^{59}$	$A_{20}^{39}$	$A_{20}^{39}$	$NR$	$NR$
4	$A_{20}^{39}$	$A_{20}^{39}$	$A_{40}^{59}$	$A_{40}^{59}$	$A_{60}^{79}$	$A_{60}^{79}$	$NR$
5	$A_{60}^{79}$	$A_{40}^{59}$	$A_{40}^{59}$	$A_{40}^{59}$	$A_{40}^{59}$	$A_{40}^{59}$	$NR$

5.8. Data sources and statistical software

The main data source for the study are the academic histories of the students. The following fields were used: student label (anonymized), code for curricular program (C-1994 or C-2010), type of admission, courses, credits and time. The software used was R (https://www.r-project.org), TraMineR “STates-Sequence” (STS). This program allowed calculation and visualization of academic trajectories (Abbott & Forrest, 1986; Gabadinho et al., 2008).

5.9. Ethical aspects

The study was in compliance with the Declaration of Helsinki for research involving human subjects’ data. Data was managed anonymously in a confidential manner. The Research and Ethics Committee of UNAM Faculty of Medicine approved the research protocol as a non-invasive minimal risk study, with registration number 065/2016.

6. Results

6.1. Descriptive statistics

A total of 6,829 students were analyzed, 4,629 (67.8%) underwent the 1994 curriculum and 2,200 (32.2%) the 2010 program. Table 4 shows the population descriptive statistics.

Table 4. Descriptive statistics of UNAM’s Faculty of Medicine student cohorts, by program, year, gender, age at admission, final grades, final credits and graduation rates (n = 6,829)

		C-1994						C-2010
		1994	1995	1996	2004	2005		2011	2012		Total
N (%)
Women		535 (57.7)	566 (63.5)	591 (64.2)	619 (66.9)	634 (65.6)		689 (65.9)	740 (64.1)		4,374 (64.1)
Men		392 (42.3)	325 (36.5)	329 (35.8)	306 (33.1)	332 (34.4)		356 (34.1)	415 (35.9)		2,455 (35.9)
Total		927 (100.0)	891 (100.0)	920 (100.0)	925 (100.0)	966 (100.0)		1,045 (100.0)	1,155 (100.0)		6,829 (100.0)
Age at admission (mean ± sd)
Women		18.2 ± 1.8	17.8 ± 2.1	18.0 ± 1.8	17.9 ± 1.4	17.9 ± 1.5		18.1 ± 1.2	18.1 ± 1.3		18.0 ± 1.6
Men		18.1 ± 1.4	18.2 ± 1.8**	18.4 ± 1.7**	18.4 ± 2.0***	18.6 ± 2.6***		18.4 ± 1.6**	18.4 ± 1.7**		18.3 ± 1.8**
Total		18.1 ± 1.6	18.0 ± 2.0	18.2 ± 1.8	18.0 ± 1.7	18.1 ± 2.1		18.2 ± 1.3	18.2 ± 1.4		18.1 ± 1.7
Final grades (mean ± sd)
Women		7.6 ± 1.3	7.9 ± 1.1	7.9 ± 1.1	8.0 ± 1.2	8.1 ± 1.0		8.3 ± 1.1	8.2 ± 1.1		8.0 ± 1.2
Men		7.9 ± 1.2***	8.0 ± 1.2	8.0 ± 1.1*	8.1 ± 1.1	8.1 ± 1.1		8.4 ± 1.1	8.4 ± 1.1*		8.1 ± 1.1
Total		7.8 ± 1.3	7.9 ± 1.1	7.9 ± 1.1	8.0 ± 1.1	8.1 ± 1.1		8.3 ± 1.1	8.3 ± 1.1		8.1 ± 1.2
Final percent of credits (mean ± sd)
Women		67.4 ± 36.6	75.4 ± 32.1	74.6 ± 33.3	69.8 ± 33.2	73.7 ± 31.0		76.4 ± 34.0	74.5 ± 34.8		73.3 ± 33.7
Men		76.4 ± 31.3***	78.6 ± 31.5	80.2 ± 30.5*	75.3 ± 32.7*	75.2 ± 31.2		82.1 ± 31.9**	81.4 ± 31.3***		78.6 ± 31.6**
Total		71.2 ± 34.8	76.6 ± 31.9	76.6 ± 32.5	71.6 ± 33.1	74.2 ± 31.1		78.3 ± 33.4	77.0 ± 33.7		75.2 ± 33.1
Graduation rate
Women		37.2	44.3	46.9	32.6	35.0		36.0	42.3		39.1
Men		41.6	52.3**	56.2**	44.4***	40.4*		53.9***	53.3***		48.9***
Total		39.1	47.3	50.2	36.5	36.9		42.1	46.2		42.7
* < 0.05; ** < 0.01; *** <0.001

6.2. Transition states

The behavior of students’ academic progress for the C-1994 cohorts (1994, 1995, 1996, 2004 and 2005) is shown in Figure 2.

Figure 2. Academic progress distribution by state, C-1994 curriculum (1994, 1995, 1996, 2004 and 2005 student cohorts), UNAM Faculty of Medicine (n = 4,629).

As can be seen in the advance in credits (Figure 2), the 1994, 1995 and 1996 classes showed a similar behavior at the end of the first academic year. Six out of ten C-1994 students managed to finish in a timely manner 100% of the first-year credits ($A_{\hspace{0.17em}}^{100}$). Only four of ten students from the 2004 and 2005 classes in the C-1994 curriculum finished the totality of first-year credits on time. In comparison, for the 2010 curriculum five of ten students finished 100% of the first-year credits on time (Figure 3). Despite the fact that in the earlier generations there was a higher regularity in the first year, in subsequent years the loss of regular students was accentuated, as can be seen in the gradual staggering of the 1994, 1995 and 1996 classes (Figure 2). This could suggest that students had higher difficulties to pass the required courses.

Figure 3. Academic progress distribution by state in the C-2010 program, 2011 and 2012 student cohorts, UNAM Faculty of Medicine (n = 2,200).

By the end of the second year of the program, about 20% of the student population in the 1994, 1995 and 1996 cohorts covered between 40% and 59% of their academic requirements. This pattern changes in subsequent generations, in the 2011 class about 40% managed to cover the same range of academic requirements. After years three and four of the program, the 1994, 1995 and 1996 classes that had academic delay were concentrated in the 60% to 79% academic advance range, a small recovery compared to the previous year. The atypical case of the 1996 class, where only 15% of the students were regular, is likely due to the university one-year 1999 strike. In the fifth year of the program, non-regular students were mostly concentrated in the $A_{80}^{99}$ state, the remaining states are distributed first, in students that achieved 60% of academic advance, and second, in students that were in the critical state of “not registered”.

Academic progress of students previous to 2000 with the C-1994 curriculum, showed a different behavior than the 2004 and 2005 classes with the same program. In earlier generations, there was a staggered fall in regularity as students advanced in the program, reaching its lower level by the end of the fourth year. In these generations by the end of the first year, about 60% of the students were regular and at the end of the 4th year this number ranged between 20% and 40%. In the time required to cover the program’s credits, half of the students finished in a timely fashion. In the 2004 and 2005 cohorts, regularity had a constant trend of about 40%. The student group that was regular by the end of the first year maintained that state throughout the program and graduated. In both groups of generations, a year after the program stipulated time to finish the credits, a large number of students in the $A_{80}^{99}$ state incorporated to the graduated level, and by the eighth year, 80% of the students that were admitted to the MD program graduated.

Figure 3 shows the student trajectories of the 2011 and 2012 cohorts that underwent the C-2010 curriculum. In these cohorts, there is a larger proportion of students that had a regular trajectory (yellow and green portion of the graphs), which was higher than those of the C-1994 curriculum cohorts. It is not possible to state that this change in pattern of trajectories’ behavior was due to the different curriculum, since curricular change is a complex and multidimensional phenomenon, but it is possible to establish metrics that allow comparison of student cohorts that undergo different programs in different times. This metric can be regularity.

In the C-2010 curriculum cohorts, the bands that represent the academic progress of 40% and 20% (delay of 60% and 80%) stand out throughout the curricular time period (Figure 3). The distribution in both cohorts of C-2010 was similar. At the end of five years, the number of students that had an advance of 60% (delay of 40%) is accentuated in the 2012 cohort in the last year of studies. The percentage of students that had zero academic progress from the second year onwards is another behavior that stood out in the sixth year of the 2012 cohort, where there was a pronounced and steep progression of students that did not register any movement. In both 2011 and 2012 cohorts, the increase in students that registered zero movement is higher than any of the C-1994 cohorts.

There were 16 generations that received the C-1994 curriculum, we analyzed five to compare them with two generations that received the C-2010 program. The inclusion of the 2004 and 2005 cohorts had two reasons: the seven-year observation period, which did not overlap with the change of curriculum in 2010; and the proximity to the moment of curricular change (Figure 4).

Figure 4. Timing of curricular change, UNAM Faculty of Medicine.

We compared the percentage of students that were in the regular state ($A_{\hspace{0.17em}}^{100}$) each year for the 2004, 2005 and 2011 classes (Figure 5). The 2011 class with the newer curriculum had a higher percentage of students in the regular state each year, compared with the previous generations. If we had compared these generations with the traditional terminal efficiency parameter, we wouldn’t have been able to observe these longitudinal changes, since in the 6th year the difference is minimal (2%).

Figure 5. Comparison of student percentages in the “Regular” state ($A_{\hspace{0.17em}}^{100}$) in classes 2004 (C-1994), 2005 (C-1994) and 2011 (C-2010).

Figure 6 shows the percentage of students that developed a regular trajectory ($A_{\hspace{0.17em}}^{100}$) throughout the program, for the 1994, 1995, 2004, 2005, 2011 and 2012 classes (the last two had the C-2010 curriculum). The percentage of students with a regular trajectory is higher in the C-2010 cohorts by about 10%. If the tendency shown in the graph is maintained, it would be expected that in a manner similar to the C-1994 curriculum, where the first generation had 28.3% of students with a regular trajectory and in the second cohort the percentage leveled off at about 36%, the C-2010 would be stabilized after the 2012 class at about 46%.

Figure 6. Comparison of the percentage of students with regular trajectory ($A_{\hspace{0.17em}}^{100}$) by class and curriculum.

7. Discussion

Education scholars have published extensively about the importance of curriculum and program evaluation, as well as the different methods for measurement of student academic trajectories (Cook, 2010; Haas, 2020). The method proposed in this study provides quantitative information useful for curriculum evaluation as well as student support and follow-up, allowing longitudinal evaluation of the implementation processes and outcomes. The process described has the following steps: information organization according to curriculum type; definition of the student states and calculation of the trajectories; and comparative analysis of academic trajectories.

Evaluation models have been predominantly focused in measuring mostly curricular outcomes, but in more recent times new models of evaluation are focusing on the dynamics of the implementation process (Shinkfield, 2007). We selected an evaluation strategy that is theoretically grounded, to illustrate the use of the method in a higher education case study. The academic trajectories were identified and visualized, showing students’ academic advancement in two different curricula. The analysis of students’ academic pathways provided information about their states and formative transit through the programs, following their academic histories.

7.1. The curriculum change phenomenon

The measurement of outcomes generated by complex phenomena is an ever-present point of contention in higher education, since outcome data are required to support decision-making even though there are always theoretical and epistemic uncertainties in the processes (Van der Bles et al., 2019). Higher education can be viewed as an interlinked system of complex subsystems, where major curricular changes are frequent phenomena (Mennin, 2010). The measure of educational changes is a difficult task, and there is a need for objective measures to evaluate their effects in the student population. A metric is proposed, the regularity of academic progress throughout a time period, built through a process of analyzing academic trajectories. This is based on the assumption that the behavior of academic trajectories captures the results derived from curricular changes in a higher education program. Changes in a curriculum are subject to diverse forces, among its more visible components are the following: participants’ resistance to change, the assumptions and models of curriculum design, faculty development, university regulations, social context and local circumstances (Bland et al., 2000).

7.2. The Method to Identify Student Academic Trajectories

The method proposes as the basic unit of measurement the number of credits per course that is established by the curricular plans under study. This measurement unit allowed the building of each transition state and academic trajectory, the methodology allowed us to quantify each and every one of the students’ academic trajectories. The uncertainty involved in the correlation between alterations in academic trajectories and the curricular changes, was attenuated by the temporal proximity and the condition of mutual exclusion between programs. Temporal proximity allows the assumption that environmental conditions among the cohorts were similar, with respect to economic, technological and political factors. The condition of mutual exclusion between curricula means that the students were subjected to one and only one curriculum.

The proposed method to identify student academic trajectories requires the standardization of data in an objective fashion. This methodology does not integrate qualitative data, which are also important to achieve and maintain curricular changes, and should be object of further study (Hamui Sutton, 2016; Harper, 2007).

7.3. Variations in student academic trajectories

The trajectories of the cohorts in the older curriculum showed two patterns of regularity: in the 1994–1996 generations regularity decreases while students advance in the program, while in the newer program, the 2004 and 2005 cohorts’ regularity is constant. Evidence from several administrative changes that the university underwent in these time periods, suggests that the populations of students are similar and that the differences in academic trajectories could be attributed, at least in part, to the different curricula. In both programs about 80% of the students finished their studies by eight years after admission, although there was a 5% higher rate of regular trajectory in the C-2010 curriculum compared to the previous (C-1994) program. The structure of both plans does not allow students to advance if they do not pass a course, so they cannot catch up until they pass the required courses for that year. The first year of the program is crucial in order to have a regular trajectory from the beginning, which determines the subsequent trajectories, similar to the findings of other authors (Bruinsma & Jansen, 2009; Jamelske, 2009). The challenges and failure of many students during the first year of college evidence some of the limitations of high-school graduates in our educational system (Labrandero et al., 2021; Rodríguez Ayán & Ruiz Díaz, 2011).

The differences in academic trajectories can be due to multiple factors: the curricular integration of contents in the newer curriculum, improved students’ learning and studying techniques, faculty development initiatives, different academic practices, use of technology and simulators (De Angelis, 2000; Dienstag, 2011; Milesi, 2010). These activities and many others can influence academic performance in students, and these should be studied in more detail in subsequent studies. The results obtained measuring academic trajectories could be an added element for analysis with different curricular stakeholders. More research should be done about curricular changes in higher education, and the methodology described in this paper can be useful to evaluate the changes that occur in students’ academic trajectories. The results of similar studies can be included in program evaluation designs to supplement the data and information obtained with more traditional sources (Komenda et al., 2015).

This study has some limitations: the transit of students was analyzed in a single school in an emerging economy country, so generalization of these results should be done with caution. Individual student socioeconomic variables that can influence academic performance were not included, and there was no further follow-up after graduation. Even though the methodology theoretically can be applied to any field of knowledge, the study was done in a health professions field, so the findings have to be replicated in other areas. On the other hand, the methodology used in this study has not been reported, as far as we know, in the area of curriculum change and program evaluation, and there is a scarcity of studies evaluating major curricular changes in higher education with objective data from before and after the modification. The behavior of academic trajectories did not integrate other context variables related to the individuals, and it is not possible to declare that the differences in the cohorts are due mainly to the concurrent curricular plan, it is reasonable however, to state that there is a differentiated behavior in the levels of regularity among curricula and academic trajectories throughout time. In our context, the method presented for description and reconstruction of academic trajectories conflates the effect of multiple actions that were implemented during the curricular change periods, so it is not possible to identify the separate elements that influence change through the behavior of students’ trajectories. In subsequent studies, the method could be escalated to research designs type II or III (Haas & Hadjar, 2020), with the intention of identifying the predictors in the frame of curricular change.

Another aspect identified in the study is the higher rate of graduation in men from both curricular periods, which coincides with a study performed in 25 cohorts of students at UNAM Faculty of Medicine in Mexico (Campillo Labrandero et al., 2021). It is important to perform further studies that analyze the influences of gender in student trajectories and academic outcomes.

The method described in this paper for the quantitative construction of student academic trajectories is innovative and useful, builds on previous academic literature about this complex issue, and is feasible for any higher education institution (Haas & Hadjar, 2020). It is an initial step to evidence-based curricular improvement and decision-making, where the adoption of curricular analytics and curriculum data visualization needs to be promoted (Hilliger et al., 2020). The method allowed the analysis and comparison of longitudinal student academic trajectories in real cohorts, and showed the quantitative and visual contrasts of the data in different curricula.

Subsequent studies should incorporate the concept of latency, to provide meaning to the collection of accumulated data in the least possible time, and decrease the gap between institutional actions oriented to the improvement of the learning process and the initial data inception and capture, through the integration of analytical tools from different scientific disciplines and epistemological frameworks (Wong & Li, 2020). In the developing field of curricular analytics, the importance of data visualization for evidence-based decision-making cannot be overstated (Konecki et al., 2018).

8. Conclusions

The proposed method allows the construction of academic trajectories to analyze, separate and quantitatively synthesize the changes that occur with curricular modifications. The analysis of students’ academic pathways offers valuable information for the evaluation of curricular changes. This information is difficult to obtain with traditional cross-sectional studies. The study does not provide proof of causality regarding the educational impact of different programs, but it can be useful to complement the array of program evaluation strategies in medical education.

Availability of data and material

The datasets used in the study are available from the corresponding author on request.

Authors’ contributions

DCS, TBG, MGM, JGMS, and MSM participated in the conceptualization of the study and its design. DCS, TBG, JGMS and MGM participated in data curation, validation, statistical analysis and data visualization. All authors participated in draft and final manuscript writing and approved the final version.

Ethics approval and consent to participate

The study was in compliance with the Declaration of Helsinki for research involving human subjects’ data. Data was managed anonymously in a confidential manner. The Research and Ethics Committee of UNAM Faculty of Medicine approved the research protocol as a non-invasive minimal risk study, with registration number 065/2016.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The study had no external funding.

Notes on contributors

Melchor Sánchez-Mendiola

Our group works at the National Autonomous University of Mexico (UNAM) in Mexico City, in the area of curriculum development and evaluation, faculty development, educational assessment, educational innovation and distance education. We have been studying some aspects of curricular development and evaluation in our higher education institutions, and exploring the effects of the pandemic in distance education, faculty teaching and students’ learning in several areas of knowledge. This study is part of a larger project that aims to develop methods for curriculum analytics and learning analytics in Mexico and Latin America.

Appendix A

Figure A1. Curricular map of 1994 MD program.

Figure A2. Curricular map of 2010 MD program.

Appendix B

Table B1. Comparative table of the general characteristics of the 1994 MD Curriculum and the 2010 program

Characteristics	Curriculum
	1994	2010
Year of approval	1993	2009
Duration (semesters)	12	13
Academic pensum	9,887 hours	9,983 hours
Total of courses	28	57
Mandatory	24	55
Elective	4	2
Total credits	449	431
Mandatory	433	423
Elective	16	8
Educational model	Flexnerian traditional model (two years of basic sciences and three years of clinical sciences)	Mixed traditional with competency-based model (generic outcome and intermediate competencies)
Stages	3 Stages Basic: two years. Clinical: two years and one year of internship. Professional: one year of mandatory social service.	4 Phases First phase: two years. Second phase: five semesters. Third phase: one year of internship. Fourth phase: one year of mandatory social service.