Paradise lost or paradise regained? Changes in admission system affect academic performance and drop-out rates of medical students

Abstract

Background: The Austrian State medical universities had to change their admission system in 2005. Until this year admission to medical studies was unrestricted. Innsbruck Medical University chose the Eignungstest für das Medizinstudium in der Schweiz (EMS) aptitude test for admission testing.

Aims and objectives: Did the implementation of a selection process affect the academic performance and drop-out rates of students according to gender?

Methods: Two groups of students were compared: ‘open admission’ (2002–2004), and ‘selected’ (2006–2009). Academic performance was tested according to results in the final exams after year 1 (SIP 1; SIP, summative integrierte Prüfung) and after year 3 (SIP 3). Drop-out rates were recorded using the registration system of the university.

Results: Both, male and female ‘selected’ students had a higher passing rate regarding SIP 1 and they passed SIP 1 in shorter time and using fewer attempts than the open admission group. The percentage in passing SIP 3 did not change due to change in admission. The drop-out rates were significantly reduced for male and female students. ‘Unselected’ female students had a significantly higher drop-out ratio than ‘unselected’ male counterparts. After EMS testing, the drop-out ratios of female and male students were not significantly different.

Conclusion: Selected applicants were more able and better motivated to study medicine.

Introduction

In Austria, the study of medicine has historically been established at three state universities (located in Innsbruck, Graz and Vienna) under a common Federal law governing admission and the uniform medical curriculum. Until 2005, Austria might have been considered as paradise for high school leavers desiring to become a medical doctor: each one could study whatever he or she wanted, provided a successful completion of high school education in Austria. For the academic institutions responsible for study organization, this system was considered rather as the opposite (‘Tartarus’), because all applicants had to be accepted, irrespective of the capacity of the medical schools. At Innsbruck Medical University (IMU), between 500 and 700 students had to be admitted annually, though, e.g. lecture halls could only accommodate 300.

This situation was completely changed in July 2005, when the European Court of Justice ruled that Austria has to treat applicants from other member states of the European Union (EU) in the same way as it treated Austrian applicants. Thus, the court ruled according to the ‘free movement of people’ principle of the EU's internal market. This decision had – among others – the effect that a few thousands of young Germans who were denied to study medicine in Germany (because of the existing numerus clausus) were likely to apply for studying medicine in Austria. This sort of overwhelming scenario convinced the political stake holders that – at least for medical studies – Austria had to leave the ‘paradise’-paradigm of open and unlimited admission. Immediately after the court's final verdict, the Austrian Parliament complied and enacted an amendment to the admission regulations. In 2005, the admission became – enforcedly, due to the very short transitional phase – only based on the point of time of application, and a number of students was admitted that corresponded to the average number of the preceding years. From 2006 onwards, the new admission system showed three major amendments.

Table

Practice points

The change in the admission procedure (introduction of a cognitive test and limiting the number of study places) is associated with a significant improvement of the academic performance. This positive effect on the academic performance is only present in the first summative exam (after year 1) but is absent in later exams (e.g. after year 3). Drop-out rates were significantly reduced after the implementation of cognitive admission testing but no effects were seen regarding forced drop-outs. Male and female EMS-selected students were not different regarding passing SIP 1, but this amendment for female students (compared to the open access period) was bought dearly by a lower admission rate in the EMS.

First, a cognitive admission testing of applicants was implemented: the medical universities in Innsbruck and Vienna opted for the Swiss medical school aptitude test (EMS, Eignungstest für das Medizinstudium in der Schweiz (Hänsgen 2007)) while the Medical University of Graz chose to establish own procedures (Reibnegger et al. 2010, 2011). Until 2011, 30,063 high school leavers have undergone the EMS for Austria (EMS-AT; completely identical in contents to the Swiss version EMS; Anonymous 2011), the identical tasks being simultaneously assigned at the German speaking Swiss universities.

Second, the legislator introduced quotas in order to ensure long-term stability for the supply of medical doctors for the Austrian health-care system. Accordingly, available study places were awarded as follows: 75% for applicants with an Austrian final high school examination (including on equal terms applicants with exams from the German speaking South Tyrol (a Province of Italy), Liechtenstein or Luxembourg for political and historical bonds); 20% for applicants with a general qualification for university entrance from another EU member state; 5% for applicants with non-EU qualification for university. Applicants were ranked by their EMS score separately within each quota. The respective necessary threshold value was always highest in quota 2 (EU member states) with the highest number of applicants per place available, e.g. 20 in 2011 (Mallinger et al. 2011). This quota regulation has been under scrutiny of the EU Commission ever since, a final decision is pending.

Third, a maximum number of newly admitted medical students per year and each institution was enacted by the Austrian government. For the medical University of Innsbruck, this number was set to 400 (360 places are reserved for human medicine and 40 places for dentistry).

In the view of the medical school organizers, the situation since 2006 may be interpreted as ‘paradise regained’, because thereafter the number of students became oriented towards educational capacity and planning reliability has become reality. However, notwithstanding a desired effect on curriculum planning, this paradigm shift has fostered constant public as well as academic debates in Austria.

We have chosen to use the opportunity of an interesting ‘natural experiment’ to study the effects of a fundamental change in the admission procedure on students’ academic success, drop-out rates and gender distribution. These particular aims appear to be of high interest also for international readers, because only few reports (Reibnegger et al. 2010, 2011; O’Neill et al. 2011a) focussing on these outcomes do exist.

Methods

Participants of the study

The study year 2002 was chosen as a starting point. In that year, a new curriculum was implemented at the medical faculty in Innsbruck which included among others a profound change in the assessment program. The former high number of predominantly oral high stakes exams was replaced by MCQ based summative integrated exams (SIP; summative integrierte Prüfung) at the end of each of the first five study years (SIP 1–SIP 5). Students who commenced in year 2005 were omitted from comparison in this study, because they underwent a unique and different selection procedure. These data shall be published elsewhere. Students starting in 2010 were omitted, because of the scarce data to be expected after only 9 months. Hence, we compared all students at IMU, who started their study of medicine according to the same new curriculum of 2002, either without admission testing (i.e. started in years 2002–2004; ‘open admission’ group or unselected group) or after having been selected by the EMS-AT admission test (i.e. commenced in years 2006–2009; ‘admission by EMS’ group or selected group).

Admission testing by the EMS

The EMS, aptitude test for medical studies (TMS) in Switzerland, as the central admission-regulating tool in the German speaking Swiss medical faculties started in 1998 – again after a phase of preparation and political debate lasting several years. Its planning phase and yearly administrations have been described in detail (Hänsgen 2012). The EMS initially resembled its German predecessor ‘Test für medizinische Studiengänge’ (TMS) closely in its nine-subtest structure, is also administered once a year, but is mandatory and can be retaken. The quality parameters of TMS and EMS are very similar; therefore, data published for the TMS are given below. The Swiss EMS-applying universities convened in 2003 to review the EMS composition and purpose which resulted in the addition of a tenth item ‘planning and organizing’ since 2005 (Hänsgen & Spicher 2005). Two Austrian medical universities (Vienna and Innsbruck) contracted the use of the EMS from 2006 onwards. The EMS has since been administered on the same day and in the same version in Switzerland and Austria.

As the literature on the aptitude tests EMS and TMS is mostly in German or French and found only in books or institutional reports, a brief overview is given in this article. For details, see synopses on the TMS in English (Trost & Ebach 1997) and German (Trost et al. 1998), respectively. The TMS contains nine-subtest (pattern recognition; basic comprehension of medical and natural sciences; perspectives of cables (note: tests for spatial sense); quantitative and formal problems; concentrated and accurate work; memorizing facts; memorizing figures; reading comprehension; graphs and tables) which make up for the overall score. The TMS is based on a three-factorial analytical model, the factors being reasoning (‘Fähigkeit zum schlussfolgernden Denken’), visual perception (‘Fähigkeit zur Verarbeitung visueller Information’) and memory (‘Merkfähigkeit’). Reliability (by split half method) of the TMS overall score has been found as 0.90 or higher, the reliability of the subtests (expressed as internal consistency) ranged from 0.66 to 0.80 (Trost 1994). The overall TMS score correlated well (r = 0.45) with the summative scores from the German State exam ‘Ärztliche Vorprüfung’ (i.e. the German state exam after year 2, usually regarded as the most selective among the German state exams). When secondary school examination marks (correlation with score in the ‘Ärztliche Vorprüfung’: r = 0.48) and TMS score were combined, that correlation was even higher (r = 0.54; Trost et al. 1998). Secondary school examination marks and TMS score correlated with r = 0.43. A gender difference in the TMS overall scores among German applicants (in Germany until 1996) has been found with 1.7–2.4 points, favouring men (normalized mean value ± SD: 100 ± 10 points; Trost et al. 1998). More than 300,000 candidates had taken the TMS (they were only allowed to take it once), when its nationwide application was discontinued due to a political decision. Since 2007, the TMS experiences a renaissance as one factor in the admission procedures of German medical faculties (Kadmon et al. 2012). By 2012, 14 faculties employ the TMS on the same premises: central development and evaluation by the Institute for Test Development and Talent Research in Bonn, administration once a year, and participation on a voluntary basis (TMS-Koordinationsstelle 2012)

Similarly for the EMS (see below), the scores in Swiss students were shown to correlate well with students’ progress through the end-of-year exams after year 1 and 2 (Hänsgen & Spicher 2001). The reliability (by split half method) of the EMS overall score has been >0.90 and the reliability of the subtests have been comparable to those found with the TMS in Germany (Hänsgen 2012). A gender difference in the EMS overall scores among Swiss applicants has been described from 1998 through 2011 measuring 1.3–2.2 points and favouring men (normalized mean value ± SD: 100 ± 10 points; Hänsgen & Spicher 2011).

Variables tested

The academic performance was analysed in four ways: (i) the proportion of students passing the summative exam at the end of the first study year (i.e. SIP 1), (ii) the number of resits necessary to pass SIP 1 (IMU study regulations allow students to resit an exam up to three times), (iii) the time elapsed until passing of SIP 1 (students who are successful in resit 1 in September may proceed in their study without delay; students passing SIP 1 afterwards will have a delay of at least 1 year) and (iv) the performance of the students in SIP 3. SIP 3 was chosen, since it marks the end of the pre-clinical phase, has tended to be more challenging and discriminating than SIP 2, and had been passed by three cohorts of the admission test group.

Both SIP 1 and SIP 3 consist of 160 MC items and have been shown to be highly reliable. The results from main examination dates in July and September always showed a Cronbach's α > 0.90. The necessary standard for passing a SIP is 60% of the number of items that remain in the assessment. Students have the possibility to ask for the elimination of items they consider to be invalid. A SIP committee consisting of 6–8 faculty members decides about these students’ requests. The number of items which are eliminated through this procedure is typically 0–4.

Data acquisition

Anonymized data were retrieved from the campus management system at IMU, which is running on Oracle's relational database system. Standard SQL as well as some pre-aggregated views were used to retrieve the required data sets. At the date of publication, the query consists of about 400 lines of code.

All data from the years 2002–2006 were originally stored in a previous, now decommissioned system and imported into the current platform during its implementation. From 2007 onwards all data were entered and created using the currently operative platform.

Definitions

‘Drop-out’ is defined as a medical student, who has voluntarily ended (either explicitly or by discontinuation of payment of tuition fees) his/her study at IMU before graduation, without being eliminated from study by law.

‘Forced Drop-out’ is defined as a medical student, who is excluded from continuing his/her study of medicine at IMU according to IMU study regulations (i.e. after having scored negatively in the 3rd resit of an exam).

Statistical methods

The proportion of students passing SIP 1 or SIP 3 and the proportion of drop-outs as well as forced drop-outs were compared between the open admission and the admission by EMS group using the Pearson χ² test.

The expected number of trials to pass SIP 1 for these students, who finally passed it, was estimated using Poisson regression.

Methods known from survival analysis were used to assess time until passing SIP 1 in different ways. Kaplan–Meier curves were created for the two admission groups showing the cumulative probability to pass the SIP 1 starting from 0 at the beginning of the observation interval and increasing at each possible SIP 1 date. The first SIP 1 is set after the first study year, which is equivalent to 9 months after study start. Dates for possible resits are set in about 2–3-month time intervals. For the drawing of the curves, students who are still enrolled at IMU and have not passed SIP 1 yet, are included. Drop-outs and forced drop-outs only contribute to the Kaplan–Meier curves as censored observations until their drop-out date. Therefore, passing rates cannot be read off this curve. However, the curves can be used to get an overview of the time until passing for the complete observation period. Median times until passing were estimated. Furthermore, Cox regression models were used to estimate the Hazard ratio (HR) between the two admission groups, which can be interpreted as the ‘chance’ to pass at each time point during study.

For the comparison of SIP 3 success rates, all students were included in the descriptive tables. However, those admitted in 2008 were excluded from statistical testing, since there was only one possible trial for SIP 3 for them in July 2011. Therefore, comparability was not given.

The observation period also influences the comparison of drop-out and forced drop-out rates. These rates were calculated for the first three complete study years. Therefore, we chose to compare only these cohorts in which an observation period of more than 3 years was possible.

All statistical analyses were performed using the statistics program R (R Development Core Team 2011).

The study was officially approved by the Vice Dean for Studies at IMU and the IMU data protection committee (note: by Austrian law, ethical committees only deal with clinical studies and patients rights so far).

Results

Effects on academic success measured as passing of summative annual exams

In Table 1, the numbers of students admitted in years 2002–2004 and 2006–2009 are presented together with the number of students who passed SIP 1 and SIP 3. The average annual number of study beginners was substantially reduced from 602 in the open admission period to 349 in the admission by EMS period. Despite this reduction, the number of students passing SIP 1 increased both in absolute and in relative terms. 71.0% of the admitted EMS-selected students passed this exam compared with 49.1% of the unselected group. This is due to both a higher percentage of students sitting this exam (83.5% compared to 69.7%) as well as a higher passing rate (85.1% compared to 70.5%). Both differences between the admission groups were highly significant (Table 1).

Table 1.  Number and percentage of unselected (open admission) and selected (admission by EMS) students sitting and passing SIP 1 and SIP 3

	Open admission (2002–2004) N = 1805 (♂ 746, ♀ 1059)	Selected (2006–2009) N = 1395 (♂ 738, ♀ 657)	p-Value (χ^2-test)
Number of students taking SIP 1 (males/females)	1258 (69.7%^a) (533/725)	1165 (83.5%^a) (625/540)	<0.001
Number of students passing SIP 1 (males/females)	887 (70.5%^b) (402/485)	991 (85.1%^b) (523/468)	<0.001
Number of students taking SIP 3 (males/females)	871 (48.3%^a) (397/474)	657 (61.7%^c) (346/311)	<0.001
Number of students passing SIP 3 (males/females)	850 (97.6%^b) (386/464)	645 (98.2%^b) (337/308)	>0.05

Notes: ^aPercentage relative to the number of admitted students; ^bpercentage relative to the number of students sitting SIP 1 or SIP 3 and ^cpercentage relative to the number of admitted students in 2006–2008.

The second criterion of academic performance regards the number of attempts necessary to pass SIP 1. Table 2 presents the absolute numbers and cumulative percentages of students passing the exam at the respective trial. The percentage of students passing SIP 1 at the first examination date in June was significantly higher for the selected student group (Table 1). However, for those students, who passed SIP 1, the expected number of attempts modelled by a Poisson regression did not differ significantly between the different admission groups (p = 0.239).

Table 2.  Number of attempts necessary for passing SIP 1

	1	2	3	4
Number of attempts	N (cum. %)^a	N (cum. %)	N (cum. %)	N (cum. %)
Open admission group	454 (51.2)	241 (78.4)	140 (94.1)	52 (100)
Selected group	576 (58.1)	182 (76.5)	166 (93.2)	67 (100)

Note: ^aCumulative percent relative to the number of students taking SIP 1 at least once.

The third criterion refers to the time students needed to pass SIP 1 which is illustrated in Figure 1 as a cumulative probability of passing curve. The figure shows that the difference between the two groups originates at the first SIP 1 term after 9 months. From then the two curves take a parallel course. The median time to pass SIP 1 was 14 months for the open admission group and 11 months for the admission by EMS group, respectively. The HR was 1.45 (p < 0.001) for EMS-admission group compared to the unselected group which means that the chance to pass SIP 1 was 1.45 times higher for students selected by EMS than for unselected students at any time point.

Figure 1. Cumulative probability passing SIP 1 over an observation period of 45 months for students, who are still enrolled. Open admission group (unselected; blue lines) vs. selected group (red lines). The dashed lines show the 95% confidence interval of the cumulative probabilities.

Table 1 also presents the percentage of students passing SIP 3 after study year 3 relative to the number of admitted students per entrance year. The percentage of EMS-selected students passing SIP 3 was significantly higher than the respective value for unselected students. When, however, proportions were related to the number of students who had passed SIP 1 there was no statistically significant difference: 91.4% of the students with open admission who had passed SIP 1 were also successful in SIP 3, compared to 92.6% in the EMS group.

Effects on drop-out and forced drop-out ratios

In the open admission period, 663 students discontinued studying medicine, many of them without ever attempting to take SIP 1. Hence, 36.7% of the admitted students were regarded as drop-outs. After implementation of the EMS the number of drop-outs was significantly reduced to 186 (17.5%). The numbers of forced drop-outs, on the other hand, were 97 (5.4%) and 63 (5.9%) for the open admission and selected group, respectively. These numbers were not significantly different.

Effects on gender ratio

The percentages of male and female admitted students are listed in Table 3 for the open access and the EMS selection period, respectively. While in the open admission group the proportion of female medical students exceeded the percentage of males, this ratio went into reverse in the EMS group. Male applicants were significantly more successful in the EMS test ranking than their female competitors; 24.2% of male EMS participants were admitted to study human medicine but only 17.2% of female (Anonymous 2011).

Table 3.  Percentage of male and female students admitted and passing SIP 1

Type of admission	Open admission group (♂ % : ♀ %)	Selected group (♂ % : ♀ %)
Gender ratio of admitted students	41.3 : 58.7	52.9 : 47.1
Gender ratio of students passing SIP 1 (% : %)	45.3 : 54.7	52.6 : 47.4

Significant differences were observed regarding the percentage of passing SIP 1 in the admission years before the implementation of the EMS; 75.4% of unselected male students passed SIP 1 compared to 67.0% of female students relative to the number of students who took the exam, leading to a significant ‘gender gap’. Because of this difference the ratio of male/female students passing SIP 1 increased compared to the admission numbers (Table 3). When EMS-selected students were compared, there was no significant gender difference (82.1% male vs. 84.3% female), in fact female students showed a slightly better performance than males. Hence, the male/female ratios did not change between admitted and passing SIP 1 (Table 3). Analogous results were obtained, when gender was tested against the other criteria of academic performance (i.e. time until passing SIP 1; number of resits necessary to pass) in the two groups. Unselected females needed a median time of 16 months for passing SIP 1 compared to 12 months for unselected males (p = 0.016). In the EMS-admission group, both male and female students needed a median time of 11 months for passing SIP 1.

For SIP 3, there was no gender difference in passing SIP 3, neither under open admission nor with admission testing.

A significant gender difference was evident for the number of drop-outs: 31.1% of male students admitted in the years 2002–2004 (unselected group) discontinued their study at an early stage compared to 40.7% of female students. This difference was highly significant (p < 0.001; Pearson χ² test with Yates’ correction). In the EMS-selected group, the number of drop-outs was significantly reduced and there was no gender effect (proportion of drop-outs: 17.8% for males and 17.2% for females). Regarding forced drop-outs, there was again no significant difference between male and female students, neither in the unselected nor in the EMS-selected group.

Discussion

The implementation of a cognitive admission test at the IMU has lead to a clear and significant amendment of the academic performance of its students and to lower drop-out rates. Third, the ‘gender gap’ favouring men observed in academic performance and drop-out in the open admission group has disappeared after introduction of the EMS. Nevertheless, as a similar ‘gender gap’ is now observed with the EMS scores. Thus, the effect of the new admission system on gender ratio appears to be quite complex.

The improvement of academic performance can be seen regarding three outcomes: first, a higher percentage of students passed the summative exam after the first year of medical study; second, the students needed fewer attempts; and third, shorter time to pass this exam. Interestingly, the difference in passing SIP 1 was only present at the first trial which takes place at the end of June. At all further examination dates, the passing ratio was not different between the two groups. This can be observed easily in Figure 1. Also, the number of students who took SIP 1 at the first possible date was higher in the selected students. Hence, the EMS has succeeded in reaching its goals namely to select for students who are more able to advance in (and eventually finish) their study. An additional cause for the improvement of the academic performance could be the reduction in the number of students. During the open admission, the average number of students was 600 which was reduced to 350 after the implementation of the EMS. Since the number of teachers stayed the same, this led to an improvement of the teacher/students ratio. Hence, one could argue that this improvement has caused (at least partially) the change in the academic performance. The teacher/students ratio was, however, only relevant for the practical courses leading there to smaller group sizes. Since SIP 1 examines almost only the factual knowledge which has been taught in lectures of year 1, it is unlikely that the better teacher/student ratio had an effect on the passing of SIP 1.

Interestingly, the implementation of a selection procedure in the admission system did not only improve the academic performance but showed also a positive effect regarding the drop-out ratio. Contrary to other countries which use selective admission systems (Eva et al. 2004) the attrition rate had been exceptionally high in Austrian medical schools during the time of open admission. Upon introduction of a selection system the number of students who stopped studying medicine during the first year was more than halved. A similar effect was noted in the Medical University of Graz, Austria (Reibnegger et al. 2011). Even though the drop-out ratio was more than halved it is still very high when compared with other countries. In the UK the year 1 drop-out rate of medical students was between 3.5% and 4.9% (Arulampalam et al. 2007).

In a recently published prospective study, O’Neill et al. (2011a) compared two admission systems and their effect on the drop-out rate of medical students in the first 2 years of study. They compared two groups of students in Denmark. Half of the students were admitted on the basis of having achieved the highest pre-university GPAs in the applicant pool. The second half were admitted on the basis of a composite admission test score. Also, in their setting, the students who were admitted by the admission test had a lower risk of dropping out. Although there are significant differences between our study and the Danish study (e.g. different definitions of drop-out, the two compared groups were admitted in the same study year in Denmark) but the main message can be compared. Students who succeeded in a competitive testing have a higher motivation to continue their studies than students who were admitted because of their high school grades.

The analysis of the drop-out rates revealed also a significant gender effect. During the open admission period, the drop-out rate was significantly higher for female students than for male. The same difference has been found in a recent study at the Medical University of Graz, Austria (Reibnegger et al. 2011). The implementation of selection in the admission procedure abolished this difference. In a systematic review, the confounding factors for drop-out were recently analysed (O’Neill et al. 2011b). They found a significant effect in only 3 of 13 studies which included gender in their analysis a significant effect. Only in a military medical program, female gender was found to be associated with higher risk for dropping out (Stetto et al. 2004). In most medical universities, the drop-out ratio is usually lower for female students (Arulampalam et al. 2004a, 2004b, 2007; Andriole & Jeffe 2010). Hence, the situation in Austria (this study and Reibnegger et al. (2011)) has been rather unique.

The question regarding gender fairness of the EMS gained substantial attention in Austrian media because the number of female medical students was significantly reduced after the implementation of the EMS test (Spiel et al. 2008). This effect which was present only in Austria but not in Germany and Switzerland where the same test is in use has to be put into perspective that the lower academic performance and higher drop-out ratio of female students was abolished through the introduction of this test. Female students who were admitted by the EMS were undistinguishable from their male colleagues regarding passing SIP 1 and dropping out of study. During the open admission period, female students had significantly lower passing rates in SIP 1 than their male colleagues. Through the introduction of the selective admission the ‘discrimination’ of female students has been shifted from SIP 1 to the admission procedure. The causes of this ‘discrimination’ have been analysed in detail in a report authorized by the Austrian ministry of science (Spiel et al. 2008). In a recent study concerning the motivation of students of medicine in Vienna, Austria, Hofhansl (2011) found out that female students were significantly more motivated by ‘patient-orientation’ than male students. This different type of motivation together with the well-known lower performance of female applicants in competitive situations (Balafoutas & Sutter 2012) might be the reason for the underperformance of female applicants in the EMS.

The abandonment of the open admission system for medical universities in Austria might have been perceived as ‘paradise lost’, at least by those who did not qualify in the EMS. The presented data show, however, that the situation for those who were admitted can be regarded as ‘paradise regained’ since the students performed better and were less likely to drop-out.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and the writing of this article.