Abstract
Background
The purpose of this study was to examine the predictive potential of multiple indicators (eg, preadmission scores, unit, module and clerkship grades, course and examination scores) on academic performance at medical school, with a view to identifying students at risk.
Methods
An analysis was undertaken of medical student grades in a 6-year medical school program at the Faculty of Medicine and Health Sciences, United Arab Emirates University, Al Ain, United Arab Emirates, over the past 14 years.
Results
While high school scores were significantly (P < 0.001) correlated with the final integrated examination, predictability was only 6.8%. Scores for the United Arab Emirates university placement assessment (Common Educational Proficiency Assessment) were only slightly more promising as predictors with 14.9% predictability for the final integrated examination. Each unit or module in the first four years was highly correlated with the next unit or module, with 25%–60% predictability. Course examination scores (end of years 2, 4, and 6) were significantly correlated (P < 0.001) with the average scores in that 2-year period (59.3%, 64.8%, and 55.8% predictability, respectively). Final integrated examination scores were significantly correlated (P < 0.001) with National Board of Medical Examiners scores (35% predictability). Multivariate linear regression identified key grades with the greatest predictability of the final integrated examination score at three stages in the program.
Conclusion
This study has demonstrated that it may be possible to identify “at-risk” students relatively early in their studies through continuous data archiving and regular analysis. The data analysis techniques used in this study are not unique to this institution.
Introduction
Admission to medical school is generally hotly contested, with many more applicants than available places.Citation1 Because medical training is resource-intensive, it is important to select students who are capable of developing into competent and safe doctors. The dilemma facing medical faculties, however, is which selection criteria best predict student “success.”Citation2,Citation3 Ultimately, success should represent physician competence,Citation4–Citation7 but this is a difficult task because many cognitive and noncognitive factors influence competence.Citation1,Citation6–Citation9 Individual medical faculties thus often develop unique entrance criteria encompassing more than one domain. Despite the increasing use of nonacademic factors, such as attitude and empathy, in the selection process, preadmission academic achievement remains an important consideration.Citation1 However, the evidence is conflicting in terms of the value of academic criteria, such as the secondary school grade, in terms of academic success at medical school, largely because of the variability in secondary school education.Citation1,Citation8,Citation10
The admission process is only the beginning of a long journey to graduation. In line with a learner-centered philosophy, once accepted, students should be continuously monitored, so that those “at risk” at different stages of their studies can be identified, and the appropriate support and remediation provided.Citation11,Citation12 Identifying “at-risk” students becomes increasingly important in the context of widening access, as would be the case for promoting ethnic minorities or addressing previous political disadvantage.Citation13–Citation15 The situation at the Faculty of Medicine and Health Sciences, United Arab Emirates University, the only federal medical school in the United Arab Emirates, would certainly be considered a case of widening access. Established in the 1980s, the Faculty of Medicine and Health Sciences has a social responsibility to train Emirati doctors to service the national (Emirati) and expatriate United Arab Emirates communities. Since graduating its first Emirati doctors in 1994, student numbers have steadily increased in line with a national emiratization program. In recent years, this has culminated in the easing of some medical school entrance criteria, necessitating the identification of potentially “at-risk” students early and tracking their progress through their studies.
Institutional context
The Faculty of Medicine and Health Sciences curriculum comprises three 2-year courses, ie, medical sciences, organ systems, and clinical sciences. During medical sciences, organ systems, and clinical sciences, students complete 10 units, 11 modules (including two clinical skills modules), and 10 clerkships, respectively (). There is continuous assessment over the 2-year period to determine eligibility for externally reviewed course examinations. During clinical sciences, students rotate through clerkships in internal medicine, surgery, pediatrics, psychiatry, obstetrics and gynecology, community medicine, and family medicine (). The externally examined final integrated examination is the high-stakes licensing examination, comprising clinical (objective structured clinical examination, patient cases) and theory (inhouse, National Board of Medicine Examiners comprehensive clinical examination) components.
Table 1 Faculty of Medicine and Health Sciences 6-year curriculum structure indicating units, modules, clerkships, course averages, and examinations in chronological order
All applicants to the English medium Faculty of Medicine and Health Sciences medical program are Emirati nationals (Arabic speakers) who attended either a public (Arabic medium) or a private (English medium) secondary school and who successfully completed a general university foundation year. Final selection depends on a student’s high school grade, a satisfactory international English examination score, a Medical College Admission Test, and an interview. Despite these selection criteria, there is a small annual attrition, and a number of students are considered “at-risk” or borderline at different stages of the program. Therefore, the present exploratory study set out to analyze historic data to identify potential predictors of success in the final integrated examination and whether there were points in the curriculum where “at-risk” students could be identified, such that support and remediation can be provided.
Two main research questions were posed in this study, ie, the extent to which available preadmission grades (including English) predict performance (or lack thereof) during the 6-year medical program, and which scores during a student’s medical studies best predict performance (or lack thereof) during the 6-year program.
Methods
Database queries of all available student grades from high school and university medical studies were conducted. Ethical approval for the study was obtained from the Al Ain Medical District human research ethics committee. Preadmission grades were high school grades, the United Arab Emirates University placement assessment, which is the common educational proficiency assessment (comprising English and mathematics), and an international English examination (eg, TOEFL® or the International English Language Testing System). In addition, Faculty of Medicine and Health Sciences grades comprise results from units, modules and clerkships; year averages, as well as end of course examinations, final integrated examination, and National Board of Medicine Examiners results. Analyses were performed on three data sets:
Ten cohorts of medical students for whom comprehensive grades were available from high school to graduation (n = 297)
All Faculty of Medicine and Health Sciences grades (unit, module, clerkships, year averages, course examinations) to determine their predictive ability for all components of the program
A multivariate linear regression of grouped grades for the first 2 years (medical sciences), the first four years (medical sciences and organ systems), and in the 6-year curriculum (medical sciences, organ systems, and clinical sciences) to determine components of the program that offer the greatest predictability of the final integrated examination grade.
Statistical analysis
Pearson’s correlation (r) with a statistical significance level of P < 0.01 was used to identify significant correlations between grades. R2 was used to identify the level of variability in performance explained by the predicting variable (predictability). R2 values range between 0 and 1, with values close to 0 indicating low predictability and values close to 1 indicating high predictability. R2 values can also be depicted as percent predictability,Citation16–Citation18 which, for the purposes of this study, have been categorized as: >60% high predictability; 40%–60% moderate predictability; and <40% as low predictability.
Results
The total number of students for the 14-year study period was 792 (228 males, 564 females), with the average graduating cohort comprising 32 students (10 males, 22 females). Because the Common Educational Proficiency Assessment, TOEFL, and the International English Language Testing System were introduced in the 2000/2001 academic year only, these scores were not available for some students. Some grades for the currently enrolled students (ie, 2005–2010 intakes) were also not available.
Grades available on admission
Although significant correlations were measured for the three preadmission grades (high school grade, Common Educational Proficiency Assessment, English), predictability was low in the short and long term (). The high school grade best predicted performance in unit 1 (13.8%). Common Educational Proficiency Assessment and English language grades generally did not predict performance.
Table 2 Predictability (percent) of various preadmission criteria and selected program grades (darker background indicates higher predictability)
Above average, average, and below average students
For the 10 cohorts for which comprehensive grades were available, students were categorized into the “above average” (top 20th percentile), “average” (middle 60th percentile), and “below average” (bottom 20th percentile, ). However, it is important to note that the average does not represent the same students in each group. Apparent from is the variability of grades for different program components. Of interest is the spread of unit 1 results following the high school grade clustering and the components that discriminate between the above average, average, and below average groups, in terms of the final integrated examination scores (eg, unit 5, unit 10, medical sciences examination, module 8, organ systems examination, clinical sciences year averages). These data were analyzed further.
Figure 1 Profile of above average (top 20th percentile), average (middle 60th percentile,) and below average (bottom 20th percentile) grades for students for whom all scores were available from the high school grade to the final integrated examination (n = 297).
Medical school grades during the six-year program
suggests that a student’s performance in the introductory unit, ie, unit 1, may be an early indicator of subsequent performance during medical sciences, with predictability being 51% for the medical sciences year 1 average, 40% for the medical sciences year 2 average, and 26% for the medical sciences examination. Similarly, unit 10 and the medical sciences examination are identified as moderate predictors of performance in many organ systems modules and high predictors of organ systems averages.
Medical sciences, organ systems, and clinical sciences examination grades were significantly correlated with the average scores for the 2-year period comprising the course (59%, 65%, 56% predictability; P < 0.05, 0.001, 0.001; n = 415, 321, 244, respectively). The averages for year 1 and year 2 of medical sciences or organ systems were significantly correlated (P < 0.001), with high predictability, ie, 83.2% for medical sciences (n = 514) and 89.9% for organ systems (n = 396). The predictability of the clinical sciences year 1 average for the clinical sciences year 2 average was lower at 45.8% (P < 0.001, n = 283).
Medical school grades and final integrated examination success
confirms the potential value of some grades identified in . To this end, unit 10 (38% predictability), the organ systems examination (41%), clinical sciences year 1 average (49%), clinical sciences year 2 average (43%), overall medical school average (56%), and the National Board of Medicine Examiners examination grade (35%) offered the greatest predictability of final integrated examination success.
Figure 2 Predictability of preadmission grades and all medical school grades for the final integrated examination.
Abbreviations: The abbreviations used in this figure are shown in .
Figure 3 Predictability of year averages and course examinations for the final integrated examination.
Abbreviations: The abbreviations used in this figure are shown in .
Because year averages determine whether a student qualifies for the course examination, and the course examinations determine a student’s progression to the next phase, their predictive ability warrants consideration. Predictability for the final integrated examination increased from the medical sciences averages and examinations (19%–27%), through organ systems (31%–35%) to clinical sciences grades (43%–49%, n = 242). The overall medical school average, available one month before the final integrated examination, accounted for the greatest predictability (56%, P < 0.001, n = 245). Final integrated examination grades were also significantly correlated (35% predictability, P < 0.001, n = 154) with the National Board of Medicine Examiners clinical sciences examination scores.
Multiple regression modeling
Having identified individual grades, grade averages, and course examinations as potential predictors of the final integrated examination results, the next undertaking was a stepwise multivariate linear regression (adjusted R2) using three sets of grades (all medical sciences grades, all medical sciences and organ systems grades, all six-year grades). All final integrated examination marks available up to, but not including, the 2009/2010 academic year were included. The analysis identified the following:
First 2 year (medical sciences) grades: unit 10 and medical sciences course final examination in order of predictive value were identified as key grades (out of 15 grades) providing 40.5% predictability of the final integrated examination (n = 128).
First 4 year (medical sciences and organ systems) grades: OSC Average, unit 10, and clinical skills year 2 in order of predictive value were identified as key grades (out of 30 grades) providing 53.3% predictability of the final integrated examination (n = 122).
6 year grades (medical sciences, organ systems, and clinical sciences): overall medical school average, organ systems year 1, medical sciences course final examination, organ systems course final examination, unit 9, MSC Average, organ systems year 2, clerk 3, and clerk 6 in order of predictive value were identified as key grades (out of 44 grades) providing 79.9% predictability of the final integrated examination (n = 119).
When the model was tested comparing the actual versus predicted 2009/2010 final integrated examination grades, the average predictive error decreased from 2.6 marks, 4 years before the final integrated examination, to 2.2 marks, 2 years before the final integrated examination, with 1.8 marks in average predictive error a month prior to the final integrated examination ().
Figure 4 Regression analysis model: Predicted versus actual final integrated examination grade using 2009–2010 grades.
Summary of results
Based on the findings of this study, unit 1 (the first learning encounter in the program) can be used to identify struggling students early
The medical sciences year 1 average is a useful indicator of performance in medical sciences year 2
Unit 10 (end of medical sciences year 2) may be a key indicator of performance in subsequent years, including the overall average 4 years later
Organ systems average and organ systems examination grades appear to be useful indicators of performance 2 years in advance of the final exit examination
Regression analysis identified key components within the program that are highly associated with final exit examination success, allowing for prediction of performance well in advance of this final exit examination
Graphic depiction of data, such as , allows one to identify anomalies in curriculum components (eg, poor discriminators of student performance, excessively high or low summative assessment).
Discussion
This exploratory study set out to investigate the potential predictive ability of preadmission grades and various summative assessment results during a 6-year medical program, with the view to identifying “at-risk” students early and at different stages of their studies. Although we did not find the “smoking gun,” ie, a single predictor of long-term student performance, we believe that this study has highlighted a number of important issues in terms of admission criteria, curriculum components that require attention, and possible remediation points during the 6-year program.
With regard to the predictive ability of preadmission grades (high school grade, English, Common Educational Proficiency Assessment), unlike some reports of the usefulness of the high school grade during medical studies and beyond,Citation2,Citation6,Citation19,Citation20 the present study found that the high school grade, a major Faculty of Medicine and Health Sciences admission criterion, is a poor predictor of performance, even in the short term. This may reflect the high school grade cutoff (>90%) and hence the low discriminatory power. Because there is evidence suggesting that premedical school science may be correlated with success at medical school,Citation1,Citation2 individual mathematics, biology, or other science grades may be more appropriate as potential predictors than the comprehensive high school grade.
The low correlation of English language scores with other grades was surprising, because much has been written about the difficulties of studying medicine as an English second language learner.Citation22–Citation28 A recent study of the transferable skills of incoming Faculty of Medicine and Health Sciences medical students who were followed up a year later, suggest that language is a barrier to academic achievement.Citation29 Like the high school grade, the low predictability of English and future performance may lie in its poor discriminatory power because applicants generally just meet the minimum requirement (500 for TOEFL, 5 for the International English Language Testing System). Notwithstanding this result, we believe that English language proficiency does influence success in the early years of medical studies. The next step is to consider the predictive ability of different components of the International English Language Testing System (ie, listening, speaking, reading, writing).
Because it is unlikely that United Arab Emirates secondary schooling will be standardized in the foreseeable future, alternative selection criteria need to be considered. Admission tests, such as the Medical College Admission Test, which probably offer greater objectivity than the high school grade, are regularly used elsewhere in the admission process. Although Lynch et al found that the UK Clinical Aptitude Test failed to predict year 1 performance at two medical schools,Citation10 the US Medical College Admission Test predicted performance in all three US Medical Licensing Examination steps.Citation21 In the early years of the Faculty of Medicine and Health Sciences medical program (ie, in the 1990s), Abdulrazzaq et al found that the grade point average, a measure of overall medical school performance, was significantly correlated with an admission test.Citation3 In recent years, the Faculty of Medicine and Health Sciences admission process has included a Medical College Admission Test-type examination but, because the results are not archived electronically, they could not be used for this study.
Having admitted students with variable secondary schooling and English language proficiency, features common to many medical schools, the next task is to identify curriculum components that predict performance early in the program, at different stages and, ultimately, in the final licensing examination and beyond graduation.Citation3,Citation4,Citation6 Although our analysis did not yield a single predictor, it has been possible to identify different points in the program where borderline students can be flagged. To this end, the first 6-week unit (introduction to medical sciences), with reasonable predictability for medical sciences year averages and the course examination, is suitable for identifying struggling students at the outset of their studies. Based on our experiences this past year, with a large number of underprepared students in terms of language and maturity, unit 1 is currently being restructured such that students are provided with extra language and study skills support as they get to grips with the fundamentals of chemistry, medical biology, and physics.
For early identification of “at-risk” students, stepwise multivariate linear regression analysis added value in that the predictive power of this model in terms of predicting success in the final exit examination exceeded that of any individual curriculum component. This translated into 40.5% predictability 4 years in advance of the final integrated examination, 53.3% predictability 2 years in advance, and 79.9% predictability a month before the final integrated examination. At each stage, students could be flagged and supported with remediation and then tracked for progress.
In undertaking such a comprehensive analysis, the correlation between two curriculum components may not always be explicable. For example, Kozar et al,Citation30 while attempting to identify students at risk of failing year 4 National Board of Medicine Examiners surgery, found that the apparently unrelated year 2 pathology was the best predictor of success for this examination. In the present study, the moderate (42% predictability) of unit 1 for the final National Board of Medicine Examiners clinical sciences examination almost 6 years later is difficult to explain, but warrants further investigation. It may be that the long vignettes in the National Board of Medicine Examiners require a good command of English, as well as critical thinking skills. Successful students in unit 1 may therefore be individuals with higher English and/or higher order cognitive abilities.
Although this study began as an exploratory exercise of analyzing 14 years of data to ascertain the predictive value of various summative assessments, the analysis yielded some unanticipated benefits. To this end, certain curriculum components should be investigated, eg, the exceptionally high grades for clerkship 2 (surgery year 1) while clerkship 10 (family medicine) appears not to discriminate between the performance of below average, average, and above average students. The significant correlation between the Faculty of Medicine and Health Sciences course examinations and the National Board of Medicine Examiners examination is encouraging, suggesting that in the cognitive domain, Faculty of Medicine and Health Sciences students’ achievements may be comparable with an international benchmark.
While this study reflects the situation at one medical school, the principles are generalizable. We hope that readers recognize the long-term and short-term benefits of electronic data archiving of all student grades. Archiving allows the generation of regular snapshots of summative assessments to ensure that a degree of uniformity of results exists. More importantly, as this study has shown, it allows medical educators to identify “at-risk” students at key points in the program such that remediation can be undertaken. The earlier such students can be flagged, the more likely these will be a positive outcome. We believe that we can do this as early as the first 6 weeks of our program, but certainly within the first 2 years.
Conclusion
Ideally, any medical educator would like to identify students requiring remediation at the earliest possible stage. With the variability in secondary schooling in most countries, the preadmission grades may not offer any discriminatory power in predicting performance during medical school. Medical schools thus need to make maximum use of data collected before and during the program. The data analysis techniques used in this study are not unique to this institution. Provided medical schools continuously archive summative assessment grades electronically, a faculty or support staff member familiar with database queries could perform such analyses and generate similar reports. We believe that this study has demonstrated that through continuous data archiving and regular analysis, it may be possible to identify “at-risk” students relatively early in their studies. As Hamdy et alCitation7 point out in their best evidence medical education review, the literature on the long-term predictive value of undergraduate assessment systems is relatively sparse and warrants attention. From the exploratory findings of the present study, while long-term prediction (ie, over 6 years) may be difficult, there is educational merit in identifying grades that offer better short-term predictability. We are relatively confident of identifying “at-risk” students 4 years in advance of the final exit examination but more confident 2 years in advance. In both instances, there is sufficient time to support, remediate, and monitor students as they progress through their studies. Unlike most medical schools, where the annual intake exceeds 150 students, cohorts at the Faculty of Medicine and Health Sciences are small but increasing. Notwithstanding, we believe that, in view of the 14-year period of data collection, the trends that have been identified are a true reflection of future students’ performances at Faculty of Medicine and Health Sciences.
Acknowledgments
We wish to thank the Faculty of Medicine and Health Sciences and the University General Requirements Unit Student Records departments for providing us with the data used in this study.
Disclosure
The authors report no conflicts of interest in this work.
References
- JamesDChilversCAcademic and non-academic predictors of success on the Nottingham undergraduate medical course 1970–1995Med Educ2001351056106411703642
- Al-NasirFARobertsonASCan selection assessments predict students’ achievements in the premedical year? A study at Arabian Gulf UniversityEduc Health (Arbingdon)200114277286
- AbdulrazzaqYMIbrahimKCould final year school grades suffice as a predictor for future performance?Med Teach1993152432518246721
- VeloskiJHermanMWGonnellaJSZeleznikCKellowWFRelationships between performance in medical school and first postgraduate yearJ Med Educ197954909916513092
- GonnellaJHojatMRelationship between performance in medical school and postgraduate competenceMed Educ198358679685
- LumbABVailAComparison of academic, application form and social factors in predicting early performance on the medical courseMed Educ2004381002100515327683
- HamdyHPrasadKAndersonMBBEME systematic review: Predictive values of measurements obtained in medical schools and future performance in medical practiceMed Teach20062810311616707291
- McManusICSmithersEPartridgePKeelingAFlemingPRA levels and intelligence as predictors of medical careers in UK doctors: A 20 year prospective studyBMJ200332713914212869457
- DeAngelisSNoncognitive predictors of academic performance. Going beyond the traditional measuresJ Allied Health200332525712665294
- LynchBMacKenzieRDowellJClelandJPrescottGDoes the UKCAT predict year 1 performance in medical schoolMed Educ2009431203120919930512
- McLeanMGibbsTJLearner-centred medical education: Improved learning or increased stress?Educ Health (Abingdon)20092228720029762
- McLeanMGibbsTTwelve tips to designing and implementing a learner-centred curriculum: Prevention is better than cureMed Teach20103222523020218837
- HolmesDEight years’ experience of widening access to medical educationMed Educ20023697998412390467
- JamesDFergusonEPowisDSymondsIYatesJGraduate entry to medicine: Widening academic access and socio-economic accessMed Educ20084229430018275417
- Van WykJMMcLeanMFutreEMUsing evaluation in a problem-based learning curriculum to promote transformationReview of Further and Higher Education and Training200714560
- PeatJBartonBMedical Statistics: A Guide to Data Analysis and Critical AppraisalOxford, UKBlackwell Publishing2008
- WikipediaCoefficient of determination Available from: http://en.wikipedia.org/wiki/R-squareAccessed December 20, 2010.
- UpshurREGMoineddinRCrightonEKieferLMamdaniMSimplicity within complexity: seasonality and predictability of hospital admissions in the province of Ontario 1988–2001, a population-based analysisBMC Health Serv Res200551315693997
- HaidingerGFrischenschlagerOMitterauerLPrediction of success in the first-year exam in the study of medicine – a prospective surveyWien Klin Wochenschr2005117827832 German16437320
- HaidingerGFrischenschlagerOMitterauerLReliability of predictors of study success in medicineWien Klin Wochenschr2006156416420 German
- JulianERValidity of the medical college admission test for predicting medical school performanceAcad Med20058091091716186610
- AhmedBAhmedBLAl-JouhariMMFactors determining the performance of medical students in the Faculty of Medicine, University of KuwaitMed Educ1988225065083226343
- Chur-HansenALanguage background, proficiency in English, and selection for language developmentMed Educ1997313123199488850
- Chur-HansenAVernon-RobertsJClinical teachers’ perceptions of medical students’ English language proficiencyMed Educ1998323513569743794
- DhaliwalGTeaching medicine to non-English speaking background learners in a foreign countryJ Gen Intern Med20092477177319381729
- HassanMOGumaaKAHarperAHeseltineGFDContribution of English language to the learning of basic medical science in Sultan Qaboos UniversityMed Teach199517277282
- HayesSCFarnillDMedical training and English language proficiencyMed Educ1993276148433662
- LucasPLenstrupMPrinzJWilliamsonDYipHTipoeGLanguage as a barrier to the acquisition of anatomical knowledgeMed Educ19973181869231106
- McLeanMShabanSMurdoch-EatonDGeneric skills development early in the undergraduate medical curriculum: What does student self-assessment tell us? (In preparation)
- KozarRAKaoLSMillerCCSchenartsKDPreclinical predictors of surgery NBME exam performanceJ Surg Res200714020420717509265