Abstract
Introduction: Up to now, student selection for medical schools is merely used to decide which applicants will be admitted. We investigated whether narrative information obtained during multiple mini-interviews (MMIs) can also be used to predict problematic study behavior.
Methods: A retrospective exploratory study was performed on students who were selected into a four-year research master’s program Physician-Clinical Investigator in 2007 and 2008 (n = 60). First, counselors were asked for the most prevalent non-cognitive problems among their students. Second, MMI notes were analyzed to identify potential indicators for these problems. Third, a case-control study was performed to investigate the association between students exhibiting the non-cognitive problems and the presence of indicators for these problems in their MMI notes.
Results: The most prevalent non-cognitive problems concerned planning and self-reflection. Potential indicators for these problems were identified in randomly chosen MMI notes. The case-control analysis demonstrated a significant association between indicators in the notes and actual planning problems (odds ratio: 9.33, p = 0.003). No such evidence was found for self-reflection-related problems (odds ratio: 1.39, p = 0.68).
Conclusions: Narrative information obtained during MMIs contains predictive indicators for planning-related problems during study. This information would be useful for early identification of students-at-risk, which would enable focused counseling and interventions to improve their academic achievement.
Introduction
A variety of selection procedures is used at medical schools to pick those candidates that are best able to perform successfully during their study and beyond (Siu & Reiter Citation2009; Prideaux et al. Citation2011; Monroe et al. Citation2013). Until recently, the focus has been primarily on predictors of cognitive academic performance (Salvatori Citation2001; Siu & Reiter Citation2009). Nowadays, however, it is clear that, besides cognitive skills, non-cognitive qualities like professionalism, time management and communication are important competencies of future medical students and doctors. Recent studies have shown that problematic professional behavior during medical school is predictive of future professional problems (Papadakis et al. Citation2005; Tamblyn et al. Citation2007). Other difficulties students often experience are in organizing and integrating large amounts of new information (Paul et al. Citation2009), time management, effectiveness of study strategies, and in self-regulatory and metacognitive skills (Winston et al. Citation2010). As a result, these students may be at risk with respect to their study progress.
Till date, it has proven to be difficult to assess non-cognitive, personal qualities in selection procedures validly. Judgment biases may influence selection outcomes in an unwanted manner. In addition, reliability is at stake due to poor sampling when applicants are subjected to only one interview (Prideaux et al. Citation2011). Recently, the so-called multiple mini-interview (MMI) procedure was introduced to assess non-cognitive skills more reliably (Eva et al. Citation2004). In subsequent studies, the MMI method was shown to be positively correlated with early medical school performance, and also with clerkship performance ratings and OSCE performance (Reiter et al. Citation2007; Eva et al. Citation2009). These studies show that multiple individual human judgments of non-cognitive skills when combined predict future performance in a sufficiently reliable way.
In 2007, the MMI method was introduced as part of the selection procedure for the four-year medical research master Physician-Clinical Investigator (P-CI) at Maastricht University (Guyaux et al. Citation2010). The selection procedure results in a ranking list, representing differences in predicted suitability to perform successfully in this research master. This list is used to offer the available training positions to the applicants with the highest ranks. Each year, 100–200 applicants participate in the selection procedure for the P-CI master, in which 30 training positions are available.
Although most selected students are successful in both cognitive and non-cognitive aspects of the study, some encounter professional lapses or problematic study behavior. Clearly, these problems were not predicted by the MMI scores or any other part of the selection procedure. Theoretically, the narrative information that is written down by the interviewers during the MMIs and stored in the student files could be a better predictor of such problems and could constitute a useful resource for the student mentors (called counselors in the P-CI master), but till now this information has been unused. If this assumption about the notes being better predictors is correct, the selection procedure may not only serve to provide a ranking list purely for selection purposes, but also be the starting point for custom-made teaching and an assessment-for-learning program right from the start (Shepard Citation2000). Such additional use of selection methods would mean an important step forward in terms of social accountability of educational institutes that are required to use their resources as effectively as possible (Boelen & Woollard Citation2009).
Therefore, we investigated whether narrative information obtained during MMIs in a selection procedure predicts problematic non-cognitive and study behavior during medical school, and, if so, which behavior is predicted.
Methods
Context
The four-year P-CI research master is a graduate-entry program that enables students to become medical doctor as well as clinical investigator. This combination makes it a challenging program for the students. Each year, a selection procedure decides which 30 students are allowed to enter this master. They must have finished a biomedical bachelor with good results; GPAs as well as a cognitive test are taken into account in the first part of the selection procedure. The second part consists of MMIs on different topics, such as motivation, past performance, empathy and communication skills. The applicants’ performances on each individual interview are graded independently by the interviewers as being “sufficient”, “doubtful” or “insufficient”, and the combination of all individual scores adds up to a ranking list. In each station, interviewers also make notes that are not used in the procedure itself; both notes and grading are completed in the time interval between individual interviews. The notes are stored for possible use in appeals, to underpin the interviewers' judgments.
Students and counselors
In this study, we focused on students who enrolled into the P-CI master in 2007 (cohort 2007; n = 30) and 2008 (cohort 2008; n = 30). In this master, each student is assigned to a counselor at the start of the first year, who mentors the student on an individual basis throughout his/her study. Each counselor typically takes care of 3–8 students per cohort. Every year, student and counselor meet at least four times. Before each meeting, the student had to update his/her portfolio and reflect on his/her performance based on test results, feedback and experiences. This way, the counselors get to know ‘their’ students very well, including their non-cognitive performance and study behavior.
Seven counselors mentored the 60 students in cohorts 2007 and 2008 (five in cohort 2007 and six in cohort 2008; four of them were active in both cohorts). In cohort 2007, two students discontinued their study within the first year, while in cohort 2008 three students dropped out within the first month. In the end, 54 out of 60 students have finished their study within four to five years, while one student is currently finishing the last part.
All students included in this study completed a consent form during the selection procedure, in which they consented to the inclusion of their data in a de-identified way in future scientific research. We collected all data during selection and in the course of regular educational activities, and collection did not influence selection or study results in any way. Protection of participants and their privacy was guaranteed because all information was de-identified so that the data could not be traced to individual students. Given these precautions, the study did not need formal ethical approval in our context.
Study design
This retrospective exploratory study was subdivided into three parts. First, the seven counselors were asked to name the three most prevalent non-cognitive problems they encountered in ‘their’ students, and grade them (3-2-1) to indicate the frequency of occurrence (3 = most frequent). From their reactions the two most highly-graded problems were selected for further analysis.
Second, two independent and blinded investigators (MoE and LS) analyzed the de-identified notes written down during the MMIs of 15 randomly chosen students out of the total of 55, and identified what they thought to be possible indicators for these two most frequent non-cognitive problems.
Third, a case-control study design was used. The counselors were asked to identify the students who exhibited either one or both of these non-cognitive problems during their study (cases). The notes of their MMIs were de-identified and screened by the same two independent and blinded investigators (MoE and LS) to investigate whether the proposed indicators of these problems were indeed present. As a control, the MMI notes of a similar number of control students from the same cohorts (without the identified non-cognitive problems) were screened for the presence of these indicators as well.
Statistical significance of the association between the group of students with the non-cognitive problems (cases) and the presence of indicators for these problems in the MMI notes was tested using odds ratios and confidence intervals. We used www.medcalc.org for calculation of odds ratios, confidence intervals and p-values (Bland & Altman Citation2000; Sheskin Citation2004).
Results
Part 1: The two most prevalent non-cognitive problems
All seven counselors independently identified three non-cognitive problems that they encountered most frequently in ‘their’ students and graded them (3-2-1; 3 = most frequent). shows the resulting list of non-cognitive problems; planning difficulties and self-reflection-related problems were mentioned most frequently and graded highest. Planning difficulties related to problems with time management, underestimation of study load, and problematic prioritizing of tasks. Self-reflection-related problems were addressed as insufficient awareness of (the consequences of) own functioning, indications of defensive behavior, and insufficient or non-effective actions to improve this.
Table 1. Grading of most frequently occurring non-cognitive problems by the seven counselors of cohorts 2007 and 2008.
Part 2: Indicators in MMI notes
The narrative information that was written down during MMIs with 15 randomly chosen students was analyzed to investigate whether indications for the two most prevalent non-cognitive problems were already present during the selection procedure preceding the master.
In the MMI notes of five students both investigators found no indicators at all for the two non-cognitive problems. In the MMI notes of the other 10 students one or more potential indicators were found. In four of them potential indicators for both planning-related and self-reflection-related problems were present. The two investigators identified the same indicators in all but one case; this case was discussed and agreement was reached.
Illustrations of quotes that both investigators felt to indicate planning difficulties during secondary school and/or the preceding bachelor study were:
“Without stress she has difficulty to start studying; external pressure is needed for serious planning”
“When he finds the content of a unit less interesting, he tends to study less”
“Her perfectionist nature forces her to read everything, which can cause planning problems”
“An interesting, extracurricular subject distracted her attention, causing bachelor grades to decline”.
Potential indicators of problems related to self-reflection were:
“Student indicates that she studies easily, while her bachelor grades are only just sufficient and her progression is limited”
“She characterises herself as a good planner, but at the same time (while discussing another item in the interview) indicates that she has trouble planning”.
The latter quote illustrates a lack of consistency in the candidate's opinion about her own functioning, suggesting limited self-reflection.
In addition, the following quotes were found in the notes of the interview involving a discussion with a simulated patient, in which students were asked to reflect on their own functioning:
“Self-reflection is primarily aimed at patient problem, not at own communication”
“Limited self-reflection”
“Self-reflection on empathy is insufficient”.
In one case, the notes pointed out that a student presented himself as an easy talker, who was very self-confident. After discussion, the investigators considered this to be a potential indicator of limited insight in own functioning as well.
As a result of this analysis, a limited number of potential indicators for planning-related and self-reflection-related problems were identified ().
Table 2. Potential indicators for planning-related and self-reflection-related problems in MMI notes of 15 randomly chosen students.
Part 3: Case-control study
Based on the above-mentioned findings, a case-control study was performed to investigate how predictive these indicators were for planning-related and/or reflection-related problems during the research master P-CI.
The seven counselors identified 23 students who exhibited planning-related and/or reflection-related problems during their study (cases). Thirteen students had planning-related problems, while six had reflection-related problems; another four students showed problems in both domains.
The MMI notes of the 23 identified students as well as the MMI notes of 21 control students (without planning- or reflection-related problems) were de-identified and screened for the indicators mentioned in . Indicators for planning-related problems were found in MMI notes of 23 of these 44 students, while the notes of 11 students contained indicators for reflection-related problems. Illustrations of quotes that were considered indicators for planning-related problems are:
“Study progress retarded; put a lot of time in non-study related business”
“Not well prepared for interviews; no priority”
“Has problems starting to study without (external) stress”
“When a subject is less interesting, study is postponed to the latest moment”
“Perfectionist, wants to read everything; can be a pitfall”.
Quotes considered as potential indicators for reflection-related problems:
“Poor self-reflection”
“Caveat: often reacts with ‘yes, but.,’, suggesting a defensive, non-self-reflective nature”
“Beware, a flood of words, knows how to tell everything in a convincing way”.
Altogether, the data indicate a statistically-significant association between the presence of indicators for planning-related problems in MMI notes and the actual occurrence of such problems during the subsequent study ( odds ratio 9.33; 95% confidence interval 2.12–41.07; p = 0.003). No such evidence was found for self-reflection-related problems ( odds ratio 1.39; 95% confidence interval 0.29–6.68).
Table 3. Odds ratios for the association between the presence of narrative indicators for planning-related (A) or self-reflection-related problems (B) in MMI notes taken during selection on the one hand and the actual occurrence of such problems during study of the same students on the other.
Discussion
The selection of applicants for a medical study is generally aimed at selecting those candidates that are most likely able to perform successfully during their study and also in their subsequent professional life. As a result, the selection procedure is merely used to decide on who is admitted and who is not. In the current study, we propose to use narrative information obtained during selection interviews to predict future problems in non-cognitive behavior with the intent to make counselors more attentive for early warning signs. This may enable early and dedicated counseling and remediation to improve the selected students’ study success. This way, selection will not only serve as an assessment-of-learning measure but also as a first assessment-for-learning step (Shepard Citation2000; Schuwirth & Van der Vleuten Citation2011). This approach may decrease student drop-out and prevent study delay.
Our study shows that narrative information written down during MMIs, as part of the selection procedure, contains indicators of planning-related problems that appear to be significantly predictive for the actual occurrence of such problems during medical school. Indicators of extrinsic instead of intrinsic motivation to study, priority conflicts, and/or a perfectionist character are predictive of problematic self-management that can lead to delayed and/or decreased study progress or even drop-out.
All seven counselors encountered planning-related problems among ‘their’ students; six of them indicated that this was the most frequently occurring non-cognitive problem. Up to now, counselors advise and stimulate students to improve the organization and management of their study from the moment planning-related problems expose themselves. With the current knowledge, however, counseling can be more focused right from the beginning of a study career, enabling specific educational and even therapeutic interventions. Literature shows that unorganized students suffer most from workload and pressure of progressing in their studies according to a predetermined timetable (Ruohoniemi et al. Citation2010). More than aptitude, time management and prioritizing are important for academic achievement (West & Sadoski Citation2011). Organized studying appears to be related to both study progress and success (Rytkonen et al. Citation2012). Therefore, early and dedicated counseling will help to prevent or diminish planning-related study problems and, as a consequence, improve study success. Retrospective analysis of study progress and success of the students under study suggests that the delay faced by some of them could have been prevented by the currently proposed approach.
Another non-cognitive problem frequently encountered by the counselors was limited self-reflection. Although potential indicators for this problem were identified in the MMI notes, the occurrence of these indicators was not significantly predictive of the actual occurrence of self-reflection-related problems during medical school. Adequate self-reflection is nowadays considered an essential attribute of competent healthcare professionals. This is why it is one of the important goals of our portfolio and counseling system to increase students’ awareness of the importance of self-reflection and to stimulate development of their reflective skills (Driessen et al. Citation2005). Since reflective practice is regarded as an important skill for medical students, and early and focused guidance is expected to stimulate development of this skill, further study is needed to better identify indicators for self-reflection problems.
Drop-out from or delay during medical school, in spite of selection efforts, is a cause for concern (Yates Citation2011; Stratton & Elam Citation2014). This is the case for the students involved who suffer from personal distress, for the university that is faced with a disproportionate amount of time and energy spent on struggling students, and for society that has to bear the financial burden for drop-out and delayed students in countries where they receive public funding. Early identification of struggling students, with focused guidance and remediation, will help to avoid some of this inefficiency.
Indeed, the additional use of selection data is attractive from a financial perspective. In countries like the Netherlands, where education is publicly funded, the gains of avoiding delay or drop-out will compensate largely for the costs of a selection procedure and counseling system. Education of students of the four-year P-CI master is financed by the Dutch government per registered year, for a maximum of four years, and after successful completion of the study. Costs of the curriculum, including the selection procedure, primarily depend on the number of hours invested by staff members. The total costs of selection through MMIs and intensive counseling for four years approximate 4500 euros per selected student, while drop-out of a student in the first year results in a loss of income that is more than three times higher. In addition, study delay by one year results in extra costs of about 1000 euros per student for additional counseling only, without extra income. Hence, in a context like ours prevention of drop-out or study delay is clearly worthwhile from a financial point of view. This might of course be entirely different in a country where students pay their own fees without public support.
Limitations of the study
There are some methodological issues to consider. The current study is performed using data from two student cohorts of the P-CI master. As a consequence, the population under study was relatively small (n = 55). On the other hand, both cohorts were taken into account entirely, which suggests that the current findings also hold for subsequent cohorts in this particular master. In addition, the P-CI master applies problem-based learning (PBL), in which non-cognitive skills like self-management and prioritizing are key elements for students to be successful (Dolmans & Schmidt Citation1996). It is questionable whether our findings are generalizable to non-PBL programs. Planning-related problems may be less prominent in more traditional, teacher-centered curricula.
Another issue to consider is the fact that students who successfully enter the P-CI program have already completed a prior bachelor with flying colors. Probably, this previous bachelor did not challenge all of them to employ good time management skills. By contrast, the P-CI program turns out to be very challenging for all enrolled students and demands good time management skills. The strength of the predictability of the narratives we found may not generalize to other contexts with other entry criteria. Yet, we think we demonstrated the idea that using information obtained in selection interviews as fully as possible in the form of a first step in an assessment-for-learning program is useful, but will have to be adapted to best serve the local context.
Conclusion
The present study shows that selection procedures can serve a wider purpose than merely to detect which students to accept and which not. Notes made during selection interviews contain predictive indicators for future study problems, which can be used to alert student mentors or learning coaches to be vigilant for early signs of non-cognitive problems during the study. This would provide an opportunity for early prevention of study delay and attrition and produce a larger return on the investment associated with selection procedures, and as such also contribute to the social accountability mandate of many medical schools.
Notes on contributors
MIRJAM G. A. OUDE EGBRINK, PhD, MHPE, Professor of Implementation of Educational Innovations, Department of Physiology, School of Health Professions Education, and Scientific Director, Institute for Education, Faculty of Health, Medicine and Life Sciences, Maastricht University, The Netherlands.
LAMBERT W. T. SCHUWIRTH, MD, PhD, Professor and Director Prideaux Centre for Research in Health Professions Education, Flinders University, Adelaide, Australia, and Professor of Innovative Assessment, Department of Educational Development and Research, Faculty of Health, Medicine and Life Sciences, Maastricht University, The Netherlands.
Acknowledgements
The authors are grateful to Janneke Guyaux and Koos van Geel for selecting the MMI notes and de-identifying them, and to the counselors of the 2007 and 2008 P-CI student cohorts for their cooperation.
Declaration of interest: The authors report no declarations of interest.
References
- Bland JM, Altman DG. 2000. Statistics notes. The odds ratio. BMJ 320:1468
- Boelen C, Woollard B. 2009. Social accountability and accreditation: A new frontier for educational institutions. Med Educ 43:887–894
- Dolmans D, Schmidt H. 1996. The advantages of problem-based curricula. Postgrad Med J 72:535–538
- Driessen EW, Van Tartwijk J, Overeem K, Vermunt JD, Van der Vleuten CP. 2005. Conditions for successful reflective use of portfolios in undergraduate medical education. Med Educ 39:1230–1235
- Eva KW, Rosenfeld J, Reiter HI, Norman GR. 2004. An admissions OSCE: The multiple mini-interview. Med Educ 38:314–326
- Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. 2009. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ 43:767–775
- Guyaux J, oude Egbrink MGA, Heeneman S, Houben AJHM, Willekes C, Schuwirth LWT, De Goeij AFPM. 2010. Selection based on a combination of cognitive and non-cognitive qualities. Choices and experiences with the research master Physician-Clinical Investigator (P-CI) at Maastricht. Neth J Med Educ 29:328–336
- Monroe A, Quinn E, Samuelson W, Dunleavy DM, Dowd KW. 2013. An overview of the medical school admission process and use of applicant data in decision making: what has changed since the 1980s? Acad Med 88:672–681
- Papadakis MA, Teherani A, Banach MA, Knettler TR, Rattner SL, Stern DT, Veloski JJ, Hodgson CS. 2005. Disciplinary action by medical boards and prior behavior in medical school. N Engl J Med 353:2673–2682
- Paul G, Hinman G, Dottl S, Passon J. 2009. Academic development: a survey of academic difficulties experienced by medical students and support services provided. Teach Learn Med 21:254–260
- Prideaux D, Roberts C, Eva K, Centeno A, McCrorie P, McManus C, Patterson F, Powis D, Tekian A, Wilkinson D. 2011. Assessment for selection for the health care professions and specialty training: Consensus statement and recommendations from the Ottawa 2010 Conference. Med Teach 33:215–223
- Reiter HI, Eva KW, Rosenfeld J, Norman GR. 2007. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ 41:378–384
- Ruohoniemi M, Parpala A, Lindblom-Ylanne S, Katajavuori N. 2010. Relationships between students' approaches to learning, perceptions of the teaching-learning environment, and study success: a case study of third-year veterinary students. J Vet Med Educ 37:282–288
- Rytkonen H, Parpala A, Lindblom-Ylanne S, Virtanen V, Postareff L. 2012. Factors affecting bioscience students' academic achievement. Instruct Sci 40:241–256
- Salvatori P. 2001. Reliability and validity of admissions tools used to select students for the health professions. Adv Health Sci Educ Theory Pract 6:159–175
- Schuwirth LW, Van der Vleuten CP. 2011. Programmatic assessment: From assessment of learning to assessment for learning. Med Teach 33:478–485
- Shepard LA. 2000. The role of assessment in a learning culture. Educ Res 29:4–14
- Sheskin DJ. 2004. Handbook of parametric and nonparametric statistical procedures. Boca Raton, FL: Chapman & Hall/CRC
- Siu E, Reiter HI. 2009. Overview: What's worked and what hasn't as a guide towards predictive admissions tool development. Adv Health Sci Educ Theory Pract 14:759–775
- Stratton TD, Elam CL. 2014. A holistic review of the medical school admission process: examining correlates of academic underperformance. Med Educ Online 19:22919
- Tamblyn R, Abrahamowicz M, Dauphinee D, Wenghofer E, Jacques A, Klass D, Smee S, Blackmore D, Winslade N, Girard N, et al. 2007. Physician scores on a national clinical skills examination as predictors of complaints to medical regulatory authorities. JAMA 298:993–1001
- West C, Sadoski M. 2011. Do study strategies predict academic performance in medical school? Med Educ 45:696–703
- Winston KA, Van Der Vleuten CP, Scherpbier AJ. 2010. At-risk medical students: implications of students' voice for the theory and practice of remediation. Med Educ 44:1038–1047
- Yates J. 2011. Development of a 'toolkit' to identify medical students at risk of failure to thrive on the course: an exploratory retrospective case study. BMC Med Educ 11:95