Abstract
Introduction
This paper compares the panel interview (PI) performance with the multiple mini interview (MMI) performance and indication of behavioural concerns of a sample of medical school applicants. The acceptability of the MMI was also assessed.
Materials and methods
All applicants shortlisted for a PI were invited to an MMI. Applicants attended a 30-min PI with two faculty interviewers followed by an MMI consisting of ten 8-min stations. Applicants were assessed on their performance at each MMI station by one faculty. The interviewer also indicated if they perceived the applicant to be a concern. Finally, applicants completed an acceptability questionnaire.
Results
From the analysis of 133 (75.1%) completed MMI scoresheets, the MMI scores correlated statistically significantly with the PI scores (r=0.438, p=0.001). Both were not statistically associated with sex, age, race, or pre-university academic ability to any significance. Applicants assessed as a concern at two or more stations performed statistically significantly less well at the MMI when compared with those who were assessed as a concern at one station or none at all. However, there was no association with PI performance. Acceptability scores were generally high, and comparison of mean scores for each of the acceptability questionnaire items did not show statistically significant differences between sex and race categories.
Conclusions
Although PI and MMI performances are correlated, the MMI may have the added advantage of more objectively generating multiple impressions of the applicant's interpersonal skill, thoughtfulness, and general demeanour. Results of the present study indicated that the MMI is acceptable in a multicultural context.
Admission to medical schools is highly competitive. Pre-admission academic ability is a prerequisite universally as it is the best predictor of academic performance during training, although its relationship with clinical performance is less clear (Citation1, Citation2). As the number of academically able applicants exceeds the number of training places available, additional measures such as the aptitude test, personal statement, and interview are used. The admission interview is used to determine whether candidates possess the requisite personal traits suitable for medical training and effective communication skills, and it also serves as an opportunity to examine whether prospective students are truly interested in medicine and the particular school applied for.
The panel or board interview is commonly used in addition to pre-admission academic qualifications to aid decisions (Citation3). Students selected by the interview have been reported to perform better, for example, in communications Objective Structured Clinical Examination (OSCE), compared with those selected by academic merit alone (Citation4). However, the reliability and validity of the panel interview (PI) have been increasingly questioned (Citation5, Citation6). Structuring the interview has been reported to enhance its acceptability and reliability (Citation7). The multiple mini interview (MMI) has been designed to be highly structured, resembling the OSCE (Citation8).
In the MMI, candidates rotate around a series of short interviews that assesses non-cognitive attributes currently assessed in the PI. The stations are usually designed in such a way that they do not require any specific learned knowledge but rather evaluate the applicant's ability to logically work through a problem and express their ideas clearly. The stations may be designed to involve discussing one's viewpoint on a particular social ethical dilemma with one or more faculty or lay interviewers, role-playing with an actor or simulated patient, or completing a specific task. Candidates may be assessed for critical thinking, communication skills, and ability to coherently discuss particular social dilemmas. It is suggested that the MMI allows a wide sampling of applicants’ competencies so that a more accurate picture of their overall ability may be assessed. Furthermore, in some MMIs, interviewers are asked to raise a ‘red flag’ or indicate a ‘concern’ if they felt the applicant was behaving inappropriately at an MMI station, that is, overly aggressive, timid, rude, immature, etc.
Typically the MMI consists of 10 stations, each lasting 8 min and assessed by one interviewer, and has been reported to be reliable, acceptable and feasible (Citation9). The multiple assessments of applicants by different interviewers across different domains increase the reliability of the MMI. Compared to the autobiographical sketch, pre-admission academic qualifications, aptitude test and PI, performance at the MMI has been reported to be a statistically significant predictor of success in early years at medical school training (Citation10, Citation11). Applicants who performed well at the MMI have also been reported to do better at national licensing examinations (Citation12). Medical schools that have documented using the MMI have reported that it is acceptable (Citation13–Citation15) and feasible (Citation16). However, most reports of MMI have been from the Western experience. Its acceptability in other contexts has rarely been reported.
Although the rise in the use of MMIs shows that the traditional PI is now becoming outdated as a method to be used in isolation, continuous research is necessary to further build the evidence base to show that the MMI is the best way to select future doctors. In this paper, we compare the PI performance of a sample of medical school applicants to their MMI performance, and evaluate the acceptability of the MMI in a multiracial and non-Western context.
Methods
Sample selection
Ethics approval was obtained from the Institutional Research and Ethics Committee. Applicants in an MBBS programme in one medical school shortlisted for the PI were invited to participate in an MMI pilot. A letter of invitation and an information leaflet on the MMI process were given to potential participants via email. The applicants were assured that the outcome of the MMI would not affect their selection for entry into the medical programme. Selection was based on academic performance and the PI. Confidentiality of MMI and PI performances was assured. Written consent was obtained from the applicants. This also included consent for the use of examination results for the purpose of data analysis. Participation was on a voluntary basis. The interviews were conducted during September/October 2013. Interviewers were recruited from academic faculty. They were given a one-day training session and an MMI assessment manual. On the days of the interviews, a reminder briefing was given.
Data collection
Those applicants who consented to participate attended on the days that both the PI and MMI were administered. Each applicant was first interviewed by a panel of two faculty interviewers for 30 min. Five panels interviewed one applicant each concurrently for 30 min, resulting in 10 applicants interviewed in 60 min. After the PI, the 10 applicants were briefed for 30 min on the MMI before they participated in the MMI. The MMI stations’ interviewers were different from those who had taken part in the PI. This was to avoid the influence of preconceived impressions of the applicants generated in the PI.
The MMI consisted of 10 stations each of 8 min, including 2 min of preparation time. Each station was assessed by one faculty interviewer. One station required the applicant to complete a task with a helper, two required the applicant to role-play with an actor, and the remaining seven stations consisted of one-to-one interviews. Each station tested a specific attribute and the attributes tested were: ability to be a team player, conscientiousness, altruism, ability to cope with stress, ability to summarise, decision-making skills, tolerance and adaptability, empathy, career motivation, and honesty and integrity.
Data were collected on age, sex, ethnicity, and academic qualifications at entrance. The various pre-admission qualifications were banded to categorise academic ability, that is, Band 1 – higher academic ability, Band 2 – moderate academic ability, and Band 3 – lower academic ability. This was necessary so that applicants may be compared using a single classification system as the pre-admission qualifications of applicants were rather diverse, such as ‘A’ levels, the national sixth form qualification, the International Baccalaureate, and the Australian and Canadian matriculation. During the PI, each interviewer assessed the ‘personal characteristics’, ‘desirable attitudes and values of the future healthcare professional’, ‘appropriate skills for the future healthcare professional’, ‘ability to make decisions and tolerate uncertainty’, and ‘communication skills’. A total maximum score of 20 was possible. The average of the two interviewers was taken as the final score. In the MMI, the score at each station was standardised to 10, giving a possible total maximum score of 100. At each station the interviewer was asked to indicate a concern if the subject was overly aggressive, timid, arrogant, rude, immature, etc. The subjects were categorised into the number of stations at which they were indicated as a concern. At the end of the MMI, the subjects were asked to complete an anonymous questionnaire on the acceptability of the MMI. Examples of questions asked were, ‘I received adequate information prior to my interview about the MMI’, ‘The pre-MMI briefing on the day prepared me for the MMI’, and ‘The instructions at the MMI stations were clear and easy to understand’. Each question was scored on a 5-point Likert scale from ‘Strongly Disagree’ (Citation1) to ‘Strongly Agree’ (Citation5).
Data analysis
The mean PI and MMI scores with 95% confidence intervals (CIs) were calculated and compared by age, sex, race, and academic ability using t test and ANOVA. The number of stations at which subjects were indicated as a concern was cross-tabulated against age, sex, race, and academic ability for association using the chi-square test. Pearson's correlation analysis was carried out to test for association between PI and MMI. The t test was carried out for association between number of stations indicated as concern and PI and MMI scores. Frequency distributions for acceptability questions were calculated. The mean score for each item was tested for association with sex and race using the t test.
Results
Of 177 participating applicants, 131 (75.1%) MMI scoresheets were fully completed. Seventy-eight (59.5%) subjects were females compared with 53 (40.5%) males, and 109 (83.2%) subjects were aged 19 or younger and 22 (16.8%) were aged 20 or older (). Ninety-two (70.2%) subjects were Chinese, whereas 12 (9.2%) were Malay, 18 (13.7%) were Indian, and 9 (6.9%) were others. Academically able (Band 1) subjects numbered 70 (53.4%) compared with 23 (17.6%) moderately able (Band 2) and 38 (29.0%) less able (Band 3) subjects. The mean PI score was 15.7 (95% CI 15.2–16.1) and mean MMI score was 72.5 (95% CI 70.8–74.1), both of which were not statistically significantly associated with sex, age, race or academic ability. One-hundred-and-twenty (91.6%) subjects were assessed as a concern at one station or not at all, whereas eleven (8.4%) were assessed as a concern at two or more stations (). Assessment of a concern was not statistically significantly associated with sex, age, race, and academic ability.
Table 1 Comparison of mean PI and MMI scores with 95% confidence intervals by sex, age, race, and pre-university academic ability (n=131)
The correlation coefficient between total PI and MMI scores was 0.438 (p=0.001), although it should be noted that for any PI score, the range of MMI scores was quite wide (). For example, for the PI score of 15, the MMI scores ranged from 60 to 90.
Fig. 1. Correlation between PI and MMI scores: Pearson's correlation coefficient = 0.438 (p=0.001).
The mean MMI score for those assessed as a concern at one station or none was statistically significantly higher at 73.5 (95% CI 71.9–75.1) compared with those assessed as a concern at two stations or more at 61.7 (95% CI 55.1–68.4, p=0.001) (). However, mean PI scores for both groups were not statistically significantly different.
Table 2 Comparison of mean PI and MMI scores (95% CI) by assessment of concern (n=131)
Most subjects strongly agreed (36.9%) or agreed (35.2%) that they received adequate information about the MMI prior to their interview (). However, relatively fewer strongly agreed (9.1%) or agreed (24.0%) that they used information from various sources to prepare for the MMI. Most strongly agreed (27.4%) or agreed (40.0%) that the pre-MMI briefing was helpful. The majority strongly agreed (30.7%) or agreed (51.7%) that the instructions at the MMI stations were clear. Similarly, subjects also strongly agreed (27.4%) or agreed (49.1%) that it was clear to them what the MMI was assessing. Over two-thirds strongly agreed (32.4%) or agreed (35.2%) that they enjoyed the MMI. Just over one-third reported that they strongly agreed (13.6%) or agreed (22.7%) that they felt stressed before the MMI. Almost all strongly agreed (59.1%) or agreed (30.1%) that the interviewers were polite and respectful during the MMI.
Table 3 Number of subjects (%) responding to the acceptability questions on the MMI (n=173)
The mean scores for each of the acceptability items were compared between sex and race categories, but no statistically significant difference in scores were detected ().
Table 4 Comparison of mean scores (95% CI) for questions on acceptability of the MMI by sex and race
Discussion
This study was carried out to compare the performance of a sample of medical school applicants at a PI and an MMI, evaluate the acceptability of the MMI in a multiracial context, and determine the logistic feasibility of the MMI compared to the PI. The key findings were that PI and MMI performances were generally not associated with sex, age, race, and entrance academic ability, PI and MMI scores are statistically significantly correlated, and assessment of a concern was not associated with PI scores. Most subjects reported that the MMI was acceptable, and there were no differences in acceptability scores.
The PI has often been criticised for being biased (Citation3), but in this study no statistically significant differences were noted between applicants who were male or female, from different age or racial groups, or pre-admission academic ability. Similarly, consistent with previous reports, the results of the present study indicated that the MMI was not biased (Citation8, Citation13) (Citation16–Citation18). This is not a universal observation as female applicants have been reported to perform better in an MMI (Citation19) administered to a sample of dental students. However, it should be noted that this does not necessarily indicate bias in the MMI. The finding that PI and MMI scores were not associated with pre-admission academic ability is consistent with reports that the admission interview assesses non-cognitive attributes (Citation8, Citation20). Less frequently some studies have reported association between academic ability and MMI performance in emergency medicine residents (Citation21) and graduate-entry dental students (Citation22).
After academic ability, the interview is the most commonly used method to assess the suitability of candidates for medical training (Citation1). The PI is the usual practice, but the MMI is increasingly adopted because of its better reliability through its highly structured format and multiple assessments by different interviewers. However, the PI may be designed to be highly structured, such that differences in performances between the PI and MMI may be insignificant. The results of the present study indicated that the subjects who performed well in the PI were statistically significantly more likely to perform well in the MMI. This suggests that the MMI assessment of applicants was no different to the PI assessment and therefore does not add any more value to the PI. However, it should be noted that of all the applicants who performed well in the PI, for example, at a score of 15 and above, their MMI scores were dispersed considerably. Thus, although there was a statistically significant association between the PI and MMI, a significant minority of poor MMI performers scored high in the PI. If current evidence for the validity of the MMI is to be accepted (Citation10, Citation12) (Citation23), then a significant minority of unsuitable candidates would have been selected for medical training.
Perhaps the more significant contribution of the MMI is the generation of multiple impressions of the candidate's interpersonal skill, thoughtfulness, and general demeanour (Citation12) to identify those who are unsuitable for medical training. And this may be generated in a relatively short period of time (Citation24). Candidates who were raised as a concern independently at two stations or more performed similarly in the PI to those who were raised as a concern at one station or none at all. Although assessment of concern was subjectively determined at each station, the multiple independent assessments of applicants by different interviewers across different stations increase the reliability and objectivity of the assessment. This suggests that the MMI may have the added advantage of more objectively identifying applicants who may not be suitable to train as doctors because of particular concerns. An interesting finding was that all 11 applicants who were identified as a concern at two or more stations were of Chinese racial background. Comments made by the interviewers on their scoresheets suggested that they were either extremely nervous, overly arrogant, or inarticulate. It was noted that just over half of these applicants were raised as a concern at the Teamwork station, in which applicants were interviewed on their perceptions and experiences of teamwork. Although the data do not allow definitive conclusions, it may be postulated that the poor performance of the Chinese applicants at the Teamwork station in this context reflected their socially more competitive instincts compared to the other racial groups (Citation25). This is explained by their perception of marginalisation as a minority group and their need to be competitive as individuals. Anecdotally, it has been observed that some Chinese students are less willing to share their learning in the classrooms. However, longitudinal follow-up of subjects assessed as a concern is necessary to determine if this assessment is predictive of future academic and clinical performance. The applicants admitted to the programme will be followed up, and their performance at the MMI will be assessed for association with future academic and clinical performance.
Consistent with previous reports, subjects in the present study generally rated the acceptability of the MMI positively. Performance at the MMI did not appear to favour subjects from particular sex, age, and racial groups, or academic ability. More importantly, responses to the acceptability questionnaire indicated that the MMI was acceptable irrespective of sex or race. Most published research on the MMI has emanated from English-speaking countries where the population might be considered to be more homogeneous on grounds of race, religion, and language than some parts of the world. Results of the present study indicated that the MMI is acceptable in a multicultural context.
The findings of this study should be considered in the context of its limitations. Firstly, identification of a concern can vary widely, especially in the context of multicultural and multinational interviewers at this university. However, as each subject was assessed independently at more than one station, better reliability may be expected. Identification as a concern does not necessarily mean that the subject will perform poorly during medical training or clinical practice. Longitudinal evaluation of this group of students is necessary to validate the usefulness of identifying candidates as being a concern.
Ethical approval
This study was approved by the International Medical University Joint-Committee on Research and Ethics.
Contributors
AP acted as a primary investigator and contributed to the conception and design of the study and the acquisition, analysis, and interpretation of data, and drafted the paper. VKML contributed to designing and vetting of the MMI stations and acquisition, analysis, and interpretation of data, and reviewed the manuscript. YSC contributed to the conception and design of the study and data acquisition, and reviewed the manuscript. CFS, RDA, and AAF contributed to designing and vetting of the MMI stations and acquisition of data, and reviewed the manuscript.
Conflicts of interest
The authors have not received any funding or benefits from industry or elsewhere to conduct this study.
Funding
This study was funded by an internal grant from IMU.
Acknowledgements
The authors thank Ms Charmaine Khoo, formally Head of Admissions, for assistance with planning and organising the MMI pilot; and Ms Kogilla Rajamanikam and Ms Elaine Teoh in coordinating the MMI pilot.
References
- Kreiter CD, Axelson RD. A perspective on medical school admission research and practice over the last 25 years. Teach Learn Med. 2013; 25: S50–6. [PubMed Abstract].
- Wilkinson D, Zhang J, Byrne GJ, Luke H, Ozolins IZ, Parker MH, etal. Medical school selection criteria and the prediction of academic performance. Med J Aust. 2008; 188: 349–54. [PubMed Abstract].
- Edwards JC, Johnson EK, Molidor JB. The interview in the admission process. Acad Med. 1990; 65: 167–77. [PubMed Abstract].
- Dahlin M, Söderberg S, Holm U, Nilsson I, Farnebo L-O. Comparison of communication skills between medical students admitted after interviews or on academic merits. BMC Med Educ. 2012; 12: 46. [PubMed Abstract] [PubMed CentralFull Text].
- Goho J, Blackman A. The effectiveness of academic admission interviews: an exploratory meta-analysis. Med Teach. 2006; 28: 335–40. [PubMed Abstract].
- Dixon M, Wang J, Dineen C, Tomlinsen E. The panel interview: a review of empirical research and guidelines for practice. Public Pers Manage. 2002; 31: 397–428.
- Patrick LE, Altmaier EM, Kuperman S, Ugolini K. A structured interview for medical school admission, Phase 1: initial procedures and results. Acad Med. 2001; 76: 66–71. [PubMed Abstract].
- Eva KW, Rosenfeld J, Reiter HI, Norman GR. An admissions OSCE: the multiple mini-interview. Med Educ. 2004; 38: 314–26. [PubMed Abstract].
- Pau A, Jeevaratnam K, Chen YS, Fall AA, Khoo C, Nadarajah VD. The multiple mini-interview (MMI) for student selection in health professions training – a systematic review. Med Tech. 2013; 35: 1027–41.
- Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, Norman GR. Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ. 2009; 43: 767–75. [PubMed Abstract].
- Husbands A, Dowell J. Predictive validity of the Dundee multiple mini-interview. Med Educ. 2013; 47: 717–25. [PubMed Abstract].
- Eva KW, Reiter HI, Rosenfeld J, Trinh K, Wood TJ, Norman GR. Association between a medical school admission process using the multiple mini-interview and national licensing examination scores. JAMA. 2012; 308: 2233–40. [PubMed Abstract].
- Hofmeister M, Lockyer J, Crutcher R. The acceptability of the multiple mini interview for resident selection. Fam Med. 2008; 40: 734–40. [PubMed Abstract].
- Campagna-Vaillancourt M, Manoukian J, Razack S, Nguyen LH. Acceptability and reliability of multiple mini interviews for admission to otolaryngology residency. Laryngoscope. 2014; 124: 91–6. [PubMed Abstract].
- Dore KL, Kreuger S, Ladhani M, Rolfson D, Kurtz D, Kulasegaram K, etal. The reliability and acceptability of the multiple mini-interview as a selection instrument for postgraduate admissions. Acad Med. 2010; 85: S60–3. [PubMed Abstract].
- Brownell K, Lockyer J, Collin T, Lemay JF. Introduction of the multiple mini interview into the admissions process at the University of Calgary: acceptability and feasibility. Med Teach. 2007; 29: 394–6. [PubMed Abstract].
- Uijtdehaage S, Doyle L, Parker N. Enhancing the reliability of the multiple mini-interview for selecting prospective health care leaders. Acad Med. 2011; 86: 1032–9. [PubMed Abstract].
- Taylor CA, Green KE, Spruce A. Evaluation of the effect of socio-economic status on performance in a multiple mini interview for admission to medical school. Med Teach. 2014; 37: 59–63. [PubMed Abstract].
- Barbour ME, Sandy JR. Multiple mini interviews for selection of dental students: influence of gender and starting station. J Dent Educ. 2014; 78: 589–96. [PubMed Abstract].
- Lemay JF, Lockyer JM, Collin VT, Brownell AK. Assessment of non-cognitive traits through the admissions multiple mini-interview. Med Educ. 2007; 41: 573–9. [PubMed Abstract].
- Hopson LR, Burkhardt JC, Stansfield RB, Vohra T, Turner-Lawrence D, Losman ED. The multiple mini-interview for emergency medicine resident selection. J Emerg Med. 2014; 46: 537–43. [PubMed Abstract].
- Foley JI, Hijazi K. The admissions process in a graduate-entry dental school: can we predict academic performance?. Br Dent J. 2013; 214: E4. [PubMed Abstract].
- Reiter HI, Eva KW, Rosenfeld J, Norman GR. Multiple mini-interviews predict clerkship and licensing examination performance. Med Educ. 2007; 41: 378–84. [PubMed Abstract].
- Dodson M, Crotty B, Prideaux D, Carne R, Ward A, de Leeuw E. The multiple mini-interview: how long is long enough?. Med Educ. 2009; 43: 168–74. [PubMed Abstract].
- Chin YM. Ethnic socialization: a case of Malaysian Malay and Chinese public universities students experiences. Int J Soc Sci Humanit. 2013; 3: 582–5.