Multiple mini-interview as a predictor of performance in the objective structured clinical examination among Physician Associates in the United Kingdom: a cohort study

Abstract

Introduction

Patient satisfaction and health care outcomes are directly linked to useful communication skills. Therefore, excellent interpersonal skills are imperative for health care professionals. Multiple mini-interview (MMI) is designed as a selection tool to assess the communication skills of applicants in medical schools during the admission process. However, objective structured clinical examination (OSCE) assesses students’ communication and clinical skills at the end of their academic terms. Recently, Anglia Ruskin University, Chelmsford, UK, adopted MMI in the selection process for the first cohort of MSc Physician Associate trainees for the academic year 2015–2016. This study aimed to determine the likelihood of MMI as a predictor of future performance of communication skills in the OSCE.

Materials and methods

The anonymous data of the average scores of communication skills attained in MMI and OSCE at the end of year 1 were collected for 30 students from the Physician Associate program team. Subsequently, Pearson’s correlation was computed to determine the relationship between the average scores of communication skills attained in MMI, and OSCE during trimester 2 and trimester 3 by the Physician Associate trainees.

Results

The study showed positive correlation between the scores of communication skills attained in MMI and OSCE during trimester 2 (r=0.956, n=30, p<0.001) and trimester 3 (r=0.966, n=30, p<0.001).

Conclusion

The study provides empirical evidence for the validity of MMI as a predictor of future performance of Physician Associate trainees’ communication skills during subsequent OSCEs.

Keywords:

• multiple mini-interview
• MMI
• objective structured clinical examination
• OSCE
• noncognitive skills
• communication skills
• Pearson’s correlation
• Physician Associate

This article is referred to by:

The role of interviews in predicting student OSCE performance: medical students’ perspective

Introduction

The process of admission in medical schools is quite demanding for both the admission committee and applicants. The wide gap between the number of applicants and the availability of places in the medical schools makes the correct selection procedures vital. The critical step of delivering a competent health care professional starts with selecting a suitable candidate using a rigorous selection process and further refining this selection with a panel interview. However, the use of multiple mini-interviews (MMIs) in their stead is growing consistently.¹ Evidence suggests that MMI can predict students’ future performance in the clinical clerkship and objective structured clinical examination (OSCE).^2–5 In 2015, Anglia Ruskin University, Chelmsford, UK, recruited the first cohort of students for MSc in the Physician Associate program using MMI.

Candidates’ future performance is of interest to authors, and many articles in the medical education literature have explored the possible future performance. According to one study, candidates’ academic merit is the best reflection of their predictive academic performance.⁶ There is a perception that cognitive skills are important, however, there is also a strongly held perception in the medical education research fraternity that the right attitude, as evaluated by an MMI assessment, significantly contributes to a productive workforce. The importance of a panel interview is highlighted by a study which concluded that students selected through the interview outperformed in communication skills during OSCE as compared to those selected by academic merit alone.⁷ The necessity of a structured admission interview to facilitate the selection of a candidate with the right attitude to work in the health care sector, is proven.⁸

In 1976, the term “noncognitive skills” was first developed to describe the various aspects of an individual’s personality other than cognitive traits which can contribute toward their behavior and success.⁹ Health care professionals need to have both cognitive and noncognitive skills. It is difficult to objectively test the noncognitive skills through traditional interviews.¹⁰

Communication skills should be an integral part of health care education programs. Experience alone cannot be attributed to a reliable factor in improving the communication skills of the students.^11,12 Effective communication is paramount to high-quality patient care and management.¹³ The majority of complaints received from dissatisfied patients at the National Health Services (NHS; UK) were attributed to poor communication.¹⁴ Therefore, it is suggested to assess both cognitive and noncognitive attributes of aspiring candidates wanting to join the health care profession.¹⁵

The MMI is conceptually derived from OSCE, but differs in being nonclinical.¹⁶ Evidence suggests that MMI may be a more impartial, reliable and valid assessment tool when compared to traditional interviews in testing noncognitive skills.¹⁷ The MMI has been positively received by the candidates, assessors and other stakeholders.^18,19

In paramedicine, MMI has been proven as a reliable and valid tool for the assessment of nonclinical attributes.²⁰ The ability of an individual to perceive, regulate and understand emotion, construct thoughts and then respond accordingly is defined as emotional intelligence.²¹ Emotional intelligence is a desirable trait in medical professionals, which may contribute to the development of professionalism and communication skills if integrated wisely in the curriculum.²² Assessing the emotional intelligence of the candidate through MMI has been proposed in the literature and is considered the most valid tool to measure noncognitive capabilities in an educational setting.²³

A comparative study of MMI versus panel interviews for medical school applicants showed that both were valid instruments. However, MMIs appeared more successful in assessing noncognitive skills and may objectively better reflect the overall persona of the applicants.²⁴ MMIs are appreciated for being innovative in finding the ways to test the compliance with the core values of the NHS in UK. It helps to avoid a rehearsed response enabling the candidate to better portray overall personality, and allows interviewers to discriminate between good and excellent.²⁵

This study aims to identify the correlation between the performance of the candidates’ communication skills during MMI and the subsequent OSCEs during their training and the effectiveness of MMI as a selection tool with the communication skills reflecting the ethos of the NHS among Physician Associates.

Materials and methods

A retrospective cohort (n=30) study was designed to examine the correlation between the average scores of communication skills of Physician Associate trainees recruited through an MMI-based selection process and subsequent two OSCEs in their first year of study (trimesters 2 and 3). The anonymous data of the average scores of communication skills attained in MMI and OSCE were collected from the Physician Associate program team. The study was approved by the ethics committee of the Faculty of Medical Sciences, Anglia Ruskin University, Chelmsford, UK.

MMI

In December 2012, “Compassion in Practice” was launched, which comprises a set of values: care, compassion, competence, communication, courage and commitment (6Cs), which are embedded in the daily care provided by the health care professionals within the NHS.²⁶

As part of the Physician Associate program, the Department of Medicine and Healthcare Sciences constructed an interview process that tested the potential candidates on their attitude toward the 6Cs, knowledge of general medical ethics and situational judgment.

It is essential to define the core communication skill for the item development, and it was based on the Calgary-Cambridge guide.^27–29 As part of the process, 10 simulated patient-based scenarios were constructed by clinicians and academics. The purpose was to evaluate the noncognitive skills, moral and ethical judgment of the candidate. The other skills that were assessed included problem solving, interpersonal communication, teamwork, flexibility and the reliability of a candidate.

Candidates were given 2 minutes to read and prepare a scenario or a question and then 8 minutes to present to an interviewer at each of the mini-interview stations. The assessors were clinical or academic members of the faculty and received MMI training prior to their assignment as an interviewer. An example of scenario is given in Box 1.

OSCE

OSCEs have become a favorite tool used in the assessment of students’ competence in both undergraduate and postgraduate medical education. This assessment method has been shown to be highly reliable when the following criteria are met: proper organization, blueprinting (the assessment should cover the learning objectives of the course), examiner training and consistent simulated patient training. It provided the opportunity to assess the communication skills and has been proven to measure the noncognitive ability of a student along with clinical competencies. During the first academic year, the students were formatively assessed for their communication skills by blueprinting and constructing OSCE scenario stations. These clinical and communication skill stations were based on the Faculty of Physician Associates’ (FPA) Curriculum Framework. Clinical consultants and academic lecturers designed the stations to meet the core standards set by the FPA and ensured a safe and robust scenario was created to test the competency of the students. These stations varied from focused clinical skills to different communication skill scenarios (eg, history taking, explanation of procedures and breaking the bad news).

Box 1 Example of an MMI station

Blueprinting criteria: communication, commitment and compassion

You are working as a Physician Associate trainee in a medical ward.

Your shift is about to end, and a patient who has been under care in your team has requested to speak to a member of the medical team.

The nurse in charge has come to you and said that you must speak with this patient, and they have a lot of questions to ask. The nurse walks off without waiting for your reply.

Put the following actions in order of most appropriateness to the least as what you would do and why?

1. Complain to someone senior about the nurse forcing this job on you.

2. Wait for the on-call team to take over and handle the situation.

3. Go home because you do not get paid for overtime.

4. Speak to the patient and explain any doubts clearly.

5. See the patient tomorrow as it is not urgent.

Abbreviation: MMI, multiple mini-interview.

Psychometric scale for both MMI and OSCE

To be consistent with the measurement of communication skills for both the MMI and OSCE, we used Explanation and Planning Scale (EPSCALE), a validated 4-point response scale designed to measure communication skills.³⁰

Statistical analyses

The statistical analyses were performed using IBM SPSS Statistics for Windows, Version 24.0 (IBM Corporation, Armonk, NY, USA). Pearson’s correlation coefficient was calculated to assess the relationship between the average scores of communication skills attained in MMI and OSCE during trimester 2 and trimester 3.

Results

There was a positive correlation between the average scores of communication skills attained in MMI and OSCE during trimester 2 (r=0.956, n=30, p<0.001) and trimester 3 (r=0.966, n=30, p<0.001; Tables 1 and 2). The results are summarized as scatter plots in Figures 1 and 2.

Figure 1 Scatter plot of the average scores for communication attained in MMI and OSCE (trimester 2).

Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Figure 2 Scatter plot of the average scores for communication skills attained in MMI and OSCE (trimester 3).

Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Table 1 Correlation between the average scores of communication skills attained in MMI and OSCE (trimester 2)

Correlations		MMI	OSCE trimester 2 communication skills
MMI	Pearson’s correlation	1	0.956**
	Significance (two-tailed)		0.000
	n	30	30
OSCE trimester	Pearson’s correlation	0.956**	1
2 communication skills	Significance (two-tailed)	0.000
	n	30	30

Note:

** Correlation is significant at the 0.01 level (two-tailed).

Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Table 2 Correlation between the average scores of communication skills attained in MMI and OSCE (trimester 3)

Correlations		MMI	OSCE trimester 3 communication skills
MMI	Pearson’s correlation	1	0.966**
	Significance (two-tailed)		0.000
	n	30	30
OSCE trimester 3 communication skills	Pearson’s correlation	0.966**	1
	Significance (two-tailed)	0.000
	n	30	30

Note:

** Correlation is significant at the 0.01 level (two-tailed).

Abbreviations: MMI, multiple mini-interview; OSCE, objective structured clinical examination.

Discussion

Health care outcome is positively correlated with active physician–patient communication. Trained in a medical model, Physician Associates are a relatively new addition to the existing health care force with communication being an essential part throughout the curriculum. Communication is a crucial bridge between patients and health care professionals which can significantly enhance the recovery of the patients.³¹ It is one of the aspects of noncognitive traits and can be learned through practice and experience. A number of medical councils all over the world including the Accreditation Council for Graduate Medical Education (ACGME) and General Medical Council (GMC), UK, have endorsed the introduction of training in communication skills in the medical curriculum.^32,33

MMI has been proven as a successful technique for selecting candidates in various healthcare domain. Studies support the adoption of MMI as an admission protocol in conjunction with other academic selection criteria. The MMI score can be used as a predictive measure for future performance in examinations as well as clinical clerkship.³⁴ The predictive validity of MMI was confirmed and attributed to the similarity in the communication aspects of both examination types and the ability of an individual to perform well under pressure in a real-life clinical setting.³ The numerical score provided by MMI allows wider dispersion during the evaluation of the strength of the applicants’ communication skills.

MMIs have been shown to be more effective than typical interviews as the number of raters involved in the assessment of each candidate allows for a more reliable objective assessment of these individuals. The way in which the MMIs are carried out can provide insight into different aspects of candidates’ eligibility for the position. Communication and listening skills are often assessed, as well as their time management when put under pressure to understand the contents of the station, formulate their thoughts regarding the matter and effectively communicate them back to the assessor or, in some cases, the actor at the station.

Specific factors need to be considered to efficiently use MMIs as a tool for value-based recruitment (VBR). The critical criteria for each station need to be defined efficiently; the scoring system should identify both positive and negative behaviors to allow for more comprehensive assessment of the candidates. Having a defined set of criteria that should be met at each station facilitates fair judgment and comparison of candidates’ performance in different aspects of the interview.

Although MMI has established its capability to assess communication skills during the admission process,³² its validity as an admission tool for the Physician Associate programs in UK is still in its infancy. In the literature, evidence for the reliability, validity and acceptability of the MMI is increasing for the postgraduate admissions. However, data on the predictive validity of the MMI in medical education are sparse.³⁵

To the best of our knowledge, this is the first study describing the use of an MMI in a Physician Associate admission process. Every stratum of health care professionals requires communication skills, and the assessment of these skills has been adopted as a crucial part of the admission process and curriculum. Medical schools are continuously working toward developing innovative solutions for teaching and assessing communication skills at all levels of medical education.^36,37 The MMI as a selection tool has long been practiced to enhance the assessment of communication skills during the admission interviews. Conversely, OSCE has been proven as a useful tool for the assessment of communication skills in medical education.^38,39 For the successful practice of medicine, expertise in both verbal and nonverbal communication is essential, and both MMI and OSCE can assess this effectively.

Limitations

This study has few limitations. First, trainee students had varied experience such as some were fresh biomedical science graduates and many with 5–7 years of work experience in health care settings. This imposes a perspective bias in their MMI performance. Second, this study was conducted at a single center, with relatively small sample size limiting the generalizability. Future studies should focus on repeating and encompassing the findings with multiple Physician Associate programs, as well as establishing the correlation between MMI and national examinations for the Physician Associates conducted by the FPA which in turn will contribute to setting a benchmark criterion for the assessment of OSCE across the Physician Associate schools, thereby contributing to the patients’ safety and care.

Conclusion

This is the first study demonstrating the positive relationship between the performance of communication skills during the recruitment process using MMI and OSCEs in the Physician Associate program in UK. The MMI scores are an excellent indicator to predict the performance of candidates’ communication skills during OSCEs.

Acknowledgments

We would like to express our earnest thanks to Dr Greg Goodman, Associate Professor, Monash University, Melbourne, VIC, Australia, for comments that significantly improved the manuscript.

Disclosure

The authors report no conflicts of interest in this work.