https://orcid.org/0000-0002-2313-1973
ABSTRACT
COVID-19 imposed significant limitations upon the 2021 U.S. National Resident Matching Program (NRMP), most important of which is the replacement of traditional in-person interviews with a virtual format. To determine the strengths, limitations, and overall utility of virtual interviews (VIs) for residency applicant selection, a 14-question electronic survey was administered to programme directors (PDs) of all American Council for Graduate Medical Education (ACGME)-accredited residency programmes, from December 2020 through March 2021. PDs were asked about their experience with VIs and the ability to assess residency applicants using the virtual format. A total of 1123 PDs (30% response rate) representing 30 different specialities responded to our survey. Compared to in-person interviews, VIs made it more challenging to assess applicants’ fit with the programme, emotional intelligence, commitment to speciality, and ability to function as a resident physician. Overall, only 15% of PDs believed that VIs were better than in-person interviews. Once travel restrictions are lifted and in-person interviews are possible, 67% of PDs plan on hosting dual-format residency interviews, while 26% and 5% of PDs will exclusively host in-person interviews and VIs, respectively. This result was significantly different between surgical and non-surgical programmes [35% of surgical PDs suggested they would offer in-person interviews exclusively, compared to 21% of non-surgical PDs, p < 0.0001]. Although proven to be cost and time-efficient, VIs were challenging in evaluating certain qualities of residency applicants. While this study was focused on U.S. residency matching, it provides important insights about the future of VIs in medical recruitment as a whole.
Introduction
The residency interview period has traditionally involved in-person interviews and pre-interview social events for applicants to gauge and choose their place of future training, and for programmes to choose their future trainees. In addition to objective assessment of the applicant’s curriculum vitae, a multitude of factors influence the selection of candidates from the programme standpoint, and likewise, a residency programme from the applicants’ perspective [Citation1,Citation2]. The time-honoured combination of pre-interview socials, interviews with members of the selection committee, and interactions between applicants and residents in both formal and informal settings presents an opportunity for applicants to become familiar with the programme and for residency programmes to assess applicants’ fit with their programme.
As a result of the COVID-19 pandemic, the pathway to residency in 2021 has been anything but a smooth ride for applicants and residency programmes alike. The significant travel restrictions and institutional limitations of remote rotations for foreign students has severely limited applicants’ interaction with residency programmes. In addition, the mandatory implementation of virtual interviews (VIs) created unique challenges for assessing applicant-programme fit. Despite being implemented for fellowship interviews in 2020, VIs remain in a relative stage of infancy for selecting prospective resident physicians with many involved parties unsure how they will be employed going forward [Citation3–6]. In this study, we seek to determine the strengths, limitations, and overall utility of VIs for applicant selection through the perspectives of programme directors (PDs) across all specialities.
Methods
A 14-question survey was administered to PDs of all American Council for Graduate Medical Education (ACGME)-accredited residency programmes. Residency programmes included in our study were identified using the ACGME website. Email addresses of PDs were collected from the ACGME and residency programme websites. PDs received an individualised link for the survey, and the responses were collected in an anonymous fashion from December 2020 until March 2021. Non-respondents received three separate reminders during this period.
The survey was designed based on an extensive review of relevant literature and group discussions and was piloted prior to electronic distribution [Citation1,Citation7]. Once the main objectives were identified, one author (MA) designed an initial draft of the survey and the rest of the authors (Programme director, surgical resident, research fellow, and medical student) individually tested it for pertinence, ease of understanding, and time needed to finish. The survey was then restructured accordingly.
PDs were asked about the number of posts offered during the 2021–2022 application cycle, the convenience of VIs from a financial and time standpoint, and the ability to assess residency candidates using the virtual format. PDs were also queried as to whether attendance at virtual social events prior to the interview was considered when inviting and ranking applicants. Finally, PDs were asked about their overall experience with VIs and whether they would like to use the virtual format in forthcoming interview cycles. For the purposes of this study, approval was obtained from the Baylor College of Medicine Institutional Research Board.
Categorical data are presented as percentages and analysed via Chi-squared and Fisher’s exact tests as appropriate. Continuous data are presented as medians and interquartile ranges (IQR) and compared via the Mann–Whitney test. JMP Pro 14 statistical package (JMP, Pro 14, SAS Institute Inc, Cary, NC, 1989–2019) was used for this analysis.
Results
A total of 1123 PDs (30% response rate) representing 30 different specialities responded to our survey (). When compared to interviews held in-person, VIs made it more challenging to assess applicants’ fit with the programme (73% of respondents), personality and communication skills (68%), and commitment to speciality along with their ability to function as a resident physician (57%). Because of the challenges posed by the VI format in the subjective assessment of applicants, 46% of PDs reported that the residency selection committee relied more heavily on objective criteria when evaluating applicants (.
Table 1. Respondents’ Speciality.
Table 2. Programme Directors Perspective on Virtual in Comparison to in-person Interviews.
The majority of PDs (95%) agreed that virtual interviews were less expensive for programmes with a reduction of a median of $3000 (Interquartile range, 1000–8000) in interview-related costs. Only 45% of responding PDs stated that VIs were less time-consuming than in-person interviews. 47% of PDs agreed that VIs allowed for more candidates to be interviewed. PDs also reported that only a median of 5% of VIs were hindered by technical difficulties (.
Figure 1. Perspectives of PDs on virtual versus in-person interviews.
2Overall, only 15% of PDs believed that VIs were better than in-person interviews. Once travel restrictions are lifted and in-person interviews are possible, 67% of residency programmes plan on hosting residency interviews in virtual and in-person settings during the upcoming residency interview cycle while 26% and 5% of PDs, respectively, would exclusively host in-person interviews and virtual interviews. This result was significantly different between surgical and non-surgical programmes [35% of surgical PDs suggested they would offer only in-person interviews compared to 21% of non-surgical PDs, p < 0.0001].
Discussion
Our study highlights the limitations of VIs in evaluating residency candidates across various specialities. Interpersonal skills, interactions with faculty, staff, and residents have been cited as the four most important factors for ranking applicants according to the 2020 NRMP (National Resident Matching Program) Program Director Survey [Citation1]. More than half of our PD respondents found it difficult to assess applicant fit and communication skills using the virtual format. This difficulty might stem from the recent implementation and unfamiliarity with the virtual interface or from differences inherent to VIs. Although VIs make it more difficult to identify cues easily apparent during in-person interaction, such as body language, demeanour, and personal attire, many programmes have tried to minimise the limitations of VIs by hosting social events and virtual happy hours which present an opportunity to get to know the applicants in a small informal setting [Citation8]. These measures also presented applicants with the opportunity to learn more about the programme. In our survey, from those who offered virtual social events prior to the interview, almost a third (29%) considered applicant attendance at such events when making decisions regarding interview invites and ranking. The limitations of assessing the applicants during the VIs likely resulted in programmes relying on more objective criteria for evaluating applicants as evidenced by our survey response.
VIs offer several advantages compared to in-person interviews for both applicants and programmes [Citation7]. Almost all PDs agreed that VIs were less expensive when compared to in-person interviews and led to a median cost reduction of $3000. This cost-reduction can be a result of decrease in time spent by faculty interviewers with resulting increase in clinical productivity, and also from the lack of a need to physically host interviewing applicants [Citation7]. This aspect of VIs proves a massive advantage for residency applicants who spend thousands of dollars on flights and lodging each year [Citation7]; however, assessment of interview finances from the applicants standpoint was not conducted in our study. Although VIs might lead to significant time-efficiency from the applicant standpoint, only half of the programmes found VIs to be less time-consuming. This finding may be the result of differences in the design and implementation of VIs across programmes. While some programmes conducted their VIs on weekends or after work hours, others dedicated full working day(s) to interview applicants, similar to the procedure used for in-person interviews. Technical aspects could be considered as a potential disadvantage of VIs [Citation9]; however, this only seemed to mildly affect interviews as only a median of 5% of interviews were interrupted by major technical difficulties.
COVID-19 has transformed our lives and its impact may continue past the immediate pandemic. Although the majority of PDs disagreed that VIs were overall better than in-person interviews, most programmes plan on offering both formats in upcoming interview cycles. Programmes that decide to employ both formats may use VIs to screen applicants before offering in-person interviews to highly considered applicants. Conversely, programmes may choose to interview two sets of applicants separately by hosting in-person and virtual interviews for highly considered applicants and subsequent tiers of applicants, respectively [Citation8]. Surgical specialities are less likely to offer only VIs or combined formats compared to non-surgical specialities. This difference might be a result of the different expectations of incoming residents and training cultures.
Offering more interviews and conducting the virtual format may contribute to a diverse applicant pool by allowing programmes to interview more candidates who normally get screened out because of applicants having low board scores or attending a lower-ranked medical school. The enhanced diversity could also help applicants from low-economic status and significantly benefit international medical graduates who could now be interviewed from their home countries. Virtual interviews have presented a novel alternative to in-person interviews and are one of the forms of adapting to the new norm. Virtual interviews are being adopted for the 2022 application cycle as well, owing to the re-emergence of the COVID-19 in our community – their long term feasibility remains a question to be answered in upcoming cycles.
Our study features several limitations. Due its cross-sectional nature, we were unable to assess the temporal changes in PDs’ perspectives of VIs following the 2021 NRMP match results. In addition, our survey’s limited response rate may not completely represent all residency programmes, yet the high congruity among PD’s responses supports the validity of this study.
List Of Abbreviations
NRMP: National Resident Matching Program
VIs:Virtual Interviews
PDs:Programme Directors
ACGME: American Council for Graduate Medical Education
External Funding
The authors report no external funding source for this study.
Prior Abstract Or Poster Presentation
This study/data/abstract have not been presented.
Prior Publications, Online Or In Print
This paper has not been published online or in print, and is not under consideration elsewhere.
Supplemental Material
Download MS Word (16.5 KB)Disclosure Statement
No potential conflict of interest was reported by the author(s).
Supplemental data
Supplemental data for this article can be accessed online at https://doi.org/10.1080/21614083.2022.2087397
References
- Results of the 2020 NRMP program director survey. NRMP. Aug 2020. Avialble at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2020/08/2020-PD-Survey.pdf Accessed on: Mar 2021.
- Results of the NRMP applicant survey by preferred specialty and applicant type. NRMP. Jul 2019. Available at: https://mk0nrmp3oyqui6wqfm.kinstacdn.com/wp-content/uploads/2019/06/Applicant-Survey-Report-2019.pdf Accessed on: Mar 2021.
- Elmorsi R, Asaad M, Ferry AM, et al. How real is a virtual interview? Perspectives of orthopaedic surgery residency directors. Eur Rev Med Pharmacol Sci. 2021 12;25(24):7829–5 doi:10.26355/eurrev_202112_27629.
- Asaad M, Elmorsi R, Ferry AM, et al. The experience of virtual interviews in resident selection: a survey of program directors in surgery. J Surg Res. 2022 02;270:208–213 doi:10.1016/j.jss.2021.09.011.
- Ferry AM, Asaad M, Elmorsi R, et al. Impact of the virtual interview format on urology residency interviews: a survey of program directors. Urol Pract . . 2021;10–1097. doi:10.1097/UPJ.0000000000000292
- Rajesh A, Asaad M, Elmorsi R, et al. The virtual interview experience for MATCH 2021: a pilot survey of general surgery residency program directors. Am Surg. 2021 Aug 13;31348211038555. doi:10.1177/00031348211038555.
- Pourmand A, Lee H, Fair M, et al. Feasibility and usability of tele-interview for medical residency interview. West J Emerg Med. 2018;19(1):80–86 doi:10.5811/westjem.2017.11.35167.
- Wright AS. Virtual interviews for fellowship and residency applications are effective replacements for in-person interviews and should continue post-COVID. J Am Coll Surg. 2020;231(6):678–680 doi:10.1016/j.jamcollsurg.2020.09.005.
- Jones RE, Abdelfattah KR. Virtual interviews in the era of COVID-19: a primer for applicants. J Surg Educ. 2020 Jul-Aug;77(4):733–734 doi:10.1016/j.jsurg.2020.03.020.