A single center evaluation of applicant experiences in virtual interviews across eight internal medicine subspecialty fellowship programs

Figures & data

Table 1. Demographic information about survey respondents

	Self-reported demographic data (n = 75) n (% survey respondents)
Gender
Female	33 (44.0%)
Male	35 (46.7%)
Non-binary	1 (1.3%)
Prefer not to answer	1 (1.3%)
No response^a	5 (6.7%)
Current year in training
Second year resident (PGY2)	1 (1.3%)
Third year resident (PGY3)	48 (64.0%)
One year post residency	12 (16.0%)
Two years post residency	0 (0%)
Three or more years post residency	7 (9.3%)
Prefer not to answer	2 (2.7%)
No response^a	5 (6.7%)
Race/Ethnicity (select all that apply)
Hispanic, Latino, or Spanish origin	7 (9.3%)
American Indian or Alaskan Native	0 (0%)
Asian	19 (25.3%)
Black or African American	6 (24.0%)
Native Hawaiian or Pacific Islander	2 (2.7%)
White	34 (45.3%)
Prefer not to answer	4 (5.3%)
No response^a	5 (6.7%)
Underrepresented in Medicine (UIM)^b
UIM	15 (20.0%)
Not UIM	51 (68.0%)
Prefer not to answer	4 (5.3%)
No response^a	5 (6.7%)
Fellowship Program
Clinical informatics	7 (9.3%)
Endocrinology	7 (9.3%)
Gastroenterology	7 (9.3%)
Geriatrics^c	4 (5.3%)
Hematology/oncology	22 (29.3%)
Hospice and palliative medicine	7 (9.3%)
Infectious Diseases	14 (18.7%)
Rheumatology	2 (2.7%)
No response^a	5 (6.7%)

^aSurvey respondents who did not respond to the demographic data were labeled as ‘no response’.

^bSee methods section for definition and inclusion criteria for UIM

^cGeriatrics/palliative combined fellowship included in the Geriatrics fellowship respondents

Table 2. Overall strengths and weaknesses of the virtual interview day experience: Results from content analysis of responses to open-ended items

Domain	Strengths^a	Weaknesses/Areas for improvement^a
Structure/organization of the virtual interview day	‘I liked the dedicated overview of the program at the beginning (better than if it was a recorded video).’ ‘It was well structured, there were enough breaks, the tip sheet beforehand was very helpful.’ ‘Appropriate length of individual interviews and overall day. The supplemental videos provided introducing applicants to the hospital and various resources were very helpful.’ ‘I very much appreciated the 15 minute breaks (as opposed to 5 min breaks or no breaks) between interview sessions and found these to be crucial.’ ‘The technical support was strong and I knew I could reach out to the program coordinator.’ ‘The virtual format allowed me to save time and money.’	‘The approach of having one general “waiting room” for all of the fellows and then sending us in and out of breakout rooms was not a good experience … There were often delays in sending us to our breakout rooms, which added to the fatigue of the day.’ ‘I would suggest making it clear that lunch time is for eating. It wasn’t clear, so we ended up having 10 minutes at the end of the “fellows lunch” period to eat something.’ ‘I think that it would have been really beneficial to send a packet or PDF with program details (e.g., program structure, healthcare benefits, etc) in advance.’ ‘Consider shortening the day, as I feel zoom fatigue is really from the total length of day rather than back to back interviews.’
Individual interviews	‘I felt like I really connected with my interviewers.’ ‘Was able to switch between interviews seamlessly and had good 5 minute and 1 minute warnings before the interviewed finished.’ ‘I liked the standard questions with the program director, it made it feel like he was trying to get to know me as a person.’	‘20 min for interviews felt short, especially with the hard cutoff time. Maybe 25?’ ‘I do not like that the interview rooms timed out automatically. This led to awkward ends to the conversations.’
Interactions with current fellows	‘It was helpful to have multiple opportunities to speak with fellows.’ ‘The strongest aspect of the interview day was the opportunity to chat one-on-one with a current fellow (i.e., a chief fellow sent an introductory email connecting us with another fellow in advance so we could talk by phone). Creating an opportunity for one-on-one private conversation with a current fellow is one of the most impactful things I think a program can do for virtual recruiting.’	‘I would have liked more one-on-one or small group time with the fellows. It is hard to ask questions in large groups.’ ‘I would have liked to talk to fellows who have children (did not get this opportunity over zoom).’
Evaluating educational components	‘Good description of the breadth of research opportunities.’	‘It would have been nice to attend a didactic session, but I don’t think this is compulsory.’
Evaluating program culture	‘They let us sit in on the working groups involving the fellows and faculty, which did give a sense of the culture.’	‘I would have liked to have the opportunity to get a feel for the intangibles of working at this program; specifically, an impression on how much people enjoyed coming to work every day and relationships between colleagues.’ ‘It was harder to get a general sense of what sort of interpersonal interactions exist in the department, sense of camaraderie among the fellows, and what sort of physical presence the fellowship has in the hospital.’
Evaluating the hospital/surrounding city	‘Enjoyed the attempt to do a tour over video.’	‘Hospital set up and facilities where not really shown on interview day. It doesn’t need to take a lot of time, but a little bit of a visual would be useful.’ ‘Could consider a zoom based walkthrough of the various facilities to simulate in-person experience (cafeteria, work rooms, hospital entry shots, etc).’ ‘Hard to assess distance between hospitals and experience of living in the city.’

^aShown are representative quotations from applicants responding to the open-ended items: ‘What were the overall strengths and weaknesses/areas from improvement of the UCSF virtual interview day experience?’, organized by themes.

Table 3. Recommended considerations for structuring an effective virtual interview day

Domain	Recommended considerations
Structure/organization of the virtual interview day	Recommend that the interview day is long enough to convey the information needed but ideally <7 hours to prevent fatigue, with adequate breaks (3–5) included Provide contact information for the program director or an administrator in case technical issues arise Create meal breaks just for eating, being sensitive to different time zones Consider emailing or mailing overview materials to applicants in advance
Individual interviews	Recommend <30 minute individual interviews
Interactions with current fellows	Recommend sufficient interactions between applicants and fellows, ideally in small groups or one-on-one Consider offering more than one opportunity for applicants to meet fellows (e.g., a virtual social lunch and a panel Q + A or teaching conference) Consider pairing applicants with current fellows via email in advance to provide the opportunity for one-on-one connection outside the virtual interview day, if desired
Evaluating educational components	Create an overview presentation to highlight the educational components of the program, ideally given in real-time (rather than pre-recorded) at the beginning of the virtual interview day
Evaluating program culture	Consider allowing applicants to attend a virtual teaching conference/didactic, particularly those that allow applicants to observe fellow/faculty interaction Directly address sensitive subjects, such as the experience having a family/balancing family obligations and the culture of DEI at the program (e.g., with a planned Q + A session)
Evaluating the hospital/surrounding city	Include a virtual hospital tour, either pre-recorded or in real time (e.g., hospital entrance, fellow work-rooms, training facilities, etc.) Highlight elements of the surrounding city, including where most fellows live and cultural/outdoor opportunities in the surrounding area

Figure 1. Features of the virtual interview day

Survey respondents reported about features of the virtual interview day including: (A) the length of the interview day (hours), (B) the number of breaks, and (C) the number of individual interviews. (D) The distributions of survey responses to Likert items about characteristics of the virtual interview day are shown here. Responses to Likert items to the right of the vertical baseline (0% axis) show the percentage of respondents who answered ‘strongly agree’ (dark green) or ‘somewhat agree’ (light green) to the statement shown. Responses to the left of the vertical baseline show the percentage of respondents who answered ‘neither agree nor disagree’ (yellow), ‘somewhat disagree’ (orange), or ‘strongly disagree’ (red) to the statement shown. Survey respondents reported about the (E) incidence and types of technical difficulties, (F) the number of fellows that they interacted with during the virtual interview day, and (G) the format(s) in which they interacted with fellows.

Figure 2. Ability to assess the fellowship components and culture, and overall assessment

(A) The distributions of applicant responses to Likert items about their ability to adequately evaluate components of the fellowship program through their virtual interview experience are shown here. Responses to Likert items to the right of the vertical baseline (0% axis) show the percentage of respondents who answered ‘strongly agree’ (dark green) or ‘somewhat agree’ (light green) to the statement shown. Responses to the left of the vertical baseline show the percentage of respondents who answered ‘neither agree nor disagree’ (yellow), ‘somewhat disagree’ (orange), or ‘strongly disagree’ (red) to the statement shown. (B) Applicant responses about their ability to adequately assess diversity, equity, and inclusion (DEI) by under-represented in medicine (UIM) status. (C) Applicant responses about whether they would choose to interview virtually again if both virtual and in-person interviews were possible. The same Likert scale and visual representation of the results described in (A) were also used for (B) and (C). (D) Applicant responses about how they anticipate the virtual interview day will affect their anticipated rank position of the fellowship program.