COVID-19 vaccination telephone outreach: an analysis of the medical student experience

ABSTRACT

Introduction

The COVID-19 pandemic diminished opportunities for medical students to gain clinical confidence and the ability to contribute to patient care. Our study sought out to understand the value of telephone outreach to schedule COVID-19 vaccines on medical student education.

Materials and Methods

Forty students engaged in telephone outreach targeting patients aged 65+ without active patient portals to schedule COVID-19 vaccines. Data consisted of a single administration retrospective pre/post survey inquiring about what students learned, expectations, other health-care processes that would benefit from outreach, and interest in a population health elective. Likert items were analyzed and open response analysis involved inductive coding and generation of thematic summaries by condensing codes into broader themes. Demographic data of patients called and subsequently received the vaccine were also collected.

Results

There were 33 survery respondents. There was a statistically significant increase in net comfortability for pre-clerkship students for documenting in Epic, providing telehealth care, counseling on common health-care myths, having challenging conversations, cold-calling patients, and developing an initial trusting relationship with patients. The majority called and who received the vaccine were non-Hispanic Black, within the high SVI category, and had Medicare and/or Medicaid. Qualitative data showed that students emphasized communication, the role of trusted messengers, the need to be open minded, and meeting patients where they are.

Discussion

Engaging students in telephone outreach early in the COVID-19 pandemic provided students the opportunity to develop their skills as physicians-in-training, contribute to combating the ongoing pandemic, and add value to the primary care team. This experience allowed students to practice patience, empathy, and vulnerability to understand why patients had not received the COVID-19 vaccine; this was an invaluable experience that helped students develop the skills to become empathetic and caring physicians, and supports the continued role of telehealth in future medical school curriculum.

KEYWORDS:

• Medical education
• COVID-19
• telehealth
• vaccination
• health equity
• value-added education

Introduction

Medical students need to acquire a vast breadth of knowledge, skills, and experience before becoming independent and competent medical professionals. During the early months of the COVID-19 pandemic, clinical teaching and training were substantially affected, and opportunities to gain clinical confidence and the ability to contribute to patient care were diminished, affecting the academic progress of some students [1]. Hueston and Petty [2] describe the challenges surrounding medical education in Wisconsin including at our institution, highlighting the importance of virtual education and other innovative student learning opportunities. Furthermore, defining the role of medical students during the pandemic was difficult because while as physicians-in-training, they potentially could form part of a health-care system’s response to public health emergencies, they are considered non-essential in clinical delivery and should be protected from unnecessary COVID-19 exposure [1]. Students felt this paradox more than ever at the beginning of the COVID-19 pandemic, especially with the Association of American Medical Colleges’ (AAMC) initial recommendation to remove students from clinical responsibilities [3]. Medical schools and medical students actively sought out safe opportunities to fulfill their role as future physicians with many highlighting telehealth as an important avenue to enhance medical student education [4].

A pre-pandemic approach called, ‘share the care’ which shares responsibilities between the health-care team members and the learners, has shown potential to enhance student learning and add value to health systems [5]. Potential areas of student integration include facilitators of communication and coordination between different health-care components of a patient’s care continuum and directly working with patients to improve health [5,6]. Further, U.S. medical students want to engage in clinical activities that add value to patient care and the health-care system [7]. Including value-added activities may build learners’ confidence and combat the feeling of being a burden, especially in the ongoing global pandemic.

The implementation of value-added activities for medical students in clinical settings would be highly significant in their education and would also yield reciprocal benefits for the current health-care system [1]. Evidence suggests that opportunities created for medical students to actively engage in the COVID-19 pandemic allowed students to cultivate values that are central in medical education, including altruism, service during a crisis period, professional solidarity, and a disposition to serve society [1].

The COVID-19 pandemic exposed vulnerable areas in the health-care system including significant health-care disparities and lack of health equity [8 – 12]. Throughout the COVID-19 pandemic, medical students throughout the nation found innovative approaches to combat these gaps [13], and expanding opportunities such as this may be key to sustaining and improving medical education during a protracted pandemic.

This paper describes how medical students were incorporated into an outreach program designed to improve COVID-19 vaccine access and vaccination rates among vulnerable patients without patient portal access and sheds light on the educational and professional development of students that engaged in this work. For this intervention, value-added student roles included providing patient outreach and education and identifying patient-specific barriers for the vaccine all through telehealth. We hypothesize that this experience for students would improve their comfortability in many important clinical domains including use of the electronic health record, having difficult conversations, utilizing telehealth, and developing trusting relationships with patients. Furthermore, we hypothesize that there would be important gains described by the students including those that strengthen communication and interpersonal skills; key areas of education that were lacking during the COVID-19 pandemic.

Materials and methods

Setting and program description

Starting on 22 January 2021, the health system deployed a strategy for COVID-19 vaccination largely driven by patient portal notification. Research suggests people who are older, less educated, economically disadvantaged, and who identify as ethnic minorities are less likely to have access to digital health information [14–16]; therefore, a telephone outreach initiative to target those without active patient portals at our institution’s General Internal Medicine (GIM) clinic was implemented to alleviate disparities in COVID-19 vaccine access. Between 1 February and 27 April 2021, medical student volunteers, internal medicine residents, primary care physicians, and community health workers provided directed telephone outreach to our GIM clinic patients aged 65 and older without patient portal access.

All medical students at the institution were invited to volunteer via mass emails through the school. Fourth year students enrolled in the Ambulatory Internal Medicine rotation were required to participate as part of their course requirement. Students worked 4-hour shifts, calling patients from a shared patient list in the electronic health record. A detailed workflow was created to standardize outreach, and backup support from attending physicians was available via telephone. Training included discussion on disparities in patient portal access. A total of 40 students who volunteered or who participated through a clinical rotation engaged in outreach program and signed up for a total of 139 shifts. During the intervention, 2,018 patients received outreach calls. This outreach project was deemed a quality improvement project via the institution’s quality improvement exemption process and issued an Institutional Review Board waiver.

Data collection methods

Demographics of the patients who received telephone outreach including race/ethnicity, social vulnerability index (SVI), and payor group (Medicare vs. Medicaid) were documented. Demographic data of those patients who received the vaccine and telephone outreach were then retrospectively collected through our patient portal data. SVI is a measure adopted by the CDC that uses 15 variables to reliably predict a community’s risk from a natural or human-caused disaster and the potential resource need of that community [17]. The SVI is based on census tract; for ease of use and integration with EMR level data, the SVI was classified based on zip code to a dichotomous variable of high or not high SVI. Zip codes were classified as high SVI if the majority of census tracts status within a zip code had a high SVI.

A retrospective pre/post survey was created and sent via email to all medical student participants at the conclusion of the outreach (Appendix). The survey was open for 1 month (14 May–15 June 2021) with students receiving their initial invite and reminders within this time frame. The survey included demographic questions, seven pre/post outreach questions using a Likert scale, and several open-ended questions regarding what students learned during outreach, their pre/post outreach expectations, other projects/programs/disease processes that would benefit from telephone outreach, and lastly, student interest in a population/preventive health management elective and why or why not. Questions were intentionally designed to understand the impact of the experience on the study hypothesis given that no quantitative measures exist in the literature to understand this impact. Only questions focused on the medical student experience and education were included in our analysis.

Quantitative analysis methods

The seven Likert scale questions used the scale ‘Extremely Comfortable’ to ‘Extremely Uncomfortable’ regarding student perspective pre/post outreach concerning different experiences during the project. The data were subdivided into two categories: the combination of pre-clerkship medical students in their first and second years (M1 and M2, respectively) and clerkship students in their third and fourth years (M3 and M4, respectively). Analysis of the Likert questions was done separately on these two subgroups. Data were analyzed in R (version 3.5.2 (20 December 2018)). A chi-square test was used to test for differences in the Likert scale between the before and after questions, and p-value <0.05 was considered significant. Data were then summarized into a new variable called Net Comfortability which is the combination of ‘Extremely Comfortable’ and ‘Somewhat Comfortable’ for the data tables. Confidence intervals for the Net Comfortability were also found in R.

Qualitative analysis methods

Analysis was an iterative process that involved inductive coding of the data, an audit of the generated codebook and application of the codes, the condensing of codes into thematic summaries, and a whole team review of the summaries [18; 19]. Three research team members worked collaboratively to inductively code the entire data set. As new codes emerged, previously coded categories were reviewed to ensure complete application of the codebook [18,19]. A coding audit was conducted using 20% of the data, and agreement was calculated at 80.1%, meeting qualitative coding standards [18; 19]. Next, a thematic summary for each analytic grouping was created by condensing the individual codes into broader themes. Finally, the entire research team read the thematic summaries and provided feedback.

Results

Quantitative results

Thirty-three out of 40 (82.5%) participants responded to the survey. Forty-five percent of participants were M1/M2 students (n = 15) while 55% (n = 18) were M3/M4 students. M1/M2/M3 students were volunteers, while 13 of the M4 students participated as a part of their Ambulatory Internal Medicine course requirement. Further demographic details are shown in Table 1.

Table 1. Demographics of survey takers.

Year of medical school	Total (N = 33)% (N)
First year (M1)	36% (12)
Second year (M2)	9% (3)
Third year (M3)	9% (3)
Fourth year (M4)	45% (15)
Required to participate	39% (13)

Thirty-two participants answered the Likert question. Net comfortability for M1/M2 students increased significantly for ‘documenting in Epic’ (P < 0.001), ‘providing telehealth care’ (P < 0.001), ‘counseling patients on common health-care myths’ (P = 0.001), ‘having challenging conversations with patients’ (P = 0.002), ‘cold-calling patients’ (P = 0.004), and ‘developing an initial trusting relationship with patients’ (P = 0.039). While there was improvement for M1/M2 students in their comfortability with dealing with ‘patients choosing a different plan than the one recommended’ (42.9% to 64.3%), the increase was not significant. For M3/M4 students, net comfortability was the same before and after outreach for ‘documenting in Epic’ and increased for the rest of the Likert questions. However, these changes were not significant. Further details are reported in Tables 2 and 3.

Table 2. First- and second-year medical student net comfortability (extremely + somewhat comfortable) survey responses.

N	Before outreach14	p-Value	After outreach14
Documenting in Epic	0% (0.0−23.2%)	<0.001	78.6% (49.2−95.3%)
Providing Telehealth Care	7.1% (35.9−99.6%)	<0.001	78.6% (0.20−34.0%)
Counseling Patients on Common Health Care Myths	0% (0.0−23.2%)	0.001	78.6% (49.2−95.3%)
Having Challenging Conversations with Patients	0% (0.0−23.2%)	0.002	57.1% (28.9−82.3%)
Cold-Calling Patients	28.6% (8.4−58.1%)	0.004	92.9% (66.1−99.8%)
Patients Choosing Different Plan than Recommended	42.9% (17.7−71.1%)	0.179	64.3% (35.1−87.2%)
Developing an Initial Trusting Relationship with Patients	21.4% (4.7−50.8%)	0.039	57.1% (28.9−82.3%)

Table 3. Third-and fourth-year medical student net comfortability (extremely + somewhat comfortable) survey responses.

N	Before outreach18	P-Value	After outreach18
Documenting in Epic	94.4% (22.7−99.9%)	0.943	94.4% (22.7−99.9%)
Providing Telehealth Care	72.2% (46.5−90.3%)	0.247	94.4% (72.7−99.9%)
Counseling Patients on Common Health Care Myths	66.7% (41.0−86.7%)	0.290	77.8% (52.4−93.6%)
Having Challenging Conversations with Patients	77.8% (52.4−93.6%)	0.978	83.3% (58.6−96.4%)
Cold-Calling Patients	44.4% (21.5−69.2%)	0.249	77.8% (52.4−93.6%)
Patients Choosing Different Plan than Recommended	66.7% (41.0−86.7%)	0.979	72.2% (46.5−90.3%)
Developing an Initial Trusting Relationship with Patients	83.3% (58.6−96.4%)	0.713	88.9% (65.3−98.6%)

Between 1 February and 27 April 2021, 2,018 patients received telephone outreach calls. Out of the 868 patients who received the COVID-19 vaccine, 382 (44%) received telephone outreach. The majority of patients called and who received the vaccine were non-Hispanic Black (68.8% and 57.0%, respectively), within the high SVI category (68.8% and 59.6%), and had Medicare and/or Medicaid (96.7% and 95.7%).

Qualitative results

Major qualitative themes, associated codes, code definitions, and representative quotes can be found in Table 4. Students described what they learned and their expectations pre- and post-outreach experience. Many focused on what they learned from the volunteer experience, their perceptions of outreach patients, and the communication skills needed and gained from the experience. In terms of technical skills, several students expressed that the telephone outreach was useful in learning how to use the electronic medical record. Overall, students described a wide range of patient attitudes toward vaccination. Most students either expected patients to be eager to or resistant to receive the COVID-19 vaccine. One student shared that patients were often ‘reluctant to discuss their reasoning,’ and another noted a ‘more pervasive fear of the vaccine in the community’ than expected. In contrast, students acknowledged that many patients simply did not have access to the vaccine or had reasonable concerns as to why they were resistant to getting the vaccine. One student described an interaction with a patient who was unsure about the safety and efficacy of the vaccine while undergoing chemotherapy, and another student stated that many patients were ‘just scared and needed more information from trusted individuals.’

Table 4. Major qualitative themes, associated codes, and representative quotes.

Theme	Code	Code definition	Example quote
Perceptions of outreach patients	Eager for vaccination	Participant either expected or did not expect patient’s enthusiasm to get the vaccine	‘I somewhat assumed that all patients would be super excited to hear they were eligible to receive the vaccine.’
	Resistant to vaccination	Participant either expected or did not expect patients to be willing to get the vaccine	‘I was pleasantly surprised by the number of individuals that had a desire to get the shot.’
	Reasonable Concerns	Participant either expected or did not expect patients to have reasonable concerns about the vaccine	‘Those who were hesitant had more reasonable concerns than I had expected.’
	Misinformation	Participant did not expect a conversation about COVID-19 vaccine misinformation	‘The extent of false information patients were hearing.’
Meeting patients where they are	Trusted Messenger	Participant either expected or did not expect a health-care professional to have such a positive impact in persuading them to get vaccinated	‘It was reassuring to hear that the patient had a great relationship with their primary care physician and trusted them to help them make the right decisions about their health care.’
	Open-minded	Participant found the best approach to conversations was to not have preconceived notions or expectations	‘I learned that it is best to approach conversations with a clean slate and no expectations.’
	Autonomy	Having to do with respecting the patient’s wishes and decisions	‘The patients are the masters of their fate, and I learned to respect this.’
Skills	Epic	Any time a participant mentions learning how to use Epic or gaining skills in the Epic interface	‘I learned how to navigate Epic. I was very unfamiliar with it before starting.’
	Communication	Having to do with persuading and using interpersonal skills to connect with patients	‘I think the best unexpected discovery for me was that through genuinely listening to people’s concerns … you can get patients to agree to the best treatment plan for them.’
Future outreach	Vaccination	Having to do with vaccination benefiting from telephone outreach	‘Other vaccination efforts.’
	Health screening	Having to do with health maintenance screening benefiting from telephone outreach	‘Health maintenance especially uncomfortable visits (scheduling colonoscopies with patients, pap smears).’
Health equity	Outreach	Participant discusses reaching out to the community	‘Great opportunity to serve the community and improve population health.’
	Underserved	Having to do with populations that are typically underserved benefiting from telephone outreach	‘Any project/program/disease process with a significant population that has challenges and obstacles to establish a connection.’
	Access	Having to do with barriers to getting the vaccine	‘Some people wanted to but didn’t yet have the chance to get the vaccine.’
	Preventive Health	Participant is interested in learning more about and participating in preventive medicine work	‘Personally I feel that the greatest impact we can have on patients is to help them live healthy lives and prevent disease before they even need more invasive health care to treat chronic disease.’

Throughout their reflections, students highlighted the importance of communication and the role of trusted messengers when engaging in community outreach of this nature. Students emphasized the need for interpersonal skills to connect with patients. For instance, students shared they learned, in the setting of cold-calling, how to ‘develop a rapport’ and ‘better counsel patients,’ and the importance of ‘genuinely listening to people’s concerns.’ Further, students acknowledged the importance of trusted messengers in health-care; one student shared, ‘the patient had a great relationship with their primary care physician (PCP) and trusted them to help them make the right decisions about their health care.’ Another student shared that patients really valued having ‘good rapport with their [PCP].’ Regardless of whether patients were eager or resistant to receiving the vaccine, students described the need for physicians to learn to meet patients where they are when discussing health and wellness issues. A student wrote, ‘I learned that it is best to approach conversations with a clean slate and no expectations. The patients are the masters of their fate, and I learned to respect this.’ Nearly every student wrote about having preconceived notions about patients and shared that their expectations changed due to the outreach experience. In addition, many students noted they had a difficult time combating the vast amount of misinformation circulating regarding the COVID-19 vaccine.

Students were asked what else could benefit from a similar telephone outreach program. Two broad themes of preventive health and access to health care emerged from the data. Preventive health included ideas for vaccination campaigns or clinics for influenza, HPV, pneumococcal, zoster, and other pediatric vaccines. Students also recommended conducting health screening outreach calls such as age-appropriate cancer screening and chronic disease screenings including hypertension and diabetes. For instance, a student recommended, ‘HIV screenings for high-risk patient[s],’ and another student suggested, ‘scheduling colonoscopies with patients … [or] pap smears.’ Access to health care was described in two ways: the importance of establishing a PCP and how to support patients who are traditionally underserved to achieve health equity. Students touched on the importance of having an established PCP to provide vaccines, health screenings, and mental health resources. A student stated, ‘any project/program/disease process with a significant population that has challenges and obstacles to establish a connection’ would benefit from having these conversations which ‘do make a difference toward changing attitudes over time.’ Responses that reflected the need to serve the underserved specifically mentioned populations that would benefit most from the outreach telephone calls. These populations included patients with increased barriers to care such as ‘minority populations,’ ‘patients who use the emergency department frequently,’ and patients who frequented safety net clinics such as the Medical College of Wisconsin’s ‘Saturday Clinic for the Uninsured.’

Since the outreach program was a population and preventive health management intervention, students were asked if they would enroll in an elective course that focused on this topic. Out of 33 participants, 15 responded yes, and most described the topic as valuable. Students noted that they wanted to gain a better understanding of the role of physicians in preventive health and opportunities to improve health outcomes through preventive care; exposure to these concepts early in medical school was also desired. One student shared that an elective would help them ‘gain a better understanding of all the things that we as physicians can do to improve the overall health of our communities,’ and another said, ‘it’s an important topic that we don’t get very much information about in our first two years of medical school.’ Students also described health equity as a reason to take a population and preventive health elective. Student descriptions of health equity included learning about and discussing the role of preventive medicine and outreach and how it can affect underserved communities and potentially improve their access to care with the goals of improving health outcomes and quality of care while decreasing health disparities. For instance, a student stated, “we can bring down the cost of healthcare in our country a lot while increasing the quality of care and the quality of patients’ lifestyles.” Another shared, ‘I feel that the greatest impact we can have on patients is to help them live healthy lives and prevent disease before they even need more invasive health care to treat chronic disease.’ One student also shared a physician’s sense of duty and responsibility and that ‘all leaders of healthcare should dedicate some time to preventive health.’ The majority of students who responded ‘no’ or ‘I don’t know’ to taking an elective on population and preventive health noted they were unsure and/or required more details about the course to help them make a decision. One student who answered ‘no’ stated, ‘policy analysis is not relevant (for the next decade of my career, at least), and neither is making calls that would be done by ancillary staff,’ and another student who answered ‘no’ responded that a certificate program would encourage them to sign up for an elective.

Discussion

This project serves as an example of value-added medical education, where powerful experiential learning experiences can add value and capacity to our health-care delivery system [20]. When done well, application of this concept has the potential to improve outcomes for patients, health-care team members, and learners in our academic medical centers. Engaging medical students in population-based COVID-19 vaccine telephone outreach provided an invaluable opportunity to not only develop their skills as physicians-in-training but also to contribute to the effort to end the pandemic and add value to the primary care team.

Through this initiative, pre-clerkship students were able to start developing their patient communication skills, many for the first time, while clerkship students, who were removed from clinical duties at the onset of the pandemic, were able to continue building upon their own communication skills. Analysis of the Likert questions highlights the impact of the telephone outreach on medical students, especially the pre-clerkship students. Pre-clerkship students showed a significant increase in net comfortability in all Likert questions except in ‘patients choosing different plan than recommended’ (Table 2). A very striking data point is the fact that before outreach, no pre-clerkship students were ‘somewhat comfortable’ or ‘extremely comfortable’ in ‘documenting in Epic,’ ‘counseling patients on common health-care myths,’ or ‘having challenging conversations with patients’ with a statistically significant increase in these categories after outreach. The added aspect of cold-calling really challenged medical students to quickly be able to develop a rapport and build trust with patients. Findings suggest telephone outreach could have a great benefit on the acquisition of knowledge, skills, and experience needed to become independent and competent medical professionals for pre-clerkship students. This is especially true during a pandemic when experiences like this were extremely limited for students. An additional benefit of telephone outreach in medical education curriculum post-initial outbreak of COVID-19 is that students were provided the opportunity to gain experience in difficult, heavy, and complex discussions such as social determinants of health without the time constraints and pressures of a traditional in-person clinical setting. Due to the inability to conduct a physical exam, telehealth heightens the need for more advanced and complex skill development of obtaining a patient history. While there was a net increase in comfortability in almost all Likert questions for clerkship students, these changes were not significant, possibly due to the small sample size (Table 3). A lack of significant change in net comfortability in M3/M4 students likely stems from these students already having clinical and patient care experience. The impact of the outreach on clinical students can be better characterized by their qualitative reflections.

The COVID-19 vaccine is unfortunately a highly politicized topic, which can influence a person’s preconceived notions. This not only applies to patients but also medical students and physicians alike. This experience allowed many students to practice patience, empathy, and vulnerability to understand why patients had not received the COVID-19 vaccine. Students learned to be okay with engaging with patients who have different opinions and to respect their decisions while understanding that each individual has their own lived experiences. These described skills are invaluable in developing empathetic and caring physicians.

As physicians and future physicians our first role is to ‘do no harm.’ With that being said, medicine has a tumultuous past with minoritized populations. Centuries of structural racism, including racism in health care, resulted in well-founded and deep-rooted mistrust in the medical establishment by communities of color, especially Black communities [8,9,12]. Part of the duty of physicians is to work to regain the trust of communities of color. Student responses demonstrated that an experience directly working to improve health equity is well-received and educational. Further experiences combating disparities in access to other preventive health care screenings including other vaccines and cancer screenings may have significant benefits for students, the health-care system, and those individuals who have been historically excluded from access.

During this experience, students learned the importance of trusted messengers and how patient trust in their PCP can have a significant impact on a patient’s choice to get vaccinated. The pervasiveness and easily distributable nature of health misinformation compounds the mistrust that communities of color already have with the medical establishment. In Dr Vivek Murthy’s first surgeon general advisory, he stated that health misinformation is a serious public health threat [21]. Students learned first-hand the important role health-care professionals have in combating the pandemic of misinformation.

More than half of the participating students responded that they would be interested in a population/preventive health management elective, further supporting the role in telephone outreach in developing student interest in this important work. While many medical schools have started incorporating health disparities and health equity education into their curriculums, the most poignant learning experiences are often the ones where knowledge is applied. Having the opportunity to work toward closing the gaps in health equity and interacting directly with the patients affected by health disparities provides a unique and valuable experience for students. The positive medical education value of this outreach initiative and student belief of the utility of telephone outreach in health screening and maintenance support a continued role of telehealth in future medical school curriculum. During our outreach intiative, medical students added value to the GIM clinic and the primary care team through decreasing the burden of PCPs. They completed COVID-19 vaccine outreach for their patient panels both providing medical education and combating health misinformation, especially for at-risk patient populations such as non-Hispanic Black patients, socially vulnerable patients, and Medicare and/or Medicaid patients. They improved the health of those patients who received the vaccine, freed up PCP time to manage other patient health problems, identified those for which location of vaccine clinics were an issue and referred those patients to social workers who could sometimes offer transportation, and flagged those who wanted to talk to their PCP more about the COVID-19 vaccine. Furthermore, early in the COVID-19 pandemic, medical students at our institution were able to pilot several other telehealth experiences, including pre- and post-primary care clinic visit patient outreach. Further proof of concept of the effectiveness of telehealth has been demonstrated by many groups of medical students across the world implementing their own telehealth programs as detailed by Grafton-Clarke et al. [22] and Park et al. [23] in their scoping reviews of the impact of the COVID-19 pandemic on medical student education. Aside from the educational benefit to medical students and added value to PCPs as described by our study, these studies showed additional ways that medical students added value to PCPs. As described in Grafton-Clarke et al., employing medical students to participate in telehealth visits decreased the burden on PCPs. Some medicals students were employed to perform scripted social determinant of health screening, which allows the PCP to provide more patient-focused care. Others conducted domestic violence screening, medication reconciliation, emergency department visit follow-up calls, triaging general medical patient complaints using a script and checklist, delivering food parcels to families, and education for patients with COVID-19 [22]. Park et al. detailed value-added medical student roles including identification of antepartum patients with new food or housing insecurity, anticipatory guidance to vulnerable populations, reconnecting those lost to follow-up, ensuring timely post-partum care in the setting of early discharges during COVID pandemic, helping orient and set up patients to telehealth technology, and preparing patients for their PCP appointments and completing pre-visit intake and ‘rooming’ patients. For medical schools across the nation beginning to incorporate telehealth as a permanent part of their curriculums, Muntz et al. using our pilot program and student feedback expertly outline a curricular strategy and roadmap to implement a telehealth curriculum [24].

There are a handful of limitations of the study that must be acknowledged. The first is the data represent the student experience of a small sample at a single institution; thus, generalizing to other contexts should be done with caution. In addition, the single post-intervention administration of the survey yielded data that allowed only for analysis of retrospective pre- and post-change. The retrospective nature of a survey allows for recall bias in recollection of telephone patient encounters. Literature has also shown only a weak correlation between self-assessment and performance with individuals often overestimating their abilities [25], potentially meaning an overestimation in improvement using the Likert scale questions. Further, the small sample size impacts quantitative power; however, the mixed methods approach provides additional qualitative insight beyond a statistical analysis.

Future directions for curriculum change at our institution include the incorporation of telehealth as a permanent component of undergraduate and graduate medical education curricula. As part of our medical school curriculum in the 2021–2022 academic year, outreach to encourage colon cancer screening is being done for patients 45–49 years of age. Our Internal Medicine Residency Program has begun employing telephone outreach in an attempt to increase rates of breast cancer screening. These are just two additional examples of the ample opportunities for expansion of telehealth activities across the medical education continuum.

Our COVID-19 vaccination outreach experience demonstrates that students are valuable members of the care team while learning important clinical and communication skills. As we engage in more population health work while prioritizing health equity, it is critical that we do not forget students in the process, and we must engage them in this exciting and impactful work.

Acknowledgments

We would like to acknowledge the medical students who participated in the COVID-19 telephone outreach initiative and those who completed the survey. We would like to acknowledge the M4 Ambulatory Course Directors Theresa C. Maatman, MD and Cynthia Kay, MD, Education Program Coordinator Mary Hoeschen, BA, and the Clinical Program Coordinator Taylor Melster, BS. We would lastly like to acknowledge the Froedtert GIM Clinic Manager Tammy Brissette, the Froedtert GIM Clinic Director Ann M. Maguire, MD, MPH, and the Section Chief for Primary Care Theodore MacKinney, MD, MPH.

Disclosure statement

The authors declare that they do not have a conflict of interest.

Additional information

Funding

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Appendix