Multi-stakeholder validation of entrustable professional activities for a family medicine care of the elderly residency program: A focus group study

ABSTRACT

Entrustable Professional Activities (EPAs) have become widely used within Competency-Based Medical Education (CBME) for the training and evaluation of residents. Little is known about the effectiveness of incorporating multiple stakeholder groups in the validation of EPAs. Here, we seek to validate an EPA framework developed for the University of Manitoba Care of the Elderly Enhanced Skills program using online focus groups consisting of five stakeholder groups. Participants were recruited to take part in one of five online focus groups, one for each stakeholder group (physician faculty, residents, non-physician healthcare professionals, administrators/managers, and patients). Each group met one time for 90 minutes over ZOOM_®. The themes arising from stakeholder feedback suggest that successful EPAs must neither be too specific nor too expansive in scope, clearly delineate appropriate means of evaluation, and indicate specific clinical settings in which each EPA should be evaluated. Cross-cutting themes included requiring trainees to collaborate with other professionals when it would optimize patient care, and preparing trainees to advocate for their patients’ health (Advocacy). The present study demonstrates that multi-stakeholder analysis yields diverse feedback that can help make EPAs more clear, easier to use in evaluation, and more socially accountable.

KEYWORDS:

• Entrustable Professional Activities
• Competency-based medical education
• Focus group
• Stakeholder
• Validation

Background

Over the last two decades, Competency-Based Medical Education (CBME) has been widely adopted into medical training programs across the globe (Van Melle et al., 2021). CBME aims to “graduate health professionals who can practice at a defined level of proficiency, in accord with local conditions, to meet local needs, in a system of fixed outcomes” (Ten Cate et al., 2021). A trainee’s progress through a competency-based program largely depends on their demonstration of an established set of standards (i.e., competencies) (Ten Cate, Schwartz, & Chen, 2020). Competencies characterize a professional’s abilities in a broad sense, using words such as “knowledge, professional attitude [and] communication skill” (12). Due to their abstract nature, implementing competencies into the training and assessment of trainees has proven to be a significant challenge (Van Loon, Driessen, Teunissen, & Scheele, 2014). To remedy this problem, the concept of Entrustable Professional Activities (EPAs) was introduced into medical education (Lohenry et al., 2017).

“EPAs are tasks or responsibilities that faculty entrust to a trainee to execute once he or she has obtained adequate competence” (Ten Cate, 2014). They are “units of work” (Ten Cate, 2014) that find their place at the “does” level of Miller’s Pyramid of Assessment (knows, knows how, shows how, does) (Ten Cate et al., 2021). EPAs have inspired a shift away from high-stakes formative assessments and toward multiple low-stakes, EPA-based assessments (Ten Cate, Scheele, & Ten Cate, 2007). In recent years, it has become increasingly evident that physicians require more than knowledge and expertise to offer exceptional care to patients. The good doctor must also “practice with professionalism and societal responsiveness, based on shared decision-making and in optimal collaboration within teams and the health care system at large” (Van Loon, Driessen, Teunissen, & Scheele, 2014). One way of fostering social accountability, interprofessional collaboration, and system-based practice in our future physicians is by acquiring input from various stakeholders in the health care system on EPA frameworks used to guide the training of residents.

Insufficient efforts to consult relevant stakeholders has contributed to the failure of many policies and programs in the past (Bryson, Humphrey, & Affairs, 2016) – a teachable lesson for those developing and implementing EPA frameworks. Regrettably, stakeholder involvement in the context of competency-based program development has focused mainly on physician perspectives (Graham et al., 2009). According to the World Health Organization (WHO), socially accountable programs should be built to consider not only health professionals, but also academic institutions, health administrators, policy makers and communities (Boelen, 2000). Specific examples of stakeholders that have been consulted in CBME include residents, physician faculty, nurses, administrators, and patients (Smith et al., 2013). However, each study has tended to only consider a limited number of distinct stakeholder groups. In addition, only a few studies have directly involved patients in the evaluation of EPAs (Bramley & McKenna, 2021; El-Haddad, Damodaran, McNeil, & Hu, 2017).

In Canada, residents can become practicing family physicians by completing a two-year Core Family Medicine residency program. Family physicians see patients of all ages and provide most of the primary care in Canada. At any point after completing the core family medicine residency program, family physicians can opt to pursue an additional year of training in providing care to elderly patients. At the end of the program, they are granted a Certificate of Added Competence in Care of the Elderly (CoE). CoE physicians, then, are family physicians who are trained to provide more advanced care to elderly patients, particularly those with complex needs. They also “augment and support the care provided by other family physicians, other specialists, and other care providers typically around issues of frailty, complexity, comorbidity, medication assessment and management, and functional decline in the elderly” (Hall et al., 2021).

The University of Manitoba Department of Family Medicine has recently articulated a framework of EPAs for its Enhanced Skills CoE Program. The EPAs were developed in 2018 by an expert group consisting of CoE physicians, Geriatricians, and learners that was led by a senior medical educator. Following an extensive review of existing competency frameworks, the group identified and articulated 10 EPAs that were introduced in the 2019–2020 academic year.

This study aims to validate the CoE EPA framework via online focus groups with representation from many stakeholder groups: physician faculty, trainees, non-physician health care professionals, managers/administrators, and patients and their family members.

Methods

Participants

Participants from stakeholder groups of interest were recruited to participate in stakeholder-specific focus groups. The focus groups included physician faculty, residents, administrators/managers, non-physician health care professionals and patients/their family members (referred to as the “patient group”). Study invitations were distributed to potential participants by e-mail. Faculty members were contacted directly through readily available faculty listings. For the remaining stakeholder groups, e-mail invitations were distributed by residency program directors, administrators within the regional health authority, professional colleges, and Local Health Interest Groups (LHIGs). Both CoE physicians and Geriatricians were invited to take part in the physician faculty group. The resident group was unique in that it consisted of individuals both in- and out-of-province due to an insufficient number of current Manitoba trainees in these programs. The patient group was defined as patients who have received or are currently receiving care, as well as family members of individuals who fit this description. The number and characteristics of individuals representing each stakeholder group are detailed in Table 1.

Table 1. Demographic composition of focus groups.

Stakeholder Group	Group Composition
Physician Faculty	6 Geriatricians 1 CoE Physician
Current Trainees	2 Geriatrics Trainees (in-province) 2 Geriatrics Trainees (out-of-province) 1 CoE Trainee (in-province)
Non-physician health care professionals	4 Physiotherapists 1 Psychiatric Nurse 2 Registered Nurses
Administrators and Managers	2 Managers 2 Administrators
Patients	6 Patients 1 Family Member

Focus group meetings

Focus group participants received the list of proposed EPAs (viewable in Table 2) more than one week in advance. Each focus group met online over ZOOM_® for a single 90-minute meeting. Group sessions were facilitated by either the principal investigator or a medical student using a semi-structured interview script. Each focus group was asked three questions regarding the proposed EPA framework: “Do you feel they describe common tasks that a Care of the Elderly Physician performs?,” “Are there EPAs that you do not regard as a key activity that a graduating Care of the Elderly Family Physician should be able to perform independently by the end of the training?” and “Are there additional EPAs that you think should be included?.” All meetings were recorded through ZOOM_® and transcribed using NVivo_® (QSR international, Doncaster, Victoria, Australia). Transcripts were later reviewed for errors and anonymized to protect confidentiality.

Table 2. Entrustable professional activities developed by the University of Manitoba Department of Family Medicine for Care of the Elderly trainees.

EPA Number	EPA Title
1	Provide patient-centered care of the older adult that optimizes function and/or well-being
2	Prevent, diagnose and manage common (key) geriatric conditions
3	Provide comprehensive medication review to maximize benefit and minimize number of medications and adverse events
4	Assist patients and families in clarifying coals of care and making care decisions
5	Provide palliative care and end-of life of care for older adults
6	Coordinate care and health care transitions for older adults with multi-morbiditiy and multiple providers
7	Provide geriatric consultation and co-management
8	Collaborate and work effectively as a leader or member of an inter-professional healthcare team
9	Advance the quality and safety and health care for the elderly through the application of quality improvement and/or research
10	Teach the principles of care of the elderly and agining-related health care issues to healthcare professionals, learners, families , and other in the community

Analysis

The results from each focus meeting were discussed by the research team to identify themes and their corresponding codes that would be used in data analysis. Transcripts were then coded using NVivo_®. The coding structure was refined iteratively by the research team twice over the period of time at which focus group meetings were taking place. Results were analyzed comparatively to explore agreements, gaps and other differences between stakeholder groups’ perspectives. The credibility and transferability of results was founded on the participation of stakeholders from many stakeholder groups, and the selection of representative quotes to support each theme and subtheme. To ensure dependability of results over time, the leader of each meeting relied solely on the semi-structured interview script for all meetings, minimizing the chance of results from preceeding meetings influencing input gained from subsequent meetings. These measures were taken to establish the trustworthiness of results (Graneheim & Lundman, 2004).

Ethics

All participants were required to sign a consent form in advance of their focus group meeting. No incentives were given for participation and all data collected was anonymized. Ethics approval was granted from the Health Research Ethics Board at the University of Manitoba (HS24748 (H2021:114[CM5])).

Aim

In this study, we aim to validate this CoE EPA framework using input from five stakeholder groups to inform future iterations.

Results

A summary of all input toward each EPA is presented in Table 3.

Among the ten EPAs, stakeholders drew the most attention to EPAs 2, 4 and 9. EPA 2 was the only EPA to be commented on by all stakeholder groups. In addition to the participants’ evaluation of each EPA, 4 themes arose that serve as lessons for future initiatives in EPA development: Specific vs. Broad, (subthemes: Operationalization and Setting), Wording Changes, Interprofessional Functioning, and Advocacy (subthemes: Systems-based Practice, Social Contexts, Social Determinants and Transitions in Care).

Table 3. Stakeholder Feedback Regarding Ten Care of the Elderly Entrustable Professional Activities.

EPA number	EPA Title	Results
1.	Provide patient-centered care of the older adult that optimizes function and/or well-being.	The resident and faculty groups agreed that the scope of this EPA is too broad.
		The faculty group felt this EPA 1 could possibly be removed
		Members of the health professionals group felt that team-based care (in particular with Occupational Therapists and/or Physiotherapists) should be emphasized to improve function and well-being for patients.
2.	Prevent, diagnose and manage common (key) geriatric conditions.	The resident and faculty groups felt that elements of this EPA may be redundant, having already been taught during the core family medicine residency.
		“Geriatric conditions” could be replaced with “geriatric syndromes” for the sake of language congruency with EPAs developed for Geriatrics by the Royal College of Physicians and Surgeons of Canada.
		Difficulties operationalizing EPA 2 were expected. Separating “assessing cognition” and “managing dementia” into two separate EPAs was suggested to remedy this issue.
		This EPA should be evaluated in multiple settings, including in long-term care. Trainees ought to also recognize how social determinants of health are implicated in the prevention, diagnosis and management of key geriatric syndromes. Evaluating loneliness and social health should also be under this EPA.
3.	Provide comprehensive medication review to maximize benefit and minimize number of medications and adverse events.	Multiple members of the health care professionals group said that trainees should also be able to communicate to patients the reasons they are taking each medication and ensure that the patient is able to take their medication.
		Trainees should be able to counsel patients on the appropriate use of natural health products and their drug interactions.
		EPA 3 should require trainees to work with allied health professionals and/or family members to ensure medications are being taken consistently and properly.
		Trainees should learn to collaborate with pharmacists when doing medication reviews.
4.	Assist patients and families in clarifying goals of care and making care decisions.	This EPA was strongly endorsed by the health care professionals group.
		As part of this EPA, trainees should give realistic expectations regarding prognosis to patients early in their care
		This EPA should be modified to be inclusive to patients who have non-family members as the primary collaborators in that person’s care.
5.	Provide palliative and end-of-life care for older adults.	Two members of the manager/administrator group suggested that this EPA should also include coordinating palliative care in a culturally appropriate way (e.g., integrating traditional healing practices).
		The patient group agreed that providing medical assistance in dying (MAID) should fall under the responsibilities of Geriatricians, but that CoE physicians should mention it as an option for patients when appropriate.
		This EPA should be evaluated across multiple settings, including in long-term care settings.
6.	Coordinate healthcare and healthcare transitions for older adults with multi-morbidity and multiple providers.	Multiple members of the health care professional group expressed appreciation for this EPA.
		CoE physicians should be trained to follow up with patients after being discharged to ensure their medications are appropriate while living at home. The resident group raised concerns over how EPA 6 would be evaluated. For example, difficulties obtaining feedback on discharge summaries from preceptors was reported. Facilitating transitions is a significant part of the family medicine program and, therefore, may be redundant. This EPA should be elaborated to specify examples of transition such that the EPA is not limited to acute care.
7.	Provide geriatric consultation and co-management.	The faculty group suggested that EPA 7 be evaluated in both in- and outpatient settings, including long-term care and virtual care.
8.	Collaborate and work effectively as a leader or member of an inter-professional healthcare team.	The Health care professionals group expressed strong appreciation for this EPA.
		EPA 8 could be merged with EPA 1, so that physicians work with other professionals to optimize patient well-being.
9.	Advance the quality and safety of health care for the elderly through the application of quality improvement and/or research.	The Administrator/manager group expressed strong appreciation for this EPA.
		Residents felt this EPA may be redundant, as participating in quality improvement is a part of core family medicine residency.
		Concern was raised by the resident and faculty groups about this EPA not being feasible in a one-year program.
		Members from the faculty group highlighted the importance of participating in quality improvement. One member suggested trainees be required to do a “small project” in quality improvement.
		This EPA should be broadened to include other non-research-related academic activities.
		Administrator/manager group felt EPA 9 could include contributing to the development/revisions of health policy.
10.	Teach the principles of care of the elderly and aging-related health care issues to health professionals, patients, families, health care providers and others in the community.	There was widespread agreement in the patient group that EPA 10 is important in the “professional development of any kind of doctor.”

Theme: specific vs. broad

Comments coded to the Specific vs. Broad theme were made by the faculty, resident and administrator/manager groups. While some EPAs were considered too broad in their wording, no comments were made suggesting that any EPA was too specific in its focus.

In the resident group, the wording of the CoE EPAs was compared to those developed for a Geriatrics Specialty program, saying,

In general, I do actually really like the way the Royal College EPAs are laid out in that they’re quite a lot more specific than the [CoE] family medicine ones.

Similarly, a member of the faculty group said,

But I also find on the care of the elderly side it’s too broad … so I think … there could be some work around narrowing that down.

Subtheme: operationalization

The resident and faculty groups expressed concerns that some EPAs would suffer shortcomings at the level of implementation and trainee evaluation (i.e., operationalization), sometimes due to the broadness of some EPA descriptions.

According to a participant in the faculty group,

The [CoE] family medicine’s [EPAs] are too broad. So, in number two, you could easily just do five falls and not be evaluated on your dementia management because there’s nothing to say it has to be at least two dementias, two falls, two incontinence, two of this.

A member of the resident group doubted as to whether EPA 6, “coordinate healthcare and healthcare transitions for older adults with multi-morbidity and multiple providers,” could be evaluated realistically.

It’s just harder to potentially get feedback [on] a system-level basis. For example, I have discharged patients, and I’ve done a large number of discharge summaries, but you don’t really get direct feedback on your discharge summaries. A lot of the time, or you might get feedback, but you’ll get feedback from your attending who also discharged the patient and didn’t see them in follow up later.

Contrary to other participants, one resident suggested that the broadness of EPA descriptions could ultimately facilitate the EPAs’ operationalization.

Sometimes when you get to the specifics of an EPA, for example, for dementia assessment, you need like one vascular, one Alzheimer’s, one Lewy body … If a resident is generally good with dementia management, there’s a lot of correlation between being good at vascular versus Alzheimer’s versus Lewy body, et cetera, and the specific need to get each of those EPA’s on target can be quite tricky, especially [since] Manitoba has a smaller catchment area.

Subtheme: setting

The faculty group highlighted the importance of certain EPAs being evaluated in different settings.

For expanding on [EPA] 6, just defining examples of transition so that, again, it’s not just acute care.

I think if you just get a little bit more specific in the evaluations, that this should at least include one virtual encounter.

Regarding EPA 7, a member of the faculty group said,

Adding to that list of important settings and for advocacy, maybe long-term care? That’s something you should have more exposure than somebody who didn’t do Care of the Elderly.

Theme: wording changes

Thoughts related to the wording of EPAs were raised by all focus groups except for the physician faculty group. Many of these comments were for the purposes of social accountability or congruence with language used in other specialties.

Just in general, with the DSM-V, often we’re using major neurocognitive disorder instead of dementia, but I know ‘dementia’ is more widely known and recognized.

Regarding ‘assisting patients and families and clarifying the goals of care and making decisions’ … I would like to see in there is that it’s a little more inclusive. Many people have different definitions of family, so it doesn’t necessarily have to be a son or daughter or mother. So, I would like to see that wording changed to something like significant other.

Theme: interprofessional functioning

The resident and health care professionals group identified EPAs that trainees should demonstrate an ability to collaborate with other health care professionals.

One member of the health care professionals group said,

Reading [EPA] 3 … collaborating with pharmacy – I think – is so important when talking about medication review, because to be honest … in long-term care … ninety-nine percent of the time, it’s the pharmacist who is the one who’s identifying all these medications.

Another member of the health care professionals group said,

But when you’re assessing fall risk, you really have to see a client in their environment … that’s where the allied health will come in.

From the resident group:

I think one of the EPAs that might be added to the care of the elderly program is actually to recognize the limitations. When would you seek additional expertise [or] expert consultation? There’s a limit in terms of how much training I’m going to get in a year versus someone with a core internal medicine background who’s been exposed to very sick patients.

Theme: advocacy

All stakeholder groups made comments on the theme of patient advocacy. These thoughts were further categorized into the subthemes of Systems-based Practice, Social Contexts, Social Determinants and Transitions in Care.

The patient, manager/administrator and faculty groups all proposed an EPA be added that involves an awareness of locally available programs and resources. A representative quote comes from the manager/administrator group:

I wonder about possibly something about being aware of the supports that are available to seniors … [for example], knowing what programs are in your area.

The manager/administrator, patient and health care professionals groups suggested that the EPAs also require trainees to integrate an understanding of social contexts into their practice, particularly as it pertains to palliative care.

A participant in the patient group said,

Especially here in Manitoba, we live with a very large indigenous population … So I think cultural awareness is really, really important.

Similarly, members of the patient and faculty groups indicated that trainees should be required to demonstrate some EPAs with an understanding of the social determinants of health.

Regarding EPA 2, a member of the patient group said,

Say, someone’s income is very low and they’re lacking housing, they may present as having a problem with their hip or something. But if they had housing, stable housing, then maybe a lot of things would be a lot better, or if they had a secure source of food.

The health care professional, manager/administrator and resident groups indicated that EPA 6 could be modified to optimize the continuity of care provided by future CoE physicians.

For example, a member of the administrator/manager group said,

Those steps of transition can be real gaping holes where things are missed … If physicians can be really mindful of that and follow up and say, you were recently at the hospital, tell me what were the things they sent you home with? What are you using now?

Ideas for additional EPAs

All stakeholder groups were asked, “Are there additional EPAs that you think should be included?” Their responses to this question, and the specific stakeholder group(s) they come from, are summarized below. Based on stakeholder input, CoE trainees should also become able to:

• Screen for and manage substance use disorder in older adults (patient, health care professionals and resident groups).

• Demonstrate an awareness of local support programs available to elderly patients (administrator/manager, patient and faculty groups).

• Begin discussions with patients about medical assistance in dying (MAID) (patient group).

• Offer trauma-informed care when appropriate (patient group)

• Offer support to patients who have experienced the loss of a loved one (patient group)

• Identify elderly patients experiencing social isolation and propose appropriate solutions (health care professionals group)

• Perform a fall risk assessment (resident group)

• Identify when patient presentations are beyond the scope of the physician’s expertise, and consult with or refer to appropriate specialists (e.g., a Geriatrician) (resident and faculty groups)

• Perform consultations virtually (faculty group)

• Demonstrate an understanding of administrative realities in long-term care facilities (faculty group)

Some of the items suggested may be more suited to being integrated within existing EPAs, rather than a stand-alone EPA.

Discussion

We aimed to evaluate the utility of multi-stakeholder consultation in validating an EPA framework for the CoE residency program. Six themes arose from the focus group meetings: Specific vs. Broad, Operationalization, Setting, Clarity of Wording, Interprofessional Functioning and Advocacy. In addition, we received feedback on all ten EPAs and suggestions for additions to the EPA framework.

Themes

Feedback coded under Specific vs. Broad and its subthemes, Operationalization and Setting, indicates that the wording and/or scope of many EPAs were overly broad. This finding represents a commonly encountered challenge in EPA development; EPAs must be delicately balanced to provide the necessary specificity to standardize training, while being “broad and flexible enough to reflect the wide range of residents’ complex and unique clinical experiences” (Martin, Sibbald, Brandt Vegas, Russell, & Govaerts, 2020; Van Loon, Driessen, Teunissen, & Scheele, 2014). Addressing shortcomings in this area is critical in ensuring the development of successful EPAs. Faculty are often hesitant to entrust residents to perform EPAs with less supervision when their wording is ambiguously broad (Emke, Park, Srinivasan, & Tekian, 2019).

Stakeholders suggested changes to the wording of some EPAs. Most of these recommendations aimed to make terminology consistent with that used by other specialties. As language inconsistencies have been the bottleneck of introducing CBME into training program (Englander et al., 2017), unifying any language incongruencies should be of high priority. Other wording changes were geared toward ensuring the language used is inclusive to the full diversity of patients residents will encounter throughout their careers. Words used by health care practitioners significantly affect how patients perceive the inclusivity of the clinical environment (Shetty, Sehovic, Green, Schabath, & Ph, 2019). EPAs should prepare residents to offer inclusive care to patients of all backgrounds and orientations.

Although EPA 8 is exclusively dedicated to collaboration with other professionals, many comments were directed toward specific EPAs on the theme of Interprofessional Functioning. Taken together, the input received suggests that integrating interprofessional functioning into multiple EPAs may be a superior approach compared to teaching the skill as a distinct EPA. Graduating residents may not be aware of the specific contexts in which involving other health care professionals would be in the best interest of patients. Integrating interprofessional collaboration into multiple EPAs could provide the granularity necessary for residents to collaborate with other professionals when appropriate.

Contributions of stakeholder groups

The unique yet overlapping contributions of the stakeholder groups sheds light on what roles different stakeholders may be able to play in the validation of EPA frameworks.

The faculty, resident and administrator/manager group made comments regarding EPA scope (i.e., Specific vs. Broad). The only stakeholder group to bring forward setting-related concerns was the faculty group. Therefore, the development of EPAs that comprehensively assess residents on their performance in different clinical contexts might depend on a thorough consultation of appropriate faculty members. Thoughts on how EPAs would be operationalized only surfaced during the faculty and resident groups. Since the educational and clinical practicalities of EPAs most directly affect residents and their preceptors, it is unsurprising that these stakeholders are most apt to comment on EPA operationalization. For example, resident focus groups have brought attention to instances where EPAs have worsened workloads and compromised educational experiences (Branfield Day, Miles, Ginsburg, & Melvin, 2020). The only comment on the theme of Specific vs. Broad that came from the administrator/manager group was a neutral statement recognizing the broadness of the EPAs’ descriptions. Overall, then, the faculty and resident groups were the primary stakeholder groups able to provide insight into how the scope of various EPAs could be improved.

As previously mentioned, all stakeholder groups touched on the theme of patient advocacy. Interestingly, there was significant overlap between the subthemes of advocacy that the stakeholder groups touched on. In other words, no subtheme was exclusive to a specific stakeholder group. However, there were differences in the emphasis that stakeholder groups placed on each subtheme and patient advocacy as a whole. By far, the most attention to advocacy-related themes was given by the patient group. The patient group’s contributions were unique in this area. Participants shared many stories and examples of how physicians can offer suboptimal care by failing to identify important patient characteristics and circumstances. Our findings corroborate earlier work which found that patient feedback on EPAs was complementary to that of physicians but provided many unique details on trainee expectations (El-Haddad, Damodaran, McNeil, & Hu, 2017). Therefore, future efforts in EPA development should not depend on other stakeholder groups (e.g., faculty) to adequately represent the expectations patients have of CoE physicians in-training.

Limitations & strengths

It is important to consider some noteworthy limitations of the methods used in this study. First, focus group studies are traditionally limited by self-response and social desirability bias (Sheth et al., 2019). Ensuring each focus group is homogenously composed of only one stakeholder group partially reduced the effects of social desirability bias (e.g., compared to having residents and physician faculty participate in the same focus groups). Second, “errors of omission” (i.e., participants having ideas that they would like to have shared but were not triggered by reading the items on a list) can occur when participants are tasked with giving feedback on a list of EPAs (Smith et al., 2013). In addition, we may have been unable to meet thematic saturation due to the limited sample size of each stakeholder group. For example, El-Haddad et al. found that thematic saturation was achieved once 14 patients participated in focus groups (El-Haddad, Damodaran, McNeil, & Hu, 2017). In comparison, our patient focus group consisted of 7 participants. Our administrator/manager group was the lowest, at four participants.

However, the suboptimal sample sizes of each stakeholder group may have been offset by the breadth of stakeholder groups consulted, consisting of 30 total participants. The feedback received was diverse and will be pivotal in revising the EPA framework to be clearer, and more effective, patient-centered, collaborative and socially accountable. As predicted by recommendations for greater stakeholder consultation in EPA development, the feedback received will ultimately produce CoE physicians that offer more holistic patient-centered care (Bramley & McKenna, 2021). In addition, trainees will be able to collaborate more effectively with other professionals to meet the needs of their patients and operate better in the system at large. The quality of the feedback received through these focus groups is attributed to the unique perspectives and values each stakeholder group offered.

Recommendations

Residency programs would benefit from involving physician faculty, residents, health care professionals, administrators/managers and patients in the development of EPAs. We suggest that this consultation occurs early in the process before EPAs are implemented, but after faculty have finished developing a draft of EPAs. Frequent revisions to existing frameworks can cause “change fatigue” in the context of medical education (Marty et al., 2020).

Lessons can also be learnt from suggestions stakeholders made regarding EPAs that could be added to the training of physicians with Geriatric-focused practices. Such residents should be trained to address geriatric substance use disorders, offer trauma-informed care, manage social isolation, be able to discuss MAID with patients, identify when consultation with specialists is necessary, and demonstrate awareness of administrative realities in long-term care and the support programs available for elderly patients locally. In response to the COVID-19 pandemic, residents should also be trained to offer virtual care to patients when appropriate.

Future research could measure the benefits EPAs refined by multiple stakeholders have on the experiences of faculty and residents and the care patients receive.

Conclusions

As predicted by earlier work, multi-stakeholder consultation contributes greatly to the validation of EPAs. Feedback received through our focus groups affirmed that EPAs ought to be appropriate in scope, have socially accountable wording that is also congruent with that used by other specialties, and prepare trainees to be effective advocates for patients at many levels.

Availability of data and materials

Full survey results have been made available by the authors on Zenodo (DOI: 10.5281/zenodo.6385646)

Ethics approval and consent to participate

This research study was conducted in accordance with the Declaration of Helsinki. All participants were required to sign a consent form in advance of their focus group meeting. The final study protocol, survey guide and data collection tools were approved prior to the commencement by the University of Manitoba Health Research Ethics Board (research study HS24748).

Acknowledgements

We thank the Winnipeg Regional Health Authority as well as its Local Health Interest Groups for assisting in the distribution of study invitations to relevant stakeholders.

Disclosure statement

BC and DF were medical students at the Max Rady College of Medicine at the time of the study. PS is a board member of Age and Opportunity (unremunerated) and has received speaking honorarium from the University of Ottawa. JF has no competing interests to declare.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

The work was supported by the Max Rady College of Medicine, University of Manitoba .