Addressing the systems-based practice requirement with health policy content and educational technology

Abstract

Duke University Hospital Office of Graduate Medical Education and Duke University's Fuqua School of Business collaborated to offer a Health Policy lecture series to residents and fellows across the institution, addressing the “Systems-based Practice” competency.During the first year, content was offered in two formats: live lecture and web/podcast. Participants could elect the modality which was most convenient for them. In Year Two, the format was changed so that all content was web/podcast and a quarterly live panel discussion was led by module presenters or content experts. Lecture evaluations, qualitative focus group feedback, and post-test data were analyzed.A total of 77 residents and fellows from 8 (of 12) Duke Graduate Medical Education departments participated. In the first year, post-test results were the same for those who attended the live lectures and those who participated via web/podcast. A greater number of individuals participated in Year Two. Participants from both years expressed the need for health policy content in their training programs. Participants in both years valued a hybrid format for content delivery, recognizing a desire for live interaction with the convenience of accessing web/podcasts at times and locations convenient for them. A positive unintended consequence of the project was participant networking with residents and fellows from other specialties.

Introduction

The Accreditation Council of Graduate Medical Education (ACGME) requires residents and fellows regardless of specialty, to develop competency in six competencies: patient care, medical knowledge, professionalism, interpersonal and communication skills, practice-based learning, and systems-based practice. Systems-based practice has been recognized as one of the more challenging programs to both teach and assess (Moskowitz & Nash 2007; Varkey et al. 2009). To address this need at an institutional level, Duke University Hospital (DUH) Office of Graduate Medical Education (OGME) and Duke University's Fuqua School of Business collaborated to offer a Health Policy lecture series to residents and fellows. Residents and fellows reported having little exposure to Health Policy content but believed it is important for their practice within the larger health care system. Two instructional strategies were offered (live and web/podcast). Residents and fellows could choose the instructional strategy they believed would work best for them. The program encouraged interaction among specialties and program levels, since few opportunities exist during Graduate Medical Education (GME) training. This study analyzed the need and or desire for health policy content, comparison of two instructional delivery methods, and the perceived benefit of interaction among specialties.

Table

Practice points

Health policy content is missing from medical education curriculum. Health policy content meets the ACGME competency of “systems-based practice.” Residents and fellows prefer a hybrid format of delivery over strictly live lecture or web/podcasts. Interdisciplinary interaction between residents and fellows enhances their educational experience.

Background

DUH is the sponsoring institution for more than 900 residents and fellows training in 74 ACGME-accredited programs and more than 50 non-ACGME accredited programs.

In 2007, Dr Victor Dzau, the Chancellor of Duke University Health System created an endowment to provide new funding to support GME and encourage GME “innovation.” The goal of the funding is to reward excellence in teaching, assessment, and program improvement by encouraging educational innovation, collaboration, and sustainability. Program directors, faculty, and residents and fellows are encouraged to submit applications on a yearly basis.

The GME and the Duke Fuqua School of Business – Health Sector Management partnered to submit a proposal to provide a health policy lecture series for residents and fellows across the institution. Both entities recognized the need for resident and fellow professional and personal development in health policy as one of the important components of systems-based practice (Shortt & Hodgetss 1997; Taylor & Chudley 2001; Allen et al. 2005).

The project was designed to:

1. Assist programs in meeting the ACGME requirement of systems-based practice.

2. Provide a mechanism to document “competency” with a multiple choice post-test and evaluation.

3. Compare two instructional strategies (traditional lecture with web/podcast lecture)

4. Assess the impact of multidisciplinary interaction through facilitated focus groups.

Methods

The first year ran from January 2008 to December 2008. The second year ran from January 2009 to December 2009. The project is currently in its third iteration, with improvements based on participant feedback. This article reports on outcomes for Year One and Year Two.

Content development

One of the authors, Schulman K, a cardiologist jointly appointed in the School of Medicine and Fuqua School of Medicine, led the Fuqua School of Business – Health Sector Management faculty in developing the initial lecture series. Fourteen topics were identified (Table 1).

Table 1.  Topics for health policy lecture series for Years One and Two

Year One	Year Two
Health Policy series	Health Policy series	Health Law, Ethics and Global Health series
Private Health Insurance	Health Macro-Economics	End of Life Care
Medicare	Physician Payment	Conflicts of Interest
Medicaid, SCHIP and the Uninsured	Private Health Insurance	Research Ethics: The Jesse Gelsinger Case
Health Care Disparities	Medicare	Global Health: Introduction to Global Health
Health Care Policy and 2008 Candidates	Medicaid, SCHIP, Uninsured	Global Health: Comparative Health Systems
End of Life and LTC	Health Policy and the Obama Administration	Introduction to Law and Medicine
Comparative Health Care Systems	Health Information Technology	Law and Medicine: Tort and Medical Malpractice
Health IT	Quality and Pay for Performance	Employment and Contract Law
Quality and Pay for Performance	Technology: Drugs and Devices	Biases
Technology: Drugs and Devices		Medical Decision-Making
Conflict of Interests
Health Care Costs and Projections
Physician Payment
Hospital Payment

Based on feedback from participants and a thorough review of curriculum content, a second series of lectures was offered during Year Two – Health Law, Ethics, and Global Health. The format changed as well to provide all content via web/podcast with quarterly panel discussions with lecturers serving as panelists (Table 1).

Instructional strategies and participants

During the first year, two instructional strategies were utilized. Participants were given the choice of attending monthly live lectures or accessing the lectures via web/podcasts. Those who chose to participate via web/podcast were provided with a new 80gb iPod classic to access Duke's iTunesU. Participants were asked to sign and submit a Participant Agreement, agreeing to attend (or listen to if in the web/podcast group) 12 of the 14 lectures, attend three focus groups and a health policy symposium, complete two post-tests highlighting the important points of each lecture, and an evaluation. Each participant's program director was also required to approve and support their participation by signing the Participant Agreement. The use of web/podcasts was built upon knowledge derived from previous institutional curricula facilitated by OGME (Andolsek et al. 2010).

The 1-hour lectures were held monthly in a large on campus lecture hall. Presentations were recorded using Camtasia Studio (http://www.techsmith.com/camtasia.asp), and edited slightly and uploaded to a password protected site on Duke's iTunesU page. Those accessing the web/podcast could view the presentation slides and hear the presenter and the audience question and answer period. The presentations could be opened on their computer or downloaded and viewed on their iPod (or MP3 player equivalent).

A wiki page was created to provide participants with course schedule, participant roster, the lecturer's PowerPoint slides, speaker biographical information, and suggested readings. Also available on the course wiki page were links to iTunesU and the online lecture evaluation, created using SurveyMonkey (SM) (http://www.surveymonkey.com/).

During Year Two, all lectures were made available on-line via Duke's iTunesU webpage. Four live panel discussions were facilitated by webcast presenters or other content experts. Panels were scheduled for 12:00 pm on a weekday based on participant responses to the most convenient times for them to attend. Participants were again asked to sign (and have their program director sign in support) a Participant Agreement, agreeing to listen to/view web/podcasts, and complete post-tests and evaluations of modules and panels.

Evaluation

Numerous tools were put in place to evaluate the quality of the lecture series and compare the two methods of delivery. All participants were required to take two electronic multiple-choice post-tests via SM. The post-test was developed by asking presenters to submit 2–4 multiple choice questions highlighting the most salient points of their talk. Participants had one opportunity to take the test and their results with the correct answers were emailed to them. Test results of the two groups of participants (live lecture and web/podcast) were compared.

During Year One, a 10-question electronic evaluation using the survey tool SM (http://www.surveymonkey.com/) was developed for each lecture. Feedback was gathered on the content value and preference of delivery format. Participants in the live lectures were asked to rate the presentation format and the group discussion. Participants using iTunesU, were asked to rate the clarity of slides and audio, timeliness of availability, ease of use, and appropriateness of the use for distance learning. During Year Two, the evaluation solicited feedback about module content, delivery method, and panel format.

Year One participants were required to attend three focus groups. Preliminary analysis was carried out alongside data collection to facilitate the ongoing elaboration of emerging themes. The facilitator used open-ended guiding questions asking participants to assess the health policy content, delivery formats, and impact of multidisciplinary interactions. Focus groups were recorded and transcribed. Identifying information was removed from the transcripts prior to analysis. Data were extracted using the process of open coding to identify recurring themes until the point of saturation (Miles & Huberman 1994; Strauss & Corbin 1998; Kennedy & Lingbard 2006).

The study was “exempted” by the Duke School of Medicine Institutional Review Board.

Results

For Year One, a convenience sample of 34 individuals (all from ACGME accredited programs) committed to participate; eight for the seated lectures and 26 for the web/podcast option. Six of the then 11 Duke GME departments were represented as were program levels 1–7. For Year Two, there was a convenience sample of 25 participants in the Health Policy Lectures series and 18 in the Health Law, Ethics, and Global Health, again representing 6 of the now 12 Duke GME departments (Dermatology became its own Department in 2009). A greater number of residents and fellows participated in Year Two.

Health policy content

Participants’ previous exposure to health policy topics varied but most had no prior formal education in this area. Some had attended one or two prior lectures and one had taken a semester-long course while in college. All reported that Health Policy content is critical for practicing physicians but that, in their experiences, does not exist formally in the undergraduate or GME curriculum. Representative comments from the Year One focus groups include:

• “This content is critical to the success of a physician.”

• “I was embarrassed I didn’t know this information prior to hearing these lectures. This should be a standard part of our training.”

• “We must help guide the policies that impact our work as physicians and we can’t do that if we don’t know anything about health policy.”

• “We weren’t exposed to this content in medical school or through our GME program curriculum. I worry about all the others who won’t get to hear these lectures.”

Ninety-four per cent of the responders reported that the level of content offered was appropriate and 97% reported the information presented included the appropriate context for the management of patients.

When asked what they would do differently as a result of participating in the lecture series participants shared the following:

• “Better assess political party plans and make an informed decision on election day.”

• “Share this information with colleagues who are missing out.”

• “Participate in hospital committees now that I have some idea what goes on around us.”

• “Be a better patient advocate.”

Content delivery method – live versus web/podcast

During Year One, 26 of the 34 or 76% of participants chose to watch and or listen to the lecture series via web/podcast as opposed to traditional seated lecture. Participants were asked during the focus groups why they chose the live lecture or the web/podcast version. See Table 2 for representative responses.

Table 2.  Participant explanation for selected method of participation

Traditional live lecture	Web/podcast
Easier to be more alert in a lecture and easier to zone out while listening to an iPod	Its all about time and convenience
More meaningful if there is direct interaction with the presenter	Podcast allows you to pause, rewind, play again
If important topic, it's important to be present	There is no time for extra lectures – can’t even get to the ones scheduled in own program
Seated is always better – if there is time	Risk falling asleep in a lecture, but can control the podcast and focus
On-line courses can be boring with no interaction	It is less distracting. In residency lectures, there are pagers going off and people moving in and out
It is important to be able to ask questions at the right time	Can listen or watch and do it any time or place
There is a huge educational value attached to the in-person interactions	Can listen while doing something else
	Ideally, content could be offered using a combination of methods

Table 3.  Evaluation outcomes

	Live lecture (n = 24)^a		Webcast (n = 21)^a		Podcast (n = 66)^a
Presentation format	Below expectations	0%	Below expectations	0%	Below expectations	3.0% (2)
	Met expectations	33.3% (8)	Met expectations	52.4% (11)	Met expectations	45.5% (30)
	Above expectations	66.7% (16)	Above expectations	47.6% (10)	Above expectations	50% (33)
Benefit to me	Below expectations	12.5% (3)	Below expectations	9.5% (2)	Below expectations	1.5% (1)
	Met expectations	41.7% (10)	Met expectations	33.3% (7)	Met expectations	40.9% (27)
	Above expectations	45.8% (11)	Above expectations	57.1% (12)	Above expectations	56.1% (37)
Use of my time	Below expectations	12.5% (3)	Below expectations	9.5% (2)	Below expectations	0%
	Met expectations	41.7% (10)	Met expectations	42.9% (9)	Met expectations	47% (31)
	Above expectations	41.7% (10)	Above expectations	47.6% (10)	Above expectations	51.1% (34)
Session overall	Below expectations	4.2% (1)	Below expectations	4.8% (1)	Below expectations	0%
	Met expectations	41.7% (10)	Met expectations	38.1% (8)	Met expectations	45.5% (30)
	Above expectations	54.2% (13)	Above expectations	57.1% (12)	Above expectations	53% (35)

Note: ^aIt is not known how many different individuals responded as these were anonymous evaluations and responses cannot be tracked to an individual.

Year One participants were asked to evaluate each lecture they attended or listened to in Duke's iTunesU. There were 109 responses in total. (It is not known how many different individuals completed the evaluations as the evaluation was intended to be anonymous and thus responses cannot be linked to individuals in any way.) The results of the evaluation were filtered by those who participated by “live lecture,” “webcast,” (on a computer), and “podcast” (using an iPod). Results are included in Table 3.

All participants reported that a combination of live and distance education would be ideal. There was unanimous agreement that listening to lectures via web/podcasts was appropriate for this content and that there would be added benefit from some live interaction with experts. Thus, the instructional strategy for Year Two was revised to make all lectures available as web/podcasts and schedule four live panel discussions facilitated by webcast presenters or other content experts.

In Year Two, participants completed evaluations after watching groups of 2–3 lectures and attending a panel discussion (A total of eight, four Health Policy series and four Health Law, Ethics and Global Health). Participants were asked to rate satisfaction with program format (listening to/watching lectures via web/podcast and attending live panel; discussions) after each panel; 33 of 48 responses, or 69% were “completely satisfied,” while the other 15 responses were “somewhat satisfied.”

Feedback from focus groups revealed that both live and web/podcast participants found it convenient to have all course information and materials available on a wiki.

Post-test performance

There was no difference in the performance on the post-test (mean score overall) between the Year One live lecture participants and webcast participants. For the group as a whole, the mean score was 82% on the 32 question post-test (mid and final combined). The range of scores was 41–100%. The mean score for the seated group on the post-test was 82% (range was 59–100%). The mean score for the webcast/podcast group was 82% (range was 41–100%).

For Year Two, the mean score for the health policy series was 85% (25 question post-test) and for the Health Law, Ethics and Global Health (31 question post-test) was 88% (Table 4).

Table 4.  Performance on post-test by participants in Years One and Two

	Year One		Year Two
	Live lecture	Web/podcast	Health policy	Health law, ethics and global health
Number of participants^a	3	21	10	4
Mean score range	50–100%	40–100%	42–100%	75–100%
Mean score overall	82%	82%	85%	88%

Note: ^aSome participants did not complete the post-test(s) (chose not to or graduated).

See “Study limitations” section for brief discussion of the small number of individuals in Year One participating in live cohort and the missing post-test results for a portion of the participants.

Interaction between specialties

Participants from Years One and Two reported (in focus groups and on evaluations) that it was beneficial to have an opportunity to meet and “mingle” with residents and fellows from other programs. Representative comments included:

• “I may go through my entire training and never interact with trainees in another discipline.”

• “I know very little about other specialties and if we are to become competent in systems-based practice, I really should.”

• “It has been an added benefit to be able to meet residents and fellows from other specialties. We are such a small fellowship, I don’t get to meet many other trainees.”

• “I have a new appreciation for ____ residents/physicians.”

• “We interact little … but our decisions affect each other greatly. I found each specialties outlook on different issues thought provoking and useful.”

All participants agreed that these interactions with residents and fellows from other specialties would improve both their training experience and the medical care they delivered, especially consultative care. One participant's story is typical. “Calling for a consult or receiving a page can be a bad experience. No one feels they have time and mostly, people call without having the right information or knowledge, resulting in a waste of people's time. After my conversations with people here, I now have a better idea what information I should have when I call and it will certainly help being able to contact a person I now know.”

Another participant said: “[m]y experience with calling for consults has not been good – and I hear and see the same watching and listening to attendings. Knowing people in these areas will help tremendously and I know I will respond back to those who page us differently than I have in the past.”

Thus, the participants recognized both personal and professional gain from interacting with residents and fellows from other specialties.

Discussion

Health policy content

Individual GME programs may struggle to provide all the necessary knowledge, skills, and experiences required to help make residents and fellows competent. As the participants report, very few of their medical schools or residency programs include health policy content as a formal part of the curriculum. This may be because of lack of faculty expertise, uncertainty of importance or resident and fellow interest, and or lack of curricular time, partly as a result of duty hour restrictions (Jagsi et al. 2006; Woodrow et al. 2006; Chudgar et al. 2009). However, there is little debate that this content greatly impacts physicians, hospitals, and patients. As health care becomes more important in the world economy, physicians need education specific to this evolving industry (Agrawal et al. 2005; Ouraishi et al. 2005; Riegelman 2006; Lee et al. 2007).

This project leveraged the institution's association with a University School of Business. It would be unrealistic for 74 individual programs to arrange to have these experts provide conferences for their own individual residents and fellows.

Participants in this study confirmed that they were willing to dedicate their own time to become familiar with and engage in dialogue about health policy. Program directors supported and encouraged this participation. Participants found the content and instructional formats useful and realistic and were able to identify ways in which they will change their behavior as a result of being exposed to this new information.

Content delivery method – live versus web/podcast

There are mixed views on the topic of when and how to effectively use distance education (Cook & Dupras 2004; Cook 2009). As demonstrated in previous studies (Kerfoot et al. 2007; Cook et al. 2008; Pereira et al. 2008), the web/podcast participants were as successful on their post-test as those participating in the live lecture (although it is not known how those who did not complete the post-test would have performed, as mentioned in the “Study limitations” section). The participants in this study are part of generation “y” and “x” (Sandars & Morrison 2007). They require active engagement and the ability to multi-task (Maag 2006). Being able to watch and or listen to a lecture at a time and place convenient for them, meets the generational expectations (Boulos et al. 2006; Sandars & Morrison 2007) as well as the practicalities of GME. In Year One, 26 out of 34 participants chose webcasts/podcasts over live lectures. Feedback from all participants suggests distance education is more convenient, realistic, and preferred, especially when combined with some opportunity for live discussion follow up. “Hybrid courses” are convenient for learners, allowing them to access a lecture when and where they choose, refer back to the material when needed, and experience some interaction (Rochester & Pradel 2008).

Interaction between specialties

One “unexpected consequence” of offering this content across specialties was the participant interaction and opportunity to learn more about one another as individuals and as specialties. Duty-hour restrictions may have resulted in less “down time” where residents and fellows simply “hang out” in the hospital. While the course was not designed specifically for this purpose, it brought together residents and fellows across specialties for shared education. This resulted in relationship building, networking, and a renewed appreciation for the work of each other. In addition to the social component, the participants surmised that their teamwork, consult interactions, and ultimately patient care would improve as a result of their interactions (Christie et al. 2007; Baldwin & Daugherty 2008; Chakraborti et al. 2008). Medicine has become increasingly more complex and specialized patient care undoubtedly would be enhanced by finding more ways for physicians to learn from, appreciate, and work collegially with those in other specialties.

Cost and sustainability

It is difficult to determine the true cost of the lectures and web/pod casts or the “break even point” based on how many residents participated. Presenters were paid on average $2000 per lecture. A higher stipend was offered as there were some additional expectations of the presenters such as introducing their topic using a case study, develop PowerPoint slides that could be easily viewed if downloaded to an ipod, and provide multi-choice questions that highlighted the key points of the lecture. Additional expenses included the administrative support of two individuals: one who worked with the participants marketing, registration, attendance, post-test completion, and completion certificates and the other who identified and served as a contact for speakers, collected post-test questions, and handled speaker payment. There was no expense for utilizing Duke's iTunesU of the wiki. SM (used for the post-tests and evaluations) is $150.

Utilizing web/podcasts as a means of content delivery allows more individuals access to a precious, finite resource (the Fuqua and Health System faculty) who could not have delivered the lectures to every residency program. There is now a library of more than 20 formal lectures captured and available to all Duke residents and fellows allowing a sustainable way to deliver this content institutional wide. The content captured is unlikely to change drastically or frequently. Thus, if residents and fellows continue to have access to this important content in a means convenient for them, it is well worth the cost and eventually, a minimal cost per user.

Study limitations

There were several limitations to this study. There were only 77 participants; all of whom were self-selected. And, the number of participants in the live lecture cohort in Year One does not lend itself to comparison with other groups. In addition, without a pre-test, it is not clear whether there was new knowledge obtained. We cannot be sure if the results are representative of the larger group of residents and fellows at Duke, or elsewhere. Finally, only a subset of the participants took and or completed the post-tests. Not knowing how they would have performed, the researchers acknowledge the analysis might be different had all participants completed the post-test.

Conclusion

We believe that this course allowed Duke OGME to partner with experts and Duke Fuqua to help programs across the institution meet the ACGME requirement of systems-based practice. We allowed participants to choose their preferred educational format in Year One and used their feedback to design a new format in Year Two. All four cohorts (Year One seated and web/podcast and Year Two Health Policy and Health Law, Ethics and Global Health) rated their experience highly and were equally successful in passing a post-test documenting successful knowledge acquisition. Resident and fellow interaction across specialties proved to be an important “side effect.” Evaluation results and participant focus group discussion provided useful feedback for program improvement. At the end of 2 years, there is a library of enduring lectures which will be available for all of residents and fellows and programs to continue to teach health policy content.

Acknowledgment

The authors thank Dr John Weinerth, Director of Graduate Medical Education at Duke University Hospital for his leadership and support of this project.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.