The hospital educational environment and performance of residents in the General Medicine In-Training Examination: a multicenter study in Japan

Abstract

Background

It is believed that the type of educational environment in teaching hospitals may affect the performance of medical knowledge base among residents, but this has not yet been proven.

Objective

We aimed to investigate the association between the hospital educational environment and the performance of the medical knowledge base among resident physicians in Japanese teaching hospitals.

Methods

To assess the knowledge base of medicine, we conducted the General Medicine InTraining Examination (GM-ITE) for second-year residents in the last month of their residency. The items of the exam were developed based on the outcomes designated by the Japanese Ministry of Health, Labor, and Welfare. The educational environment was evaluated using the Postgraduate Hospital Educational Environment Measure (PHEEM) score, which was assessed by a mailed survey 2 years prior to the exam. A mixed-effects linear regression model was employed for the analysis of variables associated with a higher score.

Results

Twenty-one teaching hospitals participated in the study and a total of 206 residents (67 women) participated and completed the exam. There were no residents who declined to participate in the exam. The mean GM-ITE score was 58 (standard deviation 8.4). The mixed-effects linear regression analysis showed that a higher PHEEM score was associated with a higher GM-ITE score (P = 0.02). Having a department of general medicine, and hospital location in a provincial community (versus an urban setting), were also shown to have a significant relationship with the higher score (P = 0.03, and P = 0.02, respectively).

Conclusion

We found that the performance of the medical knowledge base of resident physicians was significantly associated with the educational environment of their hospitals. Improvement of the educational environment in teaching hospitals might be crucial for enhancing the performance of resident physicians in Japan.

Keywords:

• outcome-based education
• postgraduate medical education
• educational environment
• general medicine
• provincial hospital

Introduction

In Japan, after completing a 6-year medical undergraduate education, students enroll in 2-year postgraduate residency training programs.¹ Resident physicians are required to undertake supervised training in general medicine, including internal medicine, emergency medicine, and primary care. The Ministry of Health, Welfare and Labor of the Japanese Government, regulates the training programs and states the set of objectives for resident physicians to achieve basic knowledge of, and skill in, general medicine. When this national policy of 2-year postgraduate residency training programs was introduced in 2004, the computer-based national matching system was introduced and students could apply to their preferred hospitals throughout the entire country.² In our previous study, the greater satisfaction of non-university residents was shown compared with university residents in Japan.³ In that study, to compare the educational environments of university and non-university hospitals, we used the Postgraduate Hospital Educational Environment Measure (PHEEM), which is a 40-item inventory, and an evaluation tool consisting of a validated questionnaire with three subscales. PHEEM is considered to be a reliable and validated instrument, to evaluate the quality assurance process for the educational environment of a teaching hospital.^4–9

It is not clear whether teaching hospitals with a better educational environment could provide an educational program for their resident physicians, to achieve greater basic knowledge, engagement, and skill in general medicine. In this context, we conducted a multicenter study with the aim of investigating the association between a hospital educational environment and the knowledge base of medicine in Japan. To assess the knowledge base, we developed and used the General Medicine In-Training Examination (GM-ITE) based on a similar methodology used to develop the In-Training Examination in Internal Medicine (IM-ITE) which has been offered annually to all trainees in US medical residency programs since 1988.¹⁰

Methods

Measurements

For the outcome assessment, we developed and used the GM-ITE by employing the similar methodology used to develop the IM-ITE of the USA. The purpose of the US-ITE is to provide residents and program directors with an objective, reliable, and valid assessment of each resident’s personal performance in a written, multiple-choice examination, and the performance of each residency program compared with that of its peers.^11,12

The GM-ITE included 100 questions testing a wide range, from clinical skills and pragmatic medical knowledge to psychosocial care of the patient, designed and written by a committee of experienced attending physicians organized by the Japan Organization of Advancing Medical Education Program (JAMEP, a non-profit organization). The examination gives a maximum score of 100 and a minimum score of zero; higher scores indicate a better performance of knowledge base of internal medicine. The examination question review was conducted and the content validity was confirmed by peer review of each member of the committee. A 2-hour fixed time-limit was set for taking the exam.

Participants and data collection

We conducted a multicenter study involving Postgraduate Year 2 (PGY-2) resident physicians at 21 teaching hospitals in Japan. The invitation to participate in the study was announced on the JAMEP website. Teaching hospitals which voluntarily participated in our study provided the mean score of the PHEEM, which was collected in a survey conducted 2 years prior to the exam.³

The program director at each hospital was required to assemble residents in a room at each hospital at a scheduled time, and provide the GM-ITE for their residents. Each program director then collected the completed examination answer sheets, and sent these back to us, in an envelope we had provided. As the academic calendar in Japan starts on April 1 and ends on March 31 of the following year, the GM-ITE was conducted in March of the 2011 academic year. This study was approved by the institutional review board of the Mito Kyodo General Hospital, Mito City, Ibaraki, Japan

Statistical analyses

We tested the hypothesis of significant association between the PHEEM score and GM-ITE score by constructing a mixed-effects linear regression model to account for the clustering of the individuals within each residency program.¹³ Covariates as potential confounders included having a department of general internal medicine (GIM) or general medicine (GM) and hospital location (provincial or urban). Hospitals with a department of GIM or GM are considered to provide better educational programs for general medicine. Hospitals located in provincial communities are considered to have a better chance of seeing primary care patients than those located in urban areas. Statistical analyses were conducted using STATA version 11 (StataCorp LP, College Station, TX, USA). A 2-tailed P < 0.05 was considered statistically significant.

Results

A total of 206 PGY-2 resident physicians-in-training (67 women, 33%) participated and completed the examination. There were no residents who declined to participate in the exam. Of the PGY-2 physicians, 34.5% were women and 65.5% men.¹⁴ Among all participants, the mean score of the exam was 57.7 (standard deviation [SD], 8.4). The mean PHEEM score was 57.6 (SD, 5.4) among 21 hospitals. There were 14 teaching hospitals with a department of GIM or GM. Thirteen hospitals (62%) were located in a provincial community, while eight hospitals (38%) were located in an urban area. The hospital characteristics of the top 10 residents are shown in Table 1.

Table 1 The hospital characteristics of the top ten residents for the GM-ITE score

Ranking # of residents	Provincial/urban	GIM/GM	The PHEEM score
1	Provincial	Yes	149
2	Urban	Yes	99
3	Provincial	Yes	149
4	Provincial	Yes	149
5	Provincial	Yes	102
6	Provincial	Yes	102
7	Provincial	Yes	121
8	Provincial	Yes	106
9	Provincial	Yes	101
10	Provincial	Yes	149

Abbreviations: GM-ITE, general medicine in-training examination; PHEEM, postgraduate hospital educational environment measure; GIM, general internal medicine; GM, general medicine.

The results of the mixed-effects linear regression analysis (Table 2) showed that: a one-point increase in PHEEM score was associated with a 0.16 (95% Confidence Interval; CI, 0.023–0.29, P = 0.02) point higher GM-ITE score with beta of 0.16; having a department of GIM or GM was associated with a 4.07 (95% CI, 1.45–7.69, P = 0.03) point higher GM-ITE score; and a hospital located in a city (versus a provincial community) was associated with a −4.59 (95% CI, −8.46 to −0.72, P = 0.02) point lower GM-ITE score. This suggests that adjusted mean PHEEM scores were about 4.6 points lower among residents in urban hospitals than among those in provincial hospitals.

Table 2 The results of the mixed effects linear regression model for the GM-ITE score

Variable	Beta	95% CI		P-value
PHEEM score	0.16	0.023	0.29	0.02
GIM/GM (vs no GIM/GM)	4.07	1.45	7.69	0.03
Located in urban area (vs provincial)	−4.59	−8.46	−0.72	0.02

Abbreviations: CI, confidence interval; GM-ITE, general medicine in-training examination; PHEEM, postgraduate hospital educational environment measure; GIM, general internal medicine; GM, general medicine; vs, versus.

Discussion

Residents in teaching hospitals with higher PHEEM scores, with a GIM or GM department, or in a provincial program, demonstrated a better knowledge base of medicine. To the best of our knowledge, this is the first study to provide the evidence that the educational environment is associated with enhanced performance of knowledge base in the area of general medicine. The educational environment in a teaching hospital is likely to determine the engagement and motivation of residents and lead to greater achievement of educational outcomes.⁹ Because the program satisfaction level of residents also depends on their engagement and motivation, teaching hospitals with a better educational environment may also lead to greater satisfaction among residents.²

The lack of a GIM (or GM) department in teaching hospitals is not rare among Japanese teaching hospitals. Historically, Japanese hospitals did not have subspecialty medical divisions such as cardiology, gastroenterology, nephrology; they had only a department of medicine before the 1980s. During the 1980s, there was a boom in subspecialty medical divisions. Reflecting that change, no general medical department existed in Japanese hospitals by about 1990. However, after the 1990s, with an aim to educate every physician to retain the generalist’s mind even if the person is a specialist, a department of GIM or GM was established in multiple teaching hospitals to contribute significantly to clinical care, research, and clinical training. Since the GIM or GM departments cover all subjects of medicine and has a broad view of patients’ symptoms and concerns, ie, not focusing on a single organ system, it is understandable that programs with GIM or GM positively affect the knowledge base of general medicine.

Medical residents in provincial hospitals performed better in the GM-ITE scores in our study. Training programs in provincial areas may offer better learning opportunities, and thus improve learning outcomes in general medical knowledge and skills. Physicians, including primary care physicians, are scarce in provincial areas, hence residents in provincial teaching hospitals have to see and solve common medical problems of primary care patients. Due to the scarcity of physicians in such areas, patient exposure may be richer and more diverse in provincial hospitals than in city area hospitals.

Limitations of our study are: first, our results might have been influenced by sampling bias. Although the participants were conveniently selected and assigned to the examination, they did not necessarily represent all Japanese residents.

Other confounding factors may be the basic academic characteristics of the residents before the postgraduate training. Whether the higher GM-ITE score was due to a better educational environment or due to hard-working students in those hospitals remains uncertain; perhaps students who are good at taking examinations have the choice of selecting good hospitals, and the remaining students go to the other hospitals.

Further study will involve more participants across Japan, which would lessen the bias. Second, with respect to the cross-sectional study design, causality cannot be determined and thus the results require careful interpretation.

Conclusion

We conclude that the educational environment in a teaching hospital has been shown to affect the engagement and motivation for clinical training among residents. Training in provincial community hospitals with a GIM division offers residents greater exposure to patients with more common medical problems than highly specialized hospitals, thereby giving these residents the chance to cultivate their clinical skills and pragmatic knowledge. The result of this exam may be useful for designing an improved model of postgraduate medical education/residency training in Japan, also by collaborating with the Ministry of Health, Welfare and Labor of the Japanese Government. Even among teaching hospitals in non-provincial settings, supportive establishment of GIM division would improve the teaching outcome of the residents.

Acknowledgments

This research was funded by the JAMEP.

Disclosure

The authors have no conflicts of interest to declare.