General physicians graduated from a PBL undergraduate medical curriculum: How well do they perform as PBL tutors?

Abstract

Background: A study was conducted on the effectiveness of general physicians recently graduated from a medical school with Problem-Based Learning (PBL) curriculum as PBL tutors to expand the school's tutor pool.

Aims: This study aims to investigate these non-staff tutors’ effectiveness in terms of student satisfaction and learning outcomes.

Method: An experimental study was conducted of 12 PBL groups of second-year medical students (n = 40). Four PBL groups were led by non-staff tutors; the other eight groups were led by staff tutors during the two PBL units. Tutor evaluation and student satisfaction questionnaires were administered and student performance scores were analysed to compare between groups led by staff tutors and non-staff tutors.

Results: The students’ overall satisfaction with the non-staff tutors on a five-point Likert-scale was high (M = 4.5 ±.638). Additionally, the student scores on written tests were comparable between groups. Yet, in one unit, the groups led by staff tutors received significantly higher scores on the group evaluation than those led by non-staff tutors.

Conclusions: The results of this study show that the non-staff tutors performed as effectively as the staff tutors did with regard to student achievement in written exams. Still, the findings of this study suggest that different tutor backgrounds and experiences might affect student performance beyond the written exam scores.

Introduction

The effectiveness of Problem-Based Learning (PBL) in medical education has been addressed extensively in the literature. However, there are several challenges to implement this innovative educational approach. Among such challenges is the large number of tutors necessary to implement the PBL curriculum. Accordingly, many medical schools with a PBL curriculum have utilised non-expert tutors, including residents and graduate students to mitigate this issue (Kwizera et al. 2001; Dolmans et al. 2002; Matthes et al. 2002; Gilkison 2003; Groves et al. 2005; Maudsley et al. 2007). Various competencies are required to be an effective PBL tutor (Schmidt & Moust 1995; De Grave et al. 1999; Neville 1999). Therefore, medical schools need to make an informed judgment on the effectiveness of non-expert tutors before they decide to use them.

Several research studies compared the effectiveness of expert tutors and non-expert tutors; yet, they varied in their definitions of the non-experts participating in their studies. Gilkison (2003) and Groves et al. (2005) compared the effectiveness of tutors with different backgrounds. They categorised medically qualified tutors as ‘expert’ and tutors without any background in medicine as ‘non-expert.’ They found that expert tutors questioned students and used their subject matter knowledge more often and non-expert tutors expected students to question each other and facilitated the group process more often. In another study, Groves et al. (2005) compared staff tutors and non-staff tutors, and they found that staff tutors were more knowledgeable in subject matters and non-staff tutors were more skilled in facilitating the PBL process.

Table

Practice points

There is an increasing need for medical schools to come up with various sources of PBL tutors to implement PBL and those with PBL curriculum can consider their own graduates as PBL tutors to expand their tutor pool. This study shows that PBL groups tutored by recent graduates of the same program with a PBL curriculum achieved learning outcomes as good as those tutored by faculty members. This study shows that general physicians recently graduated from the medical school with PBL curriculum can be effective PBL tutors due to their own experiences in PBL as medical students. This study provides evidence for the feasibility of using general physicians who recently graduated from a medical school with a PBL curriculum as PBL tutors with minimal time commitment from them.

Several studies have reported that the tutors’ content expertise and their experience with PBL tutoring had no bearing on the students’ learning outcomes in terms of their written exam scores (Dolmans et al. 1996; Matthes et al. 2002; Park et al. 2007). However, Schmidt et al. (1993) found that students tutored by content experts are likely to spend more time in self-directed study and performed better in written tests than did those tutored by non-experts. They also found that the tutors’ process-facilitation skills affected student achievement and were correlated to their content expertise, indicating that the two are necessary and closely related conditions for effective tutoring. In addition, Kassab et al. (2005) investigated the effectiveness of peer tutoring and compared the performance of PBL groups guided by expert tutors (faculty members) and by student tutors (third-year medical students). The two groups were comparable in terms of their written and practical exam scores. However, the student-led groups had more difficulty in analysing problems during the first tutorial session, indicating that student tutors require training to become effective tutors.

As one of the first Korean medical schools to adopt PBL, the medical school under investigation has a well-established PBL program and it provides a great deal of support for PBL. However, faculty members are finding it challenging to take part in PBL in the midst of increasing pressure on the clinical duties and research productivity. Furthermore, only a small number of non-clinical faculty members in this particular program can participate in PBL because they are located at a distance from the campus where students do PBL. As a result, efforts were made to find the tutor pool from those who were not faculty members yet were familiar with the PBL process and had basic knowledge in clinical sciences.

Being exposed to PBL during the 2 years of its pre-clinical program, graduates of the medical school under investigation are well versed in the PBL process. Additionally, the performance of the graduates of this program in national standardised tests has been consistently high – with virtually 0% of failure rate in the Korean medical licensing examination for the past several years. This indicates that most of the graduates likely mastered the knowledge they were expected to acquire to pass the licensing exam. Moreover, our annual surveys of new students indicated that a large number of students in this program hope to prepare themselves to become medical educators. Thus, it was our expectation that many of the recent graduates of this program were motivated to teach and they had acquired the basic knowledge in the subjects to be covered in the PBL blocks.

Accordingly, recent graduates of this medical program, particularly those who were practicing as general physicians, were suggested as prospective PBL tutors. Yet, there was no evidence on how well they would perform as PBL tutors because it had not been tried before. Therefore, a research study was conducted to investigate the effectiveness of these PBL tutors. The tutor effectiveness was evaluated at two levels in Kirkpatrick's (1994) model for the evaluation of educational programs – i.e. student satisfaction and learning outcome. The effectiveness of these non-staff tutors were compared with that of staff tutors.

Methods

Participants

A class of second-year pre-clinical medical students (n = 40) participated in the present study. The students had already gone through 1 year of a PBL program in their first pre-clinical year. This medical school has used PBL in the pre-clinical curriculum for over a decade. About 60 faculty members participate in the PBL tutorials each year as tutors or co-tutors; they are drawn from a pool of over 450 basic and clinical sciences faculty members at the school of medicine. Among these medical faculty members participated in the present study. Additionally, 16 non-staff tutors participated in the present study. These non-staff tutors were selected from recent graduates of the medical school under investigation who were practicing as general physicians at local public health centres across the nation as part of their mandatory 3-year public service duties.

To select the non-staff tutors participating in this study, all graduates from the medical school who graduated between 2003 and 2006 and were practicing as general physicians were contacted and all those who were contacted agreed to participate. Among these 16 general physicians, six had completed their internships before they began their practices and the other 10 were practicing right after they graduated from medical school. The mean Grade Point Average (GPA) scores of these general physicians for their undergraduate medical studies was 3.345 out of 4.5 (SD = 0.438). All of these non-staff tutors were male and none had taken the role of PBL tutor before. Recent graduates who were in an internship or resident training program were not included in the tutor pool for the present study due to the difficulty with their time commitments.

Study design

An experimental study was conducted for 12 tutorial groups, which spanned two PBL units that were offered in the year 2007. Students were divided into six PBL groups in each unit; two groups were led by non-staff tutors and the other four were led by staff tutors. Each unit was 1-month long and the topics covered in the units under investigation were psychiatry and the musculoskeletal system. The tutorials took place twice each week and each tutorial was 3 h long. The students were randomly assigned to a PBL group of six or seven members and they were all assigned to different groups after each unit. Therefore, none of the students were in an experimental group for both units during the study.

The staff tutors participated in the present study were recruited and assigned to a PBL group about 6 months before this study took place. These staff tutors had various backgrounds with regard to specialty. The staff tutors’ experience with PBL also varied; about one half of them had no experience in PBL tutoring before they participated in this study. The staff tutors attended weekly ‘pre-tutorial’ sessions, where they were briefed on the problem by the case writer and the PBL process was discussed with the unit chair for 30 min before the first tutorial began throughout the duration of the PBL unit. Meanwhile, the non-staff tutors were briefed on the tutorials by receiving tutor guide documents via E-mail about 2 weeks prior to the start of the unit.

Each non-staff tutor was randomly assigned to a PBL group for 1 week, as they could take part for 1 week only due to their limited time availability. Thus, four non-staff tutors were assigned to a PBL group for the duration of the unit for the experimental groups, wherein the tutor was rotated each week. In contrast, the staff tutor led the group for the whole duration of the unit for the control groups.

Data collection and analysis

Several instruments were used to collect the data for this study. Students in the experimental groups completed tutor evaluation forms each week to rate the non-staff tutors whom they worked with (n = 110). This tutor evaluation form was adapted from the one developed by Hendry et al. (2002). This questionnaire included eight five-point Likert items (i.e. ‘1 = very unsatisfactory’, 5 = ‘very satisfactory’) and one open-ended question regarding the student perceptions of the effectiveness of non-staff tutors. Additionally, the data on student overall satisfaction with non-staff tutors were collected by administrating a questionnaire to the students in the experimental groups at the end of the unit (n = 28). In this end of the unit questionnaire, the students were asked to rate their overall satisfaction level with the non-staff tutors on a five-point Likert scale. Additionally, three open-ended questions were provided to solicit their perceptions of the benefits and disadvantages of working with non-staff tutors. Besides, the non-staff tutors filled out a questionnaire that consisted of three open-ended questions about their experience as PBL tutors and suggestions for the improvement of PBL program.

The data on students’ learning outcomes were obtained from the assessment scores based upon the PBL evaluation system adopted by the medical school being studied, in which students are evaluated both individually and in groups. More specifically, students took weekly quizzes every Monday morning during the PBL unit, which accounts for up to 10% of their overall score on the unit. In addition, peer and group evaluations were conducted at the end of the unit, each of which accounts for up to 5% of the total unit score. In addition, tutors rated the students’ performance in the PBL group in student evaluation, which accounts for up to 20% of the total unit score. Finally, the students sat for a comprehensive written examination, which is mostly multiple-choice questions, at the end of the unit (up to 50% of the total unit score); attendance accounts for up to the remaining 10%.

The students’ learning outcomes in the PBL units under investigation were compared between groups led by staff tutors and non-staff tutors using independent t-test and one-way ANOVA. Additionally, correlation analyses were conducted of the tutor's credentials (GPA scores in their undergraduate medical studies, the scores in the Korean medical licensing exam) to investigate the effect of the tutor's credentials on their performance. Furthermore, the number of learning issues in each PBL group produced for a case was counted each week to compare the quantity of output of the group's learning process between the control and experimental groups.

Results

Student satisfaction with non-staff tutors

The results of student survey showed that the students’ perceptions of the non-staff tutors were positive. Student perceptions of tutor effectiveness rated on the eight Likert-scale items ranged from 4.49 to 4.68 (1 = very unsatisfactory, 5 = very satisfactory). Additionally, the mean score for the students’ overall satisfaction rate was 4.50 (SD = 0.638) using the same five-point Likert scale. The results of an ANOVA showed that the student perceptions of tutor effectiveness varied among the non-staff tutors for two items: ‘our tutor gave feedback on our group dynamics and learning process’ (F = 1.806, p < 0.05), and ‘our tutor was knowledgeable about the content covered in this week's case’ (F = 2.317, p < 0.01). However, the results of an ANOVA showed that there was no significant difference in the overall student satisfaction among the four PBL groups led by non-staff tutors (F = 1.059, p = −0.385).

Differences in student achievement between groups

Comparisons were made on the performance of the students led by staff tutors and non-staff tutors. Independent t-tests were performed of the students’ overall performance scores in the unit, which is a composite of a variety of evaluation components as alluded earlier. The results showed no significant difference in overall student performance between groups for both units. A chi-square analysis was performed to compare groups in their GPA scores and in their first-year pre-clinical studies to determine if there were any group differences in terms of their academic performance in the previous academic year. The results of the chi-square analysis showed that the students’ GPA scores were not significantly different between groups at the 0.01 level (χ² = 1.800, p = 0.031 and χ² = 3.600, p = 0.058 in each unit).

Additionally, independent t-tests were performed of weekly quiz scores, comprehensive exam scores and group evaluation scores to investigate differences in student achievement in various performance measures between groups. Quiz scores were obtained by averaging the scores of weekly quizzes administered throughout the unit. The comprehensive exam scores were obtained from the written test conducted at the end of the unit. The results of t-test showed that there were no significant differences in quiz scores and comprehensive exam scores between groups in both units. However, the group evaluation scores, which were evaluated by the unit chair on the case analysis reports and case maps that each group had submitted at the end of each case were significantly different in one unit. In this unit, the group evaluation scores of the PBL groups led by staff tutors (M = 4.48, SD = 0.40) were significantly higher than those led by non-staff tutors (M = 4.18, SD = 0.70), p < 0.01. An analysis of Cohen's d shows that these t-tests had moderate effect sizes, where d = 0.543 and 0.536 for each unit.

In addition to the student assessment scores, the number of learning issues identified by each group were recorded and analysed. The average number of learning issues reviewed by the PBL groups were comparable between the groups led by staff tutors and non-staff tutors (M = 7.1 and 6.9, respectively).

Relationships between tutor's backgrounds and student perception of tutor effectiveness

Correlation analyses were performed to investigate the impact of the non-staff tutors’ academic credentials on student perceptions of tutor effectiveness. The results showed that the tutors’ GPA scores in their undergraduate medical program were not significantly correlated with their effectiveness as tutors as perceived by students; the p values on the eight Likert-scale items in the student questionnaire for tutor evaluation ranged from 0.59 to 0.96 (Pearson r = 0.052 and 0.005, respectively). In addition, the tutors’ performance on the Korean Medical Licensing Examination was not significantly correlated with the student perception of tutor effectiveness; the p values on the same eight items in the student questionnaire ranged from 0.12 to 0.97 (Pearson r = −0.150 and 0.003, respectively). Furthermore, the results of a one-way ANOVA showed no significant differences in tutor effectiveness perceived by students between the non-staff tutors who had completed their internship and those who did not on the eight Likert-scale items in the student questionnaire on tutor effectiveness; the p values ranged from 0.615 to 0.071.

Overall perceptions of tutors and students

Students expressed their views on the benefits and limitations of being tutored by non-staff tutors in response to the open-ended questions on the student satisfaction questionnaire. Students generally noted that they enjoyed the comfortable environment created by working with those who graduated from the same program recently. Also, several students were positive about the non-staff tutors’ group facilitation skills. For instance, a student commented that ‘[the non-staff tutors] had a good understanding of us and led our discussion from our standpoint. That's probably because they had experienced PBL as a medical student’. Additionally, several students remarked that working with non-staff tutors was beneficial to their learning because ‘the tutors gave me practical advice on how to learn in the PBL curriculum’. Still, some students pointed out that the non-staff tutors were not as effective as the staff tutors were in terms of their expertise in the subject matter. For instance, one student commented that ‘[the non-staff tutors] seemed to have a difficulty giving us detailed feedback on relevant clinical knowledge’.

The non-staff tutors were also asked to answer open-ended questions on the questionnaire. Generally, these tutors expressed that it was intrinsically rewarding to have an opportunity to return to their medical school and teach the students and to experience PBL from the teacher's perspective, even though they received minimal compensation, mostly for their travel expenses. Yet, they also expressed difficulties in facilitating the PBL tutorial process, especially in facilitating student discussions. Some of these non-staff tutors pointed out the lack of group facilitation skills as an inexperienced tutor. They noted that it was ‘difficult to know when to intervene in group discussions’ and they were ‘not sure how to get passive students to participate actively in the discussions’. Some also stated that they felt it was difficult for them to facilitate group discussions effectively because they lacked content expertise. One non-staff tutor stated that ‘I wished to jump in the group discussions more often to promote active discourse among the students, but that was difficult to do because I had limited content knowledge’.

Discussion

The overall results of the present study indicate that non-staff tutors, who were drawn from general physicians recently graduated from the medical school with a PBL curriculum, performed effectively in terms of student satisfaction and student performance on written exam scores. This finding is consistent with the results from other research studies comparing students’ written exam scores between PBL groups tutored by experts and non-experts or those led by staff tutors and non-staff tutors (Kwizera et al. 2001; Matthes et al. 2002; Park et al. 2007). However, the present study also found that the group evaluation scores, compared between groups led by staff tutors and non-staff tutors, were significantly different. The group evaluation scores were assessed by the unit chair not by the tutor who guided the group, thereby allowing objective assessment of the quality of the output of the PBL groups’ learning process (i.e. the case analysis report and case maps). This preliminary finding suggests the possibility that the backgrounds of the tutors and their experiences have an impact on the quality of the group's learning outcomes. Previous studies have focused primarily on the impact of the tutors on student performance on written exams. The results of the present study indicate that it is worthwhile to explore the impact of the PBL tutor's backgrounds and experiences on student outcomes in areas beyond written exam scores, such as clinical reasoning skills and problem-solving skills, which are believed to be facilitated effectively by PBL (Musal et al. 2003). Additionally, an in-depth interview study is recommended to have a better understanding of why such differences occur in the performance of groups led by staff tutors and non-staff tutors.

The students’ perception of the environment of the PBL tutorials led by the non-staff tutors was similar to the research findings on the effectiveness of peer tutors reported by Kassab et al. (2005). In particular, the fact that the non-staff tutors participated in the present study had gone through the same program as the students were in may have been conducive to create a rapport with the students. In addition, the present study also revealed both students’ and tutors’ perceptions that the non-staff tutors lacked group facilitation skills and expertise in the content. However, the findings of the present study indicate that such limitations do not significantly affect student achievement, at least in the areas assessed by written exams. Given the fact that they had received minimal PBL tutor training and none had been PBL tutor before, the non-staff tutors’ performance found from the present study is particularly notable. It can be speculated that these non-staff tutors were able to perform competently with minimal training in PBL due to their prior experiences in PBL as medical students.

It should be recognised that there is a limitation to this study. There were differences in the duration of time that staff tutors and non-staff tutors participated in this study. The non-staff tutors were working full-time at a distance from the school under investigation and had to take a leave to travel to the campus to participate in the tutorials; as a result, each tutor was able to take part for only 1 week, whereas the staff tutor was assigned to the group for the duration of the unit. As suggested from the study by Kassab et al. (2005), inexperienced tutors likely have a difficulty in facilitating groups, especially in their first week of tutoring. Thus, this could account for the differences in the groups’ performance in group evaluation. Additionally, the impact of the tutor's background on the students’ academic achievement could not be examined because of the short time allocation by the non-staff tutors. If they were able to tutor the groups for a longer period of time, it might have been possible to investigate the effect of the differences in the tutors’ background and credentials on the students’ learning outcomes. Such findings can lead to a better understanding of what credentials are entailed to recruit effective tutors.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article.

Additional information

Notes on contributors

Kyong-Jee Kim

KYONG-JEE KIM, PhD, is an assistant research professor of Medical Education, Sungkyunkwan University School of Medicine, Korea.

Joo Heung Lee

JOO HEUNG LEE, MD, PhD, is a professor of Dermatology at Samsung Medical Center, Sungkyunkwan University School of Medicine, Korea. He also oversees the school's PBL curriculum committee.

Changwon Kee

CHANGWON KEE, MD, PhD, is a professor of Ophthalmology at Samsung Medical Center, Sungkyunkwan University School of Medicine, Korea. He is also associate dean of education and director of medical education.