Lessons learned from a study-group pilot program for medical students perceived to be ‘at risk’

Abstract

Background: At the University of New Mexico School of Medicine (UNM-SOM) we have noticed that some first year medical students have difficulty accurately assessing their academic skills and are often afraid to seek help. This leads to marginal performance and sometimes even failure. Therefore, we developed a preemptive intervention using peer-led study groups based on the personalized System of Instruction (PSI).

Aim: The goal of this pilot study was to evaluate this approach for assisting students, interms of student success, and cast benefit.

Methods: Thirteen first-year medical students considered to be ‘at risk’ of academic difficulty took part in a six-month pilot intervention. They participated in structured study groups that were facilitated by upper-level medical students. The groups met twice weekly for up to two hours each time. The at-risk students took short multiple-choice quizzes and discussed major concepts. If students did not achieve 80% or better on the quizzes, they were required to take a second quiz to demonstrate mastery. Summative exam scores from four groups of students were compared: those with Medical College Admission Test (MCAT) scores <25, who received the study group intervention; their classmates with MCAT scores >25 who did not receive the intervention; and two matched groups from the previous year, none of whom had access to the structured study groups.

Results: No significant differences in exam scores were seen between the group who received the intervention and the matched group who did not.

Conclusions: Despite this result, we learned several useful lessons about study groups and interactions between first-year and upper-level medical students: (1) Students perceived participation in the study groups as a good learning strategy, but preferred participation not be mandated. It may be preferable to train and encourage students to run their own study groups. (2) Both students and proctors acknowledged interpersonal benefits from the program but, as these benefits can be achieved by other means, an expensive proctor-based program is not, we believe, the best use of academic support resources. (3) Focus in the study groups was on content for the quizzes, but more attention to how-to-learn strategies may have had greater impact.

Introduction

In most incoming medical school classes there are students who will struggle with the demands and expectations of the curriculum. At the University of New Mexico School of Medicine (UNM-SOM), we have found that some of these students can be identified on the basis of lower-than-average Medical College Admission Test (MCAT) scores (a standardized, multiple-choice entrance exam required by most US medical colleges), or university science Grade Point Averages (GPAs). Despite the imperfection of such predictors, indicators that a student may need extra academic help should be addressed early, before the student has experienced failure. Based on our experience, the emotional and psychological consequences of failure become additional issues to deal with, and make focusing on learning content even more difficult.

Table

Practice points
•Students perceived participation in the study groups as a good learning strategy, but did not appreciate mandated group involvement, and felt it compromised their autonomy and confidence. We recommend offering training to all students on how to structure their own peer-led study groups if they wish to use this approach.
•Both students and proctors acknowledged interpersonal benefits from the program, but this was not the primary program goal. Since these benefits can be achieved by other means, an expensive proctor-based program is not, we believe, the best use of limited institutional resources for academic support.

We wanted to see if a proactive approach of ‘pre-remediation’—preventing failure before it occurs—was a preferable strategy to ‘remediation’—fixing something after it has already broken. We designed an early intervention learning support program that was built around study groups (Sobral2002; Blowers et al. 2003; Higgins 2004), and based in part on the Personalized System of Instruction (PSI) model of mastery learning (Keller 1968; Kulik et al. 1979; Ocorr & Osgood 2003). In the classic PSI model, students work their way through assigned material in a series of learning units. When the student feels confident in knowledge of that unit, he or she takes a unit quiz to demonstrate mastery of the material. More senior students in the same program act as teaching assistants, known as proctors, and administer and grade these quizzes. Failing the quiz requires that the student restudy and retake the quiz until mastery is reached.

We conducted a six-month trial of the study group program. The purpose of this communication is to describe the program, present the results, and offer our recommendations based on the lessons we learned.

Methods

Medical Education Apprenticeship Program (MEAP) study groups consisted of first-year medical students most at risk of experiencing academic difficulty, which we defined as those starting medical school with an MCAT composite score of 24 or below, and/or 7 or below on any one section of the test. We chose the MCAT score as our standard based on data collected by our evaluation office and others (Mitchell 1990; Julian 2005), which show a correlation between MCAT scores and academic performance in medical school including scores on the United States Medical Licensing Examination (USMLE), Step 1. Thirteen students, in two MEAP groups, were required to participate in the study groups for the length of the pilot program (August–February), which encompassed three subject blocks, each lasting 6–12 weeks.

In total, 37 second- to fourth-year medical students applied for employment as study group leaders (‘proctors’). Eligibility criteria included a grade of Good or Outstanding in the relevant subject area, and previous teaching experience. Seventeen of these students completed a half-day training session which covered: (1) the PSI model of mastery learning; (2) relevant learning styles and strategies; (3) communication skills; (4) program logistics. Seven were chosen as proctors based on schedules and availability. They received compensation of US$20/hour.

The study groups met twice each week for two hours. Each session began with the at-risk students taking a multiple-choice quiz, the results of which were used to structure further content discussion. The quizzes were written by proctors, and were derived from the faculty-created formative quizzes available to all first-year students. Group sessions also offered opportunities to discuss study techniques, time management and test-taking strategies.

The pilot program was assessed for academic effectiveness on the basis of student summative exam scores in each of the three subject blocks. Four groups of students were compared. Students who participated in the MEAP study groups (n = 13) were compared with a same-sized group of medical students from the previous year who had comparable incoming MCAT scores but who did not have access to structured, proctor-led study groups (the pre-MEAP class). All students in these two groups had total MCAT scores <25; the MCAT average for the MEAP students was 22, and the MCAT average for the pre-MEAP students was 21.4. The other groups were composed of the remainder of each class: all students in these groups (n = 61 for remainder of MEAP class, n = 63 for remainder of pre-MEAP class) had MCAT scores of 25 or greater, with the students in the MEAP class averaging 28.6 and the students in the pre-MEAP class averaging 29.3.

The mean final grades for each of the four groups in the first three subject blocks were compared to determine whether the MEAP study-group participants showed an improvement in exam performance relative to their matched group, and in relationship to their own classmates. Comparisons were done using analysis of variance. A p-value of < 0.05 was considered significant.

One of our proctors (CN) designed his required undergraduate medical school research project around surveying students and proctors for their views on the effectiveness of this program. Questionnaires were administered 1.5 months after the pilot study ended.

Results

A synopsis of general program problems and their solutions is presented in Table 1.

Table 1.  Problems and solutions

Problem	Solution	Question remaining
Student negative attitudes about mandatory participation in the study groups threatened to compromise student motivation to learn	Having students lead the study group helped with motivation	Should early intervention be mandatory? Will students who need extra help seek it out?
Evolution occurred within each group as the group settled. Each group became different, creating a question of quality control	Frequent meetings of the program coordinators with proctors were instituted to standardize group offerings
Students were not coming to study groups, or were attending but were not prepared	Proctors took attendance. Students were reminded by proctors to come prepared and why this was important for them and for the group to function well. Some students were contacted by program coordinators and/or dean	Should attendance be mandatory?
Students were sometimes not concerned with deep learning but with just getting a passing grade on the quizzes	Proctors worked to get students to worry less about grades and more about learning
There was a lack of integration of the study group offerings at the subject block level, and institutional curriculum level	Coordinators took turns attending undergraduate medical education committee meetings to discuss & inform faculty on program's progress/process/expectations
Proctors spent too much time preparing for their study groups, sometimes interfering with their own studies	Proctors divided up the chores of study group facilitation and quiz writing. Proctors needed to accept that they were not the teachers and learn to encourage the growth of peer-teaching in the group
Study group quiz generation from the formative quizzes was sometimes difficult because of the uneven quality of these formatives and the large number of topics covered each week. Also, the proctors were not content experts, and the proctor-written quizzes were not checked by faculty for accuracy	Proctors laboriously rewrote and tried to check the accuracy of their quizzes, and also wrote comprehensive answers to their quizzes. They made it their goal that there was ‘Step 1 relevance’ to all of their quizzes, because many of them were studying for USMLE Step 1 at that time	Are proctors the best choice to write quizzes? Could faculty be recruited for this task?
Quiz generation began as solely a proctor responsibility, which resulted in proctor learning but perhaps less student learning	Students started writing their own group quizzes by the final month of the pilot study
Faculty and staff time commitment was sometimes very high	Program coordinators divided up the tasks and whoever was available did the troubleshooting

The comparison of summative exam scores between groups is given in Table 2. No significant differences in scores were seen between the MEAP students when compared with the matched pre-MEAP students. There were also no significant differences between the matched groups with higher MCAT scores between years. These results show that the study-group intervention was not successful in its objective of improving the at-risk students’ academic performance.

Table 2.  Student final exam scores

	Pre-MEAP Class				MEAP Class
	MCAT Total		Difference in means	Effect size	MCAT Total		Difference in means	Effect size
Subject block	<25 (n = 13)	> = 25 (n = 63)			<25 (n = 13)	> = 25 (n = 61)
Human Structure Function and Development	78.4	83.0	4.6	0.7	77.3	82.8	5.5	0.7
Mechanisms of Disease	79.2	84.3	5.1	1.0	77.1	84.2	7.1	1.2
Neurosciences	79.3	83.2	3.9	0.7	75.2	83.5	8.4	1.1

Student questionnaires had a total response rate of 69%. The survey questions and average response scores are given in Table 3. Neither the students nor the proctors were aware at the time of their questionnaire responses that no academic benefit had been seen for the study-group participants. Self-selection into study groups was preferred over mandated participation. Students appreciated their proctors’ efforts, and felt the study groups did more to help them with adjustment to medical school than they did to help them grade-wise (correctly, as it turned out). Many of the participants expressed the opinion that all first-year students should have access to such groups, or at least have information on how to structure groups on their own.

Table 3.  Student response to questionnaires about participation in study groups

Question	Average
1. My initial reaction to the invitation to join MEAP/MSG was positive	2.73
2. The Personalized System of Instruction was effective in my group	4
3. My proctor worked well with my group	4.59
4. I feel I improved academically because of my involvement with the study group	3.36
5. I feel that contact with the upper-class proctor helped my adjustment to medical school	3.73
6. I would join another, similar group if it was offered	3.62

Notes: Response % = 69%. Averages were calculated by assigning 0–5 for each response: ‘strongly disagree’ (=0), ‘disagree’ (=1), ‘slightly disagree’ (=2), ‘slightly agree’ (=3), ‘agree’ (=4), ‘strongly agree’ (=5), thus an average of 4 would correspond with ‘agree’.

All seven of the proctors responded to the questionnaires (data not shown). In general, the proctors at least agreed (see scale described in Table 3) that PSI was an effective learning method in their groups and believed that their students benefited academically and in terms of adjustment to medical school by their participation. They expressed concern that the program was not available to every student, and would have preferred that participation was voluntary rather than required. They agreed that their goals for proctoring (opportunity to study for the boards, to earn money, satisfaction from teaching, desire to become involved in academic medicine) were fulfilled, and reported that they would act as a proctor again.

Discussion and conclusions

The main finding of this pilot study was that participation in structured, proctor-led study groups was not effective in improving the summative exam performance of medical students perceived to be at risk. Despite the lack of decisive academic results, there were student- and proctor-perceived benefits of the MEAP intervention. Proctors were often able to provide useful practical information and suggestions for adjustment to the strains of medical school.

Study limitations to note include the small sample size, the short duration for the pilot project, and incomplete data because the student participants have not yet taken USMLE, Step 1, nor do we yet have information on graduation rates for our participants.

Besides the study limitations mentioned above, possible reasons why we did not see a difference in outcome include: the focus in the study groups was on content for the quizzes, not on how to learn; time was spent on only a few topics each week due to time constraints; perhaps MEAP students did not prepare enough to benefit from the quizzes; the proctors may not have been sufficiently expert in either the content or in teaching; the PSI model itself was not appropriate in this context; study-group work was not sufficiently integrated with the curriculum; or, more direct faculty input was needed.

Despite our neutral result in terms of exam scores, there were positive aspects of the intervention that were not entirely anticipated. Proctors enjoyed the teaching experience and stated that it helped them to study for their boards more effectively. First-year students who were not in the structured study groups requested that they be allowed to participate, especially because they wanted access to the quizzes. And, mentoring between freshman and more senior students proved to be supportive to both groups of students. We continue to explore an optimal structure for medical student learning, through both independent activities and with structured study groups.

Notes on contributors

All authors are at the University of New Mexico School of Medicine.

PAMELA DEVOE, MA, is Program Manager, Hispanic Center of Excellence.

CHRISTOPHER NILES, ALANA BENJAMIN and ANDREW BRAINARD are third-year medical students.

NICHOLAS ANDREWS and ELIZABETH COLOMBO are MD/PhD students.

BENJAMIN DUDLEY is a fourth-year medical student.

LISA BLACKLOCK is a Resident in Diagnostic Radiology/Medicine.

CHERI KOINIS, MA, MEd, is Director, Student Learning Support.

MARCY OSGOOD, PhD, is an Assistant Professor in the Department of Biochemistry and Molecular Biology.

Acknowledgements

The authors would like to thank Jan Mines and Betty Skipper, PhD for their help with the statistical analyses. They would also like to thank Summers Kalishman, PhD for suggestions regarding the questionnaires.