Peer review: An effective approach to cultivating lecturing virtuosity

Abstract

Background: Most university faculty members are expected to teach. Many would benefit from instruction designed to improve lecturing.

Aims: To explore the impact of a program in which video-recorded lectures were critiqued by peers.

Method: Sixteen lecturers participated in this qualitative study. Four agreed to have an undergraduate lecture video-recorded for peer review. Twelve participated in review sessions wherein the lecturer and three peers viewed and critiqued the recorded lecture. All discussions were recorded and transcribed for thematic analysis. Subsequently, semi-structured interviews were conducted with each lecturer and all 12 peer reviewers. Three pairs of research team members independently conducted thematic analyses of the discussion transcripts and the interviews; then all members met to develop consensus on major emergent themes.

Results: Six themes were identified: (1) the benefits of peer review; (2) the components of successful peer review; (3) the value of reflection on teaching experiences; (4) the inherent stress in peer evaluations; (5) the elements of successful lecturing; (6) lecturing as performance.

Conclusions: The benefits of peer assessment of lecturing (PAL) were enthusiastically endorsed by all 16 participants. The PAL program is now supported by the McGill Faculty Development Committee and plans to implement regular PAL sessions in place.

Introduction

Most faculty members in today's universities are expected to teach. One critical pedagogical skill entails the ability to effectively teach large groups of learners (Srinivasan et al. 2001). Many faculty members are intimidated by large group lectures and most would benefit from targeted instruction and feedback on their lecturing skills (Hatem et al. 2006). Although feedback on the success of lectures occasionally comes from fellow faculty members or department chairs, the most prevalent source of formal and informal feedback is students. Student assessments are easy to acquire and can be helpful in improving lecturer behaviors, but they can also be problematic. Often the feedback is insufficiently specific to direct and facilitate change. It is thus not surprising that student assessments have been regarded as “over-used and limited ways of evaluating lectures” (Brown & Manogue 2001). To add to the negative perceptions of student assessments of lecturers, analysis of the instruments used in such evaluations reveals that they are often poorly constructed and may be of questionable reliability and validity (Jefferson et al. 2002; Leamon et al. 2004).

An alternative to student evaluations, particularly for formative assessment, is peer review. Peer review of faculty performance, so essential to the evaluation of academic research productivity, has played a progressively significant role in clinical teaching (Brown & Ward-Griffin 1994; Jefferson et al. 2002; Ten Cate et al. 2007; DaRosa et al. 2011; Finn et al. 2011). Peer evaluation of faculty teaching is widely accepted in nursing schools, where occurs with increasing regularity (Brown & Ward-Griffin 1994; Bennet et al. 2011) and some Pharmacy Colleges have adopted peer review for classroom teaching as well (Wellein et al. 2009). Regrettably, in Medicine “there is currently no global level of professional peer-review of excellence in teaching” (Harden & Wilkinson 2011).

Table

Practice points

Peer review of lecturing can enhance confidence and lecturing virtuosity Peers are helpful and not judgmental Peer assessment can lead to an active learning community Lecturing skills are modifiable

Peer evaluation of lecturing has particular appeal when it occurs in the lecturer's work setting, enabling specific individualized feedback while fostering collaboration (Newman et al. 2009). Situating a peer evaluation intervention in a collegial partnership can help overcome anxieties related to being “in the spotlight” since peers speak the same language as, and know the culture of, their peers. Another putative advantage of peer observation of lecturing is that it could encourage reflection on the lecturing practices of the reviewers as well as those of the individual being reviewed.

To appraise the value of peer review of lecturing in a large group setting, we conducted a qualitative study designed to gauge the perceptions of both reviewers, and individuals reviewed, of the benefits and shortcomings of peer assessment.

Methods

Participant selection

We recruited a purposive non-random sample of 16 experienced Faculty of Medicine lecturers who regularly teach undergraduate medical students. All the subjects regularly lecture in the Basis of Medicine component of the McGill Medical school curriculum. Four of the 16 agreed to have one of their scheduled lectures video-recorded for use in a peer assessment study. The other 12 were recruited to serve as peer reviewers of the recorded lectures. We purposely recruited four different groups of peer lecture reviewers with a view of acquiring broad commentary on the lecturers’ positive behaviors and areas in need of improvement. We anticipated that the 12 reviewers would learn from the process and would be able to provide valuable comments on their experiences in the subsequent interviews.

The peer assessment of lectures intervention

We video recorded four one-hour lectures, each delivered by one of the four volunteers in the regular Basis of Medicine lecture series during the 2009–2010 academic year. Within three to six months of each recorded lecture, we convened four different peer assessment of lecturing (PAL) group meetings designed to view and critically appraise each of the four lecturers. The three to six months delay between the delivery of each of the four recorded lectures and each of the four peer review sessions reflected the difficulty of organizing video review meetings to fit the participants’ busy academic schedules.

In each 90-minute video viewing session, the lecturer and the three peers watched, discussed, and critiqued segments of the lecture and gave constructive feedback to the lecturer on issues such as timing, interaction with the audience, slide quality and quantity, and speech characteristics, but not on the content of the lecture. The video was allowed to run until the lecturer or one of the peer reviewers requested it be stopped for an observation, a comment, a critique, or a question. As a guide for the 90-minute critical assessment activity, each of the group participants used a recently developed and validated instrument for peer lecturer assessment (Newman et al. 2009). The first author in the study served as moderator of the four PAL groups while a research associate audio-recorded each of the four sessions for later transcription. There was no overlap in the membership of the four PAL groups. Three to 10 months later the research associate conducted and audio-recorded semi-structured interviews with each of the four video-recorded lecturers. We designed these lecturer interviews with a view to acquiring information on the impact of the PAL intervention. In particular, we wanted to determine whether the PAL experience led to any changes in lecturing practices or in attitudes, values, and beliefs about teaching.

The research associate also conducted interviews with the 12 peer evaluators using focus groups. The discussions in the focus groups addressed participants’ perceptions of the value of the PAL experience, particularly as it influenced their own teaching practices, and their perceptions of how the experience may have influenced the lecturers who had been video-recorded.

Subsequently, three separate pairs of research team members each independently conducted an inductive thematic analysis of the transcripts from (1) the 90-minute PAL sessions, (2) the semi-structured interviews with the recorded lecturers, and (3) the focus group members who viewed and discussed the videos in the PAL sessions. In each case we used a uniform process of emergent thematic analysis. The pairs of research team members read the transcripts individually, making marginal notes on recurring issues to develop preliminary codes. They reconvened to compare notes and revise as necessary. A saturation of the data was reached and each of the three pairs arrived at a consensus on the predominant themes identified. Field notes were used to complement the transcribed data. With a view to develop a consensus on the major emergent themes across the three data sets, all six analysts met to discuss the emergent findings. Following debate and discussion, the team members reached consensus on the major themes that recurred repeatedly.

Ethics approval: We received ethical approval of this study from the McGill University Faculty of Medicine Institutional Review Board.

Results

Emergent thematic analysis of the transcripts from the 90-minute PAL sessions, the focus groups, and the individual participant interviews yielded six prevalent themes.

Theme 1: The benefits of peer review

All participants acknowledged the value of discussions with their peers and appreciated the opportunity to meet and talk about common interests in teaching and current teaching experiences.

When you read a good book, you read it but … the joy of having read the book is to be able to discuss it … to get perspective from your peers

… there is value in your peers’ feedback … because little things that other people will pick up could significantly improve your performance and give … you some help

… it was a good experience … having another perspective and having someone make you look at your slides and some of the things you do a little more critically.

… better to be judged by your peers rather than somebody you perceive making a decision about … you

Peer reviewers also positioned themselves in relation to the video recorded lecturer and reflected on how they would feel if they were in the lecturer's position. Viewing a lecture of a peer teacher stimulated teachers to think about their own teaching techniques.

It makes you think 100 percent about how effective you are at what you actually do when you’re teaching

I think the value to some extent is … when you are evaluating somebody else you are self-evaluating, “Do I do that?” or “How do I do this?” “I think it is really healthy to do this periodically.

Theme 2: The components of successful peer review

Discussion of the elements of success of the peer review resulted in the emergence of five subthemes:

• Video viewing of the lecture

The video of the recorded lecture played a major role in facilitating the peer review process. It conveyed the reality and immediacy of the lecture. The peers opined that the lecturer might not have appreciated their feedback unless he/she had seen the behaviors under discussion. “Seeing is believing,” said one peer. One of the lecturers recognized how “seeing yourself in action” dramatically prods one to think about “lecturing styles.”

Another lecturer reported, “In my former life I was a synchronized swimmer and I remember when my coach brought in video equipment for the first time and the number of times she would tell us, ‘Your leg is not straight, your leg is not straight.’ And you are thinking, yes it is! And then to actually see it you say, ‘Oh it's not’. So I think that it is quite a useful thing to be able to see yourself as cringe worthy as it is. It is a useful exercise.”

• Non-threatening environment

The PAL experience “created a space” that allowed for relaxed collegiality and comfort with peers who previously did not know each other well.

I think it doesn’t happen when it is not an assigned space. We just don’t have time to do this … it only happens in this assigned space which is unfortunate. And we really don’t have time to go to each other's lectures …

This space, separate from the usual work area, facilitated participants’ receptiveness to peer feedback and reflection to change. The participant “buy in” and the environment that allowed for the discussion of teaching were somewhat analogous to the manner in which peer review committees adjudicate research proposals.

 … we don’t ever sit around and talk about each other's teaching the way that Doctor X [the moderator) says we talk about each other's research … I think there is real value in an expansion of something like this … it adds an element of informality that really kind of relaxes you enough to actually listen to the critique that you receive.

• Role of moderator

The moderator played a significant role in assuring peers’ participation and comfort level. Participants found that he made it easy for them to engage in the discussion by asking stimulating questions and inviting both positive and negative constructive comments. During the 90-minute video-viewing sessions, the moderator periodically stopped the video to allow discussion.

He was an excellent moderator … this worked because (he was) a moderator whom we respect as an educator.

• Receptivity of the lecturer

The lecturer's desire for feedback played a pivotal role in the peers’ ability to offer their opinions.

… the person you are going to watch and try to help has to want to be helped.

From his standpoint he must have understood the position we were in as well - that we were being asked to comment and to come up with some sort of constructive criticism and that we weren’t just spontaneously calling a meeting to criticize him …

• Value of the rigorously-developed lecture assessment instrument (Newman et al. 2009) for assessing the lecture and for subsequent personal use in considering one's own lecture performance.

  I kept the sheet [the lecture assessment instrument] that had the different categories and I asked myself how … I was doing on each one … I think it gave you what each level of performance would be … if you do this, you get this mark … so I tried to grade myself and it made me think about my position.

Theme 3: The value of reflection on teaching experience

Reflection on the PAL group members’ own teaching was stimulated by viewing the video and by engaging in the discussion of the lecture with peers. Self awareness of one's own teaching and lessons gleaned from the discussion and feedback stimulated self-reflection, and, for many participants, a desire to modify current lecturing techniques:

It is helpful to see yourself through the eyes of others.

For me the main impact this has had was pushing me to critically look at how I do the lectures and finally sitting down and revising some of the stuff

It makes you definitely reflect on your own style when you participate in this kind of presentation … or else we just go through the routine and we don’t think about it all the years … this kind of stirs you up and makes you more receptive to change

Some of the video reviewers’ own teaching techniques mirrored those of the video-recorded lecturer. Discussions on those techniques were related to why the reviewers do, or not do, the same things (“lecturing errors”) as the video-recorded lecturer. Commonly noted were observations such as: “I do that” and “His issues are not my issues”, which led to questions and discussion of “what are my issues?”

… it left me wondering if there was something I was doing that I could work on … it makes me think that my teaching is a living thing that needs to be tended and improved … cultivated always.

Theme 4: The inherent stress of a peer evaluation process

Engaging in peer evaluation in an open and frank manner can engender anxiety in even the most confident and competent teacher. Participants acknowledged the significant stress but most felt that the stress contributed positively to the experience.

Signing on (to participate in a peer lecturer assessment session) … the thought crosses your mind that it can be a little bit intimidating … you don’t know what is going to happen, but I think the benefits outweigh the discomfort that you feel

… these are things we don’t talk about often

What is the best way to say complicated things?

Theme 5: The elements of a successful lecture

The PAL process afforded review and recognition of the “errors” and the positive behaviors that participants identified in themselves and others. Several participants subsequently communicated that they had made changes in their lecturing and others indicated a commitment to change. All were left thinking more about their own teaching style. Several key elements of lecturing were discussed: PowerPoint use, body language, eye contact, humour, and “presence” were prominent in the discussions.

… it gave me an impetus to make changes … maybe the order of things, some of the slides, especially some of the slides that are overly complicated

… you were not looking at the audience very much … I also like to look around the room and make eye contact …. I get a sense that … more eye contact is essential

… we are looking at the slide and the slide is not correlating with what you are talking about at the time … 

I agree that animation keeps them more aware of the points that you are making as you are going through it, … yeah the body language, move around a little bit.

You grabbed their attention at the outset, You used humour well, I think your storytelling is wonderful … it's what grabs their attention and makes the content real

Theme 6: Lecturing as performance

Lecturing entails more than the transfer of information from the notes of the lecturer to the notes of the learner. The successful lecturer pays attention to all aspects of the performance, recognizing that optimal learning requires attention to issues such as environment, pace, enthusiasm, awareness of student attention, humor, and active learning.

… (viewing the lecture on video) opens your eyes to your own performance” … “There must be basic things that we probably all agree on … you know things such as giving an organized presentation … we all know what an organized presentation is … but aside from that, a lot of teaching to me is performing … you are acting … and there are certain things which work for you which wouldn’t work for me

… but there are nevertheless some key factors which I think ought to be considered such as organization and enthusiasm. If you are “blah” about the subject then why should the students care?

Discussion

Abraham Flexner's call for a reduction in medical school lectures (Flexner 1990) combined with Spence's lukewarm comments on lecturing, indicating that “the wholesale use of lecturing is certainly not justified”(Spence 1928) might have presaged a significant reduction in the quantity of medical school lectures in the last century. In the 1984 GPEP report (Panel on the General Professional Education of the Physician and College Preparation for Medicine 1984) the authors commented that “the educational yield from lecturers is generally low,” thus augmenting the negative perceptions of the value of lectures. Despite those ominous commentaries, the lecture “persists as a predominant mode of instruction in most health science organizations and medical schools.”

The staying power of lectures argues that they have merit as a teaching modality (Sweeney et al. 1984). A lecture can provide a guiding framework for students facing information overload, exposure to a new vocabulary, new database and complex biomedical material. If delivered enthusiastically, a lecture may also inspire curiosity and serve as a motivation for learning (Hansen & Roberts 1992). If faculty members responsible for lecturing to health care students can avoid the habitual pitfalls of transferring minutiae, the lecture format affords an excellent opportunity to reach large numbers of eager learners at one time. Lectures are economical in terms of instructor–student ratios and space requirements and, if prepared conscientiously, and delivered competently, can be powerful learning experiences.

Our study confirms that a safe collegial environment in which the participating lecturers are physically and emotionally comfortable with their peers can engender enthusiasm for learning and resolving for change (Knowles 1985). All our subjects expressed an eagerness to revisit the substance and performance aspects of their lectures and to incorporate new behaviors and approaches as a consequence of their participation in the PAL exercise.

Recording a live unscripted lecture in the natural environment of the curriculum, followed by animated discussion and reflection with peers, delivers powerful messages to all involved. One participant in our study recently stated that “Teaching is something so important that, to have faculty engaged on a regular basis analyzing it the way they analyze research, changes the culture. It sends a message to all faculty members that this is an important issue.”

Subsequent to completion of our study, widespread local dissemination of the results has generated enthusiasm among lecturers and faculty leaders and this has led to plans for incorporating PAL as a regular faculty development activity at our institution.

Limitations

Although a small sample, such as we used, may lead critics to question the validity of our results, we are of the opinion that, in a qualitative research study such as ours, reliability is not central to the success of the study. Peer evaluations from different sources designed to enhance lecturing success may be equally valid even if they are not highly correlated. In our opinion the critical measure of success of our effort will be to see how sustained is the impact of the collaborative effort among colleagues who trust and respect each other (Norcini 2003; Siddiqui et al. 2007; Bell & Mladenovic 2008).

Conclusions

Interactions between the individuals who participated in our study engendered an unpredicted result – the beginning coalescence of an informal community of learners. Eraut's (2004) learning model includes social and individual determinants of learning in the workplace that is achieved through interaction with fellow workers. Eraut emphasizes the idea that tacit learning occurs through the process of socialization with peers and/or colleagues. The collegial interactions that evolved during our study contributed to the enjoyment of learning and a dedication to improving lecturing skills. There is growing international support for learning communities among teachers. These communities promote valuable peer interactions and can contribute to improved student learning (van Es 2004).

Acknowledgments

We gratefully acknowledge the financial support for this project from the Royal College of Physicians and Surgeons of Canada.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of this article