Effect of using an audience response system on learning environment, motivation and long-term retention, during case-discussions in a large group of undergraduate veterinary clinical pharmacology students

Abstract

Background: Teaching methods that provide an opportunity for individual engagement and focussed feedback are required to create an active learning environment for case-based teaching in large groups.

Aims: A prospective observational controlled study was conducted to evaluate whether the use of an audience response system (ARS) would promote an active learning environment during case-based discussions in large groups, have an impact on student motivation and improve long-term retention.

Methods: Group A (N = 83) participated in large group case discussions where student participation was voluntary, while for group B (N = 86) an ARS was used. Data collection methods included student and teacher surveys, student focus group interviews, independent observations and 1-year post-course testing.

Results: Results indicated that the use of an ARS provided an active learning environment during case-based discussions in large groups by favouring engagement, observation and critical reflection and by increasing student and teacher motivation. Although final exam results were significantly improved in group B, long-term retention was not significantly different between groups.

Conclusions: It was concluded that ARS use significantly improved the learning experience associated with case-based discussions in a large group of undergraduate students.

Introduction

There is evidence to support that active learning approaches are effective in improving learning outcomes in medical education (Michael 2006; Graffam 2007). Among the various strategies proposed to promote active learning, case-based teaching is considered to be effective in stimulating students to synthesize, apply and integrate basic knowledge in the face of real-life situations and has been recommended for a variety of clinical subjects especially in the pre-clinical training years (Novotny 1998; Michel et al. 2002; Monahan & Yew 2002).

Classically, case-based teaching scenarios have been described for small groups of students, allowing for interaction, focussed feedback and student engagement, which are key elements for success. While small group teaching is relatively accessible during clinical rotations, it is rarely feasible during the pre-clinical years in most curricula because of time, space and manpower constraints. Despite good intentions, most teachers will find that case-based discussions in a large group setting often leads to disappointing outcomes such as poor overall student participation and interaction as well as difficulties in giving focussed and pertinent feedback, all of which lead to reduced motivation and loss of active learning benefits (Gibbs & Jenkins 1992; Bruneau & Langevin 2003). When teaching to larger groups, specific tools are therefore needed to allow each student to take an active part in the discussion and to facilitate adequate and relevant feedback from the teacher (Acharya 2001; Gwee & Hoon 2001). Furthermore, since motivation is considered a key component of learning, teaching methods that aim to improve or maintain motivation among students and teachers would also be considered beneficial (Chan 2004).

Audience response systems (ARS) have been designed to electronically poll large groups of students using individual hand-held transmitters and to immediately report results in graphic form (Caldwell 2007). Thanks to the advent of more user-friendly wireless technology, these systems have gained popularity since the mid-1990s in many fields of education including health sciences where they have been used for various pedagogical activities (Miller et al. 2003; Eggert et al. 2004) and in a variety of health sciences (Uhari et al. 2003; Schackow et al. 2004; Slain et al. 2004; Johnson 2005; Latessa & Mouw 2005; Molgaard 2005; Pradhan et al. 2005; Plant 2007; Nayak & Erinjeri 2008). Many different systems are commercially available which share the ability to allow users to provide immediate feedback, and enhance the student involvement in large group lectures (Barber & Njus 2007). Many reports have been published on the use of ARS in various teaching situations. Although results concerning the impact on short- or long-term retention have been contradictory (Schackow et al. 2004; Slain et al. 2004; Pradhan et al. 2005; Plant 2007; Nayak & Erinjeri 2008), the majority of authors have found that student and teacher perceptions of this new technology are favourable (Slain et al. 2004; Johnson 2005; Molgaard 2005; Caldwell 2007; Plant 2007; Nayak & Erinjeri 2008).

Some educators have specifically described the use of ARS to facilitate case-based discussions in groups ranging from 20 to 100 students (Slain et al. 2004; Plant 2007; Nayak & Erinjeri 2008); however, these reports were based on limited numbers of participants or on student perceptions only. The purpose of this study was to evaluate whether the use of an ARS would: (1) promote an active learning environment during case-based discussions in large groups; (2) have an impact on student motivation, and as a consequence, (3) improve long-term retention.

Table

Practice points

ARS use provides an active learning environment for case-based discussions in large groups ARS increases student motivation and peer learning ARS modifies teaching approaches and attitudes towards lecturing ARS is associated with a greater investment of time and effort on behalf of teachers and students

Methods

Study subjects and context

A prospective observational controlled study was conducted on a group of undergraduate veterinary students enrolled in a compulsory veterinary clinical pharmacology course. The course was divided into three major theme blocks of 5 weeks duration each. At the end of each lecture block, a 2 h case discussion and review session was presented. Case scenarios were provided to all students 1 week in advance of the scheduled discussion sessions and students were encouraged to prepare these questions in advance alone or in groups. No marks were given for this activity; hence there is no impact on the final results of the course.

Two cohorts of students participated in the study. Group A consisted of 83 students enrolled in the winter of 2004 and Group B consisted of 86 students enrolled in the winter of 2005. Identical course outlines were used for both groups including the timing of case discussion sessions for which identical cases and questions were used. The same instructor (MD) taught both courses and used the same lecture materials (notes and slides) for each group. Only the format of the discussion sessions varied. Group A participated in large group case discussions in which open-ended questions (e.g. Which antibiotic would you prescribe?) were asked to the entire group and student participation was voluntary. For Group B, an ARS was used to poll students’ individual responses and the open-ended questions were transformed into multiple-choice questions (e.g. Among the following, which antibiotic would you prescribe?).

Group B also was presented with a series of 25 review questions designed to assess comprehension of previous subject matter in relation to the cases provided. All review questions were of multiple-choice format.

The ARS system (Interwrite PRS infrared, eInstruction, Denton, TX) used in this study was set up with two infra-red receivers for a maximum capacity of 90 clickers. Questions were incorporated in Powerpoint (Microsoft Powerpoint 2003, Microsoft Corp., Redmond, WA) and results were automatically recorded in CSV format files for further analysis using Excel (Microsoft Office Excel 2003, Microsoft Corp., Redmond, WA).

Student surveys

Approximately 45 days after the last case-based discussion session of the semester, students in both groups were asked to voluntarily complete an anonymous on-line survey (WebCT, Blackboard Inc., Washington DC) of 36 and 40 questions, respectively, for group A and group B, designed to capture their perceptions of the case-discussion sessions, the ARS system in general (group B only) and more specifically the level of interaction, case preparation strategies, general study habits, perceived learning outcomes, advantages and disadvantages and on their levels of confidence regarding the subject matter of the course.

Student focus groups

One month after the last case-based discussion session, 30 students representing a random sub-sample of 35% of group B were invited to participate in one focus group interview each designed to collect further insight on their perceptions of the use of the ARS in general. Hence, two focus group interviews involving 15 students each were conducted. Two investigators (DH and AV) facilitated the focus group sessions using a standard set of predetermined questions.

Independent observers

The principal investigator (MD) taught all case-based discussions while the other investigators (AV and DH) each attended one case-based discussion session for group B in order to complete a standardized observation form to evaluate technical issues and logistics, student–student interactions and student–teacher interactions.

Post-test (1 year)

Students in both groups were asked to participate in a review examination, composed of standard multiple-choice questions and conducted using the ARS, 1 year after the end of the course (spring 2004 for group A; spring 2005 for group B). A random sample of 50 questions taken from the 75 questions originally used for group B during the review sessions were used for both groups.

Statistical analysis

Two-sided paired T-tests, assuming unequal variances, were used to compare pre- versus post-test results for group B and for comparing post-test and final examination results as well as cumulative grades for the semester between group A and group B. The Cochran–Mantel–Haenszel statistic was used to compare categorical survey responses and population demographics data between groups. All statistical analyses were performed with an SAS version 9.1 (SAS Institute Inc., Cary, NC) using a significance level of p < 0.05.

Results

Student population demographics and academic performance levels

Group A comprised 59 female and 24 male students (female to male ratio of 2.5 : 1) with a mean age of 25.2 years ± 3.9 (S.D.) (range 21–41 years) while Group B was composed of 71 female and 15 male students (female to male ratio of 4.7: 1) with a mean age of 24.2 years ± 3.0 (S.D.) (range 22–37 years). Age distribution was not significantly different between groups (p = 0.08); however, the female to male ratio in group B was nearly double that in group A. There was no significant difference between overall grades for the semester during which the study was conducted for group A (mean 85.01% ± 8.16) compared to group B (mean 85.74% ± 7.55) (p = 0.28).

Participation

Attendance was over 90% for all case-based discussion sessions throughout the study and for both groups. Response rates for the 1 year post-test were 59% (49/83) for group A and 45.3% (39/86) for group B. Student survey response rates were 59.0% (49/83) for group A and 87.2% (75/86) for group B.

Teaching strategies

Specific teaching strategies were adapted for the use of the ARS during case discussions. When the majority of students responded correctly to questions with only one valid answer, there was less discussion needed than when there was a greater proportion of wrong answers, indicating that certain issues required clarification or further interpretation. Questions with several valid or invalid answer choices were deemed to be more useful to evaluate student's comprehension and ability to apply or integrate knowledge and to generate participation and interaction according to the course instructor (Figure 1).

Figure 1. Examples of question formats used as teaching strategies during case-based discussions in large groups using the ARS. Compiled responses are presented as a percentage of all students per answer choice. Previously identified correct answer results (blue and green striped bars) appear differently from other results (green bars). Students may optionally indicate their degree of confidence with their response which can be high (red) or low (yellow). A: Question with only one valid answer among 6 possible choices, the last of which is ‘I don’t know’ and with the majority of students having responded correctly. B: Question with only one valid answer among 6 possible choices, the last of which is ‘I don’t know’ and for which there is a greater distribution of students having responded incorrectly or having indicated that they did not know the correct answer. C: Question with several valid answers among 6 possible choices, the last of which is ‘I don’t know’ and for which there is a large distribution in student responses.

Student surveys

Prior experience with ARS was reported to be ‘none’ for 100% of respondents in group A and 98.7% of respondents in group B.

The perceived usefulness of the clinical pharmacology course for their successful completion of the DVM program was significantly greater for students in group B than for students in group A (p = 0.02) (Table 1).

Table 1.  The perceived usefulness of the clinical pharmacology course for their successful completion of the DVM programme for students having attended case-based discussions in large groups without the ARS (group A) and those with the ARS (group B)

Group	I completely disagree	I disagree	I agree	I completely agree
A (N = 49)	2	0	15	32
B (N = 76)	2	0	7	67

Notes: In general, I believe that the knowledge and skills learned in the clinical pharmacology course will be useful for the successful completion of my DVM degree.

Group B responses are significantly different from group A responses at p < 0.05.

Although the level of agreement was high in both groups, students having used the ARS (group B) agreed to a significantly greater degree with the statement that the case-based discussions were fun (p = 0.03), allowed them to auto-evaluate their knowledge (p = 0.03), improved their ability to retain the subject matter (p = 0.03) and were too long (p < 0.001) than those that did not use the ARS (group A) (Figure 2). There was no significant difference between the two groups for the perception that case-based discussions in large groups were interactive (p = 0.58), allowed them to auto-evaluate their comprehension of the subject (p = 0.83), improved their ability to understand the subject (p = 0.48), were too difficult (p = 0.21), needed more explanations (p = 0.37), were not pertinent (p = 0.63) or were not practical (p = 0.15) (Figure 3).

Figure 2. Students’ perceptions of the positive aspects of case-based discussions in large groups without ARS (group A – white) and with ARS (group B – black). Results presented as mean scores for all responses in each group. Question: I liked the case-based discussions in general because: *Group B responses are significantly different from group A responses at p < 0.05.

Figure 3. Students’ perceptions of the negative aspects of case-based discussions in large groups without ARS (group A-white) and with ARS (group B- black). Results presented as mean scores for all responses in each group. Question: I disliked the case-based discussions in general because: *Group B responses are significantly different from group A responses at p < 0.05.

When asked to specifically choose the single most useful aspect of case-based discussions, students in group A associated their experience with different types of learning activities than those in group B (p = 0.04) (Table 2).

Table 2.  Perceived usefulness of case-based discussions for students having attended case-based discussions in large groups without the ARS (group A) and those with the ARS (group B)

Group	Retain the subject	Understand the subject	Synthesize and apply the subject	Identify my personal strengths and weaknesses	Were not useful
A (N = 49)	0	8	37	3	1
B (N = 76)	2	21	39	13	1

Notes: ‘The case-based discussions were most useful to me to…’

Group B responses are significantly different from group A responses at p < 0.05.

Students’ perceptions of their own competence in various areas of therapeutic decision making or clinical application were significantly greater for group B than for group A for all topics except choosing the appropriate anti-inflammatory drugs and choosing the appropriate drugs specific to a given body system (Table 3).

Table 3.  Students’ short-term (45 days) perceptions of their competence in various skills associated with the teaching objectives of the clinical pharmacology course for those having had case-based discussions without ARS (group A) and with ARS (group B)

	Mean score
Skill	Group A	Group B	P-value
Rational drug use and dosage adjustments	2.57	2.91	0.0048
Appropriate drug use in relation to public health concerns	3.02	3.33	0.0055
Appropriate use of antimicrobial drugs	2.12	3.10	<0.00001
Appropriate use of anti-inflammatory drugs	3.10	3.18	0.39
Appropriate use of antiparasitic drugs	2.69	3.01	0.013
Appropriate use of drugs specific to a given body system	2.73	2.96	0.053

Note: Results are presented as mean scores for all responses in each group.

Concerning their study habits, group B students reported a significantly greater number of hours spent reviewing their class notes during the semester in general (p = 0.01), reviewing class notes prior to case-discussion sessions in particular (p = 0.002) and studying for the final exam (p = 0.02) compared to those in group A. However, there was no significant difference in the number of hours dedicated to preparing cases for case-based discussions between the two groups (p = 0.10).

There was a significant difference between the students’ reported frequency of preparing cases alone (p = 0.02) and preparing cases in small groups (p = 0.005) prior to case-based discussion sessions, with group B having a greater tendency to prepare cases with colleagues than group A, although mean scores for preparation with colleagues remained low overall for both groups (Figure 4).

Figure 4. Students’ reported frequency of preparing cases alone (a) and preparing cases with colleagues (b) for students having had case-based discussions without ARS (group A- white) and with ARS (group B-black). Results presented as percent respondents for each frequency. *Group B responses significantly different from group A responses at p < 0.05.

Responses to open-ended questions in the surveys revealed that the three most frequently stated advantages of ARS for group B were: fun or interesting, incentive to stay up-to-date on subject matter and better retention, while the three most frequently stated disadvantages of ARS were: technical problems related to equipment, not having enough time to prepare for the case discussions and too much added workload compared to other courses.

Focus groups

Contrary to the survey results, the majority of students (80%) participating in the focus group sessions indicated that they preferred preparing for the case discussions with other classmates rather than alone and some indicated that this was a change from their usual working habits associated with the fact that the ARS approach was forcing them to commit to a response in class. Students also felt that the case discussions (with the ARS) were likely to be helpful in achieving a better retention of the subject matter.

Nearly 100% of students indicated that the ability to evaluate their strengths and weaknesses was the single most useful advantage of using the ARS. Approximately 75% of students indicated that the ARS allowed them to overcome the fear of participating (giving their opinion) in a large group due to the anonymity of the approach. Other advantages mentioned by the students included: focussed feedback to the entire class at once, dynamic interactions between students before and after entering their responses. Many (∼60%) students also mentioned that the main drawback of the ARS was the response time (delay when too many students try to respond at once) related to the limitations of the receptors.

Independent observations

Observers agreed that setting up and initiating the sessions with the ARS system was fast and efficacious. They also observed that students appeared at ease with the clickers from the beginning and that little stress was associated with the technology. The most common technical problem observed was the inability of the two receptors in the room to capture all signals at once; hence some students were required to make several attempts before successfully entering their response.

Observations pertaining to interaction revealed that students were generally very concentrated when questions first appeared on the screen and that student–student interaction was minimal for review questions but became increasingly apparent for case discussion questions. Spontaneous consultation among groups of two to three students was observed either prior to or after individual responses were recorded (10–30% at first and eventually close to 75% at the end). Student–teacher interactions were observed to be more frequently initiated when there was a greater distribution of responses or when a large number of responses were incorrect.

Concerning participation, generally 95–100% students present were seen to respond to each question. Students were observed to be more involved and interested in the class than in standard lectures with at least 80% of students generally concentrating hard until the end of the 2 h sessions.

Final examination results and long-term retention

Final examination results for group B (92.2% ± 5.4) were significantly greater than for group A (89.0% ± 11.9) (p = 0.03). However, the overall mean number of correct responses was significantly decreased from the in-class questions score of 61.7% ± 11.0 (mean ± SD) to the 1-year post-test test results of 51.3% ± 10.0 (mean ± SD) (p < 0.0001) for group B. Furthermore, there was no significant difference between 1 year post-test mean scores for group A (51.0% ± 8.5) compared to group B (51.3 % ± 10.0) (p = 0.89).

Discussion

This study provides evidence that the use of an ARS provides an active learning environment during case-based discussions in large groups of veterinary pharmacology students and has an impact on student and teacher motivation.

While other studies in health sciences have shown similar findings based on student and teacher perceptions (Uhari et al. 2003; Johnson 2005; Latessa & Mouw 2005; Molgaard 2005) or specific educational outcomes such as improved scores on final exams or short-term recall (Schackow et al. 2004; Slain et al. 2004; Pradhan et al. 2005), only a handful of reports have described the use of an ARS in large groups, defined as comprising 50–200 students in general (Bruneau & Langevin 2003). Of these, two studies considered survey results on general perceptions alone (Johnson 2005; Molgaard 2005), while only one measured specific outcomes showing that undergraduate pharmacy students having used the ARS during lectures in large groups, but not case-based discussions, performed better in the final examinations (Slain et al. 2004).

Active learning has been described as the process of having students engage in an activity that forces them to reflect upon ideas and how they are using those ideas (Michael 2006). The principal and inter-related components of active learning include engagement, observation and critical reflection (Graffam 2007). In fact, the obligation to commit or engage to an individual response when using an ARS is thought to be one of its main benefits for improved learning (Chan 2004). As has been seen in other studies (Uhari et al. 2003; Slain et al. 2004; Molgaard 2005; Pradhan et al. 2005; Nayak & Erinjeri 2008), students using the ARS in the current study were observed to be more active and attentive throughout the case discussion sessions with the participation rate exceeding 90% of the class throughout the 2 h sessions compared to traditional large group discussions where only a handful of more outspoken students usually participated. Other ARS features can further improve participation, as shown in this study, where students indicated that the anonymity of responses allowed them to overcome the ‘fear’ of actively participating in class discussions or as others have shown, the fact that results from all respondents are made available immediately allows students to engage in the discussion of controversial issues or to ask questions more confidently when they know that others share similar opinions (Slain et al. 2004; Molgaard 2005; Nayak & Erinjeri 2008).

Using an ARS during case-based discussions in large groups also favours observation and critical reflection. In fact, although the majority of students found that case-based discussions were especially useful to help them synthesize and apply the course material regardless of ARS use, a significantly greater proportion of students who used the ARS in these sessions viewed the ability to identify their strengths and weaknesses and to better understand the course material as important benefits as well. Other studies have demonstrated similar perceptions that the use of ARS in educational activities enhances the ability to auto-evaluate and critically reflect on one's abilities (Slain et al.2004; Nayak & Erinjeri 2008). The use of an ARS in large groups also provides an opportunity for critical reflection with respect to the unique student–teacher interactions that are generated. In this study, as in others, the beneficial aspects of ARS perceived by the teacher were the ability to gauge students’ level of knowledge and abilities (Slain et al. 2004) and the resulting focussed and timely feedback that can then be provided (Uhari et al. 2003; Slain et al. 2004; Molgaard 2005). Also, the teacher involved in this study, like in others, found that the use of ARS not only made lecturing more comfortable or enjoyable (Caldwell 2007; Nayak & Erinjeri 2008), but also more importantly changed their teaching approach. In some reports, students felt that instructors who used the ARS seemed to be more aware of their needs and hence to have a more immediate and caring teaching style (Uhari et al. 2003; Caldwell 2007). Others have reported that the use of ARS allows teachers to use class time more efficaciously by spending more time explaining concepts that appear to be misunderstood by the majority of students rather than by those that ask questions individually (Slain et al. 2004). Finally, when teachers encourage students to participate more actively in class, students are more motivated to learn (Acharya 2001), which is another well-established factor in effective learning (Chan 2004).

Intrinsic motivation in adult learners is ensured when students are able to develop a feeling of competence or self-confidence, become curious or seek situations that challenge their abilities and feel that they are in control of their learning (Chan 2004). Survey results from this study demonstrated that the use of an ARS during case-based discussions significantly improved the students’ perceptions of their own competence in their ability to adjust drug dosages for rational drug use and to make therapeutic decisions related to public health concerns (such as antimicrobial resistance in animals and in humans or drug residues in food of animal origin), antimicrobial use and antiparasitic strategies, all of which often require the integration of knowledge from other disciplines and the consideration of case-specific factors. The greatest difference was seen in the area of antimicrobial use, involving a particularly complex and multifactorial decision path, where students having used the ARS appeared to be very self-confident, whereas those who had participated in traditional in-class discussions rated their competence in this particular skill as lowest of all. There are several possible explanations for these findings. First, self-confidence could have been enhanced in students who used the ARS since they were able to gauge their abilities in comparison with their colleagues. Also, as described above, the ARS provided the opportunity to present questions that challenged the students’ ability to integrate multiple factors before committing to a particular answer and then to understand, with the ensuing discussion and feedback, how there can be more than one valid response depending on the circumstances of each case. While such questions are frequent in clinical pharmacology, the benefits of the ARS in such situations can easily be transposed to other areas of health sciences requiring complex decisions and critical thinking, such as surgery, diagnostic testing or prognostic predictions among others.

The feeling that one's learning activities are purposeful and rewarding is another driving force for motivation (Miller 1990). Students who used the ARS also perceived the clinical pharmacology course as useful for the completion of their undergraduate degree to a greater degree than those that did not, although both groups had a high perception overall. Furthermore, as has been reported in numerous other investigations (Slain et al. 2004; Johnson 2005; Latessa & Mouw 2005; Molgaard 2005; Plant 2007; Caldwell 2007), the use of an ARS motivates students since they are considered to be fun, new and a welcome change of pace compared to traditional lectures. While case-based discussions in large groups were well appreciated by most students, those who used the ARS perceived them to be significantly more fun and interesting and indicated that their motivation for this activity was enhanced.

The ARS also increased student motivation by allowing them to take control of their learning and to benefit from greater peer interaction than with case-based discussions alone. It has been reported that individuals are likely to learn more when they are learning with others than when they learn alone (Michael 2006). In this study, students who used the ARS reported having prepared for the case-based discussions with other colleagues in small groups prior to the in-class sessions significantly more often than those who did not use the ARS.

Peer interaction also appeared to be enhanced with the ARS based on the observations that students spontaneously consulted with each other to a large extent during the sessions in class either prior to or after entering their individual responses to a given question. This interaction was also more obvious for case-related questions than for the more theoretical review questions.

Students in the ARS group appeared to have different study habits than those who did not use ARS with regards to the number of hours spent reading their class notes or preparing for the final examination, although these differences may have been solely due to differences in class composition since the ARS group's female to male ratio was nearly double that of the other ARS group. Furthermore, as has been reported in another study (Slain et al. 2004), there was no significant difference in the time students spent preparing the specific cases prior to in-class discussion between the two groups with the majority of students spending 5 h or less preparing for the three 2 h discussion sessions in the semester.

The results of this study could not confirm a positive effect of ARS use on long-term retention. At first glance, this finding does not support the hypothesis that if the ARS promotes an active learning environment and increases student motivation, then learning outcomes should presumably be improved (Graffam 2007). However, a review of the literature indicates that the unbiased measurement of long-term retention in medical education research remains a challenge. In a study performed with a small group of family-medicine residents, investigators found that interactive lectures improved learning outcomes similarly with and without an ARS, compared to traditional lectures although it was speculated that their results could have shown greater benefits from the ARS if they transposed their study to larger groups of students (Schackow et al. 2004). In another study, students who experienced increased interaction either with or without an ARS during lectures had significantly better results on post-lecture recall and 1 month post-lecture retention than those having attended traditional lectures, although the ARS itself did not provide any significant additional benefit (Schackow et al. 2004). Based on opinion surveys alone, other studies, as this one, have reported that students had the impression that using an ARS in various educational contexts improved their ability to retain learned concepts (Johnson 2005; Latessa & Mouw 2005; Pradhan et al. 2005; Plant 2007), although these findings were not always consistent with whether it actually improved their learning (Caldwell 2007). Several investigators have also evaluated the impact of the use of an ARS on learning based on various forms of pre- versus post-testing including final examinations (Schackow et al. 2004; Slain et al. 2004; Molgaard 2005; Pradhan et al. 2005; Plant 2007; Nayak & Erinjeri 2008), on short-term (less than one month) recall (Schackow et al. 2004; Pradhan et al. 2005; Nayak & Erinjeri 2008) and long-term retention (Plant 2007). Differences in the context of the use of ARS, types of students and tools for assessment between these different studies precludes further comparison, yet it is relevant to consider them in light of the measurement of recall versus that of true long-term retention.

With regard to short-term results or performance on final examinations, some studies have shown a positive effect of ARS use (Schackow et al. 2004; Slain et al. 2004; Pradhan et al. 2005), while others have failed to show a significant difference (Plant 2007; Nayak & Erinjeri 2008). It is important to note, however, that most of these studies involved very small sample sizes or very short-term assessments, such as immediate post-lecture quizzes as an indication of retention. In the current study, students having benefited from case-based discussions combined with review questions on pertinent course material using the ARS performed significantly better on their final examination than their study counterparts. However, despite similar academic performance levels in both groups, this improved performance on the part of the ARS group may be explained by the fact that these students reported a significantly greater number of hours spent reviewing course notes and preparing for the final examination than those in the other group. Nonetheless, based on previously published findings and this study, further investigation of the effect of ARS use on short-term retention, which is mostly due to recall (Bell et al. 2008), using a cross-over design or more homogenous study groups would be of interest.

On the other hand, as others have demonstrated, activities that appear to produce rapid learning and/or high learner satisfaction may nonetheless result in poor long-term retention (Bell et al. 2008). While most reviewers on the subject agree that there is ample converging evidence in various fields of education suggesting that clickers generally improve student outcomes (Caldwell 2007), this study could not confirm a significant impact of ARS use on long-term retention, since students having used the ARS did not perform better than those without the ARS on 1 year post-testing. Furthermore, students from the ARS group showed a 10% decrease in performance compared to their immediate in-class responses to the same questions which could be explained by the fact that, as an essential mental function, retention is known to worsen with increasing passage of time especially if the new knowledge is not used by the learner (Bell et al. 2008).

Use of an ARS in case-based discussions in large groups also had drawbacks which echoed findings from other studies. Despite the fact that observers noted that setting up and initiating ARS sessions in large groups was generally fast and simple and that students appeared to be at ease with the clickers, the teacher and a majority of students stated that minor technical problems related to the software or to the performance of the equipment (clickers and receptors) used were the main negative aspects related to ARS use as reported elsewhere (Uhari et al. 2003; Caldwell 2007; Plant 2007; Nayak & Erinjeri 2008). The fact that ARS use tended to take more time to cover the same amount of material (Slain et al. 2004; Caldwell 2007; Plant 2007; Nayak & Erinjeri 2008) and the additional time and effort required from teachers and students before and during the sessions (Slain et al. 2004; Latessa & Mouw 2005; Caldwell 2007) were also common complaints noted in other studies. As demonstrated above, ARS use brings about changes in teaching strategies and requires a minimum of adaptation to teaching approaches. To reduce the incidence of negative aspects of ARS use, it is, therefore, recommended that educators seek practical advice from experienced users or consult publications that provide useful tips (Michel et al. 2002).

The study described in this report was based on the data collected from a large number of subjects, and relatively good response rates yet has limitations that should be addressed to avoid misinterpretation. First, the use of two different cohorts of students over two consecutive semesters precluded blinding or cross-over design. Although cohorts of the veterinary undergraduate degree are relatively homogeneous from 1 year to another, this particular study design may have introduced bias due to differences in class composition. In fact, although average age and age range as well as overall academic performance were not significantly different, the difference in female to male ratios may have had an influence on students’ responses to the survey or their perceptions on the use of ARS. The significant differences between groups could therefore have been due to factors other than the use of the ARS alone. Also, as others have suspected as well (Pradhan et al. 2005), the novelty of the ARS in itself could have led to alterations in teaching methods and the primary instructor's attitude, and the level of concentration and enthusiasm in class and could have influenced students’ responses to the survey and focus group interviews, a phenomenon known as the Hawthorne effect (Caldwell 2007). Finally, recall and retention assessments were based solely on tests composed of multiple-choice questions (MCQ) in an attempt to simplify and standardize 1 year post-evaluations and obtain optimal participation rates. Although an effect on recall after a full year is less likely (Bell et al. 2008) and there was no significant difference between the groups in the 1 year post-test, group B students may have had an advantage over group A students since context of learning and context of assessment are known to be related, although this is unlikely. While classic MCQs have been shown to be highly reliable because of the large quantity of concepts that can be tested and marked (Wass et al. 2001), there has been some criticism on their validity in some cases because, if poorly designed, they appear to measure only trivial knowledge (Wass et al. 2001). Nonetheless, MCQs are considered valid to evaluate the level of knowledge (‘knows’) (Wass et al. 2001; Amin et al. 2006) and, if well designed, are also valid to assess lower levels of skill acquisition (‘know how’) which are normally considered adequate for early (pre-clinical) stages of medical training, such as in this study (Amin et al. 2006). Yet, with the general objectives of a veterinary curriculum in mind, there was no practical evaluation made for the retention of knowledge and skills in this study; therefore, conclusions on the benefits of ARS use in improving retention on a ‘shows how’ (Miller 1990) level of competence cannot be made. These limitations have also been reported by others where questions used to assess the impact of the ARS on learning were not necessarily validated to adequately measure knowledge or comprehension acquisition (Pradhan et al. 2005; Plant 2007).

More specifically, the focus of this study was on the potential impact of ARS use on the educational experience during case-based discussions in larger groups. In fact, although case-based discussions have been shown to be effective teaching tools in veterinary clinical pharmacology and other fields (Novotny 1998; Michel et al. 2002; Monahan & Yew 2002), these activities are usually conducted in small groups which are associated with greater opportunities for personal engagement and focussed feedback (Bruneau & Langevin 2003). In contrast, large groups are typically characterized by a more passive attitude on the part of the students, unfocussed feedback and decreased motivation for both students and teachers (Gibbs & Jenkins 1992; Bruneau & Langevin 2003; Slain et al. 2004). When time, manpower and curricular constraints preclude case-based teaching in small groups, this study demonstrates that the use of an ARS provides the beneficial aspects of small group dynamics to larger groups of students so that the educational objectives of case-based teaching can be achieved (Gwee & Hoon 2001).

Conclusions

Use of an ARS was effective in promoting an active learning environment while conducting case-based discussions in a large group of undergraduate veterinary students. Along with an increase in motivation and a modification in teaching approach, the overall benefit of ARS use was found to be an improved learning experience for both teacher and students.

Acknowledgements

The authors wish to acknowledge the valuable contribution of Mr André Laflamme as pedagogical consultant in this study, as well as to thank Dr Stéphanie Dugas for data entry and validation, Mr Guy Beauchamps for statistical analyses and Dr David Quirion for providing a sample ARS in the initial stages of the study.

Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the article.

Additional information

Notes on contributors

Michèle Doucet

MICHÈLE DOUCET DMV, DVSc, Dipl. ACVIM, ACVCP, is a Professor of Veterinary Clinical Pharmacology at the University of Montreal's College of Veterinary Medicine. Besides her regular teaching and research duties, she is involved in pedagogical research with a special interest in teaching tools that promote active learning in undergraduate students.

André Vrins

ANDRÉ VRINS, DMV, is an Associate Dean for professional studies at the University of Montreal's College of Veterinary Medicine and is a Professor of Equine Internal Medicine. In recent years, he has been involved in curricular reform and evaluation as well as in implementing a veterinary leadership and communication programme at this institution.

Denis Harvey

DENIS HARVEY, DMV, MSc, PhD, is a Full Professor of Bovine Medicine and Surgery, Ambulatory Clinic, College of Veterinary Medicine at the University of Montreal. He has been involved in multimedia learning system design and distant learning for many years.