The audience response system and knowledge gain: A prospective study

Abstract

Introduction: The introduction of an audience response system (ARS) in the obstetrics and gynaecology course for medical students at The University of Western Australia provided an opportunity to measure knowledge gain by ARS lecture formats compared with didactic lectures.

Methods: The study was conducted over four obstetrics and gynaecology terms, alternating the ARS and didactic format between two selected lectures each term. Students completed multiple-choice quizzes immediately post-lectures and 5 weeks later.

Results: Immediate post-lecture quiz mean scores for the ARS lectures were significantly higher compared with scores for the didactic lectures (7.5 vs. 6.7, p < 0.001). Pairwise comparisons between ARS and didactic lectures for each lecture topic showed significantly higher quiz scores for ARS lectures (preterm labour 8.3 vs. 7.4, p = 0.032; and prenatal diagnosis 6.9 vs. 6.0, p = 0.014). Quiz scores for the didactic preterm labour lecture were significantly higher than scores for the didactic prenatal diagnosis lecture (6.0 vs. 7.4, p < 0.001). Quiz results at 5 weeks showed no differences in scores between the ARS and the didactic lectures and no differences between lecture topics.

Conclusions: Use of the ARS in lectures appeared to improve knowledge gain immediately post-lecture but no difference was found after retesting at 5 weeks.

Background

Australia has recently experienced significant increases in medical school intakes designed to meet increasing demands from the health sector workforce and workplace (Joyce et al. 2007). The Faculty of Medicine, Dentistry and Health Sciences at The University of Western Australia has seen a 45% increase in student numbers over the last 6 years. This increase has presented the Faculty with new challenges in the delivery of effective and efficient educational programmes and in the use of resources.

Table

Practice points

The use of ARS technology can be an effective teaching tool that encourages active learning in a lecture format. ARS lecture format can increase knowledge gained from the lecture however, for ongoing retention of knowledge students need to integrate knowledge with self-directed and ongoing learning. Designs of ARS questions require careful consideration to encourage problem solving and deeper ongoing retention of knowledge.

Lectures are a time efficient method of presenting information to large groups of students (Uhari et al. 2003). They can provide an introduction to a subject, build on existing knowledge, provide different points of view, include up-to-date research and where relevant add personal experience (Brown & Manogue 2001). A carefully structured lecture can also be an effective way to combine and present information from multiple sources on complex topics (Richardson 2008). However, traditional didactic lectures also have the potential to merely facilitate passive learning, where students are only recipients of information presented by the lecturer, without any active engagement in the learning process (Gulpinar & Yegen 2005). Learners demonstrate limited attention spans and low retention rates of factual information in lectures where they are passive in the learning process (Fischer et al. 2004; Gulpinar & Yegen 2005).

Active learning, in contrast, is considered a powerful way to enhance learning, as improved learning occurs when strategies are used to encourage active student participation (Newble & Cannon 2001). The incorporation of an audience response system (ARS) to a lecture can help facilitate the transition from passive to active learning (Pradhan et al. 2005; Holmes et al. 2006; Caldwell 2007; Duggan et al. 2007; Alexander et al. 2009; Hoyt et al. 2010). Active participation is made possible by connecting each participant to the lecturer through an anonymous electronic response device. The lecturer poses a question to the students and provides an array of multiple-choice answers. The students then select their responses using their individual wireless keypads, and the results are displayed immediately on a visual histogram of the student's responses for both lecturer and students to see. In this format, immediate feedback on knowledge and understanding of the material is received in an anonymous, non-threatening manner (Beatty 2004; Menon et al. 2004; Caldwell 2007; Nayak & Erinjeri 2008), discussion is generated (Copeland et al. 1998; Caldwell 2007), and students’ attention span is increased (Copeland et al. 1998; Nguyen et al. 2006).

Studies in medical education of the perception of using the ARS lecture format have reported that ARS lectures were more active and engaging (Menon et al. 2004; Latessa et al. 2005; Duggan et al. 2007; Nayak & Erinjeri 2008), and allowed students to focus on the key points in the lecture (Copeland et al. 1998; Uhari et al. 2003; Nayak & Erinjeri 2008). Participants reported that an ARS lecture increased their level of confidence (Nayak & Erinjeri 2008), made it easier to gauge their level of mastery of material (Copeland et al. 1998; Uhari et al. 2003; Nayak & Erinjeri 2008; Alexander et al. 2009), and increased their ability to gauge the knowledge of other students in the class (Copeland et al. 1998; Uhari et al. 2003; Nayak & Erinjeri 2008; Alexander et al. 2009). One study reported that students felt ARS gave them confidence to verbally answer questions in subsequent lectures (Nayak & Erinjeri 2008).

However, these positive responses to ARS by participants have not consistently translated into an increase in knowledge gain when objectively assessed. Research on the direct measures of medical education (effectiveness of ARS on knowledge gain) is limited and the results variable. Some studies have found a positive correlation on knowledge gain with the use of ARS in lectures (Pradhan et al. 2005; O'Brien et al. 2006; Alexander et al. 2009), while other studies found no difference (Miller et al. 2003; Duggan et al. 2007).

Aim

The School of Women's and Infants’ Health, within the Faculty of Medicine, Dentistry and Health Sciences at The University of Western Australia currently runs a 10-week obstetrics and gynaecology term for Year 5 medical students of a 6-year program. The diverse academic timetable in the term includes weekly small group tutorials, clinical skills workshops, reflective tutorials and lectures. The introduction of ARS in the obstetrics and gynaecology term provided the School with the opportunity to study the effectiveness of using ARS in lectures. We hypothesised that short-term (immediate) and long-term (5 weeks) knowledge retention would be greater with the ARS format compared with the traditional didactic lecture system. The study provided the students and lecturers with the opportunity to comment on their perceptions of the use of ARS in an educational setting.

Method

The study was conducted prospectively over one academic year at the School of Women's and Infants’ Health. The study population was the 2009 cohort of Year 5 medical students during their obstetrics and gynaecology term. Two lecture topics were selected for inclusion in the study, one on preterm labour and the other on Ultrasound Assessment for prenatal diagnosis. The lectures were delivered by two subject-matter experts, presenting material in their area of specialist clinical practice. In preparation for the study, the lecturers altered their lectures to allow for question and discussion time.

Each lecture topic was delivered once a term over four terms. This provided a total of eight lectures in the study period. In each term, one lecture topic was delivered using the ARS format and the other lecture topic was delivered in the traditional didactic format. The use of ARS was alternated each term to the other lecture topic (delivered by the other subject-matter expert). Over the four terms, each lecture topic was delivered twice using ARS and twice using didactic format (Figure 1).

Figure 1. Study flowchart.

The lecture formats were delivered by PowerPoint slides with all students receiving a handout. Both lectures were held in the first half of the 10-week term. At the conclusion of both ARS and didactic lectures, a paper version 12-question multiple-choice quiz (five choices per question, single correct answer, 1 mark per question) was immediately administered to students (initial quiz). The 12-questions were based on content presented during the lecture. These 12-questions were different to the five questions dealt with during the ARS lecture.

Re-administration of the same 12-question quiz occurred 5 weeks later to assess retention of knowledge (5-week quiz). These quizzes were conducted prior to the end of term and administered to students who attended the particular lecture (Figure 1).

Each student and both lecturers were asked to complete a questionnaire immediately after the ARS lecture to assess their perception of using the ARS during the lecture. The questionnaire consisted of eight questions using a five-point Likert-scale (ranging from strongly agree to strongly disagree) and two open-ended questions on the benefits and disadvantages of using ARS in the lecture. To encourage uncensored answers, the students’ responded anonymously.

Approval for this study was obtained from The University of Western Australia's Ethics Committee.

Means and standard deviations (SDs) were used to summarise the quiz scores. Student perceptions were summarised with frequency distributions. Analysis of variance (ANOVA) was used to evaluate the differences in quiz scores between ARS and didactic lectures, lecture types and student terms. Bonferroni correction was used for multiple pairwise comparisons between the main effects such as lecture types or initial and 5 weeks scores. All hypothesis tests were two-sided and p < 0.05 were considered statistically significant. Statistical analysis was performed using SPSS statistical software (version 15.0, SPSS Chicago, IL).

Results

From a cohort of 170 5th year medical students, 120 (70.5%) and 122 (71.6%) students completed the immediate post-lecture quizzes and 106 (62.3%) and 108 (63.5%) students completed the 5-week follow-up quizzes. Attendance at lectures varied as some students had clinical responsibilities which precluded them from being present. Approximately 89–97% of students who attended the lectures participated in the initial post-lecture quizzes (Figure 1).

Quiz scores are summarised in Table 1. Immediate post-lecture quiz scores for the ARS lectures overall were significantly higher (mean 7.5 ± 1.7) compared with the scores for the didactic lectures (mean 6.7 ± 1.9), p < 0.001. Pairwise comparisons between ARS and didactic lectures for each lecture topic showed significantly higher quiz scores for ARS lectures (p = 0.014). Quiz scores for the didactic preterm labour lecture were also significantly higher than scores for the didactic prenatal diagnosis lecture (p < 0.001). There were no statistically significant differences in immediate quiz scores between terms (p = 0.057).

Table 1.  Quiz scores by lecture type and topic

		ARS lecture			Didactic lecture			All
Week	Lecture	Mean	SD	N	Mean	SD	N	Mean	SD	p-Value
1	Prenatal diagnosis	6.9*	1.5	65	6.0^c***	1.7	61	6.5	1.7	0.014^b
	Preterm labour	8.3**	1.6	55	7.4†	1.8	60	7.8	1.8	0.032^b
	Both lectures	7.5	1.7	120	6.7	1.9	121	7.1	1.8	<0.001^a
	p-Values	<0.001^b				<0.001^b
5	Prenatal diagnosis	7.0*	2.2	49	6.1***	1.9	54	6.5	2.1	0.222^b
	Preterm labour	6.5**	1.9	57	7.0†	2.0	54	6.8	2.0	0.999^b
	Both lectures	6.7	2.0	106	6.6	2.0	108	6.7	2.0	0.088^a

Notes: ^aANOVA p-values for presence of any groups differences; ^bpairwise contrasts between lectures and topics as appropriate; *p = 0.882, **p < 0.001, ***p = 0.705, †p = 0.244.

Data shown as mean (SD).

No differences in quiz scores between ARS and didactic lectures were present at 5 weeks post-lecture (p = 0.088; Table 1). There were no statistically significant differences in quiz scores between the four terms (p = 0.119). Final comparisons between the initial quiz scores and scores at 5 weeks demonstrated significant differences between the scores (p < 0.001; Table 1). Of note, quiz scores for ARS preterm labour lecture decreased over time (p < 0.001) while the scores for the ARS prenatal diagnosis lecture remained unchanged (p = 0.882). The quiz scores for both the didactic lectures did not change significantly over time (Table 1).

The results of the self-administered student perception questionnaire are summarised in Table 2. One hundred and nine of the participating students completed the questionnaire (90.8%). Students were asked to comment on the benefits and the disadvantages of using an ARS format. Common themes in this free text included: the anonymity of the devices; the opportunity to compare their knowledge against other students in a non-threatening way; interactive and enjoyable. Further comments from students echoed the literature on gauging knowledge retention during the lecture, highlighting important points, obtaining immediate feedback and allowing students to pick up misunderstandings made during the lecture.

Table 2.  Student perception of using ARS in a lecture

	Strongly agree	Agree	Neutral	Disagree
Using the ARS lecture format stimulated my interest in this topic	(37) 33.9%	(55) 50.5%	(15) 13.8%	(2) 1.8%
The ARS contributed to my understanding of this topic	(25) 22.9%	(57) 52.3%	(25) 22.9%	(2) 1.8%
The ARS facilitated my developing clinical reasoning process	(19) 17.4%	(46) 42.2%	(37) 33.9%	(7) 6.4%
The ARS is appropriate because it allows me to make mistakes anonymously	(46) 42.2%	(53) 48.6%	(9) 8.3%	(1) .9%
The ARS assisted me identify areas for further development	(19) 17.4%	(53) 48.6%	(31) 28.4%	(6) 5.5%
The ARS allowed me to focus better on key points in the lecture	(35) 32.1%	(52) 47.7%	(20) 18.3%	(2) 1.8%
The ARS assist me in retaining lecture information	(29) 26.6%	(57) 52.3%	(20) 18.3%	(3) 2.8%
I enjoyed participating in the ARS lecture format	(53) 48.6%	(54) 49.5%	(1) 0.9%	(1) 0.9%

Notes: Values used strongly agree to strongly disagree. ‘Strongly Disagree’ removed from table as no data in this field.

There were considerably less student responses outlining disadvantages compared to benefits. In all four teaching terms, a small number of students (total of 12 comments) expressed concern that ARS slowed the lecture especially while waiting for other student responses and that this would lead to less material being delivered.

A total of 98% of the students reported that they enjoyed participating in an ARS lecture and more than 80% felt that using the ARS format stimulated their interest and allowed them to make mistakes anonymously, while 75% of students felt ARS contributed to their understanding of the topic. Most students (80%) felt ARS assisted them in retaining lecture material and 79% agreed that the ARS allowed them to focus better on key points in the lecture. When asked if ARS helped them identify areas for further development most students (66%) agreed, 28.4% were neutral and 5.5% disagreed. In response to the statement, ARS facilitated my developing clinical reasoning process, 59.6% agreed, 33.9% were neutral and 6.4% disagreed.

One lecturer was more positive in her responses to the perception questionnaire and felt that the ARS format assisted with her adjusting her pace during the lecture and helped her gauge effectiveness in her teaching. Both lecturers felt they were able to cover the material in the time frame with the ARS format and that the ARS lecture enhanced their teaching.

Discussion

The results of the immediate post-lecture quiz showed higher mean scores for the ARS lecture format, demonstrating students initially retained more knowledge with the ARS format. Disappointingly, this initial increase was not retained over the 5 weeks. There are three main factors that may have contributed to the lack of difference, first variable clinical experience; second design of the embedded ARS questions; and third, the ability of the quiz questions to test for long-term knowledge.

It is problematic in the comparison of initial and the 5-week quiz, as the scores cannot account for individual student's clinical experience and self-directed learning. Experience in the clinical area allows students to consolidate and develop in the application of theoretical knowledge (Chan 2002) and motivate the student to further self-directed learning (Hutchinson 2003). The nature of the clinical environment means the student experience is varied. Thus, testing retention of knowledge proves difficult due to this inherent variability in clinical scenarios. This differential clinical experience may have contributed to the decrease in scores for the 5-week preterm labour quiz, as the management of preterm labour is influenced by many variables.

It is possible that the design of the ARS questions did not encourage deeper learning. When designing the ARS questions, it was decided to use a total of five multiple-choice questions in each lecture. The questions were used sparingly so the lecturer could emphasise and highlight important points (Robertson 2000; Miller et al. 2003). When designing the questions, we attempted to facilitate a range of aspects highlighted in the literature as benefits of using the ARS including assessing prior knowledge to prevent unnecessary repetition of concepts (McLaughlin & Mandin 2001); introduce new concepts (Pinsky & Irby 1997); check understanding of concepts (Beekes 2006); and to highlight key concepts (Beekes 2006).

The ARS questions also attempted to test for a variety of responses from cognitive (factual) knowledge to problem solving. Cognitive knowledge was assessed by asking direct recall questions. Testing of cognitive knowledge predicts and can correlate well with overall competence (McCoubrie 2004). The incorporation of several questions that applied a problem to a scenario, encouraged students to think about the answer in relationship to a clinical situation. A problem scenario challenges students’ clinical reasoning skills and is associated with deeper learning (McLaughlin & Mandin 2001). The responses to the 5-week quiz did not demonstrate long-term retention and it is possible the problem scenarios were insufficient to encourage deeper learning.

The answers to the initial post-lecture quiz indicated positive short-term recall of knowledge while long-term recall at 5 weeks was not demonstrated. Objective testing methods, such as multiple-choice questions are possibly ineffective to measure long-term retention or clinical reasoning (Cwiak et al. 2004). This may suggest that the study quiz questions were better designed to test for short-term factual recall rather than long-term retention.

On average, higher scores were also obtained for the preterm labour using the ARS format in comparison to the prenatal ARS lecture. Studies have shown that ARS can facilitate feedback to the lecturer about their pace, clarity, style and volume of teaching material (Caldwell 2007). In addition, it could be suggested that a particular lecturer's style of presenting or a particular topic could benefit more from an ARS lecture format.

The overall result of the student perception questionnaire is positive towards the use of ARS in lectures, this agrees with previous studies (Copeland et al. 1998; Miller et al. 2003; Uhari et al. 2003; Menon et al. 2004; Latessa et al. 2005; Holmes et al. 2006; Nguyen et al. 2006; Duggan et al. 2007; DeBourgh 2008; Nayak & Erinjeri 2008; Alexander et al. 2009). The majority of students enjoyed participating in an ARS lecture and reported developing a better understanding and retention of the material. As noted, several students in free text expressed concern that ARS slowed the lecture and thought this would lead to less material being delivered. Interestingly, both lecturers agreed that after initial adaptation of the lectures to allow for questions and discussion, during the actual ARs lectures the amount of material they were able to present was not reduced. These views are consistent with statements in ARS literature, that amending lectures to suit an ARS format can lead to prioritisation of factual content and refocusing of the lecture towards key concepts (Gauci et al. 2009).

ARS lecturers require flexibility and a wide knowledge of the topic as lectures can be diverted when addressing issues highlighted by the ARS questions (Beatty 2004). The lecturers in this study were both experienced lecturers who had presented these specific topics to medical students for at least 4 years. The lecturers’ feedback showed they enjoyed utilising this new format and perceived the ARS as a worthwhile teaching tool.

Our study has several strengths: it was carried out over four teaching terms and included four ARS and four didactic lectures. Both lecturers are senior clinicians and have been teaching for decades. The design of the study enabled the lecturers to conduct the ARS lecture and traditional lecture format twice adding to the standardisation process. The lecture formats for each topic were as similar as possible in their content and length. Lecture handouts remained the same between alternating format delivery, and the scheduled lecture times were the same each week and term.

A limitation of the study is that not all students in the cohort participated, predominantly due to clinical responsibilities preventing attendance at all lectures. Approximately 90–100% of students who attended the lectures completed the initial post-lecture quizzes, ranging from 22 to 36 students in a single session. Individual student's attendance at lectures was not recorded, therefore it is not known if the students attending the ARS lecture were the same students as those who attended the traditional didactic lecture, however considerable overlap could be expected.

Conclusions

It can be concluded that using ARS in a lecture format is an effective teaching tool and emphasises the value of audience interaction and participation. The positive feedback from the students regarding using the ARS format was encouraging and the results of the post-lecture quizzes suggest that knowledge gain is initially better with an ARS lecture and helps students obtain a better base of knowledge. For ongoing retention of knowledge, design of ARS questions requires careful consideration to encourage problem solving and deeper learning.

The ARS lecture format can make an effective contribution to a comprehensive learning programme. Clearly, students need to integrate knowledge with self-directed and ongoing learning in the clinical area.

Funding

This study was supported by a grant provided by King Edward Memorial Hospital Post Graduate Medical Research Fund, Perth, Western Australia. This contributed to payment of Audience Response System software and keypads.

Declaration of interest: The authors report no declaration of interest.