The effects of audience response systems on learning outcomes in health professions education. A BEME systematic review: BEME Guide No. 21

Figures & data

Table

Practice points

ARS may improve knowledge scores and do improve learner reaction. Findings are more striking with non-interactive teaching comparators and non-randomised studies. In postgraduates, where sleep deprivation is common, ARS may be even more beneficial (further study required). This review highlights the importance of having high quality studies with balanced comparators available to those making curricular decisions.

Table 1  Included online databases

Health-related databases	General databases
Medline (1950 to present)	Physical Education Abstracts
EMBASE (1980 to present)	SCOPUS (1823 to present)
PubMed (1950 to present)	Web of Science (1956 to present)
CINAHL (1937to present)	ERIC (1966 to present)
Cochrane Library (various dates to present)	OpenSigle (various years to present)
	Proquest Dissertations and Theses (content dates vary to present)

Note: Databases – note that all searches were limited to 1970 to July 2010.

Table 2  Search terms and strategy

Health-related databases
Health trainee education methods	and ‘ARS’
exp Education/ or exp Educational Technology/ or ‘teaching method*’.mp. or curriculum.mp. or ‘instructional method*’.mp.	‘audience response system*’.mp. or ‘classroom response system*’.mp. or ‘wireless response system*’.mp. or ‘electronic voting system*’ or ‘group response system*’ or ‘personal response system*’ or clicker* or iclicker* or ‘interactive voting system*’ or ‘student response system*’

Note: Limits: English language, human, 1970 to present.

Table 3  Search terms and strategy

General databases
‘ARS’	and ‘Health professions’
‘audience response system*’.mp. or ‘classroom response system*’.mp. or ‘wireless response system*’.mp. or ‘electronic voting system*’ or ‘group response system*’ or ‘personal response system*’ or clicker* or iclicker* or ‘interactive voting system*’ or ‘student response system*’	medic* or nurs* or ‘physical therap*’ or physician* or health or dentist* or pharmac* or ‘occupational therap*’ or doctor* or dietitician* or psychologist* or clinic*

Note: Limits: English language, human, 1970 to present.

Table 4  Inclusion and exclusion criteria applied to potentially relevant studies to determine suitability for systematic review purposes

	Inclusion criteria	Exclusion criteria
Population	Medical students Residents Physicians Nursing students/nurses Pharmacy students/pharmacists Dental students/dentists Veterinary medicine Trainees/veterinarians Dietician trainees/dieticians Clinical psychology trainees/Clinical psychologists Other allied health professionals	Non-health professions trainees
Intervention	Audience response in conjunction with: Lectures Workshops Small group learning sessions Clinical teaching Videos Other teaching sessions	Shadowing Mentoring Practice audits Feedback alone
Comparator	Any teaching method described under the inclusion criteria for ‘Intervention’ section without audience response. Any ‘standard curriculum’ without audience response
Outcome (Based on modified Kirkpatrick's 1967 model of hierarchical outcomes)	Change in patients’ health Change in behaviour Inclusion of skill in clinical practice Change in skills OSCE scores Observed assessment scores Change in knowledge Written exam scores Change in attitudes/perceptions Confidence self ratings Comfort self ratings Learner reaction Satisfaction with teaching method Satisfaction with instructor
Study type	Comparative studies, which provide primary data for any of the outcomes listed above, including the following designs: Randomised controlled trials Non-randomised control trials Cohort studies Controlled before and after studies Interrupted time series Other robust comparative studies English language (Morrison et al. 2009)	Studies reporting on needs assessments for audience response systems Studies reporting the prevalence of audience response systems Opinion papers Articles not in the English language

Figure 1. Flow diagram of included studies.

Table 5  Summary of findings

Outcome	Intervention	Comparator	Findings: Any significant difference			Study Design and Number of Participants Enrolled
			<5%	5–10%	>10%
Knowledge	Lecture with ARS	Traditional non-interactive lecture			Favours ARS	1 RCT (n = 22)
			No difference			1 RCT (n = 127)
					Favours ARS	1 RCT (n = 77)
			No difference			1 RCT (n = 20)
					Favours ARS	1 RCT (n = 17)
					Favours ARS	1 NRCT (n = 24)
		Traditional interactive lecture	No difference			1 RCT (n = 283)
				Favours ARS		1 RCT (n = 179)
				Favours ARS		1 RCT (n = 86)
			No difference			1 NRCT (n = 70)
			No difference			1 NCC (n = 28)
				Favours ARS		1 NCC (n = 254*)
					Favours ARS	3 NCC (n = 131, n = 141 and n = 131)
			Favours ARS			1 NCC (n = 169)
			No difference			1 NCC (n = 142)
			Favours ARS			1 NCC (n = 126)
			Favours ARS			1 Prospective cohort (n = 148)
		Lecture, unknown interaction		Favours ARS		1 NCC (n = 88)
				Favours ARS		1 NCC (n = 66*)
			No difference			1 Prospective cohort (n = 283)
	Tutorial with ARS	Standard tutorial (interactive)		Favours ARS		1 RCT (n = 102)
Self-confidence	Lecture with ARS	Traditional interactive lecture	Favours ARS			1 NCC (n = 169)
Reaction	Lecture with ARS	Traditional non-interactive lecture	Mixed			1 RCT (n = 127)
			Favours ARS			1 RCT (n = 77)
		Traditional interactive lecture	Favours ARS			1 NCC (n = 254*)
			Favours ARS			3 NCC (n = 131, n = 141, n = 131)
			Favours ARS			1 NCC (n = 169)
			Favours ARS			1 RCT (n = 283)

Notes: ARS = audience response system; RCT = randomised controlled trial; NRCT = non-randomised controlled trial; NCC = non-concurrent cohort.

*The exact number of participants enrolled in the study was not reported.

Table 6  Study characteristics

Citation	Institution	Design	Population	Research question/purpose	Intervention	Comparator	Primary outcomes
(Duggan et al. 2007)	The University of Adelaide, Faculty of Health Sciences	RCT	Undergraduate medical	To examine the effect an ARS when used as an integral part of a lecture, in terms of cognitive outcomes, interaction and lecturer and student satisfaction come up with their own material	ARS was used and examined in two lecture series, 3-hour blocks held every week (5–6 questions per lecture)	Traditional Lecture format	Knowledge: Immediate and 8–12 week post-quiz scores Learner reaction: Reaction to ARS and non-ARS lectures
(Elashvili et al. 2008)	University of Iowa College of Dentistry	RCT	Undergraduate dental	To evaluate student performance on written and psychomotor skill tests receiving conventional lectures versus an interactive ARS lecture	ARS was used with 12 questions asked throughout the lecture (some before and most after the concepts were taught)	Traditional lecture format with no questions	Knowledge: Immediate post-test, unit exam, final exam and practical exam scores Learner reaction: Student perception of ARS vs. non-ARS lectures
(Liu et al. 2010)	Midwestern University Chicago College of Pharmacy	RCT	Undergraduate pharmacy	To evaluate the effectiveness of an ARS on short- and long-term learning	ARS used in one lecture with five questions	A traditional lecture with the same five questions using hand raising to answer	Knowledge: Immediate and 1 month post-quiz scores
(Miller et al. 2003)	Nation-wide Study in 5 Centres (USA)	RCT	Health professionals	To examine the use of ARS as an interactive teaching tool for healthcare providers	ARS questions interspersed throughout the CRT presentation.	CRT with the same questions but with voluntary response	Learner reaction: Participant perceptions Knowledge: Immediate after quiz score
(Moser et al. 2010)	Wayne State University, Eugene Applebaum College of Pharmacy and Health Sciences	RCT	Graduate pharmacy	To evaluate the effect of utilising ARS on student understanding of material and overall class interaction	ARS was used for a 2-hour lecture with questions asked throughout	Traditional lecture with same questions being asked using voluntary participation	Knowledge: 10 Q score, exam average, block average Learner reaction: student interaction
(Palmer et al. 2005)	University of Adelaide, South Australia	RCT	Undergraduate medical	To measure the effect of the ARS on student learning and reaction within a small group tutorial	ARS used in two tutorials with multiple choice questions and multi-stage questions	Regular tutorial format with no ARS (standard questioning)	Knowledge: Pre-test, immediate and 6-week post-test scores, and retention calculation on both multiple choice and multi-stage questions Learner reaction (non-comparative)
(Plant 2007)	Oregon State University College of Veterinary Medicine	RCT	Undergraduate veterinary medicine	To evaluate the potential benefits of ARS for short-term and long-term knowledge retention and on student reaction	ARS used in 5 lectures (3–5 question slides/lecture)	Traditional lecture without the question slides	Knowledge: Immediate and delayed test scores Learner reaction (non-comparative)
(Pradhan et al. 2005)	UMDNJ-Robert Wood Johnson Medical School	RCT	Medical residents	To compare delivery methods of lecture material regarding contraceptive options by either traditional or interactive ARS lecture style	ARS used in one resident lecture on contraceptive options	Traditional didactic lecture with no interaction	Knowledge: Pre-test and post-test scores Learner reaction (non-comparative)
(Rubio et al. 2008)	Department of Radiology, Cincinnati Children's Hospital	RCT	Medical residents	To develop techniques for optimising educational time by evaluating the effect of ARS on residents’ retention of material in both the short and long term	ARS used in one lunchtime lecture (five questions asked)	Standard didactic format (no questions or interaction)	Knowledge: Immediate and 3-month post–lecture test scores Learner reaction (non-comparative)
(Patterson et al. 2009)	Widener University, School of Nursing	NRCT	Undergraduate nursing	To evaluate the effectiveness of a teaching intervention using an ARS in the classroom by examining learning outcomes	ARS used in a lecture with questions dispersed through lecture. It was used for six lectures throughout the semester	Traditional lecture with same questions being asked using hand raising	Knowledge: Unit and final exam scores Learner reaction (non-comparative)
(Schackow et al. 2004)	Department of Family Medicine, University of Illinois	NRCT	Medical residents	To evaluate if an ARS used during didactic lectures can improve learning outcomes by family medicine residents and to identified factors influencing ARS assisted learning outcomes	ARS used in monthly lectures (seven additional ARS multiple choice question slides in the presentation)	Traditional lecture format in an interactive (with 7 questions) and non-interactive (no questions) format	Knowledge: Immediate post-lecture quiz scores, 1-month post-lecture quiz scores
(Barbour 2008)	University of Bristol Dental School	Non-concurrent cohort	Undergraduate dental	To assess the student perception of ARS implementation and the impact on end-of-course examination results	ARS was used for 9 lectures and 1 large tutorial session (between 6 and 15 questions/lecture)	Traditional lecture and tutorial with standard questions	Knowledge: Final exam scores Student reaction (non-comparative)
(Berry 2009)	University of Wisconsin-Eau Claire College of Nursing and Health Sciences	Non-concurrent cohort	Undergraduate nursing	To assess the effect of ARS on exams scores, final grades and the level of participation	ARS used in lecture throughout the semester in class and at remote locations (10 questions/lecture)	Traditional lecture with pre-class quizzes (10 questions/quiz)	Knowledge: Unit exam and final exam scores, overall grades Learner reaction (non-comparative)
(Cain et al. 2009)	College of Pharmacy, University of Kentucky	Non-concurrent cohort	Undergraduate pharmacy	To see if ARS could maintain student attention, improve student comprehension, and improve student grades and reactions	ARS used during lecture (ARS used for first 28 of 42 lectures followed by 14 lectures of traditional classroom lecture)	Traditional lecture format with oral questioning	Knowledge: Final grades of class and personal averages compared with previous year Learner reaction: Overall course and instructor rating
(Doucet et al. 2009)	University of Montreal's College of Veterinary Medicine	Non-concurrent cohort	Undergraduate veterinary medicine	To evaluate whether the use of an ARS would promote an active learning environment during case-based discussions in large groups, have an impact on student motivation and improve long-term retention	ARS was used in a 2 hour case-based discussion lectures at the of the five-week blocks (three blocks)	Case-based discussion groups with open-ended voluntary participation questions	Knowledge: Final exam scores, and 1 year post-test scores Learner reaction: Student perceptions and preparation
(Grimes et al. 2010)	Austin School of Nursing, University of Texas	Non-concurrent cohort	Graduate nursing	To evaluate the usefulness of integrating ARS technology into an accelerated graduate nursing program by measuring leaner and instructor satisfaction and examining student achievement	ARS used in a weekly 2-hour lecture course for 15 weeks (up to eight questions embedded in lectures)	Traditional lecture format	Knowledge: Standardised final exam scores Learner reaction (non-comparative)
(Halloran 1995)	Nursing Department, Western Connecticut State University	Non-concurrent cohort	Undergraduate nursing	To assess the effect of computer aided instruction and keypad questions compared with traditional classroom lecture on student achievement	Computer-aided and ARS incorporated lectures (used throughout semester in most or all lectures)	Traditional lecture format (overhead, discussion and oral questioning)	Knowledge: Two midterm exams and final exam scores Learner reaction (non-comparative)
(Lymn & Mostyn 2009)	University of Nottingham School of Nursing	Non-concurrent cohort	Health professionals	To incorporate and evaluate the use of an ARS in promoting understanding and improving student performance	ARS use in lecture (127 questions were incorporated into eight lectures with 13 questions repeated two times or more)	Traditional lecture without the ARS	Knowledge: Formative and summative test scores Learner reaction (non-comparative)
(Slain et al. 2004)	School of Pharmacy, West Virginia University	Non-concurrent cohort	Graduate pharmacy	To evaluate the impact of interactive ARS on student learning, interest and satisfaction.	ARS used during lecture in three separate courses (43, 127 and 70 ARS questions used in each course, respectively, throughout the semester)	Traditional lecture format with mini case questions	Knowledge: Exam scores, final grades and specific question subsets Learner reaction: Multi-question survey
(O’Brien et al. 2006)	Case Western Reserve University (Case) School of Medicine	Prospective cohort	Undergraduate medical	To assess the impact of ARS on final exam scores and student reaction	ARS was used throughout the semester (17 of 31 large group lectures and 3–4 questions/lecture asked at the end of lecture)	The same class and questions as intervention, but those students who did not once use respond to the ARS questions	Knowledge: Final exam scores Learner reaction (non-comparative)
(Stein et al. 2006)	University of Kentucky College of Medicine	Prospective cohort	Undergraduate nursing	To determine whether the use of ARS for pre-test reviews improved student learning outcomes, to describe the steps involved in creating an ARS review, and to encourage nurse educators to implement this technology in their classrooms.	There were six of eight pre-test reviews given using the ARS. It was used in conjunction with a Jeopardy style game.	Two of eight pre-test reviews were given in the traditional review format (lecture-style overview of notes)	Learner reaction (non-comparative) Knowledge: Test scores

Table 7  Main findings of the review

Citation	Design	Outcomes	Qualitative results	Author's conclusions	Stated limitations
(Duggan et al. 2007)	RCT	Knowledge: Immediate and 8–12-week post-quiz scores Student reaction: Attitudes of ARS and non-ARS lectures	Knowledge sores were not completely reported, however, the author reported was no difference in MCQ scores between ARS and traditional lectures (p = 0.785). The cervical cancer lectures showed higher student ranking in favour of ARS in all parameters. The breast cancer lectures showed higher ranking in favour of traditional lectures in five of seven parameters (p < 0.001)	‘In this setting, EVS [ARS] technology used in large group lectures did not offer significant advantages over the more traditional lecture format’	There was poor participation, small sample size; only two lectures on two topics were examined
(Elashvili et al. 2008)	RCT	Knowledge: Immediate post-test, unit exam, final exam and practical exam scores Student Reaction: Student perception of ARS vs. non-ARS lectures	Statistically significant differences were found in favour of the ARS in scores on the immediate written post-test (mean scores 8.7/10 vs. 7.6/10, p = 0.002) and in performance bond strength testing (means of 26.7/40 vs. 23.3/40, p = 0.039) for the lecture ‘Principles of Dental Bonding’. The other examinations/skill testing showed no significant difference between the two groups. The responses to the question ‘I can easily transfer my knowledge gained from the class to the practical examination’ were significantly higher in the ARS group receiving the ‘Principles of Dental Bonding’ lecture. All other questions showed no difference	‘This technology has the potential to increase student knowledge retention and the ability to transfer the student's knowledge through psychomotor skill performance when used carefully in the context of the lecture’. These results indicate that ARS is a promising teaching tool for dental education	One lecture (with ARS) was given in the afternoon on the same day of a major exam in the morning
(Liu et al. 2010)	RCT	Knowledge: Immediate and 1-month post-quiz scores	Students who attended the ARS class scored an average 1 point higher on the immediate quiz than students who were assigned to the control group (10.7/16 vs. 9.7/16, p = 0.02). No significant difference was seen between the quiz 2 scores of the two groups (9.5/16 vs. 9.5/16, p = 0.99)	‘The use of a student response system can positively impact students’ short-term learning; however, that positive effect did not appear to last over time’	The study was non-concurrent, allowing possible communication between two groups. The small number of questions in the quiz may not allow for significant difference to be shown
(Miller et al. 2003)	RCT	Student Reaction: Participant perceptions Knowledge: Immediate after quiz score	ARS participants rated the quality of the presentation, the quality of the speaker and their level of attention significantly higher than non-ARS participants (p < 0.05). Knowledge scores were not significantly different between the two groups (3.9/5 vs. 4.3/5, p = 0.129)	The overall opinion of the ARS was favourable; however, the comparative test assessment showed no difference between groups	The study relied on self-reported data, it was not a blinded study and there was lecturer variability
(Moser et al. 2010)	RCT	Knowledge: Subset of ARS questions, exam average, overall block grade and average GPA Student reaction (non-comparative)	Statistically significant differences were found in the 10 exam questions that covered material from the 2 hours of ARS/non-ARS lecture (6.6/10 vs. 5.8/10, p = 0.03) and overall block grade (82.9% vs. 80.4%, p = 0.047) favouring ARS lectures. No difference was found in the exam average (78% vs. 76.1%, p = 0.31) or the average GPA (3.0 vs. 3.0, p = 0.409)	Use of the ARS resulted in better exam scores and increased in-class interaction between students and instructor. These results suggest that the ARS is an effective tool at promoting learning and should be utilised throughout the pharmacy curriculum	NS
(Palmer et al. 2005)	RCT	Knowledge: Pre-test, immediate post-test, 6-week post-test scores, and retention calculation on both multiple choice and multi-stage questions Student reaction (non-comparative)	Different quiz question types were analysed for ARS and non-ARS tutorials. All pre-test scores were statistically similar. For multiple-choice questions (MCQ) in the GI haemorrhage tutorial no scores were significantly different. For MCQ in the abdominal pain tutorial immediate quiz scores (7.60/11 vs. 6.94/11, p = 0.03) favoured traditional tutorials, whereas all other scores were not significantly different. For Multistage questions (MSQ) in the GI haemorrhage tutorial, both the 6-week post-quiz (5.96/12 vs. 6.72/12, p = 0.01) and retention scores (−0.28 vs. 0.52, p = 0.01) favoured the ARS tutorial. For MSQ in the abdominal pain tutorial immediate quiz scores (6.79/12 vs. 6.22/12, p = 0.03) favoured traditional tutorials, while retention scores (−0.87 vs. −0.02, p = 0.02) favoured ARS tutorials	‘Electronic voting systems can provide a stimulating learning environment for students and in a small group tutorial may improve educational outcomes’	There was a limited amount of ARS equipment (enough for 30 students)
(Plant 2007)	RCT	Knowledge: Immediate and delayed test scores Student reaction (non-comparative)	The mean short-term knowledge-retention test scores of the ARS group and non-ARS group were 81% and 78% (p = 0.32,), respectively. The mean long-term knowledge-retention test scores of the ARS and non-ARS groups were 54% and 55% (p = 0.77), respectively. The differences between groups were not significant for either time period	‘Although benefits to short-term and long-term knowledge retention were not detected in this pilot study, all students responding to the survey perceived a benefit and supported the use of ARS in the clinical veterinary medicine curriculum’	The study had a very small sample size, voluntary participation and participants not blinded to the outcome (after lecture test)
(Pradhan et al. 2005)	RCT	Knowledge: Pre-test and post-test scores Student reaction (non-comparative).	Residents who received ARS interactive lectures showed a 21% improvement (78%–95%) between pre-test and post-test scores, whereas residents who received the standard lecture demonstrated a 2% improvement (80%–82%). There was a significant difference in improvement of test scores favouring the ARS lectures (p = 0.018).	The evidence shows the effectiveness of the audience response system for knowledge retention, which suggests that it may be an efficient teaching tool for residency education.	It is possible that the effect seen was due to the novelty of the system and an increase in attention.
(Rubio et al. 2008)	RCT	Knowledge: Immediate and 3-month post-lecture test scores Student reaction (non-comparative)	Immediate post-test scores (76.4% vs. 60.0%, p = 0.02) significantly favoured ARS lectures over traditional lectures. Three-month post-scores (58.2% vs. 27.5%, p < 0.001) also significantly favoured ARS lectures	The ARS is an effective alternative to either traditional lecture or to the hot seat style. Most resident teaching use one of these two styles	The sample size was small in this study, and there were limitations that did not allow every initial learner to be retested
(Patterson et al. 2009)	NRCT	Knowledge: Unit and final exam scores Student reaction (non-comparative)	Comparing scores after ARS vs. traditional lectures; unit 1 exam scores (43.76/50 vs. 44.16/50, p = 0.562), unit 2 exam scores (40.42/50 vs. 41.19/50, p = 0.332), unit 3 exam scores (39.18/50 vs. 37.53/50, p = 0.060), and final exam scores (65.84/80 vs. 64.59/80, p = 0.340), there was no statistical difference in any test scores	There is no observed difference in achievement on exams but this may be due to limited exposure (six lectures for 15 minutes each). The study was promising because it showed increased student engagement	The study was a convenience sample. There was a limited exposure and technology implementation problem that may have affected the outcomes
(Schackow et al. 2004)	NRCT	Knowledge: Immediate post-lecture quiz scores, 1-month post-lecture quiz scores	Immediate post-lecture quiz scores were 4.25/7 with non-interactive lectures, 6.50/7 following interactive lectures without ARS and 6.70/7 following ARS lectures. The difference in scores following ARS or interactive lectures versus non-interactive lectures was significant for both (p < 0.001). Six-week post-lecture quiz scores were 3.39/7 with non-interactive lectures, 4.22/7 following interactive lectures and 4.67/7 following ARS lectures. ARS was significantly higher than basic (p < 0.05), whereas interactive was not (p = 0.11). The difference between the ARS and interactive lectures was not significant for either quiz	‘Both audience interaction and ARS equipment were associated with improved learning outcomes following lectures to family medicine residents’	This was a small study, it was not randomised and there was not 100% attendance of participants. The study was limited to family medicine residents only
(Barbour 2008)	Non-concurrent cohort	Knowledge: Final exam scores Student reaction (non-comparative)	There was no significant difference in final exam scores (71.60% vs. 69.90%, p = 0.44) comparing ARS lectures vs. traditional lectures	The ARS system proved very popular with the students. There was, however, no statistically significant impact on the results of the examination at the end of the course	This was not a controlled experimental study, and the sample size was limited
(Berry 2009)	Non-concurrent cohort	Knowledge: Unit exam and final exam scores, overall grades Student reaction (non-comparative)	Statistically significant differences were found in Exam 2 (91.23% vs. 86.93%, p = 0.000) and Corse Grades (95.03% vs. 93.33%, p = 0.000) in favour of ARS lectures. No difference was found in Exam 1 (90.57% vs. 90.48%, p = 0.880) or the Final exam (90.78 vs. 89.80%, p = 0.180)	Knowledge outcomes showed that some increase when the ARS was used. Student satisfaction supported the use of ARS as a way to engage students	Control group, although similar in admission GPA and size, were not matched for other characteristics. The study was in a rural nursing program and may not be applicable on a larger scale
(Cain et al. 2009)	Non-concurrent cohort	Knowledge: Final grades of class and personal averages compared with previous year Student reaction: Overall course and instructor rating	The final average for the ARS cohort (2008 class) was 89.9% while the comparative cohorts were 86.1% (2007, p < 0.05) and 81.8% (2006, p < 0.001). This was a significant difference favouring ARS above either cohort. Overall ARS course ratings were 3.3/4.0 vs. 2.4/4.0 for comparator. Overall ARS instructor ratings were 3.5/4.0 vs. 2.65/4.0 for comparator. The author stated significantly higher ratings favouring ARS in both categories but no p value was given	While using the ARS strategy, ‘classroom attendance improved, course grades increased, and course/instructor ratings improved’. A causal relationship cannot be established but this shows positive results for ARS use	This was not a controlled experimental study
(Doucet et al. 2009)	Non-concurrent cohort	Knowledge: Final exam scores, and 1-year post-test scores Student reaction: Student perceptions and preparation.	Final examination results (92.2% vs. 89.0%, p = 0.03) were significantly greater for the ARS lectures. One-year post-tests showed no significant difference. Student confidence with difficult skills was significantly higher for the ARS group in three of six categories of questions and do different in the other three. Student reaction results indicated that the use of an ARS provided an active learning environment by favouring engagement, observation and critical reflection and by increasing student and teacher motivation.	ARS use significantly improved the learning experience associated with case-based discussions in a large group of undergraduate students.	The study was not randomised; there was no blinding or crossing over. There was a different female to male ratio in to two cohorts.
(Grimes et al. 2010)	Non-concurrent cohort	Knowledge: Standardised final exam scores Student reaction (non-comparative)	Standardised final exam average (89.23% vs. 80.79%, p < 0.001) significantly favoured the ARS lectures over traditional lecture	Student and faculty satisfaction surveys as well as student performance outcomes all reflected positively on the use of the ARS	No comparative analysis of group demographics was done, nor was a pre-test–post-test design used
(Halloran 1995)	Non-concurrent cohort	Knowledge: Two midterm exams and final exam scores Student Reaction (non-comparative)	There was no significant difference in class averages for the ARS group vs. comparator for Midterm 1 (76.7% vs. 82.1%), Midterm 2 (80.1% vs. 82.6%), or the Final exam (83.4% vs. 78.4%). No p values were given	The ARS with computer-aided help is a viable option in nursing education. Test scores were statistically the same, however a trend toward improvement is seen in the ARS group	The study was small and was not randomised
(Lymn & Mostyn 2009)	Non-concurrent cohort	Knowledge: Formative and summative test scores Student reaction (non-comparative)	Students who had experienced ARS teaching scored significantly higher (p < 0.05) in both the formative and summative exam in comparison to students from cohort two. However, when compared with cohort one, there was no difference in summative or formative grades, suggesting a cohort effect. Exact scores were not reported numerically but were displayed graphically	The ARS has an overall positive impact on learning based on exam results and student opinion	There is a possible cohort affect between the experimental and one control group
(Slain et al. 2004)	Non-concurrent cohort	Knowledge: Exam scores, final grades, and specific question subsets Student reaction: Multiquestion survey	Three separate courses using ARS were examined. Students using the ARS had better scores on the Clinical Pharmacokinetics examination questions (mean scores, 82.6% vs. 63.8%, p < 0.001), on the cumulative final examination for Medical Literature Evaluation (82.9% vs. 78.0%, p = 0.016), and on the evaluable ‘analysis type’ examination questions in the Pathophysiology and Therapeutics course (82.5% vs. 77.4%, p = 0.0002). All other analysis of knowledge scores were not different. Students using the ISR system in all three courses were positive about the system	The ARS was a useful tool to encourage active student learning was well received by students. It is effective in gauging student understanding and can positively affect student performance	This was not a controlled experimental study
(O’Brien et al. 2006)	Prospective cohort	Knowledge: Final exam scores Student reaction (non-comparative)	Mean course exam score was 81.9% for non-participants and 85.8% for the students who used the ARS at least once. This mean progressively increased to 94.4% for the students who used the system the most. A regression analysis showed a significant (p < 0.01) relationship between the level of participation with the ARS and exam performance. There was no significant relationship (p > 0.1) on the Year 1 Comprehensive Exam, arguing against a self-selection bias for better exam performers	‘The use of an ARS with lectures for medical students seemed to improve exam performance and promote active learning’	There were a small number of participants, and only 14 students took 10 or more quizzes
(Stein et al. 2006)	Prospective cohort	Student reaction (non-comparative) Knowledge: Test scores	Average scores of the class on examinations preceded by the interactive ARS review were compared with those on examinations preceded by the more traditional, lecture-style review; no significant improvement because of ARS use was found. There were no scores reported in the study	Class examination scores did not show a significant increase when compared with standard reviews but student opinion of the ARS was very favourable	NS

Note: NS – None stated.

Figure 2. Meta-analysis of immediate knowledge-based test scores comparing ARS lectures (experimental) to traditional lectures (control).

Figure 3. Meta-analysis of long-term knowledge-based test scores comparing ARS lectures (experimental) to traditional lectures (control).

Morrison A, Moulton K, Clark M, Polisena J, Fiander M, Mierzwinski-Urban M, Mensinkai S, Clifford T, Hutton B. English-language restriction when conducting systematic review-based meta-analyses: Systematic review of published studies. Canadian Agency for Drugs and Technologies in Health, Ottawa 2009

 Google Scholar

Duggan PM, Palmer E, Devitt P. Electronic voting to encourage interactive lectures: A randomised trial. BMC Med Educ 2007; 7: 25

 PubMedGoogle Scholar

Elashvili A, Denehy GE, Dawson DV, Cunningham MA. Evaluation of an audience response system in a preclinical operative dentistry course. J Dent Educ 2008; 72(11)1296–1303

 PubMed Web of Science ®Google Scholar

Liu FC, Gettig JP, Fjortoft N. Impact of a student response system on short- and long-term learning in a drug literature evaluation course. Am J Pharm Educ 2010; 74(1)6

 PubMed Web of Science ®Google Scholar

Miller RG, Ashar BH, Getz KJ. Evaluation of an audience response system for the continuing education of health professionals. J Contin Educ Health Prof 2003; 23(2)109–115

 PubMedGoogle Scholar

Moser LR, Kalus JS, Brubaker C, 2010. Evaluation of an audience response system on pharmacy student test performance. Unpublished manuscript

 Google Scholar

Palmer EJ, Devitt PG, De Young NJ, Morris D. Assessment of an electronic voting system within the tutorial setting: A randomised controlled trial. BMC Med Educ 2005; 5(1)24

 PubMedGoogle Scholar

Plant JD. Incorporating an audience response system into veterinary dermatology lectures: Effect on student knowledge retention and satisfaction. J Vet Med Educ 2007; 34(5)674–677

 PubMed Web of Science ®Google Scholar

Pradhan A, Sparano D, Ananth CV. The influence of an audience response system on knowledge retention: An application to resident education. Am J Obstet Gynecol 2005; 193(5)1827–1830

 PubMed Web of Science ®Google Scholar

Rubio EI, Bassignani MJ, White MA, Brant WE. Effect of an audience response system on resident learning and retention of lecture material. AJR. Am J Roentgenol 2008; 190(6)W319–W322

 PubMed Web of Science ®Google Scholar

Patterson B, Kilpatrick J, Woebkenberg E. Evidence for teaching practice: The impact of clickers. Nurse Educ Today, 2009; 30(7)603–607

 PubMed Web of Science ®Google Scholar

Schackow TE, Chavez M, Loya L, Friedman M. Audience response system: Effect on learning in family medicine residents. Fam Med 2004; 36(7)496–504

 PubMed Web of Science ®Google Scholar

Barbour ME. Electronic voting in dental materials education: The impact on students’ attitudes and exam performance. J Dent Educ 2008; 72(9)1042–1047

 PubMed Web of Science ®Google Scholar

Berry J. Technology support in nursing education: Clickers in the classroom. Nurs Educ Perspect 2009; 30(5)295–298

 PubMedGoogle Scholar

Cain J, Black EP, Rohr J. An audience response system strategy to improve student motivation, attention, and feedback. Am J Pharm Educ 2009; 73(2)21

 PubMed Web of Science ®Google Scholar

Doucet M, Vrins A, Harvey D. Effect of using an audience response system on learning environment, motivation and long-term retention, during case-discussions in a large group of undergraduate veterinary clinical pharmacology students. Med Teach 2009; 31(12)e570–e579

 PubMed Web of Science ®Google Scholar

Grimes C, Rogers GJ, Volker D, Ramberg E. Classroom performance system use in an accelerated graduate nursing program. Comput Inform Nurs 2010; 28(2)79–85

 PubMed Web of Science ®Google Scholar

Halloran L. A comparison of two methods of teaching. computer managed instruction and keypad questions versus traditional classroom lecture. Comput Nurs 1995; 13(6)285–288

 PubMedGoogle Scholar

Lymn JS, Mostyn A (2009) Use of audience response technology to engage non-medical prescribing students in pharmacology. ICERI2009 Proceedings, MadridMadrid, 16 to 18 November, 2009, Spain, pp. 4518–4524

 Google Scholar

Slain D, Abate M, Hodges BM, Stamatakis MK, Wolak S. An interactive response system to promote active learning in the doctor of pharmacy curriculum. Am J Pharm Educ 2004; 68(5)1–9

 Web of Science ®Google Scholar

O’Brien TE, Wang W, Medvedev I, Wile MZ, Nosek TM. Use of a computerized audience response system in medical student teaching: Its effect on exam performance. Med Teach 2006; 28(8)736–738

 Web of Science ®Google Scholar

Stein PS, Challman SD, Brueckner JK. Using audience response technology for pretest reviews in an undergraduate nursing course. J Nurs Educ 2006; 45(11)469–473

 PubMed Web of Science ®Google Scholar