Using classroom response systems for creative interaction and engagement with students

Abstract

Recent changes to the landscape of higher education and the student environment generally has brought renewed focus on the need for pedagogical innovation. The importance of student satisfaction and the accompanying rising expectations of the technologically savvy generation of students have brought significant challenges to an academic community already busied with the pedagogically sound delivery of undergraduate taught courses. This new environment has inevitably led to a demand for tools that can assist convenors to meet those challenges without overburdening already tight workloads. This paper presents evidence on the innovative use of one such tool in the form of an over web classroom response system, introduced with the aim of meeting the challenges of the new era in higher education. We suggest that the use of this type of technology can increase student satisfaction and enjoyment as a driver for creative engagement.

Keywords:

• large group classes
• higher education
• classroom response system
• student response system
• clickers
• peer interaction
• economic tools for teaching

Public Interest Statement

Modern social media platforms such as Facebook and Twitter have been replacing the traditional channels that students have used to communicate during a taught course. Mobile applications are seen as more “in sync” with lifestyles over university-provisioned services such as discussion boards in a virtual learning environment. One tool that can bridge this recent divide between the traditional channels and the students’ own mobiles is the classroom response system (CRS). This technology allows students to respond to multiple choice questions during a lecture and initiates an interaction between lecturer and student, otherwise absent in the traditional lecture setting. Questions can be used to test understanding or to explore differences of opinion with the benefit of response-specific feedback from the convenor afterwards. In this paper, we describe our development in practice of this tool and present evidence on the student perception of the use of CRS in their teaching.

1. Introduction

In the last few decades, we have seen what we might call dramatic change in the higher education sector. Technology has driven exponential growth in opportunities for social connection, and an overwhelming array of vehicles for accessing data and sharing information. The Ericsson Mobility Report (2015 June, http://www.ericsson.com/res/docs/2015/ericsson-mobility-report-june-2015.pdf) finds that by the end of the decade, the change will be on a global scale with 75% of the world’s population owning smartphones and 90% of the population with access to mobile broadband network coverage. The report finds that in many countries, mobile broadband subscriptions already exceed populations, and this is particularly in mature markets where there is a concentration on increase of internet-enabled devices per individual rather than new subscribers. As Traxler and Wishart (2011) find, this proliferation of mobile devices has led to them being “taken for granted” in our lives, and to the educator this presents an opportunity to capitalise upon the devices of our learners and to “take education into new modes, spaces and places”.

UK universities have, in the first instance, found adapting to this technological tidal wave a challenge, particularly with the presentation of new government funding arrangements, and a drive towards increasing student satisfaction. Social media platforms such as Facebook, Twitter and WhatsApp have replaced the channels that students in the past were expected to use to communicate with each other during a taught course. The fast moving nature, popularity and ease of use associated with mobile applications have made university-provisioned services such as a Virtual Learning Environments’ discussion boards less attractive. Whilst within formal arenas computer-mediated communication seems to be at a dead end, use of these modern tools for learning is occurring at a far more grass-roots level between students, from which the lecturer is becoming disenfranchised.

This switch, where students actively select their learning tools outside the walled garden of university systems, will be of a concern to any convenor who has designed a course with some expectation of control over the learning environment. This leaves the undergraduate convenor with a dilemma: either reinforce the domain of the traditional intranet by persisting with the use of the traditional and practiced channels for course delivery; or risk a leap of faith and “roll with” the changes by embracing and capitalising upon the changes brought forth by the students. How then to bring together this distributed learning, and make inroads into fostering and engaging students with these tools while learning on a formal course? Research such as that presented by Educause’s ECAR study of 2014 points to a use of mobile technologies in formal education as not just something “that happens”. The research instead suggests that it requires some prompting usually through the vehicle of assessment.

This paper takes up the challenge by discussing the use of a classroom response system (CRS) that allows students to utilise their own web-enabled devices as a vehicle to engage with the generation that makes so-ready use of mobile technology as a second nature in their everyday lives, presenting tentative and supportive encouragement by the lecturer in seeing portable devices very much as learning tools. A further motivation for the use of CRS was to initiate a deeper interaction with students during lectures in a large cohort setting, notorious for its impersonal experience (Draper & Brown, 2004). Interacting with students inside the classroom during lectures carries obvious immediate benefits such as increased student attention, class enjoyment and enhanced learning, though it became apparent beyond this that other more surprising benefits presented themselves as students became more involved with the development of their taught course.

The CRS was introduced on two undergraduate and compulsory core economics courses at the University of Manchester from 2009: Macroeconomic Principles (ECON10042) taught in the first semester and Macroeconomics IIA (ECON20401) taught in the second semester. Both courses are taught in a large cohort setting with around 600–500 students registered on average, respectively, and making up part of the compulsory units for students specialising in economics registered across several programmes. Prior to 2009, the units were taught using a traditional approach of convenor-led lectures and graduate teaching assistant-led tutorials. A desire to address the challenges faced by students in the large cohort setting such as the feeling of being “just a number” provided the initial motivation for the adoption of CRS and also the general dissatisfaction with metrics such as feedback. Further in to this, it became clear that the best way to better interface with the students, or personalise the experience, might be through their own mobile devices, something that may have traditionally been viewed as an obstacle by convenors during lectures. The CRS system chosen was called TurningPoint Responseware which was considered most suitable to use in conjunction with mobile phones after an early spell of using the traditional handsets.

Students interacted by answering multiple choice questions embedded within PowerPoint slides. Initially, we made simple use of the technology by utilising exam-type multiple choice questions during revision sessions at the end of semester but with more practiced use, and surprisingly in partnership with students, began developing more sophisticated use of the interaction technology. One of the key benefits we have seen from the pedagogical development of our teaching is the ability to make use of the students as innovating partners in the whole process. We have now developed a whole repertoire of methods of interaction, the details of which we present below. Here, we should also mention that a unique aspect of this project has been the innovation which has presented itself from a collaboration between an academic and a technologist, an unexpected consequence and valuable consideration for HE institutions now looking for ways to drive forward pedagogical innovation.

A close collaboration between the academic and learning technologist has the potential to produce outcomes not normally imaginable by each professional working separately in different fields. Working together “in class” combined with the sharing of expertise in developments within education and technology provides a novel and highly productive approach towards innovation in pedagogy. Both professionals are more attuned to evaluating the overall delivery and consider individual elements within a holistic teaching approach. Of course, use of the technology relies on student ownership of a web-enabled device such as the touch screen mobile phone. In the earlier part of the project, this issue was overcome by the provision of hand-held hardware (something well known as clickers) to answer the obvious question of inclusivity. In later years, however, this has become much less of an issue as we have seen ownership of these kinds of devices become almost universal amongst students. Ownership of mobile phones doubled from 39 to 78% in the three academic years, 2011/2012 to 2013/2014, see an Appendix A for evidence of this technological tidal wave showing a breakdown by device type. Thus, the technological tidal wave most recently witnessed has seen a diminished need to distribute large numbers of handsets before class, resulting in increased use of the technology across the years. Increased use also brought about a refinement in pedagogical practice and we see this reflected in the evidence, some of which was collected using the CRS system itself, presented below. Satisfaction and enjoyment of course aside, we have also seen wider benefits around the student social value and enjoyed increased agility of teaching practice as a result of adopting this technology in class.

The purpose of this paper is to highlight our findings that use of a CRS system during the lecture can increase student satisfaction and engagement with their taught course, also how we might react to the seismic changes we have seen in the use of technology by students, or young people in general. After a literature review, we describe our developmental path from the adoption of CRS to a more practiced and detailed use of the technology. We present evidence of the students’ reaction to the introduction of this interaction technology and demonstrate positive impact made upon student satisfaction and enjoyment, very much aligned to the re-emphasis of National Student Survey (NSS) scoring on student engagement and collaboration. Finally, we conclude and offer our recommendations to convenors considering novel tools to further student engagement or those seeking to create an interactive classroom.

2. Literature review

The lecture as a mode of delivery on courses in higher education is not without its critics, see Bligh (1998) as a most obvious candidate. A common concern for convenors is the difficulty of including active learning elements or fostering participation. The exact format dates back to a time when a teacher was expected to transfer knowledge by broadcast or from reading the material from books. Students were not expected to be involved (Draper & Brown, 2004; Shulman, 2005). Authors have criticised the lecture format for promoting passivity or mindless note taking by students (Crouch & Mazur, 2001; Kolikant, Drane, & Calkins, 2010). The lecture has been classed as impersonal (Kolikant et al., 2010) or allowing very little in the form of feedback from the students (Boyle & Nicol, 2003; Draper & Brown, 2004).

And yet despite this deficiency, lectures are still the most favoured and lauded as a content teaching method used within higher education. As Shulman (2005) describes, this is a mode of teaching that delivers “to the largest number at the lowest cost”. As a vanilla, lectures may present these challenges, but we have all known lectures to be of high quality, thought provoking and inspiring. Many of those teachers will employ to some degree the Socratic method in their teaching, using questioning techniques to foster learning amongst their students. Here, questions are used rather than answers, with a focus on encouraging reasoning and understanding over retention and recall. See Laurillard (2013) for one such model. Draper and Brown (2004) and Bruff (2009) more specifically attribute deeper learning to situations where students are placed within an active role and Bligh (1998) favours teaching such as seminars and “buzz groups” that provide this opportunity. Given the status and longevity of the lecture format, what adaptations can be carried out to transform these towards dialogue, discussion and understanding?

As a teaching format that is geared towards mass educational delivery tending towards the didactic, the scale of teaching is inhibitive towards participation. Shulman (2005) sees this as encouraging students to adjourn “cloaks of invisibility” where their participation is concealed from others. Equally, a fear of failure in front of students is inhibitive towards experimentation by the lecturer, particularly where unit surveys leave lecturers feeling that that they are operating on a knife edge. What is needed therefore is a scaffold for allowing “safe” interaction to take place, ideally using an institutionally ratified platform, with delivery of immediate benefits to encourage and foster more outward forms of interactivity.

CRS is a teaching tool commonly promoted to lecturers looking to increase interactivity within their face-to-face teaching (Simpson & Oliver, 2007). These aids previously took the form of physical voting handsets (or “clickers”) distributed to students, but there is now a shift towards mobile application versions of the same tool. Historically, other systems have been used such as coloured flashcards or asking for a “show of hands”, but the current computerised systems are unrivalled in their speed of data collection, quality of data presentation and most importantly, their confidentiality and anonymity of student response (Kolikant et al., 2010; O’Donoghue & O’Steen, 2007). Over the last decade, universities have invested heavily in this technology, and a range of sources indicate that CRS can have an immediate novelty value for students and can create interest for contemporary learners (Broussard, 2012; Freeman, Bell, Comerton-Forde, Pickering, & Blayney, 2007; Koenig, 2010).

Most commonly, CRS are used to pose interactive multiple choice questions interspersed within the lecture content. Through setting the MCQs and examining how students have voted, gaps are bridged in understanding and differences of opinion can be illuminated. Now the lecturer is able to tailor the instruction more specifically to the cohort’s needs or interests (Crouch & Mazur, 2001; Dufresne, Gerace, Leonard, Mestre, & Wenk, 1996; Koenig, 2010). Because of the confidentiality of responses, students are also more encouraged to answer freely without fear of reproach, thereby providing the tutor with an unbiased picture of how students have voted (Draper & Brown, 2004; Freeman et al., 2007). Overall, this leads to lectures becoming far more enjoyable, with the increased dialogue stimulating both tutor and student (Draper & Brown, 2004; Freeman et al., 2007), a motivation behind our choice of survey question in the evidence presented.

CRS can also be employed within structured teaching techniques. O’Donoghue, Jardine, and Rubner (2010) describe a hierarchy of CRS uses beyond quizzes, and towards highly developed pedagogies that encourage dialogue-rich learning. One notable technique is Peer Instruction (Mazur, 1997), which uses structured questioning and small group discussion tasks. Crouch and Mazur (2001) regard Peer Instruction as having the potential to engage every single student, regardless of level and confidence through discussion, and they provide a useful outline of the process for new users. However, even modest refinement in practice can improve simple quizzing techniques such as Draper and Brown (2004) who suggest that spending time developing good question sets can provide a good return in increased benefits to teaching and learning.

There is evidence within the literature that CRS can have a catalytic effect on teaching with lecture environments moving from silence and passivity towards dialogue and interaction (Boyle & Nicol, 2003; Broussard, 2012; Crouch & Mazur, 2001; Cutts, Kennedy, Mitchell, & Draper, 2004; Gauci, Dantas, Williams, & Kemm, 2009; Kolikant et al., 2010); and, there is also a suggestion that CRS use facilitates a shift towards learner-centric approaches through reopened channels of communication (Kolikant et al., 2010).

Generally, the literature suggests a number of ways in which CRS technology can provide a positive impact upon the student experience. What follows is a developmental delineation of our use of this technology, starting from our most basic use and leading towards more specifically designed practice.

3. Evidence and discussion of CRS

Below, we discuss our specific approaches in using CRS and present the evidence collected using TurningPoint Responseware to measure the students’ reaction to the use of this technology in their learning, specifically their perception of the increase in satisfaction and level of enjoyment.

3.1. Basic use of CRS

Typically, a CRS allows students to be polled during lectures normally using multiple choice style questions, in our case, from inside Microsoft PowerPoint. The responses can usually be fed back to the presentation in some way so that the convenor can provide response-specific feedback to the students after the polling exercise. The most obvious benefit of using this system is that it allows basic interaction during the lectures and that students can check their understanding of the material during the course.

We had been making use of CRS for multiple choice style testing since 2009, but it is only since the academic year of 2011/2012 that we have been making regular use of the technology throughout the course and specifically using the over web version through the students’ own mobiles, in this case, Turningpoint Responseware. The data presented here were collected between 2011/2012 and 2013/2014 on two courses: ECON10042 (Macroeconomic Principles) and ECON201401 (Macroeconomics IIA). Progressive use has brought the intended higher satisfaction but also, and surprisingly, other benefits such as partnership, student-led innovation and a contribution towards the students’ own social development generally. We are in little doubt that these unintended benefits fed further increases in the students’ own satisfaction with the course. Students were able to check their own progress, empowered by the facility to contribute towards the design of their own course and also able to make use of the new tool to meet new people. For further comprehensive discussion of applications and benefits of this tool, see our literature review.

Evidence on the level of satisfaction was taken during revision sessions towards the end of the respective semester using the CRS itself. One benefit of using this method to collect evidence was the speed of use and lack of disturbance to the class and also that the system protected the anonymity of the students. In the spirit of the National Student Survey, students were asked the question to what level they disagreed/agreed with the statement “The voting system has enhanced my level of satisfaction with the programme 1-5” with 1 and 5 being associated with disagree and agree, respectively. Evidence from the three academic years, 2011/2012–2013/2014, for both courses is presented in Table 1.

Table 1. Survey results from UoM students on the use of CRS

Qu: To what extent do you agree or disagree with the following statement? “The voting system has enhanced my level of satisfaction with the programme.” 1–5

For both the first-year Microeconomic Principles and second-year Macroeconomics IIA, there is clear evidence that this tool is seen as a satisfaction enhancer with between 79 and 92% of students agreeing across both courses and all three years. Interestingly, the proportion of students agreeing increases over time with a range of 79–84% in the academic year 2011/2012 to 92% for both courses in 2013/2014. The year-on-year increase could be explained by the increased use of the technology and our development in practice, a finding consistent with Nielsen, Hansen, and Stav (2013) who connect experience of use with student satisfaction. Students disagreeing with the statement on satisfaction are in the small minority, but are a reminder that any innovation can never be universally popular, especially in a large cohort setting where the distributions of opinion can display significant variance. Below, we discuss three techniques that allowed us to further engagement with the students using the CRS.

3.2. Peer instruction

The evidence presented in the previous section reflects the use of CRS and also our own development in practice over time. We now discuss these developments in more detail and present evidence from the students’ perceived level of enjoyment of these activities. From 2010 onwards, we introduced a more sophisticated technique first introduced by Mazur (1997) that developed our use of CRS from the presentation of simple multiple choice exam-type questions to a structured framework designed to nurture a deeper learning process. Students were polled a suitable question but with the results hidden to avoid biasing the following poll; after this, they were asked to discuss their responses to the question in the previous slide with their neighbours in the theatre. Lastly, the students were re-polled again with the same question after their discussion/arguments before the lecturer ran through the aggregated responses to feedback to the whole class.

Polling in this way led to a significantly higher level of interest in the material being used for the exercise with students very keen to find out if their initial guess and also their discussions were correct. In the first couple of years, we introduced this tentatively so as to minimise any negative problems that might arise from using peer interaction in a large cohort environment. In the lectures that we introduced this technique, we used two peer interaction questions closely aligned to the material presented during the same lecture and aligned with the final assessment. Students were prepared for the exercise by the careful design of instructions immediately before and during the exercise. Being used in a large cohort, this way presented obvious challenges such as the need to restore the class to quiet after their excitement generated by the session, but practice led to a small repertoire of fixes such as playing music during discussion and allowing the end of the music to provide a natural cue to end discussions.

Below, we present our survey evidence from students exposed to the discussion activities we have referred to as peer interaction. For consistency with the earlier survey, we asked to what extent they agreed/disagreed with our statement aligning the exercise with their level of enjoyment, 5 to 1 (5 being agree and 1 disagree).

From the results presented in Table 2, we can see a clear and positive view from the students that the use of discussion-type, peer interaction questions made their course more enjoyable, with 86% agreeing with the statement on Macroeconomic Principles and 71% agreeing on Macroeconomics IIA. Interestingly, the second years had a higher “neutral” score for this question and the scores were slightly lower for the same cohort than when asked about the use of CRS as a satisfaction enhancer. One reason for this might be that the second-year students found the discussions disruptive or that our design of PI questions was more suitably aligned for a split opinion when written for the first-year course. Our general experience is that a split in opinion is always optimal in terms of generating interest. Overall, the survey results suggest that students enjoyed the activity, but that enjoyment was not universal. One could argue that by making the class more enjoyable, the students felt a higher level of engagement with the course.

Table 2. Survey results from UoM students on the use of peer interaction

Qu: To what extent do you agree or disagree with the following statement? “Taking part in the discussion activities made the course more enjoyable.” 1–5

3.3. Developing engagement with CRS

One immediate development in our practice that arose through our use of Peer Instruction was in the form of opportunities to capitalise upon the shift of opinion within the student body. Initially, this benefit was felt through taking the time to examine flawed reasoning and misconception revealed by the MCQ. It is a very easy trap for new and anxious users of this technology to skip over more thorough examination of how the class has voted, but this goes hand in hand with question design. Peer Instruction allows you as an instructor to re-evaluate the benefit of MCQs within live teaching, and you quickly find that questions with no clear majority provide much greater scope for learning to occur. Questions and their answers can be deliberately designed to split opinion, and good questions which split opinion can be archived for future use or refinement, year on year.

We found that the discussion exercises amplified student engagement, an interest in exploring reasoning behind correct and incorrect answers. Contrary to initial predications, the discussion did not always provide a shift in distribution towards the correct answer. When this occurred, we were surprised to see that students were not disappointed but were instead more eager to go through the example, and examine their learning at a metacognitive level. The benefits of a computer-based CRS is that it is a quick operation to compare results pre- and post-discussion, and the TurningPoint platform we were using provided a “Comparative Linking” template for this purpose. It became clear that the comparative link was a valuable addition to the peer interaction exercise, with students reacting by displaying further interest and appearing more engaged, something we see reflected in the evidence taken from students on the enjoyment of their studies in Table 2.

Further to this, we have begun to explore how we can capitalise upon this form of student “buy in”. It is not always appropriate to use discussion activities, and students will have varied tastes throughout their studies. However, the value of having students commit to an answer prior to teaching cannot be underestimated. This is something we have termed “holding attention to ransom” as a stand-alone technique; students are polled at the beginning of the lecture with a question aligned to the main learning outcome of the lecture. There does not have to be a sole right answer to the question as the shift in opinion displayed at the end of the lecture does the job of reinforcement. Having invested by commitment, the students’ interest in the material is enhanced until the second poll and discussion of opinion shift at the end of the lecture. The students’ attention has been held to ransom, as they anticipate closure on the problem presented.

3.4. Completing the feedback loop

The last area for us to discuss is that of student feedback. We have found that one of the richest streams of innovation has come from the students themselves and so it seems intuitive that to open up communication with students is to tap into this. One useful provision of our particular CRS system was the availability of an anonymous communication from the student provided via a menu alongside the polling options during the sessions. This was discovered by the students who used it in a fun way to provide us with anonymous and uninhibited, but friendly remarks during the lecture. Alongside the data collected from the actual responses to the multiple choice questions was a list of free-text comments generated into an Excel report for later viewing by the convenor. The feature of anonymity was particularly encouraging for the students and undeterred from asking/communicating questions about their understanding of the material.

Students began using the tool to convey their level of understanding on the taught material. This facility proved particularly useful in reflecting on material presented and also on how certain areas might benefit from further explanation. Our particular CRS also provided the details of exactly which slide was showing when each particular comment was made, making it particularly easy to identify the associations of the comments. Further to this, students developed their own ways of using the tool: favourable comments, practical requests and even proposals for pedagogical innovation. One such example of this is the suggestion by a student to use audio clips to provide to lighten the atmosphere, something that also proved good at drawing discussions to a close.

Most importantly, on top of receiving and reacting to the feedback provided is to complete the loop by reacting to this in the aggregate during the lectures by discussing this with them the following week. Students, aware of the power of their feedback, were further stimulated and encouraged to feedback further. Effectively, students had become further engaged with their course and also partners in its development.

4. Conclusion

This paper has presented survey evidence taken from students exposed to the use of a CRS, introduced to encourage interaction and to foster engagement on two core courses in economics at the University of Manchester. We have found through over five years of practice that use of CRS can help initiate and further interaction with students, an objective that is notoriously difficult in the large cohort settings. From a basic introduction, we have seen how development in the use of CRS can also further engagement with students as they are encouraged to connect via exercises such as peer-to-peer interaction and through reinforcement of their learning outcomes. Apart from the data presented here, we have also seen the positive attitude of students to the innovations in their learning reflected in the end of semester online evaluation questionnaires scores, see Appendix A, and through the students’ open comments.

An unforeseen benefit of this exercise presented itself in the form of the student input to the development of practice. We would consider with hindsight that the contribution from the students in terms of development of delivery has been most important. Students themselves are well placed to assist the convenor, certainly in terms of ideas, and so opening up this channel would be highly recommended. Another recommendation would be to start, tentatively and gradually develop, the tools most suitable to your own course. One common objection to taking up a tool such as a classroom voting system might, quite rightly, be any negative consequence. What happens when things go wrong? In our experience and perhaps surprisingly, the students are always very forgiving, knowing that boundaries are being pushed. For this reason, we would also recommend advertising the presence of a pilot and a gradual introduction to the higher use of the technology to allow time to acquire expertise in practice.

Looking forward in our research agenda, we plan to further our engagement with students by extending the learning environment from inside to outside the classroom and so our current work involves the use of social media as a teaching tool in higher education. We see this work as a part of an ongoing process of continuous improvement in our research on delivery in higher education. Any journey of innovation can never be promoted as a costless exercise, but we can happily say that this one has added to our own value, not just that of our students.

Additional information

Funding

Funding. The authors received no direct funding for this research.

Notes on contributors

Paul Middleditch

Paul Middleditch is a senior lecturer in Macroeconomics at the University of Manchester, associate member of the Economics Network and fellow of the Higher Education Academy (FHEA). His research interests include monetary policy, macroeconomic time series and also innovation in the area of pedagogy, particularly the use of technology in lectures to foster a more interactive learning environment. http://staffprofiles.humanities.manchester.ac.uk/Profile.aspx?Id=paul.middleditch

Will Moindrot

Will Moindrot is an educational technologist at the Liverpool School of Tropical Medicine. He has worked within the field of technology-enhanced learning for 10 years, previously working for City University London and the University of Manchester. His research interests include classroom interaction tools, media technologies and the effect of these as a change agent to support professional development in higher education roles. http://uk.linkedin.com/in/willmoindrot

Appendix A

Table

	What type of device did you use to participate in the interaction lectures?
	2011/2012			2012/2013				2013/2014
Sample	62			119				78
Mobile	24	39%		74		62%		61		78%
Laptop	22	35%		14		12%		11		14%
Tablet	7	11%		28		24%		6		8%
Clickers	9	15%		3		2%		0		0%
		1. Overall, I would rate this unit as being excellent
		2011/2012			2012/2013				2013/2014
ECON10042		4.34/5.00			4.12/5.00				4.57/5.00
ECON20401		4.50/5.00			4.32/5.00				4.65/5.00
		2. The feedback that I received on my work was helpful
ECON10042		4.25/5.00			3.80/5.00				4.28/5.00
ECON20401		4.17/5.00			3.75/5.00				4.43/5.00
		3. My lecturer Paul Middleditch was excellent
ECON10042		4.53/5.00			4.66/5.00				4.73/5.00
ECON20401		4.63/5.00			4.69/5.00				4.85/5.00