The use of virtual learning environments and their impact on academic performance

Abstract

Links have been reported in the literature between lecture attendance and academic performance. The effect of a student’s use of a virtual learning environment (VLE) on their academic performance has not been the subject of extensive research, but is likely to be of interest to both students and the lecturers who develop resources for a VLE. This paper presents an examination of two modules from the 2010/11 final year BEng Civil Engineering course at Loughborough University. Data from the students’ academic performance is compared with the VLE’s usage logs in order to identify relationships. Correlations differed for the two modules but were generally weak.

Introduction

Virtual learning environments (VLE) have recently emerged as an important topic in education theory and practice (Weller, 2007). In theory, a well-maintained VLE should enable students of all learning styles to receive the best possible education, in a way that they may not in an exclusively lecture-based environment which tends to be focused on auditory learners only (i.e. those who learn best by listening (Williams and Fardon, 2005; Vigentini, 2009)). If the resources on a VLE do not cater to the needs of the students - both in terms of their format (text files, audio files, videos etc.) and their content - the VLE is effectively rendered useless as it does not add to the students’ learning experience. Ideally (for co-located students rather than distance learners) it should contain just enough information to allow students to reinforce their work in lectures and gain a broader and deeper understanding of the subject (Ofsted, 2009), preferably through a range of presentational styles (such as video or audio).

A student’s experience in the classroom or lecture theatre depends on how well the teaching style fits with their individual learning style. The same issues exist with virtual systems and a variety of resources is needed in order to provide adequately for all learning styles. Many higher education institutions are increasingly fragmented and lacking in cohesion or unity (Whitworth, 2005) and regard VLEs as a way to help overcome the problems introduced by the national increase in the number of students coming to university and the higher workload that this entails for lecturing and support staff (Richardson, 2001). There is, however, a danger that, if the resources available online are too comprehensive, students could cease attending lectures (Bromage, 2003), preferring instead to access lecture materials online and to carry out their studies from the comfort of their homes (Boyle et al., 2008). It can therefore be said that there are three types of VLE: those with too much information (which discourage students from attending lectures), those with too little (rendering the VLE largely useless), and those with just enough to allow students to reinforce their work in lectures and gain a broader understanding of the subject (Ofsted, 2009).

This paper aims to link students’ academic performance data with VLE usage logs. Although direct causality may not be inferred between VLE use and academic performance, any correlations between VLE use and academic performance for different classes of students or different types of modules can yield insights into the effective use of VLEs. Two modules from the final year of the BEng Civil Engineering programme at Loughborough University were studied: the Design Project and Geotechnics 3. Using students’ VLE usage logs and their corresponding final grades for the modules (both coursework and examinations), evidence was sought for the existence of a link between their use of the VLE and their academic performance in each module. Two interviews with current students were also conducted in order to ascertain whether the quantitative data aligns with student views.

Related research

Links have been established between lecture attendance and academic performance in fields as diverse as biological sciences (Gatherer and Manning, 1998), economics (Stanca, 2006), geography (Stewart et al., 2011) and psychology (Gunn, 1993). On the other hand, some research argues against relying on lectures alone. Lake (2001) found that physical therapy students in an “active” learning environment performed better than those in a more passive, purely lecture-based environment, whilst St. Clair (1999) questions some of the research reported in the literature and identifies the case against compulsory lecture attendance.

The emergence of VLEs has added momentum to pedagogical research on the use of information technology. The few long-term studies of their usage focus on the benefits of a VLE to a learning institution or the pedagogical deficiencies they help to overcome (Coates et al., 2005).

Regardless of how well structured a VLE is, there will always be a difference in how diverse students use the resource. In a study of 209 students at Coventry University, it was found that 57% of the students read learning materials, 52% downloaded learning materials but only 14% accessed external web pages (Bromage, 2003). This implies that students tend to look at VLE resources simply as online versions of lecture notes rather than as a source of additional information to further their understanding. Mimirinis and Bhattacharya (2007) also note the challenge of designing a VLE for deep learning. Von Konsky et al. (2009) report that uploading audio recordings of lectures to the VLE does not impact on lecture attendance and students who attended lectures and listened to online recordings tend to perform better. The positive impact of multimedia educational content is also indirectly reported by Issa et al. (1999) and Porta et al. (2008). The more general question of the link between VLE use and academic performance has received less direct attention. Maltby and Mackie (2009) analysed links between academic performance and VLE “click count tracking data” in a group of Business School undergraduates. The introduction of the VLE was found to have mixed effects. Lee et al. (2009) observed a clear positive correlation between VLE use and academic performance in the study of Japanese language. Similar results were reported in psychology (Beaman and Harvey, 2005) and medicine (Johnson, 2008).

Research methodology

Assessment records for the Design Project and Geotechnics 3 modules in the academic year 2009/10 were anonymised by a member of academic staff who assigned unique identifiers to each student. The same identifiers were also applied to render the VLE usage logs anonymous. Each student’s sex and the first two characters of their university registration number (denoting the year they joined the university only) were retained during the process of data preparation. This differentiated those students who had completed an industrial placement before taking the modules from those who proceeded directly from the second to the third year. The Design Project module had 56 students in total; Geotechnics 3 had 101. Of those cohorts, 54 students took both modules. The instructors from both modules adhered to the university’s policy of making as many resources as possible available on the VLE and encouraging (but not mandating) VLE use by the students.

For all visitors to the VLE over the duration of the module, the VLE logs detailed the action taken (e.g. looking at “course view”), the date and time at which the action was taken, what information was accessed and the IP address of the user. In order to establish the number of visits of each student, the dates and times of their actions were grouped together in blocks. If more than 30 minutes elapsed between actions, the student was considered to have ended a visit and any subsequent actions would be counted as a separate visit. Not all actions taken were views of resource material posted to the site by a lecturer. Other actions, such as looking at the module homepage ("course view") or at the list of students enrolled on the module ("user view"), were included in the analysis, despite not necessarily aiding the students academically. The definition of an “action” was therefore refined to include only resource, forum and user views and exclude course and blog views. These were the most commonly occurring actions and were judged to be those which could most benefit the students. When displayed graphically it was clear that few students used either “forum view” or “user view” and that by far the most common action was “resource view”. Although forums are generally considered to be a useful tool (Stimson, 1997; Crook, 2000; Bromage, 2003), the low number of students using them and low frequency of use (no student viewed the forums more than three times in Geotechnics 3 and 11 times in the Design Project) lead to the conclusion that little useful information is being conveyed via the forum pages and that they therefore have a negligible impact on students’ academic performance. This low use of forums is not unique to these modules and has been observed in different studies (Bromage, 2003; Crook, 2000; Wells et al., 2008). Subsequent analysis therefore considered only the number of resource views (when accessing resources such as tutorial sheets, web links or exam papers and answers).

Following on from the graphical analysis of the data, Spearman’s Rank-Order Correlation Coefficients and Pearson’s Product-Moment Correlation Coefficients were calculated. Two interviews were conducted in order to establish a link between the quantitative data and the students’ actual perceptions. A sample size of two was deemed necessary and sufficient to supplement the quantitative data with subjective, qualitative perceptions. A convenience sampling strategy was adopted, with the sample stratified by sex so that one male student and one female student were interviewed. The interviews were audio recorded and transcribed.

Figure 1. Histogram showing frequency of visits to the VLE by Geotechnics 3 students

Figure 2. Histogram showing frequency of visits to the VLE by Design Project students

Figure 3. Histogram showing marks obtained by Geotechnics 3 students

Figure 4. Histogram showing marks obtained by Design Project students

Figure 5. Number of VLE visits vs. final marks for Geotechnics 3 students

Figure 6. Number of VLE visits vs. final marks for Design Project students

Figure 7. Frequency of various types of views by students in the Geotechnics 3 VLE

Figure 8. Frequency of various types of views by students in the Design Project VLE

Figure 9. Number of resource views vs. final marks for Geotechnics 3 students

Figure 10. Number of resource views vs. final marks for Design Project students

Results and discussion

Figures 1 and 2 are histograms showing the distribution of VLE visits among students from the Geotechnics 3 and Design Project modules respectively.

It is clear from Figures 1 and 2 that few students in either module made more than 20 visits to the VLE, with an average view count of 9.7 per student for the Geotechnics 3 module and 10.7 for the Design Project. Although it may appear that the Design Project was visited more often, it must be noted that it ran over two semesters with a total of 24 taught weeks and eight weeks of holidays (Easter and Christmas), while Geotechnics 3 ran for one semester with 12 taught weeks and a four week holiday at Christmas. It could be expected that the Design Project VLE would be visited twice as often as that for Geotechnics 3 (as it is twice as long) but this is not the case. Figures 3 and 4, which show the distribution of final grades for the Geotechnics 3 and Design Project modules respectively, show that the final grades achieved in the Design Project were in a tighter grouping (with the majority of students receiving 60-70%), compared with a spread of 50-80% in the Geotechnics 3 module.

Figures 5 and 6 show the final marks against the VLE visits for Geotechnics 3 and Design Project students respectively. Figure 6 shows a moderate positive correlation between the number of times a student visited the Design Project pages on the VLE and their final grade in that module. However, Figure 5 shows a weak negative correlation between the two variables for Geotechnics 3, suggesting that the more frequently a student visits the VLE, the lower their final grade in this module becomes. Logically, this is unlikely to reflect a causal relationship, as one would expect that increased use of the VLE resources for a subject would either increase a student’s grade or have no impact at all and is highly unlikely to lower their grade. A more in-depth analysis of the data is required to establish why the Geotechnics 3 module suggested a negative correlation.

One of the most common actions taken on any visit is to look at the module homepage ("course view"), from which point a student may access any part of the VLE relating to that module. As this is merely a gateway to relevant information, and does not contain any academic resources, it was discounted from the analysis. The frequencies of the three remaining actions - resource, forum and user view - are shown in Figures 7 and 8.

Given that the vast majority of actions were resource views (which would have the strongest impact on student performance and are therefore the only views considered in further analysis), Figures 9 and 10 show the number of resource views against the final marks received for each student in the Geotechnics 3 and Design Project modules. The figures are refinements of Figures 5 and 6 respectively, this time considering resource views only.

This affects the data for Geotechnics 3 (Figures 9). There is now a slight positive trend between the two variables which is at odds with the negative trend in Figure 5. Little change can be observed between the two graphs for the Design Project (Figures 6 and 10). Additionally, Figures 9 and 10 also show that few students on either module made more than 30 resource views in total, with an average of 0.8 views per week for the Geotechnics 3 module and 0.4 for the Design Project. Once again, the difference in the number of views between the two modules is not consistent with the difference in their duration. The two modules are in some respects not directly comparable, as the Design Project is a 100% coursework-assessed module whereas Geotechnics 3 consists of both coursework and an exam.

VLE use in the Design Project appears to have a far greater correlation with students’ marks. Whilst the Geotechnics 3 module is based largely around lectures, with much of the information that the students need being given out as lecture notes or handouts, the majority of the work for the Design Project takes place outside of lectures in the students’ own time, with far less paper-based information being distributed. The Design Project students will therefore have a greater reliance on the VLE as the primary source of information. Having the majority of the required resources available via the internet allows the group members to work independently, and meet solely to “touch base” and assist each other as required. This flexibility of working is aided by other synchronous and asynchronous communication media such as email, social networking sites and mobile phones.

Following on from the graphical analysis, Spearman’s Rank-Order Correlation Coefficients and Pearson’s Product-Moment Correlation Coefficients were calculated between VLE resource views and final marks for the Design Project module marks only, the Geotechnics 3 marks only, and both sets of module marks combined, as shown in Tables 1 and 2.

The Spearman’s Rank-Order Correlation Coefficients demonstrate weak positive correlations, particularly for the Design Project. As a value of 1 indicates a strong positive correlation between the two variables, -1 a strong negative correlation and 0 no correlation, there appears to be little benefit for students’ grades in increased viewing of the VLE resources for Geotechnics 3. Although the value for the Design Project of 0.248 does not suggest a strong correlation, it does show that there is a positive relationship between the VLE resource views and final marks. Further analysis of this could suggest methods of improving this correlation, allowing the students to improve their performance in that module.

Pearson’s Product-Moment Correlation Coefficients were calculated to test whether the correlations were linear. The Pearson’s Coefficients suggest similar relationships, with a small positive correlation of 0.238 for the Design Project. A negligible correlation is also observed in Geotechnics 3. For the Design Project, the fact that the Pearson’s Product-Moment Correlation Coefficient is lower than the Spearman’s Rank-Order Correlation Coefficient implies that little correlation observed is not linear, and the linear relationships represented in Figures 9 and 10 may not be statistically significant.

Table 1. Spearman’s Rank-Order Correlation Coefficients between final marks and VLE resource views for various combinations of module marks

Marks from Design Project only	0.248
Marks from Geotechnics 3 only	0.005
Marks from both modules	0.077

Table 2. Pearson’s Product-Moment Correlation Coefficients between final marks and VLE resource views for various combinations of module marks

Marks from Design Project only	0.238
Marks from Geotechnics 3 only	0.044
Marks from both modules	0.077

Interviews

Two students were interviewed and there was some agreement between their responses. Both said that the VLE was best used as a source for additional information (rather than as a primary teaching resource) and, for modules assessed by exams, found the past papers to be the most useful resources available. However, they also noted that they tended to use them more during exam seasons, often ignoring the VLE for the rest of the semester and focusing instead on lecture notes (handouts or personally transcribed). Neither the time and date of students’ actions nor the number of views of past exam papers compared with other resources were considered in any of the above analyses, so few direct comparisons may be made. This may, however, help to explain the poor relationship between the number of resources viewed and marks attained in the Geotechnics 3 module as the majority of students are apparently not making full use of links or background information posted to the VLE and are instead relying primarily on lecture notes for their studies for most of the semester.

Neither interviewee was aware of the forums on the VLE, with both stating that they would have used them if they had been told about them at the beginning of the course. As Crook (2000) notes, many students avoid posting on forums for fear that their questions might make them appear foolish to their peers. With minimal student use of the forums, lecturers will also be less inclined to post there or check for queries and the forums can quickly become obsolete. Conversely, if anonymous posting or posting under usernames is permitted, students can feel over-secure in posting, leading to the forum boards becoming a tool for bullying, as found at the University of Edinburgh by de la Varre et al. (2005).

When it came to preferred methods of contacting lecturers with queries, one interviewee indicated a preference for talking to a lecturer face-to-face whilst the other preferred to email the lecturer, but would meet in person if further explanation of an issue was required. Inevitably, different students will have different preferences which will be affected by their individual circumstances. One interviewee mostly worked away from campus, so felt it was easier to use email to ask questions than having to travel onto campus to meet a lecturer in person.

Whilst both students found the past papers and tutorials highly useful for Geotechnics 3, they felt that the resources were better in the Design Project. One interviewee felt that this was partly because there was less contact time for this module, so lecturers had little choice but to distribute information via the VLE or email. It was also noted that the availability of the information on the VLE allowed the students to work on their coursework wherever and whenever they chose, without having to meet to share paper resources between their coursework groups.

Conclusions

The study has found that the VLE has very little effect on students’ academic performance within the Geotechnics 3 module, but a moderate effect on their performance in the Design Project in the final year of the BEng Civil Engineering programme at Loughborough University. Both the Spearman’s and Pearson’s Correlation Coefficients show negligible relationships between VLE use and academic performance within Geotechnics 3 and a small positive relationship with the Design Project.

By far the most common action taken by students visiting the VLE was “resource view”, although the study did not specifically investigate the question of the most utilised resource. The forums were used far less, although this may be due more to lack of awareness rather than an indication of their value. Both students who were interviewed reacted positively to the idea of forums and believed that they could be a useful tool. They also said that neither module in the study made use of alternate media in the presentation of resources uploaded to the VLE. It may be that if lecturers were encouraged to develop a wider range of resources for the VLE and make use of the forums as a method of answering students’ queries, the VLE could be used to its full potential and may better aid students in achieving intended learning outcomes.

A limitation of this research is its comparison of two modules with contrasting subject matter and assessment. A suggested direction for future research is to compare more similar modules, isolating the effect of VLE use and reducing other confounding factors.