Engagement with Online Pre-exam Formative Tests Improves Exam Performance and Feedback Satisfaction

Abstract

The National Union of Students (NUS) National Student Experience Report identified examination feedback as an area where students had particular concerns. This finding was echoed in the authors’ institution and triggered an action research project to investigate ways of improving students’ perceptions of pre- and post-exam feedback. We report the results of part of the project aimed at improving student satisfaction with pre-exam feedback and preparedness for the end of module examination, where we used an ‘Assessment for Learning’ (AfL) strategy. Students on a second year human genetics module (LSC-20050) were supplied with a series of 10 formative online tests with instant feedback. Our results show that students who completed more of the online tests performed significantly better in the end of module exam than students who completed < 4 tests. Comparing the LSC-20050 exam results with other bioscience modules indicates that the students who took fewer tests did no better in LSC-20050, but the students who did more tests showed significantly enhanced performance in LSC-20050. Students who did the tests reported that they found the tests a useful way to learn and felt better prepared for the exam. Thus, engagement with online assessments can improve pre-exam feedback satisfaction with the added bonus of improving exam performance.

Keywords:

• pre-exam
• feedback
• online tests
• formative

Introduction

It is now well established that students can take different approaches to learning, described as the surface (internalising/reproducing), strategic (achieving) and deep (utilising/meaning) approaches (Marton & Säljö 1976, Biggs 1979, Entwistle et al. 2001). The students who do best are those who can self-regulate their learning. Pintrich & Zusho (2002) stated that:

‘Self-regulated learning is an active constructive process whereby learners set goals for their learning and monitor, regulate, and control their cognition, motivation, and behaviour, guided and constrained by their goals and the contextual features of the environment.’

An aim of higher education is to produce individuals who have not only amassed factual knowledge in a particular discipline but who are also independent thinkers, equipped with the skills to find meaning in, and make critical judgements of, new information and assimilate it with their existing knowledge. To achieve this, our curricula, teaching and assessment strategies must help facilitate the self-regulatory and deeper learning approaches through which students can develop these attributes.

There is considerable evidence in the literature that student learning is driven primarily by assessment (Snyder 1971, Miller & Parlett 1974, Sambell & McDowell 1998) and that student effort is concentrated around the summative assessment period (Brown et al. 2003, Gibbs & Simpson 2004, Chevins 2005). According to Gibbs & Simpson (2004):

‘Exams can have the effect of concentrating study into a short intense period at the end of the course with little study of, for example, lecture notes, until many weeks after the lecture.’

They suggest that regular formative tests can help to distribute student effort across the course and develop self-regulated learning. Formative assessment does not contribute to the module mark, but is intended to generate feedback to help improve future performance and accelerate learning (Sadler 1998). The concept of using assessment as a learning tool has been referred to as either ‘Assessment for Learning’ (AfL) (Gipps 1994) or as ‘Assessment as Learning’, a term used by Earl (2003) to encompass students’ active participation in directing their own learning and engagement in self-assessment. Chevins (2005) exploited this approach in a study where, over three-week cycles, he replaced formal lectures in animal physiology with prescribed reading. Students then tested their knowledge in a paper-based formative short answer test for which generic feedback was supplied, followed by a summative test (containing some of the same questions) a week later. Over a three-year period, students reported increased study hours (concentrated primarily around the summative tests) and showed improved marks in the end of module essay-based exam (Chevins 2005). Other examples where engagement with formative assessment has led to improved performance in the summative assessment include: economics students being given the opportunity to take an online practice test (Sly 1999); health science students from four Spanish universities taking a mid-term formative assessment (Carrillo-de-la-Peña et al. 2009) and physiology students attempting a practice essay (Carvalho & West 2011).

The evidence is clear that engagement with formative assessment is effective at improving performance, but is it improving learning or self-regulation skills? McDowell et al. (2011) carried out an in-depth analysis of the effects of AfL approaches at their institution. By analysing the results of a questionnaire they developed, they showed that in modules where an AfL approach was taken, students were more likely to take deep approaches and reported an overall more positive learning experience. They also demonstrated that the student experience was centred on staff support and feedback, active engagement, interaction with peers and module design.

The format of the formative assessment can play a part in encouraging surface or deep approaches to learning and thus affect the efficacy of AfL strategies. Biggs & Tang (2011) stated that multiple-choice questions (MCQs) and short answer assessments almost invariably encourage a surface approach. However, we must beg to differ on this point, especially as both the examination and formative online tests we have developed for this study are of the short answer variety. We believe that carefully designed short answer questions can be used to differentiate students who have taken surface approaches and those who have deeper level understanding. Indeed, Gibbs & Simpson (2004) proposed a set of conditions whereby formative assessment could support self-regulated learning and each of these conditions has been addressed in the design of the online tests for this current study.

There is an intimate link between assessment and feedback. Even if students have well-developed self-regulation skills, in order for them to fully judge how they have performed in a task or assessment and how they can improve, they need some form of external feedback. The evidence that quality feedback, when properly acted upon by the student, can enhance student performance is overwhelming (Higgins et al. 2002, Glover & Brown 2006, Weaver 2006, Hattie & Timperley 2007, Poulos & Mahony 2008). Indeed, in an extensive meta-analysis of 87 studies on factors affecting student achievement, feedback emerged as the most powerful single influence (Hattie 1987). Nicol & Macfarlane-Dick (2006) distilled seven key principles of good feedback practice showing feedback as a dynamic process that should elicit responses in the student, not just as a delivery of what is right or wrong with a students work. Feedback that is simply corrective is unlikely to have any influence on student learning (Fazio et al. 2010). The student does however have to interact with the feedback and we do not really need the confirmation in the literature to know that students do not always read feedback or try to understand it (Hounsell 1987, Ding 1998, Lea & Street 1998) – the piles of uncollected work speak for themselves! In this study, the feedback is presented immediately upon submission of the online tests, which should hopefully encourage the students to engage with it.

Unfortunately, despite the excellent advice of Nicol & MacFarlane-Dick (2006), the National Student Survey (NSS) showed that feedback is an area of the educational experience where students across the sector feel under-supported (Unistats 2012). This is perhaps not surprising as research has shown that there is often a gap between student and tutor perceptions of what feedback is (Bevan et al. 2008) and how useful coursework feedback is (Brown & Glover 2006, Lizzio & Wilson 2008, Holmes & Papageorgiou 2009, Bailey & Garner 2010, Ferguson 2011, Orsmond & Merry 2011). A criticism of the questions relating to feedback in the NSS was that they were too vague to give any real indication of the root problems. Through the Student Experience Report (National Union of Students, NUS 2011) and the Feedback Amnesty (Porter 2010), the National Union of Students (NUS) has gone some way to analysing students’ feedback experiences in-depth and suggesting what students can reasonably expect from feedback. One thing that emerged was that examination feedback was an area of extreme dissatisfaction for students, something that has not been specifically addressed to any great extent in the literature. This was echoed in a survey carried out in the authors’ institution (School of Life Sciences (SLS), Keele University) and was the trigger for an action research project, which was designed to explore ways of improving students’ perceptions of, and the learning impact of, pre- and post-examination feedback. In this report, we are specifically evaluating the effects of a pre-exam feedback AfL intervention on a second year human genetics module, namely the introduction of a series of pre-exam online tests with instant feedback. We show that the students who fully engaged with the tests felt more supported and performed better in the end of module examination than the students who chose not to do the tests. Moreover, by comparing the marks achieved on other modules, there is evidence that although we are detecting a ‘better student’ effect, those better students have significantly enhanced their performance in the human genetics module.

Methods

Setting up the question bank

Keele University uses BlackBoard 9 (BB9) for its virtual learning environment (KLE). The question bank of about 60 questions was built taking advantage of the wide variety of question types available in BB9, such as MCQ, fill in missing words, multiple response, numerical, text based and matching. At the time of question input, there is also the opportunity to input general response feedback for the whole question and feedback specific to a given answer choice, as appropriate. Many textbook publishers supply banks of test questions to accompany their books and some of our questions were adapted from these. However, the majority of questions were written ab initio. The questions were not necessarily the same type as would appear in the exam paper, meaning that surface and strategic learners could not just memorise the answers. Questions were designed to test whether students: had acquired essential factual knowledge; showed evidence of understanding; could apply the numeracy skills to different situations and could test a hypothesis. Some sample questions are supplied in supplementary file 1.

The questions were organised into tests that became available immediately after the appropriate teaching session. As soon as the student began the test it was shown as ‘In Progress’ in BB9 Grade Centre and the mark was shown upon submission of the test. Some of the free-text questions required manual marking and by checking Grade Centre every few days, tests that needed marking could be identified.

Ethical review

Ethical review was carried out with reference to the British Educational Research Association (BERA) Ethical Guidelines for Education Research (BERA 2011) and submitted to the ethics review committee, along with a brief project proposal and the information sheet and consent form that was going to be supplied to the students (see supplementary file 2 – note that only the pre-exam interventions are being discussed in this report).

The study cohort

Students on a second year human genetics module (LSC-20050) were invited to participate in the project. As S.A.H. is responsible for all the teaching and assessment on this module, the AfL approach (lectures are supported by problem solving workshops) and promotion of online activities (e.g. students produce a wiki page) should render the students receptive to online learning activities. The cohort of approximately 90 students is drawn from three of the five degree programmes offered in the School of Life Sciences: Biology (B) and Human Biology (HB) which are both dual honours programmes and single honours Biomedical Science (BMS), giving a wide representation of the SLS student population.

Inviting students to participate

Project information sheets and consent forms were distributed in class. The project was also described verbally in detail and students were given the opportunity to take part in as much or as little of the project as they liked. Fifty-two students, with representatives from all three subject cohorts, wished to participate in the project, 46 of whom consented to having their test and exam scores analysed. Attendance was significantly reduced on the day the consent forms were distributed (less than 75% of the 92 cohort), which may have had an impact on the number of students who agreed to participate in the project.

Collecting and analysing survey responses

The post-exam survey to determine students’ perceptions of the online tests was hosted on Survey Monkey (http://www.surveymonkey.com/). The advantages of using Survey Monkey are that responses are anonymous, there is a range of standard tried and tested question formats (e.g. Likert Scale, multiple response, single response, free-text response), results are collated online and can be downloaded, a link can be posted on the KLE page and for small surveys (< 10 questions, < 100 participants) it is free. The questions specifically relating to this part of this study are shown in supplementary file 3.

The link to the survey was placed in the same folder as the exam feedback and the responses were collected over a four-week period.

Collecting and analysing test data

BB9 Grade Centre automatically records test scores and the number of attempts (with dates). The whole cohort’s data were downloaded to an Excel spreadsheet and the non-participants’ data were deleted. Exam marks were collected for LSC-20050 for all participants and for other bioscience modules only for the 26 students who had taken > 3 bioscience exam-based modules. This decision was taken due to the diverse nature of the student cohorts e.g. some B or HB dual honours students may only take one other exam-based bioscience module, whilst others and the single honours BMS cohort will have taken up to six. The student names were removed to anonymise the data and the participants were grouped into categories depending on the number of tests they had completed.

All graphs were produced in Excel (Microsoft Office 2007) and statistical analyses were performed in Minitab®16 (http://www.minitab.com). Pearson Correlations were calculated to explore relationships between the number of tests attempted and exam scores, and also between mean test scores and exam scores (whole cohort and subject cohorts). Two-sample two-tailed t-tests were used to determine if the differences between the categories were statistically significant, thereafter one-tailed t-tests was used to test the hypotheses such as ‘is the exam mean lower in the 0–2 test group than the 8–10 test group?’. Comparisons were also made between the exam scores in LSC-20050 and other modules using one-tailed paired t-tests to test the hypothesis that the mean score in LSC-20050 is higher than the mean score in the other module exams.

Results

Timing, number of tests completed and number of attempts at each test

Students had access to one compulsory test and 10 non-compulsory tests, which became available on the KLE the week of the relevant teaching session. They could complete the tests whenever and as often as they wanted. Table 1 shows a snapshot of activity at week 6, when a total of five tests were available to the students. It is clear that activity drops off as the semester progresses with only four students having looked at test 5. By the end of the module, 38 of the 46 participants completed one or more of the non-compulsory tests, with most students attempting the tests in the week before the exam. Most of the students completed the tests only once, although about 25% of the students had multiple attempts (up to four) at one or more of the tests. It tended to be the same students who had multiple attempts at the individual tests. One student completed two tests four times and the marks improved from 72% to 87% to 96% to 100% in one test and from 31% to 44% to 84% to 100% in the other.

Table 1 Snapshot of test activity during the first six weeks of semester

Completion status	Test 1 Compulsory	Test 2	Test 3	Test 4	Test 5
Number of tests completed	44	15	7	5	0
Number of tests In progress	1	6	1	2	4

What did the students think about the tests?

It was important to ask the students how they used the tests and reflect upon how useful or otherwise they found them and this was addressed in the post-exam online survey (supplementary file 3). The responses (n = 19) are summarised in Table 2. Darker grey cells represent questions about level of engagement, timing of tests, etc., the lighter grey cells are questions about the perceived usefulness of the tests and the white cells are miscellaneous questions. Free text responses are shown in the bottom cell.

Table 2 What did the students think of the tests?

Response	Percentage
I have done some or all of the revision tests.	100
I did one or more of the tests more than once.	47.4
I did the revision tests within a week of the lecture.	15.8
I did the revision tests within a month of the lecture.	26.3
I did the revision tests when I was revising for the exam.	94.7
I did not do the tests.	0
I prefer to do all of my revision just before the exam, rather than learning the material when it is delivered.	15.8
I find it easier to revise for the exam, if I have ensured my understanding of the material when it is delivered.	68.4
I only did the tests that backed up lectures that I understood less well.	0
If I understood the lecture, I did not feel it was necessary to the test.	5.3
I felt better prepared for the examination having done some or all the tests.	94.7
The tests help re-enforce my understanding of the lecture material.	89.5
The tests made me aware of material that I did not understand from the lectures.	68.4
The instant feedback helped me see where I had gone wrong with the questions	100
I found the tests a useful way to learn.	94.7
The standard of feedback was better in this module than other modules.	100
They do not count towards the module mark, so what is the point doing them?	0
I am more likely to do a test if it is online than if it is in the module handbook.	78.9
‘It was a very easy way to see how my revision was progressing by first doing the test and getting the mark, then redoing it after the revision of that lecture checking the mark and the feedback on the questions.’
‘The online tests were very useful in directing my learning and revision, highlighting areas that I lacked knowledge.’

Is there any evidence of improved exam performance in students who completed the tests?

To determine if there was any relationship between the number of non-compulsory tests attempted and the exam score, a scatterplot was produced, plotting the LSC-20050 exam score against the number of tests each survey participant had completed (Figure 1). There was a significant positive correlation (r = 0.681, P < 0.001, n = 46), with the exam score likely to be higher in students who had completed a greater number of tests.

Figure 1 Relationship between the number of tests attempted and exam performance. Scatterplot to show how the exam score relates to the number of tests attempted. The exam scores were plotted on the y-axis depending on the number of tests completed (x axis). The whole survey cohort is included (n = 46). There is a significant positive correlation (r = 0.681, P < 0.001), with exam performance better in the students who did the most tests.

Having confirmed a relationship between the number of tests attempted and the exam score, the data were examined more closely, placing the students into categories depending on the number of tests they had completed (0 versus > 0; 0–4 versus 5–10; 0–2 versus 8–10). The numbers in each category were: 0, n = 8; > 0, n = 38; 0–4, n = 23; 5–10, n = 23; 0–2, n = 17; 8–10, n = 17. Figure 2 shows the mean exam scores for the students in each category for the whole cohort. Two-sample t-tests were performed to determine if there were significant differences between the exam scores in each of the categories. The 0 tests mean was significantly lower than the > 0 tests mean (P = 0.009). For the 0–4 tests versus 5–10 tests and the 0–2 tests versus 8–10 tests comparisons, the former mean was significantly lower than the latter mean in each case (P < 0.001).

Figure 2 Mean exam score versus the number of tests attempted. Column chart showing the mean exam scores for students grouped into categories based on the number of tests attempted. Note that some of the categories overlap (e.g. the 0 test group is included in the 0–2 test group, which is itself included in the 0–4 test group). Numbers in each category: 0, n = 8; > 0, n = 38; 0–4, n = 23; 5–10, n = 23; 0–2, n = 17; 8–10, n = 17 Error bars are ± SE.

The data were then examined at the level of subject cohorts – BMS versus HB and B pooled. HB and B cohorts were pooled because these two cohorts have completed a genetics module in the first year, whilst the BMS students have no prior exposure. For the HB and B students pooled, the means for the 0–4 tests and 5–10 tests were 58.7% [n = 12, standard error (SE) 3.9] and 72.6% (n = 16, SE 3.6), respectively, which are significantly different (P = 0.016). For the BMS students 0–4 tests and 5–10 tests had means of 59.3% (n = 11, SE 4.5) and 78.6% (n = 7, SE 1.4), which were significantly different with P = 0.002. There was no significant difference between the 0–4 test means for HB/B and BMS (P = 0.914) and although the mean for BMS 5–10 tests was higher than the mean for HB/B 5–10 tests, this was not significantly different (P = 0.15).

It is clear from Figures 1 and 2 that the students who took more of the tests did better in the exams, but are there any confounding factors, such as attendance or the ‘better student’ effect?

Does attendance at teaching sessions have an effect exam performance?

To determine whether attendance had an effect on exam score, the participants were grouped by their attendance levels and the exam scores were compared. All the study participants attended between 9 and 11 sessions out of a possible 11. As only three students attended nine times their data have been pooled with the seven students who attended 10 times in Figure 3. There is no significant difference between the mean marks in the 9–10 and 11 groups (P = 0.749).

Figure 3 Relationship between number of sessions attended and exam score. Column chart showing the mean exam scores of students grouped by attendance. For 9–10 times n = 10 and for 11 times n = 36. Error bars are ± SE.

Are we detecting the ‘Better student’ effect?

It may be the case that the better students are more likely to have taken advantage of the online tests to help their revision and would have performed well in the exam anyway. The scatterplot in Figure 4 shows a positive correlation between test score and exam score showing that the students who did better in the tests also did better in the exam (r = 0.431, P = 0.007). A paired t-test to compare mean test scores (60.9%, n = 38, SE 2.3) with mean exam scores (69.5%, n = 38, SE 2.3), showed a significant difference (P = 0.001), indicating that students have done better overall in the exams than in the tests.

Figure 4 Relationship between mean test score and exam score. Scatterplot to show how the mean score in the tests relates to the exam scores. Data are only included for students who did > 0 test (n = 38). There is a statistically significant positive correlation (r = 0.431, P = 0.007) between the mean test mark and the exam performance.

The only way to test whether doing the tests improved a good student’s performance over and above what they might have been expected to achieve is to look at their performance in other modules. Given the diverse nature of the whole cohort, only students taking > 3 exam-based LSC-code modules were included in this analysis (n = 26). Figure 5 shows the mean exam mark for LSC-20050 and the mean exam mark for the other modules. At 67.9% the mean exam score for LSC-20050 was higher than the mean exam score for the other modules 61.3% and a paired t-test shows that this is significant (P = 0.006). Did students perform better in LSC-20050 because the exam was easier? In Figure 6, students were grouped into two categories – those who had taken up to four tests, or those who had taken 8–10 tests (no students took 5–7 tests). Between the 0–4 and 8–10 categories there were significant differences in the exam scores, irrespective of whether we were looking at LSC-20050 (P < 0.001) or the other modules (P = 0.004), indicative of a ‘better student’ effect i.e. it is the better students who have availed themselves of the tests. However, when one-tailed paired t-tests were used to compare the differences within the categories, the LSC-20050 exam scores were not significantly higher than the other module exam scores in the 0–4 tests category (P = 0.187) but importantly, in the 8–10 tests group, the LSC-20050 scores were significantly higher (P < 0.001). This suggests that the reason that the LSC-20050 exam mean was higher than the other module exam means was due to the inflated scores of the students who carried out 8–10 tests and not because the exam was easier.

Figure 5 Comparison of exam score for LSC-20050 with other modules. Column chart showing the mean score for LSC-20050 and the mean scores for other LSC-coded modules for students who took >3 LSC-coded exams (n = 26). Error bars are ± SE.

Figure 6 Comparison of mean exam scores for LSC-20050 and other modules versus number of tests attempted. Students were grouped into two categories dependent on the number of tests they completed and the LSC-20050 scores (dark blue bars) were compared with the mean exam scores of other bioscience modules (light blue bars). For the 0–4 tests group n = 15 and for the 8–10 group n = 11. Error bars are ± SE.

Discussion

The aims of this study were to determine whether engagement with online tests providing instant feedback could improve human genetics students’ perception of pre-exam feedback and improve exam performance.

How are students using the tests and what benefits do they feel they have had in completing the tests?

The students had access to a series of 10 formative online tests covering material from the teaching sessions. Provision of extra questions for this module is not a new idea. Until this year, about 20 additional questions were supplied in the module handbook, with the answers being released on the KLE page during the revision period. Given that the students also had the questions in their workshop handbook, we were rather surprised by a comment in last year’s module evaluation saying that there were not enough practice questions. This made us question as to whether the students actually knew that they were there and was the main reason we decided to put the questions on the KLE page. Interestingly, in the survey completed by the project participants at the end of the module, 78.9% of students indicated that they were more likely to do tests if they were available online as opposed to being in the module handbook (Table 2). The online tests included many more questions than previous years and provided instant feedback. The tests were badged as pre-exam feedback to get the students into the mindset of thinking that revision tests are feedback.

Data were collected on the test and exam scores, and on when, how many and how often the participants completed the tests. Students were invited to complete an online survey post-exam and one of the questions asked how they had used the tests and what benefits they thought completing the tests might have had. We had hoped that students would attempt the tests within a week of the lecture to help embed the material they had just covered, but as can be seen from the snapshot of test activity taken at week six (Table 1), test activity drops off as the semester proceeds. Whilst 44 of the 46 students had completed the compulsory test in week one, only four students had even looked at test five by the end of week six. These data correlate well with the survey results shown in Table 2 – approximately 16% of the participants looked at the tests within a week of the lectures and just over a quarter within a month. However, 95% of the survey participants said that they completed the tests during the revision period. Why the majority of students did not complete the tests earlier is unclear. For some, it may just be that they have assessments to complete for other modules and so do not have the time to do the tests or it may just be that the tests are formative and the students are behaving as predicted by Chevins (2005) i.e. concentrating their efforts on preparing for the summative assessment. A paired t-test comparing the mean score in the tests versus the mean score in the exam shows that the students do better in the exam suggesting that the students are indeed working harder for the exam.

Interestingly, 68% of students said that they find their revision easier if they try to understand the course material when it is delivered and only 16% said that they prefer to do their revision just before the exam, rather than learning the material at the time it is presented. This does not tally with the timing of taking the online tests and suggests that many of the students are using the tests strategically as a revision tool, rather than, in the true essence of AfL strategies, to help them identify gaps in their knowledge at an earlier stage i.e. just after the lecture. Only about 25% of the students did > 1 of the tests more than once, but several students had multiple attempts at the tests and with each attempt, the score increased. The free comments are shown in Table 2.

‘It was a very easy way to see how my revision was progressing by first doing the test and getting the mark, then redoing it after the revision of that lecture checking the mark and the feedback on the questions.’

and

‘The online tests were very useful in directing my learning and revision, highlighting areas that I lacked knowledge.’

These comments suggest that at least these students were self-regulating by using the tests to identify gaps in their knowledge, then going back to revising the material and then re-attempting the test. As Boud (2000, p158) states

‘The only way to tell if learning results from feedback is for students to make some kind of response to complete the feedback. This is one of the most often forgotten aspects of formative assessment. Unless students are able to use the feedback to produce improved work, through, for example, re-doing the same assignment, neither they nor those giving the feedback will know that it has been effective.’

Despite the fact that most of the test activity was during the revision period, the survey results show that the students did perceive the benefits of the tests. All of the students saw the value in doing the tests, even though the tests did not themselves contribute to the module mark. When asked to compare the pre-exam feedback on LSC-20050 with other modules, 100% rated the LSC-20050 feedback better, a significant improvement in satisfaction. They found the tests a useful way to learn (94.7%), to help re-enforce their understanding (89.5%) and to highlight areas of incomplete understanding (68.4%). Furthermore, 100% of the survey respondents thought that the instant feedback helped them see where they had gone wrong. Importantly, they reported that doing the tests had helped them feel better prepared for the exam. Falchikov & Boud (2007) wrote about the influences of emotion on learning and in this instance we may be seeing the effects of positive emotions (feeling prepared) on performance. Even if there was no evidence that the tests actually helped improve performance, the fact that the students perceived them to be beneficial makes us feel that the tests have been a successful intervention.

Evidence that engagement with the tests improves exam performance

In order to determine whether taking the online tests helped exam performance, the exam scores were plotted against the number of tests the students had performed and as the scatterplot showed a clear positive correlation, the students were grouped into categories according to the number of tests performed. In accordance with earlier findings (Sly 1999, Chevins 2005, Carrillo-de-la-Peña et al. 2009), those students who did more tests did better in the end of module exam and that this effect was seen at the whole class level and at the individual subject cohort. It was important to look at the level of the subject cohort because the HB/B and BMS students have had very different pre-exposure to genetics teaching, with HB/B students having already done a first year genetics module. HB/B students may therefore have a more solid grounding in genetics and may be expected to do better. There was no significant difference, however, between the cohorts. The results in Figures 1 and 2 do not demonstrate that it was (directly) the engagement with the tests that improved the exam scores – there may be a number of confounders. Firstly, it may be that the students who have done more tests have attended more of the teaching sessions, but as shown in Figure 3, no difference can be attributed to attendance. Possibly, we are detecting the ‘better student effect’ – the students who did more of the tests are ‘better students’, who would have done better in the exam anyway. The average exam scores show a positive correlation with the test scores (Figure 4) i.e. the weaker students are not out-performing the stronger students and academic rankings are being maintained. Sly (1999), suggested that his results did not indicate the ‘better student’ effect, in that many of the weaker students opted to take the tests, whereas some of the best students did not. We are not entirely convinced by his argument, especially as he does not compare performance across other modules. Although he may have considered his students weaker academically, they clearly had aspirations to improve their scores by practising and did indeed improve their performance. There is evidence in our study that some ‘better’ students (scores of > 70) did not do the tests (Figure 1). These may be students who are already confident in their own ability and do not feel the need for extra practice. The critical point is that students who did not do the tests did no better in LSC-20050 than they did in their other modules, whilst the students who took 8–10 tests did significantly better (Figure 6). Figure 5 shows that at the level of the whole cohort, the students did better in the LSC-20050 exam, which may just indicate that the LCS-20050 exam was easier, or that the students have understood the module better. We cannot judge the latter point, but the results in Figure 6 indicate that the ‘easy exam’ hypothesis is very weak. When split into 0–4 tests and 8–10 tests, there is no difference in the mean exam scores for LSC-20050 and other modules in the 0–4 test group, but there was a significant difference in the group that did 8–10 tests. If the exam was easier we would have expected to see better performance in both groups. We believe there is evidence of the ‘better student’ effect in our study – Figure 6 also shows that the students who did 8–10 tests did significantly better in their other modules than the students who did 0–4 test. However, the results support the interpretation that by doing the tests, those ‘better’ students have improved their performance in LSC-20050 over and above expectations and it was their inflated exam scores that are responsible for the observed increase in exam score at the level of the whole cohort.

How suitable is this approach for bioscience teaching?

The online tests used in this study were seen as pre-exam support for a short answer based exam and although some of the calculation style questions were of a similar format in both the online tests and exam, many of the question styles used for the online tests (e.g. true/false, matching, etc.) were not mirrored in the end of module exam. However, our results clearly show that full engagement with the tests improved exam performance and so we would certainly recommend a similar AfL approach to help students prepare for short answer exams. However, there are many different bioscience examination formats in our own school and other institutions (e.g. MCQ, short answer, essay, combination of short answer and essay, paper comprehension). Could similar short answer tests help improve performance in more diverse exam formats, such as essays? We are currently investigating this with essay-based exams and there is some evidence in the literature that engagement with short answer tests can help improve performance in essay style exams (Chevins 2005). Whilst attempting short answer tests is not direct practice for writing an essay, carefully designed questions can help students judge their knowledge acquisition and understanding and identify areas where they need to concentrate their revision efforts. It should be possible to design questions specific to key required reading e.g. the questions may be based on the data presented in a research paper. This will help students go into their exams with greater confidence that their knowledge base is sound.

Writing suitable questions and building a test question bank can require a big up-front time commitment, depending on the number of questions required. However, once created, the questions can be shared with other modules and little or no tutor input is required as the feedback is provided automatically when a student submits their attempt.

Do the test questions have to be delivered online? Not necessarily, but our results show that students are more likely to do tests online than in hard copy. Our students have easy access to the virtual learning environment (BB9) and the tests are conveniently placed in a folder containing all the links to the relevant teaching materials. More importantly, feedback and grades are provided instantly and staff and students can monitor progress in the BB9 Grade Centre (or the equivalent in other virtual learning environments).

Conclusions

Students who engaged with online pre-exam tests were more satisfied with pre-exam feedback, felt better prepared for the exam and improved their performance in the end of module exam.

Acknowledgements

The results presented in this paper are part of an action research project carried out by S.A.H. as a credit-bearing module for an MA in Learning and Teaching in Higher Education. Thanks to Lin Norton and Jackie Potter, who supervised the action research module. Special thanks to the 2011 LSC-20050 cohort, whose participation was fundamental for this project.

Appendix: Supplementary Data

Supplementary file 1 – sample questions

Supplementary file 2 – Student information sheet and consent form.

Supplementary file 3 – relevant post-exam questions