Use of Feed-forward Mechanisms in a Novel Research-led Module

Abstract

I describe a novel research-led module that combines reduced academic marking loads with increased feedback to students, and allows students to reflect on and improve attainment prior to summative assessment. The module is based around eight seminar-style presentations (one per week), on which the students write 500-word ‘news & views’ style articles (short pieces highlighting new results to a scientific audience). Students receive individual written feedback (annotated electronically on the work), plus an indicative mark, on their first submitted report. For subsequent reports, only a subset is marked each week, such that each student receives feedback on two further submissions. Simultaneously, they have access to written feedback on their peers’ reports (a total of two reports per student enrolled on the module). Students are encouraged to read and apply the general and specific messages from all the feedback to their own subsequent work (using it as feed-forward). At the end of the module, students self-assess their eight submissions and select the two they believe are their best pieces to put forward for summative assessment. Combining data from three cohorts, student attainment increased throughout the module, with higher marks for the two chosen reports than for the two marked reports or their first report. Students selecting previously unmarked reports also showed a greater increase in their mark for the module than students selecting reports that had previously received a mark. Module evaluation forms revealed that the students found access to feedback on others’ work helpful in shaping their own assignments.

Keywords:

• formative assessment
• summative assessment
• feedback
• feed-forward
• workload

Introduction

Feedback plays a key role in supporting student learning (Hattie et al. 1996, Black & Wiliam 1998), and there is a wealth of guidance identifying good practice in providing feedback to students, stressing particularly factors such as timeliness, utility and the provision of ‘feed-forward’ that allows students to improve on subsequent work (e.g. Higgins et al. 2001, Gibbs & Simpson 2004, Nicol & Macfarlane-Dick 2004, Orsmond et al. 2005, Duncan 2007). The Quality Assurance Agency’s (QAA’s) Code of Practice on Assessment of Students states ‘Institutions provide appropriate and timely feedback to students on assessed work in a way that promotes learning and facilitates improvement …’ (QAA 2006).

However, students do not always engage effectively with feedback (Glover & Brown 2006). While the majority of students claim to read and think about feedback (Bevan et al. 2008, McCann & Saunders 2009), more limited numbers actively use feedback when preparing the next assignment (e.g. Orsmond et al. 2005, Glover & Brown 2006, Scott et al. 2011), perhaps because they lack appreciation of feedback as feed-forward (Chanock 2000). For students to learn, they must act on the information provided in feedback (Nicol 2010), and thus we need to encourage students to interact with feedback comments in a more active way.

For staff, the provision of timely and good quality feedback is time-consuming, and increasing student numbers make it more difficult for instructors to maintain the amount and quality of feedback and meet internal deadlines (Gibbs & Simpson 2004, Hope 2011). Consequently, there has been a reduction in the frequency of assignments and in the quality, quantity and timeliness of feedback (Gibbs & Simpson 2004). Tutors often believe that students take little or no notice of feedback (Glover & Brown 2006, Crisp 2007), which contrasts sharply with student perceptions (Bevan et al. 2008, McCann & Saunders 2009). To reduce marking loads, coursework assessment is generally used as both formative and summative assessment (Brown et al. 1997, Weaver 2006), although some evidence suggests formative feedback becomes less valuable when it is combined with summative assessment (Atkins 1995, Brown et al. 1997). Formative assessment tasks results in an opportunity for students to practice assessment types, which (combined with effective feedback) can have large positive effects on learning (Black & Wiliam 1998), particularly for some groups of students (Gibbs & Simpson 2004).

Here, I describe a module with an assessment strategy that (a) provides effective feedback to students that can be immediately applied to subsequent assignments (i.e. acts as feed-forward) and (b) reduces academic marking workloads by reducing the number of assignments that receive feedback each week (detailed in the description of the module later). I assess the effectiveness of the assessment strategy, and demonstrate that the students can use the information provided in the feedback to improve on their own work.

Description of module and assessment strategy

‘Topics in Biodiversity and Evolution’ is a final year undergraduate module (catering for approximately 32 students) designed to give students insight into the research process, and to link their knowledge and understanding of concepts developed during their degree programme to current research. Students are often unaware of the link between the research being carried out at their institution and the content of their courses (Jenkins et al. 1998, Brew 2006); this module highlights those links.

The module teaching strategy takes the form of a series of eight seminars presented by active researchers, including postdoctoral researchers (providing them with potentially valuable teaching experience; åkerlind 2005). The seminars are based on their research, including two publications that are made available to the students approximately one week prior to the session. The choice of the research is up to the speaker, providing it has been published and is available to the students, but speakers generally present research published in the last five years. Each speaker is allocated approximately an hour to present their research, in a format and at a level that can be understood by the audience, and which may include some discussion with the students. Following the presentation, a further hour is available for a question-and-answer session with the speaker. To encourage participation in this session, 5% of the module marks are associated with asking questions and answering those posed by the speaker.

Speakers are asked to chair the question-and-answer sessions and ensure that all students have the opportunity to ask questions, but are free to choose the method used to do this. During the sessions, the module coordinator notes the number of questions asked by each student during the session (the small numbers of students ensures that this is possible). Marks are awarded based on the number of sessions in which the student asked a question (50% of the available marks) and the total number of questions asked to a maximum of two per session (50% of the available marks). Thus, a student asking an average of two (or more) questions per seminar would be awarded the full five marks, and marks are rounded up to the nearest whole number. No judgment is made as to the quality of questions to encourage students to ask questions.

After each seminar, the students write a 500-word ‘news and views’ style article, aimed at a general scientific audience. Each piece is submitted two weeks after the seminar, such that each student submits a total of eight pieces of work. The first session of the module introduces the students to the key features of ‘news and views’ articles, and they are given a number of examples, together with the original paper on which they are based, to provide guidance on the coursework they are expected to produce. ‘News and views’ type articles feature in a number of scientific journals that the students should be familiar with, including Nature, Trends in Ecology and Evolution, and Journal of Experimental Biology.

The key feature of the module assessment strategy is that not all pieces of work are marked and fed back on. However, for their first submission, each student receives individual feedback annotated on to a paper or electronic copy of their work, together with a completed feedback grid (Table 1) and an indicative mark to allow them to more easily judge the quality of their work. This is returned within one week to allow them to apply the feedback to the next submission. For all subsequent submissions, I give feedback on only a subset of the work (the remainder is assessed on a pass/fail basis only; see later). I annotate a selection of reports electronically with feedback, using the comment function in Word, add the completed feedback grid (Table 1), award an indicative mark, and post to the Virtual Learning Environment for all students to see. Prior to posting, reports are checked carefully to ensure anonymity for the students, and any details that could potentially identify a student are removed. Students may also request feedback on a specific aspect of their report by asking a question identifying an area with which they would like specific help (McKeachie 2002). Written feedback comments focus on the strengths and weaknesses of the report, with specific suggestions for improvement, with the aim of enabling learning rather than judging achievement (Maclellan 2001). Again, this feedback is provided before the next submission is due. Students are expected to apply the feedback to subsequent reports, not amend previous ones.

Table 1 Feedback grid accompanying written comments. The grid is accompanied by a statement explaining the marking as follows: ‘The table below should give you an idea of the areas in which more work is needed and those that are done well in this report. Each statement below is given a score of 0 to 3 where 3 = excellent and 0 = not done at all. Note that not all of these may be relevant to all pieces of work, and the mark you receive for the work is not based entirely on these scores – the idea is to give you an indication of what is done well and where the work needs improvement.’

Score	Area
	Concise, snappy title
	Just enough background to understand the topic
	Clear statement of scientific question/hypothesis tackled
	Any specialised terms clearly and concisely explained
	Focal study introduced early in the piece
	Focal study introduced by a statement of the key findings/conclusion
	Novelty of study highlighted clearly
	Details of study findings following statement of findings
	Methodological detail limited to essential points
	Secondary findings discussed
	Mechanisms behind the findings discussed
	Article contains a concluding paragraph
	Conclusion ties back to scientific question
	Suggestions for wider implications and/or further research
	Careful use of additional referencing
	Study referenced properly in text and in the reference list
	Article written using scientific voice, avoiding chattiness/colloquialisms

Reports that are to be marked are selected at random, but with the condition that each student should receive feedback on one of reports 2–4, and on one of reports 5–8. Thus, each student receives individual feedback on three reports (the first report, and two randomly selected reports), but has access to feedback on two further pieces of work by every student on the module. Over the course of the module, students have access to both high and low quality examples of work. It is a condition of passing the module that seven out of eight submissions are of passing grade, and so all submissions are assessed on a pass/fail basis and students are notified if they have submitted a piece of work that would be below passing grade (this has not yet happened, as the first session provides extensive detail on producing the coursework). Thus, all reports are read, but the student only receives detailed feedback on three of their own pieces of work. The total time spent marking associated with this component of the module is reduced by approximately two-thirds (1500 words per student rather than 4000), as only one-third of the reports are provided with feedback each week, and the assessment of the remainder on a pass/fail basis can be achieved very rapidly.

At the end of the module, students self-assess all their submissions, and select the two that they think are of the highest quality for summative assessment (worth 60% of the module mark), using the assessment criteria and the feedback on their own and other marked examples to guide them. The chosen reports may or may not include work that has previously been marked, but reports cannot be amended from the previous submission as the student is expected to use and apply the feedback as feed-forward and improve their work in this way. All summative submissions (selected reports) are marked without reference to any previous assessments, and a subset is check-marked in accordance with University guidelines. To accompany the two chosen news and views articles, the students are asked to write a 300-word reflection that outlines their reason for selection of these articles (10% of the module mark). This should focus on the features of the article that make it of high quality, demonstrating how the student has applied the feedback to their work, but may also touch on other areas (such as the student’s understanding of the subject area).

The remaining 25% of the module marks come from a defended poster, presented in pairs. The poster is based on a published paper from the recent literature (from the year in which the module is taking place, and the previous year). Students are asked to produce a scientific poster, as might be presented by the authors of the paper at a conference, and are directed towards online resources on poster design, and the numerous scientific posters around the department. Posters are presented in a poster session at the end of the module, and are assessed by 2–3 independent markers, using standardised marking criteria.

Analysis of ‘news and views’ marks

Is there evidence that students are using feedback to improve the quality of their ‘news and views’ submissions and select their best quality work?

I wished to determine whether the ‘news and views’ component of the assessment strategy was an effective way of allowing students to produce and select their best work. With this aim in mind, student marks were collated from the first three years of the module (2010/2011, 2011/2012 and 2012/2013 academic years). Table 2 outlines the number of students enrolled on the module, and the number of reports submitted in each category.

Table 2 Numbers of students enrolled on the module and first, randomly selected and chosen reports assessed, for the three cohorts.

Year	2010/2011	2011/2012	2012/2013
Enrolled	23	33	31^2
First	23	33	29
Randomly selected	37^1	66	60
Chosen	46	66	60

¹The strategy for selection of reports during the module differed slightly in 2010/2011 and was refined following student feedback.

²In 2012/2013 one student failed to submit the first report, and one student’s marks have been excluded from analysis as the student did not follow the procedure for selecting reports. The reflection submitted by the student stated two reports had been chosen at random for summative assessment, rather than through self-assessment.

If students were able to use the feedback on their own and other students’ work effectively, then I would expect:

1. An increase in marks awarded between the first and randomly selected reports as student work improves in quality.

2. For the marks awarded to the chosen reports to be higher than both the randomly selected and first reports.

All analyses were carried out in R 2.13.0 (R Development Core Team 2011). To assess whether there were differences between the marks awarded to students’ first, randomly selected and chosen reports, I used linear mixed effects models to investigate the effects of report type (first, random or chosen), cohort, and their interaction, on the awarded mark. Percentage marks were divided by 100 to give a value between zero and one, and arcsin-square root transformed to meet the assumption of normality of model residuals. Student identity (anonymous) was added as a random factor to account for the repeated measures nature of the data. The non-significant interaction (report type * cohort; F_4,328 = 1.161, p = 0.328) was removed following Crawley (2007), and only the model with main effects is presented here. Subsequently, each pair-wise comparison between report categories was made using paired t-tests.

Is there evidence that students are self-assessing unmarked work to improve on module marks?

Next, I was interested in whether there was any evidence that students were self-assessing unmarked work and using it to improve on their overall attainment in the module. To investigate this, I looked firstly at the number of previously marked reports selected by the student for final submission, and found that while 16 students selected from among their three previously marked reports (2 marked reports), 49 chose one report that had not previously been marked (1 marked report), and 21 students chose to submit two reports for which they had not previously received a mark (0 marked reports). If students are able to independently select their best work, then I would predict that those students choosing 1 or 0 marked reports should show greater improvement in marks between their first and randomly selected reports, and their final chosen reports.

To assess whether the number of previously marked reports chosen by a student for their final submission had an effect on their overall attainment, I first calculated the average mark for the two chosen reports, and the average mark for the first and randomly selected reports. I then calculated the difference between these values. I then assessed whether this mark difference differed between students choosing 0, 1 and 2 previously marked reports using analysis of variance (ANOVA) on log(mark difference + 10), followed by Tukey HSD post hoc tests. A constant was added to the mark difference value to ensure all numbers were positive (one student chose two previously marked reports that did not include their best mark), and data were log-transformed to meet the assumptions of normality.

Results

Is there evidence that students are using feedback to improve the quality of their submissions and select their best quality work?

There was a significant effect of report type on the mark awarded (F_1,332 = 52.192, p < 0.001), but no effect of cohort (F_1,83 = 0.350, p = 0.706). Consequently, data from the three cohorts were pooled, and post hoc tests carried out on this pooled data. There were significant differences in the marks awarded between first and randomly selected reports (F_1,161 = 7.974, p = 0.005), between first and chosen reports (F_1,170 = 87.995, p < 0.001) and between randomly selected and chosen reports (F_1,248 = 59.336, p < 0.001). Students were awarded significantly higher marks for their chosen reports than for their first or randomly selected submissions, and marks for randomly selected submissions were significantly higher than for the first reports (Figure 1).

Figure 1 Mean (± 1 standard error) mark (%) awarded to students first, randomly selected and chosen reports. Stars and dashed lines indicate where significant differences lie.

Is there evidence that students are self-assessing their work to improve on module marks?

There was a significant effect of number of reports chosen that had previously been marked on the calculated difference in marks (F_2,83 = 5.516, p = 0.005). Students selecting 1 or 0 previously marked reports had a significantly higher difference in marks than students selecting 2 previously marked reports (Tukey HSD, adjusted p-values, 1 marked report: p = 0.004, 0 marked reports: p = 0.024). Students selecting 0 marked reports did not have a significantly greater mark difference than students selecting 1 marked report (p = 0.985; Figure 2).

Figure 2 Mean (± 1 standard error) difference in the average marks awarded for the two chosen reports and the average marks awarded for the first and two randomly selected reports, for students selecting 0, 1 and 2 reports that had previously been given a mark. Stars and dashed lines indicate where significant differences lie.

What are the student impressions of the module?

Feedback on the module (collected via standard departmental end-of-module evaluation forms) was generally positive. The structure and questions asked on these forms changed between cohorts, but some general themes remained. Of the students completing the questionnaire, 11/14 (2009/2010) and 16/20 (2010/2011) responded to the statement ‘the feedback on course work I received was useful’ with a positive answer. In 2012/2013 the module evaluation form changed and this question was removed. Those that did not find the feedback useful expressed (in the free-text section of the forms) a lack of confidence in their ability to select their own reports for summative assessment, and a perception that the unmarked work was a ‘waste of time’.

Of those that did find the feedback and assessment strategy useful, comments included:

I enjoyed this module, especially the general feedback and feedback on others work. It enabled me to refine my own work well. It has also given me skills which are transferable to other modules. (comment from 2011/2012)

Reading other people’s feedback has been helpful. (comment from 2010/2011)

Students also enjoyed the research-led aspect of the module, and the variety of topics covered. For example:

The module was highly informative and I liked how the coursework material was based on different research directions as I think it has allowed me to explore areas of biology I normally wouldn’t have read. (comment from 2011/2012)

In 2012/2013, the students were specifically asked which aspects of the module they found most enjoyable. Here, a number of students included comments on the research-led angle:

The module was highly informative and I liked how the coursework material was based on different research directions as I think it has allowed me to explore areas of biology.

Having a different lecturer each week talking about their most recent research was very interesting and kept each week fascinating.

… diversity of presentations covering different areas of genetics and ecology.

Students in 2012/2013 also provided positive feedback on the assessment strategy, with some individuals commenting on the benefits of small weekly assignments, being able to select assignments, and the amount of feedback. One student commented:

This has been the most valuable module I have taken in my degree. It has improved the way I read papers by changing the way I find the most important information in articles. I have therefore been able to apply this to other modules. The amount of feedback has been excellent and the class participation has improved the way I listen in lectures.

When asked how the module could be improved, this cohort (2012/2013) continued to express some discomfort in choosing their own assignments, and requested more guidance on how to do so, and more ‘general’ feedback.

Discussion

There was a significant increase in students’ marks between the first and the randomly selected reports, suggesting that the students’ ability to write the reports in line with the assessment criteria increased over the duration of the course. The final chosen reports gained significantly higher marks than both the first and randomly selected reports, suggesting that the students were able to use the feedback provided to select better work for summative assessment than that selected at random. These results suggest that the feedback supported their learning (Hattie et al. 1996, Black & Wiliam 1998) and that they actively used it to prepare subsequent reports (Orsmond et al. 2005, Glover & Brown 2006, Nicol 2010, Scott et al. 2011). A focus on multiple low-stakes assessment tasks can enhance student motivation to learn, compared to high-stakes tasks (Nicol & Macfarlane-Dick 2004), which may have been an influence here.

Students that chose no previously unmarked reports (i.e. selected two out of the three reports that had been marked during the course) showed a small positive mark difference compared to the average mark for all three, suggesting that they were able to select the best two of three, although it is highly likely that this decision was made on the basis of the indicative marks awarded only. Students selecting one or more reports that had previously not been marked, however, showed a significantly greater mark difference, suggesting that (1) some students chose to use the information they had learned during the course from the feedback and apply it to their own work (Chanock 2000, Nicol 2010), and (2) that those students were able to effectively use this information and select work that resulted in an increase in their final mark for the course, suggesting that the feedback also facilitated the development of self-assessment skills (Nicol & Macfarlane-Dick 2004).

Feedback comments are usually directed towards the student who has produced the assignment, but viewing comments that are directed at other students’ work can provide a wealth of information to all students (Nicol 2010). Comments on other students’ work are often shared through a collated list, which may derive from previous years’ assignments (Nicol & Macfarlane-Dick 2004). In the module described here, students are able to build a database of not only feedback comments, but also study the specific aspects of the reports on which they are based. Feedback is also provided regularly and is detailed (Gibbs & Simpson 2004), and can be applied directly to subsequent assignments within the module, closing the ‘feedback loop’ (Sadler 1989, Boud 2000). Together, these assessment and feedback strategies allow students to perform well on the module.

Comments on the end-of-module evaluations reflected some negative impressions of the assessment strategy, and thus there are improvements that can be made. In the first year, I did not do enough to manage student expectations (which are linked to student satisfaction; Appleton-Knapp & Krentler 2006), and so subsequently I have spent increased amounts of time explaining the assessment strategy and its benefits to the students. For students that are used to summative work only, as is the norm for the majority of the modules that they take, receiving only formative feedback on their work is unexpected and unusual, and many students do not fully appreciate the benefits of being able to ‘practice’ assignments in a formal way before the summative submission. The sector-wide decline in formative assessment (Atkins 1995, Higher Education Academy 2012) suggests that this could be a common problem.

Many students expressed a lack of confidence in their ability to choose their own reports, although the data suggest that the majority of students are able to do this. I suggest that incorporating peer-review of reports (Orsmond 2011) prior to the final submission may give students more confidence in selecting their own reports, as well as relying on the guidance of peers. Evidence suggests that peer-review is an effective means of assessment, and that (given clear marking criteria), peer marking closely reflects that of academic staff (Stefani 1994, Orsmond et al. 1997, Falchikov & Goldfinch 2000). Furthermore, combining self-, peer- and co-assessment can encourage students to become more responsible and reflective learners (Dochy et al. 1999), suggesting benefits beyond report selection by using this approach.

For the more general structure of the module, a concern may be that staff are unwilling to discuss their own research with students (Brew 2006), yet I did not find it a problem to recruit the required number of volunteers, and informal discussions with staff suggested that they enjoyed teaching on the module, particularly the question-and-answer session at the end of each seminar. The fact that staff were able to use already-prepared conference or seminar material, and that there was a low marking load (staff were asked only to contribute to the poster marking) may have contributed to this willingness.

Although I attribute the increase in student marks to the module feedback and assessment strategy, it is possible that other aspects of their experience are the underlying cause of the improvement. However, there is no other module that is taken by all students enrolled on Topics in Biodiversity and Evolution, and the students themselves have linked their improvements in other modules to this module, rather than vice versa. The small weekly assessments are unusual within the degree programmes, as other modules, including those taken simultaneously, generally have final summative coursework and/or examinations, rather than feedback provided within the module.

The assessment strategy I describe for this module provides a mechanism for generating rapid feedback that can be effectively used by students as feed-forward to improve their learning and attainment in the module. The module structure, and particularly the feedback strategy could be applied across subject areas, and is not restricted to research-led teaching in ecology and evolutionary biology. However, the feedback strategy is novel to the students, and runs counter to the experiences of assessment and feedback they have previously received during their degree programme. The feedback from students who did not value the format suggests that the expectations of students need to be carefully managed (Appleton-Knapp & Krentler 2006) and the value of the approach within the context it is used clearly explained.

Acknowledgements

The author thanks the students and staff who participated in the module over the past three years, and to two anonymous referees whose comments helped improve the manuscript.