How does online peer feedback improve argumentative essay writing and learning?

ABSTRACT

This study investigates the effects of unscripted, scripted and guided online peer feedback on students’ argumentative essay writing, argumentative feedback quality and domain-specific knowledge acquisition in the field of educational sciences. Participants were 52 students who were randomly divided over 26 dyads and randomly assigned to three conditions (unscripted, scripted and guided peer feedback conditions). An online peer feedback platform, named EduTech, was designed and these instructional supports were embedded within this platform. Students were asked to write an argumentative essay (individually), to engage in argumentative peer feedback with their learning partner (collaboratively), and finally to revise their essays based on feedback received (individually). The findings indicate that students in the scripted online peer feedback condition outperform the other two conditions in terms of argumentative essay writing, argumentative feedback quality and their domain-specific knowledge acquisition. Implications and suggestions for future research are discussed.

KEYWORDS:

• Online peer feedback
• argumentative writing
• argumentative essay
• collaborative eLearning environments

Introduction

Writing essay is a common activity for higher education students (Mei, 2006). For example, students who follow courses on environmental education, sociology, human nutrition and health, biotechnology and molecular life sciences, or educational technology are frequently asked to write essays on ‘global warming’, 'feminism’, ‘vegetarian vs. meat eaters’, ‘the use of mobile devices in classrooms’ and ‘Genetically Modified Organisms (GMOs)’. Despite popularity of essay writing tasks for students in higher education (see Noroozi, Biemans, & Mulder, 2016; Noroozi, Kirschner, Biemans, & Mulder, 2018), teachers often are not happy with regard to the quality of such essays and they often blame students for the lack of solid argumentation and sound reasoning in their argumentative essays (Kellogg & Whiteford, 2009). From teachers’ point of view, a clear and high-quality argumentative essay should encompass a clear claim supported by argumentation, reasoning and evidence, followed by acknowledgement of counter-arguments against the original claim, and integrations of the arguments and counter-arguments which could lead to a final conclusion on the issue (see Noroozi et al., 2016). This implies that writing such essays require solid argumentation strategies (Mei, 2006; Wingate, 2012).

Students often fail to follow these argumentation strategies in their written argumentative essays. There could be several reasons for this important issue: First, due to lack of knowledge, some students do not know the characteristics of a good argumentative essay (Bacha, 2010). Second, even if they know about these characteristics, they may struggle to put them into practice when they write such argumentative essays (Noroozi, Teasley, Biemans, Weinberger, & Mulder, 2013; Valero Haro, Noroozi, Biemans & Mulder, 2019a). Third, since features, terminologies and epistemologies of argumentative essays may vary from discipline to discipline (Andrews, 1995; Wingate, 2012), students often struggle in writing such essays for different argumentation tasks in different courses (see Noroozi et al., 2018; Wingate, 2012). All these challenges imply that students need additional support for writing high-quality argumentative essays.

Online peer feedback is one of the promising educational strategies to improve student’s argumentative essay writing and learning. For example, researchers have shown that peer feedback can improve students’ writing quality (Huisman, Saab, van Driel, & van den Broek, 2018; Noroozi & Hatami, 2019; Noroozi et al., 2016; Valero Haro, Noroozi, Biemans & Mulder, 2019b), students’ feedback quality (Gielen & De Wever, 2015; Noroozi et al., 2016), domain-specific knowledge gain (Valero Haro et al., 2019b) and students’ attitudes towards the subject at hand (Noroozi & Mulder, 2017). Although these studies have confirmed the effectiveness of online peer feedback, there are some main criticisms on the way in which peer feedback is implemented. For example, there are concerns about the quality of peer feedback because of students’ limited knowledge, experience and language ability (Noroozi et al., 2018; Saito & Fujita, 2004). Also, there are emotional and psychological issues with giving and receiving critical feedback (Andriessen, 2006). These challenges suggest that implication of peer feedback in classrooms needs to be supported to fully guarantee its effectiveness (Noroozi et al., 2016).

Many online learning environments have been designed and developed to support peer feedback processes. Such environments allow students to submit their essays, provide feedback to their peers’ works reciprocally and anonymously without restriction of time and space (Lin, Liu, & Yuan, 2001, 2018; Noroozi et al., 2016; Tsai, Lin, & Yuan, 2002; Yang, 2011). Also, implementing peer feedback in online environments increase timeliness of feedback for learners to reflect on one’s own and their peers’ work (Chen & Tsai, 2009; Corgan, Hammer, Margolies, & Crossley, 2004). The most important benefit of such online environments is that they allow for embedding various types of instructional supports (scripting and guidance) that can guide students provide their peers with more relevant and high-level quality feedback (Noroozi, Biemans, Busstra, Mulder, & Chizari, 2011).

Different instructional approaches have been proposed to support online peer feedback. These approaches include sentence starters, open text-boxes, assigning and rotating roles, peer interactions, input text fields and question prompts (Noroozi, Weinberger, Biemans, Mulder, & Chizari, 2012). All of these approaches fall under the name ‘scripting’. Although studies have shown the effectiveness of scripting approaches on improving students’ academic writing and learning, some scholars point out that this approach can be challenging. For example, overly detailed scripts or ‘over-scripting’ can impose cognitive load and impede learning (Dillenbourg, 2002). Furthermore, scripts may limit the freedom and creativity of students to think wider and various points of view of the issue at stake (see Noroozi et al., 2013). To cope with this challenge, scholars have proposed to use simple instructional strategies to reduce cognitive load such as textual guidance and instruction prior to peer feedback processes (see Noroozi et al., 2016). For example, Valero Haro et al. (2019b) showed that the effectiveness of such simple instruction and guidance prior to peer feedback can improve students’ argumentative essay writing quality. So far, the effectiveness of such online unscripted, scripted and guided peer feedback has not been compared to each other in relation to various aspects of learning processes and outcomes. Furthermore, it is not clear how students respond to such online peer feedback settings when it comes to writing argumentative essays. This study compares the impacts of online unscripted, scripted and guided peer feedback on students’ learning outcomes such as students’ argumentative essay writing, argumentative feedback quality and domain-specific knowledge acquisition. Furthermore, this study reports students opinions on the use of online peer feedback system. We thus answer following research questions:

1. What are the effects of online unscripted, scripted and guided peer feedback on students’ argumentative essays quality?

2. What are the effects of online unscripted, scripted and guided peer feedback on students’ argumentative feedback quality?

3. What are the effects of online unscripted, scripted and guided peer feedback on students’ domain-specific knowledge acquisition?

4. What are the opinions of students on the use of online peer feedback system?

Methods

The study took place at Kharazmi University, Tehran, Iran. The participants were 52 BSc students who enrolled for the course ‘Applying Computer in Educational Sciences’, and were randomly divided over 26 dyads and were assigned to unscripted (9 dyads), scripted (8 dyads) and guided (9 dyads) conditions. Students in the unscripted condition were regarded as a control group without any support during the online peer feedback. Students in the scripted condition were supported by argumentative scripts in the form of a question prompts during their online peer feedback. Students in the guided condition received online instruction on ‘how to provide an argumentative peer feedback’ prior to the feedback phase.

The mean age of the students were 20.21 (SD = 1.51) years. All participants were female. The topic for discussion was ‘Mobile Learning’. Students were asked to write an argumentative essay on the following statement: ‘The use of mobile phones and tablets in the classroom should be banned’. They were provided with some theoretical text as background information. Then, they engaged in argumentative peer feedback, and finally revised their essays based on feedback received. A self-made online environment (EduTech) was designed and used in this study. This online learning environment had a series of steps (see Figure 1).

Figure 1. The procedure of the study embedded in EduTech platform

All these steps had to be completed individually except for the peer feedback step. The feedback features in EduTech for scripted condition was designed on the basis of argumentation literature (Andrews, 2010; Noroozi et al., 2016; Schneer, 2014; Toulmin, 1958; Wingate, 2012) and a high-quality argumentative essay in the field of Educational Sciences. Therefore, a series of meetings were held with the experts to define the elements of a high-quality argumentative essay for students in the field of Educational Sciences. The panel of experts concluded that a high-quality argumentative essay in the field of Educational Sciences should include: the expression of a clear position on the topic, expressing the context of topic (introduction), the arguments and evidences (examples, facts, expert opinion, etc.) for and against the topic, integration of various pros and cons, and the final conclusion on the initial position. We then designed our scripted and guided peer feedback on the basis of these items and embedded them in EduTech. This design resulted in a new form of feedback called ‘argumentative peer feedback’. Argumentative peer feedback in this study refers to the feedback on the topic that is backed up and justified as opposed to superficial feedback such as feedback on personal qualities and/or outcome-based feedback that lack justifications and sufficient instructional value to improve writing and learning (Hyland, 2000; Muncie, 2000).

The study lasted about 5:30 h and consisted of five main phases (each phase in one session in five consecutive weeks). 1) During the introduction and pre-test phase, students received introductory textual and verbal explanations on the EduTech and completed a survey on demographic variables, and their domain-specific knowledge (about 30 min). 2) In the study and draft phase, students were asked to read theoretical text and articles on the topic and to search the Internet based on keywords bolded in the text (40 min), and to write an argumentative essay (80 min). 3) In the peer feedback phase, each student was asked to read the draft argumentative essay of the learning partner and to provide feedback on them (50 min). 4) During the revision phase, students were asked to read the comments of their learning partner and then to revise their draft argumentative essay (60 min). 5) Finally, in the post-test phase, students were asked to fill out several questionnaires to assess their domain-specific knowledge on the topic and also their opinions regarding the process (15 min).

Measurements

The quality of student’s written argumentative essays (in the draft and the revision phases) was measured using the coding scheme developed by the authors. The scheme considers the features of a high-quality argumentative essay in the field of Educational Sciences and was developed in conformity with the literature (Andrews, 2010; Noroozi et al., 2016; Schneer, 2014; Toulmin, 1958; Wingate, 2012). The scheme included eight components as following: 1) Position on the topic, 2) Introduction (attention grabber) on the topic, 3) Claims in favour of the topic, 4) Evidence to support claim(s) in favour of the topic, 5) Claims against the topic, 6) Evidence to support claim(s) against the topic, 5) The weighing benefits and drawbacks, and 8) Conclusion. A single score was assigned for each of these components. Each student was given no point for each level 1 assessment (e.g. not mentioned), one point for each level 2 assessment (e.g. non-elaborated), and two points for each level 3 assessment (e.g. elaborated). Thus, for each component, students’ scores varied between zero and two. Subsequently, all points assigned to each student were added together and served as the final score for students’ written argumentative essay quality. The test of reliability index of inter-rater agreement for 10% of the data for two coders resulted in identical scores in 80% of draft and 82% of the revision phases. The same scheme was adjusted to assess students’ feedback quality with two coders resulted in identical scores in 83% of the data.

The pre- and post-test questionnaire, which was completed by students before draft phase and after revision phase consisted out of 10 multiple-choice questions to measure students’ domain-specific knowledge. For these questionnaire, students needed to choose one answer out of four options. Each correct answer was given one point and as a result each student could receive 10 points at maximum for both pre- and post-test.

Students’ opinions regarding the EduTech platform were measured using a questionnaire designed by Noroozi and Mulder (2017). This questionnaire included four main sections, 36 items in total. All responses were rated on a 5-point Likert-type scale ranging from ‘almost never true’, to ‘almost always true’. The first section measured students’ perceived effects of the domain‐specific learning outcomes (6 items). The second section measured students’ perceived effects of the domain‐general learning outcomes (10 items). The third section measured students’ opinions on the ease of use of the module (7 items). The last section measured students’ appreciation of the module (13 items). The reliability was high for all four scales of this instrument (Cronbach α = 0.84, 0.88, 0.87 and 0.89, respectively).

Findings

Results for research question 1

Repeated measures ANOVA test showed that the written argumentative essays’ quality of students in all conditions improved significantly from the draft phase to the revision phase, F (1, 49) = 114.18, p < 0.001, η2 = 0.70. Also, there was a significant difference among the conditions in terms of argumentative essay quality, F (1, 49) = 18.80, p < 0.001, η2 = 0.43. The post hoc Tukey HSD test showed that the mean score for the scripted condition was significantly higher than unscripted condition, p < 0.001. In addition, this test showed that the mean score for the guided condition was significantly higher than unscripted condition, p < 0.001. Also, students in the scripted condition scored higher than students in the guided peer feedback condition in terms of argumentative essays quality, p < 0.001 (see Table 1).

Table 1. Differences among various conditions in terms of dependent variables

Variables		Conditions		M (SD)	F	Sig	Tukey HSD
Quality of students argumentative essays		Unscripted	Draft	3.50 (1.15)	18.80	0.001	Scripted and Guided > Unscripted Scripted > Guided
			Revision	4.55 (0.70)
		Scripted	Draft	4.75 (1.43)
			Revision	10.43 (2.47)
		Guided	Draft	4.22 (1.43)
			Revision	7.27 (2.10)
Quality of students argumentative feedback		Unscripted		1.66 (2.02)	31.77	0.001	Scripted and Guided > Unscripted Scripted > Guided
		Scripted		12.06 (3.85)
		Guided		7.11 (4.94)
Domain-specific knowledge		Unscripted	Pre-Test	2.83 (0.98)	4.43	0.01	Scripted and Guided > Unscripted Scripted = Guided
			Post-Test	4.33 (1.90)
		Scripted	Pre-Test	3.43 (1.09)
			Post-Test	6.93 (1.76)
		Guided	Pre-Test	3.38 (1.46)
			Post-Test	6.27 (1.56)
Motivation and Satisfaction with the Learning Experiences	Perceived effects of the domain‐specific learning outcomes	Unscripted		3.69 (0.42)	8.18	0.01	Scripted and Guided > Unscripted Scripted = Guided
		Scripted		4.12 (0.46)
		Guided		4.24 (.38)
	Perceived effects of the domain‐general learning outcomes	Unscripted		3.99 (0.27)	5.83	0.005	Scripted and Guided > Unscripted Scripted = Guided
		Scripted		4.35 (0.45)
		Guided		4.40 (0.42)
	The ease of use of the module	Unscripted		4.23 (0.62)	1.26	0.29	Scripted = Guided = Unscripted
		Scripted		4.29 (0.61)
		Guided		4.52 (0.45)
	Appreciation of the module	Unscripted		4.11 (0.47)	1.54	0.22	Scripted = Guided = Unscripted
		Scripted		3.98 (0.51)
		Guided		4.26 (0.42)
	Total	Unscripted		4.01 (0.32)	4.92	0.01	Guided > Unscripted Guided = Unscripted Scripted = Guided
		Scripted		4.19 (0.33)
		Guided		4.35 (0.32)

Results for research question 2

One-way ANOVA showed a significant difference among various conditions in terms of argumentative feedback quality, F (1, 49) = 31.77, p < 0.001. The post hoc Tukey HSD test revealed that the mean score for the scripted condition was significantly higher than unscripted condition, p < 0.001. In addition, the mean score for the guided condition was significantly higher than unscripted condition, p < 0.001. Also, students in the scripted condition scored higher than students in the guided condition in terms of argumentative feedback quality, p < 0.001 (see Table 1).

Results for research question 3

Repeated measures ANOVA test showed that the domain-specific knowledge of students in all conditions improved significantly from the pre-test to the post-test, F (1, 49) = 87.70, p < 0.001, η2 = 0.64. Also, there was a significant difference among conditions in terms of the domain-specific knowledge, F (1, 49) = 4.43, p < 0.001, η2 = 0.15. The post hoc Tukey HSD test revealed that the mean score for the scripted condition was significantly higher than unscripted condition, p < 0.001. In addition, the mean score for the guided condition was significantly higher than unscripted condition, p < 0.004. However, there was no significant difference between the scripted and guided conditions, p < 0.62 (see Table 1).

Results for research question 4

As shown in Table 1, students’ opinions on the use of online peer feedback platform appeared to be sufficiently high for all students. This indicates that, on average, students scored 4 out of 5 with regard to their opinions on the use of EduTech platform for online peer feedback when writing argumentative essays. One-way ANOVA showed a significant difference among the three conditions in terms of students’ opinions on the use of EduTech, F = 4.92, p < 0.01. The post hoc Tukey HSD test showed that these significant differences were only related to perceived effects of the domain-general and domain‐specific learning outcomes in favour of scripted and guided conditions as opposed to unscripted condition. However, there was no difference among conditions in term of the ease of use of and appreciation of the EduTech.

Discussions, conclusions and implications

This study investigated the effects of unscripted, scripted and guided online peer feedback on students’ argumentative essay writing, argumentative feedback quality and domain-specific knowledge acquisition in the field of educational sciences. Overall, regardless of conditions, the online learning environment (EduTech platform) led students engage in detailed argumentative reception and provision, write higher quality argumentative essay, and improve their domain-specific knowledge about the subject at hand. This is in line with previous findings that emphasise positive effects of peer feedback on various aspects of learning processes and outcomes (see Jurkowski, 2018; Noroozi, 2018; Noroozi et al., 2016; Topping, 2009). Engaging in argumentative peer feedback processes allowed students to elaborate and reflect on learning materials included in EduTech.

In this study, for most dependent variables, students in the scripted condition outperformed students in the guided and unscripted conditions. This is in line with previous studies that emphasise the positive effects of scripts on the quality of students’ written argumentative essays (Huisman et al., 2018; Noroozi et al., 2016). Giving and receiving a high-quality feedback provided students with criteria that help them assess partners’ essays clearly with justified arguments and reasoning. Therefore, using argumentative peer feedback, students were able to assess their peers’ essays based on predesigned argumentation criteria, not on their personal perspective.

Students in the guided condition outperformed students in the unscripted condition for most dependent variables. The detailed instruction on how to provide high-quality argumentative peer feedback and also how to write argumentative essays allowed students in the guided condition to engage in higher cognitive processing such as argumentation, justification and clarification resulting in higher quality argumentative feedback and essay (Noroozi & Mulder, 2017; Valero Haro et al., 2019b). However, students in the guided condition were not as successful as students in the scripted condition. The reason may be that, although students in the guided condition learned how to provide argumentative feedback and how to write argumentative essay, they were not prompted in the feedback phase to provide a high-quality feedback equal to scripted condition. Scientific literature suggest that although students may possess knowledge on argumentation, they are not always able to put their knowledge in practice during the learning processes (see Noroozi et al., 2018).

Furthermore, all students seemed to be happy with regard to the use of online peer feedback environment in this study. This is in line with previous studies that emphasise the positive effects of online peer feedback environments on students’ motivation and satisfaction with the learning experiences (Noroozi & Mulder, 2017). Overall, students perceived that working on various parts of the module was easy, clear and understandable. Also, they reported that navigating through the EduTech was easy and had no technical problem. Students appreciated the instructional supports embedded in the EduTech that helped them write a well-structured and sound argumentative essay and gain domain‐specific knowledge. Overall, user-friendliness design supported with clear guidance led to a high mean score of students’ appreciation of the EduTech. The opinions of students in the scripted and guided conditions were more positive than students in the unscripted condition regarding domain‐general and domain-specific learning outcomes. The plausible explanation is that students in the scripted and guided conditions felt that with such support they are in the right direction towards engaging in high-quality peer feedback processes compared to students in the unscripted condition who were left on their own without additional support.

We used an existing course in a real educational setting with a high level of ecological validity and its own dynamics. Such authentic setting also put some constraints on the possibilities to experiment such as sample size and the gender. In this study, only 52 students enrolled for the course and all of them were female. Scientific research shows that males might perform differently than females in terms of their argumentative learning processes and outcomes (see Noroozi, Hatami, Biemans, van Ginkel, & Bayat, 2019; Noroozi et al., 2012). Further research under more stringent conditions regarding a larger sample size with mix groups of students in terms of gender is needed to test the extent to which the results of this study can be generalised.

We only used quantitative data analysis and measurements to investigate the effects of online peer feedback on the quality of argumentative peer feedback, essay writing and learning. However, we acknowledge that qualitative analysis would also add values to the findings of this study. Therefore, we suggest that future research employ qualitative data analysis to avoid reductionist approach and to truly explore how students’ feedback quality, essay writing and learning can be improved using different online peer feedback designs.

Disclosure statement

No potential conflict of interest was reported by the authors.

Additional information

Notes on contributors

Saeed Latifi

Saeed Latifi is a PhD graduate of Educational Technology at Tarbiat Modares University, Iran. His research interests include E-Learning and Distance Education, Computer-Supported Collaborative Learning (CSCL), Argumentative Knowledge Construction in CSCL, Argumentation-Based CSCL, CSCL Scripts and Transactivity.

Omid Noroozi

Omid Noroozi is a faculty member of Educational Technology at the Education and Learning Sciences Chair Group, Wageningen University and Research, The Netherlands. His research interests include Collaborative Learning, E-Learning and Distance Education, Computer-Supported Collaborative Learning (CSCL), Argumentative Knowledge Construction in CSCL, Argumentation-Based CSCL, CSCL Scripts and Transactivity.

Javad Hatami

Javad Hatami is a faculty member of Educational Technology at Tarbiat Modares University, Iran. His research interests include Educational Development and Evaluation, E-Learning, MOOC, Concept Map, Curriculum in Education, Teaching Methods and Techniques, Measurement and Assessment.

Harm J.A. Biemans

Harm Biemans is a faculty member of Learning Sciences at the Education and Learning Sciences Chair Group, Wageningen University and Research, The Netherlands. His research interests include Collaborative Learning, E-Learning and Distance Education, Computer-Supported Collaborative Learning (CSCL), Learning Environments, Learning Sciences.