How does the type of online peer feedback influence feedback quality, argumentative essay writing quality, and domain-specific learning?

ABSTRACT

This study explored the effects of directed and undirected online peer feedback types on students’ peer feedback performance, argumentative essay writing skills, and acquisition of domain-specific knowledge. The study used a pre-test and post-test design with four conditions (feedback, feedforward, a combination of feedback and feedforward, and undirected feedback). In this exploratory study, 221 undergraduate students, who were randomly assigned to dyads, engaged in discussions about the pros and cons of “Genetically Modified Organisms (GMOs)”, provided feedback to peers, and wrote an argumentative essay regarding the topic. Results indicated significant differences among the conditions in terms of the quality of provided feedback. This implies that the peer feedback quality can be enhanced or diminished depending on its type. Results also revealed a significant improvement in students’ argumentative essay performance and domain-specific knowledge acquisition without significant differences among conditions. We discuss how the such increase in the quality of essays and learning outcomes might be related to the power of peer feedback regardless of the feedback type. We also discuss why using multiple instructional scaffolds may result in over-scripting that may diminish the power of peer feedback and the effects of the scaffolds themselves in online learning environments.

KEYWORDS:

• Argumentative essay writing
• directed peer feedback
• feedforward
• online learning
• peer learning
• scripting
• undirected peer feedback

1. Introduction

Argumentation is an essential competence across domains and aspects of daily life. Argumentation competence comprehends the capacity to argue, think critically, and reason logically for justifying and contrasting our positions and opinions against the positions and opinions of others. Argumentation in academic settings encompasses the capacity to carry out comparable tasks and continue learning in the future (Noroozi et al., 2018). Acquiring argumentation competence is crucial for students in higher education as it is a representation of students’ high-level cognitive process thinking skills to critically argue complex and controversial issues in a certain scientific field of study (Fan & Chen, 2021; Mokhtar et al., 2020). One of the most common ways to practice argumentation skills in higher education is to write an argumentative essay on controversial scientific topics (Liunokas, 2020; Wingate, 2012). According to the literature, higher education students’ argumentative essay writing skills are typically below the proficiency level required to deliver writing tasks (Cooper et al., 1984; Kellogg & Whiteford, 2009).

A good argumentative essay writing should include intuitive opinions and feelings on the topic. The reasoning behind this is, that students, and most people, possess gut feelings and intuitive opinions on the various controversial scientific issues, although they are not familiar with the topic (Noroozi et al., 2016). Students’ opinion is followed by arguments and data supporting them. Moreover, essays should incorporate arguments opposing or weakening the opinion (counter-arguments), and consider and refute the point of view of opponents. Next, the arguments in favor and against the topic should be integrated considering the opinions of the advocates and the opponents of the topic in question (Andrews, 1995; Noroozi et al., 2016; Toulmin, 1958; Wood, 2001). After the integration, students should provide their conclusions since it is common that students’ final opinion on the topic remains unclear after arguing in favor and against the topic (Noroozi et al., 2016). Finally, the presentation and specifics of the aforementioned elements should be tailored to the domain in question as variations exist from domain to domain (Wingate, 2012).

Unfortunately, sound argumentation and depth of elaboration are rather infrequent in students’ essays (Cooper et al., 1984; Kellogg & Whiteford, 2009). Such poor performance in argumentative essay writing could be due to different reasons. In some cases, students are not aware of the characteristics of a good argumentative essay (Noroozi et al., 2016). In some other cases, while students possess argumentation knowledge, they have difficulties putting their argumentation knowledge into practice such as writing an argumentative essay (Noroozi et al., 2013). Other scholars such as Driver et al. (2000) and Osborne (2010) argue that the problem cause is that argumentation is frequently developed indirectly and informally in the classroom. Cooper et al. (1984) argue that the root of the problem is insufficient task practice in the curriculum, along with the considerable time and effort required in grading essays to provide feedback to students. Even when argumentation is considered in the classroom, a teacher can only supervise and support one or a small group of students (Bloom, 1984). In large-size classes which is common in online learning settings, it requires an extreme and almost unaffordable workload for teachers to provide personalized effective feedback on essay writing (Banihashem et al., 2021, 2022; Er et al., 2021). To address the issue of insufficient argumentative essay writing skills, researchers, teachers, and practitioners have looked for instructional practices to foster students’ writing motivation and types that improve the quality of essay writing and the acquisition of domain-specific knowledge through argumentation (Bruning & Horn, 2000; Noroozi et al., 2016).

In the literature, peer learning was found as one of the most relevant instructional practices to improve students’ essay-writing capacities (Baker, 2016; Boud et al., 1999; Topping, 2005). In peer learning “students learn with and from each other without the immediate intervention of a teacher” (Boud et al., 1999, p. 413). According to the literature, the provision of peer feedback in peer learning is a powerful instructional strategy to foster learning (Gabelica et al., 2012; Hattie & Gan, 2011; Hattie & Timperley, 2007; Noroozi et al., 2020; Shute, 2008). Peer feedback is defined as the action taken by a peer to provide information regarding some aspect(s) of one's task performance or understanding (Hattie & Timperley, 2007). In the context of argumentative essay writing, peer feedback is an effective instructional strategy, particularly in large-size online courses where a large cohort of students participate and teachers are left in a difficult position in providing one-by-one feedback on students’ essay performance due to extreme workload (Er et al., 2021; Noroozi et al., 2022).

Prior studies suggest that using peer feedback can improve students’ argumentative essay writing skills (Latifi et al., 2020; 2021a; Morawski & Budke, 2019; Noroozi et al., 2016; 2022a). For example, Morawski and Budke (2019) found that different types of peer feedback including online written and oral peer feedback can improve students’ argumentative essay writing performance in terms of the quality of complexity in arguments, quality of content-wise arguments, and quality of evidence provided in arguments. According to the literature, different peer feedback types can be used for enhancing students’ argumentation skills in essay writing (Latifi & Noroozi, 2021). In general, peer feedback types can be categorized into directed and undirected peer feedback (Gielen & De Wever, 2015; Latifi et al., 2021b). Directed peer feedback means that students receive support on how to provide feedback and this support can be in different forms such as training, scripts, worked examples, question prompts, sentence openers, templates, and checklists (e.g. Latifi et al., 2020). The main idea behind this type of feedback is that students do not have enough knowledge and skills in giving feedback themselves and they should be directed (Noroozi et al., 2016). On the other hand, undirected feedback means that students should not receive support when providing feedback to their learning partners and the logic behind this idea is that this support could limit students’ creativity in delivering effective feedback (Jermann & Dillenbourg, 2003; Tchounikine, 2008). Although both types have their pros and cons, the review of the literature does not provide sufficient evidence on which peer feedback type is more effective, especially in online learning environments. In online learning environments, students’ feedback performance can be recorded and tracked via online learning systems (Filius et al., 2019; Xie, 2013). This creates opportunities to track and monitor students’ performance and learning process during peer feedback activities. In online learning settings, students have flexibility in providing feedback anytime and anywhere (Du et al., 2019), and they can also have opportunities for anonymous feedback which can influence students’ honesty and level of criticism in delivering feedback (Aghaee & Hansson, 2013; Coté, 2014).

It is critical for teachers to know which peer feedback type is more effective in fostering students’ argumentation skills in essay writing given attention to the online context. For example, in online contexts, peer feedback can be delivered synchronously and asynchronously (Chen, 2016; Shang, 2017) in both directed and undirected formats (Latifi et al., 2021b; Van Popta et al., 2017). Some studies have shown that providing feedback on peers’ argumentative essays is a complex task and it requires high-level cognitive processing thinking skills (Foo, 2021; Van Popta et al., 2017). Therefore, directed and asynchronous peer feedback is a good option since students are guided and can take their time to review their peers’ essays (e.g. Chen, 2016; Latifi & Noroozi, 2021). While some other critics that directed peer feedback do not always result in successful performance (e.g. Dillenbourg, 2002; Dillenbourg & Tchounikine, 2007; Noroozi, 2013). For example, studies have shown that rigidly directed peer feedback can work as an inhibitor in the creation of natural interaction among peers (Dillenbourg & Tchounikine, 2007). Or, directed peer feedback can limit students’ freedom and autonomy in providing feedback which could negatively impact their learning processes and outcomes (Dillenbourg, 2002; Jermann & Dillenbourg, 2003; Noroozi, 2013; Tchounikine, 2008).

The review of the literature shows that only a few studies focused on comparing the effectiveness of different peer feedback types on students’ argumentative essay writing performance and domain-specific knowledge acquisition (e.g. Latifi et al., 2020; 2021b; Latifi & Noroozi, 2021). In a study, Latifi et al. (2020) compared the effects of peer feedback with worked examples and scripted peer feedback on feedback quality, essay quality, and acquisition of domain-specific knowledge. They found that scripted peer feedback entailed higher feedback quality with higher positive impacts on students’ argumentative essay writing quality compared to peer feedback with worked examples. However, there was no difference between the two types in terms of improvements in the quality of students’ domain-specific learning. In another study, (2021b) focused on comparing the effects of different peer feedback types on students’ learning processes and outcomes, and essay quality. They found a significant impact of different peer feedback types including peer feedback and peer feedforward on students’ learning processes and outcomes, and essay quality, however, the difference among different supported peer feedback types was not significant (Latifi et al., 2021b).

Furthermore, the literature suggests that, in practice, peer feedback has been traditionally seen as a strategy to provide remedial diagnostics on problems rooted in peers’ work and to clarify points for improvement, while recommendations and action plans on how to improve peers’ work from now on are somewhat overlooked (Latifi et al., 2021a; Taghizadeh Kerman et al., 2022; Wimshurst & Manning, 2013). In general terms, these constructive recommendations that can feed peers’ work to be improved are called peer feedforward (Wimshurst & Manning, 2013). Peer feedforward refers to the pieces of advice provided by a learner to a learning partner that can also include clear action plans on how to uptake the given advice (Latifi et al., 2021a). Providing peer feedforward is challenging for students and normally they do not provide constructive types that can guide peers in obtaining the desired goal (Noroozi & Hatami, 2019). The main reason for that is students lack sufficient knowledge and skills in advising on peers’ work and to understand what is next and how to get there (Kerman et al., 2022).

One may argue that feedforward is also a part of peer feedback and why these two should be seen separately. This argument is theoretically valid as we expect from high-quality peer feedback to not only give information on the current performance and related issues (peer feedback) but also include information on possible suggestions and action plans for further improvements of the work (peer feedforward) (Nelson & Schunn, 2009; Patchan et al., 2016; Wu & Schunn, 2021). This means that peer feedback regardless of its quality can take place in the form of feedback, feedforward, or a combination of both. However, in practice, this is not the case and students usually focus on the feedback part (what and how has been done) and struggle or ignore providing feedforward (where to go next and how to get there) (Latifi et al., 2021a). This issue raises a need to create and examine different peer feedback types that not only explicitly guide students on how to give feedback on the actual performance but also direct students on how to give constructive feedback that can lead peers to attain desired goals.

Although prior studies provide insights into how different peer feedback types influence students’ learning processes and outcomes, argumentative essay writing skills, they did not specifically provide information on how directed peer feedback types (feedback, feedforward, a combination of feedback and feedforward) compared to undirected peer feedback type can impact students’ performance in feedback, essay, and domain-specific knowledge acquisition. Previous studies mainly focused on comparing directed peer feedback types where students are guided and supported in different ways, while undirected peer feedback type where students can provide their feedback with freedom and in a natural way is not studied. There is a need to explore students’ peer feedback performance while they are supported and guided in providing feedback compared to when they provide feedback with freedom and autonomy. In addition, the literature review suggests that only a few studies have been conducted in this research area with a small number of students in one course which can make the results less reliable for other contexts. In addition, the results of the prior studies are mixed. While, in one study, a difference was found between supported peer feedback types (Latifi et al., 2020), the other study did not show such a difference (Latifi et al., 2021b). More studies are needed in this regard to provide more valid evidence. Therefore, the current study was conducted to compare the effects of directed peer feedback types (feedback: FB, feedforward: FF, and combination of feedback and feedforward: FB+FF) and an undirected peer feedback (UF) type on students’ provided peer feedback quality, argumentative essay writing skills, and acquisition of domain-specific knowledge. Due to the exploratory nature of this study, we addressed the gap in the literature in the form of research questions rather than hypotheses. Accordingly, the following research questions are formulated to guide this study.

1. What is the quality of students’ peer feedback under FB, FF, FB+FF, and UF conditions?

2. What are the effects of FB, FF, FB+FF, and UF types on students’ argumentative essay writing quality?

3. What are the effects of FB, FF, FB+FF, and UF types on students’ domain-specific knowledge acquisition?

2. Method

2.1. Context and participants

The study took place at a university in the Netherlands that is specialized in life sciences. The participants were 221 Bachelor of Science (BSc) students enrolled in a course named “Introduction to Molecular Life Sciences and Biotechnology” and followed an online learning module. The mean age of the participants was 18.42 years (SD = 1.34, MIN = 16, MAX = 28). Students were mostly Dutch (97%). About 70% of the participants were male and 30% were female.

2.2. Study design

An exploratory study with a pre-test and post-test design was conducted. Students were randomly grouped in pairs and assigned to four conditions with different peer feedback types, namely, feedback (FB), feedforward (FF), feedback with feedforward (FB+FF), and undirected feedback (UF). The FB type directs the feedback to address the progress that has been made toward the goal (i.e. How am I going/doing?). Students in this condition were provided with question prompts that can guide them in explaining the current state of their peer's essays and identifying problems in the essay. The FF type directs the feedback to provide advice on activities that need to be undertaken to make better progress (i.e. where to go next?). In this condition, students were given question prompts that assist them with providing suggestions with action plans to peers on how to improve their essays. The FB+FF version of the feedback combines the two aforementioned types, that is, it directs the provision of feedback to both the current state of the essay and advice on activities to make better progress (Hattie & Timperley, 2007). Finally, the UF type does not guide or direct the feedback, thus it is left up to the student. This means that in the UF condition, students did not have any forms of guidelines and support (e.g. question prompts, scripts, training, etc.) that could direct them in providing feedback to their peers regarding their essays (see Table 1). The design of directed peer feedback types was based on the scripts suggested by Noroozi et al. (2016) and Latifi et al. (2021b). However, the types used in the current study were further elaborated to guide the feedback process and facilitate the provision of directed (FB, FF, and FB+FF) feedback and UF. Students in four conditions were asked to provide peer feedback on their peers’ argumentative essay writing. Students’ feedback on their peers’ essays was built on the structure of high-quality argumentative essay writing (Noroozi et al., 2016; Toulmin, 1958) including a) intuitive opinion on the topic, b) arguments and data in favor, c) counter-arguments and opposing points of view (if existing) and data against, d) integration of the pros and cons considering the opinions of advocates and opponents of the topic, and e) a final conclusion (Table 1). The mode of peer feedback activity and writing an argumentative essay was asynchronous and students had to do it individually. Therefore, students had the convenience of completing each phase at their own pace from any location at any time within the stipulated time frame. In addition, students’ identity was anonymous within the system as usernames were generic (e.g. “Student 1”) (see Table 2).

Table 1. Elements of a high-quality argumentative essay (left), directed peer feedback strategies (feedback “FB”, feedforward “FF”, and their combination “FB+FF”), and undirected peer feedback strategy (UF).

Elements of a high-quality argumentative essay	Directed Peer feedback			Undirected peer feedback
	Feedback (FB)	Feedforward (FF)	FB+FF	UF
Intuitive opinion on the topic	Did your learning partner present his/her intuitive opinion on the topic? Please explain.	What is your advice to your learning partner to (better) present his/her intuitive opinion on the topic? Please explain.	In this condition, students received both question prompts in FB and FF conditions as guidelines to give feedback to their peers.	In this condition, students had the freedom to give their feedback on their peer's essays without following any specific format or guidelines. However, students were required to give comments with a minimum of 350 words and a maximum of 450 words.
Arguments in favor of the topic (pros)	Did your learning partner provide arguments in favor of the topic? Please explain.	What is your advice to your learning partner to (better) provide arguments in favor of the topic? Please explain.
Scientific facts in favor of the topic (pros)	Did your learning partner provide scientific facts (evidence, examples, figures, etc.) in favor of the topic? Please explain.	What is your advice to your learning partner to (better) provide scientific facts (evidence, examples, figures, etc.) in favor of the topic? Please explain.
Arguments against the topic (cons)	Did your learning partner provide arguments against the topic? Please explain.	What is your advice to your learning partner to (better) provide arguments against the topic? Please explain.
Scientific facts against the topic (cons)	Did your learning partner provide scientific facts against the topic? Please explain.	What is your advice to your learning partner to (better) provide scientific facts against the topic? Please explain.
Opinion on the topic considering various pros and cons.	Did your learning partner take into account the integration of various pros (arguments in favor of the topic) and cons (arguments against the topic) in forming his/her opinion of the topic? Please explain.	What is your advice to your learning partner to (better) form his/her opinion of the topic taking into account the integration of various pros (arguments in favor of the topic) and cons (arguments against the topic)? Please explain.
Scientific facts to support opinion regarding the integration of various pros and cons of the topic	Did your learning partner provide scientific facts to support his/her opinion on the integration of various pros and cons of the topic? Please explain.	What is your advice to your learning partner to (better) provide scientific facts to support his/her opinion on the integration of various pros and cons of the topic? Please explain.
Conclusion and statement on the topic	Did your learning partner come to a conclusion on the topic based on his/her arguments? Please explain.	What is your advice to your learning partner to (better) come to a conclusion on the topic based on his/her arguments? Please explain.

Table 2. The procedure of the study and design of the online learning module.

Phase	Estimated time	Start date	Deadline
Introduction	20 min	Sep 20th
Pre-test	35 min	Sep 20th 08:00hrs	Sep 20th 20:00hrs
Write an argumentative essay	105 min	Sep 21st 08:00hrs	Sep 21st 23:30hrs
Peer feedback	90 min	Sep 22nd 08:00hrs	Sep 22nd 23:30hrs
Revise the essay, post-test, and debrief	100 min	Sep 23rd 08:00hrs	Sep 23rd 23:30hrs
Total	350 min

2.3. Procedure and design of the online learning module

A tailor-made online learning module was implemented based on a regular course assignment that was redesigned to work as a digital module. The module offers information to students in various formats such as texts, tables, figures, and images. The module only presents necessary and relevant information, and clear instructions and messages to allow students to focus on the activity at stake. For this study, we used the asynchronous format of learning as we wanted students to take their time in writing argumentative essays and providing peer feedback since these were considered complex skills that demand high cognitive processing (Van Popta et al., 2017). In addition, this online module only allows students to engage with learning materials and tasks anonymously. Therefore, students were anonymously involved in peer feedback tasks. This choice was made to avoid possible emotional distress among students that could happen when they receive critical comments from their peers (Noroozi et al., 2016). Moreover, for each input box (i.e. essay and feedback) the system checked that the number of words was within the lower and upper bounds; if that was not the case, the system provided textual and visual feedback. In addition, each text field had a word counter. For authentic online educational classrooms, this means less low-order workload for teachers as technology can do the job of checking out the structure for essay and feedback tasks.

The specific learning topic for this module was “insect cells for cultured meat manufacturing” which falls under the overarching theme of Genetically Modified Organisms (GMOs). The online module was comprised of four main phases over a period of four consecutive days as described in the section on the procedure of the study (see Figure 1).

Figure 1. The procedure of the study.

On the first day, students received a verbal introduction to the module during class time (20 min). This introduction included all the required information and instructions for students (e.g. information on the research set-up of the study, expected goals, instructions on how to follow the modules, word limits for essays and feedback, deadlines, etc.). Students were also requested to complete questionnaires on socio-demographic information and domain-specific knowledge (35 min).

On the second day, students had to individually read a text about “how to write an argumentative essay” followed by an example of an argumentative essay (30 min), search online for more information sources such as daily papers, periodic journals, and scientific papers (30 min), and write an individual argumentative essay of ca. 500 words (min. 450, max. 550 words) about: “Insect-cell biomass infected with genetically modified baculovirus is a healthy meat alternative” (45 min). Students’ original essays were considered as the pre-test.

On the third day, dyads were formed and each student in the dyad groups had to individually provide feedback to his/her learning partner using a feedback form (90 min). No specific assignment strategy was used to form dyads of students. That means that students were randomly assigned to dyads and they were asked to give feedback to each other. Students in four different conditions performed their peer feedback task. The feedback form for students in the FB and FF conditions had a question and a text field of ca. 50 words (min. 40, max. 60 words) for each element of a high-quality essay (see Table 1). Students in the FB+FF condition received a feedback form comprising both the FB and the FF forms. In contrast, the feedback form for students in the UF condition was comprised of a request to provide feedback and a text field to write their feedback of ca. 400 words (min. 350, max. 450 words). Students in the UF condition did not have to follow any guidelines and they had the freedom in providing their comments.

On the fourth and last day, students were invited to read the feedback from their learning partner (15 min), and then individually revise their argumentative essay (45 min), (ca. 500 words, min. 450, max. 550 words). Students’ revised essays were considered as the post-test. In addition, students completed a questionnaire on domain-specific knowledge (equal to a pre-test) (35 min). Finally, students were debriefed (5 min). The total time investment for the students was ca. 335 min.

2.4. Measurements

2.4.1. Measurement of students’ feedback quality

An adjusted version of a coding scheme developed by Noroozi et al. (2016) was used to assess the quality of the feedback given by the students (see Appendix 1). The scheme was in line with the peer feedback script that students used to provide feedback to their learning partner's argumentative essay. This coding scheme was developed according to the literature (Andrews, 1995; Toulmin, 1958; Wood, 2001) and the characteristics of a complete and sound argumentative essay in the context of biotechnology (see Table 1). A series of consultation meetings with a panel of experts and teachers were used to validate the coding scheme (Noroozi et al., 2016). The coding scheme is comprised of a set of variables with different levels of proficiency that describe the quality of the student's feedback. Each level has a label, points, and description. The feedback given by all students was coded and scored in terms of the following variables: Intuitive opinion, claims in favor of the topic, justification for claim(s) in favor of the topic, claims against the topic, justification for claim(s) against the topic, integration of pros and cons, and conclusion. A score, between zero and two, was given for each of the aforementioned variables as follows: zero points if feedback was not present, one point for non-elaborated feedback, and two points for elaborated feedback. All given points for these elements were summed up together and indicated the student's total score for the quality of the given feedback. Two coders (i.e. the first author and a trained coder) participated in analyzing the essay data based on the coding scheme. The inter-rater agreement between two coders was calculated randomly by selecting 5% of the students’ feedback (equally distributed for FB, FF, FB+FF, and UF conditions). To assure the reliability of the coding process, the coder was trained on such processes including the coding rubrics. Then, the first author and the coder independently coded 5% of the data. The interrater agreement was substantial (Cohen's Kappa = 0.68) according to Landis and Koch (1977). Discrepancies were resolved through discussion until an agreement was reached on how to resolve them. Afterward, the coder coded the remaining data.

2.4.2. Measurement of students’ argumentative essay writing quality

“The quality of students” written argumentative essays on the topic was measured using an adjusted version of a coding scheme developed by Noroozi et al. (2016) (see Appendix 2). The coding scheme validity was obtained in the same way as the validity of the scheme to measure students’ feedback quality, that is, from a series of consultation meetings with a panel of experts and teachers (Noroozi et al., 2016). The scheme consists of a set of variables with various levels of proficiency that depict the quality of the student's argumentative essay. Each level is defined with a label, points, and description to facilitate the coding. The original and revised essays of all students were coded in terms of the variables: Intuitive opinion, arguments in favor of the topic (pro-arguments), scientific facts in favor of the topic (pro-facts), arguments against the topic (con-arguments), scientific facts against the topic (con-facts), opinion on the topic considering various pros and cons (integration of pros and cons), scientific facts to support opinion regarding the integration of various pros and cons of the topic (integration of pro- and con-facts), and conclusion. A score of zero, one, or two was given for each of the aforementioned variables. The assessment was done as follows: zero points were given if the element was not mentioned, one point if the element was non-elaborated, and two points if the element was elaborated. All given points for these elements were summed up together and indicated the student's total score for the quality of the written argumentative essay. The coding scheme was used in two phases. In the first phase, it was used to assess students’ original essays and in the second phase, it was used to assess students’ revised essays. The gain in the quality of students’ argumentative essays was measured based on the mean score differences between the original essay and the revised essay. Likewise, in coding feedback data, the same two coders (i.e. the first author and a trained coder) participated in analyzing the essay data based on the coding scheme. The inter-rater agreement was calculated following the same process used before to calculate the inter-rater agreement for the quality of students’ feedback. The results showed a high level of agreement between the two coders (Cohen's Kappa = 0.87) according to Landis and Koch (1977). Discrepancies were resolved through discussion until an agreement was reached on how to resolve them. Then, the coder coded the rest of the data.

2.4.3. Measurement of students’ domain-specific knowledge acquisition

Students’ domain-specific knowledge acquisition was measured during the pre-test and post-test by using a multiple-choice questionnaire. This questionnaire was developed by the course coordinator and it was comprised of 17 items (e.g. “what is a continuous animal cell line?”, “insects that are commercially cultivated include … ”, “a baculovirus is … ”, and “what is a “master cell bank”? For each question, students received a point, for a total of 18 points. Then, the domain-specific knowledge score was calculated for each student on a scale from 0 to 1 (#points/18) and then multiplied by 10 to have scores on a scale from 0 to 10. The result was used as the domain-specific knowledge score for the given test.

2.5. Analysis

A one-way analysis of variance (ANOVA) test was used to compare the mean differences in students’ feedback quality in different peer feedback types. Post hoc comparisons were performed using Bonferroni's test to see the extent to which students’ peer feedback quality differs among different peer feedback types. A one-way multivariate analysis of covariance (MANCOVA) was conducted to compare the effectiveness of the directed peer feedback script on the quality of students’ written argumentative essays and domain-specific knowledge acquisition. Such an omnibus test was used to avoid getting an artificially inflated alpha as a result of conducting multiple statistical tests on the same sample. The quality of the received feedback received was used as the covariate in the analysis. To remove the effect of the condition on the covariate, the covariate was corrected. The covariate corrected for a student is given by the quality of feedback received minus the feedback quality mean of the student group condition, that is, covariate corrected = CC_ij = C_ij-C ̅_i, where i is i-th condition and j is j-th student within the condition. Finally, to reduce the impact of potential sources of bias we used the method of winsorizing, that is, outliers were substituted with the highest value that was not an outlier.

3. Results

3.1. What is the quality of students’ peer feedback under FB, FF, FB+FF, and UF conditions?

ANOVA tests indicated a statistically significant difference in the quality of feedback provided for the four conditions, Welch's F(3, 92.15) = 46.83, p < .005, η2 =  .39, with a large effect (Cohen, 1988, pp. 284–287). Post hoc comparisons indicated that the mean score of the FB condition (M = 14.26, SD = 2.08) was significantly different from the FF condition (M = 7.95, SD = 3.75), the FB+FF condition (M = 11.25, SD = 2.75), and the UF condition (M = 8.98, SD = 3.45). Similarly, the FB+FF condition was significantly different from the FF and UF conditions. Finally, the UF condition was significantly different from the FF condition. Such a result indicated that the directed peer feedback script was more effective in supporting and directing the creation of feedback and a combination of feedback and feedforward. In contrast, the script was not very effective supporting and directing the creation of feedforward.

3.2. What are the effects of FB, FF, FB+FF, and UF types on students’ argumentative essay writing quality?

The covariate, the quality of feedback received, presented a small significant relationship with the quality of writing argumentative essays, F(1, 165) = 4.14, p = .043, η2 =  .026, indicating that the quality of writing argumentative essays is influenced by the quality of the feedback received. Moreover, there was not a significant effect of the directed and undirected peer feedback types on the quality of writing argumentative essays gain after controlling for the effect of the quality of feedback received, F(3, 155) = .231, p = .875, that is, the gain was similar for all the conditions. A comparison between the estimated marginal means showed that the biggest gain, yet not significant, was obtained by the FB condition (M = 1.74) followed by the FB+FF, FF, and UF conditions (M = 1.50, 1. 68, 1.20 respectively), see also Table 3.

Table 3. Quality of writing argumentative essays scores for the original and revised essays for all conditions.

Conditions		Original essay		Revised essay
	N	M	SD	M	SD
UF	45	10.17	2.09	11.37	2.63
FB	47	8.87	2.63	10.61	2.67
FF	48	8.79	2.39	10.47	2.36
FB+FF	44	9.65	1.79	11.15	1.86

3.3. What are the effects of FB, FF, FB+FF, and UF types on students’ domain-specific knowledge acquisition?

The covariate, the quality of feedback received, presented a small significant relationship with the acquisition of domain-specific knowledge, F(1, 155) = 4.84, p = .02, η2 =  .03, indicating that the acquisition of domain-specific knowledge is influenced by the quality of the feedback received. In addition, there was not a significant effect of the directed and undirected peer feedback types on domain-specific knowledge acquisition after controlling for the effect of the quality of feedback received, F(3, 155) = .38, p = .76, indicating that the gain was similar for all the conditions. A comparison between the estimated marginal means showed that the biggest gain, yet not significant, was obtained by the FF condition (M = 11.92) followed by the FB, UF, and FB+FF conditions (M =  9.84, 9.83, and 9.09 respectively), see also Table 4.

Table 4. Domain-specific knowledge scores for the pre-test and post-test for all conditions. Scores were transformed such that the maximum possible score was 100.

Conditions		Pre-test		Post-test
	N	M	SD	M	SD
UF	41	49.17	9.35	59.00	12.86
FB	45	51.20	9.49	61.04	9.58
FF	48	50.06	11.36	61.98	11.10
FB+FF	42	50.48	9.29	59.57	11.25

4. Discussion

4.1. Discussion on RQ1

The results for the first research question showed that the quality of provided feedback among directed and undirected peer feedback types is different. Students’ feedback quality in FB and FB+FF conditions was higher than students in the UF condition. This finding indicates that directed peer feedback types namely FB and FB+FF led to better peer feedback quality than the undirected peer feedback type. The result also suggests that if the directed peer feedback type only focuses on providing feedforward, it can be less effective than undirected peer feedback. This finding is supported by prior studies where high feedback quality is reported for directed peer feedback types including scripted peer feedback and feedback and feedforward combination (Latifi et al., 2020; 2021b).

In general, the directed peer feedback types guided and helped students to provide high-quality feedback to their learning partners which is favorable to improving their writing skills (DeNisi & Kluger, 2000). In addition, the script supported the provision of feedback related to the task rather than on personal evaluations or the effect on the learning partner which is considered to be less effective (Hattie & Timperley, 2007). Moreover, the directed peer feedback types allowed the provision of structured and sequential feedback on each of the different elements of a high-quality argumentative essay, thus facilitating the improvement of each of the elements of the argumentative essay. However, it is important to take into consideration that all students received argumentative essay theory and an example of an argumentative essay and that may have played a role in the feedback provision process. Such statement is supported by the fact that the UF condition, which was not scripted, outperformed the FF condition. The previous information suggests that scaffolding the feedback provision process with theory and examples is effective to foster high-quality feedback, and a further combination with scripts to support and direct the feedback is even more effective. Yet, is necessary to understand why the script was not as effective in the FF condition. One possible explanation is that it is not customary for students to receive feedback containing possible directions to pursue or about alternative types to follow, but rather corrective feedback on their performance on the actual task. As such, it may have been difficult for the students to provide feedback containing alternative directions or types. To conclude, effective feedback should 1) reduce the gap between what is understood and what should be understood, and 2) increase students’ effort, motivation, and engagement to reduce such gap (Hattie & Timperley, 2007). In addition, effective feedback must include 1) information related to the actual task and/or performance taking as reference an expected standard, prior performance, and/or the success or failure on (part of) the task (How am I going/doing?), and 2) information related to possible directions or alternative types to follow (Where to go next?) (Hattie & Timperley, 2007).

4.2. Discussion on RQ2

There were increases in the quality of writing argumentative essays from the original essay to the revised essay in all conditions. In addition, the quality of feedback received presented a small significant relationship with the quality of writing argumentative essays. These results are in line with previous research claiming positive results of peer feedback on writing skills (Gabelica et al., 2012; Kellogg & Whiteford, 2009; Latifi et al., 2020; 2021b; Noroozi et al., 2016). During the peer feedback process, students contrasted their solutions with the ones from their learning partners. As a result, students were able to identify and rectify mistakes and misconceptions (Shute, 2008), broaden and deepen their reasoning and understanding (Yang, 2010), understand the differences between the current and the expected state, and what to do and how to do it to improve and do better (DeNisi & Kluger, 2000; Hattie & Timperley, 2007). Moreover, the peer feedback process guided students learning (Orsmond et al., 2005), facilitated problem-solving skills and self-regulation (Shute, 2008), and triggered reflection (Phielix et al., 2010).

In this particular study, no significant differences were found between directed (i.e. FB, FF, and FB+FF) and undirected (i.e. UB) feedback in terms of the quality of writing argumentative essays. These results differ from previous literature indicating that the effectiveness of feedback is influenced by its type and the way it is provided (Hattie & Timperley, 2007). Our results may be explained if we consider that all conditions received theory on the composition of an argumentative essay and an example of an argumentative essay. In previous studies on argumentation scaffolds to foster argumentation knowledge, students received argumentation theory before engaging in argumentative discourse activities, but the effects of providing theory were not investigated (Kollar et al., 2007). Therefore, it is important to put in context the effects of theory and examples on the learning outcomes. We believe, that the provision of theory and an example may have diminished the effect of the directed peer feedback scripts, and may have made the scripts redundant and even unnecessary. This reasoning is in line with educational psychology (Wittwer & Renkl, 2010) and cognitive load theory (CLT) (Sweller et al., 1998) literature indicating the positive effects of theory, or instructional explanations, along with worked examples (also known as example-based learning). For instance, providing theory and examples can prevent misconceptions and inconsistencies, and facilitate understanding (Wittwer & Renkl, 2010). In addition, the theory-example combination can be effective at fostering the acquisition of meaningful and flexible knowledge (Van Gog et al., 2004) since both the product-oriented information, i.e. the how, and the process-oriented information, i.e. the rationale or the why some solution steps should be conducted (Wittwer & Renkl, 2010), are provided. Similarly, previous research found that example-based learning is more effective when it is accompanied by problems to be solved (Pashler et al., 2007). Therefore, students supported with instructional scaffolds that combine theory, examples, and practice might profit more. Nevertheless, the effectiveness of worked examples is diminished as students get more experience, i.e. the expertise-reversal effect (Kalyuga et al., 2003). In addition, experienced students might not need further instructional support, as they have to invest cognitive resources in redundant information. Redundant information might hamper learning as it might result in unnecessary overloading and suboptimal learning processes due to the redundancy effect (Sweller et al., 1998). The redundancy effect is in line with the idea of over-scripting (Dillenbourg, 2002), which can occur due to too much scaffolding, or when the provision of external support deters the self-regulated application of the student's internal script (Fischer et al., 2013). Hence, the provision of instructional support should consider the student's internal script and should be decreased over time to foster the learning of self-directed learning skills (Noroozi & Mulder, 2017). To conclude, we believe that providing theory and an example nullified the effects of the directed peer feedback scripts in this study. In addition, caution should be exercised while piling up or combining instructional scaffolds, as the effects of some scaffolds may be nullified or may result in suboptimal learning processes due to overloading, e.g. redundancy effect and over-scripting. Last, the theory-example combination seems to be a powerful instructional support to foster argumentative essay writing.

4.3. Discussion on RQ3

The results also revealed an increase in the quality of domain-specific knowledge acquisition from pre-test to post-test in all conditions. This finding means that peer feedback regardless of its type is an effective instructional strategy to increase students’ domain-specific knowledge acquisition. Such results are in line with previous research claiming positive results of peer feedback on domain-specific knowledge acquisition (Latifi et al., 2021b; Noroozi et al., 2022b; Noroozi & Mulder, 2017; Valero Haro et al., 2019). One reason that might explain this finding is that peer feedback, in general, is seen as a process-oriented pedagogical activity in classrooms (Kerman et al., 2022; Shute, 2008) that, more or less depending on its design, encourages students to critically engage in collaborative discussions on a specific topic, recall their prior knowledge on the discussed topic, analyze and review peers’ works, identify gaps and problems, and suggest points for improvements (Topping, 2009). In addition, peer feedback allows students to reflect on their own knowledge conceptions and confirm, complement, overwrite, or restructure them (Valero Haro et al., 2019). Involving in such an informative knowledge-shared collaborative learning process might tend to improve students’ domain-specific knowledge acquisition. Another plausible reason to explain this finding is that in this particular study, students in all conditions received theory on the discussed topic. This means that all conditions were given an equal chance to acquire some basic knowledge on the topic and this can partly explain students’ domain-specific knowledge gain in all conditions. Therefore, it is possible that giving theoretical knowledge may have neutralized the effect of the directed peer feedback types. Similar to the discussion on RQ2, this reasoning can be valid in the case of domain-specific knowledge acquisition as well: combining theory and examples could positively impact the acquisition of meaningful and flexible knowledge (Van Gog et al., 2004). That is to say that students improved their knowledge acquisition in all conditions probably due to the combination of given theory and peer feedback (regardless of its type).

5. Limitations and suggestions for future research and practice

There are some limitations of this study that need to be acknowledged and considered for future studies. First, this study was conducted in vivo. This setting provided advantages and disadvantages. An advantage is the high practical relevance and high ecological validity of the study due to the real educational setting instead of laboratory settings in which motivational aspects may be affected due to synthetic learning environments and rewards upon successful completion of the tasks and activities. In contrast, to level the field of play for all the students, all the conditions received theory on the composition of an argumentative essay and an example of an argumentative essay. The latter may have affected the effectiveness of the directed peer feedback scripts, making the scripts redundant or even unnecessary. Therefore, we make a call to exercise caution while combining multiple instructional scaffolds, as some scaffolds may nullify others, which may result in suboptimal learning processes due to overloading, e.g. redundancy effect and over-scripting. The second possible criticism of the study is related to the scale and the coding scheme used to measure the quality of peer feedback. The scale, which was developed and successfully used before by Noroozi et al. (2016), has a scale ranging from 0 to 2. The scale may not offer a large spectrum of variation in contrast to a scale with more points, yet it is able to provide some insight into the quality of peer feedback. Moreover, we created a rubric, and used examples (depicting the characteristics that should be met to assign a score) to ensure consistency of the measures and assess change equally well across the entire range of the construct. In addition, the coding process of the quality of feedback disregarded if the actual feedback provided by the student corresponded, either completely or partially, to the scripted feedback type. Such type of analysis may have influenced the results. Yet, we believe that analyzing the extent to which the feedback script fosters the provision of a given feedback type deserves a deeper analysis and thus further research to understand the possible reasons and processes behind such behavior. Third, the present study only measured the effect of the intervention in the short term but not in the longer term. Therefore, future research should investigate the effects of theory and worked examples in contrast to scripting (e.g. feedback type). In addition, the effectiveness of the aforementioned instructional scaffolds should be evaluated considering the educational level of the students as their expertise and cognitive capacity may play a role.

Fourth, this study is limited regarding the feedback approach as it is designed from a one-way perspective while formative behavior goes beyond that, and includes an argumentative dialogue, in which questions are being asked to confront or further sharpen the argumentative process of the student, and continuous assessment of the further argumentation process of the student (peer) is conducted by the teacher (or peer). Future studies should focus on the impacts of dialogic and interactive peer feedback types on feedback quality, essay quality, and domain-specific knowledge acquisition. In addition, there is no 0-measurement in the assessment of feedback behavior in this study. Probably, students differ a lot in their capacity of giving (different types of) feedback. However, there is no possibility to report the effect scores of the feedback instructions on feedback behavior, in terms of standardized effects scores for the four feedback types. Moreover, as we mentioned before, we acknowledge the risk of over-scripting and redundancy effect in this study which might have happened due to overloading students with too much theoretical information combined with examples, scripts, and work examples. It might be interesting for future studies to do a longitudinal study to examine the interaction effect between the type of feedback and the type of support over time. Future research can study the optimal intensity of feedback support for novel and experienced students. Last, future studies should have a longitudinal design to assess student learning, internalization of the constructs, and its application in the same and different contexts.

Despite its limitations, the present study offers implications for future education practice in online higher education settings. First, we found that students’ feedback quality in online higher education can be influenced by the types of feedback support. Based on this finding, we suggest teachers create rubrics or scripts in their online courses that provide instructions and guidelines for students on how to give feedback on peers’ essays. In this way, students are most likely expected to perform better compared to a situation where they do not receive relevant instructions and guidelines. Second, although we found no significant differences between directed and undirected peer feedback types for improving peers’ essay quality and domain-specific knowledge acquisition, directed peer feedback namely FB strategy showed a bit higher impact compared to other peer feedback types. This finding implies that even though teachers are welcome to adopt different peer feedback types based on their own and students’ preferences, we recommend using directed peer feedback. However, one critical note for consideration is that teachers should not overwhelm students with too many guidelines and instructions as it can impose the over-scripting risk which can negatively impact students’ learning processes and outcomes.

6. Conclusions

This study provided insights into how different types of peer feedback influenced students’ quality of peer feedback and argumentative essay writing, and also their acquisition of domain-specific knowledge in an online learning environment. We found that the quality of feedback provided by students differed significantly depending on the type of peer feedback. The directed peer feedback was more effective in supporting students with the provision of feedback only or a combination of feedback and feedforward. However, this directed peer feedback was less effective in supporting and directing students with the creation of feedforward alone. Such results indicate that the quality of peer feedback can be enhanced or diminished depending on the extent to which peer feedback is directed and how this is done. The results also showed an increase in the quality of argumentative essays and domain-specific knowledge acquisition in all conditions. However, no significant differences in quality gain were found between the different conditions. These results indicate that giving peer feedback, in nature regardless of its format and type, is a powerful and effective learning strategy to enhance students’ learning outcomes.

Our results add value to the existing literature where the effects of implementing different peer feedback types on the quality of peer feedback and associated learning outcomes within online settings have been investigated. The results support the evidence that different peer feedback types can lead to different levels of peer feedback quality in online learning environments. These results are important for future practice in online higher education contexts in the sense that they provide a conclusive picture of the role of directed and undirected peer feedback types for enhancing students’ learning processes and outcomes.

Availability of data

The authors declare that the data can be available based upon on a request.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Anahuac Valero Haro

Anahuac Valero Haro is a PhD graduate from the Education and Learning Sciences (ELS) Chair Group, Wageningen University, the Netherlands. His research interests include e-learning and distance education, computer-supported collaborative learning and computer-supported collaborative argumentation to facilitate acquisition of domain-specific knowledge and argumentation competence.

Omid Noroozi

Omid Noroozi is an Associate Professor at the Education and Learning Sciences (ELS) Chair Group, Wageningen University, the Netherlands. His research interests include collaborative learning, e-learning and distance education, computer-supported collaborative learning (CSCL), argumentative knowledge construction in CSCL, argumentation-based CSCL, CSCL scripts and peer feedback.

Harm J. A. Biemans

Harm J. A. Biemans is an Associate Professor at the Education and Learning Sciences (ELS) Chair Group, Wageningen University, the Netherlands. His research interests concern competence development, competence-based education, educational psychology, educational development and evaluation, and computer-supported collaborative learning.

Martin Mulder

Martin Mulder is an Emeritus Professor from the Education and Learning Sciences (ELS) Chair Group, Wageningen University, The Netherlands. His research interests include competence theory and research, human resource development, computer-supported collaborative learning, competence of entrepreneurs, competence of open innovation professionals, and interdisciplinary learning in the field of food quality management education.

Seyyed Kazem Banihashem

Seyyed Kazem Banihashem is an Assistant Professor at Open University of the Netherlands and a researcher at Wageningen University and Research. His research interests include technology-enhanced learning, learning analytics, learning design, argumentation, formative assessment, and feedback.

Appendices

Appendix 1. Coding scheme to analyse the quality of students' feedback process with variables, points, labels, and corresponding descriptions (Noroozi et al., 2016)

Table

Variables	Points	Labels	Descriptions
Intuitive opinion	Zero	Not mentioned	No feedback on intuitive opinion is presented.
	One	Not elaborated	Feedback on intuitive opinion is presented but justification for the feedback is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Feedback on intuitive opinion is presented and justification for the feedback is discussed or elaborated on in a significant way.
Claims in favor of the topic	Zero	Not mentioned	No feedback on pro-claim(s)is presented.
	One	Not elaborated	Feedback on pro-claim(s) is presented but justification for the feedback is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Feedback on pro-claim(s) is presented and justification for the feedback is discussed or elaborated on in a significant way
Scientific facts in favor of the topic	Zero	Not mentioned	No feedback on scientific facts in favor of the topic is presented.
	One	Not elaborated	Feedback on scientific facts in favor of the topic is presented but justification for the feedback is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Feedback on scientific facts in favor of the topic is presented and justification for the feedback is discussed or elaborated on in a significant way.
Claims against the topic	Zero	Not mentioned	No feedback on con-claim(s)is presented.
	One	Not elaborated	Feedback on con-claim(s) is presented but justification for the feedback is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Feedback on con-claim(s) is presented and justification for the feedback is discussed or elaborated on in a significant way.
Scientific facts against the topic	Zero	Not mentioned	No feedback on scientific facts against the topic is presented.
	One	Not elaborated	Feedback on scientific facts against the topic is presented but justification for the feedback is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Feedback on scientific facts against the topic is presented and justification for the feedback is discussed or elaborated on in a significant way.
Integration of pros and cons	Zero	Not mentioned	No feedback on integration of pros and cons is presented.
	One	Not elaborated	Feedback on integration of pros and cons is presented but justification for the integration is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Feedback on integration of pros and cons is presented and justification for the integration is discussed or elaborated on in a significant way.
Conclusion	Zero	Not mentioned	No feedback on conclusion is presented.
	One	Not elaborated	Feedback on conclusion is presented but justification for the feedback is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Feedback on conclusion is presented and justification for the feedback is discussed or elaborated on in a significant way.

Appendix 2. Coding scheme to analyse the quality of students' argumentative essays with variables, points, labels, and corresponding descriptions (Noroozi et al., 2016).

Table

Variables	Points	Labels	Descriptions
Intuitive opinion	Zero	Not mentioned	No intuitive opinion is presented.
	One	Not elaborated	Intuitive opinion is presented but it is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Intuitive opinion is presented and it is discussed or elaborated on in a significant way.
Arguments in favor of the topic (pro-arguments)	Zero	Not mentioned	No argument is presented in favor of the topic.
	One	Not elaborated	Only one argument is presented in favor of the topic.
	Two	Elaborated	Multiple arguments (two or more) are presented in favor of the topic.
Scientific facts in favor of the topic (pro-facts)	Zero	Not mentioned	No justification for pro-argument(s) is presented. None of the pro-claim(s) are justified. Pro-argument(s) are just being presented without any back up/support in terms of presenting scientific facts, evidence, examples, figures etc.
	One	Not elaborated	Justification for pro-argument(s) is presented. Pro-argument(s) are being backed up/supported (with scientific facts, evidence, examples, figures etc.) but they are not strongly connected to the pro-argument(s). Justification for pro-argument(s) are not discussed or elaborated on in a significant way.
	Two	Elaborated	Justification for pro-argument(s) is presented. Pro-argument(s) are being backed up/supported (with scientific facts, evidence, examples, figures etc.) and they are strongly connected to the pro-argument(s). Justification for pro-argument(s) are discussed or elaborated on in a significant way.
Arguments against the topic (con-arguments)	Zero	Not mentioned	No argument is presented against the topic.
	One	Not elaborated	Only one argument is presented against the topic.
	Two	Elaborated	Multiple arguments (two or more) are presented against the topic.
Scientific facts against the topic (con-facts)	Zero	Not mentioned	No justification for con-argument(s) is presented. None of the con-argument(s) are justified. Con- argument(s) are just being presented without any back up/support in terms of presenting scientific facts, evidence, examples, figures etc.
	One	Not elaborated	Justification for con-argument(s) is presented. Con-argument(s) are being backed up/supported (with scientific facts, evidence, examples, figures etc.) but they are not strongly connected to the con-argument(s). Justification for con- argument(s) are not discussed or elaborated on in a significant way.
	Two	Elaborated	Justification for con-argument(s) is presented. Con-argument(s) are being backed up/supported (with scientific facts, evidence, examples, figures etc.) and they are strongly connected to the con-argument(s). Justification for con-argument(s) are discussed or elaborated on in a significant way.
Integration of pros and cons	Zero	Not mentioned	No integration of pros and cons is presented.
	One	Not elaborated	Integration of pros and cons is presented but justification for the integration is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Integration of pros and cons is presented and justification for the integration is discussed or elaborated on in a significant way.
Conclusion	Zero	Not mentioned	No conclusion is presented.
	One	Not elaborated	Conclusion is presented but it is not discussed or elaborated on or it is discussed in an insignificant way.
	Two	Elaborated	Conclusion is presented and it is discussed or elaborated on in a significant way.