The quality of argumentation and metacognitive reflection in engineering co-Design

ABSTRACT

Argumentation and metacognitive reflection are required for effective thinking and convincing argumentation in engineering co-design. This study investigated engineering students’ argumentation and metacognitive reflection in their final group reports and their correlation with the quality of their work in co-design. The groups practiced and gained experience in ideation in the product development process during academy-industry collaboration. Iterative theoretical content analysis and argumentation models were integrated to create a framework for identifying acceptable argumentation. The results suggest that the groups co-constructed acceptable arguments containing claims, reasons, and warrants with references. Meanwhile, the frequency of counterarguments, conclusions in advanced argument structures, and metacognitive reflection in the reports was low. The study suggests that instruction of more developed argumentation models and metacognitive reflection must be carefully integrated into engineering design education to foster convincing, advanced argumentation.

KEYWORDS:

• Argumentation
• engineering education
• engineering design
• metacognition ∙reflection‌‌‌

Engineering education places a greater value on technical knowledge and skills than communication and collaboration (Naukkarinen and Bairoh 2022). However, it is important that engineering students develop the generalisable critical thinking skills and dispositions necessary for effective and professional reasoning through complex engineering issues and questions they will inevitably face (Paul, Niewoehner, and Elder 2013). After all, engineers need technical knowledge and argumentative skills to be appropriately convincing in their professional work with various industrial customers (Dym et al. 2005; Lappalainen 2013; Rittel 1987).

The European Society for Engineering Education has emphasized that critical, creative, and reflective thinking are necessary skills for engineers and that such skills should be acquired during their engineering studies and continuously refined throughout their professional careers (Murphy et al. 2016). According to Ennis (2015), argumentation skills relate to critical thinking as they involve reasonable, reflective thinking focused on deciding what to believe or do, offering clear reasons, considering different points of view, on the use of math and visual representations to reach and justify a position, judging the credibility of sources, using metacognitive skills to reflect on their thinking, and applying appropriate rhetorical strategies in presentations.

Argumentation, used to create and justify ideas, is an important element in engineering practice (i.e. Almudi and Ceberio 2015; Andrews 2010; Erduran and Villamanan 2009; Foutz 2019; Garcia 2017; Jonassen and Cho 2011). Argumentation is an especially relevant part of the design, during which members of a group justify a proposition by presenting and weighing its pros and cons to arrive at a certain standpoint on an issue (Rittel 1987). Argumentation relates to generic and discipline-specific skills (Andrews 2010). Engineering-related argumentation is a reasoning process used to construct knowledge with its own epistemological criteria and constraints (Andrews 2010; Wilson-Lopez et al. 2020).

Previous studies have focused on engineering student groups’ argumentation in their final thermodynamics reports and found that students had problems with writing arguments (Erduran and Villamanan 2009). Jonassen and Cho (2011) explored the argumentation and counter-argumentation put forth by individual undergraduate engineering students during ethical problem solving, in both an online learning and essay writing context. They found that students constructed higher quality counterarguments to arguments provided by someone else than when arguing against their own solution.

Engineering education has long used Aristotle’s (1991) model of argumentation centred on ethos (character and morals), pathos (emotional appeal), and logos (true and probable arguments) as the three modes of persuasion (Leite et al. 2011). Logos is emphasised in argumentation rather than pathos and ethos in written assignments (Leite et al. 2011). However, more research on groupś argumentation and other models than Aristotle during engineering co-design is needed. To address this gap in the research, this study analysed the quality of groups’ argumentation in their final reports in co-design in the capstone-project course. The groups reported on the design ideas they generated in response to a case study or a problem introduced by an industry partner. The final reports represent the capstone-type projects that are important for students who are learning to develop professional skills and to collaborate with potential employers (Picard et al. 2021). Final group reports are a popular form of assessment that prepares students for collaboration, individual efforts to take responsibility in a group, and participating in group work in their careers (i.e. Davies 2009).

Domain-specific evaluation of argumentation provides knowledge of the conceptual and epistemological content of argumentation (Henderson et al. 2018). It also provides an understanding of the effectiveness of argumentation and metacognition as elements of critical thinking in a given discipline (Claris and Riley 2012; Ennis 2015; Paul, Niewoehner, and Elder 2013).

The modified Toulmin’s argument pattern (TAP) (Toulmin 2003; Toulmin, Rieke, and Janik 1984) was constructed to identify the argument structures used by the groups. The tools that help to rebuild the argumentative text from the pragma-dialectical model (van Eemeren and Snoeck Henkemans 2017), the different types of argumentation schemes (Walton 1995; Walton, Reed, and Macagno 2008), and the standards of reasoning in engineering (Paul, Niewoehner, and Elder 2013) were all utilised in the evaluation of content in argument structures in the groups’ reports.

This qualitative discourse analysis revealed that the argument structures varied between the final reports, showing that written argumentation is a complex process if argumentation has not been taught systematically in the course. Some groups can promote clear and acceptable advanced argumentation that leads to a conclusion and/or solution; such argumentation can be limited if the advanced argumentation skills of all participants in the groups are not developed.

Argumentation models in engineering

Engineering education has long used Aristotle’s (1991) model of argumentation centred on ethos (character and morals), pathos (emotional appeal), and logos (true and probable arguments) as the three modes of persuasion (Leite et al. 2011). Logos is emphasised in argumentation rather than pathos and ethos (Leite et al. 2011). Rittel (1987) was the first to develop the concept of argumentation in engineering design by defining argumentation as reasoning that can generate a standpoint on design after assessing pros and cons.

The current models of argumentation have been extensively developed in the philosophy and communication sciences. The pragma-dialectical model (van Eemeren 2010), Walton’s (2006) method of argumentation, and Toulmin Argument Pattern (TAP) (Toulmin, Rieke, and Janik 1984) are not general theories of reasoning, but they offer criteria and methods for evaluating argumentation in oral and written contexts.

TAP (Toulmin, Rieke, and Janik 1984; Toulmin 2003) diagrams a classical argument structure that includes data that are classified as either a ground (facts, evidence), a warrant (supports a claim combined with grounds), a backing (generally supports the warrant), a modality (evaluates the warrant’s support for a claim, such as a presumable nature), a rebuttal (indicates circumstances in which the warrant must be set aside), a claim (makes an assertion), or a conclusion (makes a warranted conclusion) (see Appendix). Osborne, Erduran, and Simon (2004) modified TAP to identify science teachers’ argument structures because they found methodological challenges in separating the data into claims, warrants, and backings. Stronger argumentation uses data or evidence as warrants and rebuttals as counterarguments. Erduran (2018) argues that the Toulmin model still offers a relevant framework for science education because it can be modified to the specific context and purpose of the discourse.

Engineering studies have used the Toulmin (2003) model to evaluate students’ answers in example statistics problems (Foutz 2019) and the simplified Toulmin model to assess students’ written reports on thermodynamics (Erduran and Villamanan 2009). Almudi and Ceberio (2015) utilised Sampson and Clark’s (2009) model to explore the quality of arguments, evidence, and reasoning that students used in their explanations of electromagnetic problems. Jonassen and Cho (2011) utilised the pragma-dialectical approach (van Eemeren, Grootendorst, and Snoeck Henkemans 2002) and argumentation-counterargumentation as a refutation strategy (Nussbaum and Schraw 2007).

The pragma-dialectical approach to argumentation (van Eemeren and Snoeck Henkemans 2017; van Eemeren et al. 1993) offers a tool to rebuild implicit and unclear argumentation that orients to disagreement resolution by highlighting the elements necessary to solve differences of opinion in oral or written task. Acceptable written argumentation must convince readers by removing their doubts and it cannot contain inconsistencies (van Eemeren, Grootendorst, and Snoeck Henkemans 2002).

Engineering education has its intellectual standards to assess and critically reflect inferences as elements of reasoning. The standard for inferences related to the conclusion in the chain of argumentation is: (a) clarity (conclusions are clear), (b) logic (data support the conclusion, alternative conclusions are assessed, and conclusions are clearly stated), (c) significance (conclusions are important), (d) accuracy (separation of facts and speculation), (e) evaluation of depth (acknowledgement of complexity), and (f) validity of conclusions should be critically evaluated (Paul, Niewoehner, and Elder 2013).

Written argumentation

Argumentative writing requires reasoning, epistemological, and linguistic skills to generate, evaluate, and connect arguments and counterarguments and to present a conclusion or solution in a complex, coherent sentence (Coirier, Andriessen, and Chanquoy 1999). Learning through writing arguments is a way students can engage in universities’ epistemic practices; the way students write explicit arguments gives instructors insight into students’ tacit knowledge embedded in the writing process (Takao and Kelly 2003). Well-written argumentation includes the use of multiple and converging lines of evidence, clear reasoning, and data as warrants; poor argumentation involves vague reference to data, poorly evidenced arguments, and overreliance on implicit evidence (Kelly, Regev, and Prothero 2008). In engineering, writing good arguments requires technical content knowledge, logical reasoning, and persuasion, all of which are promoted by groupwork pedagogy (Yalvac et al. 2007).

Undergraduate students are often weak in constructing claims, warrants, and rebuttals, using data as evidence (Heng, Surif, and Seng 2014), attempting to construct valid conclusions from irrelevant evidence (Almudi and Ceberio 2015; Erduran and Villamanan 2009), or insufficiently evaluating counterarguments (Leite et al. 2011).

Metacognition and reflection

Iiskala et al. (2011) used the concept of socially shared metacognition or shared regulation to refer to the monitoring and regulation of joint cognitive processes in a group towards a common goal that demands collaborative problem-solving.

Metacognition, such as the ability to use relevant concepts, valid evidence, and strategies to construct counterarguments and avoid explanation, supports argumentation competence (Rapanta, Garcia-Mila, and Gilabert 2013). Reflection emphasises asking new questions and assessing self-knowledge to create new meanings for personal transformation (Claris and Riley 2012). Reflection is often conceptually referred to as metacognition, even though the concepts have subtle differences. Lousberg et al. (2020) found that undergraduate engineering students’ reflection on their own work promotes a greater understanding of design during academic studies. In engineering design, Cropley (2015) defined metacognitive processes in thinking that allow groups to reorganise plans when necessary, reflecting upon and monitoring own progress, being aware of alternative ways to do things, and understanding the costs and benefits related to changes in the project. In an early study, Flavell (1979) clarified that metacognition is the awareness and monitoring of one’s own thinking and action in learning. Metacognition comprises (a) metacognitive knowledge of personal awareness, tasks, and strategies, and (b) metacognitive experiences. Metacognitive experiences can affect knowledge and tasks and can activate strategies. Personal awareness relates to one’s knowledge of one’s own cognitive performance, while task knowledge is what a person knows about the difficulties and goals of a task. Finally, the strategy includes a person’s knowledge of strategies and how to use them in learning. Barzilai and Zohar (2014) integrated the dimensions of metacognition and epistemology in an epistemic metacognition framework. Accordingly, persons may reflect on the adequacy or certainty of their knowledge and on the coherence of their epistemic processes, and they may monitor and evaluate their own epistemic strategies (e.g. checking the validity of the sources used). Thus, I integrate metacognition and reflection to form the concept of metacognitive reflection.

The research questions of this study are as follows: (a) Do the structures co-constructed in the group arguments lead to conclusions or solutions in co-design? (b) What types of warrants do groups co-construct to support their argumentation in co-design? (c) Do groups demonstrate metacognitive reflection in co-design?

Materials and methods

Course design

The study was conducted in the context of the International Telecommunication graduation course at the School of Electrical Engineering in Finland. The course was selected for the experiment because the course instructor was interested in exploring the quality of arguments and counterarguments in final reports to assess the validity of reasoning in design. Verbal informed consent was obtained from all participants. Argumentation was not taught to students during the course because it was not a familiar instructional pedagogy in the school of engineering.

The groups studied ideation, product development, and group processes in theory during the spring term. The same group division was used in the experimental component during the fall term. Exceptions included some students who wished to change groups and students who did not continue in the fall term. The groups practiced ideation and worked in groups to develop a telecommunication service or product for an industrial customer. The course was taught in English.

This study focuses on idea generation, the second stage of the design process, as part of the real-life product development process that took place during the fall module. The instructional goals of the fall module were to (a) become acquainted with real-life product development processes, (b) understand and use ideation in product development problems, (c) identify possible applicable solutions, (d) examine and validate the more promising solution(s) in greater detail if possible; and (e) learn to cultivate and present knowledge in multidisciplinary groups, which was expected to boost creativity.

The groups completed two types of reports within two months of the end date of the fall workshop. For the University Report, the groups were advised to document the framework of the problem and to either accept or reject arguments using ideation methods, tools, and reliability assessments. In the Industry Partner Report, preliminary markets, technical and business assessments, solutions, bases for solutions, risks, techno-economic analyses, and quality estimates were required.

The groups were provided with the following instructions as special notes for reporting:

• The product/service description, marketing plan, and project plans can be used to verify the practical applicability of your plan.

• Strive to verify the quality and risks of your solution.

• Carefully document the methods you applied to obtain your solutions.

• Document intermediate steps in the path to your group’s solution(s)—this is required, especially for the University Report.

In particular, report:

• ­ Why the final solutions were selected?

• ­ Report/analyze your solution in as much detail as you can to verify your claims.

The groups were not given any instructions regarding the division of the writing task between the members; this was decided within the teams by the members.

Participants

Fifty-five Finnish undergraduate students (seven women and 48 men) and six international undergraduate students (two women and four men) formed eight groups in the experimental component of the authentic university course during the fall term. Each group comprised six to eight students. Groups 1, 2, 3, and 5 had at least one international student. Groups 6 and 7 comprised only men. The international students were from China, South Korea, Spain, Turkey, and Thailand. The majority of the students (n = 48) were preparing to write a thesis in Finnish or English for a bachelor’s or master’s degree in communications engineering in the Department of Communications and Networking at the School of Electrical Engineering. One student was from the School of Science, and one student represented another national research institute. There were one or two students from the Department of Electrical Engineering and Automation, Signal Processing and Acoustics, and Electronics and Nanoengineering.

Data

The eight groups’ final reports to the university and to the industrial partner from the fall-term workshop (n = 16) were chosen for the analysis. The student numbers were only written in the reports and were removed prior to the analysis. The industrial partneŕs name also was made anonymous. A purposive sampling method was employed (Miles, Huberman, and Saldaña 2020). The University Reports ranged from six to 22 pages, and the Industry Partner Reports ranged from five to eight pages. The Industry Partner Report was limited to six pages without appendices to condense the essential findings.

Data analysis

The research design is a case study by using a mix of qualitative content and quantitative data. A case study is appropriate when exploring a phenomenon based on data from a group of people who form a case of interest within a bounded system in its natural environment (Yin 2017).

The data analysis was conducted in two phases. In the first phase, the unit of analysis was the structure of the argument in a sentence or paragraph (Walton 2006). The data analysis presented some challenges due to the reports having been written by non-native speakers of English. Nonetheless, the researcher applied the pragma-dialectical approach rules for the reconstruction of argumentation (van Eemeren et al. 1993; van Eemeren and Snoeck Henkemans 2017) to understand how to make unexpressed standpoints (claims) or premises (grounds) explicit, use indicators of argumentation (e.g. thus, therefore, so, because, since), and use context to make valid interpretations of argumentative structures during reconstruction. An example of the reconstruction or building of an argument is shown in an example of metacognitive reflection. A theory-driven, iterative argumentation process was employed to analyse and review the data over multiple rounds using a coding framework we developed based on the cluster analysis method (Miles, Huberman, and Saldaña 2020). The framework for coding the argument structures was based on the modified TAP (Toulmin, Rieke, and Janik 1984; Toulmin 2003) (see Table 1). TAP was developed to analyse a structure because it contains more elements than the pragma-dialectical approach (van Eemeren and Snoeck Henkemans 2017) or grounds (premises) and a conclusion in Waltońs argumentation schemes (Walton, Reed, and Macagno 2008). An argument structure needs to be coherently ordered to validate the whole argument from claim to conclusion. The elements can be of different orders.

Table 1. Coding framework for argumentation structures.

Code	Description
Claim	Taking a stance in a text
Counterclaim	The standpoint against the claim
Ground	Evidence and data
Warrant	Provision of additional affirmation and support for a claim and grounds; can contain an example, an analogy, a reference, etc.
Conclusion	Conclusion of the chain of argumentation
Solution	A way to solve a problem or issue

I have clarified the claim, grounds, conclusion, rebuttal as a counterclaim and integrated warrant and its backing in TAP (Toulmin, Rieke, and Janik 1984; Toulmin 2003). I have also added the solution to the model. The structures were classified as ‘clusters’ based on the modified TAP that Osborne, Erduran, and Simon (2004) used to identify argument structures in a sentence or a paragraph. For example, the ‘claim-ground’ structure can be found in the sentence of a paragraph that formed one of the clusters (see Table 4). The claim-ground form cluster is less sophisticated than the ‘claim-grounds-warrant’ form cluster or the ‘claim-grounds-warrants-conclusion-solution-counterclaim’ forms cluster. Claims without justification were classified as ‘other structures’. Qualifiers in the TAP were not identified in the data.

When I identified the grounds and claims, I looked at how the grounds supported claims or counterclaims. The pragma-dialectical model was employed to break down the argument structure to analyse the strength of the grounds (premises) and claims (standpoints) as follows: (a) coordinated argumentation, in which the grounds jointly support the claim; (b) multiple argumentation, in which the individual grounds support the claim; or (c) grounds that depend on one another in subordinate argumentation (van Eemeren and Snoeck Henkemans 2017).

Types of argumentation schemes (Walton, Reed, and Macagno 2008) were used within the Toulmin model to assess warrants together with backings consistent with Duschl (2008) findings. The types of warrants are listed in Table 2. ‘Argument from expert opinion’ (Walton, Reed, and Macagno 2008) was changed to ‘warrant by references’ in this study to describe arguments supported by references, figures, and tables.

Table 2. Types of warrants.

Warrant by	Definition	Example
Consequences	A policy or an action has positive or negative consequences	When this interactivity is made easier and cheaper, the chance of reaching a noticeably wider audience is great [Group 6/IPR].
Example	A description of a real case study	Amateur commentators could be, e.g. the fans of an ice hockey team which is playing on TV [Group 4/IPR].
References	A reference can be an article, website, book, figure, or table.	As the preliminary market assessment shows [ground/warrant by reference] that there is a potential market for this kind of TV program [Group 8/UR and IPR].
Analogy	A case is similar to another case in certain respects	… Still this is something that, technology-wise, would be easy to implement with today’s knowledge, [2.1e an unclear ground], but as the DVB-T boxes at the moment are not supporting this kind of technology [3. clear counter claim], it must more or less be considered a train that has left the station [warrant by analogy] [Group 1/IPR].
Cause to effect	Shows causality in a situation; if A occurs, then B will occur	… if there are only 2500 votes given per a show or if the profit per an SMS is as low as 15 cents per a message, the Net Present Value over three year period would be so low that it might not be worth implementing the service [Group 4/UR].
Alternatives	The case in which X is chosen as a case, instead of Y	This technology has some major advantages compared with more traditional server-client models [2. clear claim/warrant by alternatives], but it has some downsides as well [3. unclear counter claim]. The major advantage is to share resources [2.1 an unclear ground] and … [Group 3/IPR].
Position to know	The reference source is assumed to know about the things in their domain	Based on our own experiences … [Group 4/UR].
Appeal to general opinion	Everybody in the group accepts this generally acknowledged thought	Our intention is to develop a device that could be plugged in[into] an IP network using Ethernet [1.1d a clear reason]. ‘Everything on IP’ is the trend that we decided to follow [1.1e a clear reason/appeal to general opinion] [Group 6/UR and IPR].
Verbal classification	Definition of concept	Brainstorming is a technique by which a group tries to find a solution for a problem [by its members’ spontaneously creating ideas] All ideas produced are equally valuable and no idea is bad. The idea is to come up with as many ideas as possible, and the ideas of other members are further developed [Group 5/UR].

Table 3 lists the criteria for evaluating the content of arguments. The content was analysed systematically, and data analysis was validated by calculating the frequency of occurrence of each criterion (Neuendorf 2017). From the perspective of engineering education, relevance, clarity, and logic were chosen as the relevant criteria for engineering reasoning (Paul, Niewoehner, and Elder 2013) (see Table 3). According to Johnson and Blair (2006), a good argument must satisfy the criteria of relevance, sufficiency, and acceptability. The grounds (premises) need to provide sufficient support for the conclusion, and the grounds must be acceptable. Relevance relates to textual coherence, a reasonable and clear standpoint on a topic to fulfil the purpose of a text in the context of a domain.

Table 3. Criteria and definitions.

Criteria	Definition
Acceptability	Arguments must be acceptable. The statements’ acceptability can be verified by references or commonplace values.
Clarity	The arguments are understandable and clearly presented.
Logic	The connection between the grounds and a claim and/or conclusion is logically valid.
Sufficiency	The grounds need to provide sufficient support for the conclusion.
Relevance	An argument relates to an issue.
Rhetoric	Rhetoric refers to the use of adjectives, rhetorical figures, and emotionally loaded words (e.g. heavy, expensive, light).

Logical inconsistency in the pragma-dialectical approach (van Eemeren and Snoeck Henkemans 2017) is related to the standards of logic for reasoning in engineering education (Paul, Niewoehner, and Elder 2013). Logical inconsistencies are contradictions between a particular standpoint (a claim) and its premises (grounds). A standpoint cannot be true if the grounds are false. Pragmatic inconsistencies are contradictions in the state of affairs in the real world. Sufficiency relates to the number and structure of grounds used to support a claim or conclusion with various relations between a claim and its grounds (the coordinated, multiple, and subordinate argumentation mentioned previously). Rhetoric refers to ways of influencing the knowledge of an audience through language (van Eemeren 2010).

Unacceptable arguments were identified as fallacies and weak arguments (Walton 1995), but they were not reported in this study.

Metacognitive reflection was identified in the groups’ awareness of their knowledge about their cognitive performance, their knowledge about the task or goal, and how they monitored their progress (Barzilai and Zohar 2014; Flavell 1979; Iiskala et al. 2011). For example, group 4 was aware of their competence as knowledge of their cognitive performance, as the following reconstructed argument demonstrates:

The biggest drawback of working in pairs [reconstructed claim 1: there were multiple challenges to working in pairs] [addition: because] the group’s competence was divided into smaller segments [1.1a accurate ground], and it was more challenging to achieve the same quality in pairs as with the whole group working together. [1.1b accurate ground] However, this way of working was more efficient. [2. accurate counterclaim] (University Report)

The above passage exemplifies the pragma-dialectical approach (van Eemeren et al. 1993) to reconstructing arguments, rebuilding an argumentative structure to identify the structural elements it comprises. Superfluous words can be removed, and words can be added to rebuild a claim or ground, or rearrange structural elements. In the above example, the claim is reconstructed to increase clarity.

The same group also monitored their progress:

The results from every phase were always ratified by the entire group [1.1 ground] so that any big mistakes or errors in the service idea or calculations could not easily occur. [1. claim] … 

Group 7 reflected on the task in the University Report:

We felt that this study was more like an ideation and innovation exercise rather than a real product development study. [1. claim/warrant by position to know] This is because we were in such a hurry to just generate some ideas.

Intercoder reliability

Cohen’s kappa (Neuendorf 2017) was used multiple times to assess intercoder reliability while developing frameworks for coding the argument structure and content. The data were assessed by the author and an independent undergraduate student who graduated during the study, but the student did not do the Masteŕs thesis for this project. The student received instructions for argumentation analysis and analysed some examples before performing the final analysis. In the final reliability analysis, the raters individually assessed the structure and content of the argument in the one group’s both types of reports. Coheńs kappa was moderate for categorical coding for claim = 0.78, ground = .83, warrant  = .79, conclusion  = .56, solution  = .79, counterclaim =  .74, acceptable content = 70, multiple ground  = .54, and coordinative ground = .62. The final reliability analysis was based on multiple-level categories, including the elements in the structure and evaluation of the content. The multiple-level category evaluation showed the nature of argumentation in texts even though it is challenging process to implement. Claims, grounds, warrants, solution, and content, coordinated ground were substantiated, but conclusion and multiple grounds were moderate. The moderate and substantiate values lead to tentative and limited conclusions which need to be interpreted cautiously. Differences in raterś knowledge and skills have impacted analysis. More retraining and adding linguistic knowledge of argumentation will promote raterś agreement.

Results

Variation in the types of acceptable structures

The groupś argumentation was acceptable, relevant, and logical. The quality of argumentation was identified based on the levels defined in the structure shown in table 4.

Table 4. Frequency of Clear and acceptable arguments by type and levels

Structure	Group								Total
	1	2	3	4	5	6	7	8
Level 0
Other structure	3	5	2	4	8	0	5	2	29
Level 1
Claim-ground	1	1	3	5	2	0	0	6	18
Claim-grounds	5	2	7	5	6	1	6	4	36
Claims-ground	0	0	0	1	1	0	1	1	4
Counterclaim-grounds	1	0	0	0	0	0	0	0	1
Level 2
Claim-grounds-warrant(s)	5	4	6	9	9	2	8	8	51
Claim-counterclaim-grounds-warrant(s)	2	0	0	4	0	0	1	2	9
Counterclaim-grounds-warrant(s)	0	0	1	2	0	0	0	0	3
Claim-counterclaim-grounds	1	0	0	0	1	0	2	1	5
Level 3
Claim-grounds-conclusion	0	0	2	1	3	1	0	2	9
Claim-counterclaim-grounds-conclusion	0	0	0	2	0	2	0	1	5
Claims-counterclaims-grounds-solution	0	0	0	0	0	0	0	1	1
Claim-grounds-warrant(s)-conclusion	1	2	1	4	0	1	0	0	9
Claim-grounds-warrant(s)-solution	0	1	0	2	1	2	1	4	11
Claim-counterclaim-grounds-warrant(s)-conclusion	0	0	0	1	1	0	0	2	4
Grounds-conclusion	0	1	0	0	0	0	0	0	1
Grounds-warrant(s)-conclusion	0	1	0	1	0	1	0	0	3
Counterclaim-grounds-warrant(s)-conclusion	0	0	0	1	0	0	0	0	1
Total	19	17	22	42	32	10	24	34	200

Table 4 shows the frequencies of clear and acceptable arguments in each group’s reports. The different combinations of the argument claim-grounds-warrants-conclusion-solution-counterclaim represent patterns or clusters in levels 1–3. Levels 1–3 show argument patterns or ‘clusters’ in which the same type of arguments is grouped. Level 0 represents another structure. Grounds are missing, or a sentence can contain a claim or claims or warrants. Level 1 is a combination of claim-ground(s) that represent a less sophisticated form of an argument than a claim-ground(s)-warrant(s) or claim-counterclaim-grounds in Level 2. A structure of claim-counterclaim-grounds is more complex than a structure of claim-grounds. Combinations that comprised solution and conclusion were added to the structure as an instance of cluster 3 (level 3). There were exceptions of short structure such as grounds-conclusion and grounds-warrant(s)-conclusion. These structures contained a conclusion or a solution that requires advanced argumentation. The number of grounds and warrants may vary in levels 2 and 3. For example, the complex chain can contain only one ground or different types of warrants. The groups’ arguments varied mostly between Levels 1–2.

The groups mostly co-constructed claims, grounds, and evidence as warrant(s), even though there were variations in the argumentation structures between the groups. The advanced level 3 structures that contained conclusions and solutions alongside counterclaim(s) were found only in Groups 4 and 8.

The claims and grounds were supported by nine types of warrants (n = 162), presented from most to least frequently occurring as follows: reference, example, cause to effect, consequence, position to know, analogy, verbal classification, appeal to general opinion, and alternatives. Only one group did not have the warrants of reference in their argumentation. The number of warrants by analogy was low. Analogy is important in ideation in the early stages of real-world product development and problem-solving in engineering (Cropley 2015).

Sufficiency is related to the relationship between a claim and its grounds. The groups addressed different independent perspectives more by using multiple argumentation (n = 59) than coordination argumentation (n = 52) or subordinative argumentation (n = 5) separately. They also co-constructed multiple argumentations more than they co-constructed coordinative and multiple argumentation in the same argumentation chain (n = 10). The groups used rhetoric in a moderate manner to convince the industrial partner.

Figure 1 is a visualisation of Group 4’s advanced argumentation (claims-grounds-warrants-conclusion-solution-counterclaim) to reject the proposed idea in the University Report (see Figure 1).

Figure 1. Pros and cons of the discarded idea.

Analysis

The group suggested a digital video broadcasting program with an interactive plot (DVB interactive plot), where the viewer is given a chance to impact a fictional TV show plot, for instance, in a children’s program.

Their first counterclaim was supported by coordinated arguments (coordinative argumentation structure). They highlight the difficulty of technical implementation, the lack of existence of new receivers that viewers would need to participate, the high cost of content production, and the resulting low profitability. The group raised the point that an interactive service was offered previously in the 1990s, which was a warrant by example. The failure of the interactive TV program was used to support an objection regarding the profitability of the interaction programs. They also constructed a counterclaim to their claim that interactive service is difficult to implement and expensive. They justified their position on multiple grounds with a multiple argumentation structure (2.1a, 2.1b, 2.1c, 2.1d). At the end of the chain, the group concluded that it was still separately supported on multiple grounds. The argument was logical and acceptable. Rhetoric (‘really difficult and expensive to implement’) was relied on to persuade the arguments. More references were required to make counterargumentation convincing.

An example of the basic acceptable structure (claim-grounds-warrant) was constructed by Group 1, which focused on providing evidence (warrants) for their claims with multiple grounds (Figure 2) in their University Report.

Figure 2. Group 1 evaluated the reliability of the design and ideation based on the criteria.

Group 1 wrote that they chose the ISO 9000 family of standards to validate their project, which was clearly a constructed claim. This claim was sufficiently justified by the multiple argumentation structure. Their grounds 1.1. and 1.3. were inaccurate because the group presented them at a general level. The rhetoric used in the first ground (the system is efficient) was superficial. The warrant by reference was ISO 9000, a set of standards that helps organisations create quality management systems that meet the needs of their customers and other stakeholders.

Metacognitive reflection

We identified metacognitive reflection as related to groupś metacognitive knowledge of their groupś performance, task knowledge, and self-monitoring in their estimation of the reliability of innovation in the University Reports. Groups 2, 4, 7, and 8 justified their performances and wrote task-related comments (n = 7), and Group 4 wrote two comments on monitoring.

Discussion

It is critical to impart argumentation skills to students during engineering education. This study used argumentation models (Toulmin 2003; van Eemeren and Snoeck Henkemans 2017; Walton, Reed, and Macagno 2008) to identify and assess the types of argument structures and acceptability of argument content in the final co-designs. The epistemological criteria acceptability, clarity, logic, and relevance were used to assess the argument content, the reasoning (from a claim to a conclusion), and the quality of the evidence provided. Rhetoric and socially shared metacognition were additional dimensions of the assessment.

The results showed that the group’s argumentation was convincing. Argumentation was logical, acceptable, and evidence-based. The most significant challenge was evidence; some groups used rhetoric instead of evidence to convince the industrial partner.

The frequency of conclusions, solutions, and counterclaims in the argument chains was low. This result is similar to that of Heng, Surif, and Seng (2014), who also found the basic argument structure (claims-grounds-warrant) in participating groupś argumentation as well as to the Almudi and Ceberio (2015) and Erduran and Villamanan (2009) studies on undergraduate engineering studentś problems in formulating valid conclusions. Counterarguments were also rare in the reports (Leite et al. 2011).

The types of warrants provide concrete tools for faculty and students. Warrants give more evidence to support together with the grounds of the claim or the warrant(s); the grounds and the claim support the conclusion. The groups used mostly warrants by reference, example, or causal reasoning. The warrant by reference showed that groups constructed evidence and knowledge-based argumentation. The groups did not see themselves as an expert, indicated by a low number of warrants by position to know. It also referred to lower use of metacognitive reflection, which needs to be addressed in future studies and pedagogy. Argumentation requires engagement with ill-structured problems (Jonassen and Cho 2011; Rittel 1987). This study confirms this claim. Ideation in the course of product development provided opportunities for students to engage in argumentation by justifying their solutions for industrial customers in a real-world context. The groups were able to construct acceptable arguments that could be partly understood based on the nature of the discipline, which were then used to present technical data and to calculate pros and cons during their studies (i.e. Eide et al. 2018). These results are in line with a previous study (Jonassen and Cho 2011) that identified argumentation as an important factor in addressing ill-structured problems.

The results of this study have several pedagogical implications. Our findings confirm previous research suggesting that argumentation should be integrated into discipline-specific course curricula in engineering education (Almudi and Ceberio 2015; Erduran and Villamanan 2009; Foutz 2019; Jonassen and Cho 2011; Leite et al. 2011; Yalvac et al. 2007). Furthermore, engineering design education is a particularly appropriate field for the integration of argumentation because engineering design students need to apply knowledge to problems or other situations of varying complexities and identify solutions, constraints, and criteria (Eide et al. 2018; Rittel 1987). To that end, engineering school faculty should learn dialogical pedagogies to support engineering students’ engagement in argumentation (Andrews 2010; Garcia 2017; Henderson et al. 2018; Jonassen and Cho 2011; Lappalainen 2013; Rapanta, Garcia-Mila, and Gilabert 2013). The students could also benefit from metacognitive instruction to help develop their metacognition and advanced argumentation skills (Barzilai and Zohar 2014; Rapanta, Garcia-Mila, and Gilabert 2013). Reflection, critical questions, and self-directed learning engage in personal transformation (Claris and Riley 2012).

This study has several limitations that should be addressed in future studies. First, the small data was collected from the one academic course because the modified TAP (Toulmin, Rieke, and Janik 1984; Toulmin 2003) was developed and the other argumentation models (van Eemeren and Snoeck Henkemans 2017; Walton 1995, 2006; Walton, Reed, and Macagno 2008) were utilised in data analysis to identify structures and assess the validity of the content. Second, the student population in this study was mostly Finnish, with mainly English as their second language. Diverse student cohorts in different contexts can be examined in future studies. Third, we did not track individual group members’ performances but only asked the team leaders their thoughts about the group work after the report writing was completed. The group leaders from Groups 2, 5, and 6 reported issues with scheduling, the passivity of ethnic minority students in the groups, communication problems in English (a second language for most participants), missing team leadership, and idea domination issues. These challenges impacted the groups’ work; therefore, those elements need to be controlled when group work is the only choice in large classrooms (i.e. Davies 2009).

Future studies should focus on instruction and assessment of individual studentś and groupś argumentation, during which students engage with each other’s ideas and assess their own and others’ thinking (Garcia 2017; Henderson et al. 2018; Jonassen and Cho 2011). Examples are helpful when learning to present a counterargument (Jonassen and Cho 2011). Explicit writing-based support and prompts in graphic organisers can promote the learning of engineering-related argumentation (Wilson-Lopez et al. 2020), while argument-diagramming tools can help assess argumentation and fallacies (Rapanta and Walton 2016).

This qualitative discourse study revealed the groupś reasoning in written argumentation. Qualitative research should be promoted in engineering education to develop researchers’ skills to gain deeper insight and understanding of argumentation practices in engineering education (Kellam and Cirell 2018).

This study identified a low level of metacognitive reflection, which also needs to be addressed in future studies. Awareness of individual differences in epistemic metacognition and metacognitive regulation among group members (Khosa and Volet 2014; Rapanta, Garcia-Mila, and Gilabert 2013) are needed to help students develop advanced argumentation (level 3) skills during academic studies.

The teaching of fallacies would also promote students’ evaluation of their own and others’ argumentation. Study of fallacies might reveal obstacles in their reasoning and argumentation that hinder learning and also teach them how to identify spurious arguments when they encounter them (Gallant et al. 2021; Lodge et al. 2015).

Conclusion

As this study has shown, for success in co-design, engineering students need to develop advanced skills in group argumentation. This is an important finding for educators in engineering design. It suggests that teaching effective argumentation skills for convincing is beneficial to learners. Considering its importance in engineering practice, argumentation has not received sufficient attention in engineering education. This study has also shown that metacognitive reflection has an important role in argumentation and design, which needs to be addressed in teaching argumentation skills.

The most important contribution of this qualitative study is its use of an integrated argumentation framework (Toulmin, Rieke, and Janik 1984; Toulmin 2003; van Eemeren and Snoeck Henkemans 2017; Walton 1995; Walton, Reed, and Macagno 2008) as a tool for identifying acceptable arguments.

This will open new research and educational opportunities in engineering design and education. Engineering-related argumentation is a promising instructional approach that should be further clarified in a domain-specific manner for different engineering fields through observation and analysis of engineering students’ argumentation as they learn epistemic practices and analyse their design problems (Wilson-Lopez et al. 2020).

Declaration of interest statement

The author has no conflicting interests to declare.

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

Marita Seppanen

Marita Seppanen is a doctoral student in the Faculty of Educational Sciences at University of Helsinki, Finland. She studies the reasoning and argumentation of undergraduate students and of employees in firms.