Students’ use of formalisations for improved logical reasoning

ABSTRACT

Logical reasoning as part of critical thinking is becoming more and more important to prepare students for their future life in society, work, and study. This article presents the results of a quasi-experimental study with a pre-test-post-test control group design focusing on the effective use of formalisations to support logical reasoning. The participants were 163 pre-university secondary school students (11th and 12th graders). For the experimental group we designed a course in logical reasoning intending to develop students’ use of formalisations in formal reasoning and everyday reasoning tasks. We show that students learned to use the intended formalisations. Overall, students from the experimental group had significantly better results on the post-test compared to the control group. Positive correlations were found between item scores and the use of Venn and Euler diagrams and scheme-based strategies. Our results support our hypothesis that the use of formalisations improves logical reasoning skills.

KEYWORDS:

• twenty-first century skills
• logical reasoning
• formalisations

Introduction

To fully participate in today's society, students need to learn higher-order thinking skills, such as critical thinking as, for example, emphasised by P21′s Framework for twenty-first century Learning and the OECD (Brookhart, 2010; P21, 2015; Vincent-Lancrin et al., 2019). Critical thinking includes reasoning in formal and everyday situations, for which it is important to build, analyse, and verify reasoning, and identify and prevent incorrect arguments. For example, citizens who evaluate the validity of claims in commercials when deciding about the buying of a car or who critically review journalists’ stance and arguments in newspaper articles. Or, in professional life, a judge who properly weighs all evidence to evaluate the most likely scenario of a crime. In all these situations formal and visual representations might provide an overview of the relevant reasoning, and, thus, structure the reasoning process (Halpern, 2014; McKendree et al., 2002; Van Gelder, 2005). This shows the need for students to learn to structure their reasoning in all kinds of contexts. However, not much is known about the effective use of formal and visual representations in the reasoning of students at secondary level. The teaching of reasoning is part of mathematics education, and, as we will show, not only refers to reasoning in strictly formal and mathematical systems, it should also support reasoning in everyday situations (e.g. NGA Center and CCSSO, 2016).

Logical reasoning

The common factor that connects reasoning in formal and everyday situations is logical reasoning (Liu et al., 2015) which is the focus of this article. Formal reasoning always gives a valid conclusion within a formal system as long as one applies the pre-defined rules (Schoenfeld, 1991). These rules are often taken from mathematics or formal logic based on fixed and unchanging premises (Teig & Scherer, 2016).

Reasoning in everyday situations is often called informal reasoning in the literature, but this term is used ambiguously (e.g. Voss et al., 1991). Informal reasoning does not provide one conclusive solution, because the problem situations are often open-ended. By identifying and justifying acceptable premises, the results will be plausible within the context, but can be open for discussion, for instance, by providing new information (e.g. Galotti, 1989; Johnson & Blair, 2006; Kuhn, 1991). In an earlier paper, we concluded that informal reasoning can be defined as “reasoning in ordinary language to construct an argument which requires a critical review of the given premises and transforming of information, as well as finding additional or similar information provided by the problem solver or by external sources” (Bronkhorst et al., 2020, p. 1676). For this study, we will use this definition accordingly.

Both, formal and informal reasoning, involve selection and interpretation of information as key elements in reaching a conclusion via a series of reasoning steps. Therefore, in an earlier paper, we concluded that formal and informal reasoning can be combined into one definition of logical reasoning that involves several steps. We defined logical reasoning as “selecting and interpreting information from a given context, making connections, and verifying and drawing conclusions based on provided and interpreted information and the associated rules and processes.” (Bronkhorst et al., 2020, p. 1676). In this study, we will build on this definition, which is a workable definition and bounds the context of our research. It also clearly shows the steps needed in working towards a conclusion in logical reasoning.

Formalisations

We have already mentioned that formal and visual representations, which we call formalisations, can support logical reasoning. Formalisations include, among others, all sorts of symbols and formal notations, schematisations, and Venn and Euler diagrams and will be used in this article accordingly (e.g. Bronkhorst et al., 2020; Halpern, 2014; Van Gelder, 2005).

From earlier studies, we discovered that most students (11th and 12th graders) did not use formalisations, but preferred informal reasoning and made mistakes in all sorts of reasoning tasks (Bronkhorst et al., 2018, 2020). Our results were in line with reported difficulties concerning logical reasoning among different age groups elsewhere in the literature (e.g. Daniel & Klaczynski, 2006; Evans, 2002; Galotti, 1989; Stanovich et al., 2016). In this article, we present results of an experimental study with a pre-test-post-test control group design, using a specially designed intervention aimed at improving students’ logical reasoning skills focusing on the use of formalisations. To study the role of formalisations within different situations, it is important to differentiate between types of tasks. For that, we will use Galotti’s (1989) division between formal reasoning tasks and everyday reasoning tasks. Formal reasoning tasks are closed tasks in which all premises are provided, which intend to lead to one conclusive answer. Those tasks can be stated formally (with symbols) or non-formally in ordinary language (without symbols). Everyday reasoning tasks are open-ended problems, often with implicit or non-mentioned premises. For example, reviewing arguments in short newspaper articles. This division is visualised in Figure 1.

Figure 1. Division of tasks.

Relevance

The experimental group of our study consisted of Dutch students for whom the domain “logical reasoning” is explicitly mentioned in the most recent mathematics curriculum for pre-university education (College voor Toetsen en Examens, 2016). This domain specifically aims at improving students’ reasoning about societal issues outside the field of mathematics. In other words, it should stimulate general educational objectives as a preparation for all sorts of situations in society (cTWO, 2012). If we study the mathematics curricula from English speaking countries, we recognise similar intentions, which correspond to our definition of logical reasoning. In other words, reasoning in mathematics should not be synonymous with formal deductive reasoning only (NCTM, 2009, Chapter 1). The United States’ Standards for Mathematical Practice state for example: “Mathematically proficient students are also able to compare the effectiveness of two plausible arguments, distinguish correct logic or reasoning from that which is flawed, and—if there is a flaw in an argument—explain what it is” (NGA Center and CCSSO, 2016, p. 7). This clearly shows the possibility of placing plausible arguments next to strict formal conclusive reasoning. The National Curriculum for England for Key Stage 4 states that students should be able to “assess the validity of an argument and the accuracy of a given way of presenting information” as part of their mathematical reasoning skills (Department of Education UK, 2014). Furthermore, the United States Standards and the new Australian curriculum explicitly mention adaptive reasoning as one of the five strands for mathematical proficiency (McChesney, 2017). Adaptive reasoning represents the “capacity for logical thought, reflection, explanation, and justification” (National Research Council, 2001, p. 5), which, together with productive disposition, should show that mathematics is useful and supporting. Altogether, these examples show the importance of logical reasoning in different contexts as part of mathematics education.

Intervention

We implemented an intervention, which specifically stimulates the use of formalisations, including visualisations and schematisations, in logical reasoning. The hypothesis behind our intervention is that the use of formalisations is useful and teachable, and therefore, should improve students’ reasoning (Adey & Shayer, 1993; Van Aalten & De Waard, 2001) as also suggested by our prior research (Bronkhorst et al., 2018, 2020). Our intervention should not be confused with a course in formal logic, because formal logic demands an enormous time investment to reach a satisfactory level (Hansen & Cohen, 2011). Additionally, it might not even be effective at all for improving logical reasoning skills (e.g. Cheng et al., 1986). For our target group of non-science students, we expected that a course requiring a high level of technical, mathematical skills would not be adequate (Carrascal, 2011). Furthermore, research from Attridge et al. (2016) suggest that, for conditional inferences in particular, prior experiences with logic are required for significant improvement. Consequently, in our intervention, we focus on relevant formalisations, including visualisations and schematisations, that structure and support the reasoning process (Halpern, 2014; McKendree et al., 2002; Van Gelder, 2005).

Previous research has shown that classroom discourse and social interactions among students are supportive for improved reasoning (e.g. Rojas-Drummond & Mercer, 2003; Yackel et al., 1991). There should also be sufficient attention to students’ own solution methods with meaningful tasks related to everyday experiences (Grouws & Cebulla, 2000; National Research Council, 1999). These factors are comparable with teaching standards of Realistic Mathematics Education where students “come up with their own solutions, explain and justify their solutions, [and] try to understand the explanations and solutions of their peers” (Gravemeijer, 2020, p. 220). It is important that teachers facilitate these opportunities for constructing and understanding by providing adequate tasks and posing suitable questions (Gravemeijer, 1999, 2020; Halpern, 1998; NCTM, 2009). Together with the focus on formalisations, we incorporated this in our intervention, which resulted in the following design characteristics:

1. Students are given opportunities to reason in formal and everyday reasoning tasks.

2. Students are encouraged to invent their own formalisations, including schematisations and visualisations.

3. Students will encounter different types of arguments (strict correct, valid, plausible, open-ended, et cetera).

4. Students are encouraged to explain and compare their solutions, reflect on their reasoning, and to provide each other with feedback.

5. Different solution methods are discussed during classroom discourse.

6. The teacher will provide formative feedback regularly.

An overview of the lessons for our intervention, consisting of 10 lessons of 50 min, is found in Table 1. After some introductory explorations (first lesson), students worked a whole lesson on judging evidence in a legal case presented in a newspaper article (everyday reasoning task). They had to find a way to present the premises and proofs clearly before a possible reasoning scheme was introduced. In the following lessons, students were encouraged to invent their own structures, visualisations, and schematisations with or without the use of symbols in small tasks, often concerning syllogisms and if–then statements (mainly formal reasoning tasks). After students had the opportunity to invent their own formalisations, letter symbols, Venn and Euler diagrams, and logical symbols were introduced during lessons 3–5. In lessons 7–8, students constructed their arguments from newspaper articles, giving them the opportunity to use their discovered and learned formalisations in open-ended everyday reasoning tasks.

Table 1. Overview intervention.

Lesson	Description and tasks	Learning goals
1	Introductory lesson with exploration of field of (logical) reasoning with formal and everyday reasoning tasks, such as puzzles, mathematical examples, and reasoning in a short newspaper article.	Students are introduced to the concepts of (logical) reasoning. Students are introduced to different types of reasoning. Through interactive tasks via Socrative*, students are made curious to the domain logical reasoning and experience that reasoning errors can occur quickly.
2	Students explore reasoning in a large context, a newspaper article about a court case, as relevant in the societal debate (everyday reasoning). The assignment is open-ended, but terms as premises, reasoning steps, conclusion, correct, complete, and incomplete are introduced.	Students discover how a logical argument is structured. Students are introduced to (logical) reasoning in the societal debate. Students can identify different parts in a text: premises/assumptions, reasoning step(s) and conclusion(s). Students explore/invent first schematisations of a larger newspaper article. Students know the concepts correct, complete, and incomplete.
3	Students explore syllogisms in ordinary language and study the difference between valid and true conclusions. They also try to formalise/structure the provided syllogisms in their own way and compare those ideas with peers before an example with letter symbols is shown.	Students learn to work with syllogisms in ordinary language (all, some, a few, not) and know the term “premise”. Students can distinguish truth and validity. Students explore conclusions for if-then claims.
4	Students are encouraged to come up with their own visualisations first before Venn and Euler diagrams are introduced and applied on syllogisms and propositions.	Students are introduced to Venn and Euler diagrams. Students can draw appropriate diagrams for propositions and syllogisms. Students can use Venn and Euler diagrams to falsify claims.
5	Logical symbols are introduced and combined with drawing and verifying conclusions for if-then-statements.	Students learn to work with the logical symbols: ∧, ∨, ⇒, and ⌋. Students can combine the theory of previous sections with the new symbols and diagrams. Students can draw and verify conclusions for if-then-statements. Students can visualise if-then-statements in Venn and Euler diagrams.
6	Further practice and discourse on if-then-statements.	Further practice of working with logical symbols, if-then-statements, and visualising in Venn or Euler diagrams. Students know the difference between sufficient condition and/or necessary condition.
7	First half of this lesson is dedicated to practice and discussion of an assignment in which everything from the previous sections is combined. The second half of the lesson, students explore newspaper articles in which the reasoning contains uncertainty.	Students combine knowledge from all the previous sections. Students can recognise reasoning with uncertainty.
8	In lesson two, students explored reasoning in a large newspaper article. After all sorts of short tasks in lessons 3 until 8, students analyse reasoning in large contexts again on which they can apply new techniques.	Students can analyse reasoning from large, societally relevant, contexts. Students practise with if-then, letter symbols, (logical) symbols, and visualisations.
9	Final interactive quiz and class discussion afterwards.	Students test their knowledge in a short interactive Socrative* quiz.
10	Final lesson with practice exercises, mainly longer formal reasoning tasks, but often stated in ordinary language.	Students practise their logical reasoning skills in a variety of exercises.

*Classroom app for Quizzes and formative tests: https://socrative.com/

In general, students were almost always asked to come up with their own solutions for each exercise first. Second, they discussed their solution methods in pairs or small groups of three or four. Third, during classroom discourse different solution methods were compared not only on correctness but also on usefulness in general with specific attention to the comparison of students’ used formalisations. Homework was limited to one or two exercises.

The design of the materials followed two iterative cycles (Van den Akker et al., 2013). Materials were developed in collaboration with a group of teachers and were tested in a pilot study, including pre- and post-tests. After the pilot intervention and teacher evaluations, various adjustments were made, mainly in providing sufficient time for students to develop their own solutions and to discuss their solutions in groups. The collaboration was not only important for the development of the materials, but even more important for discourse with the participating teachers about the exercises and how to use those in class.

The implementation guidelines for the lesson sequence on logical reasoning were provided in a teacher manual, but also thoroughly discussed during the preparation sessions with participating teachers.

Research questions

To investigate the effective use of formalisations, which also include schematisations and visualisations, in the reasoning of non-science students at secondary level, this study was guided by the following research questions:

1. To what extent do students use formalisations in formal reasoning tasks (formally and non-formally stated) and in everyday reasoning tasks in the pre- and in the post-test?

2. What is the effect of a course in logical reasoning (with a focus on the use of formalisations) on students’ performance in formal reasoning tasks (formally and non-formally stated) and in everyday reasoning tasks compared to a group without that course?

3. Is the use of formalisations positively associated with the correctness of students’ answers?

Method

We used a quasi-experimental pre-test-post-test control group design (e.g. Cook & Campbell, 1979). Students in both groups followed a pre-university non-science track (11th and 12th graders; 16- to 18-year old) in Dutch schools. Both, the experimental and control group took classes in mathematics, albeit with some differences in maths topics. The experimental group took lessons in logical reasoning with our specially designed course, while the control group worked on more advanced applications of functions and their derivatives from their regular mathematics book. Our course did cover all the learning outcomes as required by the national curriculum (College voor Toetsen en Examens, 2016), but, as described above, focused on effective use of formalisations that students could apply in formal and everyday reasoning tasks. Normally, students are taught in a more teacher-centred way with a textbook chapter on logical reasoning, which introduces types of problems and solutions that students apply in, often short, exercises individually. Venn and Euler diagrams are usually introduced at the end of the chapter on logical reasoning, while in our intervention Venn and Euler diagrams were already introduced in an early stage, after which the students had sufficient time to practise with the diagrams and to apply those on larger open-ended everyday reasoning tasks.

Due to the fact that the control group did not receive lessons in logical reasoning, we can verify for changes over time and retest effects on the use of formalisations.

Participants

The recruited teachers from nine schools from all parts of the Netherlands participated voluntarily. They committed to participate in the collaboration with the other teachers and came together six times prior to the intervention. Six teachers administered pre- and post-tests in experimental and control groups. Two teachers were not able to organise test sessions for control groups and only participated with an experimental group. For one teacher the experimental group for the final intervention was too small, so he only arranged pre- and post-tests for control students. Participating teachers filled in a logbook containing surveys with Likert-type items (scale 1–5) on student and teacher activities in each lesson.

The average class size for the eight participating experimental classes was seven students and varied from three to 12. The small class sizes are explained by the fact the course including logical reasoning is an elective course. Most students take an alternative course with more advanced applications of functions. Participating students matching this condition varied from six to 23 students per school. Because of this circumstance we had to settle for an unequal number of participants in both conditions: the experimental group condition consisted of 56 students in total and the control group of 107 students.

Tasks

We used identical pre- and post-tests for this study. This choice was justified because we could verify for retest effects. The tasks were mainly selected from other tests, and adapted and selected after an exploratory study (Bronkhorst et al., 2020) and a pilot study.

The test consisted of eight tasks with two tasks with two parts, so in total there were 10 items. For each category mentioned in Figure 1, we used at least three items:

1. Formal reasoning tasks, non-formally stated: items 1, 2A, 2B, and 8.

2. Formal reasoning tasks, formally stated: items 3, 4A, and 4B.

3. Everyday reasoning tasks: items 5, 6, and 7.

This variety of tasks corresponds with the characteristics and goals of the intervention in relation to our definition of logical reasoning. In all tasks, students had to explain their answers to make their reasoning visible. The items, originally written in Dutch, can be found in an English translation in the Appendix of this article and are explained briefly below.

In the syllogism tasks (items 1 and 3, adapted from Stanovich et al., 2016) students had to verify the provided conclusion based on given premises. We considered these as counter-items, because they were non-formally stated and formally stated, respectively, but are about similar problem situations with similar conclusions. We conjecture that Venn and Euler diagrams are highly beneficial visualisations in these tasks, because those diagrams may represent the different sets provided and their underlying relations.

In items 2 and 4 (adapted from Geerlings, 2008; Stanovich et al., 2016) students had to explain for which situations a conclusion is possible/justifiable based on the if–then-claim provided, respectively non-formally stated and formally stated. A table provided four possible situations: affirming and denying the antecedent, and affirming and denying the consequent. Although all the four different situations occur within one context, we decided to split tasks 2 and 4 into two sub-items instead of taking the context as a whole like in other tasks: Item A about the antecedent and item B about the consequent. This was done based on results from our pilot intervention where we observed differences in students’ reasoning about the antecedent and reasoning about the consequent.

Item 5, a short everyday reasoning task, was based on an implication for which students had to explain why the reversal is not always allowed. We conjecture that a Venn or an Euler diagram is a highly beneficial visualisation for this task.

Items 6 (adapted from Roodhardt & Doorman, 2012, p. 42; “Rotterdammers leven anderhalf jaar korter,” 2008) and 7 (adapted from “Hof: Wilders moet worden vervolgd,” 2009; Reuling, 2014) contained short newspaper articles for which the journalists’ reasoning had to be analysed and supplemented with hidden assumptions. We assumed that our participants have sufficient prior knowledge on these topics. We conjecture that a scheme-based strategy is highly beneficial in these tasks to show connections between premises, reasoning steps, and the conclusion(s).

Item 8 (adapted from College voor Toetsen en Examens, 2013, p. 7) contained a larger formal reasoning task, stated in ordinary language, in which students had to solve a murder case. This task is suitable for a scheme-based strategy to get a clear overview of the situation and claims.

Due to identical pre- and post-tests, we performed a test-retest reliability study. Among the control group, the Pearson Correlation between the two measurements, two months apart from each other, is r(107) = .71, p < .001, which shows an acceptable reliability.

Scoring procedure

All items were scored on correctness of the final answer (0 or 1 point) as well as on the quality of explanations, based on sample solutions. Each reasoning step was scored with one point. Maximum scores per item are 3 or 4 points (see Table 2).

Students’ use of formalisations was also coded, such as Euler/Venn diagrams, implication arrows, letter symbols, logical symbols, schemes, et cetera. The codes were based on the pilot study, although new codes could arise from the analysed data. Students could use more than one type of formalisations per task.

Example

Figure 2 shows a solution from student Amy from the experimental group. Amy used letter symbols for roses (R), flowers (B), and fading (V). Thereafter, she drew two different possible options: one where roses do not necessarily fade, and one where some roses do fade. In both cases, Amy clearly showed that all R (roses) are B (flowers), but that the set of flowers (B) is much larger. Her answer gets one point for being correct (“not necessarily”) and two points for the provided explanation; one for the insight that roses are part of the set flowers and one for showing that fading does not necessarily intersect with the set roses. The sum score for this item is equal to the maximum score of three points. Concerning the formalisations, we coded for the use of letter symbols and for Euler diagrams.

Figure 2. Example solution item 1, student from experimental group, English translation in purple.

Inter-rater reliability

To analyse the inter-rater reliability of the scoring, the first two authors of this article scored the complete work of 10 students from one group. Krippendorff's alpha (Hayes & Krippendorff, 2007) was computed over the total score for all items and resulted into α = .92 and α = .82 respectively for the pre- and post-test, which is more than sufficient for a reliable analysis (≥ .80). Observed differences between raters were discussed to modify the scoring model to allow for 100% agreement. Based on those discussions the analysis model was supplemented. Thereafter, all students’ answers were scored by the first author.

Analysis

For the strategies and formalisations used (first research question), percentages of students that used those formalisations are visualised per item in bar charts for a comparison between the experimental and control group. The effect of the intervention on students’ application of formalisations on separate items was analysed by means of logistic regression analyses using observation of the application on the pre-test as control variable. The significance of effects was assessed by computing the Wald statistic and the extent of the effect is expressed in the form of odds ratios of using specific formalisations (Sieben & Linssen, 2009).

For the effect on students’ overall performance (second research question), repeated measurement analyses, including the effect size partial eta-squared (Draper, 2019), and t-tests were deployed to investigate the size and significance of differences between the experimental and the control conditions on individual items, groups of items, and total test scores. To control the family-wise error rate due to multiple comparisons, we used Holm's sequential Bonferroni procedure to adjust the raw p-values (Holm, 1979). Cohen's d is used to calculate effect sizes (Lakens, 2013).

Finally, for the relationship between the test scores and the conjectured beneficial formalisations (third research question), we performed Kendall's Tau (Cohen et al., 2011). We investigated the correlation between the use of Venn and Euler diagrams and the correctness of students’ answers for corresponding tasks as well as the use of scheme-based strategies and the correctness. Again, we used Holm's sequential Bonferroni procedure to adjust the raw p-values for multiple comparisons (Holm, 1979).

Results

Logbooks showed that the lessons were implemented according to plan. One teacher deviated somewhat from the planning due to time constraints, but the other teachers taught the 10 lessons prior to the post-test and reported high student participation in their logbooks. During most lessons, students received sufficient time to work on their own solutions and discussed their strategies in small groups. Teachers gave, according to self-reported logbooks, more and more attention to formalisations by explicitly asking students to explain their formal methods in groups and during classroom discourse. They also encouraged students to reconsider their solutions by suggesting the use of Venn or Euler diagrams.

In this section, we follow the order of the research questions. We present students’ use of formalisations in the different tasks first. Second, we will report on students’ overall performance indicating a large intervention effect. Third, we present the relation between students’ use of formalisations with the correctness of the answers.

Use of formalisations

In this subsection, we will illustrate students’ use of formalisations in the different tasks supplemented with logistic regression analyses for beneficial formalisations.

Syllogism tasks, formal reasoning tasks, formally and non-formally stated

Figures 3 and 4 show that the intervention had a large influence on the use of formalisations in the syllogism tasks (items 1 and 3): students from the experimental group used more formalisations in the post-test than in the pre-test while there was very little change in the control group. For the non-formally stated version, 57.1% of the students in the experimental group used letter symbols for (parts of) the given premises in the post-test compared to 10.3% of the students in the control group. Between pre- and post-test, the use of Venn and Euler diagrams increased strongly for the formally stated version (from 5.4% to 57.1%) as well as in the non-formally stated version (from 3.6% to 53.6%) in the experimental group with hardly any changes in the control group. The Wald statistics reveal that the odds of using a Venn or an Euler diagram in the post-test items are 30.2 and 36.2 times higher respectively for students in the experimental group compared to students in the control group (Wald statistic = 34.6, p < .001 and Wald statistic = 30.9, p < .001 respectively).

Figure 3. Item 1, use of formalisations, non-formally stated syllogism task.

Figure 4. Item 3, use of formalisations or (informal) examples, formally stated syllogism task.

If–then statements, formal reasoning tasks, formally and non-formally stated

Figure 5 shows the use of formalisations for the task concerning a non-formally stated if–then-statement (items 2A & 2B). Although we found some increase in the use of formalisations among the students in the experimental group, for example from 0% to 14.3% between pre- and post-test for the use of Euler/Venn diagrams and 10.7% to 30.4% for the use of inequality signs, the majority of the students still did not use formalisations. For the task with the formally stated if–then-statement (items 4A & 4B), the use of formalisations in students’ answers was similar (see Figure 6). Notable is that a short formal expression (e.g. A ⇒ 3), which would have provided a clear visible implication for this task, was only used by 28.6% of the experimental students in the post-test. Further analysis reveals that the odds of using a short formal expression in the post-test item are 9.66 times higher for students in the experimental group compared to students in the control group (Wald statistic = 14.6, p < .001).

Figure 5. Items 2A/2B, use of formalisations, non-formally stated if-then statement.

Figure 6. Items 4A/4B, use of formalisations, formally stated if-then statement.

Everyday reasoning tasks

Figure 7 shows that there is a big change in the approach to tasks that involve the analysis of a short everyday reasoning proposition (item 5) among the students from the experimental group, while there are hardly any changes in the control group. Between pre- and post-test, the use of letter symbols increased from 3.6% to 55.4% in the experimental group and 23.2% used logical symbols in the post-test. Similar percentages were found for the use of Euler/Venn diagrams: from 3.6% to 53.6%. Further analysis reveals that the odds of using a Venn or an Euler diagram in the post-test item are 113 times higher for students in the experimental group compared to students in the control group (Wald statistic = 20.6, p < .001).

Figure 7. Item 5, use of formalisations or (informal) examples, short everyday reasoning task.

Figure 8 shows the use of formalisations for the task on revealing hidden assumptions in a newspaper article (item 6). Many students from both groups used bullet points to separate the three premises (reasons) in the pre- and post-test. For the connection between the premises and the hidden assumption(s) or conclusion(s), the use of implication arrows almost doubled within the experimental group to 39.3% in the post-test, compared to 26.2% in the control group. None of the students in the experimental group used a scheme-based strategy in the pre-test, which changed to 17.9% in the post-test. However, if explicitly asked for the different reasoning steps in another newspaper article task (item 7), many more students from the experimental group used a scheme-based approach (16.1% in the pre-test to 71.4% in the post-test, see Figure 9), providing structured answers: from premises via reasoning steps towards a conclusion. Further analysis reveals that the odds of schematising in the post-test item are 24.3 times higher for students in the experimental group compared to students in the control group (Wald statistic = 47.4, p < .001).

Figure 8. Item 6, use of formalisations, everyday reasoning task (newspaper article).

Figure 9. Item 7, use of formalisations, everyday reasoning task (newspaper article).

Murder case, formal reasoning task, non-formally stated

For the larger non-formally stated formal reasoning task, a murder case in item 8, we do not see much difference between pre- and post-test in both groups (see Figure 10). However, a change, from 51.8% to 66.1%, is visible in the experimental group for the strategy “structuring the data per person”, which is a formal approach to obtain an overview. Further analysis reveals that the odds of “structuring the data per person” in the post-test item are 3.12 times higher for students in the experimental group compared to students in the control group (Wald statistic = 9.08, p = .003).

Figure 10. Item 8, use of formalisations or (informal) examples, formally stated task (murder case).

Summary use of formalisations

In all tasks, we observe larger differences in students’ use of formalisations between pre- and post-test in the experimental condition. Further analyses by means of logistic regression show that the changes, in using Venn and Euler diagrams and scheme-based strategies in particular, between pre- and post-test are significant compared to the control group.

Overall test performance

Figures 11–14 show the effect of our intervention. The differences between the two groups between the pre- and post-test are shown for the mean test scores as well as for grouped item scores per category of tasks (see division in Figure 1). The vertical axis is scaled to the maximum score for the total test, the formally stated formal reasoning tasks, the non-formally stated formal reasoning tasks, and the everyday reasoning tasks, respectively. Except for the formally stated formal reasoning tasks, the lower endpoint of the 95% confidence interval for the experimental group is larger than the upper endpoint from the control group for the post-test, while for the pre-test the results are quite similar. The interaction effect, denoting the difference between the groups in terms of the change between pre- and post-test scores is significant for all categories of tasks (p < .05): F(1, 161) = 31.99, p < .001, ${\eta }_p^2$ = .17 for the total test score, F(1, 161) = 4.09, p = .045, ${\eta }_p^2$ = .025 for the formally stated formal reasoning tasks, F(1, 161) = 20.06, p < .001, ${\eta }_p^2$ = .11 for the non-formally stated formal reasoning tasks, and F(1, 161) = 20.47, p < .001, ${\eta }_p^2$ = .11 for the everyday reasoning tasks. For the total test score, the effect size is greater than .14, indicating a large intervention effect (Draper, 2019).

Figure 11. Interaction effect for total test.

Figure 12. Interaction effect for formally stated formal reasoning tasks.

Figure 13. Interaction effect for non-formally stated formal reasoning tasks.

Figure 14. Interaction effect for everyday reasoning tasks.

Further analyses per item are shown in Table 2. This table shows the mean scores and standard deviations for each task for the experimental and the control group as well as the results of a Paired Samples t-test for repeated measures in both conditions, indicating the direction and the significance of the change between pre-test and post-test. To control the family-wise error rate due to multiple comparisons, we used Holm's sequential Bonferroni procedure to compute the adjusted p-values. For the combined scores of each group of tasks as well as for the total test scores, the increase is significant for the experimental group and not significant for the control group (see Table 2). The effect sizes of these significant increases are, based in Cohen's d, even large for the grouped everyday reasoning tasks and for the total test score.

Table 2. Test scores pre- and post-test experimental and control group.

		Score pre-test		Score post-test		T-test* t(55) resp. t(106)
Item	Max score	Exp	Cont	Exp	Cont	Exp	Cont
1. Syllogism task  (Non-formally stated)  M  SD	3	1.13 1.16	1.15 1.08	1.48 1.24	1.31 1.13	t = 2.04 p̃ = .14 d = 0.27	t = 1.89 p̃ = .74 d = 0.18
2A. Implication, antecedent  (Non-formally stated)  M  SD	4	3.20 0.96	3.40 0.79	3.71 0.68	3.35 0.85	t = 3.71 p̃ = .004 d = 0.50	t = −0.58 p̃ = 1.00 d = −0.06
2B. Implication, consequent  (Non-formally stated)  M  SD	4	2.46 1.29	2.30 1.22	3.00 1.14	2.23 1.23	t = 2.79 p̃ = .029 d = 0.37	t = −0.59 p̃ = 1.00 d = −0.06
3 Syllogism task  (Formally stated)  M  SD	3	0.48 0.89	0.64 0.99	1.18 1.27	0.80 1.15	t = 3.83 p̃ = .003 d = 0.51	t = 1.78 p̃ = .85 d = 0.17
4A. Implication, antecedent  (Formally stated)  M  SD	4	2.84 1.47	2.93 1.32	2.63 1.78	2.89 1.54	t = −0.95 p̃ = .34 d = −0.13	t = −0.32 p̃ = 1.00 d = −0.03
4B. Implication, consequent  (Formally stated)  M  SD	4	1.36 1.43	1.40 1.37	2.21 1.50	1.76 1.47	t = 3.46 p̃ = .007 d = 0.46	t = 3.15 p̃ = .030 d = 0.30
5. Converse implication  (Everyday reasoning)  M  SD	3	0.70 0.78	0.51 0.74	1.30 0.95	0.49 0.69	t = 4.11 p̃ = .001 d = 0.55	t = −0.38 p̃ = 1.00 d = −0.04
6. Analysis article  (Everyday reasoning)  M  SD	4	1.96 1.41	2.20 1.34	2.32 1.38	2.40 1.35	t = 1.97 p = .14 d = 0.26	t = 1.73 p̃ = .85 d = 0.17
7. Reasoning steps article  (Everyday reasoning)  M  SD	4	2.61 0.95	2.45 1.02	3.18 0.90	2.43 0.99	t = 3.51 p̃ = .007 d = 0.47	t = −0.17 p̃ = 1.00 d = −0.02
8. Murder case  (Non-formally stated)  M  SD	4	1.63 1.45	1.62 1.37	2.30 1.26	1.68 1.24	t = 3.44 p̃ = .007 d = .46	t = 0.45 p̃ = 1.00 d = 0.04
Formal Reasoning Tasks: Formally stated (items 3, 4A, 4B)  M  SD	11	4.68 2.47	4.97 2.41	6.02 3.47	5.45 2.87	t = 2.91 p̃ = .026 d = 0.39	t = 2.46 p̃ = .20 d = 0.24
Formal Reasoning Tasks: Non-formally stated (items 1, 2A, 2B, 8)  M  SD	15	8.41 2.96	8.47 2.78	10.5 2.74	8.57 2.91	t = 5.59 p̃ < .001 d = 0.75	t = 0.40 p̃ = 1.00 d = 0.04
Everyday Reasoning Tasks (items 5, 6, 7)  M  SD	11	5.27 2.06	5.16 1.81	6.80 1.91	5.32 1.90	t = 6.50 p̃ < .001 d = 0.87	t = 0.87 p̃ = 1.00 d = 0.08
Total score  M  SD	37	18.36 4.95	18.60 5.03	23.32 6.03	19.34 6.03	t = 7.48 p̃ < .001 d = 1.00	t = 1.78 p̃ = .85 d = 0.17

*adjusted p-values based on Holm's sequential Bonferroni procedure.

Concerning the separate tasks, the scores for item 2B changed significantly with medium effect sizes with p̃ < .05 for the experimental group and with p̃ < .01 for items 2A, 3, 4B, 5, 7, and 8. For the control group we only found a significant change for item 4B, but with a small effect size.

Notable is the difference between items 4A and 4B, both concerning a formally stated if–then-statement. Reasoning with the consequent given shows a significant increase in the scores within the experimental group (p̃ < .01), but the scores for reasoning with the antecedent given did not show significant changes (p̃ = .34).

Formalisations and correctness

We conjectured that the use of Venn and Euler diagrams in formal reasoning tasks will improve students’ reasoning, and that everyday reasoning tasks will benefit from scheme-based strategies. In this subsection, we will investigate the relation between these formalisations and the corresponding tasks.

Venn and Euler diagrams

Venn and Euler diagrams are useful in, respectively, a non-formally stated syllogism task, a formally stated syllogism task, and a short everyday reasoning task (items 1, 3, and 5). Kendall's Tau coefficients were calculated to verify for a positive correlation (one-tailed) between the test score and students’ use of Venn/Euler diagrams, see Table 3. To control the family-wise error rate due to multiple comparisons, we used Holm's sequential Bonferroni procedure to compute the adjusted p-values. As shown in Table 3, we found a positive correlation between the use of Venn and Euler diagrams and the scores on the syllogism tasks (items 1 and 3). For the experimental group the correlation is fairly strong (τ_b = .61, p̃ < .001) in the post-test for the formally stated version. However, using a Venn or an Euler diagram in a short everyday reasoning task (item 5) does not show a significant correlation with the scores.

Table 3. Correlation test scores and use of Venn/Euler diagram.

	Pre-test		Post-test
Item	Exp	Cont	Exp	Cont
1. Syllogism task	τb = .15	τb = .28	τb = .39	τb = .25
(Non-formally stated)	p̃ = .49	p̃ = .012	p̃ = .008	p̃ = .023
3 Syllogism task	τb = .25	τb = .30	τb = .61	τb = .25
(Formally stated)	p̃ = .17	p̃ = .006	p̃ < .001	p̃ = .023
5. Converse implication	τb = −.034	τb = .096	τb = .12	τb = .14
(Everyday reasoning)	p̃ = .49	p̃ = .49	p̃ = .49	p̃ = .32

*adjusted p-values based on Holm’s sequential Bonferroni procedure.

Scheme-based strategy

We hypothesised that in everyday reasoning tasks about the analysis of newspaper articles (items 6 and 7) a scheme-based strategy is beneficial as well as for larger non-formally stated formal reasoning tasks (murder case in item 8). Kendall's Tau coefficients were calculated to verify for a positive correlation (one-tailed) between the test scores and students’ use of scheme-based strategies, see Table 4. Again, we used Holm's sequential Bonferroni procedure to compute the adjusted p-values. As shown in Table 4, we did not compute the correlation for item 6 for the pre-test of the experimental group, because none of the students used a scheme-based strategy, but in the post-test, we found for items 6 and 7 positive, although weak, correlations. However, for the larger non-formally stated formal reasoning task, a scheme-based strategy has a clear positive correlation for the experimental group in the post-test, increasing from τ_b = .18, p̃ = .36 to a significant τ_b = .44, p̃ = .002.

Table 4. Correlation test scores and use of scheme-based strategy.

	Pre-test		Post-test
Item	Exp	Cont	Exp	Cont
6. Analysis article  (Everyday reasoning)	unavailable	τb = −.065 p̃ = .46	τb = .20 p̃ = .36	τb = .10 p̃ = .39
7. Reasoning steps article  (Everyday reasoning)	τb = .18 p̃ = .36	τb = .16 p̃ = .27	τb = .20 p̃ = .36	τb = .27 p̃ = .016
8. Murder case  (Non-formally stated)	τb = .18 p̃ = .36	τb = .23 p̃ = .042	τb = .44 p̃ = .002	τb = .032 p̃ = .46

*adjusted p-values based on Holm's sequential Bonferroni procedure.

Conclusions and discussion

In this article, we reported on an intervention study on the feasibility of teaching students to use formalisations in their reasoning in formal and everyday reasoning tasks. This study was guided by three research questions, concerning (1) the use of formalisations in formally and non-formally stated formal reasoning tasks and in everyday reasoning tasks in pre- and post-test, (2) the effect of the intervention on students’ performance, and (3) the association between the use of formalisations and correctness of answers. The conclusions below focus on the experimental group unless stated otherwise.

Students’ use of formalisations

Answering the first research question on the use of formalisations, we found an increase in the use of formalisations in the post-test compared to the pre-test in the experimental condition, both in formal reasoning and everyday reasoning tasks. Students readily used letter symbols (see Figures 3, 5, 7, 10 about items 1, 2A/2B, 5, 8 respectively) providing them with an overview of the information. Also, more complicated formalisations were applied by the students: more than 50% of the students from the experimental group used, for example, Venn and/or Euler diagrams in items 1, 3, and 5 (see Figures 3, 4, and 7). In the experimental group we also found an increase of the use of logical symbols and/or implication arrows in the post-test compared to the pre-test, although less than 25% of the students used these formalisations (see Figures 3, 4, and 7). We did not observe changes in the use of formalisations in the control group.

For everyday reasoning tasks, scheme-based strategies are useful for structuring student's reasoning. If explicitly asked to show reasoning steps (item 7), over 70% of the students in the experimental group used a scheme-based formal strategy in the post-test, while only a minority of the students used a scheme-based strategy in the pre-test (see Figure 9 about item 7). However, in another everyday reasoning task without an explicit reference to reasoning steps (item 6), the use of a scheme-based strategy was much lower (0% in the pre-test to 17.9% in the post-test). Again, we did not observe many changes in the use of formalisations in the control group.

Overall, we can conclude that students from the experimental group used formalisations, and Venn and/or Euler diagrams in particular, more often than students from the control group after the intervention. This difference is significant shown by logistic regression with the pre-test as control variable. In general, students from the experimental group are able to apply those strategies in all sorts of tasks, both in formal reasoning tasks and everyday reasoning tasks.

Development of logical reasoning

Answering the second research question about the effect of the course in logical reasoning with a focus on the use of formalisations, sufficient attention for students’ own solution methods, social interactions, and the application of discovered and learned formalisations in open-ended everyday reasoning tasks, we showed a large intervention effect (${\eta }_p^2$ = .17 > .14 for the total test score). We also showed that students’ performance in all groups of tasks mentioned in Figure 1 (formally stated formal reasoning task, non-formally stated formal reasoning tasks, and everyday reasoning tasks) as well as the total test scores increased significantly for the experimental group.

Remarkable is the difference between items 4A and 4B in which we asked students to reason with antecedents and consequents given an if–then-statement. The focus on if–then-statements during the lessons improved students’ cautiousness when dealing with if–then-claims. If a consequent is given, they have to evaluate the if–then-claim carefully before they approve a conclusion. In the formally stated task as well as the non-formally stated task, we saw significant improvement for reasoning with a given consequent (items 4B and 2B) within the experimental group. However, in the formally stated version of this task (item 4A), although the scores are reasonably high (see Table 2), we did not see improvement when reasoning with a given antecedent, either in the experimental or in the control group. An explanation might be that students from the experimental group were too careful due to their increased knowledge for the situation of a provided consequent whereby a reversal of the if–then-claim is not allowed.

Some might link these items concerning if–then to the famous Wason selection task (Wason, 1968), which has received much criticism (Moshman, 1996; Stanovich et al., 2016; Wagner-Egger, 2007). We had, however, no intention to reopen those debates and, therefore, we did not use the original wording and presentation of the task, nor did we analyse answers separately. We used if–then-statements together with explanation boxes for certain antecedents and consequents. In that way, we forced students to carefully review each situation together with demanded justifications for their reasoning, so that their use of formalisations, became visible. We come back to this in the Limitations below.

Overall, this shows that for our specific target group of pre-university non-science students an approach with attention for supportive methods that structure the reasoning instead of focusing on formal logic is successful and this finding might explain the smaller effect size among undergraduate students in previous studies focusing on formal logic and conditional reasoning (e.g. Attridge et al., 2016; Cheng et al., 1986). Despite the attention on everyday reasoning tasks, it would be interesting to investigate if the logical reasoning skills students learned in this course are sustainable and can be transferred to other contexts to get more insights into the way these reasoning skills contribute to their future life in society, work, and study, which is essential for twenty-first century learning (NCTM, 2009; P21, 2015; Vincent-Lancrin et al., 2019).

Altogether, we can conclude that a course of 10 lessons in logical reasoning with specific attention to formalising and visualising statements and reasoning, in particular by means of Venn and Euler diagrams, is highly beneficial.

The role of formalisations on logical reasoning

Answering the third research question, we found positive correlations between the use of formalisations and the correctness of students’ answers (see Tables 3 and 4). In formal reasoning tasks, the correlation is the highest: positive correlations are found between the use of Venn and Euler diagrams and the scores for the syllogism tasks (items 1 and 3) in the post-test, and between the use of scheme-based strategies and the scores for the non-formally stated murder case (item 8). But the latter was only significant for the experimental group.

Although the post-test scores for the syllogism tasks in this study are still relatively low, the increase of scores within the experimental group is significant and for the formally syllogism task are scores on average 2.5 times higher than in the pre-test. This, combined with the positive correlations, shows that using Venn and Euler diagrams is highly beneficial and, thus, should be incorporated in the daily classroom discourse as also suggested by our earlier conducted exploratory study (Bronkhorst et al., 2020).

Contrary to our expectations, we did not find high correlations between the use of scheme-based strategies and the scores for the newspaper article tasks, despite improved total scores on the tasks themselves in the experimental group (everyday reasoning tasks, items 6 and 7) and even significantly for item 7. Probably, differences between these two tasks are an explanation for this finding: item 6 is more open and both items demand different prior knowledge. Although we assumed that our participants had sufficient prior knowledge on these topics, it shows, again, the complex matter of (research on) everyday life reasoning (Galotti, 1989).

Limitations

In our research, the control group did not receive teaching in logical reasoning, so that we could verify for changes over time and for retest effects. Because of practical reasons we were not able to compare the experimental group with groups that were taught in logical reasoning with their regular mathematics books, nor with science students. The most used Dutch textbook (Dijkhuis et al., 2017) mainly provides step-by-step plans to solve logical reasoning exercises without specific attention for the discovery of own formalisations. This way of teaching is still common practice in most countries, like the UK and USA (Nesmith, 2008; Pampaka & Williams, 2016; “Traditional teaching methods still dominant in maths classrooms,” 2012). If our teaching strategy shows significant improvements compared to the more traditional ways of teaching, it might provide further evidence that it is important to incorporate meaningful tasks related to everyday experiences with room for students’ own solutions and explaining/justifying those to their peers (Gravemeijer, 2020; National Research Council, 1999). Keijzer and Terwel (2003), for example, explain that meaningful tasks and formalising as an activity for exploring fractions should also be used by low-achieving students because they are also entitled to insightful learning. That is possible as long as those students will not be confused by other arithmetic and mathematical topics that are presented as step-by-step recipes. A precondition is, however, the quality of the teacher, for whom these activities probably demand different teaching strategies, which many teachers are not familiar with (KNAW, 2009, Chapter 2).

The test we used was validated with a pilot study and had a more than sufficient inter-rater reliability, but it is important to note that the predetermined weighing of the items affects the total scores of the students. The maximum score per item is 3 or 4 points based on the reasoning steps needed, because we were interested in the actual reasoning and not only in the final answers. On the other hand, the specific demand for explanations might have triggered students enough to review the given situations more carefully, which complicates comparisons with other studies. Still, when we looked at conditional reasoning specifically, we found significantly improved scores for dealing with the consequent. Quintana and Correnti (2019), for example, did not find differences in test scores in determining if such arguments are logically valid in closed questions, despite valuable results concerning everyday arguments. Our results also contradict previously mentioned findings from Cheng et al. (1986), but supplement findings from Attridge et al. (2016), who showed that improvement in logical reasoning is possible, in particular if students had previous experience with logic. Nevertheless, it might be interesting to compare experimental groups with and without specific demands for explanations in their pre- and post-tests.

Future research and recommendations

Overall, this study shows positive effects of teaching logical reasoning with a focus on the use of formalisations, including visualisations and schematisations. The observed increase in test scores on all kind of reasoning tasks, including everyday reasoning tasks, seems promising for developing critical thinking skills as part of the 21st century skills (P21, 2015) and thus for students’ future lives, not only in tertiary education, but also in their subsequent professions and for everyday life situations. We therefore recommend to use the described design characteristics for courses on logical reasoning, with the aim to develop critical thinking skills in and outside school. For future research we recommend to investigate long-term effects to see if students will continue using formalisations to decide if conclusions in, for example, newspapers are valid.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work is part of the research programme Doctoral Grant for Teachers with project number 023.007.043, which is (partly) financed by the Netherlands Organisation for Scientific Research (NWO) and is approved by The Research Ethics Committee (CETO) of the University of Groningen with application number 58772634.

Appendix

Pre- and post-test logical reasoning, translation into English

Note: Adapted from The rationality quotient: Toward a test of rational thinking (pp. 340–341), by K. E. Stanovich, R. F. West, and M. E. Toplak, 2016, Cambridge, MA: The MIT Press.

Note: Adapted from “Werkstuk vwo-5 wa1/wa1,2 ‘Logisch redeneren’,” by J. Geerlings, 2008. Retrieved from www.fi.uu.nl/ctwo/lesmateriaaldir/ExperimenteelLesmateriaal/VWO%20Wiskunde%20C/Logisch%20Redeneren/Oud%20materiaal/toetsideeen/20080404_Gregorius_WerkstukLogica_vwo5_a1_a12.doc

Note: Adapted from The rationality quotient: Toward a test of rational thinking (pp. 335–336), by K. E. Stanovich, R. F. West, and M. E. Toplak, 2016, Cambridge, MA: The MIT Press.

Note: Adapted from Experimentele uitgave voor Logisch redeneren, vwo, wiskunde C (p. 42), by A. Roodhardt, and M. Doorman, 2012, Utrecht, the Netherlands: cTWO. “Rotterdammers leven anderhalf jaar korter.” (2008). NRC Handelsblad. Retrieved from https://www.nrc.nl/nieuws/2008/02/08/rotterdammers-leven-anderhalf-jaar-korter-11483688-a374237

Note: Adapted from “Materiaal bij workshop ‘Logische redeneren’ in het programma vwo wisC 2015 wisC-dag (SLO),” by H. Reuling, 2014. Retrieved from https://prezi.com/embed/9hs2d9d48y83/. “Hof: Wilders moet worden vervolgd.” (2009). NRC Handelsblad. Retrieved from https://www.nrc.nl/nieuws/2009/01/21/hof-wilders-moet-worden-vervolgd-11671820-a414693

Note: Adapted from “Examen VWO 2013 wiskunde C (pilot), tijdvak 1.” (2013, p. 7). College voor Toetsen en Examens. Retrieved from https://www.examenblad.nl/examendocument/2013/cse-1/wiskunde-c-vwo/opgaven-2/2013/vwo/f=/vw-1026-f-13-1-o.pdf