Students’ use of formalisations for improved logical reasoning

Figures & data

Figure 1. Division of tasks.

Table 1. Overview intervention.

Lesson	Description and tasks	Learning goals
1	Introductory lesson with exploration of field of (logical) reasoning with formal and everyday reasoning tasks, such as puzzles, mathematical examples, and reasoning in a short newspaper article.	Students are introduced to the concepts of (logical) reasoning. Students are introduced to different types of reasoning. Through interactive tasks via Socrative*, students are made curious to the domain logical reasoning and experience that reasoning errors can occur quickly.
2	Students explore reasoning in a large context, a newspaper article about a court case, as relevant in the societal debate (everyday reasoning). The assignment is open-ended, but terms as premises, reasoning steps, conclusion, correct, complete, and incomplete are introduced.	Students discover how a logical argument is structured. Students are introduced to (logical) reasoning in the societal debate. Students can identify different parts in a text: premises/assumptions, reasoning step(s) and conclusion(s). Students explore/invent first schematisations of a larger newspaper article. Students know the concepts correct, complete, and incomplete.
3	Students explore syllogisms in ordinary language and study the difference between valid and true conclusions. They also try to formalise/structure the provided syllogisms in their own way and compare those ideas with peers before an example with letter symbols is shown.	Students learn to work with syllogisms in ordinary language (all, some, a few, not) and know the term “premise”. Students can distinguish truth and validity. Students explore conclusions for if-then claims.
4	Students are encouraged to come up with their own visualisations first before Venn and Euler diagrams are introduced and applied on syllogisms and propositions.	Students are introduced to Venn and Euler diagrams. Students can draw appropriate diagrams for propositions and syllogisms. Students can use Venn and Euler diagrams to falsify claims.
5	Logical symbols are introduced and combined with drawing and verifying conclusions for if-then-statements.	Students learn to work with the logical symbols: ∧, ∨, ⇒, and ⌋. Students can combine the theory of previous sections with the new symbols and diagrams. Students can draw and verify conclusions for if-then-statements. Students can visualise if-then-statements in Venn and Euler diagrams.
6	Further practice and discourse on if-then-statements.	Further practice of working with logical symbols, if-then-statements, and visualising in Venn or Euler diagrams. Students know the difference between sufficient condition and/or necessary condition.
7	First half of this lesson is dedicated to practice and discussion of an assignment in which everything from the previous sections is combined. The second half of the lesson, students explore newspaper articles in which the reasoning contains uncertainty.	Students combine knowledge from all the previous sections. Students can recognise reasoning with uncertainty.
8	In lesson two, students explored reasoning in a large newspaper article. After all sorts of short tasks in lessons 3 until 8, students analyse reasoning in large contexts again on which they can apply new techniques.	Students can analyse reasoning from large, societally relevant, contexts. Students practise with if-then, letter symbols, (logical) symbols, and visualisations.
9	Final interactive quiz and class discussion afterwards.	Students test their knowledge in a short interactive Socrative* quiz.
10	Final lesson with practice exercises, mainly longer formal reasoning tasks, but often stated in ordinary language.	Students practise their logical reasoning skills in a variety of exercises.

*Classroom app for Quizzes and formative tests: https://socrative.com/

Figure 2. Example solution item 1, student from experimental group, English translation in purple.

Figure 3. Item 1, use of formalisations, non-formally stated syllogism task.

Figure 4. Item 3, use of formalisations or (informal) examples, formally stated syllogism task.

Figure 5. Items 2A/2B, use of formalisations, non-formally stated if-then statement.

Figure 6. Items 4A/4B, use of formalisations, formally stated if-then statement.

Figure 7. Item 5, use of formalisations or (informal) examples, short everyday reasoning task.

Figure 8. Item 6, use of formalisations, everyday reasoning task (newspaper article).

Figure 9. Item 7, use of formalisations, everyday reasoning task (newspaper article).

Figure 10. Item 8, use of formalisations or (informal) examples, formally stated task (murder case).

Figure 11. Interaction effect for total test.

Figure 12. Interaction effect for formally stated formal reasoning tasks.

Figure 13. Interaction effect for non-formally stated formal reasoning tasks.

Figure 14. Interaction effect for everyday reasoning tasks.

Table 2. Test scores pre- and post-test experimental and control group.

		Score pre-test		Score post-test		T-test* t(55) resp. t(106)
Item	Max score	Exp	Cont	Exp	Cont	Exp	Cont
1. Syllogism task  (Non-formally stated)  M  SD	3	1.13 1.16	1.15 1.08	1.48 1.24	1.31 1.13	t = 2.04 p̃ = .14 d = 0.27	t = 1.89 p̃ = .74 d = 0.18
2A. Implication, antecedent  (Non-formally stated)  M  SD	4	3.20 0.96	3.40 0.79	3.71 0.68	3.35 0.85	t = 3.71 p̃ = .004 d = 0.50	t = −0.58 p̃ = 1.00 d = −0.06
2B. Implication, consequent  (Non-formally stated)  M  SD	4	2.46 1.29	2.30 1.22	3.00 1.14	2.23 1.23	t = 2.79 p̃ = .029 d = 0.37	t = −0.59 p̃ = 1.00 d = −0.06
3 Syllogism task  (Formally stated)  M  SD	3	0.48 0.89	0.64 0.99	1.18 1.27	0.80 1.15	t = 3.83 p̃ = .003 d = 0.51	t = 1.78 p̃ = .85 d = 0.17
4A. Implication, antecedent  (Formally stated)  M  SD	4	2.84 1.47	2.93 1.32	2.63 1.78	2.89 1.54	t = −0.95 p̃ = .34 d = −0.13	t = −0.32 p̃ = 1.00 d = −0.03
4B. Implication, consequent  (Formally stated)  M  SD	4	1.36 1.43	1.40 1.37	2.21 1.50	1.76 1.47	t = 3.46 p̃ = .007 d = 0.46	t = 3.15 p̃ = .030 d = 0.30
5. Converse implication  (Everyday reasoning)  M  SD	3	0.70 0.78	0.51 0.74	1.30 0.95	0.49 0.69	t = 4.11 p̃ = .001 d = 0.55	t = −0.38 p̃ = 1.00 d = −0.04
6. Analysis article  (Everyday reasoning)  M  SD	4	1.96 1.41	2.20 1.34	2.32 1.38	2.40 1.35	t = 1.97 p = .14 d = 0.26	t = 1.73 p̃ = .85 d = 0.17
7. Reasoning steps article  (Everyday reasoning)  M  SD	4	2.61 0.95	2.45 1.02	3.18 0.90	2.43 0.99	t = 3.51 p̃ = .007 d = 0.47	t = −0.17 p̃ = 1.00 d = −0.02
8. Murder case  (Non-formally stated)  M  SD	4	1.63 1.45	1.62 1.37	2.30 1.26	1.68 1.24	t = 3.44 p̃ = .007 d = .46	t = 0.45 p̃ = 1.00 d = 0.04
Formal Reasoning Tasks: Formally stated (items 3, 4A, 4B)  M  SD	11	4.68 2.47	4.97 2.41	6.02 3.47	5.45 2.87	t = 2.91 p̃ = .026 d = 0.39	t = 2.46 p̃ = .20 d = 0.24
Formal Reasoning Tasks: Non-formally stated (items 1, 2A, 2B, 8)  M  SD	15	8.41 2.96	8.47 2.78	10.5 2.74	8.57 2.91	t = 5.59 p̃ < .001 d = 0.75	t = 0.40 p̃ = 1.00 d = 0.04
Everyday Reasoning Tasks (items 5, 6, 7)  M  SD	11	5.27 2.06	5.16 1.81	6.80 1.91	5.32 1.90	t = 6.50 p̃ < .001 d = 0.87	t = 0.87 p̃ = 1.00 d = 0.08
Total score  M  SD	37	18.36 4.95	18.60 5.03	23.32 6.03	19.34 6.03	t = 7.48 p̃ < .001 d = 1.00	t = 1.78 p̃ = .85 d = 0.17

*adjusted p-values based on Holm's sequential Bonferroni procedure.

Table 3. Correlation test scores and use of Venn/Euler diagram.

	Pre-test		Post-test
Item	Exp	Cont	Exp	Cont
1. Syllogism task	τb = .15	τb = .28	τb = .39	τb = .25
(Non-formally stated)	p̃ = .49	p̃ = .012	p̃ = .008	p̃ = .023
3 Syllogism task	τb = .25	τb = .30	τb = .61	τb = .25
(Formally stated)	p̃ = .17	p̃ = .006	p̃ < .001	p̃ = .023
5. Converse implication	τb = −.034	τb = .096	τb = .12	τb = .14
(Everyday reasoning)	p̃ = .49	p̃ = .49	p̃ = .49	p̃ = .32

*adjusted p-values based on Holm’s sequential Bonferroni procedure.

Table 4. Correlation test scores and use of scheme-based strategy.

	Pre-test		Post-test
Item	Exp	Cont	Exp	Cont
6. Analysis article  (Everyday reasoning)	unavailable	τb = −.065 p̃ = .46	τb = .20 p̃ = .36	τb = .10 p̃ = .39
7. Reasoning steps article  (Everyday reasoning)	τb = .18 p̃ = .36	τb = .16 p̃ = .27	τb = .20 p̃ = .36	τb = .27 p̃ = .016
8. Murder case  (Non-formally stated)	τb = .18 p̃ = .36	τb = .23 p̃ = .042	τb = .44 p̃ = .002	τb = .032 p̃ = .46

*adjusted p-values based on Holm's sequential Bonferroni procedure.