Identifying obstacles to transfer of critical thinking skills

Figures & data

Table 1. The logic behind the procedure used.

Problem/step in the transfer process	Performance on posttest transfer items
Recognition-only	No support < Recognition support = Free recall = Recall support
Suboptimal recall	No support = Recognition support = Free recall < Recall support or No support < Free recall < Recall support Within free recall: positive correlation with retrieved information
Application scaffold	No support = Recognition support = Recall support = Free recall

Figure 1. Overview of the study design. The four conditions differed in amount of support received while completing the near and far transfer items of the posttest.

Figure 2. Violin plots with the full distribution per condition and test moment (i.e. pretest and posttest) on performance on learning items (maximum total score of 12) in Experiment 1.

Figure 3. Violin plots with the full distribution per condition and test moment (i.e. pretest and posttest) on performance on near transfer items (maximum total score of 7.5) in Experiment 1.

Figure 4. Violin plots with the full distribution per condition and test moment (i.e. pretest and posttest) on performance on far transfer items (maximum total score of 9) in Experiment 1.

Table 2. Experiment 1: mean (SD) of test performance (number of items correct) on learning (0–12), near transfer (0–7.5), and far transfer items (0–9) and mean (SD) of time-on-task (in seconds) on learning, near transfer, and far transfer items per condition.

		Level of support
		No support	Recognition support	Free recall	Recall support
Test performance
Learning	Pretest	5.38 (2.39)	4.90 (2.42)	4.61 (2.35)	5.45 (2.39)
	Posttest	8.34 (2.85)	7.65 (2.79)	7.75 (2.77)	7.69 (2.66)
	Pretest–posttest	2.96	2.75	3.14	2.24
Near transfer	Pretest	1.83 (1.70)	1.85 (1.45)	1.76 (1.73)	2.20 (1.83)
	Posttest	4.77 (2.14)	4.25 (2.17)	4.60 (2.03)	4.70 (1.96)
	Pretest–posttest	2.94	2.40	2.84	2.50
Far transfer	Pretest	1.65 (1.87)	1.40 (1.68)	1.49 (1.70)	1.56 (1.62)
	Posttest	3.17 (2.84)	3.14 (3.01)	2.56 (2.73)	3.87 (3.23)
	Pretest–posttest	1.52	1.74	1.07	2.31
Time-on-task
Learning	Pretest	80.46 (37.74)	82.02 (38.62)	81.12 (40.81)	79.07 (32.79)
	Posttest	51.05 (18.93)	52.82 (24.97)	57.32 (22.02)	49.83 (19.88)
	Pretest–posttest	−29.41	−29.20	−23.80	−29.24
Near transfer	Pretest	107.97 (48.51)	101.01 (48.41)	101.83 (46.72)	109.12 (55.26)
	Posttest	77.95 (32.49)	76.27 (28.38)	91.16 (31.13)	95.94 (31.05)
	Pretest–posttest	−30.02	−24.74	−10.67	−13.18
Far transfer	Pretest	84.45 (37.17)	83.77 (38.04)	89.01 (42.77)	91.26 (46.35)
	Posttest	46.92 (18.28)	49.25 (30.88)	57.43 (26.13)	62.08 (36.77)
	Pretest–posttest	−37.53	−34.52	−31.58	−29.18

Table 3. Experiment 1: Pearson correlation matrix (p-value) for the learning and transfer measures.

	1	2	3	4	5	6
1. Performance on learning items pretest	–
2. Performance on near transfer items pretest	.39* (<.001)	–
3. Performance on far transfer items pretest	.32* (<.001)	.17* (.023)	–
4. Performance on learning items posttest	.41* (<.001)	.29* (<.001)	.30* (<.001)	–
5. Performance on near transfer items posttest	.21* (.003)	.29* (<.001)	.26* (<.001)	.64* (<.001)	–
6. Performance on far transfer items posttest	.31* (<.001)	.23* (<.001)	.50* (<.001)	.47* (<.001)	.44* (<.001)	–

*p < .05.

Figure 5. Graphical representation of the relationship between retrieved information during free recall and posttest near transfer performance in Experiment 1. Two measures of retrieved information were used: nothing written down was either coded as no recall or as missing value.

Figure 6. Graphical representation of the relationship between retrieved information during free recall and posttest far transfer performance in Experiment 1. Two measures of retrieved information were used: nothing written down was either coded as no recall or as missing value.

Figure 7. Violin plots with the full distribution per condition and test moment (i.e. pretest and posttest) on performance on learning items (maximum total score of 12) in Experiment 2.

Figure 8. Violin plots with the full distribution per condition and test moment (i.e. pretest and posttest) on performance on near transfer items (maximum total score of 7.5) in Experiment 2.

Figure 9. Violin plots with the full distribution per condition and test moment (i.e. pretest and posttest) on performance on far transfer items (maximum total score of 9) in Experiment 2.

Table 4. Experiment 2: mean (SD) of test performance (number of items correct) on learning (0–12), near transfer (0–7.5), and far transfer items (0–9) and mean (SD) of time-on-task (in seconds) on learning, near transfer, and far transfer items per condition.

		Level of support
		No support	Recognition support	Free recall	Recall support
Test performance
Learning	Pretest	5.16 (2.25)	4.20 (2.26)	4.90 (2.33)	4.20 (2.78)
	Posttest	7.43 (2.23)	7.52 (2.87)	8.29 (2.38)	8.28 (3.13)
	Pretest-posttest	2.27	3.32	3.39	4.08
Near transfer	Pretest	1.64 (1.26)	1.21 (0.97)	1.94 (1.60)	1.96 (1.48)
	Postest	3.50 (1.98)	3.61 (1.97)	3.92 (2.52)	4.65 (2.11)
	Pretest-posttest	1.86	2.40	1.98	2.69
Far transfer	Pretest	2.11 (1.69)	2.16 (2.13)	0.92 (1.04)	0.89 (1.74)
	Posttest	2.95 (2.46)	3.65 (3.05)	3.08 (2.23)	3.00 (3.18)
	Pretest-posttest	0.84	1.49	2.16	2.11
Time-on-task
Learning	Pretest	90.02 (21.27)	80.10 (27.34)	82.85 (21.55)	90.84 (39.81)
	Posttest	63.42 (19.17)	53.08 (18.98)	57.64 (20.88)	59.51 (22.58)
	Pretest-posttest	−26.60	−27.02	−25.21	−31.33
Near transfer	Pretest	115.02 (39.28)	101.69 (33.06)	106.63 (36.83)	113.69 (40.27)
	Posttest	73.21 (23.68)	76.50 (19.88)	92.52 (27.54)	95.91 (30.35)
	Pretest-posttest	−41.81	−25.19	−14.11	−17.78
Far transfer	Pretest	107.40 (33.89)	101.53 (36.51)	103.32 (36.16)	106.26 (37.14)
	Posttest	58.73 (17.41)	63.38 (22.82)	56.07 (21.04)	73.63 (35.06)
	Pretest-posttest	−48.67	−38.15	−47.25	−32.63

Table 5. Experiment 2: Pearson correlation matrix (p-value) for the learning and transfer measures.

	1	2	3	4	5	6
1. Performance on learning items pretest	–
2. Performance on near transfer items pretest	.50* (<.001)	–
3. Performance on far transfer items pretest	.37* (<.001)	.13* (.026)	–
4. Performance on learning items posttest	.30* (.003)	.35* (<.001)	.16 (.125)	–
5. Performance on near transfer items posttest	.25* (.014)	.38* (<.001)	.07 (.514)	.63* (<.001)	–
6. Performance on far transfer items posttest	.32* (.001)	.20 (.054)	.48* (<.001)	.25* (.012)	.18 (.085)	–

*p < .05.

Figure 10. Graphical representation of the relationship between retrieved information during free recall and posttest near transfer performance in Experiment 2. Two measures of retrieved information were used: nothing written down was either coded as no recall or as missing value.

Figure 11. Graphical representation of the relationship between retrieved information during free recall and posttest far transfer performance in Experiment 2. Two measures of retrieved information were used: nothing written down was either coded as no recall or as missing value.

Table

Affirming the antecedent Statement 1: If P, then Q Statement 2: P Conclusion: Therefore Q (valid)	Affirming the consequent Statement 1: If P, then Q Statement 2: Q Conclusion: Therefore P (invalid)
Denying the antecedent Statement 1: If P, then Q Statement 2: Not P Conclusion: Therefore not Q (invalid)	Denying the consequent Statement 1: If P, then Q Statement 2: Not Q Conclusion: Therefore not P (valid)

Table

	Multiple-choice answer	Explanation
	Correct 0,5 point	Correct 1 point	Partially correct 0.5 point	Incorrect 0 points
Affirming the consequent If P, then Q Q Therefore P	B. The conclusion does not follow logically from the two premises.	If one of the underlined sentences is mentioned: If the first premise is met, Q automatically follows. The second premise states that Q is affirmed. But it does not mean that Q is caused by P. There might be another cause than P. The presence of Q does not necessarily mean that P is the cause.	If only one of the following explanations is given: The rule is “if P, then Q”. Not “if Q, then P”. Affirming the consequent.	If none of the preceding arguments is mentioned.
Denying the consequent If P, then Q –Q Therefore –P	A. The conclusion follows logically from the two premises.	If one of the underlined sentences is mentioned: If the first premise is met, Q automatically follows. The second premise states that Q is denied. Therefore, P is not present. Because if P had been present, Q would have always followed. Thus, if Q is absent, then P is also absent.	If only one of the following explanations is given: Because Q is denied (e.g. because the employees are not Dutch). Denying the consequent.	If none of the preceding arguments is mentioned.
Affirming the antecedent If P, then Q P Therefore Q	A. The conclusion follows logically from the two premises.	If the underlined sentence is mentioned: If the first premise is met, Q automatically follows. The second premise states that P is affirmed and, thus, Q follows. Because P causes Q. Although this is not in line with my existing knowledge / this is unbelievable.	If only one of the following explanations is given: Because P is affirmed (e.g. because the report contains numbers). Affirming the consequent. Rules of logic are in this case more important than personal experiences / existing knowledge.	If none of the preceding arguments is mentioned.
Denying the antecedent If P, then Q –P Therefore –Q	B. The conclusion does not follow logically from the two premises.	If one of the underlined sentences is mentioned: If the first premise is met, Q automatically follows. The second premise states that P is denied. But it does not mean that Q is not present. It is possible that something else causes Q. The absence of P does not necessarily mean that Q is not present.	If only one of the following explanations is given: Q can still occur (e.g. health benefits can still be achieved). Denying the antecedent.	If none of the preceding arguments is mentioned.
Wason selection task The rule is: If P, then Q. Which […] should you read /test/turn to check the rule?	A. P B. not P C. Q D. not Q	If the underlined explanations are given: By turning over card A, I test whether the rule is violated: if Q is not on the back, the rule is violated. Because if P is present, then Q follows. Although this is not in line with my existing knowledge / this is unbelievable. I can also test the rule by turning over card D: if P is on the back, the rule is violated. Because if P is present, then Q should have followed. If Q is absent, P is also absent. Note: If one explains a rule incorrectly (e.g. card A, C and D chosen and incorrectly explained that card C can confirm the presence of P), then s/he loses 0.25 point per incorrect rule.	If only one of the underlined explanations is given: By turning over card A, I test whether the rule is violated: if Q is not on the back, the rule is violated. Because if P is present, then Q follows. Although this is not in line with my existing knowledge / this is unbelievable. I can also test the rule by turning over card D: if P is on the back, the rule is violated. Because if P is present, then Q should have followed. If Q is absent, P is also absent. Note: If one explains a rule incorrectly (e.g. card A and C chosen and incorrectly explained that card C can confirm the presence of P), then s/he loses 0.25 point per incorrect rule. Note: If one correctly states why card C is not chosen, s/he earns 0.25 point (e.g. only card A selected and correctly explained that card C does not give information). If the following explanation is given: Affirming the consequent and denying the antecedent.	If none of the preceding arguments is mentioned.

Table

	Correct 1 point	Partially correct 0.5 point
Affirming the consequent If P, then Q Q Therefore P	Observation linked to the conclusion and assessment of the validity of the conclusion: Q, therefore P (invalid). Q, P does not have to be present.	Only the observation and the validity of the conclusion. Affirming Q is invalid. Then-part present is invalid.
Denying the consequent If P, then Q –Q Therefore –P	Observation linked to the conclusion and assessment of the validity of the conclusion: Not Q, therefore not P (valid).	Only the observation and conclusion or the observation and the validity of the conclusion. Not Q, thus not P. Denying Q is valid. Then-part absent is valid.
Affirming the antecedent If P, then Q P Therefore Q	Observation linked to the conclusion and assessment of the validity of the conclusion: P, therefore Q (valid).	Only the observation and conclusion or the observation and the validity of the conclusion. P, thus Q.Affirming P is valid. If-part present is valid.
Denying the antecedent If P, then Q –P Therefore –Q	Observation linked to the conclusion and assessment of the validity of the conclusion: Not P, therefore not Q (invalid). Not P, Q may be present.	Only the observation and the validity of the conclusion. Denying P is invalid. If-part absent is invalid.

Data availability statement

The datasets and script files are stored on an Open Science Framework (OSF) page for this project, see osf.io/ybt5g.