Exploring core ideas of procedural understanding in scientific inquiry using educational data mining

Figures & data

Figure 1. Scientific thinking and its sub-competences, competence-aspects as well as influencing factors (Arnold et al. 2018; Mayer 2007).

Figure 2. Exemplary concept cartoon ‘Why do we need a hypothesis?’ (Arnold, Kremer, and Mayer 2017; translated).

Table 1. Coding manual for concept cartoon ‘Why do we need a hypothesis?’.

Category Sub-Category	Description ● Exemplary answers
Validity	Students describe the purpose of a hypothesis, making reference to criteria of validity.
The hypothesis is used to test/ensure that what is to be investigated is investigated	It is clear from the student’s expression that the hypothesis serves to test or ensure that what is to be investigated is investigated.
	B wants to try out, but in order to come to a result, B must know whether he is investigating what he wants to investigate (validity).
	so that you really measure what you want to measure
	You cannot start without a hypothesis, because you need it to know what you want to check.
The hypothesis focuses on variables/serves to define variables	It is clear from the student’s expression that the hypothesis serves to focus and/or define the variables to be investigated.
	The hypothesis that is put forward directs the focus to individual independent variables
	We agree with statement C the most, because before an experiment can be conducted, the hypothesis/question must be established in order to find the independent and dependent variable.
Objectivity	Students describe the purpose of a hypothesis, making reference to criteria of objectivity.
The hypothesis serves the independent consideration/(intersubjective) comprehensibility of the investigation	It is clear from the student’s expression that the hypothesis serves to look at the results independently or to make the experiment and/or the results comprehensible to (others).
	and so that other people understand exactly what assumption you are investigating.
	We find a hypothesis very important, because without a hypothesis the result cannot be considered independently.
	Furthermore, a hypothesis made after the test would be influenced by the test result.
Other	Students describe a hypothesis or its purpose without (implicit) reference to scientific quality criteria. The aim of an experiment is to prove the hypothesis. Therefore, it would be pointless to carry out an experiment without first having made an assumption. He needs a hypothesis to orientate oneself.

Table 2. Concept cartoon-questions (Arnold 2015; Arnold, Kremer, and Mayer 2016).

Cartoon Number	Focus Question	Interrater-Agreement (Cohen’s κ)
1	Why do you need hypotheses?	.64
2	Why should you plan experiments?	.73
3	Why should you think twice about measuring-type and – times of the dependent variable?	.65
4	Why should you think twice about the variation of the independent variable?	.55
5	Why should you describe data?	.85
6	Why do you need alternative Hypotheses?	.68
7	Why should you control confounding variables?	.79
8	Why should you repeat experiments?	.86

Table 3. Ideas of procedural understanding, abbreviations, and matching quality criteria.

Idea of Procedural Understanding	Abbreviation	Quality Criterion	Number of mentions/total of statements
Cartoon 1: The hypothesis …
… helps to ensure that one investigates what is to be investigated.	HY1	V	10/121
… directs focus on the variables under investigation.	HY2	V	11/121
… serves to make the study transparent.	HY3	O	6/121
Cartoon 2: One needs alternative hypotheses, …
… because there may be other variables worth considering as cause.	AH1	V	7/88
… as the influencing factor may act in the other direction.	AH2	V	2/88
… because, in this way, limitations of the hypothesis can be uncovered.	AH3	V	1/88
… in order to exclude contrary (other) hypotheses.	AH4	V	13/88
… because that way, one can preserve openness to the outcome of the investigation.	AH5	O	10/88
… because there are several possibilities for the outcome of the experiment, which should be considered.	AH6	O	17/88
… because there are different perspectives that should be considered.	AH7	O	7/88
… because they stimulate discussion and detailed analysis of the results.	AH8	O	3/88
Cartoon 3: The planning helps to …
… make sure that is investigated what should be investigated.	PL1	V	15/142
… take into account confounding variables or external circumstances.	PL2	V	13/142
… ensure traceability.	PL3	O	17/142
… ensure measurement accuracy.	PL4	R	6/142
Cartoon 4: The type and the time of measurement of the dependent variable should be carefully considered, …
… to make sure that is measured, which is to be measured.	DV1	V	8/116
… in order to obtain meaningful results.	DV2	V	4/116
… to exclude confounding factors.	DV3	V	11/116
… to make sure that is always measured in the same way and at the same time.	DV4	V	4/116
… to choose measurable dependent variables.	DV5	V	6/116
… to enable reproducible results without subjective influences.	DV6	O	13/116
… to ensure accurate measurement.	DV7	R	20/116
… to choose adequate intervals.	DV8	R	6/116
Cartoon 5: The type of variation of the independent variable should be carefully considered, …
… to make sure that is measured, which is to be measured.	IV1	V	2/90
… in order to obtain meaningful results.	IV2	V	14/90
… to exclude confounding factors.	IV3	V	5/90
… to ensure that a comparison value is reached.	IV4	V	7/90
… to choose a measurable independent variable.	IV5	V	1/90
… to ensure accurate measurement.	IV6	R	11/90
… to select sufficient appropriate intervals.	IV7	R	21/90
Cartoon 6: Confounding variables should be controlled, …
… to make sure that is measured, which is to be measured.	CV1	V	12/74
… since they influence the experiment, the dependent variable.	CV2	V	37/74
… since the omission may lead to inconclusive results.	CV3	V	6/74
… to ensure the transparency of the investigation.	CV4	O	5/74
… to ensure accurate measurement.	CV5	R	6/74
Cartoon 7: Experiments should be repeated, …
… to increase the explanatory power of the experiment.	RE1	V	33/87
… to rule out external or confounding factors.	RE2	V	12/87
… to make sure the repeatability of the investigation.	RE3	O	2/87
… to increase the measurement accuracy.	RE4	R	7/87
Cartoon 8: The description of the data serves …
… the traceability.	DE1	O	25/112
… to avoid misinterpretation.	DE2	O	4/112
… to not jump to conclusions.	DE3	O	8/112

Note: O = Objectivity; R = Reliability; V = Validity.

Figure 3. Concept cartoon ‘Why do we need a hypothesis?’ with rating scale.

Figure 4. Percentage of the agreement to concept-cartoon-items (the bars represent the combined ‘strongly agree’ and ‘agree’ answers).

Figure 5. Significant correlations (p < .05) between the ideas of procedural understanding (Arnold, Kremer & Mühling, 2017).

Note: green = directly related to the concept of confounding variables; red = indirectly related to the concept of confounding variables.

Table 4. Concepts that differentiate good from bad performers on the procedural-understanding-test (p < .05).

Item	Performance in PU-Test	N	M	SD	t	df	p (2-sided)	Effect size (r)
AH1	low	39	3.67	1.36	2.22	62	.030	0.27
	high	25	4.36	0.95
AH2	low	38	2.74	1.06	2.70	61	.009	0.32
	high	25	3.52	1.23
AH4	low	38	3.61	1.48	−2.71	61	.009	0.33
	high	25	2.60	1.38
PL3	low	38	3.39	1.03	4.10	60	.000	0.47
	high	24	4.42	0.83
CV3	low	38	3.89	1.33	2.37	61	.021	0.29
	high	25	4.60	0.82
CV5	low	38	3.53	1.48	2.17	61	.034	0.27
	high	25	4.28	1.10

Arnold, J. C., W. J. Boone, K. Kremer, and J. Mayer. 2018. “Assessment of Competencies in Scientific Inquiry through the Application of Rasch Measurement Techniques.” Education Sciences 8 (4): 184. doi:10.3390/educsci8040184.

 Web of Science ®Google Scholar

Mayer, J. 2007. “Erkenntnisgewinnung als wissenschaftliches Problemlösen [Knowledge acquisition as scientific problem solving].” In Theorien in Der Biologiedidaktischen Forschung [Theories in Biology Education Research], edited by D. Krüger and H. Vogt, 177–186. Berlin: Springer.

 Google Scholar

Arnold, J., K. Kremer, and J. Mayer. 2017. “Scaffolding beim Forschenden Lernen - Eine empirische Untersuchung zur Wirkung von Lernunterstützungen.” Zeitschrift für Didaktik der Naturwissenschaften 23 (1): 21–37. doi:10.1007/s40573-016-0053-0.

 Google Scholar

Arnold, J. 2015. Die Wirksamkeit von Lernunterstützungen beim Forschenden Lernen: Eine Interventionsstudie zur Förderung des Wissenschaftlichen Denkens in der gymnasialen Oberstufe. Berlin: Logos.

 Google Scholar

Arnold, J., K. Kremer, and J. Mayer. 2016. “Concept Cartoons als diskursiv-reflexive Szenarien zur Aktivierung des Methodenwissens beim Forschenden Lernen.” Biologie Lehren und Lernen– Zeitschrift für Didaktik der Biologie 20 (1): 33–43. doi:10.4119/UNIBI/zdb-v1-i20-324.

 Google Scholar

Arnold, J., K. Kremer, and A. Mühling. 2017. “Denn sie wissen nicht, was sie tun« Educational Data Mining zu Schülervorstellungen im Bereich Methodenwissen.” Mathematisch und naturwissenschaftlicher Unterricht (MNU) 70 (5): 334–340.

 Google Scholar