Learners’ challenges in understanding and performing experiments: a systematic review of the literature

ABSTRACT

In today’s world shaped by technology and the natural sciences, knowledge and skills related to experimentation are crucial, especially given growing public debates about science-related topics. Despite a strong emphasis on experimentation in science curricula worldwide, learners still encounter diverse challenges when experimenting. This paper provides a systematic review of empirical research on learners’ challenges during the following inquiry phases: stating research questions, generating hypotheses, planning and conducting an experiment, analysing data and drawing conclusions. A database search and an analysis of two prior narrative literature reviews identified 368 studies, of which 66 were used for further analyses after screening for eligibility using specific inclusion criteria. The analyses revealed 43 challenges at the conceptual, procedural and epistemic level that not only elementary but even university students face during experimentation. Additionally, cognitive biases and preconceptions are identified as a source of such challenges. Overall, the findings demonstrate a lack of in-depth research on stating research questions despite its importance for experimentation, whilst learners’ abilities in the other inquiry phases have been intensively investigated. The results offer valuable information for science education research and provide a basis for tailored scaffolding in the science classroom or the design of effective instructional interventions.

KEYWORDS:

• Challenges
• experiment
• inquiry-based learning
• learners
• literature review

1. Introduction

To cope with the demands of the twenty-first century and become more critical citizens, learners need to be able to use their understanding of science to contribute to public debate and develop informed opinions on science- and technology-based questions (OECD, 2018). Underpinning this important goal is the notion of scientific literacy (Bybee, 1997). Scientific literacy allows people to address, make decisions and take action on complex problems in a value-driven, scientifically knowledgeable and empowered manner (Abd-El-Khalick et al., 2004; Adams et al., 2018). Engaging learners in activities in which they apply scientific methods similarly to professional scientists (Keselman, 2003) is seen as one important way to foster scientific literacy and has therefore become a fundamental component of science teaching and learning (Rönnebeck et al., 2016). Conducting experiments is considered a particularly central investigative activity within scientific methods (Pedaste et al., 2015). Consequently, science standards and curricula worldwide emphasise the importance of experiments in the science classroom and seek to systematically foster knowledge and skills necessary for learners to understand and perform experiments (e.g., DfES & QCA, 2004; KMK, 2005a, 2005b, 2005c; NRC, 1996, 2000, 2012, 2013). One prominent science education strategy to engage learners in experimentation is inquiry-based learning (IBL; Rönnebeck et al., 2016). By directing their own investigative activities in the form of experiments, learners can gain insights into how scientific knowledge is produced (Arnold et al., 2014) and how scientists establish the credibility of the claims they advance (Osborne, 2014). However, while experimentation opens up rewarding learning opportunities, it also presents great challenges¹ to learners. In this regard, numerous studies show that experimentation is a complex task, and learners’ abilities to state research questions, generate hypotheses, plan and conduct experiments, analyse data and draw conclusions are critical components for successful IBL (De Jong & van Joolingen, 1998; Zimmerman, 2007).

Identifying and analysing learners’ challenges will not only render learners’ implicit conceptions or errors explicit, but also allow these difficulties to be directly addressed in order to challenge and transform them, while acknowledging them as an important antecedent for learning (Siler & Klahr, 2012). This can be achieved if challenges are used to derive potential from them: learners’ challenges can potentially serve as a template for creating effective guidance and scaffolding and later for adapted lesson planning (Baur, 2015; Van Uum et al., 2017). Moreover, deeper knowledge of learners’ challenges is crucial not only in science education, but in science education research as well. Identifying and analysing learners’ challenges makes it possible to examine the reasons for their existence in greater detail. In this way, tailored support methods can be developed and empirically tested for their effectiveness in overcoming specific challenges in order to develop successful IBL concepts and interventions. In addition, challenges can potentially be linked to specific knowledge and skills needed in the investigative process.

Thus, the objective of this systematic literature review is to summarise and describe empirical research findings on learners’ challenges with respect to understanding and performing experiments. For this purpose, the article first presents theoretical background information on IBL and experiments. Next, it identifies knowledge and skills required for experimentation, which serves as a crucial baseline for analysing related challenges. Afterwards, the methodology and results of the newly conducted review are presented. Finally, the results are discussed and implications are derived.

2. Inquiry-based learning (IBL) and experiments

IBL has been considered as an essential component of science education for more than 50 years (Stender et al., 2018). Due to its long-lasting importance and the extensive literature on the topic, the term IBL is accompanied by a variety of descriptions and connotations (Abrams et al., 2008; Anderson, 2002; Blanchard et al., 2010). In this regard, Rönnebeck et al. (2016) argues that inquiry is not only understood as a way for individuals to learn science and scientific ways of obtaining knowledge through different contents and materials but also as an instructional approach (Bybee, 2006; Furtak et al., 2012). Inquiry as an instructional approach can be defined along different forms of instruction (Blanchard et al., 2010; Furtak et al., 2012; Hmelo-Silver et al., 2007), ranging from minimally guided, discovery-oriented approaches in which learners engage in hands-on activities (e.g., Kirschner et al., 2006) to more structured forms, like traditional (verification) laboratory instruction, where a distinctive list of activities is predefined (Blanchard et al., 2010). The National Research Council (NRC, 1996) defines IBL as activities in which learners develop knowledge and understanding of scientific ideas, as well as an understanding of how scientists study the natural world and propose explanations based on the evidence derived from their work. This definition focuses on aspects from the learner’s point of view, not only on acquiring knowledge, but also on developing ways of scientific thinking and working as well as gaining an understanding of those methods. Furthermore, for this review we define IBL in terms of three dimensions (Duschl, 2008; Furtak et al., 2012): the conceptual, epistemic and procedural domain. In the PISA assessment, it is referred to as content, epistemic and procedural knowledge (OECD, 2019). The conceptual domain contains declarative knowledge, also referred to as substantive knowledge, consisting of learners understanding facts, theories, and principles of science, meaning science as a body of knowledge. If investigation methods are explicit learning content (gaining an understanding of those methods), then the concepts of these methods (such as: ”What exactly is an experiment?” or ”What is a research question?”) can also be understood as a part of declarative knowledge and must therefore be seen as a part of this specific content domain. The epistemic domain is based upon learners’ comprehension of how scientific knowledge is generated, meaning an understanding of the role of specific constructs and defining features essential to the process of building scientific knowledge (Duschl, 2008). ‘Epistemic knowledge includes an understanding of the function that questions, observations, theories, hypotheses, models and arguments play in science’ (OECD, 2019, p. 100). Furthermore, it describes an understanding of the rationale for the common practices of scientific inquiry, the status of claims that are generated and the meaning of foundational terms such as hypothesis, variable and theory (OECD, 2019). The focus lies on learning to develop explanations for phenomena and to understand that gained scientific knowledge is subject to change in the face of new evidence or new interpretations of old evidence, similar to the practice of professional scientists. In relation to IBL, in the literature procedural knowledge is referred not only to the knowledge of how to do something, of techniques and specific methods of a discipline, but also to the knowledge of scientific processes used to establish scientific knowledge (e.g., Gott & Duggan, 1995; Roberts et al., 2010). Hence, in this interpretation, it includes the knowledge of the practices and concepts on which empirical inquiry is based (see, Millar & Lubben, 1996). In this literature review, we define experimentation as a learning content and therefore focus mainly on the practices, the – procedural knowledge – in the procedural dimension. The procedural domain focuses in this review especially on hands-on performances and practically performing the scientific investigation.

As already explained above, a common and important type of scientific investigation in IBL settings is the experimental method (Osborne et al., 2003). The term experiment is often used as a synonym for other scientific investigations because it can include methods such as describing, observing, comparing or classifying (Mayer & Ziemek, 2006). However, an experiment goes beyond observation for example, as an experimenter controls the artificially altered conditions and intentionally intervenes in the process. Thus, although the term ‘experiment’ is used for different types of investigations (Arnold et al., 2014), it is generally considered the method of choice for investigating causal relationships between dependent and independent variables (Wellnitz & Mayer, 2011). This is also referred to as ‘fair testing’ (Gott et al., 2008).

Experimentation is often described as a problem-solving process (e.g., Klahr & Dunbar, 1988) in which ideas are tested (Osborne et al., 2003) to draw valid inferences about causal hypotheses based on the evidence derived (NRC, 2013). Depending on the goals, content, as well as the institutional setting, different types of experiments can be distinguished (e.g., Mayer & Ziemek, 2006; Roberts & Gott, 2006). The experiment is subject to the quality criteria of scientific work: objectivity, reliability and validity. The objectivity of an experiment ensures intersubjective comprehensibility, reliability ensures the measurement accuracy and reliable repeatability of an experiment, and validity ensures that the experiment really measures what it is intended to measure (Gott et al., 2008). The experimental process follows a hypothetico-deductive method (Popper, 1966) that typically consists of several logically connected phases, also known as inquiry phases. From a pedagogical point of view they guide students through the experiment as they subsequently organise the inquiry process and draw attention to important features of scientific thinking (Pedaste et al., 2015). These inquiry phases require learners to actively participate in inquiry activities; coordinating related knowledge and skill simultaneously (NRC, 2013). Although inquiry phases are logically connected, inquiry cannot be seen as a uniform linear process, but rather as a cycle in which multiple links, overlaps and interconnections between phases are possible (Pedaste et al., 2015; Rönnebeck et al., 2016). Descriptions of inquiry cycles by different researchers sometimes use different terms for phases that are essentially the same (Rönnebeck et al., 2016). Nevertheless, the successful application of inquiry in different settings suggests an underlying commonality that is at the core of IBL (Pedaste et al., 2015).

As our focus in this review lies on experimentation, the goal was to cover the main phases of this kind of scientific investigation. Thus, a conceptualisation was needed which targeted key components of experimentation instead of more broader concepts such as scientific inquiry or inquiry in general (e.g., Pedaste et al., 2015; Rönnebeck et al., 2016). The model used in this review is a synthesis of existing activity-based conceptualisations, which have lead to four main phases and related activities of IBL through experimentation: 1) stating scientific questions, 2) generating hypotheses 3) planning and conducting an experiment, and 4) analysing data and drawing conclusions (Arnold et al., 2014; Bruckermann et al., 2017; Hofstein et al., 2005; Kremer et al., 2013, 2019; Mayer, 2007; Mayer et al., 2008; Meier & Mayer, 2012). These inquiry phases can be found in many other conceptualisations (e.g., Bybee, 2006; Pedaste et al., 2015; Rönnebeck et al., 2016). Furthermore, this conceptualisation was selected because it is applicable to the conceptualisations used by empirical studies found in the data search from different disciplines covering IBL and experimentation in particular. Because of the overlaps with many other conceptualisations and its interdisciplinary nature, the conceptualisation seemed like an adequate basis for this review. To carry out each of these inquiry phases, learners need a variety of skills and knowledge during experimentation, which are described in the following paragraphs:

Stating scientific questions

Formulating research questions is important in problem-solving procedures (Cuccio-Schirripa & Steiner, 2000), because it usually represents the start of the research process (Bell et al., 2005), the pursuit of which advances and extends knowledge on a certain topic (Chin & Osborne, 2008; White & Frederiksen, 1998). The research question underlying an experiment focuses on the cause of an observed phenomenon (Pedaste et al., 2015) and creates a link to relevant prior knowledge. As an experiment investigates causal relationships between the dependent and independent variable(s), research questions ask about those causal relationships and can then be answered through the subsequent experimental process.

Generating hypotheses

Hypotheses provide plausible, justified answers to research questions (Eastwell, 2014). Hypotheses seeking to explain the investigated phenomenon are consistent with prior knowledge, theories or principles (Arnold et al., 2018) and align with the stated research question (Klahr & Dunbar, 1988). Accordingly, hypotheses are predictions about a presumed relationship between an independent variable (or variables) and its effect on a dependent variable (Gijlers & de Jong, 2005). Therefore, in order to formulate a hypothesis, the dependent and independent variable must first be identified by observing a phenomenon (Pedaste et al., 2015). Moreover, hypotheses are formulated as testable conclusions to inform the subsequent research process (Kremer et al., 2019).

Planning and conducting an experiment

Planning and conducting an experiment is done to answer the stated research question and test the generated hypotheses (Arnold et al., 2018). Setting up an experiment includes choosing the object of investigation and specifying the sample by determining the characteristics to be measured (Pedaste et al., 2012). This inquiry phase also includes determining the time periods, number of measurements and repetitions to be conducted (Germann et al., 1996). To ensure that the experimental results are comparable, test and control conditions (Germann et al., 1996) or a series of measurements need to be included. After methods and procedures are logically outlined, proper measurement equipment needs to be set up, safety precautions followed, and ultimately a sufficient number of trials to validate the results must be considered (Chang et al., 2011). When carrying out the experiment, it is crucial to manipulate variables in a scientific manner: whilst the independent variable is systematically varied to investigate its potential effect on the dependent variable, all other potential variables need to be held constant or ‘controlled’ across experimental conditions. This principle is often referred to as the control-of-variables strategy (Chen & Klahr, 1999). Furthermore, the experiment is recorded in the form of a laboratory report, documenting how the experiment was planned and conducted (Garcia-Mila et al., 2011).

Analysing data and drawing conclusions

The gathered data is analysed to establish evidence and build a link between the evidence and the conclusion through logical thinking, ultimately forming a model or an explanation (Chang et al., 2011). A first step in this process is to present the data gathered during the experiment in a clear manner by converting it into visualisations such as tables or graphs (Croner, 2003). Furthermore, the data quality is examined to assess certainty and limitations before determining to what extent the data supports the hypothesised relations (Millar & Lubben, 1996). Only then, it is possible to synthesise the results correctly and find meaningful patterns. Afterwards, the findings can be placed in relation to the previously stated research questions in order to evaluate the hypotheses regarding the causal relationship between the dependent and independent variables. Drawing inferences about causal hypotheses can contribute to explaining the cause of a phenomenon (Klayman & Ha, 1989). The data analysis also often leads to new research questions or modifications to the original hypotheses, which must then be tested again (Arnold et al., 2014), meaning that the inquiry cycle might begin anew.

3. Research goals

The sustained attention paid to IBL and experimentation in science education research over the past decades is reflected in two prior narrative reviews on a similar topic (De Jong & van Joolingen, 1998; Zimmerman, 2007). Whilst both have experimentation as their area of focus, their areas of application differ: De Jong and van Joolingen’s review (De Jong & van Joolingen, 1998) sought to determine the effectiveness and efficiency of discovery learning in simulation environments and identify problems learners encounter during experimentation in this specific kind of learning setting. Zimmerman’s (2007) literature review sought to describe the development of scientific thinking and scientific inquiry, mainly in elementary and middle school, including experimental design, evaluating evidence and drawing inferences. Neither of the earlier reviews specifically and solely focuses on challenges learners face when experimenting, but both have overlapping content in this regard. Another reason for completing a new review is that the inquiry phase ‘state research question’ was not included in either of the previous reviews. We therefore conducted a new systematic review with three goals in mind. First, given the different priorities of the previous narrative reviews and when they were carried out (15 and 24 years ago), the current systematic review aims to not only provide a more up to date overview but also expand and deepen knowledge of learners’ challenges during experimentation, making it the primary focus of the review. Second, the present systematic review analyses issues of both theoretical (conceptualisation of challenges related to specific inquiry phases) and practical relevance (e.g., information about assessment type, context domain), resulting in a systematical overview that captures the challenging components of each of the aforementioned inquiry phases during experimentation. Third, the results should be valuable for future research, as they can reveal current areas of emphasis and outstanding desiderata to inform the development of effective learning interventions and scaffolds. Likewise, the findings can be valuable for practical and conceptual work in schools, as they should provide detailed insights into IBL processes and experimentation in particular, which can be used to conceptualise effective lessons and tailored scaffolds to support learners.

4. Method

For the reasons explained above, we conducted a new systematic literature review by searching four databases representing different disciplines of interest (e.g., education, psychology, social sciences), the two previous literature reviews (De Jong & van Joolingen, 1998; Zimmerman, 2007) and analysing the resulting data set. As systematic reviews aim to make sense of large quantities of information by identifying salient themes and gaps in existing knowledge (Littell et al., 2008), we used Petticrew and Roberts (2006) method of review for the social sciences to systematically review the literature on learners’ challenges in understanding and performing experiments. Thus, we documented the research process in detail, including the definition of keywords, the selection of papers for analysis, the definition of inclusion criteria to refine the search, and the analysis of the data itself (see, Figure 1), which ensures transparency and replicability, as illustrated in other reviews (e.g., Rönnebeck et al., 2016). All publications included in the data analysis were read several times by the first and second authors and reviewed with respect to the inclusion criteria. Finally, descriptions of the reported challenges were extracted from the original studies. The identified challenges and their assignment to the different inquiry phases was verified via expert ratings and their interrater reliability. Study properties were coded in accordance with a coding scheme; interrater reliability was calculated here as well.

Figure 1. Flowchart of the search procedure and data analysis used for the systematic literature review, following PRISMA guidelines (Page et al., 2021) .

4.1 Search procedure

Initially, all studies analysed by De Jong and van Joolingen (1998; n = 21) and Zimmerman (2007; n = 26) describing learners’ challenges during experimentation were included in our sample of potentially relevant studies. Three duplicates were excluded. Another publication (from De Jong & van Joolingen, 1998) was excluded because the study was not available. The remaining 43 studies were added to a dataset. For the database search, we selected databases representing different disciplines that address IBL, science education and psychology: Education Resource Centre (ERIC) by the US Department of Education (educational science), Fachportal Pädagogik (a German educational science database) by the Leibniz Institute for Research and Information in Education, Google Scholar (interdisciplinary, including the educational, social, and natural sciences, humanities and psychology) and Web of Science by Thomson Reuters (social and natural sciences, humanities). To obtain relevant articles, the following criteria were used to formalise the search procedure: (1) relevant search terms (keywords) were used; (2) the publications must have been peer reviewed; (3) the publications had to be journal articles; (4) the articles had to have been published after 1998 (earliest option in the database (ERIC)). Keywords for the database search were defined in order to include all potentially relevant articles for the literature review. Therefore, in choosing keywords, we focused both on broader terms related to inquiry and experimentation (e.g., ‘experimentation’ or ‘scientific experiment’) as well as more specific terms describing the different inquiry phases (e.g., ‘planning an experiment’) or combinations of these (e.g., ‘stating research question’ AND ‘experiment’). The keywords chosen for the literature search were: scientific experiment, scientific investigation, scientific inquiry, experiment, state research questions, generate hypotheses, plan an experiment, conduct an experiment, data analysis, draw conclusions, control of variables strategy, confounded experiments and variations of these. The queries were altered or translated depending on the database language.

4.2 Data analysis

After checking titles and abstracts, we found 325 studies that fit our search procedure criteria and added these to the database. Next, all peer-reviewed full-text articles, totalling 368, (including 43 studies by De Jong and van Joolingen (1998) and Zimmerman (2007)) were examined in order to select eligible articles based on the following inclusion criteria:

1. The article was related to the diagnosis, development or promotion of learners’ understanding and/or ability to perform experiments.

2. The article reported original research findings regarding challenges during experimentation. Studies describing only participants’ general learning performance or the effects of intervention studies were excluded, as they do not fall within the objectives of this literature review. A reported finding was noted as a challenge of interest for this work if it compromises at least one of the inquiry phases mentioned above (see, Section 2).

3. At least one of the described challenges could be related to one or more of the four inquiry phases used to conceptualise this paper (see, Section 2).

4. The article presented empirical data. Theoretical papers or descriptions of activities were excluded.

5. The study participants were elementary and/or secondary school students or enrolled in post-secondary education (e.g., university students).

6. The participants were learners without learning disabilities.

Of the 368 articles assessed for eligibility, 67 articles fulfilled all of the inclusion criteria, including 12 articles analysed by De Jong and van Joolingen (1998), 18 analysed by Zimmerman (2007), and three articles that were part of both reviews. Articles presented in the literature reviews by De Jong and van Joolingen (1998) and Zimmerman (2007) were integrated into the analysis and marked as such. A summary of the study selection procedure is presented in Figure 1.

All articles included in the analysis were read several times and reviewed with respect to the reported challenges. Afterwards, descriptions of the challenges mentioned in the original articles were extracted. Overall, 52 challenges were identified. The identified challenges and their assignment to the different inquiry phases were then verified via expert ratings. The selected experts are all characterised by considerable experience in experimentation and/or IBL (as both practitioners and researchers) and stem from different disciplines of science education (biology, chemistry, and physics education) to represent the range of approaches to IBL and experimentation in science education. In addition, the goal was to achieve a compilation of international experts that would lead to an assessment that is as broadly applicable as possible, for example, not tied to a specific national educational framework. Following these selection criteria, 17 international experts (from Austria, Cyprus, Finland, Germany, and Switzerland) with large experience in the field of experimentation and/or IBL stemming from various fields of science education were asked to assign the 52 identified challenges to the four inquiry phases. The experts achieved a mean agreement of 87% (median of Krippendorff’s alpha = 0.85). Based on the experts’ modal response each challenge was then assigned to one of the four inquiry phases. Acknowledging, however, that there are not always sharp distinctions and these inquiry phases are often very closely related in practice (NRC, 2013), multiple responses by individual experts for each challenge were accepted. In the case of low interrater reliability (Krippendorff’s alpha < 0.60), a discussion between the first and second author took place. In such cases, the expert response patterns (see Appendix) and experts’ comments were taken into consideration to ensure the consistent application of the inclusion criteria. Based on this process, it was determined that eight challenges with low interrater reliability could be related to at least two inquiry phases; they were therefore assigned to the category ‘multiple inquiry phases’. Another eight challenges were excluded because the experts did not classify them as a challenge or the description (definition) seemed unclear. As a result, one study from Zimmerman’s review was excluded because none of its identified challenges made the final cut (n = 66). In total, 43 challenges could be assigned to the phases of stating research questions, generating hypotheses, planning and conducting an experiment, analysing data and drawing conclusions, or multiple inquiry phases. Based on the results of the expert ratings and authors’ discussions, eleven of the 43 identified challenges were classified as specific learners’ approaches (marked with an asterisk in the results section). In line with Baur (2018, 2021) these approaches are understood as procedures that do not contradict the experimentation process or compromise a particular inquiry phase or the experimental results in general. However, they provide the opportunity to make the experimental process more difficult or could lead to complications in the long run. Hence, specific learners’ approaches do not represent errors, but may impede the further development of competencies for experimentation. An example of a specific learners’ approach categorised as such by the experts is ‘learners working without a hypothesis’ (H2*; e.g., Dunbar & Klahr, 1989). Although this approach comprises core activities falling within the inquiry phase of generating hypotheses, it is still possible to conduct an experiment without stating a hypothesis and does not automatically compromise the outcome of the experiment per se. Nevertheless, not generating hypotheses makes it hard to coordinate theory (hypotheses) with the experimental results in order to solve the problem, which is why in educational settings attention needs to be drawn to this kind of learner approach.

The 66 studies reporting on learners’ challenges during experimentation were further analysed by the authors with regard to assessment type(s), context domain(s), sample size(s), grade level(s) and target group(s). The first author and a trained research assistant both coded 14 studies, or about 20% of the total. They reached a mean level of agreement of 97%, with a median Cohen’s Kappa of 1.0.

5. Results

The review of 368 articles led to 66 incorporated studies and 43 challenges learners encounter during the respective inquiry phases (see, Figure 2). Eleven of the 43 identified challenges are classified as specific learners’ approaches (marked with an asterisk; see, Section 4.2). The analysed studies were published between 1960 and 2018. The main target populations were K-12 students (n = 43) in secondary schools (n = 14), elementary schools (n = 15) or other educational institutions (n = 14), as well as university students (n = 19). Three studies compared school or university students with non-student populations. The sample sizes varied, with studies ranging from 4 to 1006 participants; about half of studies had 50 or fewer participants. All studies were published in English except for four studies that were only available in German. The analysis results are presented in five sections, relating to each of the four inquiry phases as well as challenges related to multiple inquiry phases. When contextualising the results, it is important to keep in mind that studies can address multiple target groups and context domains and include different types of assessments.

Figure 2. Number of challenges during the inquiry phases and number of studies reporting each challenge (studies can report more than one challenge).

5.1 Stating research questions

Five articles addressed two challenges learners can have when stating research questions (Q1 – Q2*; see, Figure 2 and Table 1). All five studies were part of the database search conducted for the current review. No studies included in De Jong and van Joolingen (1998) or Zimmerman’s (2007) literature review addressed challenges during the inquiry phase of stating research questions. Four of the five studies reported that learners do not state causal research questions (Q1) and therefore ask more factual type questions. Factual questions cannot be investigated in an experiment, as experiments generally investigate potential causal relationships between variables. An example of a factual question would be: ‘How is a heart constructed?’ or ‘What is in this lotion?’. The sample sizes of the studies in this category ranged from 53 to 181 participants. They all examined secondary school students in grades 7, 10 and 12 and conducted paper-pencil tests. Whilst Cuccio-Schirripa and Steiner (2000) examined several context domains (biology, earth and space science and physics), the other three studies (Hofstein et al., 2005; Neber & Anton, 2008a, 2008b) used phenomena from the discipline of chemistry. The second challenge identified was reported by Hofstein et al. (2004), who noted that learners state questions that are qualitative in nature (Q2*). An example of a qualitative type question would be: ‘Does a plant produce oxygen with the help of sunlight?’. However, while Hofstein et al. (2004) consider qualitative questions to be lower-level research questions, qualitative questions are not necessarily unscientific or untestable, as experimentation can also investigate qualitative relationships. Particularly, if little is known about the context being investigated, an exploratory approach can provide valuable insights and information about a system. However, if learners only state research questions that are qualitative in nature, this could reflect that they have difficulty proposing quantitative and therefore higher-level research questions (Hofstein et al., 2004). An example of a quantitative question would be: ‘What effect does the intensity of sunlight have on the production of oxygen in a plant?’. Ideally, research goal and type of research question should match. Hofstein et al. (2004) observed this challenge when examining 25 groups in grades 11 and 12 of secondary school. They assessed the inquiry phases by analysing laboratory reports produced during experimentation and by observing learners in experimental settings in the context domain of chemistry.

Table 1. Overview of empirical studies reporting about learners’ challenges when stating research questions during experimentation and selected properties.

Stating research questions
Challenge		Reference(s)	Review-source^a	Sample size(s)	Grade	Target group(s)^b	Assessment(s)^c	Context domain(s)^d	Covered challenge(s)^e
Q1	Not stating causal research questions	Cuccio-Schirripa and Steiner (2000)	CR	181	7	S	PP	B, ES, P	Q1
		Hofstein et al. (2005)	CR	111	12	S	PP	C	Q1
		Neber and Anton (2008a)	CR	58	10	S	PP	C	Q1
		Neber and Anton (2008b)	CR	53	10	S	PP	C	Q1
Q2*	Stating questions that are qualitative in nature	Hofstein et al. (2004)	CR	25 (groups)	11, 12	S	LP, OB	C	Q2

Note. Learners’ specific approaches are marked with an asterisk, as they are not necessarily comprising the inquiry phase or an experiment in general, but need to be reflected in the further learning process. ^aReview source: ‘CR’ current review; ^btarget group(s): ‘S’ secondary school students; ^cassessment(s): ‘LP’ analysis of laboratory reports produced during experimentation, ‘OB’ observation of learners in experimental settings; ‘PP’ paper-pencil-test; ^dcontext domain(s): ‘B’ biology (including life sciences), ‘C’ chemistry (including natural and processed materials), ‘ES’ earth and space science (including geography), ‘P’ physics (including energy and change); ^dcovered challenge(s): overview of all challenges reported by the study.

5.2 Generating hypotheses

A total of 16 studies reported seven challenges learners encounter when generating hypotheses (H1 – H7*; see, Figure 2 and Table 2). Seven studies originate from the current review, one from De Jong and van Joolingen (1998), seven from Zimmerman (2007), and one from both literature reviews. Sample sizes ranged from 6 to 498 participants, and grade levels ranged from third to eighth grade. Ten studies took place at the school level, meaning in secondary (n = 4) and elementary school (n = 4) as well as not further specified school levels (n = 2). Four studies were conducted with university students, whilst two studies compared children with university students and adults. Four studies describe challenges that children and even university students still have when generating hypotheses: they do not know what a hypothesis is (H1; Njoo & de Jong, 1993), they work without a hypothesis (H2*; Park, 2006), they operate abductively (H3; García-Carmona et al., 2017), and they only generate hypotheses that seem plausible to them (H6*; Penner & Klahr, 1996b). Two studies compared different target groups, such as adults and secondary school students (Klahr et al., 1993; Schauble, 1996). These studies applied various assessment types, including paper-pencil tests, in which, for example, elementary school students had to predict the effects of differences in the speed of a swing, which presumably relates to simple pendulum motion (Kwon et al., 2006). Also video-recorded laboratory sessions were used as assessment tools, where two practical experiments with a similar structure, a covariation and a non-covariation case, were conducted (Kanari & Millar, 2004). One of these was also a pendulum task, in which students were asked to investigate the relation between the length and weight of a pendulum and its swing time. In the other experiment, students were asked to investigate the relation between the mass of a small box and the area of its bottom surface on the one hand and the force needed to pull it along a level surface on the other. The students could choose among a predetermined set of hypotheses to investigate. Interviews were conducted afterwards, which suggested that students only selected hypotheses predicting the independent variable to have an influence on the dependent variable (H4*), which is also supported by other studies (Klahr et al., 2007; Schauble, 1990, 1996). Overall, the following assessment types were used: audio recordings of laboratory sessions (n = 2), computer simulations (n = 5), interviews (n = 8), laboratory reports produced during experimentation (n = 8), observations of learners in experimental settings (n = 3), paper-pencil tests (n = 7) and video recordings of laboratory sessions (n = 4). Not surprisingly, most test instruments concerned specific science contexts: physics (n = 7), biology (n = 2), logic (n = 2), the natural sciences (n = 1), everyday life (n = 1), earth and space science (n = 1), and mechanical engineering (n = 1). One study did not explicitly mention a specific context domain. Most studies (n = 6) reported that learners only generate hypotheses that seem plausible to them based on expected data to avoid choosing hypotheses with a high chance of being rejected (also known as fear of rejection; H6*; e.g., Echevarria, 2003). In such cases, preconceptions or beliefs have a strong influence on what kinds of hypotheses are formulated. When this phenomenon arises in a single experiment, it does not necessarily mean that the hypothesis is not testable or the outcome of the experiment is compromised, but it could affect whether learners seek out further evidence or only confirmatory evidence. The same is true of learners’ working without a hypothesis (H2*; e.g., Darus & Saat, 2014): learners can still experimentally investigate the relationship between at least two variables in such a case, but it could also indicate a lack of awareness that hypotheses or predictions are valuable and often part of the experimental process, as they give a reason to tie predictions back to theory or previous experimental results. Likewise, only selecting hypotheses predicting the independent variable will have an influence on the dependent variable and therefore not selecting hypotheses that predict no effect (H4*; Kanari & Millar, 2004), could still lead to a successful experimental outcome in any individual case. However, it might also indicate that one presumes an effect always exists or that a relationship just needs to be proven by an experiment, which could affect reasoning strategies further along in the experimental process. In addition, the following specific learner approaches could be identified as well: generating hypotheses that include more than one independent variable (H5*; e.g., Valanides et al., 2013) and learners not being aware that more than one hypothesis is possible (H7*; e.g., Klahr et al., 1993).

Table 2. Overview of empirical studies reporting about learners’ challenges generating hypotheses during experimentation and selected properties.

Generating hypotheses
	Challenge	Reference(s)	Review-source^a	Sample size(s)	Grade^b	Target group(s)^c	Assessment(s)^d	Context domain(s)^e	Covered challenge(s)
H1	Not knowing what a hypothesis is	Darus and Saat (2014)	CR	6	3	E	INTV, LP, OB	P	H1, H2, M2
		Njoo and de Jong (1993)	JJ	17 (study 1), 91 (study 2)	-*	U	A, CS, LP, OB (study 1), CS, LP, PP (study 2)	ME	H1, PC7
		Zhai et al. (2014)	CR	161	4	E	INTV, OB, PP, V	NS	H1
H2*	Working without a hypothesis	Darus and Saat (2014)	CR	6	3	E	INTV, LP, OB	P	H1, H2, M2
		Park (2006)	CR	6	-*	U	INTV	P	H2, H3, DC9
		Valanides et al. (2013)	CR	498	4, 6	E	INTV, LP	L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2
H3	Operating abductive	Kwon et al. (2006)	CR	290 (study 1), 34 (study 2)	5	E	PP	P	H3
		García-Carmona et al. (2017)	CR	66	-*	U	PP	P	H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Park (2006)	CR	6	-*	U	INTV	P	H2, H3, DC9
H4*	Selecting only hypotheses that predict an influence of the independent variable on the dependent variable	Kanari and Millar (2004)	Z	60	n/a	n/a	INTV, V	P	H4, PC10, DC3, DC6, DC7, M1, M6
		Klahr et al. (2007)	Z	56	7, 8	S	PP	EL	H4
		Schauble (1990)	Z	22	5, 6	n/a	CS, INTV, LP	n/a	H4, H6, PC6, PC7, DC8, DC13, M9
		Schauble (1996)	Z	20	5, 6	A, S	INTV, LP	P	H4, H6, PC3, DC7, DC9, DC12, DC13, M1
H5*	Generating hypotheses that include more than one independent variable	Baur (2018)	CR	18	5, 6	S	LP, V	B	H5
		Valanides et al. (2013)	CR	498	4, 6	E	INTV, LP	L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2
H6*	Only generating hypotheses that are plausible and predict expected data (fear of rejection)	Echevarria (2003)	Z	44–48	7	S	A, CS, PP, V	B	H6
		Klahr et al. (1993)	JJ; Z	64	3, 6	n/a, U	CS, LP	ES	H6, H7, PC7, DC8, DC13
		Penner and Klahr (1996b)	Z	120 (study 1), 59 (study 2)	-*	U	PP	L	H6, DC7
		Schauble (1990)	Z	22	5, 6	n/a	CS, INTV, LP	EL	H4, H6, PC6, PC7, DC8, DC13, M9
		Schauble (1996)	Z	20	5, 6	A, S	INTV, LP	P	H4, H6, PC3, DC7, DC9, DC12, DC13, M1
		Zimmerman et al. (2003)	Z	14	6	S	INTV	P	H6, DC6
H7*	Not aware that more than one hypothesis is possible	Klahr et al. (1993)	Z	64	3, 6	n/a, U	CS, LP	ES	H6, H7, PC7, DC8, DC13
		Valanides et al. (2013)	CR	498	4, 6	E	INTV, LP	L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2

Note. Learners’ specific approaches are marked with an asterisk, as they are not necessarily comprising the inquiry phase or an experiment in general, but need to be reflected in the further learning process. ^aReview source: ‘CR’ current review, ‘JJ’ De Jong and van Joolingen (1998), ‘Z’ Zimmerman (2007); ^bgrade: if marked with a -* no grade level can be depicted, because the study did not take place at school level; ^ctarget group(s): ‘E’ elementary school students, ‘S’ secondary school students, ‘U’ university students, ‘A’ adults, ‘n/a’ students target group not further specified; ^dassessment(s): ‘A’ audio recording of laboratory session(s); ‘CS’ computer simulation; ‘INTV’ interviews; ‘LP’ analysis of laboratory reports produced during experimentation; ‘OB’ observation of learners in experimental settings; ‘PP’ paper-pencil-test, ‘V’ video recording of laboratory session(s); ^econtext domain(s): ‘B’ biology (including life sciences), ‘C’ chemistry (including natural and processed materials), ‘EL’ everyday life”, ‘ES’ earth and space science (including geography), ‘L’ logic (including rule discovery), ‘ME’ mechanical engineering, ‘NS’ natural sciences (no specification), ‘P’ physics (including energy and change), ‘n/a’ context domain not clearly given; ^dcovered challenge(s): overview of all challenges reported by the study.

5.3 Planning and conducting an experiment

Overall, out of the 34 studies addressing challenges while planning and conducting an experiment, 16 studies originate from the current review, seven are part of De Jong and van Joolingen (1998) literature review, ten stem from Zimmerman’s (2007) literature review, and a single study was reported by both of them. At 34, most of the studies analysed in this review relate to challenges in the inquiry phase of planning and conducting an experiment, although only 12 unique challenges in this inquiry phase were identified (P1 – P12; see, Figure 2 and Table 3). This is because the present literature intensively describes two challenges: learners not controlling variables (PC3; n = 17) and the engineering approach (PC7; n = 11). An analysis of the studies’ target groups shows that learners struggle with controlling variables from early elementary school (e.g., Chen & Klahr, 1999) up to the university level (e.g., Dasgupta et al., 2016) and into adulthood (e.g., Shute & Glaser, 1990). This challenge has been assessed with a range of different assessment types and seems to be a domain-general difficulty. The other well-documented challenge from early childhood on (third grade), through to secondary school (5^th and 6^th grade) and into university concerns the engineering approach (PC7), in which learners create an effect as an outcome of an experiment instead of investigating a presumed relationship between variables (e.g., Carey et al., 1989; García-Carmona et al., 2017; Schauble et al., 1995). In this sense, the engineering approach has the (implicit) goal of creating a desired outcome rather than investigating the origin of an observed phenomenon. Because the scientific and engineering approach have different objectives, they follow different processes: The scientific approach takes for example, previous experimental results into account when making subsequent predictions, or considers certain experimental variations as more informative in a particular situation than others, whilst the engineering approach follows the goal of a creativity-based engineering design process (e.g., Schauble et al., 1991b). This concept is discussed in eleven studies, mostly among school students (n = 5) but also at the university level (n = 3) or in comparisons of different education levels (n = 3).

Table 3. Overview of empirical studies reporting about learners’ challenges planning and conducting an experiment and selected properties.

Planning and conducting an experiment
	Challenge	Reference(s)	Review-source^a	Sample size(s)	Grade^b	Target group(s)^c	Assessment(s)^d	Context domain(s)^e	Covered challenge(s)
PC1	Changing conditions throughout the ongoing experiment	Baur (2018)	CR	18	5, 6	S	LP, V	B	PC1, PC5, PC8, PC12, DC10, M3, M4
PC2	Limiting the experiment on just a few independent variables	Hammann et al. (2010)	CR	1006	5, 6	S	PP	B	PC2, PC5, PC7, DC7, M4
		Wu & Wu (2011)	CR	68	5	E	INTV, LP, PP, V	P	PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
PC3	Not controlling variables	Arnold et al. (2014)	CR	96 (study 1), 12 (study 2)	12 (study 1), 11 (study 2)	n/a	OB, PP	B	PC3, PC9, PC10
		Chen and Klahr (1999)	CR	87	2, 4	E	INTV, PP, V	P	PC3
		Dasgupta et al. (2016)	CR	4	-*	U	INTV, PP	B	PC3, PC5, PC10, DC4, DC10, M2
		Erdosne Toth et al. (2000)	CR	87 (study 1), 77 (study 2)	2, 3, 4	E	INTV, LP, PP	P	PC3
		Garcia-Mila et al. (2011)	CR	34	6	E	INTV, LP	B	PC3, DC11, M5
		Van Joolingen and de Jong (1991)	JJ	31	-*	U	CS, PP	C	PC3
		Klahr and Nigam (2004)	Z	112	3, 4	E	INTV, PA	P	PC3
		Kuhn (2007)	CR	30	4	E	CS	ES	PC3
		Kuhn and Dean (2005)	CR	30	6	S	CS	ES	PC3, DC10
		Reimann (1991)	JJ	8	-*	U	A, CS	P	PC3
		Schauble (1996)	Z	20	5, 6	A, S	INTV, LP	P	H4, H6, PC3, DC7, DC9, DC12, DC13, M1
		Schauble et al. (1991a)	JJ	22	-*	U	A, CS, OB, LP	P	PC3
		Shute and Glaser (1990)	JJ	30 (study 1), 530 (study 2)	-*	A, U	PP, CS	ES	PC3
		Siegler and Liebert (1975)	Z	60	5, 8	n/a	PA, PP	L	PC3, M5
		Tairab (2015)	CR	128	8, 10, 12	n/a	PP	NS	PC3
		Valanides et al. (2013)	CR	498	4, 6	E	INTV, LP	L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2
		Wu & Wu (2011)	CR	68	5	E	INTV, LP, PP, V	P	PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
PC4	When testing positive and negative hypotheses different strategies are used	Croker and Buchanan (2011)	CR	144	n/a	E	INTV, PP	B	PC4
PC5	Working without a control condition	Baur (2018)	CR	18	5, 6	S	LP, V	B	PC1, PC5, PC8, PC12, DC10, M3, M4
		Dasgupta et al. (2016)	CR	4	-*	U	INTV, PP	B	PC3, PC5, PC10, DC4, DC10, M2
		Hammann et al. (2010)	CR	1006	5, 6	S	PP	B	PC2, PC5, PC7, DC7, M4
PC6	Producing desired experimental results	Kuhn et al. (1995)	Z	32	4	n/a, U	INTV	P, S	PC6, DC13
		Kuhn et al. (1992)	Z	12 (study 1), 20 (study 2)	4, (study 1), 5, 6 (study 2)	E (study 1), n/a (study 2)	INTV, LP	EL	PC6, PC11, DC8, DC13
		Schauble (1990)	Z	22	5, 6	n/a	CS, INTV, LP	EL	H4, H6, PC6, PC7, DC8, DC13, M9
PC7	Engineering approach	Carey et al. (1989)	Z	27	7	E/S	INTV	NS	PC7, M5
		García-Carmona et al. (2017)	CR	66	-*	U	PP	P	H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Hammann et al. (2010)	CR	66	-*	U	PP	P	H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Klahr et al. (1993)	Z	64	3, 6	n/a, U	CS, LP	ES	H6, H7, PC7, DC8, DC13
		Njoo and de Jong (1993)	JJ	17 (study 1), 91 (study 2)	-*	U	A, CS, LP, OB (study 1), CS, LP, PP (study 2)	ME	H1, PC7
		Schauble (1990)	Z	22	5, 6	n/a	CS, INTV, LP	EL	H4, H6, PC6, PC7, DC8, DC13, M9
		Schauble et al. (1991b)	JJ, Z	16	5, 6	S	A, INTV, LP	P	PC7
		Schauble et al. (1995)	JJ	21 (study 1); 35 (study 2)	6	S	INTV	ES, P	PC7
		Tschirgi (1980)	Z	48	2, 4, 6	n/a, U	PP	EL	PC7, M9
		Valanides et al. (2013)	CR	498	4, 6	E	INTV, LP	L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2
		White (1993)	JJ	79, 41, 45 (study 1), n/a (study 2)	6, 11 (study 1), 12 (study 2)	S	A, CS, INTV, OB, PP (study 1), CS, PP (study 2)	P (study 1), NS (study 2)	PC7
PC8	Handling material and substances indiscriminately	Baur (2018)	CR	18	5, 6	S	LP, V	B	PC1, PC5, PC8, PC12, DC10, M3, M4
PC9	Lack of determination of test times	Arnold et al. (2014)	CR	96 (study 1), 12 (study 2)	12 (study 1), 11 (study 2)	n/a	OB, PP	B	PC3, PC9, PC10
		Duggan and Gott (2000)	CR	179	-*	U	INTV, PP	NS	PC9
		García-Carmona et al. (2017)	CR	66	-*	U	PP	P	H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
PC10	Lack of repetition of measurement	Arnold et al. (2014)	CR	96 (study 1), 12 (study 2)	12 (study 1), 11 (study 2)	n/a	OB, PP	B	PC3, PC9, PC10
		Dasgupta et al. (2016)	CR	4	-*	U	INTV, PP	B	PC3, PC5, PC10, DC4, DC10, M2
		García-Carmona et al. (2017)	CR	66	-*	U	PP	P	H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Kanari and Millar (2004)	Z	60	n/a	n/a	INTV, V	NS	PC10, DC3, DC6, DC7, M1, M6
		Wu & Wu (2011)	CR	68	5	E	INTV, LP, PP, V	P	PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
PC11	Planning and conducting the same trials several times (not with the aim of repeated measurements)	Kuhn et al. (1992)	JJ	12 (study 1), 20 (study 2)	4 (study 1), 5, 6 (study 2)	E (study 1), n/a (study 2)	INTV, LP	EL	PC6, PC11, DC8, DC13
		Schrempp and Sodian (1999)	CR	44	2	E, U	INTV	EL	PC11
		Wu & Wu (2011)	CR	68	5	E	INTV, LP, PP, V	P	PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
PC12	Do not involve all test trials while conducting their experiment	Baur (2018)	CR	18	5, 6	S	LP, V	B	PC1, PC5, PC8, PC12, DC10, M3, M4
		Valanides et al. (2013)	CR	498	4, 6	E	INTV, LP	L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2

Note. ^aReview source: ‘CR’ current review, ‘JJ’ De Jong and van Joolingen (1998), ‘Z’ Zimmerman (2007); ^bgrade: if marked with a -* no grade level can be depicted, because the study did not take place at school level; ^ctarget group(s): ‘E’ elementary school students, ‘S’ secondary school students, ‘U’ university students, ‘A’ adults, ‘n/a’ students target group not further specified; ^dassessment(s): ‘A’ audio recordings of laboratory session(s), ‘CS’ computer simulation, ‘INTV’ interviews, ‘LP’ analysis of laboratory reports produced during experimentation, ‘OB’ observation of learners in experimental settings, ‘PA’ performance assessment test; ‘PP’ paper-pencil-test, ‘V’ video recordings of laboratory session(s); ^econtext domain(s): ‘B’ biology (including life sciences), ‘C’ chemistry (including natural and processed materials), ‘EL’ everyday life”, ‘ES’ earth and space science (including geography), ‘ME’ mechanical engineering, ‘NS’ natural sciences (no specification), ‘P’ physics (including energy and change); ‘S’ social sciences, ^dcovered challenge(s): overview of all challenges reported by the study.

Overall, the sample sizes ranged from 4 to 1006 participants, and the students participating in the studies ranged from second up to twelve graders. 21 studies took place among K-12 school students, seven studies involved university students. There were also five studies comparing school students’ ability to plan and conduct an experiment with that of adults and university students (Klahr et al., 1993; Kuhn et al., 1995; Schauble, 1996; Schrempp & Sodian, 1999; Tschirgi, 1980). A single study compared university students and adults, who were not enrolled at university (Shute & Glaser, 1990). Similarly to the previous inquiry phases, most studies identified challenges when learners plan and conduct experiments using paper-pencil tests (n = 17). Even though this inquiry phase has more procedural components than the others, only two studies used performance assessments in which learners were actually required to practically conduct an experiment (Klahr & Nigam, 2004; Siegler & Liebert, 1975). In a similar vein, one study focused on students’ cognitive and manual abilities in video-recorded laboratory sessions (Baur, 2018). It was found that students handle laboratory materials and substances indiscriminately, as students did not consider the comparability of substances or materials (e.g., use of test tubes with different volumes) across different conditions (PC8).

Altogether, participants were interviewed in 18 studies, while 12 studies used computer simulations. Laboratory reports from experiments were analysed in eleven studies, followed by audio recordings (n = 5), videos of laboratory sessions (n = 4),  and observations of learners in experimental settings (n = 4). The context domains were mostly physics (n = 12), biology (n = 6), the natural sciences (n = 5), earth and space sciences (n = 5) and everyday life contexts (n = 4). The other context domains were chemistry (n = 1), logic (n = 2), mechanical engineering (n = 1) and the social sciences (n = 1).

5.4 Data analysis and drawing conclusions

Fourteen challenges with data analysis and drawing conclusions were reported by 31 studies (DC1 – DC14*; see, Figure 2 and Table 4).16 of the 31 studies originate from the current review, seven from De Jong and van Joolingen (1998) and eight from Zimmerman’s (2007) literature review. Three of the 14 challenges in this category were considered to be specific learner approaches: The first such approach is that learners become uncertain in drawing conclusions when confronted with variability in the measured data, such as that resulting from large sample sizes in repeated measurements (DC2*; Masnick & Morris, 2008). The second learner approach identified and labelled DC9* is similar in nature. In sum, the findings categorised as DC9* show that anomalous data lead learners to assume that there are flaws in the experimental procedure. For example, in the study by Schauble (1996), children and adults generated quantitative data by exploring tasks involving hydrodynamics and hydrostatics. Repeated experimental trials led to variation in the measured data. Because these were duplicate trials, learners were uncertain about what caused the differences in the data. In this case, students had to decide which differences were ‘real’ and which represented data variability. It could be shown that prior expectations about the experimental results strongly influenced learners’ interpretations. Different outcomes in duplicate trials were interpreted as indicating an effect if that effect was expected, but interpreted as measurement error if an effect was not expected (Schauble, 1996). On the one hand, this might be seen as a facet of confirmation bias (Wason, 1960); on the other hand, it could indicate that learners are willing to reflect on experimental procedures and therefore look for potential flaws, which is in line with the scientific principles underlying experimentation. The third specific approach, based on the expert ratings, is that learners do not connect their hypotheses to previous results of an experiment (DC14*). In this case, learners do not state hypotheses on the basis of data gathered and do not correct or adapt their hypotheses even if data contradicts it (Klahr & Dunbar, 1988). Overall, the sample sizes of studies reporting challenges with data analysis and drawing conclusions ranged from 4 to 1006 participants. Most studies took place at the school level (n = 19), but challenges with data analysis and drawing conclusions were also reported among university students (n = 6). Seven studies compared children to university students or adults regarding this inquiry phase and reported the challenges each group encountered (e.g., Aoki, 1991; Masnick & Morris, 2008). The students examined in these studies stemmed from second to eighth grade, with most in elementary school (n = 8). No studies presented challenges for children in higher grades. The assessment types used were mostly interviews (n = 16), followed by paper-pencil tests (n = 14), laboratory reports produced during experimentation (n = 10), computer simulations (n = 7), video recordings (n = 7), audio recordings of laboratory sessions (n = 3) and observations of experimental settings (n = 3). Not surprisingly, all of the studies investigated this inquiry phase with respect to specific context domains, namely physics (n = 10), followed by biology (n = 8) and logic (n = 4). The following context domains were examined less frequently: everyday life (n = 4), earth and space sciences (n = 2), the natural sciences (n = 2), chemistry (n = 1) and the social sciences (n = 1).

Table 4. Overview of empirical studies reporting about learners’ challenges analysing and drawing conclusions during experimentation and selected properties.

Data analysis and drawing conclusions
	Challenge	Reference(s)	Review-source^a	Sample size(s)		Grade^b		Target group(s)^c			Assessment(s)^d		Context domain(s)^e		Covered challenge(s)
DC1	Confusing between observation and interpretation of data	Boaventura et al. (2013)	CR	135		4		E			A, OB, PP, V		B		DC1, DC10
		García-Carmona et al. (2017)	CR	66		-*		U			PP		P		H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Wu & Wu (20112)	CR	68		5		E			INTV, LP, PP, V		P		PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
DC2*	Variability in measured data leads to uncertainty	Masnick & Morris (2008)	CR	133 (study 1), 101 (study 2)		3, 6 (study 1), 6 (study 2)		n/a, U (study 1), n/a, U (study 2)			INTV, PP		EL		DC2
DC3	Not calculating the arithmetic mean	Kanari and Millar (2004)	Z	60		n/a		n/a			INTV, V		P		H4, PC10, DC3, DC6, DC7, M1, M6
		Masnick and Klahr (2003)	CR	49		2, 4		E			INTV, PP, V		P		DC3
		Wu & Wu (2011)	CR	68		5		E			INTV, LP, PP, V		P		PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
DC4	Diagrams that are not suitable	Croner (2003)	CR	60		7		S			INTV, OB, LP		P		DC4, M7
		Dasgupta et al. (2016)	CR	4		-*		U			INTV, PP		B		PC3, PC5, PC10, DC4, DC10, M2
		Durmaz and Mutlu (2017)	CR	43		7		E			LP, PP		NS		DC4
DC5	Difficulties when reading graphs	Mokros and Tinker (1987)	JJ	125		7, 8		S			CS, INTV, OB, PP		NS		DC5
DC6	Drawing conclusions when the result shows no covariation is difficult	Aoki (1991)	Z	152		4, 5, 6		n/a, U			PP		P		DC6
		Kanari and Millar (2004)	Z	60		n/a		n/a			INTV, V		P		H4, PC10, DC3, DC6, DC7, M1, M6
		Valanides et al. (2013)	CR	498		4, 6		E			INTV, LP		L		H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2
		Zimmerman et al. (2003)	Z	14		6		S			INTV		P		H6, DC6
DC7	Confirmation bias	Dunbar (1993)	JJ	40		_*		U			A, CS		B		DC7, DC14
		Hammann et al. (2010)	CR	1006		5, 6		S			PP		B		PC2, PC5, PC7, DC7, M4
		García-Carmona et al. (2017)	CR	66		-*		U			PP		P		H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Kanari and Millar (2004)	Z	60	n/a		n/a				INTV, V		P		H4, PC10, DC3, DC6, DC7, M1, M6
		Penner and Klahr (1996b)	Z	120 (study 1), 59 (study 2)		-*	U					PP		L	H6, DC7
		Quinn and Alessi (1994)	JJ	179		n/a	n/a					CS		B	DC7
		Schauble (1996)	Z	20		5, 6	A, S					INTV, LP		P	H4, H6, PC3, DC7, DC9, DC12, DC13, M1
		Valanides et al. (2013)	CR	498		4, 6	E					INTV, LP		L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2
		Wason (1960)	JJ	29		-*	U					PP		L	DC7
		Wu & Wu (2011)	CR	68		5		E			INT, LP, PP, V		P		PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
DC8	Ignoring anomalous data	Amsel and Brock (1996)	Z	236		2, 3, 6, 7		A, E, U	INTV				B		DC8
		Greenhoot et al. (2004)	CR	211		-*		U	PP				EL, P		DC8
		Klahr et al. (1993)	JJ	64		3, 6		n/a, U	CS, LP				ES		H6, H7, PC7, DC8, DC13
		Kuhn et al. (1992)	JJ	12 (study 1), 20 (study 2)		4 (study 1), 5, 6 (study 2)		E (study 1), n/a (study 2)	INTV, LP				EL		PC6, PC11, DC8, DC13
		Wahser and Sumfleth (2008)	CR	172		7		S	PP, V				C		DC8, M4
		Schauble (1990)	Z	22		5, 6		n/a	CS, INTV, LP				EL		H4, H6, PC6, PC7, DC8, DC13, M9
DC9*	Anomalous data leads to the assumption that there are flaws in the experimental procedure	Park (2006)	CR	6		-*		U	INTV				P		H2, H3, DC9
		Schauble (1996)	Z	20		5, 6		A, S	INTV, LP				P		H4, H6, PC3, DC7, DC9, DC12, DC13, M1
		Wu & Wu (2011)	CR	68		5		E		INTV, LP, PP, V			P		PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
DC10	Drawing incorrect inferences	Baur (2018)	CR	18		5, 6		S	LP, V				B		PC1, PC5, PC8, PC12, DC10, M3, M4
		Boaventura et al. (2013)	CR	135		4		E	A, OB, PP, V				B		DC1, DC10
		Dasgupta et al. (2016)	CR	4		-*		U	INTV, PP				B		PC3, PC5, PC10, DC4, DC10, M2
		Kuhn and Dean (2005)	CR	30		6		S	CS				ES		PC3, DC10
DC11	Neglecting the protocol leads to incorrect inferences	García-Mila et al. (2011)	CR	34		6		E	INTV, LP				B		PC3, DC11, M5
DC12	Learners make no exclusion inferences	Schauble (1996)	Z	20		5, 6		A, S	INTV, LP				P		H4, H6, PC3, DC7, DC9, DC12, DC13, M1
DC13	Learners do not give up less advanced strategies when interpreting results	Klahr et al. (1993)	Z	64		3, 6		n/a, U	CS, LP				ES		H6, H7, PC7, DC8, DC13
		Kuhn et al. (1995)	Z	32		4		n/a, U	INTV				P, S		PC6, DC13
		Kuhn et al. (1992)	Z	12 (study 1), 20 (study 2)		4 (study 1), 5, 6 (study 2)		E (study 1), n/a (study 2)	INTV, LP				EL		PC6, PC11, DC8, DC13
		Schauble (1990)	Z	22		5, 6		n/a	CS, INTV, LP				EL		H4, H6, PC6, PC7, DC8, DC13, M9
		Schauble (1996)	Z	20		5, 6		A, S	INTV, LP				P		H4, H6, PC3, DC7, DC9, DC12, DC13, M1
DC14*	Not connecting hypotheses to previous results	Dunbar (1993)	JJ	40		-*		U	A, CS				B		DC7, DC14
		García-Carmona et al. (2017)	CR	66		-*		U	PP				P		H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Klahr and Dunbar (1988)	JJ	20 (study 1), 10 (study 2)		-*		U	A, CS, V				L		DC14
		Valanides et al. (2013)	CR	498		4, 6		E	INTV, LP				L		H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2

Note. Learners’ specific approaches are marked with an asterisk, as they are not necessarily comprising the inquiry phase or an experiment in general, but need to be reflected in the further learning process. ^aReview source: ‘CR’ current review, ‘JJ’ De Jong and van Joolingen (1998), ‘Z’ Zimmerman (2007); ^bgrade: if marked with a -* no grade level can be depicted, because the study did not take place at school level; ^ctarget group(s): ‘E’ elementary school students, ‘S’ secondary school students, ‘U’ university students, ‘A’ adults, ‘n/a’ students target group not further specified; ^dassessment(s): ‘A’ audio recordings of laboratory session(s), ‘CS’ computer simulation, ‘INTV’ interviews, ‘LP’ analysis of laboratory reports produced during experimentation, ‘OB’ observation of learners in experimental settings, ‘PP’ paper-pencil-test, ‘V’ video recordings of laboratory session(s); ^econtext domain(s): ‘B’ biology (including life sciences), ‘C’ chemistry (including natural and processed materials), ‘EL’ everyday life”, ‘ES’ earth and space science (including geography), ‘L’ logic (including rule discovery), ‘MT’ maths, ‘NS’ natural sciences (no specification), ‘P’ physics (including energy and change), ‘S’ social sciences; ^dcovered challenge(s): overview of all challenges reported by the study.

5.5 Multiple inquiry phases

19 studies in total described the eight challenges assigned to this category (see, Figure 2 and Table 5). These varied in their degree of abstraction: from learners not being able to identify variables (M2), which is not only needed when generating hypotheses about potential causal relationships between variables, but is also important when planning and conducting an experiment; to learners not being aware of the concept of measurement error (M6). Two challenges in this category were labelled specific learner approaches, as they do not necessarily compromise the outcome of an experiment in all cases. The first of these is changing hypotheses during an ongoing experiment (M3*). Baur (2018) notes that students change their hypotheses during the experimental process before obtaining a result. After changing their hypothesis, however, students do not throw out the previously established experimental setting, but expand it with additional structures. Changing hypotheses without obtaining a result that disconfirms or confirms one’s earlier hypothesis can lead learners to overlook causality. In addition, broadening the set-up of an experiment that has already begun can make experiments unsystematic. The second specific learner approach is not constructing tables (M7*), which may help learners illustrate and present the data they have obtained, but it is not always a prerequisite for scientific experimentation, because not every experiment requires data to be organised in tables.

Table 5. Overview of empirical studies reporting about learners’ challenges related to multiple inquiry phases and selected properties.

Multiple inquiry phases
	Challenge	Reference(s)	Review-source^a	Sample size(s)	Grade^b	Target group(s)^c	Assessment(s)^d	Context domain(s)^e	Covered challenge(s)
M1	Not investigating rival hypotheses	Kanari and Millar (2004)	Z	60	n/a	n/a	INTV, V	P	H4, PC10, DC3, DC6, DC7, M1, M6
		Penner and Klahr (1996a)	Z	30	n/a	n/a	INTV, V	P	M1
		Schauble (1996)	Z	20	5, 6	A, S	INTV, LP	P	H4, H6, PC3, DC7, DC9, DC12, DC13, M1
M2	Not being able to identify variables	Darus and Saat (2014)	CR	6	3	E	INTV, OB, LP	P	H1, H2, M2
		Dasgupta et al. (2016)	CR	4	-*	U	INTV, PP	B	PC3, PC5, PC10, DC4, DC10, M2
		García-Carmona et al. (2017)	CR	66	-*	U	PP	P	H3, PC7, PC9, PC10, DC1, DC7, DC14, M2
		Ramnarain (2012)	CR	42	9	S	INTV, OB, V	P	M2
		Valanides et al. (2013)	CR	498	4, 6	E	A, INTV, LP	L	H2, H5, H7, PC3, PC7, PC12, DC6, DC7, DC14, M2
M3*	Changing hypotheses throughout the experiment	Baur (2018)	CR	18	5, 6	S	LP, V	B	PC1, PC5, PC8, PC12, DC10, M3, M4
M4	Trying out	Baur (2018)	CR	18	5, 6	S	LP, V	B	PC1, PC5, PC8, PC12, DC10, M3, M4
		Hammann et al. (2010)	CR	1006	5, 6	S	PP	B	PC2, PC5, PC7, DC7, M4
		Wahser and Sumfleth (2008)	CR	172	7	S	PP, V	C	DC8, M4
M5	Lack of documentation	Carey et al. (1989)	Z	27	7	E/S	INTV	NS	PC7, M5
		Garcia-Mila and Andersen (2007)	Z	31	4	n/a, U	INTV, LP	P, S	M5
		Garcia-Mila et al. (2011)	CR	34	6	E	INTV, LP	B	PC3, DC11, M5
		Siegler and Liebert (1975)	Z	60	5, 8	n/a	PA, PP	L	PC3, M5
M6	No awareness of measurement error	Kanari and Millar (2004)	Z	60	n/a	n/a	INTV, V	P	H4, PC10, DC3, DC6, DC7, M1, M6
		Wu & Wu (2011)	CR	68	5	E	INTV, LP, PP, V	P	PC2, PC3, PC10, PC11, DC1, DC3, DC7, DC9 M6
M7*	Not constructing tables	Croner (2003)	CR	60	7	S	INTV, OB, LP	P	DC4, M7
M8	Value of the results determines the seeking for evidence	Schauble (1990)	Z	22	5, 6	n/a	CS, INTV, LP	EL	H4, H6, PC6, PC7, DC8, DC13, M9
		Tschirgi (1980)	Z	48	2, 4, 6	n/a, U	PP	EL	PC7, M9

Note. Learners’ specific approaches are marked with an asterisk, as they are not necessarily comprising the inquiry phase or an experiment in general, but need to be reflected in the further learning process. ^aReview source: ‘CR’ current review, ‘Z’ Zimmerman (2007); ^bgrade: if marked with a -* no grade level can be depicted, because the study did not take place at school level; ^ctarget group(s): ‘E’ elementary school students, ‘S’ secondary school students, ‘U’ university students, ‘A’ adults, ‘n/a’ students target group not further specified; ^dassessment(s): ‘CS’ computer simulation; ‘INTV’ interviews; ‘LP’ analysis of laboratory reports produced during experimentation; ‘OB’ observation of learners in experimental settings, ‘PA’ performance assessment test, ‘PP’ paper-pencil-test, ‘V’ video recordings of laboratory session(s); ^econtext domain(s): ‘B’ biology (including life sciences), ‘EL’ everyday life”, ‘L’ logic (including rule discovery, ‘NS’ natural sciences (no specification), ‘P’ physics (including energy and change), ‘S’ social sciences; ^dcovered challenge(s): overview of all challenges reported by the study.

6. Discussion

In summary, the current literature review includes 66 studies, 13 of which were published between 1999 and 2007 and not included in the reviews by De Jong and van Joolingen (1998) or Zimmerman (2007). The current review also includes 20 studies that were not in the two previous reviews because they were published after 2007. By analysing all of these studies, this review provides an extended and up-to-date systematic overview of empirical research on 43 challenges encountered during the four inquiry phases of ‘stating a research question’, ‘generating hypotheses’, ‘planning and conducting experiments’, and ‘data analysis and drawing conclusions’. From our review, it is clear that an increasing number of published studies are investigating challenges related to planning and conducting an experiment (for example, the following studies: Klahr & Nigam, 2004; Tairab, 2015; White, 1993) as well as data analysis and drawing conclusions (for example, the following studies: Aoki, 1991; Klahr & Dunbar, 1988; Masnick & Morris, 2008). Research on learners’ challenges in stating research questions is quite young, as we only found the following studies by Cuccio-Schirripa & Steiner (2000), Hofstein et al.,(2005, 2004) and Neber & Anton (2008a, 2008b). All studies were done on or after the turn of the millennium. In contrast, research on learners seeking out evidence confirming their hypotheses (confirmation bias) leads back to a study by Wason (1960) sixty years ago. Overall, the findings reveal an intensive research focus on neglect of the control-of-variables strategy (for example, the following studies: Chen & Klahr, 1999; Dasgupta et al., 2016) and the engineering approach (for example, the following studies: García-Carmona et al., 2017; Schauble et al., 1991b), both of which are not only found among younger students at the elementary school level (e.g., Erdosne Toth et al., 2000), but remain challenging in adulthood (e.g., Dasgupta et al., 2016). This review’s findings demonstrate that empirical evidence on challenges during experimentation exists in different fields, including science education, psychology and education research, highlighting not only the importance of this topic across disciplines but also the interdisciplinary and domain-general nature of the identified challenges during experimentation.

Although numerous studies have identified a variety of challenges in learning to experiment, their analyses show that these difficulties share common origins. Their different facets and manifestations can be described in line with the three dimensions of knowledge: (1) the conceptual, (2) epistemic and (3) procedural domain (Duschl, 2008; Furtak et al., 2012; OECD, 2019). In the following, we describe the identified challenges along these different facets and manifestations of IBL and therefore condensed into the following key statements:

(1) In some cases, learners lack conceptual knowledge about the scientific concepts underlying the specific inquiry phases and activities of an experiment. Some types of challenges could be related to an inadequate concept about experiments and experimentation. In our opinion challenges in this vein include: ‘Not knowing what a hypothesis is’ (H1), ‘Not controlling variables’ (PC3), ‘Working without a control condition’ (PC5), ‘Confusing observation and interpretation of data’ (DC1) and ‘Learners make no exclusion inferences’ (DC12). (2) Learners also demonstrate a lack of epistemic knowledge related to how scientific knowledge is generated, meaning an understanding of the role of specific constructs and defining features essential to the process of building scientific knowledge. Challenges related to the epistemic domain reflect difficulties in understanding the rationale of experimentation and developing explanations for phenomena under the circumstances that scientific knowledge is subject to change in the face of new evidence or new interpretations of old evidence. Furthermore, the understanding of the function that questions, hypotheses, interpretation and conclusion play in science is affected. Based on this description, the following challenges load into the epistemic domain: ‘Lack of repetition of measurement’ (PC10), ‘No awareness of measurement error’ (M6) and ‘Not connecting hypotheses to previous results’ (DC14*). (3) Some of the challenges show that learners do not know how to conduct scientific investigations and therefore lack procedural skills. Through real-life handling of experimental objects such as equipment, chemicals or, in biology, living organisms, learners can gain practical experience. Challenges of this sort occur in hands-on performances and practically performing the experiment. Challenges revealing this kind of deficit are in our opinion: ‘Handling material and substances indiscriminately’ (PC8), ‘Planning and conducting the same trials several times (without the aim of repeated measurements)’, (PC11), ‘Do not involve all test trials while conducting their experiment’ (PC12). However, there is only limited evidence related to learners’ procedural challenges based on the relatively low occurrences of these kinds of challenges in the current review (PC8-PC12). Although, as just shown, challenges can be distinguished along the three IBL domains (conceptual, procedural, epistemic), scientific literacy requires all three forms of scientific knowledge and skills (OECD, 2018). This also suggests that not every challenge can always be clearly assigned to a single domain, as its characterisation encompasses difficulties in more than one domain. However, the central commonality of the identified challenges across the domains reflects their similarity, which can be helpful, for example, in aligning the goals of IBL and scaffolds in science education settings.

(4) In addition to these three domains there are challenges that stem from cognitive biases and preconceptions that develop early in students’ school careers (for example, the following studies: Carey et al., 1989; Valanides et al., 2013; Wu & Wu, 2011). Cognitive biases and preconceptions impact the conceptual, epistemic and procedural domains of an investigation. Therefore, often not only a single phase is affected, but rather multiple phases of inquiry (for example, as reported in the following studies: Dunbar & Klahr, 1989; Schauble et al., 1991a; 1993). Challenges demonstrating this include ‘Only generating hypotheses that are plausible and predict expected data (fear of rejection)’ (H6), ‘Producing desired experimental results’ (PC6), or ‘Ignoring anomalous data’ (DC8). Learners are guided by preconceptions and assumptions, not by evidence, in planning and conducting experiments and drawing conclusions. In this regard, Zimmerman (2007) points out that learners’ assumptions and beliefs about the goals of experimentation influence how they learn in myriad ways: prior knowledge not only influences their understanding of the nature of science and what experimentation is on a meta level, but might also explain more specific approaches such as the tendency to focus on desirable effects and neglect undesirable ones.

In a broader context, those challenges indicate that there are many obstacles for learners on the way to engage not only in experiments, but with science-related issues that experiments can focus on. In this regard, experimentation relies not only on teaching content knowledge and facts, as these can hardly embrace the character of science (Arnold et al., 2014). In addition, an individual’s understanding of scientific concepts, phenomena and processes, and one’s ability to apply this knowledge to new and, at times, non-scientific situations can promote learners’ scientific literacy (OECD, 2019). By enabling learners to critically examine experiments and their limitations, science education can contribute to an understanding of the characteristics of science and the ability to think scientifically as a prerequisite for critical participation in the public debate. This is especially important at this place in time, where many science-related topics, such as anthropogenic climate crisis, require a certain degree of agreement or consensus in society about scientific evidence (Kranz et al, 2022; Sharon & Baram‐Tsabari, 2020). These complex public discourses are not necessarily a new phenomenon, but they have become especially demanding as science-related topics these days do not only influence everyday life decisions but will also impact personal living circumstances of individuals and entire societies in the near and far future (Lee & Brown, 2018). These topics often elicit persistent public controversy, not uncommonly because many people are misinformed about scientific facts or not able to distinguish between scientific and non-scientific information (Winter et al., 2022 Sharon & Baram‐Tsabari, 2020). Hence, to cope with these demands of the twenty-first century, learners should be able to acquire and then use their developed understanding of science, where experiments play a pivotal role, to contribute to public debate and form informed opinions on science- and technology-based questions in order to become more critical citizens (OECD, 2018).

6.1 Implications for educators and researchers

With regard to the challenges learners encounter when experimenting in IBL settings, a more challenge-oriented approach involving scaffolding measures (e.g., Hmelo-Silver et al., 2007; Sawyer, 2006) has been proven to be effective for overcoming these hurdles (Furtak et al., 2012). The characteristic feature of scaffolding measures is that they enable learners to successfully complete tasks that otherwise – without appropriate assistance – would not be manageable for them (Hmelo-Silver et al., 2007). The results of this review can be used to design specific scaffolds that address the challenges learners face. In that sense, formative assessments that point directly to students’ challenges could also be helpful for fostering learners’ skills and knowledge in experimentation or inquiry-based learning. Addressing the presented challenges in everyday teaching – giving more explanations or offering special instruction on challenges or specific learner’s approaches – could be also very beneficial. For example, if an IBL unit is being implemented to teach concrete scientific concepts (e.g., control-of-variables strategy, Chen & Klahr, 1999) it is advisable to use fully pre-structured tasks at the beginning: providing a pre-specified research question or guidelines for planning and documenting an investigation. This reduces requirements in the procedural and epistemic domains, allowing learners to focus on the subject content. This can ensure that the targeted scientific concept is actually addressed (e.g., Abrams et al., 2008; Blanchard et al., 2010; Vorholzer & von Aufschnaiter, 2019). In contrast, tasks that give learners greater responsibility are suitable when focusing on the application of procedural and epistemic competencies (e.g., Abels, 2015; Fang et al., 2016). Thus, IBL can take on numerous forms along a broad spectrum depending on the instructional objectives and learners’ competence. However, if learners are not given opportunities to experience inquiry activities as comprehensive procedures, they will not be able to fully comprehend scientific methods nor fully appreciate the nature of scientific knowledge itself (NGSS Lead States, 2013). Likewise, our analysis shows that it is not enough to only consider research results on learners’ conceptual and epistemic abilities if the goal is to identify learners’ understanding of and ability to perform experiments. Thus, more research is needed on learners’ procedural knowledge and skills (e.g., handling laboratory materials, chemicals, etc.) and associated challenges when actually conducting experiments in practice. In that sense, practical performance assessments, which are already used to some extent in international school assessment (e.g., Harmon et al., 1997), could provide insights into students’ development of manual skills and abilities. However, 64 out of 66 studies examined in this review used observations, video analyses or other non-practical oriented formats. These types of assessments (non-performance tests) might simplify or omit certain inquiry activities (Ludwig et al., 2018; Schecker & Parchmann, 2021; Shavelson et al., 1999), providing an incomplete picture of experimentation.

In sum, the results reveal an imbalance in the number of studies reporting on learners’ challenges during the different inquiry phases. Many more studies examine challenges learners face when generating hypotheses, planning and conducting an experiment or analysing data and drawing conclusions (n = 61) compared to stating research questions. Until 2008, only five studies had investigated children’s ability to state research questions. This could lead to two conclusions: on the one hand, this finding could illustrate that formulating research questions is not such a demanding research phase as to pose major challenges to learners, or on the other hand, there could be a lack of in-depth research on learners’ understanding of and ability to actually state research questions despite its importance in the inquiry process (Cuccio-Schirripa & Steiner, 2000). The latter would also align with the fact that research questions in IBL settings are often predefined in order to not overburden learners with a too-open research setting (Blanchard et al., 2010) and to ensure that different learners make comparable progress in the inquiry process. However, gaining more insight into how learners can successfully generate an experimentally verifiable research question – instead of leaving it as a blind spot that is largely set by default – should be a focus of future research.

6.2 Limitations

Literature reviews are generally limited by the spectrum of literature they examine based on their database search and subsequent selection procedure. For this literature review, we decided to limit the search to peer-reviewed articles in research journals indexed in any of the four databases used (ERIC, FIS, Web of Science and Google Scholar). We are aware that this decision may have limited the available literature (i.e., if articles are not indexed in the respective databases) and forced us to exclude certain publications (e.g., doctoral dissertations, research reports) that may have provided valuable insights into learners’ challenges when experimenting. However, by searching in four databases that index some of the most visible journals, we aimed to find the majority of important contributions to this research field. Moreover, given the relatively large number of articles included in the current review, a review of all inquiry-related publications might not have been feasible. Thus, we elected to only incorporate peer-reviewed articles to ensure that the reviewed contributions attained a certain level of quality by relying on the journals’ policy to ensure a rigorous peer review process.

Another limiting aspect of our review is the decision to limit the literature search to contributions published in the last 24 years (after 1998). To reduce the possibility of missing relevant contributions published before 1998, we consulted two prior literature reviews with a similar objective (De Jong & van Joolingen, 1998; Zimmerman, 2007), which consider relevant research dating back to before 1998. Thus, these two reviews seemed like an appropriate foundation for the current review’s literature search.

6.3 Conclusion

Overall, the results confirm that the experimental method in general and within IBL in particular represents a complex task for learners. In that sense, the effectiveness of IBL has been questioned in the past (Kirschner et al., 2006; Klahr & Nigam, 2004). Along these lines, recent reviews and meta-analyses of intervention studies contrasting IBL and more explicit or guided teaching methods have found that IBL has no overall positive benefit on student achievement. In fact, the degree of guidance (or openness; Author, 2020; Bell et al., 2005) does influence learning (Alfieri et al., 2011; Lazonder & Harmsen, 2016; Minner et al., 2010): Whilst unguided inquiry learning settings are unlikely to be effective because they neglect for example, the limitations of working memory and executive control (Kirschner et al., 2006), there is also considerable evidence that IBL can indeed improve cognitive achievement, critical thinking and attitudes towards science (e.g., Anderson, 2002; Haury, 1993; Minner et al., 2010; Osborne & Dillon, 2008; Schroeder et al., 2007) and improve learners’ ability to understand and perform experiments (e.g., Blanchard et al., 2010; Carey et al., 1989; Chen & Klahr, 1999; Furtak et al., 2012; Hofstein et al., 2005; Kuhn & Dean, 2005; Schwichow et al., 2016). In this regard, stringently designed and activity-centred learning environments have been proved as particularly effective (Blanchard et al., 2010; Bybee et al., 2006; Minner et al., 2010). According to Kirschner et al. (2006) guidance like this can lower the high demands on working memory and executive control that are required and thus enable learners to encode and store novel information in long-term memory. In light of this debate, the challenges identified in this review could serve as groundwork for the development of scaffolds to support learners’ transition from closed to more open-ended IBL approaches. In a next step, these tailored learning approaches can be empirically investigated, for example, in the form of experimental intervention studies. In our view exploring both – learners’ abilities and challenges – is critical not only for developing IBL interventions to better inform research approaches in science education, but also for their application and implementation by practitioners in science classrooms. In sum, learners’ challenges during the inquiry phases of an experiment identified and analysed in the current review can be viewed as springboards for further learning progress to contribute to critical citizens in a scientific laden world.2018

Acknowledgments

We would like to acknowledge the following experts who contributed to developing the framework for assigning challenges to specific inquiry phases and identifying their relations to the inquiry-based learning process through their ratings, comments and discussions: Till Bruckermann, Markus Emden, Katharina Groß, Marcus Hammann, Kerstin Kremer, Anja Lembens, Antti Lehtinen, Jürgen Mayer, Susanne Metzger, Pasi Nieminen, Marios Papaevripidou, Markus Rehm, Iris Schiffl, Martin Schwichow, Andreas Vorholzer, Nikoletta Xenofontos, Zacharias Zacharia. We also thank Marko Lüftenegger for his valuable feedback on earlier versions of this manuscript.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Johanna Kranz

Johanna Kranz is a Post-Doc-researcher at the Rhineland-Palatinate Competence Center for Climate Change Impacts at the Forest Ecology and Forestry Research Institute (Germany). Her main research interests are inquiry-based learning with a focus on experimentation and the control-of-variables strategy. She also engages in research on climate change education.

Armin Baur

Armin Baur is Professor for Biology and Biology Education at the University of Education in Heidelberg (Germany). His main research interests are inquiry-based learning, experimentation and teachers’ professional development.

Andrea Möller

Andrea Möller heads the Austrian Educational Competence Centre for Biology (AECC Biology) and is professor for Biology and Biology Education at the University of Vienna (Austria). Her main research interests are inquiry-based learning with a focus on experimentation. She also engages in research on environmental education and climate change education.

Notes

1. While other articles use terms such as problems, difficulties or errors (e. g., De Jong & van Joolingen, 1998; Zimmerman, 2007) to describe aspects that comprise the experimentation process, we follow Glazer (2011) and use the term ‘challenge’ to emphasise the potential that arises from working through a challenging issue, rather than simply pointing out deficits and errors (Steuer et al., 2013; Tulis et al., 2016).

Appendix

Results of the expert rating assigning learners challenges to the inquiry phases of an experiment

Note. Learners’ challenges and their descriptions identified in the original studies were assigned by experts (n = 17) to an inquiry phase (stating research questions, generating hypotheses, planning and conducting an experiment, data analysis and drawing conclusions) or to the category multiple inquiry phases of an experiment. The number of expert ratings for each challenge, the frequency of assigning the challenge to the respective inquiry phase in percent (based on the mode), the mean answers per rater (multiple answers were possible) and krippendorff’s alpha are depicted in the table below. In the category ‘multiple phases’ the mode is displayed in squared brackets (1 = stating research questions; 2 = generating hypotheses; 3 = planning and conducting an experiment and 4 = data analysis and drawing conclusions). Challenges marked with an asterisk are specific learner approaches that do not necessarily comprise the successful outcome of an inquiry phase or the outcome of an experiment in general but need to be reflected and addressed in the learning process to prevent subsequent difficulties.

Table

Challenge		Description	Expert ratings	Frequency of assignment (%)	Mean answers per rater	Krippendorff’s alpha (nominal)
Stating research questions
Q1	Not stating causal research questions	Learners only ask factual type questions that cannot be investigated by means of an experiment. Example of a factual type question: How is a heart constructed? What is in this lotion?	16	100	1.0	1.0
Q2*	Stating questions that are qualitative in nature	Learners state questions that are not quantitative type questions. Example of a qualitative type question: Does a plant produce oxygen with the help of sunlight? Example of a quantitative type question: Which effect has the intensity of sunlight on the production of oxygen in a plant? (Which factors are necessary for photosynthesis?) Which effect has the intensity of sunlight on the production of oxygen in a plant? (Which factors are necessary for photosynthesis?)	17	100	1.0	1.0
Generating hypotheses
H1	Not knowing what a hypothesis is	Learners do not have a scientific correct concept about what a hypothesis is.	17	100	1.0	1.0
H2*	Working without a hypothesis	Learners do not generate hypotheses during an experiment.	17	100	1.1	1.0
H3	Operating abductive	Learners do not include their existing content knowledge or knowledge about theories to generate a hypothesis.	16	100	1.0	1.0
H4*	Selecting only desired hypotheses that predict an influence of the independent variable on the dependent variable	Learners do not formulate hypotheses that predict no effect. Hence, they select hypotheses that predict an influence of the independent variable on the dependent variable. For example: ‘salt has an influence on yeast fermentation’ rather than ‘salt has no influence on yeast fermentation.’	17	94.1	1.1	0.80
H5*	Generating hypotheses that include more than one independent variable	Learners state hypotheses that include more than one independent variable. Example of a multivariate hypothesis: Yeast needs salt and flour for fermentation.	16	100	1.1	0.93
H6*	Only generating hypotheses that are plausible and predict expected data (fear of rejection)	Learners just generate hypotheses that are plausible for them and avoid hypotheses with a high chance of being rejected. Existing preconceptions or beliefs have a strong influence on what kind of hypotheses are formulated.	17	100	1.0	1.0
H7*	Not aware that more than one hypothesis is possible	Learners do not generate more than one hypothesis.	17	100	1.0	1.0
Planning and conducting an experiment
PC1	Changing conditions throughout the ongoing experiment	Learners intervene and change experimental conditions throughout the ongoing investigation. Example for changes: add substances in a mixture after the experiment has already started; rearrange the position of an animal in the middle of a behavioural experiment.	17	100	1.0	1.0
PC2	Limiting the experiment on just a few independent variables	When learners are required to investigate multiple independent variables in an experiment, they limit the investigation on just a few independent variables. For example, when asked to investigate ten types of substrates for germination they only test two of them.	17	100	1.0	1.0
PC3	Not controlling variables	Learners vary many variables at the same time and do not control potential confounding variables across experimental conditions (= confounded experiment, no valid inferences possible).	17	100	1.1	0.87
PC4	When testing positive and negative hypotheses different strategies are used	When testing a hypothesis, that predicts a positive outcome (e.g., honey leads to a ‘good cake’), learners use the less valid HOTAT strategy (Hold One Thing At a Time): baking another cake using the same sweetener (i.e., honey), but changing the fat and flour. When testing a hypothesis, that predicts a negative outcome (e.g., honey leads to a ‘bad cake’), learners use the more valid VOTAT strategy (Vary One Thing At a Time; or CVS): using a different sweetener (i.e., sugar), but the same fat and flour.	16	87.5	1.1	0.70
PC5	Working without a control condition	Learners are just working with a single test condition and without a control (= results cannot be compared).	17	100	1.1	0.93
PC6	Producing desired experimental results	Learners design confounded experiments, as a mean to produce a desired outcome.	17	94.1	1.1	0.79
PC7	Engineering approach	Learners create an effect as an outcome (‘engineering approach’) or just conduct experiments that are fun.	16	93.8	1.2	0.66
PC8	Handling material and substances indiscriminately	Learners do not consider the comparability of substance amounts or laboratory material (e.g., using test tubes with different volume) between different conditions.	17	100	1.0	1.0
PC9	Lack of determination of test times	Learners do not determine test times, as well as start- and endpoints and/or periods or intervals.	17	100	1.0	1.0
PC10	Lack of repetition of measurement	Learners do not repeat trials of an experiment. They are more likely to repeat measurements when the independent variable does not have an influence on the dependent variable.	17	100	1.1	0.93
PC11	Planning and conducting the same trials several times (not with the aim of repeated measurements)	Learners design and conduct the same trials (unintended) several times.	16	100	1.0	1.0
PC12	Do not involve all test trials while conducting their experiment	Learners do not carry out all necessary test-trials.	16	100	1.0	1.0
Data analysis and drawing conclusions
DC1	Confusing between observation and interpretation of data	When learners are asked to describe their observations, they interpret the results and when they are asked to interpret the results, they describe their observations.	17	94.1	1.2	0.70
DC2*	Variability in measured data leads to uncertainty	Large sample sizes coming from repeated measurements lead to uncertainty in drawing conclusions.	15	93.3	1	0.83
DC3	Not calculating the arithmetic mean	In repeated measurements, learners choose the first and the last measured value or a measured value that is in between the highest and the lowest measured value.	17	100	1.1	0.87
DC4	Diagrams that are not suitable	When learners are required to create diagrams, they do not create them at all or the diagrams are not suitable for the interpretation of data.	17	94.1	1.0	0.85
DC5	Difficulties when reading graphs	Learners have difficulties comparing and interpreting graphs.	15	100	1.0	1.0
DC6	Drawing conclusions when the result shows no covariation is difficult	Learners struggle with drawing conclusions in the situation, when the independent variable has no influence on the dependent variable. They distort or reinterpret the data, or state that non-covariation results are inconclusive.	17	100	1.0	1.0
DC7	Confirmation bias	Learners seek for information confirming their hypothesis instead of trying to disconfirm their hypothesis.	17	88.2	1.2	0.65
DC8	Ignoring anomalous data	Learners ignore data that is inconsistent with their current belief and therefore make judgements consistent with prior beliefs, even when the evidence does not support those beliefs. Therefore, the hypothesis directs the interpretation of data.	17	94.1	1.0	0.85
DC9*	Anomalous data leads to the assumption that there are flaws in the experimental procedure	Anomalous data leads to the assumption that there are flaws in the experimental procedure. When in doubt, different outcomes on a duplicate trial are interpreted as consistent with an effect if that effect was expected, but interpreted as measurement error if an effect was not expected.	17	94.1	1.0	0.85
DC10	Drawing incorrect inferences	Learners draw incorrect inferences from cohesive experiments.	17	100	1.0	1.0
DC11	Neglecting the protocol leads to incorrect inferences	Learners who not review their notes make invalid inferences.	17	82.4	1.0	0.60
DC12	Learners make no exclusion inferences	Learners struggle to make exclusion inferences and inferences of indeterminacy. Example of an exclusion: Salt has no influence on fermentation. Example of an inference of indeterminacy: It is not sure that salt has an influence on fermentation.	17	82.4	1.0	0.60
DC13	Learners do not give up less advanced strategies when interpreting results	Learners make inferences that are consistent with prior beliefs, based on a single instance of covariation (or non-covariation), or based on one level of the causal factor.	16	100	1.1	0.94
DC14*	Not connecting hypotheses to previous results	Learners do not state hypotheses on the basis of data gathered and do not correct or adapt their hypothesis even if data contradicts it.	17	94.1	1.1	0.74
Multiple inquiry phases
M1	Not investigating rival hypotheses	Learners demonstrate the correctness of one hypothesis rather than setting up experiments to decide between rival hypotheses.	17	58.8 [3]	1.1	0.24
M2	Not being able to identify variables	Learners have difficulties understanding the concept of a variable. They cannot name the types of variables, state the number of variables or determine the specific variable in an experiment.	15	60.0 [3]	1.3	0.25
M3*	Changing hypotheses throughout the experiment	Learners spontaneously change the hypothesis throughout their experiment without changing the experimental setup.	17	70.6 [3]	1.1	0.37
M4	Trying out	Learners are not following a systematic approach when experimenting and therefore are not proceeding in a scientific way.	17	88.2 [3]	1.4	0.44
M5	Lack of documentation	Learners do not take notes about the experiment. If they write protocols, they are incomplete.	17	58.8 [3]	1.1	0.24
M6	No awareness of measurement error	Learners are not aware of the idea of measurement error.	17	50.0 [3, 4]	1.2	0.43
M7*	Not constructing tables	Learners do not organise their data in tables.	17	70.6 [4]	1.0	0.45
M8	Value of the results determines the seeking for evidence	The value of the outcome determines whether learners seek either disconfirming or confirming evidence. Example: When learners are faced with a negative experimental result, they will systematically search for the one thing they can change in order to change the negative result into a positive one.	16	68.8 [4]	1.1	0.38