Enhancing ecological hierarchical problem-solving with domain-specific question agendas

ABSTRACT

Promoting problem-solving in students is an important aim of secondary science education. There is a mismatch, however, between the complex, ill-structured nature of realistic scientific problems, versus the well-structured problems students are generally confronted with. The current study investigates a teaching-learning strategy that resolves this mismatch by combining a focus on hierarchical problem-solving strategies for complex, authentic problems with the use of domain-specific question agendas that represent scientific perspectives. Three design principles were applied to develop an exemplary lesson series that was implemented in two Dutch pre-university classes. A pre-/post-test research design was followed in which data was collected in the form of sets of student-generated research questions. Additionally, audio recordings of lessons and student interviews were collected and analysed. Results indicate that participating students became more proficient at applying hierarchical problem-solving strategies like (1) reducing complex ecological problems to more manageable subproblems by formulating productive research questions and (2) identifying types of ecological problems by connecting them to domain-specific question agendas (problem-abstraction). A qualitative analysis of the teaching-learning process and student interviews informed potential refinements of the design principles, namely the use of more diverse contexts and a greater focus on collaborative learning.

KEYWORDS:

• Ecology education
• higher-order thinking
• problem-solving

Introduction

In today’s world of growing complexity, there is a mismatch between problems students are confronted with in the science classroom and the challenges of modern-day society (Osborne et al., 2018). Whereas real-life problems are often complex and ill-structured, current secondary science education mainly teaches students how to solve well-structured problems (Milbourne & Wiebe, 2018). The well-structured problems students are confronted with in science lessons are generally clearly formulated, what knowledge or skills needed to solve these problems is known beforehand and it is easy to determine the prerequisites for a ‘correct’ solution. Examples of such problems are determining pH in chemistry or calculating electrical resistance in physics.

However, many real-life problems in science are structured to a much lesser extent (Jonassen, 1997; Law et al., 2020; Van Merrienboer, 2013). Real problems often have less clearly defined features, it is often unclear what kind of knowledge would lead to productive problem solving and determining criteria for a ‘correct solution’ is difficult. An example of such an ill-structured problem is the plastic pollution of the oceans, and determining how this can best be countered.

The difference between well-structured and ill-structured problems is not a dichotomy but can better be regarded as a continuous scale (Reed, 2016; Simon, 1973) with the highly structured problems students are confronted with on one end and highly ill-structured problems, sometimes referred to as ‘wicked problems’ (Rittel & Webber, 1974) or swamps (Schön, 1987) on the other end. Although it has been a prime focus of educational research to investigate how students solve well-structured textbook assignments, this has not been the case for solving ill-structured problems (Jonassen, 1997; Law et al., 2020; Van Merrienboer, 2013). Additionally, when students are proficient in well-structured problem-solving, this does not necessarily mean that they are also proficient in ill-structured problem-solving (Milbourne & Wiebe, 2018) which means that findings from studies about well-structured problem-solving cannot be extrapolated to ill-structured problem-solving.

If we want to promote ill-structured problem-solving in secondary science education, a teaching-learning strategy is needed that focuses on these skills. For the development of such a strategy, we build on insight from two separate literary fields: studies concerning hierarchical problem-solving and studies concerning scientific perspectives.

Firstly, we base our study on research about problem-solving that concerns strategies that are useful both for well-structured and ill-structured problems (Chi, 2011; Reed, 2016; Simon, 1973), but which have predominantly been investigated only for well-structured scientific problems (Chi et al., 1982; Milbourne & Wiebe, 2018). Such strategies can be summarised with the term hierarchical problem-solving. In hierarchical problem-solving, a concrete problem is first identified as belonging to a certain category of problems with similar problem representations. Identification of the problem on an abstract level facilitates a process in which the problem is divided into smaller, more manageable, subproblems. Potential solutions to these subproblems then guide the problem-solving process for the concrete original problem.

Secondly, research on scientific perspectives provides us insight into how scientific experts apply hierarchical problem-solving strategies to approach ill-structured problems (Giere, 2010; Love, 2013; Wimsatt, 2007). These studies show scientific perspectives, that can be visualised as domain-specific question agendas, to be promising scaffolds to guide the development of student hierarchical problem-solving strategies for domain-specific ill-structured problems.

In this study, we combine knowledge from both fields, hierarchical problem-solving and scientific perspectives, in a teaching-learning strategy focused on bridging the gap between secondary science education and the real world. Domain-specific question agendas, question hierarchies that represent scientific perspectives and show how knowledge is structured, are used as scaffolds for student development of hierarchical problem-solving strategies for ill-structured problems. We focus on the subject of ecology, as this is a domain in which ill-structured problems are highly prevalent (see also ‘Focus on ecology’ below). We aimed to answer the following research question: Does a teaching-learning strategy that uses domain-specific question agendas promote ecological hierarchical problem-solving in pre-university students?

Concretely, design principles were developed for designing lessons in which students are supported to gradually develop domain-specific question agendas while being explicitly guided through activities of hierarchical problem-solving. The design principles pertained to three key features: (1) offering students realistic, ill-structured scientific problems; (2) asking them to formulate research questions, and (3) asking students to rephrase their research questions so that they can be situated in a domain-specific question agenda. The design criteria were applied to develop an exemplary lesson series for the subject of ecology at the pre-university level. The design was implemented in a pilot in which both the design principles and the lesson series were iteratively refined (supplemental table 1). Finally, data was collected from an implementation of the design in two pre-university level classes and analysed to gain insight into the realised learning trajectory and the effects on students’ hierarchical problem-solving strategies.

Theoretical framework

Hierarchical problem-solving

Research into effective problem-solving has a longstanding tradition, originating with the founding work of Herbert Simon and Allen Newell (Chi, 2011; Newell & Simon, 1972; Nokes et al., 2010; Ohlsson, 2012; Reed, 2016; Van Merrienboer, 2013). This line of research initially focused mainly on abstract puzzles but has shifted in time to problems that contain more realistic contexts and require more domain-specific content knowledge. These studies have shown that differences in problem-solving competency are generally not dependent on having a better memory or on general problem-solving skills. Instead, the consensus is that problem-solving proficiency is determined to a large extent by if and how a problem-solver has organised knowledge into productive knowledge structures or cognitive schemas (Chi, 2011; Nokes et al., 2010; Ohlsson, 2012). Proficient problem-solvers appear to have cognitive schemas containing abstract elements that they can attune to concrete problems. Take for instance the problem of planning a vacation. Such a problem has many factors that are dependent on the specific vacation one is planning: the destination, choice of transport to the destination, the maximum budget, etc. These factors can be different each time, so to successfully plan a particular vacation a problem-solver needs to account for the concrete ‘values’ of each factor for that specific vacation. Thinking back and forth between a concrete problem and an abstract problem representation is at the basis of hierarchical problem-solving.

Hierarchical problem-solving can be divided into three sub-processes that are facilitated by a cognitive schema: problem abstraction, problem reduction and problem refinement (Chi et al., 1982; Nokes et al., 2010; Ohlsson, 2012; Reed, 2016). Different names are used for these processes in different studies, in this study we consistently use the terms given above. We will now briefly go into how cognitive schemas facilitate these problem-solving strategies.

Suppose the solution to a particular problem is not immediately recognised. In that case, the problem first needs to be regarded in a more abstract problem space (problem abstraction): what kind of problem is this? Problem-identification facilitates determining what kind of questions need to be answered to reduce problem space. To generate workable questions, the problem first needs to be reduced to more manageable subproblems (problem reduction). The extent to which a problem solver is able to successfully reduce a problem is largely determined by the quality and range of the questions posed. For problem reduction to be successful, the questioning process should be (1) sufficiently broad so that the multifaceted nature of the problem is accounted for and (2) sufficiently deep so potential refinement of the problem is facilitated. Finally, it should be determined how answers to individual questions are relevant to the ill-structured problem at hand and how they contribute to possible solutions (problem refinement).

This process of hierarchical problem-solving has been elaborated on in detail for relatively well-structured problems in science education (Chi, 2011). However, it has been less thoroughly explored how such strategies could be formalised to benefit ill-structured problem-solving (Nokes et al., 2010). For inspiration, we will next look into strategies of scientific experts in solving ill-structured problems.

Scientific perspectives as scaffolds for ill-structured hierarchical problem solving

In contrast to the well-structured problems science students are confronted with, experts generally are tasked with solving problems that are ill-structured. These could be, for instance, problems for which essential knowledge has yet to be developed or problems for which key knowledge is available but where is difficult to determine what kind of knowledge would be relevant. In such cases, scientific perspectives turn out to be key factors in facilitating hierarchical problem solving. At its core, a perspective is an abstract schema that describes how knowledge is structured for a certain domain, which can guide hierarchical problem solving (Bechtel & Richardson, 2010; Craver & Darden, 2013; Giere, 2010; Thagard, 2012; Wimsatt, 2007). Below, we will first illustrate how scientific perspectives can guide problem-solving by way of a hypothetical example. Subsequently, we go into how the abstract structure contained in a scientific perspective can be made explicit in the form of a question agenda in order to make it an educational scaffold for promoting hierarchical problem-solving.

To illustrate how scientific perspectives can guide hierarchical problem-solving strategies, we first need to consider how they serve as filters through which experts explore problems. As this study focuses on ecology, we will consider the example of a hypothetical ecologist that thinks about a current ecological situation: the northward spread of the Asian tiger mosquito through Europe. First, the ecologist has to identify the problem on an abstract level in order to focus subsequent dissection of the problem. Problem recognition guides which scientific perspectives are relevant and what will be productive questions (problem abstraction). The ecologist could argue that species will only migrate to other regions if those regions contain sufficient food sources, if there is adequate opportunity to reproduce and if the abiotic circumstances are favourable. By formulating these lines of reasoning, the ill-structured problem is divided into smaller, more manageable, subproblems (problem reduction) that can then be reformulated into specific research questions about the issue at hand. For example: ‘What do mosquitos feed on and are these food sources present?’, and ‘Have abiotic circumstances changed in the areas the mosquitos have spread to?’. Answers to these questions can then be synthesised into more general notions that can be applied to the original problem (problem refinement). For example, the expert could hypothesise that higher temperatures and migrating patterns of certain bird species contributed to the spread of the Asian tiger mosquito, whereas a factor like disease was less relevant.

In the example above, a domain-specific perspective facilitated hierarchical problem-solving. An ecological perspective is of course broader than the questions that the ecologist pursued in the example. After it was determined which parts of the perspective were relevant to the problem at hand, this guided the exploration of the problem, with the perspective serving as a ‘search light’. Finally, answers to specific questions were synthesised into more general hypotheses about the problem. Such a strategy requires the problem solver to have a certain level of mastery over relevant scientific perspectives, something that students have yet to develop. To show students the hierarchical thinking structure implicit to scientific perspectives, explicit representations of abstract perspectives are needed that can be used in educational settings.

A scientific perspective denotes an abstract schema that captures a core reasoning (Landa et al., 2020; Thagard, 2012). This core reasoning is a formulation of the idea at the basis of a scientific perspective. Landa et al. (2020), for example, identify the ‘particle perspective’ as one of four scientific perspectives that constitute the basis of the Dutch secondary school chemistry curriculum and formulate the following core reasoning for this perspective:

The properties of substances can be explained by the nature of the particles of which it consists, the forces between them, and the movement and organization of those particles.

The bolded words in this core reasoning are variables that could refer to many different kinds of concepts. For instance, the word ‘forces’ could refer to physical force, magnetic force or van der Waals force. A core reasoning serves as a template that concisely states how the multitude of concepts in a knowledge field meaningfully interact.

To make the abstract structure of a scientific perspective explicit to students, a scientific perspective can be visualised as a set of hierarchically connected questions: a domain-specific question agenda (Love, 2013). A core reasoning can guide the design of such a question agenda. Earlier studies suggest that domain-specific question agendas could be used in educational settings to scaffold and structure student thinking (Janssen et al., 2019; Landa et al., 2020). For instance, Landa et al. (2020) designed a question agenda that illustrates how forces, movement and organisation of particles interact and influence each other. We propose that question agendas that represent a scientific perspective can specifically be used to scaffold hierarchical problem-solving for ill-structured problems, in this study for ecological problems. We used the methodology of Landa et al. (2020) to create three-question agendas that represent ecological thinking (Figure 1).

Figure 1. Ecological question agendas for the ‘organism-perspective’ (left), ‘population-perspective’ (middle) and ‘ecosystem-perspective’ (right).

In sum, we propose that question agendas that represent scientific perspectives can serve as domain-specific scaffolds for teaching students how to apply strategies like problem reduction, problem abstraction and problem refinement when thinking about scientific ill-structured problems.

Focus on ecology

The subject ecology has inherent characteristics that fit the focus of this study. Ecological problems are often ill-structured: ecological circumstances usually contain a high degree of interdependence, individual factors are hard to measure definitively and the relative importance of specific factors is often unclear. Additionally, thinking about ecological problems requires thinking from different ecological perspectives by default, for instance the perspective of single organisms, populations or whole ecosystems (Walter & Hengeveld, 2000). As such, when students learn to apply hierarchical problem-solving strategies from the viewpoint of ecological perspectives, this could make them more adept ecological problem-solvers.

For this study, three ecological question agendas were developed: one for the organism-level, one for the population-level and one for the ecosystem-level (Figure 1). The question agendas correspond with common views of ‘auto-ecology’ (the study of individual organisms), ‘population ecology’ and ‘ecosystem ecology’ (Walter & Hengeveld, 2000). The ecological question agendas were developed by first performing a content analysis of four major university level ecology textbooks. Core reasonings were formulated that informed the formulation of a main question for each perspective and which ecological concepts were of relevance. The core reasonings were then cross-checked with other textbooks that contained sections about ecology. Then, biological concepts were reformulated to succinct questions and connected to each other and the main question in order to construct a comprehensive question agenda. Finally, the three-question agendas were presented to four pedagogical content experts whose feedback informed further refinements.

Design principles

To identify design principles, we took as starting point the logic of scientific perspectivism as formulated in Giere (2010). Perspectivism states that problem-solving in the scientific field entails being confronted with challenging, ill-structured problems and to approach them as experts would, from varying perspectives. Since students do not yet have extensive problem representations like experts do, productive strategies to approach such problems need to be made explicit to them. To practice these strategies in class, they need to be confronted with realistic ill-structured problems and asked to adopt an attitude of scientific inquiry. Three design principles were extracted from the literature to be consistent with this rationale; relevant research is discussed for each design principle below. Figure 2 is a schematic representation of the order of lesson segments in the final design.

Figure 2. Schematic representation of the order of lesson segments (LS) for one or multiple consecutive lessons. In our study, each cycle of all five lesson blocks was performed in two to three lessons, three cycles were performed in total.

Design principle 1: start with a complex, ill-structured problem

Our first design principle was taken from the whole-task-first educational approach (de Graaf et al., 2019; Janssen & Van Berkel, 2015; Van Merrienboer, 2013). The central idea behind this approach is that students are first primed to think about an authentic, generally ill-structured, problem that requires higher order thinking and synthesis of knowledge, after which students work on part-tasks that are relevant to the central task. This methodology fits the purpose of this study as it provides the students domain-specific, ill-structured problems to reason about. Also, the real-life relevance of lesson content is emphasised. In this study, we provided ecological problems at the start of blocks of two to three lessons (Figure 3).

Figure 3. Timeline of the study. QN = data used for quantitative analysis, QL = data used for qualitative analysis.

Design principle 2: students are invited to adopt the role of researchers to induce a focus on formulating questions

Following Van Aalsvoort (2004), students are instructed to assume the role of ecological researchers in charge of a research team, and to formulate research questions. The role of ecological researcher was selected to put the students in the frame of mind of generating questions, instead of providing answers, as they are generally are not used to this (Chin, 2002). Earlier studies emphasise the potential of students’ questions to enhance learning, but at the same time acknowledge that it can be difficult for students to generate productive questions as this requires a certain level of domain-specific knowledge (Chin & Osborne, 2008). Design principle 3 addresses this finding.

Design principle 3: domain-specific question agendas are gradually constructed as scaffolds for hierarchical problem-solving

Domain-specific question agendas can be used to scaffold the students’ problem abstraction (Landa et al., 2020; Love, 2013). However, they have to be developed in class as students do not yet have the prerequisite domain-specific thinking skills needed. To achieve this, the teacher explicitly guides students to rephrase their research questions about an ill-structured problem at a more abstract level so they can be added to a gradually expanding ecological question agenda (the organism-, population- or ecology-agenda). In doing so, the students experience how their questions fit and extend each of the three-question agendas and how they contribute to the hierarchical structure contained within. When students internalise the structure of the ecological question agendas, they create cognitive schemata that can subsequently guide all steps of hierarchical problem-solving (problem abstraction, reduction and refinement) for similar ecological problems they are faced with at a later point in time.

In the methods section, we discuss how the three design principles were elaborated concretely in a lesson series.

Methods

The design principles we formulated informed the design of an exemplary lesson series on ecology in which students are given opportunity to practice hierarchical problem-solving in class. The data we gathered provides us insight into this process. The quality of the research questions students generate when confronted with ill-structured ecological problems determines the extent to which they are able to divide ill-structured problems up into more manageable subproblems (problem reduction). The quality of student questions was assessed with a rubric and the students’ progress in their questioning ability was evaluated using a pre/post-test design. Additionally, to evaluate the students’ proficiency at problem abstraction, the extent to which students were able to correctly categorise their questions with the three-question agendas was determined.

To gain insight into how the development of the ecological question agendas was integrated with the hierarchical problem-solving steps, student-student and teacher-student interactions at key moments were recorded and analysed. To gain insight in student experiences, groups of three to four students were interviewed in semi-structured interviews after completion of the lesson series.

Finally, grades for a school test were compared between participating classes and two non-participating control classes in order to determine how students who participated in the study performed concerning Dutch attainment targets for ecology. This provided us insight into the feasibility of the teaching-learning strategy in a realistic teaching context. The test was created by biology teachers of the participating school who did not partake in the study and consisted of questions comparable to questions for the Dutch national exams for biology.

Participants

Two classes containing 51 11th-grade pre-university students (28 boys, 23 girls) from one Dutch school participated in the study. The school is situated in an urban area in the Netherlands and enrols over 800 students. Two other 11th-grade classes were included as control classes for comparing learning outcomes with respect to Dutch attainment standards for ecology (n = 41; 27 boys, 14 girls). Most students were between 15 and 17 years old and had a rudimentary level of knowledge of ecology, as an introductory course on the subject had been previously taught in the 8th grade.

Lessons for participating classes were performed by the main researcher, their own teacher was present in most of the lessons in a supporting capacity. Control classes were taught the same content as the lessons for the participating classes, but didactical choices were made by individual, non-participating teachers.

Informed consent

Before the start of the study, students were given the option to participate or to opt out of specific data being collected (audio and video recordings, worksheets). Students (or a parent for students who were younger than 16) were asked consent. No participating students or parent denied consent. The study was performed in accordance with the Helsinki declaration.

Lesson series

The lesson series consisted of eight lessons of 45 minutes per lesson and spanned the regular school curriculum content for ecology. Figure 3 shows a general timeline. For practical reasons, the biology textbook was used as a structuring element: the lesson series followed the order of topics in the textbook, and textbook exercises were assigned as homework. Some textbook exercises were reworked to fit the ecological context. For instance, a textbook homework assignment that had students draw a food web for a forest ecosystem was reworked to fit an ecological context that was used in the study.

An ecological issue that recently garnered a great deal of attention in Dutch national news was selected to build the lesson series around (design principle 1). The problem concerns a large population of big herbivores (mainly deer) that lives in Oostvaardersplassen, a nature reserve in the Netherlands. Because of a shortage of food for a population of that size, many deer are at risk of starvation every winter. Several solutions to this problem have been suggested, ranging from hunting the herbivores to not interfering at all. The ill-structured and topical nature of the problem make it a good fit for the lesson series.

For each ill-structured problem, the students are invited to adopt the role of researchers (design principle 2). In the course of the lesson series, they are first tasked with determining if introducing a pack of wolves would be a suitable way of dealing with the herbivore problem. The organism-agenda (Figure 3) was developed during this phase. Second, as a related problem, the students are tasked with determining how large the wolf pack should be in order to manage the herbivore problem in the long term. The population-agenda (Figure 3) was developed during this phase. Finally, students were confronted with two additional problems in new ecological contexts in the last lessons of the lesson series: the problem of non-reproducing giant pandas in China and the problem of globally diminishing coral reefs. Novel ecological contexts were selected for the last two problems to prevent the phenomenon where students specifically link knowledge to examples that were used in class. The ecosystem-agenda (Figure 3) was developed during this phase.

As part of the lessons, the teacher guides the students in gradually constructing the ecological domain-specific question agendas (design principle 3). The lesson series starts with incomplete question agendas in which only the main questions are given. The three-question agendas (organism, population and ecosystem) are then elaborated at specific moments. When groups of students are instructed to generate research questions that could provide relevant answers for an ill-structured problem (problem reduction; Figure 2, box 2), the questions students formulate form the basis for a teacher-guided dialogue. Each group of students is asked for the questions that they considered to be most important to the problem and to situate their questions by indicating which of the three ecological question agendas (organism, population or ecosystem) they think their questions belong to. Then, students are instructed to rephrase their questions in a more general manner so that they can be added to a question agenda (problem abstraction; Figure 2, box 3). Students concomitantly add questions to individual print-outs of (incomplete) ecological question agendas, so that each student draws their own copy of the three-question agendas. Finally, the teacher reflects with the students on how possible answers to questions in the question agenda could contribute to coming up with solutions to the original problem (problem refinement; Figure 2, box 5). After this cycle is completed three times throughout the lesson series, all three ecological question agendas have been developed into complete versions (Figure 1).

Data collection

For the pre-/post-test of student questions (Figure 3, box 1 and 5), individual students were given worksheets before and after the lesson series. The worksheets contained an ecological problem, instructed students to adopt the role of ecological researchers and to write down every research question they considered relevant to the problem. Finally, they were asked to categorise their questions as belonging to either the organism-, the population- or the ecosystem-question agenda. The students were confronted with similar problems before and after the lesson series: the pre-test tasked students with investigating a sharp decline of the sparrow population in the Netherlands and the post-test tasked students with investigating possible causes of bees dying at an accelerating rate.

Additionally, grades for a school test for ecology were collected for participating students and students in two control classes, in order to analyse the effect of the lesson series on traditional learning objectives (Figure 3, box 6).

Qualitative data was collected to gain insight into the realised learning trajectory. To investigate how students reduced ill-structured problems, they were divided into groups of three to four that were recorded while they cooperatively generated research questions for ecological problems (Figure 2, box 2). In order to determine what characteristics of student discourse were associated with high learning gains, a group of students that showed a large mean pre-/post-test improvement in question quality was selected for further analysis.

Recordings of classroom dialogue (Figure 2, box 3) were used to determine how student questions were discussed and used to elaborate question agendas (problem abstraction).

Semi-structured interviews were conducted with three randomly selected groups of four students after conclusion of the lesson series (Figure 3, between box 5 and 6). The students were interviewed in groups in order to create an informal setting. The interviews focused on how the students experienced the teaching-learning process compared to their regular curriculum and how, if, they thought the question agendas contributed to their understanding of the theoretical content.

Analysis of student research questions

Data analysis on student research questions focused on (1) measuring the extent to which students were able to formulate question sets that could achieve problem reduction, and on (2) measuring the extent to which students were able to achieve problem abstraction by categorising their questions with one of the three-question agendas. A question set was defined as ‘all questions a student formulates in response to an ecological problem’. The quality of individual research questions produced by the students in the pre- and post-test was determined with a rubric. For the analysis of pre-/post-test differences, 13 students had to be excluded due to there being data available only for the pre- or the post-test. Thirty-eight students were included for the final analysis. SPSS v23.0 was used to perform all analyses. To illustrate how data analysis was performed for one question set, an example is provided in the supplementals (S3).

Analyzing quality of question sets (problem reduction)

To determine the quality of individual questions, a rubric was developed. An initial 3-item version of the rubric was drawn up by the main researcher and subsequently refined to the final 2-item rubric after multiple discussions in the research group (see supplemental table 2). With the rubric, each question was graded with a score of 0–2 on two items: specificity of the question and relevance of the question to the specific ill-structured problem (a 4-point maximum/question). These characteristics were considered to be prerequisites of questions that could possibly achieve problem reduction. If a question is not specific, for instance, when formulated ambiguously, it is impossible to determine how answering it would achieve problem reduction. Instead, if a question is not relevant to the problem, potential answers will certainly not lead to problem reduction. Student mean scores for the specificity and relevance of their questions were compared using a paired-samples t-test.

After being graded for specificity and relevance, each question was then categorised by the researchers as belonging to one of the three-question agendas (organism, population or ecosystem). This categorisation facilitated the analysis of whole question sets, measuring problem reduction, but also the analysis of student question categorisation, measuring problem abstraction (see below). A ‘question agenda score’ was calculated per student for each of the three-question agendas by adding the scores of the three highest-scoring questions per question agenda. The limit of three questions per question agenda was introduced to disentangle the quality and quantity of questions formulated by a student within an agenda.

The question agenda scores for the organism-, population- and ecosystem-levels in the pre- and post-test were then used as dependent variables for the analysis of complete student question sets. The quality of complete question sets was compared between the pre- and post-test with a repeated measures MANOVA using the question agenda scores for the organism-, population- and ecosystem-levels in the pre- and post-test as dependent variables. In order to gain insight into potential differences between problem reduction proficiency on the organism-, population- and ecosystem-levels, pre- and post-test scores for individual question agendas were compared separately using a paired-samples t-test.

For the analysis of deep and broad problem reduction, the generation of questions from multiple dissimilar question agendas was regarded as broad problem reduction and the generation of multiple questions within one question agenda was regarded as deep problem reduction. Pre- and post-test minimum and maximum question agenda scores were used as indicators of, respectively, broad and deep problem reduction and were compared using a paired-samples t-test.

Categorising questions with question agendas (problem abstraction)

To gain insight into the proficiency of students to situate their questions in a more abstract problem space, the students’ skill in categorising their questions with specific question agendas was measured by calculating the overlap (in percentage) between the researchers and each student in regard to the allocation of questions to a specific question agenda. These scores were then compared between the pre- and post-test using a paired-samples t-test.

Interrater reliability

A sample of 26 question sets (17% of all question sets collected in the pilot and the main study) was graded by 2 independent raters to determine an interrater reliability for the rubric. As the items for which we wanted to determine the reliability were ordinal measures, we applied weighted Cohen’s Kappa for this analysis (using an SPSS extension). This analysis demonstrated a moderate level of agreement for the ‘relevance’ item (90% agreement, κ = 0.41),¹ a good level of agreement for the ‘specificity’ item (81% agreement, κ = 0.70) and an excellent level of agreement for the categorisation of questions with certain ecological question agendas (86% agreement, κ = 0.81).

Analysis of scores for the school test

Scores for the school test were compared between students from the participating classes and control classes using an independent samples t-test.

Key aspects in the teaching-learning process

The qualitative analysis focused on three key aspects: (1) how students reduced complex ecological problems by formulating research questions (Figure 2, box 2), (2) how question agendas were elaborated in teacher-guided problem abstraction (Figure 2, box 3) and (3) how students reported to value the use of hierarchical problem-solving and question agendas in the student interviews (Figure 3, between box 5 and 6). Our aim was to extract examples that were representative for the realised teaching-learning process and could provide context to the main results. The recordings were first transcribed verbatim and subsequently analysed by the main researcher using an inductive approach: first transcriptions were scrutinised to identify common themes, after which findings were substantiated with excerpts that exemplified those themes.

The analysis on problem reduction focused on the discourse in a student group that had a high mean pre-/post-test improvement in question set quality. Audio recordings were analysed in order to gain insight into if and how their reductive questioning process could be linked to their considerable learning gains. Findings were discussed with the other researchers in order to identify common themes (Braun & Clarke, 2012). Finally, excerpts from the student discourse were selected in order to provide an illustrating example for each finding.

Transcripts of recordings of the plenary classroom discussion in which students situated their questions in the question agendas were analysed analogously to the analysis on problem reduction: first common themes were distilled from the recordings, then findings were discussed in the research group and finally excerpts were selected in order to provide an illustrating example for each finding.

The analysis of the student interviews focused on if and how students describe aspects of hierarchical problem-solving as part of the realised teaching-learning process and if and how the students describe the ecological question agendas as a structuring tool for thinking about ecological problems. Again, common themes identified by the main researcher were first discussed in the research group after which illustrative excerpts were selected.

Results

Implementation of the design

The implementation of the design largely followed as planned. The teaching-learning process depicted in Figure 2 was completed three times, each time with a different ecological question agenda as the main theme. The largest deviation from the planned trajectory occurred when, due to a school trip, many students were missing from the lesson in which the third and fourth ill-structured problems were introduced. As a result, we did not use data that was collected during this lesson.

Student questions

Specific results of the pre-test vs. post-test scores are shown in Figures 4–6. Mean scores for complete question sets were significantly higher in the post-test vs. the pre-test (Wilks’ Lambda = 0.83; p = .032). This difference was mainly due to an improvement in deep questioning (mean maximum question agenda score of 9.8 vs. 10.8 points; t(37) = −2.14, p = .039), while broad questioning did not improve (mean minimum question agenda score of 2.3 vs. 2.2 points; t(37) = 0.28, p = .78). More specifically, mean question scores were significantly higher in the post-test for the population-level (t(37) = −3.027, p = .004), while question scores for the organism-level (t(37) = −1.16, p = .26) and ecosystem-level (t(37) = 1.24, p = .22) did not differ between the pre- and post-test. This indicates that students improved in asking questions on the population-level by asking more or higher quality questions, whereas their questioning skill for the organism- and ecosystem-levels did not improve. The number of questions in each question set showed a trend towards students asking more questions in the post-test, compared to the pre-test (6.2 vs. 7.0 questions per question set, t(37) = −2.02, p = .051).

Figure 4. Mean scores for complete question sets (±SD) in the pre- and post-test, stratified by individual question agenda scores.

Figure 5. Mean scores for the ‘specificity’- and ‘relevance’-items in the pre- and post-test.

Figure 6. Correct categorisation of student questions in question agendas (±SD) in the pre- and post-test.

Concerning specific question characteristics, students’ questions in the pre- and post-test did not differ in average specificity (mean score of 1.74 vs. 1.82, t(37) = −1.38, p = .18) or relevance (mean score of 1.80 vs. 1.88, t(37) = −1.63, p = .11).

The classification of questions by students as being on the organism-, population- or ecosystem-level significantly improved during the course of the study (t(37) = −3.22, p = .003), indicating improved problem abstraction. The majority of categorisation errors in both the pre- and post-test (respectively 59% and 54.9% of all errors), concerned questions that the students incorrectly categorised as being on the ecosystem-level, instead of on the organism- or population-questions level.

Concluding, the mean question set scores improved during the course of the study. This change resulted from improved deep problem reduction at the population-level (as scores for the organism- and ecosystem-level and broad problem reduction did not significantly change). The trend towards more questions in the post-test makes it likely that the improvement was mainly attributable to students asking more questions as opposed to more specific or more relevant questions. Additionally, the students’ proficiency in categorising their own questions with specific question agendas, indicating problem abstraction, improved considerably.

School test

Student scores on the school test with final exam questions did not differ significantly between the test and control groups (mean score of 17.46 (±4.27) vs. 17.29 (±3.80) respectively; t(78) = −0.187, p = .85).

The teaching-learning process

This section concerns the unfolding of the realised teaching-learning process. The first part discusses how a group of students reduced ill-structured ecological problems by formulating research questions. The second part focuses on how ecological question agendas were developed in teacher-guided problem abstraction. The final part focuses on how students experienced the lessons.

Cooperative problem reduction

The group of which the discourse was transcribed consisted of four students whose mean question set scores improved considerably in the pre- vs. the post-test (18.2 vs. 24.8 points). Table 1 shows the research questions this group formulated in response to the problem whether or not wolves should be introduced in Oostvaardersplassen as a solution to the herbivore problem (Figure 2, box 2).

Table 1. Student questions that were written down in response to the ill-structured problem.

Group 1
What kind of food relations does a wolf have?
Can the wolf survive in the circumstances?
Which organisms does a wolf attract?
At what rate does a wolf reproduce?
How many deer does a wolf eat?
Does the wolf have an alternative for deer?
Is the wolf a prey itself?

The first finding was that the group selectively focused on a series of isolated notions about the problem, that they each think through to an endpoint before continuing with the next one. The excerpt below exemplifies this process. They discuss the possibility of introducing a singular wolf or wolves of only one sex, but can quickly dismiss both ideas as they recognise this would not lead to a ‘sustainable’ solution for the problem. In doing so, they have effectively reduced problem space:

Student 1:

We are tasked with investigating if introducing [wolves] would be a good idea.

Student 2:

Ah, right. Hmmm.

Student 3:

This could lead to an abundance of wolves, if they have a lot of food to their disposal.

Student 1:

If you only take one wolf, then you won’t get anymore.

Student 3:

But that singular wolf is not going to help with the problem, it will only eat a few.

Student 1:

Right, so in any case you should have reproducing wolves. How many children does a wolf get?

Student 3:

Or just male or female wolves, then they can’t reproduce.

Student 1:

No, because they want it to be a self-regulating ecosystem so it should have some sustenance. The idea is: if you let them die because of hunger, that is not possible, but if you introduce a wolf then it keeps on regulating itself and that makes it kind of sustainable.

When student 1 argues that introducing only male or only female wolves will not solve the problem, because ‘they want it to be a self-regulating ecosystem so it should have some sustenance’, she reduces the problem space for the whole group by adding the prerequisite that the wolves should at least be able to reproduce.

Another interesting theme was the collaborative nature of the discourse of the group. When one of the students formulates a question, the other students in the group collaboratively reformulate the question such that answers to it would plausibly reduce problem space. The excerpt below illustrates this:

Student 1:

How about abiotic factors? Light, water, temperature, air.

2:

Ehm, maybe wolves, they also need water, maybe there is not enough water. I don’t know, somehow it doesn’t feel that unlikely.

3:

Are the circumstances ideal for wolves?

1:

Yes, can the wolf survive in the current circumstances?

 … 

 

1:

You also have scavengers. So if the wolf leaves leftovers. But we have to think of questions.

3:

We could say: What other influence could [introducing the wolf] have, next to a decrease in the deer population, right?

2:

Yeah, do they also eat other animals? Rabbits or something like that.

3:

I think they will.

2:

Will animals like scavengers also move to the area?

1:

Ok. What kind of organisms will the wolf attract?

This example shows how the students start with a general question or statement, ‘How about abiotic factors?’ and ‘You also have scavengers’, which they then collaboratively refine into workable research questions like ‘can the wolf survive in the current circumstances?’. In doing so, they have successfully reduced the problem by defining parts of the larger problem that could be explored separately.

Teacher-guided problem abstraction

The excerpt below illustrates an example of problem abstraction: how the teacher guides students in reformulating their questions in a more abstract problem space with the goal of adding their questions to question agendas (Figure 2, box 2). A student starts with a simple question about a specific organism. The teacher then prompts the students to reformulate the question so that it could be applied to a broader range of organisms. Finally, the teacher reformulates the resulting question slightly by changing the wording and adding ‘ … does the organism have?’. The excerpt ends with the teacher writing a new question in the question scheme. This illustrates a cycle that was repeatedly followed during class discussion after each time students formulated research questions.

Teacher:

What more do you want to know about the wolf?

Student 1:

What he eats.

Teacher:

Very good, what does a wolf eat. And if we wanted to make this question less specific?

Student 1:

Uhm, is the wolf a carnivore or a herbivore?

Teacher:

Yes, that is good, but can you make it even less specific? Because when you say: is it a carnivore or a herbivore, then you are specifically talking about animals, but it would not concern plants. Can you make it even less specific?

Student 2:

What are the sources of nutrition?

Teacher:

Excellent! ‘What sources of nutrition does the organism have?’ *Teacher writes the question in the question agenda*

In prompting the students to formulate the question so that it can be applied to other organisms, the teacher makes explicit how this line of reasoning could be applied to other ecological contexts. By writing the question in the question agenda, the teacher shows how this question fits in the hierarchical structure of ecological thinking. These strategies scaffold the development of an ecological cognitive schema in students and thus could promote their problem abstraction proficiency.

Hierarchical problem-solving in student interviews

In all three interviews, students remarked that they felt their skill to generate relevant questions improved. All groups also remarked about the reductive nature of the questioning process. As one student put it:

[Generating questions] mainly teaches you to see what is the most essential [to the problem], because for one problem you might say: ‘Ok, food sources are really important’, but if there is a really weird climate over there, then climate is also kind of important. Like, can they even survive in this area? So you could estimate what is essential to a certain situation and that will not always be the same thing.

Another student answered: ‘Well, [you learn to] ask a lot of questions, and then cross out what you don’t need. Yeah, just asking questions to answer an even bigger question’.

All groups referred to the hierarchical structure of the subject matter when they were asked how they would define the way of thinking that was central to the lessons. For instance, one student remarked that she learned ‘how to approach research, that you start by thinking in a general manner and subsequently specify your thoughts in a certain way’. When asked how she would describe ecological thinking, she said ‘[It is] the bigger picture, … for a certain problem in nature, that you zoom in at a particular aspect. That you look at possible causes and that you think of all kinds of questions you need to ask yourself’. Another student responded: ‘you learn to think in a certain way, you see a question and then you know what steps you have to take to arrive at an answer’.

These examples show students describing elements hierarchical problem-solving. They describe hierarchical properties of the problem space when they talk about ‘the bigger picture’ and ‘zooming in’. They describe reductive thinking processes like selecting questions and ‘specifying thoughts’. They talk about refining problems when they mention ‘asking questions to answer a bigger question’ or ‘cross[ing] out what you don’t need’.

Ecological question agendas in the student interviews

Two of the three groups mentioned that the question agendas provided structure for connecting concepts. One student remarked:

[It shows you] a main category and that you can see what actually belongs to what. … It also [shows] the cohesion between the different concepts. That may help in remembering the content, because if you remember one concept you just think: ‘Oh, but that other one is linked to it’.

Two of the three groups also mentioned how question agendas can specifically help with generating questions. As one student put it: ‘It gives you an idea of the thinking process with a certain question. So you can think: at what level is this and what are subquestions that I could ask?’.

To conclude, students generally describe ecological question agendas as providing cohesion in both their knowledge acquisition and their question-generating abilities. As such, the results indicate that these students regard the ecological question agendas as structuring elements for learning.

Conclusion and discussion

Current secondary science education lacks teaching students how to reason about realistic, ill-structured scientific problems productively. The general way in which problem-solving is presented misrepresents how experts navigate scientific problems by way of problem abstraction, reduction and refinement while using scientific perspectives as domain-specific filters. Students have yet to develop useful cognitive schemata that guide their problem-solving. Lessons that promote the development of domain-specific hierarchical problem-solving strategies by using scientific perspectives as scaffolds could be beneficial to this process.

With this study, we investigated a teaching-learning strategy that aims to promote ecological hierarchical problem-solving strategies in pre-university students. A literature analysis and pilot study informed the formulation of three design principles: (1) start with a complex, ill-structured problem, (2) students are invited to adopt the role of researchers to induce a focus on formulating questions and (3) domain-specific question agendas are gradually constructed as scaffolds for hierarchical problem-solving. The design principles were operationalised in a lesson series for 11th-grade Dutch pre-university students in which data was collected.

The main analysis showed that complete student question set scores improved during the study, indicating improved problem reduction. This improvement was mainly attributable to improved deep questioning on the population-level. Additionally, students became more proficient at correctly categorising their questions with certain question agendas, suggesting improved problem abstraction. Therefore, we conclude that the current design, and by extension the three design principles, were successful in promoting the students’ ability to productively think about complex ill-structured problems. More detailed analysis of the data informs several refinements to the design principles that we will discuss briefly in the following section.

The observation that students mainly improved in asking questions on the population-level deserves further attention. A possible explanation could concern the fact that the complex problems the students were confronted with all concerned shrinking populations of certain species. As such, these problems could have mainly elicited questions about factors that directly influence population size. This finding signifies the importance of using a diverse range of contexts that focus on dissimilar (parts of) question agendas as ill-structured tasks when implementing design principle 1.

Another interesting finding is that there was considerable variance in individual student results. Analysis of student discourse showed that a large pre- to post-test improvement co-occurred with a pattern of highly collaborative co-construction of research questions. The potential learning benefits of collaborative learning strategies are well-documented (Davidson & Major, 2014), but not all students are equally proficient in working together and building on each other’s ideas. This suggests that inviting students to take the role of researchers does not necessarily ensure that they collaborate productively ‘as researchers’. Thus, design principle 2 might be further elaborated with a greater focus on collaborative learning strategies.

Analysis of the implementation of the teaching-learning strategy revealed a pattern of plenary teacher-guided problem abstraction in which students collaboratively reformulate their own research questions such that they can be used to develop ecological question agendas. The teaching-learning strategy could have contributed to the students’ improved problem abstraction, but this cannot be ascertained as the students’ mastery of the question agendas was never (formatively) assessed. As a domain-specific question agenda is essentially a representation of a cognitive schema, developing such an agenda in class would be expected to go in tandem with students developing a corresponding cognitive schema in their minds. Indeed, all student groups that were interviewed reported the hierarchical structure of the question agendas to have been helpful in seeing coherence in the subject matter. Nevertheless, future implementations could benefit from a larger focus on formatively assessing students’ knowledge of structural elements in the question agendas, in order for a teacher to be able to provide targeted feedback on the development of their cognitive schemata.

General limitations

Although this study yielded promising results, findings should not be overgeneralised. The effectiveness of the teaching-learning strategy is expected to be dependent both on the content of the lessons and on the educational context in which it is implemented. Additionally, because of the exploratory nature of the study we did not include a control sample for our main analysis. A study that compares this teaching-learning strategy to regular practice, for example by using control classes, would be an interesting direction for further research.

Another limitation of this study is the relatively small sample size for quantitative analyses. This results in loss of power and could explain some of the non-significant results. However, the fact that analyses of the main outcomes of this study still yielded significant results despite of this underscores their substantivity.

Theoretical and practical contributions

The main contribution of this study is that it provides a blueprint for a teaching-learning strategy that combines two novel educational concepts: a focus on hierarchical problem-solving and the development of domain-specific question agendas that represent scientific perspectives as scaffolds for this process.

Hierarchical problem-solving strategies received considerable attention in literature that concerns effective problem-solving. For instance, there are many studies on how such strategies can optimise search in computer systems (for an example: Bacchus & Yang, 1994). The main contribution of this study is that it illustrates how hierarchical problem-solving strategies can be promoted in secondary science education by implementing question agendas as domain-specific scaffolds. A focus on hierarchical problem-solving shifts emphasis from the endpoints of a thinking process (providing ‘correct answers’) to the thinking process itself (applying scientific perspectives to explore problems). Domain-specific question agendas that represent scientific perspectives serve as a link between abstract cognitive schemata and concrete contexts. A question agenda can show students that scientific knowledge is not a combination of isolated notions but generally amounts to meaningful rationales that can be applied to real-life phenomena. This study shows that if students learn to apply domain-specific hierarchical problem-solving strategies, this could make them more flexible problem-solvers, especially when faced with complex problems with a high degree of ill-structuredness.

The use of question agendas in this study bears resemblance to the role educational literature attributes to advance organisers (Ausubel, 1960): both are representations of cognitive schemata that are used in educational contexts to provide meaning to content. Where they differ is that advance organisers are presented to learners in complete form before actual content is discussed, whereas question agendas are gradually refined in tandem with discussion of the content. Additionally, domain-specific question agendas always contain hierarchically structured questions, whereas this is not a prerequisite for advance organisers. The question-based structure makes question agendas particularly suited to the goal of teaching how to reduce problems by formulating productive research questions. The gradual construction of the question agendas facilitates opportunities for students to practice teacher-guided hierarchical problem-solving in class.

Finally, this study can be regarded as a contribution to a growing base of literature on problem-based learning in general, and specifically how ill-structured problem-solving can be deployed in educational settings to enhance learning outcomes. Studies that have similarities to ours have investigated the educational potential of ill-structured problems to promote higher order thinking in other domains such as chemistry (Overton & Potter, 2011) or physics (Erceg et al., 2013). For instance, particular similarities can be observed when our teaching-learning approach is contrasted with the dynamic problem-based learning of Overton and Potter (2011): both teaching-learning strategies have students divided into groups and put into the role of experts who are consulted to assist with a real, domain-specific ill-structured problem, after which they brainstorm about the nature of the problem and how best to approach it. Our study contributes to the body of work on ill-structured problem-solving by introducing domain-specific question agendas as a way to make the hierarchical properties of productive scientific thinking explicit to students and guide the development of their problem-solving skills.

In conclusion, teaching students hierarchical problem-solving strategies with the help of domain-specific question agendas can be instrumental in making learners more adept ill-structured problem solvers. This study is a promising proof-of-concept that lays the groundwork for a teaching-learning strategy and provides a case study that shows how this strategy can be operationalised within the constraints of a fixed national curriculum. Promising future directions for further research would be to investigate how this teaching-learning strategy would impact student learning trajectories when it is implemented over a longer period of time, with different subjects or in different educational contexts.

Ethical approval: Ethical approval was obtained from the local review board of the Faculty of Behavioral & Movements Sciences of Vrije Universiteit Amsterdam (VCWE #2019.151).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was supported by funding from the Dutch Ministry of Education, Culture and Science via the DUDOC programme.

Notes

1 The discrepancy between the high percentage of agreement between the raters and the low Cohen’s Kappa for the ‘relevance’-item is due to the fact that both raters predominantly scored students’ questions as being ‘very relevant’ (the highest rating for that item), causing a high a priori chance for maximum points for this item. For a more detailed explanation of this phenomenon see Feinstein and Cicchetti (1990).