Stepping stones to success: A qualitative investigation of the effectiveness of adaptive stepped supporting tools for problem-solving in organic chemistry to design an intelligent tutoring system

ABSTRACT

Supporting students in complex problem-solving tasks is intricate due to the variety of difficulties students face. Although adaptive support has proven effective in addressing various difficulties in different disciplines, its application and exploration in science education, particularly through intelligent tutoring systems (ITSs), is limited. This study aimed to investigate the effectiveness of adaptive support in fostering students’ problem-solving processes in organic chemistry, focusing on deriving insights for designing an ITS. Adaptive support, in the form of adaptive stepped supporting tools (ASSTs), was developed and implemented in problem-centred interviews with 19 undergraduate chemistry students. The analysis indicated that ASSTs generally supported students’ problem-solving processes effectively. Additionally, certain factors were identified that influenced the effective processing of ASSTs. The results suggest that an effective ITS should feature highly flexible support pathways, a prior knowledge assessment, the explicit communication of identified difficulties, a drop-out option, and discipline-specific task formats. However, the results also raise questions about the assumed extent of prior knowledge in adaptive support and the point at which the amount of required support becomes impractical.

KEYWORDS:

• Adaptive support
• chemistry education
• higher education
• problem solving

Introduction

Problem-solving

Problem-solving is one of the most relevant competencies across all STEM disciplines. According to Mayer (1998), a problem consists of a present state, a desired state, and the obstacles involved in moving from the first to the second. While various definitions of problem-solving exist, it can generally be described as the process of moving from the present state to the desired state (Anderson, 2005; Hayes, 1989). Problem-solving requires cognitively demanding operations because complex problems are characterised by high complexity, interconnectedness, dynamics, and intransparency (Dörner & Kreuzig, 1983; Funke, 2010).

Successful problem-solving relies on two fundamental aspects: the availability and activation of prior knowledge (Anderson, 1993; Ausubel, 1968; Hammer et al., 2005). According to the resource-based framework proposed by Hammer et al. (2005), learners’ prior knowledge consists of multiple resources that they can utilise for problem-solving. However, prior knowledge is not a static construct that is permanently readily available to learners. Instead, the resources prior knowledge consists of can be considered as knowledge pieces that are interconnected and stored in organised structures (their mental models) and must be specifically activated by the problem-solving task at hand for an effective utilisation (DiSessa & Wagner, 2005; Hammer et al., 2005). Hence, mental models are dynamic systems whose structure is context-dependent, as different resources within the mental model and connections between them are activated depending on the context, such as the problem-solving task at hand (Gupta et al., 2010).

Problem-solving in organic chemistry

In organic chemistry, problem-solving presents a challenge, as common tasks embody complex problems. They simultaneously place multiple demands on students, which vary in terms of the type of knowledge required to solve them and the application of problem-solving competencies (Flynn, 2014; Moloney, 2015). Common requirements of tasks related to organic reaction mechanisms include, for example, recognising reactants in representations, determining chemical concepts that influence the reaction process, weighing up these mutual influences, and providing mechanistic rationales for the reaction process (Asmussen et al., 2023; Dood & Watts, 2022; Flynn, 2014; Graulich & Bhattacharyya, 2017).

To meet these requirements, students must have the resources assumed by the tasks and the tasks must effectively activate it (Anderson, 1993; Ausubel, 1968; Hammer et al., 2005). However, the availability of resources and the degree of their activation both depend on the individual student (Asmussen et al., 2023). Consequently, problem-solving and the difficulties students encounter while problem-solving vary among students. In organic chemistry, a substantial amount of research has focused on these difficulties (Dood & Watts, 2022, 2023). For example, studies have shown that students often rely on alternative problem-solving strategies, including heuristics, rote memorisation, and product-orientation (Bhattacharyya & Bodner, 2005; Dood & Watts, 2023). Additionally, students refer to surface features of representations, use univariate reasoning, and struggle to provide justifications. These strategies impede developing a deep understanding of organic chemistry, as they neglect relevant aspects and include irrelevant aspects in students’ reasoning (Graulich & Bhattacharyya, 2017; Popova & Bretz, 2018; Watts et al., 2021). Students also struggle with specific chemical concepts; for example, they have difficulties with defining concepts, as well as with deciding when to use them in a task. Moreover, they have fragmented knowledge pieces as resources and relate those resources poorly (Asmussen et al., 2023; Popova & Bretz, 2018).

Supporting students’ problem-solving

In an approach to investigate how to support students with these difficulties, Asmussen et al. (2023) grouped students’ difficulties in problem-solving in organic chemistry and classified them using Bloom’s revised taxonomy (Anderson & Krathwohl, 2001). The findings indicated that students require individual support for problem-solving in organic chemistry. However, supporting students is complex, as it involves considering the diversity of difficulties and the individualised nature of their occurrences (Asmussen et al., 2023). Insights from educational psychology suggest that support is most effective when tailored to the individual needs of the students and their difficulties. Support that is either too advanced or too simple can result in cognitive overload or can fail to activate students’ resources. Neither case benefits learning (Homer & Plass, 2010; Sweller et al., 2019). Salden et al. (2010) noted that providing support that no longer aligns with students’ needs can even have negative effects. Hence, opting for a one-size-fits-all support approach is not recommended; instead, the idea of adaptivity should be implemented. While diverse definitions and associated terms of adaptivity exist, such as differentiated, personalised, or individualised learning (Aleven et al., 2017; Major et al., 2021; Schlatter et al., 2022; Walkington & Bernacki, 2020), we define adaptivity as tailoring learning environments to learners’ current learning performance and needs (Plass & Pawar, 2020). To implement adaptivity, learning environments can adapt to several learner variables, which Plass and Pawar (2020) classified as cognitive, motivational, affective, and socio-cultural. Our definition primarily focuses on cognitive variables, which include, for example, prior knowledge, learning strategies, and developmental level.

A form of support that adapts to students’ cognitive variables is scaffolding. Scaffolding is characterised by providing learners with support that enables them to independently solve tasks they initially struggled with (Pea, 2004; Wood et al., 1976). Once learners can solve these tasks without assistance, the support is gradually removed (Puntambekar & Hubscher, 2005; van de Pol et al., 2010). Scaffolding requires analysing learners’ abilities, investigating what they can do on their own, what they can achieve with support, and what remains challenging even with support. The area in which students can achieve learning objectives with support is known as the zone of proximal development (ZPD; Vygotsky, 1978). According to this theory, support must target the ZPD to be effective. Continuously examining each student’s ZPD and tailoring support to it makes scaffolding a time-consuming process (Cagiltay, 2006). Therefore, including scaffolding in lectures is challenging. In this context, digital tools such as intelligent tutoring systems (ITSs) offer a promising approach to meet the individual needs of all students (Belland, 2017; Graesser et al., 2018; Plass & Pawar, 2020). While the overall effectiveness of ITSs is well established, the use of ITSs and research on adaptive support in science education is limited compared to other disciplines (Kulik & Fletcher, 2016; Mousavinasab et al., 2021; Schlatter et al., 2022).

The descriptions of scaffolding, adaptive support, and ITSs raise questions about when, how much, and in what form students should receive support to enable them to solve tasks. Koedinger and Aleven’s (2007) concept of the ‘assistance dilemma’ addresses the questions of timing and the amount of support in ITSs. They emphasise that these questions should be addressed with clear conditions. Determining the form of support also appears complex due to the multitude of possibilities, including prompts, hints, feedback, or providing parts of the solution. Moreover, these can also be presented in different ways (van Merriënboer et al., 2003). All these considerations are crucial when designing adaptive support. In this study, we embedded adaptive support for complex problem-solving in organic chemistry, and investigated it as a representative example for science disciplines. To address the raised questions regarding the development of the necessary adaptive support, we built upon a previous study in which difficulties were classified according to Bloom’s revised taxonomy (Asmussen et al., 2023). As Bloom’s revised taxonomy is used to classify learning objectives, we have defined difficulties as unachieved learning objectives in our study. Bloom’s revised taxonomy is two-dimensional and consists of four knowledge dimensions (factual, conceptual, procedural, and metacognitive knowledge) and six cognitive process dimensions (remember, understand, apply, analyse, evaluate, and create). While the knowledge dimension describes the type of knowledge, ranging from concrete to abstract, from factual to metacognitive knowledge, the cognitive process dimension describes how this knowledge is used. The cognitive processes increase in complexity from remember to create, without forming a strict hierarchy. For a detailed description of all knowledge dimensions and cognitive process dimensions see Anderson and Krathwohl (2001). Drawing on the results of the classification, we developed content-appropriate support formats and divided them into several steps. We chose prompts and information that consisted of hints and parts of the solution. This support was provided to students as adaptive stepped supporting tools (ASSTs; Broman et al., 2018; Hermanns & Schmidt, 2018) whenever they encountered difficulties in their problem-solving process until either they overcame the difficulties or no content-appropriate ASSTs remained.

The current study

The purpose of the current study was to qualitatively investigate the use of adaptive support in science education, using organic chemistry as an example. The aim was to obtain results that will serve as a basis for developing an ITS for organic chemistry. To develop and use a valuable ITS that provides students with adaptive support during the problem-solving process, it is crucial to examine how students interact with this support. Therefore, the emphasis was on two key aspects: first, determining the advantageous types of adaptive support (prompts or information) and the underlying factors necessary for their efficacy; and second, identifying the limitations of adaptive support and comprehending how these factors shape the design of an ITS. Therefore, support was developed and implemented as ASSTs. The ASSTs were integrated into problem-centred interviews. The main research questions were:

1. To what extent do ASSTs effectively foster problem-solving in organic chemistry?

2. What (person- and task-related) factors can be identified in student problem-solving that influence the effectiveness of ASSTs?

Materials and methods

Sample

To address the research questions, 19 undergraduate chemistry students (9 male, 10 female; no other gender was indicated; aged 20–21) were recruited from a German university during spring, 2022. The students volunteered to participate in semistructured, problem-centred interviews and received financial compensation. As a prerequisite for participation, students needed to be currently enrolled in Organic Chemistry I or finished with the course. This was to ensure that the topics of the problem-solving tasks were familiar to the students.

Interview tasks

The problem-solving tasks used during the interviews had been previously examined in another study (Asmussen et al., 2023). The tasks focused on common requirements students have to fulfil in organic chemistry (e.g. recall chemical concepts from memory, recognise implicit properties in representations, apply mechanisms to predict the process and the products of a reaction, or weigh up chemical concepts) and covered nucleophilic substitution and elimination reactions. In total, four task sets were employed. Each task set consisted of two tasks that involved predicting the products of reactions, with slight variations between them, and one task that required students to contrast the two predict-the-product tasks (Figure 1). A comprehensive overview of all tasks is provided in the Appendix. Although predict-the-product tasks are common problem-solving tasks in organic chemistry, they have the disadvantage that students often solve them through rote memorisation. In contrast, case-comparison tasks are less familiar but require deeper reasoning, ensuring that students’ resources are activated (Alfieri et al., 2013). In the predict-the-product tasks, students were instructed to: (a) write down the mechanism and the products of the reaction, and (b) explain and justify the reaction process. In the case-comparison tasks, students were instructed to: (a) compare both reactions and look for differences and similarities, and (b) decide which reaction has a higher reaction rate and justify their decision.

Figure 1. Task set of a nucleophilic substitution reaction used in this study, with the predict-the-product tasks on the left and the case-comparison task on the right.

Interview procedure

Before conducting the interviews, the interviewer (first author) selected a task set to start each interview, ensuring nearly equal coverage of all problem-solving tasks. At the beginning of the interviews, students were informed about their rights. Subsequently, students worked on the paper-based problem-solving tasks while thinking aloud. Whenever the students got stuck, faced uncertainty, or provided incorrect or incomplete responses, the interviewer provided appropriately matched ASSTs (also on paper) to support their problem-solving adaptively. When students explicitly expressed that they got stuck or requested support, they were immediately provided with ASSTs within the ongoing problem-solving process. In cases where such expressions or requests were not made, support was only provided after the students had completed the (sub-)task, allowing them to recognise and resolve their difficulties independently and as well as to avoid interrupting their problem-solving process. After completing a task set, the interviewer and the student decided together, considering the elapsed time, whether there was time for another one of the four task sets. Out of the 19 interviews, two task sets were completed in four interviews, while one task set was completed in 15 interviews. Overall, the interviews lasted 63–128 min. The interviews were audio-recorded and the desk (i.e. all materials as well as students’ notes and drawings) was video-recorded to trace students’ problem-solving and the moments when students received ASSTs.

Adaptive support

The development of the adaptive support was based on the results of a previous study (Asmussen et al., 2023). In this study, the difficulties students faced while working on the given problem-solving tasks were identified and classified using Bloom’s revised taxonomy (Anderson & Krathwohl, 2001). For this purpose, the difficulties were formulated as unachieved learning objectives: ‘The students are not able to [verb] [objective].’ The verb indicates the cognitive process dimension and the object the knowledge dimension. While some problems were classified to reflect students’ difficulties in remembering specific factual knowledge, difficulties predominantly emerged in the knowledge dimension of conceptual knowledge, across the cognitive process dimensions of remember, understand, apply, analyse, and evaluate, independent of the chemical content (Asmussen et al., 2023). As the difficulties primarily pertained to conceptual knowledge and occurred in combination with various cognitive processes, the development of support focused on conceptual knowledge while differentiating only among cognitive processes. Drawing on the literature, support formats suitable for each combination of conceptual knowledge with different cognitive processes were derived and selected. For this purpose, an analysis of the characteristics of difficulties classified as the combination of a particular cognitive process with conceptual knowledge was conducted. Support formats addressing these specific characteristics were then selected from the literature (Appendix). Difficulties in understand conceptual knowledge, for example, are characterised by the fact that students are not able to identify (understand) reactants (conceptual knowledge) within a representation. The support format developed for understand conceptual knowledge thus focuses specifically on practicing this process.

Because the goal of the adaptive support was to offer the precise amount of guidance that students actually needed, each support format was divided into multiple steps that matched in terms of content. These steps were implemented as ASSTs. An ASST provided either a prompt or an information, like a hint or parts of the solution. The first ASST was always a prompt. The prompts aimed to activate relevant resources that had not been utilised yet for problem-solving (Hammer et al., 2005). The information aimed to provide resources that are not part of students’ prior knowledge or to establish interconnections between unrelated resources (DiSessa & Wagner, 2005; van Merriënboer et al., 2003). The steps of each support format were implemented as ASSTs for all chemical concepts relevant in the problem-solving tasks. In total, 523 ASSTs were developed. The developed ASSTs were examined in pilot studies for comprehensibility, coherence, and usefulness, and received positive evaluations. These studies also used problem-centred interviews, in which students received support either adaptively or in a predefined sequence.

Figure 2 illustrates the steps involved in the development of adaptive support, using understand conceptual knowledge as an example. Descriptions of all support formats, the steps they were divided into, and examples of their implementation as ASSTs are provided in the Appendix.

Figure 2. Steps taken in the development of adaptive support, using the example of understand conceptual knowledge. The further steps of the subdivision of the support format as depicted in Step 5 and their implementation as ASSTs are presented in the Appendix.

While the students worked on the problem-solving tasks, the interviewer simultaneously analysed their statements and notes for emerging difficulties and classified these according to the cognitive process dimension of Bloom’s revised taxonomy. The analysis was based on the criteria and coding schemes from a prior study (Asmussen et al., 2023), which are available in the Appendix. Following that, the interviewer chose an appropriate ASST from the ASSTs of the support format developed for the corresponding cognitive process dimension of Bloom’s revised taxonomy and provided it to the student. To enable students to resolve their difficulty as independently as possible and to avoid providing support that may not be necessary, the initial ASST aimed to provide a prompt. However, in cases where this approach was not suitable – such as when students’ statements indicated that a prompt would not support in overcoming the difficulty, when the prompt’s request was already part of the students’ problem-solving process, or when the difficulty was triggered by the provision of a prompt and the prompt therefore could no longer serve as support – an ASST containing information developed for the support format was provided instead. If the chosen ASST did not resolve the difficulty, additional ASSTs were provided until the difficulty was resolved or no further appropriate ASSTs were available. To ensure the consistent, accurate, and simultaneous analysis and selection of suitable ASSTs, all interviews were conducted by the same person.

The goal of the adaptive support was to encourage students to integrate and weigh up all relevant aspects of the task in their problem-solving process. Therefore, ASSTs were also provided when students’ responses were correct but incomplete. ASSTs were not provided for accurate and complete answers.

Data analysis

For data analysis, the interviews were transcribed verbatim, and the allocation of the ASSTs was indicated at the corresponding points in the interview. Then, the interviews were analysed by qualitative content analysis. The content analysis primarily focused on how the ASSTs provided were processed. For this purpose, all text passages with difficulties were identified and the steps following the difficulty were determined. All interviews were coded deductively according to the main codes difficulty, provision of the ASST, processing of the ASST, and resolution of the difficulty. Additionally, subcodes were established. For the main code difficulty, two types of subcodes were employed; One determined the type of difficulty, while the other determined whether the difficulty was occurring for the first time or had been mentioned before. For the other main codes, the subcodes specified whether the ASST included a prompt or information and to what degree the ASST was utilised for the problem-solving process. Table 1 shows the corresponding coding scheme. The detailed coding scheme for the subcodes is provided in the Appendix.

Table 1. Coding scheme for identifying the steps between the first occurrence of students’ difficulties and their resolution.

Main code	Description	Example
Difficulty	Participant  … says or writes something incorrect.  … does not mention or utilise information that is relevant for problem-solving.  … expresses uncertainty about their approach or statement.	When it comes to elimination, I understand that it initiates a halt in a reaction. For instance, in a radical chain reaction, elimination serves to terminate the radical chain reaction by bringing two radicals together.
Provision of the ASST	Interviewer  … provides an ASST to the participant.	Provision of Remember 1 E2
Processing of the ASST	Participant  … interacts with the provided ASST after receiving it.	If I understand correctly, this actually indicates that, whenever we have an elimination, we also have an addition. This can be observed here: Hydrogen is added while bromine is eliminated.
Resolution of difficulty	Participant  … corrects the previous approach or statement.  … adds information.  … confirms their approach or statement.	It’s dawning on me that I've been saying this wrong all along. It’s about the negative inductive effect, not the positive inductive effect. I hadn’t even noticed that.

To examine whether the processing of the ASSTs differed among different types of difficulties, a second step involved classifying all identified difficulties by the cognitive process dimension of Bloom’s revised taxonomy. Although this analysis was also conducted in real time during the interviews, it was done on-the-fly and without systematic documentation. This analysis was also based on the criteria and coding schemes from the prior study (Asmussen et al., 2023).

For coding, the inter-rater reliability was determined based on 20% of the data and using two raters. A Cohen’s kappa coefficient of 0.93 was calculated, indicating high agreement. Hence, only one rater continued coding the remaining data.

Results

During the interviews, the interviewer identified and addressed a total of 369 difficulties, with an average of 19 difficulties per student (ranging from 6 to 31). Looking at individual tasks, there was an average of 7 difficulties per task (ranging from 0 to 18). Notably, the number of difficulties varied between the predict-the-product and the case-comparison tasks. For the former, an average of 3 difficulties per task (ranging from 0 to 12) was observed, while the latter placed more complex demands on the students, resulting in an average of 10 difficulties per task (ranging from 1 to 18).

To what extent do ASSTs effectively foster problem-solving in organic chemistry?

To assess the effectiveness of the ASSTs, we initially calculated the number of resolved difficulties. Overall, the students successfully overcame, on average, 85% of all difficulties using the ASSTs. For a detailed assessment of the effectiveness, difficulties were classified according to the cognitive process dimension of Bloom’s revised taxonomy, grouping similar support formats together. This analysis revealed that the number of difficulties varied among the cognitive process dimensions. Furthermore, the percentage of resolved difficulties varied between these dimensions. While apply had the lowest resolution rate at 75%, understand had the highest at 100%. The resolution rates for the other three cognitive process dimensions ranged between 85% and 88%. Table 2 presents the number of difficulties and the proportion of resolved difficulties for each cognitive process dimension.

Table 2. Numbers of difficulties, resolved difficulties, and required ASSTs for each cognitive process dimension and overall.

Cognitive process dimension	Number of difficulties	Total rate of resolved difficulties	Number of required ASSTs M (SD)	Range of required ASSTs
Remember	103	88%	1.81 (0.64)	1–4
Understand	15	100%	2.60 (1.67)	1–6
Apply	82	75%	2.34 (2.17)	1–11
Analyse	30	86%	2.68 (2.38)	1–10
Evaluate	139	88%	4.52 (5.08)	1–20
Overall	369	85%	3.06 (3.58)	1–20

A closer examination of the individual ASSTs revealed varying degrees of effectiveness of ASSTs within a support format or combinations of these in resolving difficulties. Students needed different ASSTs and varying quantities across all cognitive process dimensions to resolve their difficulties (Table 2), which resulted in diverse support pathways. Figure 3 shows these diverse support pathways and the frequency with which they occurred for each cognitive process dimension. The top level of the figure represents the number of difficulties and reclassifications within a cognitive process dimension. Reclassifications were necessary when a difficulty was initially misclassified based on students’ initial statements. However, upon processing the provided ASST, it became evident that the difficulty was actually rooted in another cognitive process dimension. Difficulties were reclassified at most once; there were no cases of going through multiple cognitive process dimensions before identifying the correct dimension. The middle level illustrates the support pathways, starting with the initial type of ASST (prompt or information) the students received after a difficulty was classified and moving on to the subsequent type of ASST they received (self-loop or the other type) if they needed further support. Cases in which more than two ASSTs were required were grouped as multiple ASSTs, without specifying the exact number and sequence of each type. The bottom level indicates whether navigating the individual support pathways resulted in the resolution or non-resolution of the difficulty. Given that the provision of ASSTs led to the resolution of the difficulty in the majority of cases, and reclassification was not necessary after each new ASST, the development of the ASSTs appears to align with the characteristics of the cognitive process dimensions.

Figure 3. Support pathways for each difficulty across cognitive process dimensions. Each cognitive process dimension is depicted in a different colour in separate graphics. The numbers and the thickness of the arrows indicate the frequency of each pathway. The support pathways progress from top to bottom, processing from the type of difficulty to the provided ASST, and concluding with the (non-)resolution.

Various support pathways were evident across all cognitive process dimensions, but the support pathways in the different cognitive process dimensions exhibited minimal differences. One difference was, for example, that in understand, more difficulties were solely supported by prompts compared to the number of difficulties supported by information or multiple ASSTs. This was not observed in any other dimension. Considering the highest resolution rate (Table 2), prompts in understand seem to have directly contributed to problem-solving, which was less common in other dimensions. However, it is noteworthy that there were both more and less frequent support pathways across all cognitive process dimensions. Among the more frequent support pathways were the pathway from prompt to information to (non-)resolution, from information to (non-)resolution, and from multiple ASSTs to (non-)resolution. Less frequent support pathways were from single or multiple prompts directly to (non-)resolution, from information to prompt to (non-)resolution, and from multiple pieces of information to (non-)resolution.

In addition, across cognitive process dimensions, as the complexity of the difficulty increased, the initial ASST more frequently took the form of information, and the proportion of multiple ASSTs increased. As expected, the increasing complexity of the difficulty corresponded to an increasing amount of required support (Table 2).

Regarding the non-resolution of difficulties depending on the support pathway, it appears that unresolved difficulties generally followed information and not prompts. This is due to the format of the ASSTs. When students could not overcome a difficulty using a provided prompt, they received the corresponding information. However, if using this information also did not resolve the difficulty and no more appropriate information was available, the difficulty remained unresolved. This raises the question of why the same ASST led one student to overcome the difficulty, while another student continued to struggle and required additional support. To address this question, the following section will focus more on examining specific sequences within a support pathway where individual ASSTs proved ineffective than on analysing the entire support pathway.

What (person- and task-related) factors can be identified in student problem-solving that influence the effectiveness of ASSTs?

When analysing sequences where ASSTs proved ineffective, several factors emerged that influenced the effectiveness of the ASSTs. These factors included students’ prior knowledge, students’ acceptance of the information provided by the ASSTs, the nesting of difficulties, students’ ability to process the ASSTs, students’ awareness of their own difficulties, and the fit of the ASSTs provided. These factors manifested individually and in combination.

To illustrate these factors, Figure 4 compares three cases in which providing the same ASST for the same type of difficulty resulted in a different processing of the ASST and, consequently, varying subsequent pathways. In all three cases, students did not consider energetic aspects when reasoning about the reaction rate in the case-comparison task. Consequently, they all received the same ASST, prompting them to establish a relationship between their previous reasoning and the activation energy. The ASST aimed to activate the necessary resources related to activation energy in students’ mental models, enabling them to subsequently integrate activation energy into their reasoning.

Figure 4. Example for the impact of prior knowledge on the effectiveness of ASSTs. The three students encountered the same difficulty and initially received identical ASSTs, but, due to varying levels of prior knowledge, they processed the support differently, which resulted in varying subsequent pathways.

For Thomas, the intended purpose of the ASST was immediately fulfilled. After establishing the connection between his previous reasoning, on the basis of nucleophilicity and activation energy, he independently linked activation energy to reaction rate, thereby expanding his reasoning. This allowed him to overcome the identified difficulty. In Thomas’s case, the task context failed to activate the resource of activation energy. However, because this resource was within his mental model, the prompt supported its activation to effectively link it to the task context.

In contrast, the processing of the ASST differed for Hannah and Chris. While Hannah, like Thomas, was able to establish the connection between her previous reasoning and activation energy, she did not independently integrate activation energy into her reasoning. She needed an additional ASST that explicitly prompted her to do so. In contrast, Chris was unable to make the connection. He struggled with conflicting information he had received in school and university, remaining unsure about which information was correct. Consequently, he received an ASST containing the necessary information that addressed one of his ideas. However, this did not support him to overcome his difficulty, leaving the difficulty ultimately unresolved. Chris’s resources were not coherent, and the ASST activated interconnections to two other different resources. The conflict between these resources was too pronounced to be resolved through one piece of additional information. Instead, Chris expressed a desire to consult relevant textbooks.

A case that shows that prior knowledge interacts with other influencing factors is the example of Eva (Figure 5). Her case illustrates the relevance of accepting the information provided by the ASST. Eva formulated the mechanism incorrectly, which led to incorrect products. She received multiple ASSTs that explained how the mechanism should proceed. However, because these ASSTs were not effective, she received an ASST that showed her exactly how the mechanism was formulated. While going through the mechanism, she realised that she should split off the leaving group. However, she expressed her dislike for the resulting product and decided to omit that reaction step. Only when Eva received an ASST depicting the final solution for the entire task she accepted that her mechanism and products were incorrect. She mentioned that she could have seen this in the first ASST but had resisted incorporating it into her problem-solving process. This resistance stemmed from her aversion to have charges in products, causing her to carry out the mechanism differently. This highlights the interplay between prior knowledge and accepting the information provided by the ASSTs. Because the information provided in the ASST did not fit her mental model, she refused to incorporate it.

Figure 5. Example for the impact of the acceptance of the information provided by the ASSTs on the effectiveness of the ASSTs. The student refused to integrate the information from the first ASST due to personal dislike.

Another influencing factor that interacts with prior knowledge is the nesting of difficulties. Often, the provided ASSTs aimed to activate specific resources or assumed a certain amount of prior knowledge. If these resources were absent or incorrect, the ASSTs frequently led to an additional difficulty. This difficulty would not have arisen without the provision of the ASSTs, as the corresponding resources would not have been activated. These difficulties were not inherently caused by the ASSTs but were revealed by them. To move the problem-solving process forward, these additional difficulties needed to be addressed beforehand, allowing the triggering ASST to become effective. Figure 6 illustrates the nesting of difficulties, sorted by the cognitive process dimensions. For each cognitive process dimension, it depicts how many difficulties contained difficulties from another cognitive process dimension. It becomes evident that, in particular, difficulties in the cognitive process dimension evaluate contained additional difficulties.

Figure 6. Nesting of difficulties between cognitive process dimensions. The numbers and the thickness of the arrows indicate the frequency of each nesting.

In total, nesting occurred for 73 difficulties, which contained 186 other difficulties. This indicates that many difficulties contained multiple other difficulties, with a single difficulty containing up to 10 other difficulties. Furthermore, difficulties were nested multiple times in some cases, where one difficulty encompassed another difficulty, which in turn contained additional difficulties. Nesting of up to four difficulties was observed. These two aspects also occurred in combination. As the embedded difficulties had to be resolved additionally, the nesting increased the number of ASSTs required. Consequently, the problem-solving process was prolonged, leading to highly complex support pathways and problem-solving processes (Figure 3). The increased quantity of ASSTs that resulted from the nesting and the extension of the problem-solving process also affected students’ ability to process the ASSTs, as both the number of ASSTs that needed to be processed and the processing time increased.

The students’ ability to process the ASSTs manifested primarily in the facets mentioned above. Firstly, students stated that the design of the ASSTs was too text-heavy. They suggested figures or bullet points to highlight relevant information and reduce the overall amount of text. Students reported that not only the amount of text but also the amount of information, including new information in general, was challenging to process. Secondly, this factor was also evident in the attention required to process the ASSTs. Due to the lengthy texts and the significant amount of information, there were cases in which students overlooked information or did not read the ASSTs to the end.

Maria’s case (Figure 7) exemplifies how the ineffectiveness of the ASSTs could be attributed to the large amount of information and the attention required to process it. Maria incorrectly explained the influence of electronegativity on basicity. She received an ASST that illustrated, among other influencing factors on basicity, the relationship between electronegativity and basicity. However, she overlooked this information that could have helped her overcome the difficulty. Additionally, Maria’s case suggests that an awareness of one’s own difficulties benefits the success of the ASSTs. Maria’s processing of the information in the ASST was not deep enough to recognise the conflict between her statement and the information provided. With a higher awareness, she could have processed the ASST more purposefully to actively seek the necessary information and could have avoided overlooking critical details.

Figure 7. Example for the impact of students’ ability to process the ASSTs and students’ awareness of their own difficulties on the effectiveness of the ASSTs. The student, unaware that her explanation was incorrect, overlooked the information in the ASST that contradicted her understanding.

The last influencing factor, the fit of the ASSTs provided, manifested in two facets. Firstly, there were cases in which the interviewer selected the ASSTs incorrectly. In these cases, the interviewer either made a mistake during the simultaneous analysis of the interview and misidentified the difficulty or chose an ASST that was either too challenging or too simple. Secondly, this influencing factor resulted from the predevelopment of the ASSTs. For example, in some cases, the difficulty was identified correctly, but no suitable ASST was available for that specific difficulty. Consequently, the interviewer had to provide a less fitting ASST as support. However, these two unintended cases were infrequent overall, with the first occurring in 1.9% and the second in 3.3% of all cases. Figure 8 illustrates the second case. In his justification, Oliver omitted an aspect that influenced the reaction. When he received support to supplement this aspect, it turned out that the resources necessary to understand a term used in the ASST were not present in his mental model. To address this unexpected difficulty, a ‘Remember addition reaction ASST’ would have been necessary, but such an ASST did not exist. As a result, an ASST was provided that employed the term in another context but did not explain it. The ASST was not suitable for the difficulty and, in the end, did not lead to its resolution.

Figure 8. Example for the impact of the fit of the ASSTs provided on the effectiveness of the ASSTs. The student received a less suitable ASST because no appropriate ASST existed for the difficulty the student was experiencing.

Discussion

Regarding the first research question, which pertained to the effectiveness of ASSTs for problem-solving in organic chemistry, the results indicate that, in most cases, the individual support formats implemented as ASSTs effectively contributed to students overcoming their difficulties. This suggests that the analysis of students’ ZPD was successful in most cases, and the ASSTs were well adapted to the students (Plass & Pawar, 2020; Vygotsky, 1978).

However, not all ASSTs were equally effective. Prompts alone were often not sufficient as support and, therefore, not as effective as providing information. The prompts were designed to activate resources not utilised by students during the task. However, if these resources were absent, prompts could not activate them, thus necessitating the use of additional ASSTs (Anderson, 1993; Ausubel, 1968; Hammer et al., 2005).

Regarding the second research question, several factors that influenced the effectiveness of ASSTs were identified. Many of these factors can be attributed to cognitive student characteristics, including prior knowledge, students’ acceptance of the information provided by the ASSTs, the nesting of difficulties, and students’ ability to process the ASSTs. The first three factors were strongly interconnected and rooted, in part, in students’ mental models. The variation in how students processed the same ASST can be attributed to differences in the resources within their prior knowledge and how these were interconnected. This variability arose because either the assumed resources were not activated by the ASST, they were linked to incorrect resources, the existing resources contained inaccurate information, or they were entirely absent (Anderson, 1993; Ausubel, 1968; DiSessa & Wagner, 2005; Hammer et al., 2005). Changing inaccurate information in students’ prior knowledge can be challenging, particularly when the activated resources within their current mental models do not align with the information provided by the ASSTs. Merely presenting correct information might not be sufficient in these cases, as the information lacks a connection point within the students’ mental models (Chi, 2013). This can lead to students not accepting or even rejecting the information provided by the ASSTs.

The characteristics of complex problems become particularly evident in the nesting of difficulties. Nesting demonstrates that problem-solving is highly complex and that individual steps are connected (Dörner & Kreuzig, 1983; Funke, 2010). Nesting was not triggered by the ASSTs but was revealed by them. Whenever nesting occurred, more ASSTs were needed, thus prolonging and complicating the problem-solving process. This required students to process more information, potentially making the problem-solving process cognitively demanding, which influenced the effectiveness of the ASSTs (Sweller et al., 2019). For example, when students needed to simultaneously process multiple new pieces of information, establish connections between them, and deal with additional information, it could have led to a high cognitive load (Sweller et al., 2019). Here, a limit of adaptive support becomes apparent. While it is generally effective, students require large amounts of support in some cases. It still needs to be clarified how much adaptive support should be provided to students during problem-solving and at which point the amount of adaptive support becomes too cognitively demanding for students, so that the problem-solving process should be discontinued to revisit the topic.

Students’ awareness of their own difficulties can be attributed more to metacognitive student characteristics than to cognitive ones. A lack of awareness complicates supporting students, as they themselves do not realise that the resources they activate and their current mental model do not fit the problem-solving task (Chi, 2013). Being aware of one’s own difficulties is important to use a support system (Keller & Hermanns, 2023).

The factors illustrated and discussed so far are rooted in the interplay between ASSTs and students’ characteristics. The last influencing factor, the fit of the ASSTs provided, results more from the interviewer and the ASSTs themselves. When the interviewer chose inappropriate ASSTs, it could be because the interviewer may have recognised students’ ZPD incorrectly (Vygotsky, 1978). However, it could also be because the ASST assumed resources, such as the knowledge of specific terms, that were not part of students’ prior knowledge. In these cases, less suitable ASSTs were provided. Although the results regarding effectiveness suggest that the ASSTs, on the whole, were suitable and the development method based on the classification using Bloom’s revised taxonomy worked well, they still raise the question of how much prior knowledge and what resources should be assumed in the development of support and to what extent the support should go in terms of content.

Limitations

When interpreting the current results, two major limitations should be considered. Firstly, the determined effectiveness of the ASSTs pertained only to the immediate problem-solving process. No information regarding the transfer of acquired knowledge through the ASSTs, either in the short or the long term, was obtained. Furthermore, although qualitative evidence indicated their effectiveness, the quantitative effectiveness was not investigated.

Secondly, the allocation of the ASSTs relied on human judgement; this introduced a potential bias into the ASST allocation. Although efforts were made to keep the allocation as consistent as possible, such as selecting an interviewer with high expertise regarding the ASSTs, maintaining the same interviewer for all interviews, and using a criteria and coding scheme for difficulty analysis, manual allocation inherently leaves room for error.

Implications for designing an ITS

Overall, the results suggest that embedding ASSTs into an ITS has great potential for supporting problem-solving because ASSTs successfully reduced the tasks’ complexity to a level within the students’ ZPD. Moreover, the results offer valuable insights into aspects to consider when designing an ITS that assists students in coping with the characteristics of complex problems. These considerations are illustrated in this paper using the example of organic chemistry, but they could also be applied to other disciplines where complex problem-solving tasks are integrated and supported within an ITS.

Firstly, in the design of the ITS, it is crucial to avoid rigid support pathways for specific difficulties. Instead, the system should allow highly flexible support pathways. As problem-solving is a dynamic process, and our results demonstrate that reclassifications of difficulties were necessary, it is not always possible to identify and classify difficulties or to select suitable support solely on the basis of the initial notes and statements of the students. Hence, the system should not follow a predetermined support pathway after the initial classification but should, instead, allow flexibility for adjustments. Moreover, students, despite experiencing similar difficulties, exhibited individualised support pathways. Implementing this level of flexibility requires a dynamic system. Even though a dynamic system requires a high-performing digital tool, high flexibility should be enabled to make it possible to adapt to students’ individual needs.

Secondly, problems, nested problems, and follow-up problems occur sequentially but it is difficult to foresee them. Therefore, incorporating a prior knowledge assessment before students engage in problem-solving tasks appears sensible to address this intransparency in students’ initial approaches. This assessment could facilitate predictions about when prompts can effectively activate the necessary resources and identify difficulties that might be nested. Utilising the results of the prior knowledge assessment for the allocation of ASSTs could prevent ASSTs from being ineffective and could address embedded difficulties proactively before students start the actual problem-solving task.

To counteract students’ lack of awareness of their own difficulties, the identified difficulties should be explicitly communicated. In the current study, attempts were made to initiate students’ independent reflection on their problem-solving process through practice tasks or by highlighting relevant concepts, thereby aiming to prompt them to notice their own difficulties autonomously. However, as this approach did not prove effective in every case, directly communicating the identified difficulties to students could be beneficial. This approach may also enhance students’ acceptance of the information provided by the ASSTs by demonstrating that they have received support for specific reasons rather than randomly.

Moreover, incorporating a drop-out option seems sensible. Especially in cases in which there is a significant nesting of difficulties and providing support leads to more and more difficulties, allowing students to opt out should be considered. Otherwise, the problem-solving process becomes laborious and potentially frustrating for them, as the ASSTs further increase the already high proportion of interconnected steps. Furthermore, students must process a large amount of information and constantly link new information, making the problem-solving process more complex. In such cases, an alternative approach might be more reasonable, such as addressing the topic through worked examples instead of adaptive support. However, the point at which adaptive support should be discontinued due to an excessive number of ASSTs remains unclear. This also raises the questions of whether only ASSTs that address content relevant to problem-solving tasks should be implemented, how much foundational information should be included as additional support, and which resources can be assumed as prior knowledge.

Lastly, it should also be noted that the task formats used in the ITS need to be customised for the specific discipline in which the ITS is employed. For instance, in organic chemistry, it might be crucial to formulate and draw reaction mechanisms. In other disciplines, such as physics, the ability to perform calculations could be essential. Therefore, when designing an ITS, careful consideration of discipline-specific task formats and their digital implementation is crucial.

Conclusion

This paper describes the development, integration, and investigation of ASSTs for problem-solving in organic chemistry. The goal was to examine the effectiveness of adaptive support in science education and to derive implications for the development of an ITS. The results show that ASSTs, generally, were beneficial for the immediate problem-solving process and supported students in overcoming difficulties. Furthermore, factors were identified that influence the effectiveness of the ASSTs. These include prior knowledge, acceptance of the information provided by the ASSTs, nesting of difficulties, ability to process the ASSTs, awareness of difficulties, and fit of the ASSTs provided. For the development of an ITS, the results suggest that highly flexible support pathways, a prior knowledge assessment, the explicit communication of the identified difficulties, a drop-out option, and discipline-specific task formats should be implemented. Nevertheless, it remains to be clarified which resources and how much prior knowledge should be assumed and at which point adaptive support becomes impractical due to the amount of support needed.

Acknowledgements

We would like to thank Elisa Gelz for her support in participant recruitment and data analysis. This study was approved by the institutional ethics committee (IRB, study ID 2020_AS41).

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the German Research Foundation (DFG: Deutsche Forschungsgemeinschaft) [grant number 506181503].