Teachers’ understanding of assessing computational thinking

ABSTRACT

Background and context

Assessment is one of the challenges related to the introduction of Computational thinking (CT) in compulsory education.

Objective

This study aims to explore teachers’ understanding of CT assessment and the associated possibilities and challenges when CT is integrated into school subjects like mathematics and science.

Method

We conducted interviews with teachers from Norway and Finland, where CT is included in the national core curricula. The interviews were analysed thematically by adapting a framework for CT integration into subjects.

Findings

Teachers found CT assessment to be challenging, due to its broader scope beyond coding. Formative assessment was found to be suitable for assessing CT, whereas national exams could become prescriptive amid assessment uncertainties.

Implications

Definitions of CT and vague curricular goals are not enough for teachers to assess CT. Professional development or clearer assessment guidelines could ameliorate the difficulties with assessing CT..

KEYWORDS:

• Computational thinking
• programming
• assessment
• teachers

1. Introduction

The field of education is adapting to the demands of a digital society, and one of the changes that is reflected in the field of education is the introduction of computational thinking (CT) in school curricula. CT is described as a transversal skill that prepares students for the 21^st century and equips them with the skills needed in the digital world (Bocconi et al., 2022). Despite the trend of including CT in education, there is no consensus on the definition of CT in the field (Tikva & Tambouris, 2021). Nevertheless, CT has been established as a prominent concept in the literature (Brennan & Resnick, 2012; Grover & Pea, 2013; Shute et al., 2017). The growth of CT in education is noticeable in the US (Yadav et al., 2018) as well as in Asia (So et al., 2020) and Europe (Bocconi et al., 2022). Bocconi et al.'s (2022) review describes the inclusion of CT in curricula in 29 European countries. They found that 25 out of the 29 countries have adopted CT in their curricula, with 18 of the adopting 25 countries revising their curricula with respect to CT between 2016 and 2021, highlighting the rapid growth of CT in curricula in recent years.

The integration of a new concept, such as CT, into compulsory education poses challenges not only to teaching but also to assessment (Bocconi et al., 2022). As Weintrop et al. (2021) highlighted, the rise of CT in curricula necessitates unpacking CT assessment as a topical issue. Challenges in assessment are well understood in the realms of subjects such as mathematics and science, unlike in computer science, where there are gaps in teachers’ assessment literacy (Yadav et al., 2015). Although CT assessment has, to some extent, been studied in programming and computer science courses (de Araujo et al., 2016; Tang et al., 2020), Yadav, Gretter, et al. (2016) indicated that teachers in computer science found teaching and assessing their subject challenging and pointed to difficulties both with respect to content and pedagogy. There is even less emphasis on assessing CT when it is integrated into other subject domains; thus, there is a call to align and understand assessment with respect to subject matter knowledge and CT (Tang et al., 2020).

Yadav, Hong, et al. (2016) pointed out the pressing need to address in-service professional teacher development regarding the assessment of CT. There is also a need to further develop formative assessments of CT and programming (Grover, 2021). Given that teachers’ intentions are essential in formative assessments (Black & Wiliam, 2009), teachers’ understanding of CT assessment is an important aspect to consider in developing teaching and assessment guidelines, designs, or teacher professional development.

Most European countries have adopted CT into their curricula but use different approaches in the integration (Bocconi et al., 2022). In this paper, the focus is on the integration of CT in Norway and Finland, where CT and programming have been included as compulsory elements in curricula. Studying teachers’ adoption of CT in these two countries can illustrate the different opportunities and challenges posed by CT integration. Norway represents a country where CT integration is solely in the existing subjects, in this case mathematics, science, arts and crafts, and music. In Finland, CT integration is also transversal, beyond its integration in mathematics and crafts (FNBE, 2014). Studying the two countries can provide insight into both the subject-level and transversal aspects of CT integration.

The assessment of CT is described as an unresolved matter (Bocconi et al., 2022; Shute et al., 2017), and as shown in the next section, research on the teacher’s perspectives on the assessment of CT is scarce. Therefore, the aim of this study is to explore teachers’ understanding of CT assessment in their own practice. To this end, we raise the following research questions:

1. How do teachers understand the assessment of CT?

2. What opportunities and challenges do teachers identify with respect to CT assessment?

The paper is structured as follows. An overview of CT in an educational context and the assessment of CT are presented in Section 2. The analytical framework of CT assessment and integration is presented in Section 3 along with descriptions of the curricula in Norway and Finland. The research design is presented in Section 4 and in Section 5, we outline the findings. The findings are discussed in Section 6 and the conclusions are presented in Section 7.

2. Background

In this section, CT in an educational context is presented, along with studies on CT assessment and teachers’ experiences with CT assessment.

2.1. Defining CT in an educational context

Despite the ambiguity in definitions of CT (Tikva & Tambouris, 2021), general characteristics of what CT refers to can be drawn from the literature. Brennan and Resnick (2012) outlined a CT definition for the block-based coding environment Scratch, encompassing three dimensions of CT. The first entails CT concepts, such as sequences, loops, parallelism, events, conditionals, operators, and data. The second describes CT practices, such as being incremental and iterative, testing and debugging, reusing and remixing, and abstracting and modularising. The third dimension are CT perspectives, which relate to working with CT, such as expressing, connecting, and questioning. To address the connection of CT being closely related to scientific practice, Weintrop et al. (2016) proposed a definition of CT for mathematics and science classrooms comprising four categories: data practices, modelling and simulation practices, computational problem-solving practices, and systems thinking practices. To map the scope of CT and provide a definition and framework of CT, Shute et al. (2017) reviewed 45 studies of CT in education and proposed that CT is represented by six facets: decomposition, abstraction, algorithm design, debugging, iteration, and generalisation. An effort to further consolidate the existing CT definitions was undertaken by Tang et al. (2020), who divided CT definitions in an educational context into (1) CT related to programming and computing concepts and (2) competences that are needed for domain-specific knowledge and general problem-solving skills. This study is informed by the definitions above, with CT being both related to programming and computing concepts (Tang et al., 2020) and consisting of decomposition, abstraction, algorithm design, debugging, iteration, and generalisation (Shute et al., 2017).

The nature of CT is multi-faceted, as described by the outlined approaches. Against this background, the introduction of CT into curricula can take many forms, and hence curricular operationalisations, policy documents, or exams can become steering. Since the definitions of CT vary, a question that can be raised is whether the assessment of CT is difficult for teachers.

2.2. Assessment of CT

The assessment of CT is an unresolved issue within the field of CT and education (Román-González et al., 2019; Shute et al., 2017; Weintrop et al., 2021). Assessment serves several purposes, ranging from monitoring individual students’ progression to institutional and national accounting purposes (Newton, 2007). Formative assessment can increase engagement and improve students’ learning outcomes (Wiliam, 2011) while also providing feedback to the teacher about what is working and what is not. According to Yadav et al. (2015), CT can be a difficult concept to assess when there are multiple correct solutions and the complex concepts, making traditional assessment unsuitable. Therefore teachers need support in developing an assessment literacy of CT. Given the lack of educational and pedagogical knowledge of CT assessment both in practice and in research (Grover, 2021), CT assessment is a field that needs to be expanded.

In terms of CT assessment, two issues are pertinent: what is assessed and how it is assessed. What to assess is connected to the lack of consensus about the definition of CT and to the integration of CT into subject domains. For example, Tang et al. (2020) found that the field of CT assessment is dominated by assessing cognitive constructs, such as CT concepts, skills, and programming. de Araujo et al. (2016) found that problem solving was an element mentioned by 96% of the studies on CT assessment, with most studies teaching CT through programming courses. The authors of the study further found that problem solving, developing algorithms, and applying abstraction were the aspects that were mostly assessed. Cutumisu et al. (2019) indicated similar findings in their review of empirical studies from 2014–2018: the most frequently assessed concepts in CT were algorithms, abstraction, problem decomposition, logical thinking, and data.

The question of how to assess CT was also mentioned in review studies. However, none of the reviews explicitly focused on formative assessment. Tang et al. (2020) found that most assessment studies employed paper – pencil tests and portfolio assessments. Other assessment types were selected- and/or constructed response questions, surveys, and interviews. Although the review also indicated that portfolio assessments aid formative assessments, the focus appears to be on summative assessments in most studies. de Araujo et al. (2016) found that assessments were conducted through assessing code or through multiple-choice questions. Cutumisu et al. (2019) found that most assessments were not programming language-specific, and that they were conducted using multiple-choice and open-ended questions. This indicates that CT assessment involves a broad range of methods, with a majority of studies focusing on multiple-choice questionnaires, while paper and pencil tests, portfolio assessment, and interviews are also identified as means of assessing CT. Similarly, Poulakis and Politis (2021) conclude in a review of CT assessment that multiple methods of assessment are preferred. The suggested methods include interviews, think-aloud protocols, field observation, evaluation rubrics and portfolios. The authors propose, based on their review, that knowledge transfer is the essential and a common criterion in studies on assessing CT. Da Cruz Alves et al. (2019) reviewed studies on automatic CT assessment in K-12 and found that most automatic CT assessments targeted algorithms and programming sub-constructs. They found gaps in research with respect to assessing broader 21^st century skills, which also are a component in CT. These studies propose that CT cannot be assessed solely through a narrow assessment of programming. Since the introduction of CT in education, it has been suggested that CT skills are transferable across contexts (Denning & Tedre, 2021) and the transferability of CT skills across domains has driven policy makers to include CT into national curricula (Vinnervik, 2023). The empirical support regarding transfer of CT is, however, not conclusive (Scherer, 2016). Therefore, it appears that assessing CT as problem solving, as transfer of knowledge and as a broader 21^st century skill remains an intricate issue.

2.3. Teachers assessing CT

The different interpretations of CT lead to assessment of different concepts (Poulakis & Politis, 2021) and the multitude of definitions makes assessment of CT complex issue. Since there is no unified definition as such the responsibility for deciding what to assess might become the responsibility of teachers. If there are national curricula, teachers can be guided by these, but this depends on how elaborate and detailed these are. Some national curricula, such as the Norwegian curricula (NOU 2015:8, 2015, p. 68), follow the principle of giving teachers freedom to choose the teaching methodologies. As a consequence, the curriculum offers few guidelines for how to meet the curricular aims. A lack of a unified CT definition therefore puts more pressure on teachers, and how they understand CT and CT assessment becomes a question that needs to be addressed.

In integrating CT in schools across disciplines, developing the field of CT assessment is a crucial aspect (Shute et al., 2017), and successful integration of CT into curricula requires assessment (Grover & Pea, 2013). Arguably, this highlights the need for robust assessment practices. In addition to summative assessment practices, which have to some extent been mapped in the field (Tang et al., 2020), there is a need for studies on formative assessment (Grover, 2021) and a need to incorporate broader and holistic assessments of CT (Grover, 2017). Including teachers’ understanding of the assessment of CT will aid in understanding the field of CT assessment and its challenges. Furthermore, to the best of our knowledge, studies linking CT, assessment and teachers’ perspectives are limited. In the following paragraphs, we draw on the few related articles we identified in the literature.

How teachers perceive the assessment of CT has not been extensively researched. Cabrera (2019) argued that teachers’ preconceptions about teaching practices may influence their teaching, and consequently impact how students learn. Common challenges that teachers report when teaching CT are teachers’ own subject knowledge and challenges with respect to differentiation to meet different levels of ability (Sentance & Csizmadia, 2017) and misalignment between assessment practices and CT curriculum, affecting the teachers’ preparedness to teach and assess CT (Irons & Hartnett, 2020).

Findings from CT assessment studies indicate that teachers perceive code quality as difficult and unimportant to assess (Crow et al., 2020; Kirk et al., 2021), while feedback on syntax is perceived as easier to give than feedback on style and semantics (Crow et al., 2020). Teachers also perceive that students do not understand the value of feedback on style and semantics (Crow et al., 2020). Regarding abstraction, often described as an essential component in computational thinking, Liebe and Camp (2019) found that teachers do not use rubrics when assessing abstraction and are generally uncomfortable with assessing abstraction. These aspects indicate some areas in CT that teachers perceive as difficult to assess. Thus, the teacher perspective on assessing CT needs to be further explored to fully understand the potential constraints and aspects of CT to emphasise in assessments.

3. Analytical framework

Educational standards provide guidelines for students’ expected outcomes and are often reflected in curricula and descriptions of assessment in curricula. Both curricular standards and assessments have been found to shape and form school practice (Binkley et al., 2012). The assessment of CT is one area in which curricular standards and teacher practices intersect. There are no evident theories that connect CT assessment and the curricular demands that teachers face. Therefore, we draw on two analytical frameworks, one for the integration of CT in education by Tannert et al. (2022), and one for assessment (Black & Wiliam, 2009; Wiliam & Thompson, 2008). This allows for the consideration of the curricular and pedagogical aspects of CT while combining them with existing theories of assessment.

3.1. Assessment

In educational settings, assessments are essentially inferences drawn from the subject outcomes (Black & Wiliam, 2018). If the inference focuses on the actions needed to help the student learn in the future, the assessment is formative, and if it is an inference that relates to the current status of a student, it is summative (Black & Wiliam, 2018). This means that how the information provided by an assessment is used distinguishes formative from summative assessment; thus, in theory, the same assessment instrument can be used for either summative or formative purposes (Dixson & Worrell, 2016). The purpose of summative assessment is to evaluate learning outcomes, and the assessment is often formal and conducted after a period of instruction. By contrast, the purpose of formative assessment is to improve teaching and learning and to diagnose student difficulties, and it is usually informal and ongoing (Dixson & Worrell, 2016). However, all formative assessments entail summative assessments, and the boundaries between these can be blurry in practice.

Formative assessment entails information used to decide the further steps in instruction to improve student achievement (Black & Wiliam, 2009; Wiliam, 2011) and provide information about the activities that will likely improve performance (Wiliam, 2011). Although practices and interpretations of the term vary to some degree, there is evidence that formative assessment enhances student learning (Hattie & Timperley, 2007; Wiliam, 2011). In this study, we use the framework of formative assessment introduced by Wiliam and Thompson (2008). This entails the following assessment processes:

• The teacher clarifies and shares learning intentions and criteria for success.

• The students understand and share learning intentions and criteria for success.

• The teacher enables classroom discussions and tasks to reveal student understanding.

• The teacher provides feedback that guides the learner forward.

• The students are activated as resources for another.

• The students are activated as owners of their own learning.

Formative assessment involves teachers and learners, and guides the learner into understanding the current status of their learning, what the aim or goal is, and the steps needed to reach the goal (Wiliam & Thompson, 2008). In this study, we consider assessment from the perspective of the teacher; for example, the teacher clarifies and shares learning intentions and criteria but can also facilitate students’ activation as resources for another.

3.2. CT integration and assessment framework

CT can be integrated into schools as a subject of its own (such as computer science, see Lodi and Martini (2021)), or through integrating CT into or across existing subjects (Tannert et al., 2022). Integrating CT into subjects means that CT is included in one or several subject curricula, for example, mathematics, science, or art, while integrating CT across subjects means that it is a transversal skill that is an overarching goal across subjects (Bocconi et al., 2022). Challenges in CT assessment arise from its broad interpretation and the various roles it is given in curricula. Therefore, the different aspects of CT integration were analytically divided into three parts. First, in assessing CT, teachers need to relate their practice to national curricula, while the pedagogical choices are guided by the teachers themselves. Second, they need to decide whether the emphasis is on the subject in which CT is integrated, or if CT is subject of assessment. Third, the teacher faces a choice about which aspects of CT that are assessed. To consider all the dimensions in play, we developed an analytical framework, inspired by Tannert et al. (2022), who outlined issues of integrating CT into curricula. Our framework is an adapted version of Tannert et al. (2022) framework, where the focus is on CT integrated in and across subjects, allowing for organizing our findings while accounting for multiple aspects of CT in schools.

The analytical framework (Figure 1) describes how CT knowledge is connected to the teaching of CT. The framework comprises two dimensions: a curricular-pedagogical dimension on the vertical axis, and CT as a primary or secondary objective on the horizontal axis (Figure 1). Within the two dimensions, CT knowledge is divided into two different compartments: core CT knowledge and applied CT knowledge.

Figure 1. The analytical framework, inspired by Tannert et al. (2022).

3.2.1. Curricular level – pedagogical level

For teachers, the national curriculum is a framework for teaching CT, but within a classroom setting, many processes are in play. This is described by the two levels on the vertical axis. The curricular level depicts curricular aspects of CT, such as syllabus, learning goals, and assessment standards. The pedagogical level describes the instructional and pedagogical choices the teacher makes, such as how to teach and assess CT. These two levels illustrate the different dimensions that teachers face when CT is introduced in schools.

3.2.2. Core and applied CT knowledge

CT knowledge is divided into two strands: core CT knowledge and applied CT knowledge. The former is directly related to coding and programming, while the latter means that the core CT knowledge is applied, for example, in innovation or problem solving. Tannert et al. (2022) presented a third level, humanistic knowledge, entailing values and ethical, emotional, and cultural competencies. However, humanistic knowledge is not subject of CT assessment in the Norwegian and Finnish curricula as such (FNBE, 2014; NDET, 2019) and are thus omitted. Informed by the curricula (FNBE, 2014; NDET, 2019), we apply a broad understanding of applied CT, which includes elements such as creativity and perseverance, and which may be relevant for assessment and feedback.

The CT definitions in Section 2.1 are mapped into the analytical framework. The first category of definitions, “programming and computing concepts” (FNBE, 2014; NDET, 2019; Tang et al., 2020), are to be considered core CT, while the second category, “competences that are needed in domain-specific knowledge and general problem-solving skills”, is considered applied CT knowledge.

3.2.3. Primary and secondary objectives

The horizontal axis of the framework describes the objectives of teaching CT. Primary objectives refer to CT knowledge as the foregrounded goal of teaching, while secondary objectives refer to the CT knowledge being used with the aim of learning the subject, such as mathematics. This axis acknowledges that even when integrating CT across or within subjects, the objective of the learning situation can be to develop CT skills and knowledge, for example, programming or debugging strategies. CT as a primary objective is exemplified through the fifth-grade competence aim of the Norwegian mathematics curriculum, “create algorithms and express them using variables, conditions and loops” (NDET, 2019), in which CT is foregrounded. Secondary objectives account for applications of CT as a means of learning the subject, for example, simulating or modelling a science phenomenon or exploring mathematics problems (NDET, 2019). The objective is therefore the subject knowledge itself, and not the CT skills specifically.

Adapting and applying Tannert et al. (2022) framework on assessment offers a tool for characterising what kind of CT knowledge is being assessed (CT integration framework) and how it is assessed (assessment theory). This situates the analysis of teachers’ understanding of CT assessment in the context of different dimensions that teachers need to navigate: the curricular-pedagogical dimension, the objectives of teaching CT, and the CT knowledge itself.

3.3. The cases of Norway and Finland

Finland introduced CT in the curriculum in 2014 (effective from 2016), and Norway introduced CT in the curriculum in 2020. Studying the approaches of the two countries provides perspectives on both CT as a subject integration and as a cross-curricular theme.

In the Norwegian curriculum, CT¹ and programming are not separate subjects but are integrated into the curricula of mathematics, science, music and arts, and crafts (NDET, 2019). CT is mentioned specifically only once in the mathematics curriculum, in the core element “Exploration and problem solving”, while programming is the term used throughout the rest of the curriculum. In mathematics, students are introduced to the basic concepts of programming (loops, conditions, and variables) and the use of programming in problem solving. These skills are to be transferred to arts and crafts, music, and science, all of which contain competence aims involving programming and CT. Although the Norwegian curriculum uses the notion of programming, it has a broader meaning than coding and comprises explorative and creative problem solving. Although assessment is given a prominent place in the curriculum, the guidelines are however very vague and do not offer prescriptions on how to assess CT. In addition to the curricular aims, the Norwegian Directorate of Education provided a supplementary material to the curriculum, outlining a CT definition based on Barefoot Computing (2016). The definition includes the CT concepts of logic, algorithms, decomposition, patterns, abstraction, evaluation alongside the approaches of tinkering, creating, debugging, preserving, and collaborating.

The Finnish core curriculum includes CT in various forms as an aspect of 21^st century skills. In the Finnish curriculum, there is no single “CT definition”. However the Finnish curriculum includes descriptions of both subject-specific and transversal 21^st century competences (Korhonen et al., 2022). Programming is mentioned as one aspect of transversal competency related to ICT, and additionally it is mentioned in the mathematics and crafts curricula. For example, one of the 20 learning objectives in grades 7–9 mathematics is to guide students in developing algorithmic thinking² in applying mathematics and programming to solve problems. Thus, despite the lack of an explicit curricular CT definition, the Finnish curriculum includes programming and CT as an aspect of development of 21^st century skills. To support the implementation of the curricula, an additional resource for teachers was developed (National Audiovisual Institute, & Finnish National Agency for Education, 2022), in which programming skills are defined as (1) computational thinking,³ (2) explorative work and production, and (3) knowledge of and operation in programmed environments.

Overall, the two countries illustrate how CT can be integrated when it is embedded in existing subject disciplines. Several aspects in the Finnish curriculum emphasise 21^st century skills, while programming also has a central role in the mathematics objectives. “The Computational Thinker”, based on the Barefoot Computing (2016) definition by the Norwegian Directorate for Education and Training, defines CT more explicitly, including several concepts and practices present in Shute et al.'s (2017) definition. Both curricula therefore emphasize CT as a competence in both domain-specific and general problem-solving skills, with additional resources and policy documents that strongly focus on programming skills.

4. Research design

This paper draws on two sets of data from Norway and Finland. This study is part of a larger Nordic project on CT in K–12 mathematics and science education. The project is a design-based research project (DBR), and this study is part of the initial problem analysis (McKenney & Reeves, 2018). The data used in this paper were collected through one focus group interview with Norwegian teachers participating in the project and seven semi-structured individual interviews. Four lower secondary school teachers from Norway were interviewed individually as well as three lower and upper secondary school teachers in Finland (Table 1). Three of the teachers (two from Norway and one from Finland) were part of the project. The other teachers were recruited through the teachers who were in the project and through professional development courses. This set of interview participants is not an “average teacher” selection, but participants were recruited due to their interest in CT, thus being able to elaborate on the topic of CT assessment. The interview protocol was piloted with one teacher, but no significant changes were made to the protocol after the pilot; therefore, the pilot interview was included in line with the other interviews.

Table 1. Overview of participant teachers.

Pseudonyms	Country	Subjects	Grades	Experience (years)
Eddie	Finland	Physics, mathematics, biology and chemistry	Lower and upper secondary school	9
Peter	Finland	Mathematics, physics	Lower secondary school	15
Aaron	Finland	Mathematics, IT, STEAM	Lower and upper secondary school	8
Martha	Norway	Science, mathematics, arts and crafts, physical education	Primary and lower secondary school	8
Mikaela	Norway	Mathematics, science	Lower secondary school	4
Stephen	Norway	Mathematics, arts and crafts, programming	Lower secondary school	2
Tori	Norway	Mathematics, science, programming	Lower secondary school	10

The group interview and individual interviews with the Norwegian teachers were conducted in Norwegian, while the individual interviews with the Finnish teachers were held in Finnish, allowing for the teachers to provide reflections in their native languages. The interviews were conducted online, lasting between 40 and 60 minutes, and audio recordings of the interviews were transcribed verbatim. All authors understand Norwegian. The first author is fluent in Finnish and analysed the transcripts in Finnish, while the other authors accessed transcripts that were translated from Finnish to English.

4.1. Data analysis

The analysis process is inspired by thematic coding (Braun & Clarke, 2006, 2019), guided by “finding patterns of shared meanings across the data set” (Braun & Clarke, 2019, p. 592). The analysis of qualitative data commonly consists of preparing and organising the data, reducing the data into themes through a coding process, and finally representing the data (Creswell, 2013, p. 180), a strategy we followed in the analysis. All authors read through the transcripts and familiarised themselves with the data. The unit of analysis was agreed to be segments of text addressing a specific meaning spanning from short sentences to longer statements. Thereafter we started with open, inductive coding, guided by the research questions of exploring teachers’ understanding of CT assessment and identifying opportunities and challenges. The objective was to code with proximity to the data, remaining open to multiple theoretical directions. Two of the authors coded the data independently. After the coding the analysis included two steps, the first was to form categories of the codes, and the second was grouping the categories into themes. A requirement for a category was that it was found in the transcripts in more than one interview, and that all authors agreed that this was a category.

The process of categorisation involved several iterations and discussions among the authors. Current CT definitions and assessment literature did not comprehensively cover the topics discussed by teachers in our study, such as the type of CT knowledge, curricular and pedagogical tensions and objectives in subject integration. We therefore developed the analytical framework (see section 3.2, Figure 1) and used it to interpret and organize the categories into larger themes. The framework thus provided a structure for grouping the categories into themes and for presenting and discussing the findings. An overview of the themes, categories and corresponding codes is outlined in Table 2.

Table 2. The themes, categories and code examples of the data analysis process.

Theme	Category	Codes (examples, not exhaustive)
Assessing core CT	whether the code works	debugging, end result, using correct programming structures
	level of abstraction	perfecting code, differentiation between “good” and “bad” code, perfectioning, hard coding
	students’ understanding of code	explaining code, commenting code, self-reflection about code, flow charts
Assessing applied CT	problem decomposition	analyse problem, real-life problem, algortihmic steps, break down problem, instructions
	problem solving	applying across contexts, applied tasks, transfer of knowledge
	collaboration	ask friends, listen to peers, group work, student discussions
	creativity	creativity, several correct solutions, personalising the project
Subject integration	integrating CT into subject	exams test programming, vague curriculum, change in mathematics teaching
Assessment and curricula	curricula	curricular operationalisations, exploration, guiding documents, learning trajectory
	exams	exams steering, misalignment, criteria, no exams
	assessment strategies	self-reflection, observation, interacting with students, projects
	assessment criteria	task-specific, commenting code, observation, feedback, student engagement, difficult to set

5. Findings and analysis

In this section, the findings are presented through the lens of the analytical framework (Figure 1). First, we present the findings on how teachers understand the assessment of CT through their reflections on their practice (Section 5.1). This section is further structured following the main themes from the data analysis, namely assessing CT as core CT knowledge (Section 5.1.1) and as applied CT knowledge (Section 5.1.2). Second, the teachers’ reflections on the opportunities and challenges related to the assessment of CT are presented (Section 5.2). These include Subject integration (Section 5.2.1) and Assessment and curricula (Section 5.2.2).

5.1. Teachers’ understanding of CT assessment

The teachers’ understanding of their own assessment of CT shows a partition into two themes, namely assessing the core CT knowledge and assessing applied CT knowledge. These two themes create a tension in the type of CT knowledge assessment can and should focus on, whether being the core CT knowledge closely related to programming skills, or the more intricate matter of assessing applied CT knowledge.

5.1.1. Assessing core CT knowledge

Our findings indicate that teachers’ understanding of assessing CT was connected to assessing core CT knowledge. The following categories were identified as categories of assessing core CT knowledge: whether the code works, the level of abstraction, and students’ understanding of their own code.

Whether the code works was discussed by three of the teachers (Martha, Tori, and Peter). Martha reflected on the formative and summative assessment of CT with respect to the process and the result:

[…] formative assessment is more like walking around and looking at the work being done. Are they on the right track or not? See and adjust a bit, and give some hints. I think the assessment part is quite simple, at least in the end. It either works, or it doesn’t. At least in some tasks.

Martha’s reflections on whether the students are “on the right track” depict practices of formative assessment, which include observing, adjusting, and giving hints based on observations. Such forms of feedback and instruction were mentioned by several of the teachers (Martha, Aaron, Stephen, and Peter). Ensuring that students are on the right track indicates an understanding of formative assessment as a way of guiding them forward and improve their performance. However, in the end, Martha highlighted that the assessment was based on the result and whether the code was correct – that is, a summative assessment. Tori said that she looked for whether the students “[…] manage to perfect their programming code so that it works exactly as they want, without bugs”. Assessing the end-product and whether the code works represent assessing cumulated core CT knowledge, thus being summative assessment.

A code that works is most often a result of debugging, as mentioned by several teachers (Stephen, Tori, and Eddie). Stephen reflected that debugging was the easiest aspect to assess. Tori highlighted debugging as an important element in CT assessment:

A bit like in math: there’s a mistake, and I just check the answer, or I just use the calculator. While here [in programming], it’s like there’s a mistake, but why? You must find out for yourself. Managing to debug a little yourself is actually part of the assessment criteria or competence goals in programming.

Tori compared debugging to processes in mathematics assessment, in which students check the answer or use the calculator, and in which clear criteria enable students to find their own and each other’s errors. Having the students discover their errors is a way of activating students as resources for each other and as owners of their learning. Assessing debugging can supplement the assessment of whether a code works, and the iterative nature of debugging can provide a fruitful environment for the assessment of CT. It is considered an effective approach to CT assessment (Zhong et al., 2016), but by itself, it is insufficient for assessing CT.

The second finding captures the teachers’ reflections on what a good code is, or the level of abstraction. In this context, it refers to various aspects of the functionality of the code and as such can be regarded as an element of core CT knowledge. The teachers in this study had different interpretations of what constituted a “good” code, from “perfectioning the code and eliminating bugs” (Tori) to “[…] the perfect code is somehow the shortest code […] or the one that can be done with the fewest commands […] but that they [students] experience that it can be done in many ways” (Mikaela). Mikaela’s interpretation of a “perfect code” also highlights an interesting aspect in which creativity and code quality are mutually enhanced. Stephen differentiated between completing the task by, for example, hard coding and using a more sophisticated, automated, and generalised algorithm, thereby assessing the level of abstraction. To a certain extent, understanding CT assessment as assessing levels of abstraction allowed the teachers to differentiate between performance regarding basic CT knowledge, but they expressed that they found it challenging to differentiate between high and low achievement.

Furthermore, our findings indicate that teachers’ understanding of the assessment of CT is closely related to the assessment of students’ understanding of code. Several of the teachers (Stephen, Martha, and Aaron) encouraged students to reflect on how they had solved tasks, and to explain their actions. Stephen said that he asked students to write about “What do you think that you mastered? How did you solve the sub-goal? What did you perhaps not really master?” Furthermore, Stephen illustrated that the students were, for example, asked to explain the purpose of a given function: “They are in a way to explain what this function in my algorithm does, for example, and what the purpose of the function is and how it works”. Mikaela outlined similar methods for assessing students’ CT skills but stated that she was unsure of how to set the criteria for a good code and how to assess if the answer was correct. Two of the Norwegian teachers (Stephen and Tori) also brought up students’ commenting their own code, indicating that it was important not only to produce the result but also to be able to explain and reason. In outlining how they assess CT, the teachers emphasised producing a correct result or a code that runs as well as uncovering the underlying understanding of the students. Commenting and explaining code can provide a means for both formative and summative assessments, depending on when in the process they are utilised, and can provide insights into whether students have understood the content.

The use of real-world problems and flow charts were frequently mentioned as tasks that revealed students’ understanding of the code. Translating a real-world problem into an algorithm or flow chart (Eddie) and further interpreting a flow chart in terms of the code was recognised by several teachers (Stephen, Aaron, and Peter) as a means of assessing CT. However, both Stephen and Peter also pointed out that a flow chart does not capture the richness of CT. Peter worried that limiting assessment to simple tasks is problematic:

Probably like making some tests, or something like making a flow chart, like “how would you break down this problem” and such mechanical work. But then I don’t know on a larger scale [if it makes sense]. It is … that’s probably the biggest challenge … assessing this [CT as a] whole.

Peter highlighted the role of flow charts in facilitating algorithm design and breaking down a problem (problem decomposition), and he recognised that these can be assessed through tests. However, he saw this as somewhat contradictory to assessing CT as a broader concept. Martha viewed the assessment of programming as focused on a correct result, while she described CT assessment as a formative assessment of the process. Stephen also problematised that assessing the program is not sufficient in assessing CT. Similarly, Peter and Stephen highlighted a discrepancy between CT as both core CT knowledge and a broader problem-solving competence, which, as Peter pointed out, is more challenging to assess. Stephen, Aaron and Peter all expressed confidence in programming, which indicates that this issue is more far-reaching than the teachers’ need for more experience with programming. Rather, it points towards a need to address both what CT entails and how it can be assessed if it is interpreted as broader than coding. Teachers’ understanding of CT influenced its assessment. They understood CT as broader than coding or programming; therefore, its assessment became challenging and a tension between core CT and the following section of applied CT knowledge can be discerned.

5.1.2. Assessing applied CT knowledge

Applied CT knowledge represents the application of core CT skills. The first two categories, problem decomposition and problem solving, represent aspects of CT that can require the application of various types of core CT knowledge. The two other categories cover the more generic approaches of creativity and collaboration, which relate to applying core CT knowledge in a broader context.

Several teachers highlighted problem decomposition as an aspect that is assessed in CT. Stephen mentioned that he often broke up a problem into sub-problems, as he experienced that the students had a hard time solving a problem that was not decomposed in some way. Dividing a larger problem allows for the assessment of sub-tasks. However, he added that he could perhaps let go of some control and try to assess problem decomposition instead. Learning programming is not easy, and Stephen’s reflections indicate that there might be some tension between learning to decompose a problem and not making the task too difficult for the students. Decomposing a problem for students may represent a challenge when designing tasks that facilitate the assessment of the decomposition process. As an example of a task, Eddie mentioned decomposing an everyday problem into conditions and codes. In similar ways, several of the teachers considered problem decomposition to be a way of assessing applied CT knowledge. Problem decomposition can be interpreted as strictly related to coding but can also be viewed in a larger context, as it is inherent in all problem-solving processes.

Aspects of CT assessment beyond core CT knowledge share many similarities with problem solving and applying core CT knowledge. Eddie reflected on the transfer of knowledge to new contexts as a possible basis for tasks:

[…] Computational thinking is a skill that can be applied in a wider variety of situations, so then I think it would be interesting to see and evaluate how the students are able to apply it in such non-programming contexts, or in clearly non-mathematical contexts […]. Then it becomes much more challenging to assess what it [CT] should contain and what kinds of things should be revealed in students’ skills.

Eddie’s reflections above pose CT as applicable in different contexts, which is at the heart of applied CT knowledge. Aaron indicated that he used tasks that required applying core CT knowledge, and that he noticed that the students were “applying what they have just learned somewhere, like in some new context”. This indicates that a design that changes the context of the task can facilitate the transfer of knowledge and provide a means for assessing applied CT knowledge. However, this suggests that the construct of assessment is unclear in applied CT knowledge. If it is difficult to articulate what is being assessed, assessing it can become somewhat binary – being reduced to whether students can or cannot complete such tasks. In such situations, with unclear intentions and criteria, formative assessment can become challenging and may be reduced to more summative feedback.

Given that CT integration is new, it might be difficult for teachers to be explicit about and share intentions and criteria. This may obscure the formative assessment and lead both teachers and students to reach for clearer indicators, such as “whether the code works”. However, teachers are aware that these are insufficient measures when it comes to CT. The teachers discussed problem decomposition and the transfer of knowledge as prominent aspects of CT assessment. Well-designed tasks were recognised as a possible contribution to the assessment of the application of (core) CT knowledge.

The teachers brought up several aspects that are not unique to CT but are, however, an inherent part of the CT process. Four of the teachers included creativity (Tori, Stephen, Mikaela, and Aaron) in discussing CT assessment. Tori’s students programmed a game, and along with the assignment, she also shared the assessment criteria. She described how she used the assessment criteria to encourage the students to show a higher level of creativity:

[…] You have a very low goal achievement on creativity. What can you do with it? Can you explore it? Can you make it a little bit your own? Help them [the students] a little bit towards higher goal attainment along the way.

Tori explained creativity in the process as making the game more personal. She linked creativity to the exploration of the task and explicitly included applied CT knowledge in the assessment criteria. She described how shared criteria and learning intentions can point students towards goal attainment and make creativity an objective of formative assessment.

Several of the teachers (Eddie, Martha, Aaron, and Peter) indicated that they designed the learning situation so that it would allow for collaboration. Collaboration was not directly mentioned as a construct of assessment by others other than Martha but was mentioned by several teachers as an aspect of working with CT and can thus facilitate formative and peer assessment. Martha viewed collaboration as the most difficult aspect of CT to assess, exemplifying criteria such as “ … Did you make a good effort? Were you good at listening to others?” She contrasted the assessment of the end-product or result with assessing collaboration, the work process, and listening to peers, indicating that these were important aspects of CT but are more difficult to capture in a CT assessment. Collaboration allows for activating students as resources for one another. One question that arose here is whether collaboration needed to be assessed, or whether it could be a way of working with CT but not explicitly as a construct to be assessed.

In summary, applied CT knowledge, was perceived as more difficult to assess in comparison to assessing some aspects of core CT knowledge. The findings add further insights into the tensions between assessing programming related CT and as a broad set of problem-solving skills.

5.2. Opportunities and challenges

The teachers navigated between the different levels of curricular demands, pedagogical approaches, and embedding CT in the subject. Several opportunities and challenges were identified through the teachers’ reflections on their CT assessment practices. First, we present and discuss findings in the realm of the assessment of CT as a primary or secondary objective (the horizontal axis in Figure 1). Second, we discuss these aspects with respect to the curricular and pedagogical realms (the vertical axis in in Figure 1).

5.2.1. Subject integration

The teachers’ reflections on assessment revealed interesting aspects about how they understood the connections between CT and the subject into which it was integrated. The Norwegian teachers were aware that the curriculum outlined that students were supposed to use the core CT knowledge in the subject, such as mathematics. Stephen explicitly addressed this tension:

There are some [resources], but often, they are not linked to mathematics in the way that one might want it to be in the context of mathematics. Or you find things that are partly related to mathematics but … ultimately not about that.

In his teaching design and assessment practice, Stephen addressed CT as a primary objective, while he recognised a need for assessing CT as a secondary objective – that is, assessing CT in the context of mathematics. He further criticised the standards indicated by the national exam, in which the task, including CT, asks students to interpret a flow chart and construct a program based on the diagram. He identified an incoherence between the exam task, which “tests whether students actually can write a computer program”, and the curriculum, which prescribes “exploring mathematical properties and relationships”. This identified incoherence is a direct example of the tension between primary and secondary objectives when integrating CT into a subject. The other teachers did not bring up this tension explicitly, but drawing on the findings in Section 5.1.1, we observed that several of the teachers understood CT assessment as assessing core CT knowledge. Mikaela’s interpretation of teaching CT, by contrast, illustrates a different approach. In her view, the connection between mathematics and CT is that “it [CT] is not in addition, but in a way intertwined. It actually enhances the teaching if you can do it right”. Although she did not strictly refer to assessment but to teaching, she perceived a connection between CT and subjects, which in theory could weaken the dichotomy of CT as either a primary or secondary goal.

CT as a primary or secondary goal in, for example, mathematics raises not only the question of what to assess but how to assess it. Several of the teachers (Peter, Aaron, Martha, and Tori) underlined the differences between the assessment of CT and mathematics. Peter observed that students were used to a specific way of working in mathematics: “[…] if, for example, they have been practicing math with a book and a notebook for six years, it may appear strange that the teaching of math becomes different”, implying that the explorative nature of teaching and learning CT differs from more traditional mathematics. Mikaela and Martha both emphasised that students were used to looking for answers at the back of the book. Martha said that “they [the students] are more concerned with getting it right”. Tori mentioned that the students were not interested in finding out why a solution might be right or wrong, while in programming, errors became evident. Thus, CT interfered with the students’ standard result-oriented approach. Not providing the answer to the students could allow or even motivate an approach in which students seek feedback and thereby are activated as resources for others and owners of their own learning. Changing the learning culture in mathematics can thereby allow for opportunities for formative assessment, although it is perceived as challenging by students.

The incorporation of CT into subjects urges the need for meaningful integration and well-designed tasks. Whether the assessments should focus on CT as a primary objective or on the subject, thus making CT a secondary objective, should be considered. However, CT integrated into subjects has the potential to foster a culture of exploration in the absence of correct results in textbooks.

5.2.2. Assessment and curricula

Teachers’ practice is directed by national guidelines such as curricula. At the same time teachers also have freedom to interpret curricula, design their own assessments and make pedagogical decisions. The curricula, exams, assessment criteria and assessment strategies presented both opportunities and challenges to the teachers.

The ambiguous formulations in the curriculum were perceived by many of the Norwegian teachers as both a challenge and an opportunity. Stephen considered it positive that the goals in the curriculum were vague, thus allowing for the teachers’ own interpretations. Mikaela’s opinions were in line with Stephen’s, and she thought that the goal of “explor[ing] mathematical properties and relationships by using programming” (NDET, 2019) is advantageous, since it aims not only at finding the correct solution but also at exploring the possibilities of problem solving with CT and programming. Using the concept of exploration further demonstrates the difficulties with interpretation during CT assessment. Stephen, being otherwise confident in programming, found that assessing CT was difficult due to the emphasis on exploration:

Have you explored well, or have you explored poorly, right? So you have to break it down into something more concrete and tangible then. [I: Yes.] And then you just say that we interpret that as exploring. Are you able to use variables? Are you able to use lists? Are you able to use if-else loops, and how are you able to use them?

Stephen operationalised the curricular goal of exploring mathematical expressions through assessing the use of variables, lists, and conditionals. Along with Mikaela, Stephen exemplified that he has to make choices that were guided by the curriculum but in the end were his interpretations.

Some of the teachers discussed exams. In particular, Mikaela and Stephen reflected on the role of national exams in Norway. As CT is relatively new to the curriculum, the teachers mentioned that it would affect their own assessment. Mikaela indicated that “there will be a central exam in tenth [grade] in a couple of weeks [I: mm], so I thought maybe I’d see what appeared there, [I: mm] then see if we put it on our own exam”. Stephen was even more explicit and said that it was important to teach how to analyse a flow chart because that was what the exam included with respect to CT. He pointed out that the curricular goals of exploring mathematical aspects with CT were not reflected in the national exam. This describes a situation where the curriculum and the exams do not reflect each other, and the teachers are left to navigate between curricular and pedagogical tensions.

Peter did not want to have centralised tests but related assessment in mathematics to the standard criteria in Finland. He said that the curriculum was quite open, but that the foundation of the assessment of CT was connected to the 21^st century skills. He stated that he did not wish to have more standardised exams but that an “arc or learning trajectory” would be helpful, accompanied by some criteria. Aaron, by contrast, said that he did not find assessment problematic, and since there was no regulation, his approach was to differentiate and assess through designing tasks that required applying CT knowledge. The two Finnish teachers’ reflections showed how curricular and pedagogical tension can be negotiated in different ways – through task design or through a more explicit learning trajectory and criteria.

One way of operationalising the overarching aims of CT is through assessment criteria. Some teachers developed their own criteria and were, to a varying degree, explicit on how they formulated them. Stephen operationalised the assessment of CT into his own criteria, emphasising core CT knowledge as the primary objective. Martha constructed criteria related to the result: how the Micro:Bit car should drive, but she offered no criteria directly related to core CT knowledge. Mikaela’s criteria for a “good code” are yet another example. Mikaela and Stephen also stated that setting their own criteria was a difficult task. Mikaela described a “good code” as “a short code”, while Stephen related it to levels of abstraction in code. Eddie emphasised independent work and the ability to analyse the problem at hand, and Peter focused on students’ honest trying. This reflects the fact that the lack of criteria drove the teachers to work with their own interpretations. Although teachers are given freedom, a lack of shared criteria and intentions may threaten transparency and coherence in assessments. CT assessment accentuates the demanding situation that teachers face when curricular goals are open to interpretation.

An opportunity identified in the teachers’ responses was that integrating CT as a criterion in mathematics assessment broadened the range of assessment. Aaron highlighted that some students who were good at programming but did not do so well on the mathematics tasks, achieved higher grades when programming was part of the overall mathematics assessment. Using programming as a compensation for weaker results in mathematics was also underlined by Martha, indicating that the introduction of CT in subjects may broaden the scope of competences and, consequently, assessment in the subject.

One important aspect of the pedagogical level described by the teachers is the variety of assessment strategies that they use in connection with CT and programming, most of which reflect versatile formative assessment strategies, such as debugging and commenting code. Several of the teachers described their assessment strategies with an emphasis on students’ self-reflection (Peter, Aaron, Mikaela, and Stephen), which suggests that the teachers activate the students as owners for their own learning. Observation of students’ work and interaction in groups was also mentioned by two of the teachers (Eddie and Aaron). Many of the teachers engaged in dialogue with students through feedback, both in connection to formative and summative assessments. Aaron and Mikaela described that an indicator of students’ learning is that they are excited about the tasks. Aaron emphasised that one goal of learning CT is that the students get an experience of working with CT and being enthusiastic, and that the learning will be a by-product of this experience. The focus on student engagement further emphasises that teachers work on engaging students to take responsibility for their learning. Aaron also emphasised that assessing CT is a qualitative task, which indicates a strong focus on formative assessment. However, most teachers indicated that rich formative assessment requires time and may be difficult to realise. Overall, the teachers used different forms of assessment and look for versatile signs of learning when assessing CT.

6. Discussion

The aim of this study was to explore how teachers understand the assessment of CT through reflections on their own CT teaching practices when CT is embedded in subjects in lower secondary schools in Norway and Finland. Earlier studies have shown that teachers find CT a difficult topic to teach (Cabrera, 2019). Our findings indicate that teachers also find the assessment of CT difficult to varying degrees and that the vagueness of the national curricula resulted in different implementations of CT assessment. Li et al. (2020) pointed out that the focus on research on CT in schools has been dominated by a computer science-oriented context, and despite efforts to conceptualise CT in the STEM fields (Weintrop et al., 2016), the research about CT integrated into non-computer science disciplines is still in development. As the results of this study demonstrate, the assessment of CT integrated in subjects is thus also an emerging field.

The integration of CT into subjects allows for an interpretation of CT beyond programming. Our findings indicate that several of the teachers shared the understanding that CT was broader than programming. However, it was difficult for teachers to put this into practice, and assessing whether the code worked along with levels of abstraction were common assessment practices. A pertinent question is what can be assessed beyond core CT skills. While describing CT as a real-world problem-solving skill is easy, assessing these skills punctures the abstract discourse and makes it evident that teachers need tangible examples. The teachers in this study did not want standardised tests or quizzes, which are common ways of assessing CT (Tang et al., 2020). The teachers focused on the need to understand the progression of CT as a trajectory throughout the school years. Furthermore, the Norwegian teachers focused on the need to understand how to assess the specific components of the CT definition, such as decomposition. Our results highlight the need for a deeper understanding of what the integration of CT in subjects entails to allow for its assessment by teachers.

The teachers’ understanding of CT assessment can be described as a continuum that ranges from assessing coding, through higher levels of abstraction, such as problem decomposition, to aspects such as problem solving, creativity and collaboration. Teachers are thus torn between assessment of programming related concepts which are considered more concrete and easier to evaluate, and practices and approaches like collaboration, which they want to assess but were unsure about how. Lai and Wong (2022) and Pérez (2018) illustrate how collaboration and other dispositions support learning CT and the integration of CT in subjects. Our findings highlight that acknowledging CT as more than programming makes assessment of CT a complex task. Facing the variety of broad CT definitions and CT’s ambiguous relationship to programming (Tikva & Tambouris, 2021), teachers act as interpreters of CT. Viewing CT as a continuum is in line with teachers’ understanding of CT, and the CT continuum can be utilised when designing assessments, thereby helping them grasp the concept and eventually becoming confident in assessing it.

Some of the difficulties teachers face when teaching CT have been identified in previous studies. Teachers’ preparedness to teach CT is affected by the misalignment of assessment and curriculum (Irons & Hartnett, 2020) and teachers lack support in making curricular decisions (Korhonen et al., 2022). Additionally, teachers’ curricular needs should be accounted for in teaching CT (Yadav, Hong, et al., 2016). In our study, we observed similar tendencies. Norwegian teachers are, according to the curriculum, to assess students’ exploration through programming, which is an example of what we have called applied CT, but how this should be assessed is not outlined. Washback from national exams can become prominent (Holm & Kousholt, 2019), and as the findings in this study suggest, the vagueness in curriculum might enhance this effect, as teachers look to the exams to prepare their students to achieve good grades. Meanwhile, our findings also show that teachers appreciate the open curricular descriptions of CT. Burying teachers in lists of criteria might be problematic as well, especially when working with 21^st century skills. At the heart of assessing 21^st century skills, the objective should not be to only achieve pre-determined goals and “working backwards from goals” but discovering new objectives (Scardamalia et al., 2012, p. 231). This study highlights the disconnection between curricular goals and national exams, and the teachers’ understanding of the assessment of CT. This resulted in a tension that the teachers had to negotiate, leading to different emphases on the CT constructs that were assessed. In the Nordic countries, teachers traditionally have professional freedom and autonomy (Mausethagen & Mølstad, 2015) and national curricula are not highly detailed. Thus, detailed curricula might threaten teachers’ professional autonomy to make judgements about assessment. However, detailed assessment guidelines might ameliorate the problems teachers face with assessing CT. One could argue that when a new concept, such as CT, is introduced, clearer assessment descriptions could facilitate assessment practices. Furthermore, detailed descriptions may contribute to ensuring transparency and equal opportunities for students as well as cohesive assessment. Clear assessment descriptions can be viewed as part of operationalizing the broad CT concept, which could help teachers in their practice.

The Finnish teachers’ understanding of CT assessment differed somewhat from the Norwegian teachers’ understanding. One of the Finnish teachers who had a strong background in computer science from the university perceived CT assessment as a qualitative and process-oriented task with a strong focus on formative assessment. He was the only one of the teachers who explicitly said that the CT assessment was not difficult. This aligns with the findings from Kalelioglu and Sentance (2019), who found that more experienced teachers relied on observation. Another of the Finnish teachers discussed CT and CT assessment primarily through the lens of 21^st century skills, highlighting a change in the process of learning, assessment, working with problem solving, and transfer of learning. These somewhat differing understandings of what CT assessment entails might be a reflection of not having a “national definition” such as the Norwegian teachers have. However, as we see from the Norwegian teachers’ experiences, a definition does not provide much support for assessment if the meaning of the definition is not contextualised. We agree with Yadav et al. (2017) that the meaningful integration of CT into curricula and practice requires teacher educators to provide future teachers with content pedagogy and instructional strategies. Vinnervik (2023) raised concern about how lacking professional development coupled with vague curricular guidance is concerning. We agree with this and conclude with adding that is a need to educate teachers on what the elements of CT entail and how to interpret them, even when a CT definition is used.

An issue that can be raised with regard to the assessment of CT is whether to focus on the CT process or the product. Focusing on assessing the end-product – that is, the project or code – has been recognised as problematic in CT (Fagerlund et al., 2020). Some of the teachers viewed CT as a process in itself, and that assessing it comprised formative assessment with observation, guidance, and feedback. Other teachers focused more on the content of the process, such as problem decomposition, and how this should be specifically assessed. CT has several procedural elements, and the connection to process was also prominent in the teachers’ descriptions of CT assessment. This was identified as differing from a traditional mathematics setting, in which there is an answer in the back of the textbook. Several of the teachers observed that formative assessment was what enabled “genuine” learning of CT. As demonstrated by the teachers’ reflections in this study, conceiving CT as almost a formative process of its own makes it an especially fruitful environment for fostering a formative assessment culture. A possible way forward is to further develop practices and frameworks for formative assessment of CT, as Grover (2021) suggested, starting with formative assessment of core CT knowledge, and thereafter broadening the framework to encompass formative assessment of applied CT knowledge.

Exploring teachers’ understanding of CT assessment helped us identify some challenges and opportunities they face. Assessing CT beyond coding or core CT knowledge was perceived as difficult by most teachers. The assessment through tasks that require applying knowledge, along with encouraging higher-order thinking, were identified as an opportunities in connection with the assessment of CT. This implies that designing tasks where core CT knowledge is applied allows for the assessment of CT as a broader concept. Designing tasks that include both mathematics and CT and require applying knowledge in a way that is meaningful and characteristic for the subject could therefore be one answer to how to assess CT within the mathematics subject. Since CT aspects, such as abstraction, generalisation, and problem decomposition, are often intertwined in the process, assessing CT through task design is one viable option. Open-ended tasks in which the assessment is performance-based could align with the inquiry-based nature of CT (Yadav et al., 2015). This, however, can require some extensive work with designing tasks that fully account for CT and the subject into which it is integrated. The design and assessment of such tasks may find support in more subject-specific CT frameworks, such as that of Weintrop et al. (2016), and represents a different approach compared to the more generic tasks studied by, for example, Román-González (2015).

This study presents a state of the field in two Nordic countries, captured through the teachers’ lens. Although Nordic teachers find programming and CT both meaningful (Korhonen et al., 2022) and challenging (Nordby et al., 2022), this study sheds light on how teachers understand assessment and the opportunities and challenges the integration of CT presents. There is the need for future studies to focus on examining the practice of in-classroom assessment in order to characterise the type of assessment that teachers use in classrooms where CT is integrated into subjects.

7. Conclusion

Teachers, as practitioners, are an important voice in the discussion of the assessment of CT. Three contributions are made to the field of assessing CT from a teacher perspective. First, in creating an overview of teachers’ CT assessment practices, our findings indicate that teachers found core CT skills, often related to programming, easier to assess than applied CT. At the same time, they view CT as a broader concept than solely core CT skills. A concern is that some of the frequently used assessment methods cannot capture the full concept of CT, creating a tension in CT assessment. This is an element that needs to be taken into consideration in discussion and design of CT assessment. Second, our findings demonstrate that giving teachers a definition of CT, including established CT elements such as abstraction, problem decomposition, generalisation, does not necessarily translate into an assessment practice that is aligned with a broader understanding of CT. This highlights the need for CT definitions to be coupled with professional development in the assessment of CT. Finally, an issue that can be raised is whether, and how, the tensions highlighted in this paper can be harnessed in order to further support teachers in their CT assessment.

Acknowledgments

We thank the anonymous reviewers for their careful reading of our manuscript and their insightful comments and suggestions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by The Research Council of Norway under Grant [320322]

Notes on contributors

Aino Ukkonen

Aino Ukkonen is a PhD candidate at the Faculty of Education and International Studies, Oslo Metropolitan University, Norway. Her research interests are within computational thinking, mathematics education and computer science education, especially the integration of programming into existing subject disciplines. She has worked as a researcher with mathematical modeling in environmental and transport sciences with published work within these fields.

Katarina Pajchel

Katarina Pajchel is an associate professor at Oslo Metropolitan University, Norway. Her main research interest is science education, teacher education, including research on research-based teacher education, programming, and computational thinking in science education.

Louise Mifsud

Louise Mifsud is a professor at the Faculty of Education and International Studies, Oslo Metropolitan University. She holds a PhD in Education from the University of Oslo. Her research and teaching interests are in the areas of technology enhanced learning in schools and teacher education, more specifically cyber ethics and computational thinking. She is currently lead PI on a Norwegian Research Council funded project on computational thinking in schools and teacher education. She has published several peer-reviewed articles, book chapters and conference papers within the field of technology enhanced learning.

Notes

1. The Norwegian term used is algorithmic thinking (“algoritmisk tenkning”), but the English version of the curriculum uses the term computational thinking.

2. The Finnish translation is “algoritminen ajattelu”.

3. The Finnish word is “ohjelmoinnillinen ajattelu” and the Swedish word used in Finland is “datalogiskt tänkande”.