Unravelling the numerical and spatial underpinnings of computational thinking: a pre-registered replication study

ABSTRACT

Background

Key to optimizing Computational Thinking (CT) instruction is a precise understanding of the underlying cognitive skills. Román-González et al. (2017) reported unique contributions of spatial abilities and reasoning, whereas arithmetic was not significantly related to CT. Disentangling the influence of spatial and numerical skills on CT is important, as neither should be viewed as monolithic traits.

Objective

This study aimed (1) to replicate the results of a previous study by Román-González et al. (Computers in Human Behaviour 72), and (2) to extend this research by investigating other theoretically relevant constructs. Specifying the contribution of reasoning (i.e. numerical, figural), numerical skills (i.e. arithmetic, algebra), and spatial skills (i.e. visualization, mental rotation, short-term memory) helps to better understand the cognitive mechanisms underlying CT.

Method

We investigated a sample of 132 students from Grades 7–8 (age range 12–15 years). Participants completed the Computational Thinking test, as well as a variety of psychometric assessments of reasoning, numerical, and spatial skills. To determine which cognitive skills are relevant for CT, we calculated bivariate correlations and performed a linear regression analysis.

Findings

Results confirmed unique contributions of figural reasoning and visualization. Additional variance was explained by algebraic skills.

Implications

We conclude that CT engages cognitive mechanisms extending beyond reasoning and spatial skills.

KEYWORDS:

• Computational thinking
• cognitive abilities
• computer science education
• numerical processing
• spatial processing
• reasoning

Introduction

There is a growing interest in teaching children computational thinking (CT) to prepare them for the demands of our increasingly digital society. Key to optimizing the instruction of computational thinking is gaining a precise understanding of the underlying cognitive skills. To this aim, the current study addressed the relation between CT and reasoning, as well as spatial and numerical skills in secondary school students. We set out to replicate and extend findings by Román-González et al. (2017): specifically, we aimed to 1) replicate parts of the original study, namely the contribution of figural reasoning, visualization and arithmetic performance, as well as gender differences, and 2) extend this research by investigating the contribution of numerical reasoning, algebraic skills, mental rotation and visuo-spatial short-term memory. Additionally, we wanted to determine whether gender differences in CT can be explained by gender differences in mental rotation and other cognitive skills.

CT refers to a set of cognitive problem-solving skills supporting the acquisition of programming skills. It was conceptualized as a general set of skills suitable for solving a wide range of problems, including everyday tasks (Wing, 2006). As an umbrella term, it has been characterized as the ability to identify, analyze, decompose and solve problems (Mohaghegh & McCauley, 2016). In an effort to identify common characteristics of previous definitions, Shute et al. (2017) defined CT as “the conceptual foundation required to solve problems effectively and efficiently (i.e. algorithmically, with or without the assistance of computers) with solutions that are reusable in different contexts” (p. 142). However, there is still no clear consensus about the precise nature of these cognitive problem-solving strategies, and there are only few reliable and valid tests available to specifically assess CT in children and adolescents. Consequently, the last years have seen an increasing demand for methodological rigor in the field of computer science education research (e.g. Margulieux et al., 2019). To advance theoretical accounts of CT, it is important to unravel the nature of its underlying cognitive mechanisms. Increasing our understanding of the specific cognitive processes underlying CT is also highly relevant from a practical viewpoint if we aim to foster CT abilities in various age groups and student populations.

Recently, Román-González et al. (2017) took an important first step by investigating the contribution of different cognitive abilities to CT. While most studies on the cognitive predictors of CT included only small samples or focused on single predictors, Román-González et al. (2017) conducted a large-scale study with students from Grade 5 to Grade 10 (age range 10–16 years). Their primary aim was to collect normative data for their novel, theory-driven Computational Thinking test (CTt), but they also conducted a validation study by investigating the association between CTt scores and four distinct cognitive predictors (i.e. numerical skills, spatial skills, verbal skills, and reasoning). The CTt’s test items address one of the following seven core computational concepts of computer programming: basic directions and sequences, loop repeat times, loops repeat until, if simple conditional, if/else complex conditional, while conditional, and simple functions. These concepts are based on some of the concepts proposed by Brennan and Resnick (2012) and the CSTA Computer Science Standards for Grades 7 and 8 (Seehorn et al., 2011).

Whilst an impressive sample of 1,251 students took part in the standardization of the CTt, the validation studies are based on smaller subsamples: The authors reported data from 135 students¹ completing the Primary Mental Abilities (PMA; Cordero et al., 2007, based on Thurstone & Thurstone, 1949), a large Spanish-language test battery assessing spatial, verbal, numerical abilities, as well as reasoning. Specifically, the spatial task provided a measure of visualization skills: Participants were shown an abstract two-dimensional figure and were asked to decide which of six other drawings correspond to this figure in two-dimensional space. The verbal task involved identifying the correct synonym of a given word out of four given options. The numerical task provided a measure of arithmetic performance: Participants had to quickly determine whether the given sum of three two-digit numbers was correct or not. Finally, the PMA reasoning task involved inductive reasoning: Participants selected a letter constituting the logical continuation of a series of letters. Moreover, 56 students completed an additional task (RP 30; Seisdedos, 2002): The RP 30 problem-solving test involved figural reasoning, spatial abilities and short-term memory: Participants were shown a series of five figures containing a number of geometrical objects together with a set of logical conditions (e.g. the number of white squares and black hexagons is not equal), and indicated the figures fulfilling the given logical conditions.

Román-González et al. (2017) reported moderate positive associations between CT and reasoning, as well as spatial skills. They also found a small significant association between CT and verbal skills. In contrast, the authors did not find a significant association between CT and numerical skills, leading them to conclude that numerical skills are not particularly relevant for CT. Entering performance on all cognitive skills assessed by the PMA in a regression analysis revealed that only reasoning and spatial abilities accounted for unique variance, whereas numerical and verbal abilities did not. Due to the small sample of participants completing the RP 30 problem-solving test, this assessment was not included in the regression model. However, bivariate correlations with the CTt were high. Lastly, drawing on their full standardization sample of 1,251 students, Román-González et al. (2017) reported small-size gender differences in favor of the male participants. Based on their results, the authors concluded that CT is fundamentally based on general mental ability, and, to a lesser extent, on specific cognitive skills such as spatial abilities.

However, from a theoretical point of view, neither reasoning nor spatial or numerical abilities should be viewed as monolithic traits (McGrew, 2009). They rather consist of distinct subskills that may have a differential impact on CT. We argue that CT can be linked to specific cognitive skills when theoretically relevant constructs are considered. Thus, we aimed to extend the original research of Román-González et al. (2017) by examining the specific contribution of reasoning, as well as spatial and numerical subskills on CT. Specifically, we first planned to replicate the original study by assessing the same cognitive skills, namely figural reasoning, visualization and arithmetic performance. In addition to that, we extended this research by investigating additional skills that are potentially highly relevant for CT, namely numerical reasoning, algebraic skills, mental rotation and visuo-spatial short-term memory.

Reasoning

In the well-established Cattel-Horn-Carroll model of intelligence (McGrew, 2009), reasoning or fluid intelligence (Gf) is defined as the ability to recognize structural relations between stimuli, understand implications, and draw logical conclusions (McGrew, 2009; Schneider & McGrew, 2012). As many researchers agree that CT can be described as a set of cognitive problem-solving skills (Shute et al., 2017), it appears reasonable to assume a close link between reasoning and CT. However, factor analytic research has indicated that reasoning is not strictly unidimensional, but is best modelled by means of a general mental ability factor and orthogonal factors reflecting figural, numerical, and verbal reasoning (Carroll, 1993; Wilhelm, 2004). For instance, numerical reasoning enables individuals to deliberately solve novel problems with numerical material, whereas figural reasoning enables individuals to recognize structural relations between spatial stimuli, understand implications, and draw logical conclusions (McGrew, 2009; Schneider & McGrew, 2012).

On an empirical level, there is ample evidence for an association between figural reasoning and CT and programming outcomes. In a recent study comparing expert programmers to a control group, figural reasoning was the only cognitive variable differing between groups: Programmers showed significantly higher figural reasoning skills than non-programmers (Helmlinger et al., 2020). Concerning the association with CT, Román-González et al. (2017) reported a significant association between figural reasoning and CT in secondary school students. In a younger sample from primary school, Tsarava et al. (2019) found a significant association between figural reasoning and CTt performance. In summary, empirical findings strongly support theoretical considerations that CT and programming are associated with figural reasoning.

Numerical reasoning may arguably be especially relevant for CT. Numerical reasoning is often assessed with number sequencing tasks which require understanding the regularities and principles of a number sequence. Numerical reasoning may be fundamentally linked to algorithmic thinking, which likewise involves identifying the structure underlying a certain problem. The original study by Román-González et al. (2017) employed a reasoning task requiring the continuation of a sequence of letters, and performance in this task was a unique predictor of CT together with spatial abilities. Similar to number sequencing tasks, this letter sequencing task can be considered as having a numerical component given that the rank or position of the letters in the alphabet formed the basis of the sequential organization (e.g. “a” is the first letter of the alphabet, “b” comes in the second position). However, when comparing programmers to a control group, Helmlinger et al. (2020) did not find any group differences in numerical reasoning. Given that empirical evidence is mixed, the present study aims to put the theoretical account of a specific contribution of numerical reasoning to an empirical test.

Although there has not been a lot of work on the relation between verbal reasoning and CT, neither theoretical frameworks nor empirical evidence suggest that this construct constitutes one of the core cognitive foundations of CT. Verbal reasoning is commonly assessed with a deductive reasoning task, requiring participants to draw a logical conclusion based on given verbal statements. Tsarava et al. (2019) reported a non-significant relation between performance on a verbal reasoning task and CT in children from primary school (Tsarava et al., 2019), and Helmlinger et al. (2020) did not find any group differences in verbal reasoning between programmers and non-programmers. Thus, we did not consider verbal reasoning as a predictor of CT in the current study.

Numerical abilities

In the Cattel-Horn-Carroll model of intelligence (McGrew, 2009), numerical abilities are represented within the quantitative knowledge (Gq) factor, representing an individual’s store of acquired mathematical knowledge. When formally introducing the concept of CT, Papert (1980) suggested to view it as a form of mathematical thinking, and that mathematics could be fostered by acquiring programming skills. However, more recent theoretical accounts clearly distinguish between mathematical thinking and CT, despite pointing out common processes (Shute et al., 2017; Wing, 2008). Furthermore, numerical skills such as arithmetic and counting were theorized to be conceptually unrelated to CT (Shute et al., 2017).

While on a theoretical level, it appears highly plausible that arithmetic should not be directly linked to CT, empirical evidence is mixed: While Román-González et al. (2017) did not find any association at all between mental calculations and CT performance, Tsarava et al. (2019) reported a medium-size relation between arithmetic fluency and CT in a small sample of younger children from primary school. However, this conflicting finding may be partly due to methodological differences: Tsarava et al. (2019) used an adapted version of the CTt which their participants had to complete in only 20 minutes. This adapted version included only 75% of the total amount of items of the original CTt by Román-González et al. (2017), but administration time was less than 50%. In the younger sample of primary school children reported in Tsarava et al. (2019), this may have provoked a strong load on processing speed. As processing speed plays an important role for arithmetic fluency, the observed association between arithmetic and CT may have been caused by processing speed.

Other numerical abilities may be more closely linked to CT. For instance, Prat et al. (2020) observed a significant contribution of numeracy to learning outcomes in a Python course. Numeracy, as an umbrella term, can be viewed as the ability to draw on mathematical knowledge to process numerical information. Among the wide range of numerical abilities, algebraic skills may be especially important for CT, as they enable individuals to deal with variables and functions. For instance, some CTt items contain simple functions nested within a repeat times loop. Taking this into account, programming instruction is currently integrated into the instruction of algebra within the mathematics lessons across different grade levels in Sweden (Bråting et al., 2020). Concerning the contribution of algebra to CT and programming, there is first empirical evidence that algebraic skills can predict learning outcomes in an introductory programming course at university (Graafsma, 2019). Nonetheless, to the best of our knowledge, it is still unclear whether this relation can also be observed in younger students at the start of secondary school. In light of theoretical considerations and recent empirical evidence, we conclude that it is important to further investigate the relation between arithmetic and CT, while additionally addressing the contributions of algebraic skills and number sequencing.

Spatial abilities

In the Cattell-Horn-Carroll model of intelligence, spatial abilities such as visualization and mental rotation belong to the visual processing (Gv) component, allowing an individual to perform figural or geometric tasks requiring perception and transformation of visual stimuli (McGrew, 2009).

Parkinson and Cutts (2019) proposed a theoretical model of the relations between spatial skills and programming. They argued that certain spatial skills may play specific roles in reading and identifying key points in code or problems, as well as the mental models constructed when trying to understand problems or generate programs. According to the authors, two spatial factors may be particularly relevant for generating mental models: visualization and mental rotation. According to Parkinson and Cutts (2019), visualization is particularly important for creating a mental model which may need to be developed and restructured (i.e. mentally transformed) as required. Mental rotation may be involved when individuals try to grasp complex elements of a mental model, enabling them to visualize how these elements would look like in a different orientation.

Visualization has been defined as the ability to create a mental representation of an object, to transform this mental representation and retain it throughout the entire mental transformation process (e.g. Carroll, 2003). A large body of evidence supports significant associations between visualization and CT and programming outcomes. Fincher et al. (2005) reported a significant association between performance on a visualization task and final grade in an introductory computer science course in a large sample of 177 university students. Bockmon, Cooper, Gratch et al. (2020) assessed visualization skills in university students prior to participation in an introductory programming course. Results showed that visualization was a predictor of interindividual differences in programming skills after completion of the course. Training studies suggest a causal relation between visualization skills and programming ability: After an intervention designed to foster spatial skills, college students demonstrated improved performance on both a visualization task and programming skills compared to a control group (Bockmon, Cooper, Koperski et al., 2020). Another study specifically aimed to train spatial skills in a group of computer science students with low visualization skills: This training group showed improved visualization skills, as well as an increase in class rankings (Parkinson & Cutts, 2020). Concerning the association with CT, Román-González et al. (2017) found a significant predictive contribution of spatial abilities (as assessed by a visualization task) in secondary school students. In summary, empirical findings strongly support theoretical considerations that CT and programming rely on visualization.

Mental rotation is the ability to perform complex rotations in three-dimensional space (e.g. Carroll, 2003). Although evidence is still scarce, first studies indicate that mental rotation is related to CT and programming (Ambrósio et al., 2014; Città et al., 2019). In a relatively large sample of 92 children from primary school, Città et al. (2019) observed a positive association between CT and mental rotation. A small-scale exploratory study with 12 university students enrolled in a computer science course (Ambrósio et al., 2014) found that performance on a 3D mental rotation test was related to academic performance. The contribution of mental rotation may be highly dependent on the task format: There is reason to assume that the ability to mentally rotate objects may be particularly relevant for the present CTt, given that its items are spatial in nature, consisting of mazes and geometric patterns. Since mental rotation shows a reliable gender effect (Linn & Petersen, 1985; Voyer et al., 1995), this could explain the observed advantage of boys on the CTt (Román-González et al., 2017). On the other hand, it is possible that mental rotation plays an important role for CT irrespective of gender or task format. Therefore, we aim to further explore its contribution to CT in the current research.

Margulieux (2019) argued that by developing spatial skills, individuals can increase their ability to process information by chunking (i.e. grouping) it in meaningful entities. She expected this to have a beneficial effect of performance in STEM domains: Training spatial skills such as visualization may help to form strategies to encode mental representations of spatial information more efficiently. Such strategies increase the amount of new information processed within a given time span. This is important because short-term memory has a limited capacity allowing to maintain a number of elements of information in mind and it plays a role in various complex cognitive tasks such as reasoning and mathematics (e.g. Attout & Majerus, 2015; Peterson et al., 2017). Alleviating memory load thus allows individuals to allocate more resources to other relevant tasks.

Importantly, short-term memory can be subdivided into different modalities (e.g. verbal and spatial). Being able to spatially retain information appears especially critical when creating a hierarchically organized mental representation of an algorithm. Visuo-spatial short-term memory enables retaining a limited amount of spatial information in memory for a brief period. Empirical evidence on the contribution of short-term memory to CT is scarce. Surprisingly, Akar and Altun (2017) did not observe a significant correlation between visual memory and performance in an introductory computer science course. However, this study used a complex task as a measure of visuo-spatial memory (i.e. not only involving short-term maintenance of information, but also manipulation and keeping information in mind across multiple trials). In contrast, a recent study showed a significant association between CT and performance on a simple verbal short-term memory task, in which participants had to recall a series of digits in the correct order (Spieler et al., 2020). To clarify the contribution of visuo-spatial short-term memory, we argue that it is important to employ a “pure” measure of visuo-spatial short-term memory and have therefore included such a measure in the current study.

The present study

The current replication project aimed to further disentangle the influence of cognitive skills on CT. We had separate expectations for the replication as well as the extension part of the current study. From the replication perspective, we expected to replicate the following results: We expected CT to be positively associated with reasoning as measured by a figural reasoning task, with spatial abilities as measured by a visualization task, but not associated with mental arithmetic. We also expected to replicate the previously reported gender effect, with boys showing higher CT performance.

From the extension perspective, we first aimed to further clarify which reasoning abilities play an important role in CT by distinguishing between numerical and figural reasoning. Our expectation was that both numerical and figural reasoning are uniquely related to CT. Further, we expected complex algorithmic skills to be positively related to CT. Finally, we aimed to disentangle the role of spatial skills for CT. Our expectation was that not only visualization, but also mental rotation and visuo-spatial short-term memory would play a central role for CT.

The study proposal was registered with the journal. This means that the research questions, methods and analysis plan of this study were reviewed and revised before data collection and analyses were carried out.

Method

Participants

The present sample consisted of 132 students from Grades 7 and 8 (mean age 12 years and 6.51 months; SD = 8.95 months) from Austria. 74 identified as female and 58 identified as male. Initially, 169 students took part in the study, but 24 had to be excluded because data of the cognitive assessments were lost due to technical issues. Moreover, a class of 13 students decided to discontinue participation because of organizational issues.

Participants were recruited through their schools. We received ethical clearance for the study from our institutional ethics committee (ethics vote no. 39/102/63). Written informed consent was granted by parents or legal guardians.

As it is not clear how age influences the association between CT and the measured cognitive variables, we focused on a smaller age range than the original sample. Román-González et al. (2017) reported a sample of students with a relatively wide age range from Grade 5 to 10. By keeping variance in age low, we could assess the association between variables relatively independent of a possibly confounding influence of age. We only recruited students from Grades 7 and 8, as this is the age range the CTt was originally designed for (Román-González et al., 2017). It is important to mention that in Austria, mandatory programming instruction does not take place in schools until Grade 9. This allowed us to investigate a population of students mostly without prior formal programming experience.

Power

The appropriate sample size was determined by conducting power analyses with the “pwr” package (Champely, 2020) in R (R Core Team, 2020). To obtain a conservative estimate, we decided to base our target sample on the largest sample size estimate we obtained. Whenever available, we considered relevant effect sizes from previous empirical studies. Note that in many instances (i.e. relation between CT and numerical reasoning, algebraic skills and visuo-spatial working memory), there was no estimate of effect sizes available in the literature. We set power to .80 and the probability of alpha-error to .05, corresponding to the convention by Cohen (1988).

For the bivariate relations between CT and numerical, as well as spatial variables, we considered the smallest significant effect size reported in the literature reviewed above. Thus, we conducted a power analysis based on the size of the relation between mental rotation and CT based on results by Città et al. (2019). In this study, the size of the correlation coefficient ranged between r = .27 and r = .32. In order to obtain a conservative estimate, we considered the smallest effect size of r = .27. Power analysis revealed a minimum sample size of 105 participants.

For the regression analysis, we considered the effect size of the regression coefficient of spatial abilities reported by Román-González et al. (2017). The standardized regression weight corresponded to an f² of .15. Sample size was computed for a regression model with seven predictors of interest, yielding a minimum sample size of 103 participants. Following reviewers’ suggestions, we decided to aim for a sample similar in size as the original study in which a total of 135 participants were included in the regression analysis.

Tasks

Replication

For the replication part of our study, we assessed CT, reasoning (as measured by a figural reasoning task), numerical abilities (as measured by a mental arithmetic task) as well as spatial abilities (as measured by a visualization task).

Computational thinking (CT)

We developed a German-language translation of the original Computational thinking test (CTt; Román-González et al., 2017)². This computerized multiple-choice test mainly consists of logical and visuo-spatial problems, such as Pac-Man mazes or drawing geometric patterns. Importantly, these can be solved without any prior experience with a programming language. Each item addresses one or more of the following seven computational concepts: Basic directions and sequences (4 items); Loops-repeat times (4 items); Loops-repeat until (4 items); If-simple conditional (4 items); If/else-complex conditional (4 items); While conditional (4 items); Simple functions (4 items). Based on the CT framework (Brennan & Resnick, 2012), the following cognitive tasks are required for solving the items: Sequencing (14 items), completion (9 items), and debugging (5 items). In total, the CTt contains a total of 28 items to be solved within a time limit of 45 minutes.

Cognitive abilities

Originally, we had intended to use German-language paper-and-pencil tests (CFT 20-R; Weiß, 2019; DEMAT 9; Schmidt et al., 2012; DIRG; Grube et al., 2010) to assess the cognitive predictors of CT, which were chosen because they closely matched the Spanish-language tasks PMA test battery (Cordero et al., 2007) and the RP 30 problem-solving test (Seisdedos, 2002) employed by Román-González et al. (2017). However, since submitting our preregistration, we had to make some adaptations to our original plans for assessing cognitive abilities due to the pandemic situation, preventing face-to-face assessments in schools. Thus, we chose a German-language computerized test battery (INSBAT 2; Arendasy et al., 2020) with excellent psychometric properties allowing an assessment of fluid reasoning, visualization, and mental arithmetic.

The Intelligence-Structure-Battery-2 (Arendasy et al., 2020) is based on the Cattell-Horn-Carroll model (McGrew, 2009; Schneider & McGrew, 2012) and is widely used in German-speaking countries. This test was chosen for the following reasons: (1) Prior research (Arendasy et al., 2020) indicates a good fit of the Cattell-Horn-Carroll model and the latent correlations between all stratum two factors were in line with a recent meta-analysis (Bryan & Mayer, 2020). (2) All subtests have been calibrated by means of the 1PL Rasch model (Rasch, 1980) and exhibit measurement invariance across gender, age and educational level (Arendasy et al., 2020). For all subtests, a person parameter is estimated, which considers not only the correctness of the responses, but also the item difficulty of the items received by the test-taker. The person parameter thus allows a direct comparison of participants’ performance. (3) The subtests have been constructed based on well-validated cognitive processing models using automatic item generation (Arendasy & Sommer, 2012; Irvine & Kyllonen, 2002) and research using the Linear Logistic Test model (Fischer, 1995) indicates that the item design features linked to these cognitive processes account for ≥ 75% of the variance in the 1PL item parameters (Arendasy et al., 2020). This implies that the cognitive processes hypothesized to be used by the test-takers account for ≥ 75% of the variance in test-takers’ test scores. Taken together, these studies provide profound evidence for the construct validity and fairness of this measure.

The INSBAT 2 subtests were administered as computerized adaptive test (CAT; Van der Linden & Glas, 2000) with a target reliability corresponding to Cronbach’s α = .70. Only the mental arithmetic test was administered as a fixed-item linear test with an approximately equal number of items with low, medium and high levels of complexity.

Figural reasoning

Participants were shown incomplete 3 × 3 figural matrices filled with geometrical figures (e.g. rectangles and circles). The number and arrangement of these figures followed certain rules, and the bottom right field was always empty. Participants were required to infer the rules and complete the matrices by selecting one of six alternatives to fill the gap. Similar to the numerical reasoning task, this test was constructed on the basis of well-validated cognitive models (e.g. (Carpenter et al., 1990; Rasmussen & Eliasmith, 2011).

Mental arithmetic

Participants were given 20 complex multi-step mental arithmetic problems requiring the application of basic arithmetic operations. This task was constructed on the basis of well-established difficulty factors in mental arithmetic problem solving such as the problem size effect (e.g. Ashcraft, 1995; Campbell & Xu, 2001; Groen & Parkman, 1972), the number of arithmetic operations (e.g. Geary et al., 1993; Verhaeghen et al., 1997), the number of carry-over or borrowing steps (e.g. Deschuyteneer et al., 2005; Furst & Hitch, 2000; Klein et al., 2010; Widaman et al., 1989), and the difficulty of the arithmetic operations (Campbell & Xu, 2001; Voyer et al., 1995). These difficulty factors have been shown to jointly account for 83.2% of the differences in the 1PL item difficulty parameters. Cronbach’s α reliability for the present sample was .89. There was an item-wise time limit of 45 seconds.

Visualization

Each item consisted of a geometrical figure cut into multiple parts. Participants were asked to indicate which geometric figure could be assembled from the available parts by choosing one of five response alternatives. The test items were constructed on the basis of well-validated cognitive processing models (e.g. Embretson & Gorin, 2001; Ivie & Embretson, 2010; Pellegrino et al., 1985).

Extension

In the extension part of our study, we assessed numerical reasoning, algebraic skills, mental rotation, and visuo-spatial short-term memory. Similar to the replication part of our study, we had to adapt our original pre-registered plans for assessing these skills due to the pandemic situation, and used further subtests from the INSBAT 2 (Arendasy et al., 2020), instead of paper-and-pencil tests.

Numerical reasoning

Participants were presented with a sequence of six numbers following certain rules. They were asked to discover these rules and continue the number sequence by typing in the seventh number. The test items were constructed on the basis of well-validated cognitive processing models for number series problem solving (e.g. Holzman et al., 1983, 1982).

Algebraic skills

Participants were asked to complete complex arithmetic problems by supplying the missing arithmetic operation signs. The test items were constructed based on a well-validated cognitive processing model (Arendasy et al., 2007).

Mental rotation

Participants specified by which angle and in which direction the start view figure must be rotated to match the figure in the finish view. The test items are based on well-validated cognitive processing models (e.g. Arendasy, 2000; Arendasy & Sommer, 2010; Just & Carpenter, 1985). Notably, there is evidence that the item design features prevent test-takers from solving the test items by means of cognitive processes other than mental rotation (Arendasy, 1997, 2000; Arendasy & Sommer, 2010; Arendasy et al., 2020). Furthermore, this subtest has been shown to load on a visual processing factor in several independent studies and there is evidence that this subtest is highly correlated with Shepard and Metzler figures which are often considered as a prototypical complex mental rotation task (Arendasy, 1997, 2000; Arendasy et al., 2008; Arendasy & Sommer, 2010).

Visuo-spatial short-term memory was assessed with a computerized task based on the Block Recall Forward subtest of the Working Memory Test Battery for children (WMTB-C; Pickering & Gathercole, 2001). Participants were presented with nine identical white blocks. Each trial, a sequence of blocks lit up in red, and participants were asked to indicate this sequence by clicking the blocks in the same order. Sequence length started at three blocks and continually increased if four out of six trials were successfully recalled. Once three errors per sequence were made, the task was stopped.

Procedure

Since submitting our preregistration, we had to make some adaptations to our original plans for data collection. Originally, we had intended to conduct face-to-face assessments at participants’ schools. As this was impossible due to social distancing measures during the ongoing pandemic, we collected the data online, with students completing all tasks on a standard computer or laptop. Data collection required three school lessons of 50 minutes within three school weeks, and all tasks were administered in a classroom setting in participants’ school supervised by school teachers. Note that data collection in the current study was more extensive than in the original study, which only required a total of two school lessons of 60 minutes in two consecutive weeks. In line with the original study by Román-González et al. (2017), tasks were given in a fixed order, and students completed the CTt before receiving the cognitive ability assessments.

Results

Descriptive statistics

Descriptive statistics of all study variables are displayed in Table 1, and the distribution of scores of the Computational Thinking test (CTt) is illustrated in Figure 1. As can be seen, the distribution is slightly left-skewed: There is a higher frequency of higher scores than lower scores, suggesting that the test was generally fairly easy for the present population. As can be seen in Table 1, for all cognitive assessments, kurtosis and skewness values were between −1 and 1 and therefore, the distribution of values can be considered as not deviating substantially from normal distribution.

Figure 1. Distribution of computational thinking test (CTt) scores.

Table 1. Descriptive statistics of Computational Thinking test (CTt) scores.

	Mean	SD	Min	Max	Skewness	Kurtosis
CTt scores^a	19.30	4.60	6	27	−0.67	0.19
Figural reasoning^b	0.01	1.39	−4.33	4.10	0.82	0.12
Mental arithmetic^b	0.60	1.71	−3.73	5.14	−0.06	−0.09
Visualization^b	−0.93	1.06	−5.04	1.71	−0.34	1.02
Numerical reasoning^b	−0.17	2.03	−6.31	4.41	−0.56	0.90
Algebraic skills^b	−1.83	1.48	−5.97	2.77	0.90	0.13
Mental rotation^b	−1.51	1.02	−4.72	1.88	0.00	0.86
Visuo-spatial STM^a	15.77	5.52	1	30	−0.37	−0.15

STM: Short-term memory.

^aNumber of correct responses.

^bPerson parameters.

Gender differences

We found a statistically significant gender difference in CT performance (t(130 = −2.50, p = .014, d = .44), with boys (M = 20.41, SD = 4.69) achieving higher scores on the CTt than girls (M = 18.43, SD = 4.36) by 1.98 points on average. Following the classification of effect sizes by Cohen (1988), this can be considered as a medium-size effect.

Correlational analyses

Zero-order correlations between CT and its predictors are shown in Table 2. According to the classification of effect sizes by Cohen (1988), correlation coefficients between .10 and .30 can be considered as small, correlation coefficients between .30 and .50 can be described as moderate, and correlation coefficients above .50 can be considered as large. The association between CT and numerical reasoning was small, whereas figural reasoning was moderately related to CT. There was a small significant relation between mental arithmetic and CT, and a small-to-moderate association between algebraic skills and CT. Visualization was moderately related to CT. In contrast, the associations between CT and mental rotation, as well as visuo-spatial short-term memory were weak. As expected, all predictors were significantly related to CT.

Table 2. Correlations between study variables.

Variable	1.	2.	3.	4.	5.	6.	7.	8.
1. Computational Thinking		.47**	.33**	.48**	.38**	.42**	.24**	.22*
2. Figural reasoning			.22**	.33**	.54**	.35**	.12	.22*
3. Mental arithmetic				.19*	.24**	.45**	.12	.13
4. Visualization					.20*	.31**	.42**	.14
5. Numerical reasoning						.33**	.15	.15
6. Algebraic skills							.30**	.17*
7. Mental rotation								.09
8. Visuo-spatial short-term memory

*p < .05; **p < .01; *** p < .001.

With regard to inter-correlations between predictors, we found moderate associations between numerical and figural reasoning, as well as mental arithmetic and algebraic skills. While we observed a substantial association between visualization and mental rotation, visuo-spatial working memory was not significantly related to the other spatial skills.

Predicting CT

To examine the cognitive predictors of CT, we conducted a hierarchical multiple linear regression analysis with the “enter” method (Table 3). In a first step, gender explained 4% of variance in CT. Introducing the cognitive predictors in a second step contributed 35% of additional variance to the prediction of CT. Thus, this model explained 39% of variance in CT. Visualization, figural reasoning and algebraic skills were identified as significant unique predictors of CT. Notably, gender remained a significant predictor of CT.

Table 3. Hierarchical linear regression analysis predicting CT by gender, numerical and figural reasoning, mental arithmetic, algebraic skills, visualization, mental rotation, and visuo-spatial short-term memory.

	β	sr	p
Model 1: Gender
Gender	.22	.22	.014
Model Fit: F(1,130) = 6.27, p = .014, adj. R^2 = .04
Model 2: Cognitive predictors
Gender	.15	.14	.049
Figural reasoning	.20	.16	.020
Mental arithmetic	.07	.06	.358
Visualization	.30	.26	< .001
Numerical reasoning	.10	.09	.210
Algebraic skills	.19	.15	.028
Mental rotation	−.02	−.02	.756
Visuo-spatial short-term memory	.08	.08	.251
Model Fit: F(7,123) = 11.62, p = .001, adj. R^2 = .39

Discussion

The current project investigated the influence of cognitive skills on CT in students from Grades 7 and 8: First, we wanted to replicate the contribution of reasoning, spatial abilities and numerical abilities by assessing the same subskills as Román-González et al. (2017). Second, we aimed to extend this research by investigating the contribution of other theoretically relevant subskills, namely numerical reasoning, algebraic skills, mental rotation, and visuo-spatial short-term memory. Our third goal was to determine whether gender differences in CT can be explained by gender differences in mental rotation and other cognitive skills.

Replication

Concerning the replication part of our project, our results support the view that CT is underpinned by reasoning, as assessed by figural reasoning, and spatial abilities, as assessed by visualization. However, in contrast to the original study, we also found a significant association between CT and mental arithmetic.

We observed substantial bivariate associations between CT and both figural reasoning and visualization (r = .47 and r = .48, respectively), replicating findings reported by Román-González et al. (2017). Note that the size of the correlation between figural reasoning and CT is somewhat lower than the size of the correlation reported in the original study (r = .67). However, this discrepancy can be explained by the fact that the RP30 problem-solving test employed in the original study does not provide a “pure” measure of figural reasoning, but also requires spatial abilities and short-term memory.

Our observation of a small positive association between mental arithmetic and CT does not replicate previous findings by Román-González et al. (2017). This is interesting, as previous accounts proposed that numerical abilities might only play a role for CT in young learners from primary school (Tsarava et al., 2019). It was speculated that once a certain level of numerical skills was achieved through schooling, its contribution to CT might be diminished. However, for older students, we still found an association between mental arithmetic and CT, suggesting an ongoing contribution of mental arithmetic in early secondary school. Recently, another study with children aged 10–15 years also reported an association between arithmetic fluency and complexity of a game designed in a summer course (Spieler et al., 2020). Thus, it is possible that mental arithmetic plays a persisting and largely age-independent role for certain CT tasks and skills. Discrepancies between our results and the original study by Román-González et al. (2017) may be partly task-related: Whereas the original study employed an arithmetic verification task in which participants had to indicate whether a given sum was correct or not, the present study used a task requiring exact calculations with basic arithmetic operations (additions, subtractions, multiplications and divisions).

Extension

The present study critically extends the research literature on the cognitive mechanisms of CT by specifying the contribution of different subskills belonging to three components of the Cattell-Horn-Carroll model of intelligence (McGrew, 2009; Schneider & McGrew, 2012): Gf (fluid reasoning), Gq (quantitative knowledge) and Gv (visual processing). To this aim, we assessed two reasoning skills (i.e. figural and numerical reasoning), two numerical skills (i.e. arithmetic and algebraic skills), as well as three spatial skills (i.e. visualization, mental rotation and short-term memory). In line with our expectations, all of these skills were positively related to CT (r = .22 – r = .48). This provides new insights because some of these subskills had previously not received much attention as predictors of CT.

The critical aim of this study was to identify which cognitive subskills uniquely contribute to the prediction of CT. To this aim, we chose a regression-based approach. This is important, given that a substantial overlap could be expected between the cognitive variables we considered. Indeed, we observed medium-size correlations between subskills belonging to the same component of the Cattell-Horn-Carroll model (i.e. figural and numerical reasoning, arithmetic and algebraic skills, visualization and mental rotation). Only visuo-spatial short-term memory was unrelated to the other spatial skills. This is not entirely surprising, given that the Cattell-Horn-Carroll model (McGrew, 2009; Schneider & McGrew, 2012) proposes that short-term memory constitutes a separate component, irrespective of the precise nature of the information processed (e.g. visuo-spatial or verbal).

Regression analyses revealed that figural reasoning contributed unique variance to the prediction of CT, whereas numerical reasoning did not. This corroborates previous findings pointing towards a close link between figural reasoning and CT (e.g. Román-González et al., 2018; Tsarava et al., 2019), as well as programming (e.g. Ambrósio et al., 2014; Helmlinger et al., 2020; Prat et al., 2020). Note that numerical reasoning also involves problem-solving abilities and shows significant associations with both figural reasoning and CT. As already pointed out by Helmlinger et al. (2020), both figural reasoning and numerical reasoning subtests of the INSBAT 2 battery are inductive reasoning tests, and the only distinction is that figural reasoning specifically involves processing of spatial stimuli. Thus, it appears reasonable to assume that CT may particularly require an understanding of the structural relations between spatial material.

A major finding of the present study is that algebraic skills served as a unique predictor of CT. This extends first evidence reporting significant associations between algebra and programming performance in university students (Graafsma, 2019) to a younger sample from secondary school. Although these correlational findings do not inform us about the precise mechanisms underlying the causal relation between algebraic skills and CT, algebraic reasoning is about understanding the functional affordances and constraints of mathematical operations and functions. Computational thinking may require a similar line of reasoning to sort out how elements of a program relate to each other and function integratively while adhering to the basic rules of the domain in question. For this reason, algebraic reasoning may be more closely related to computational thinking than other numerical skills such as mental arithmetic, which was considered in the original study as well as in our replication study.

Addressing this explanation provides an exciting avenue for future research. Although the precise causal mechanisms remain to be unraveled, the current results suggest that the role of numerical abilities for CT should not be discounted. This is important, given that previous studies argued that numerical abilities are not particularly relevant for CT based on the lack of an association between CT and more basic arithmetic skills (e.g. Román-González et al., 2017). Although both arithmetic and algebraic skills can be subsumed under the broader construct of numerical abilities, the present study suggests that the latter is more relevant for CT.

Among the spatial abilities we investigated, only visualization emerged as a unique predictor of CT. From a theoretical and practical viewpoint, it is interesting that visualization was a more prominent predictor of CT than mental rotation, given that both involve mental transformations of spatial material. While visualization involves transforming and maintaining mental representations across multiple processing steps, mental rotation requires complex mental transformations in three-dimensional space. Thus, the current results suggest that the ability to perform three-dimensional transformations is not of central importance for CT. However, the specific contribution of mental rotation may depend on the precise nature of the CT assessment: While the CTt does require rotations in two-dimensional space (e.g. mazes), none of the items explicitly required three-dimensional rotations. The finding of a prominent contribution of visualization is in line with correlational studies (Fincher et al., 2005; Román-González et al., 2017), as well as intervention studies indicating that spatial trainings specifically fostering visualization skills in beginning programmers led to an improvement in programming outcomes (Bockmon, Cooper, Koperski et al., 2020; Parkinson & Cutts, 2020).

We observed significant gender differences in favor of the boys in our replication study, which corroborates findings from the original study (Román-González et al., 2017), as well as evidence based on a sample of older high-school students (Guggemos, 2021). Gender remained a significant unique predictor of CT even when the cognitive variables were introduced into the regression model. The amount of variance shared between gender and CT did decrease in the second step of the model, but there still remained a substantial overlap that could not be accounted for by cognitive skills (Model 1: sr = .22, Model 2: sr = .14). This is interesting, given that Román-González et al. (2017) suggested that gender differences in CT were explained by a male superiority in the required cognitive abilities. The present results seem to indicate that gender differences in CT are largely independent of gender differences in cognitive skills. Thus, it is important to consider other explanations, such as an initial stereotype threat. Girls are less likely than boys to enroll in introductory computer science courses (Master et al., 2016), despite there being some evidence that these activities can successfully foster CT (e.g. Tsarava et al., 2019). Recently, Guggemos (2021) reported that computer literacy acted as a significant predictor of CTt scores in high school students, and that more than 50% of the observed gender difference in CT could be explained by lower self-concept, computer literacy, and self-determined motivation in female students. Although the precise mechanisms linking computer literacy and CT are still unclear, Guggemos (2021) argued that knowledge about the capabilities of computers may be beneficial given that CT involves representing a problem in way a computer can solve it. Therefore, future research addressing the gender gap in CT and programming outcomes should consider cognitive variables as well as motivational and affective factors.

Limitations

It is important to acknowledge some limitations of the present study: the scope of our study is narrowed by the study design, the tasks employed, and the sample investigated.

First, we could not employ the same measures as the original study. The cognitive test battery employed in the original study did not include any subtests assessing specific reasoning skills, mental rotation or algebraic skills, which we were interested in to extend the scope of the original study. Moreover, we could not conduct face-to-face assessments due to the ongoing pandemic. Thus, we had to carefully choose a German-language test battery allowing a reliable and valid supervised online assessment of all constructs measured by the original study, as well as the additional subskills addressed in the present study.

As already pinpointed by Román-González et al. (2017), some limitations of the CTt should be considered: this assessment only provides a one-dimensional measure of CT and does not cover all facets of CT recognized in the literature. For instance, with regards to Brennan and Resnick’s (2012) framework, the CTt mainly focused on “computational concepts”, partly considers “computational practices”, but “computational perspectives” are not covered.

Moreover, it is important to point out that the items of the CTt are spatial in nature: test-takers are required to navigate through a maze or complete a geometrical pattern. This may have over-emphasized the contribution of figural reasoning and visualization in the present work, as well as in previous studies (e.g. Román-González et al., 2017; Tsarava et al., 2019). However, given the relatively minor association found between mental rotation and CTt performance, it is unlikely that the reliably observed gender differences in CTt performance are caused by the spatial nature of the items.

Finally, it is important to mention that our sample may not be entirely comparable to the Spanish normative sample from Grades 7 and 8. Although both samples were highly similar in age and duration of formal schooling, the present sample showed a higher average performance on the CTt (+ 3 points). Although the current study does not provide insight into the underlying reasons, it is possible that cohort effects play a role. At the time point of data collection, students in Austria had already gained extensive experience with digital media due to online teaching during the pandemic. As a side effect, online teaching may have promoted students’ CT. Moreover, the last years have seen an ongoing effort to develop and promote computer science education in schools. Further investigations are clearly needed to replicate these findings. However, if similar improvements are visible in different samples, it would seem advisable to collect new test norms for the CTt.

Conclusions

We set out to identify which cognitive subskills contribute to the prediction of CT. We could confirm previous research by showing that figural reasoning and visualization are important predictors. Additionally, the present work points to an incremental role of complex numerical abilities, as assessed by an algebraic task. Together, these findings advance our theoretical understanding by showing that cognitive mechanisms of CT extend beyond fluid reasoning and visualization, pointing towards a unique contribution of algebraic skills.

Acknowledgments

The authors acknowledge the financial support by the University of Graz. We would like to thank Marcos Román-González for kindly sharing with us the original Computational Thinking test (CTt), as well as Josef Guggemos and Katerina Tsarava for their German-language adaptations. We would also like to thank Anna Exel very much for helping with administrative issues. Finally, we would like to thank students and teachers from the following schools supporting our research: BG/BRG Knittelfeld, BG/BRG Pestalozzi, BRG Kepler, KLEX Graz and MS Kepler.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Data is publicly available on the Open Science Framework and can be accessed at https://osf.io/3c9mw

Additional information

Funding

This work was supported by internal funding (Route 63) from the University of Graz and the Graz University of Technology.

Notes

1. Although Román-González et al. (2017) did not explicitly report the number of participants considered in the regression analysis, we were able to derive this information from on the degrees of freedom of the regression models.

2. Previously, there were two German-language adaptations of the Computational Thinking Test (CTt) for other age groups: 1) Guggemos et al. (2019) developed a more difficult version for high-school students by replacing the five easiest CTt items with five harder items. 2) Tsarava et al. (2019) designed a version for children from primary school by selecting the 21 easiest CTt items and reducing administration time from 45 to 20 minutes. The novel version developed for this study constitutes a direct translation of all 28 items of the original CTt by Román-González et al. (2017).