Methods, understandings, and expressions of causality in educational research

The opening pages of Pearl and Mackenzie’s volume The Book of Why: The New Science of Cause and Effect (2018) herald their captivating romp through causality by referring to a “ladder of causation” (p. 28) that starts from association (by seeing and observing), moves up to intervention (by doing and intervening), and thence to counterfactuals (by imagining, retrospection, and understanding). Each rung of the ladder establishes causality more certainly.

Humans think causally. Causality can be studied by many methods. Here, Pearl and Mackenzie (2018) state that statistical analysis does not simply concern data and their methods of analysis; rather, there is a need for an “understanding of the process that produces the data” (p. 85). Such “understanding” comes from introducing causality, as causality yields something additional to the original data. “Methods” of data analysis are informed by an “understanding” of causality, as this Editorial shows. Pearl and Mackenzie write that if we remove the understanding of causation from statistical analysis, all that we are left with is data reduction, which does not tell us much. The papers in this issue move forward from “methods” to “understanding” data with regard to causality. Further, the Editorial indicates how easily it is to find expressions of causality in articles; this should caution researchers to take care in the wording that they use. The Editorial below draws attention to wording in deliberately italicising causal words in quoting from the articles in this issue. For example, is causality really being demonstrated, or, like Pearl and MacKenzie’s lowest rung of the ladder, is there merely association?

Causality, be it post hoc or ante hoc, is self-evidently important in education. However, how we adduce causality is far from straightforward, and the papers in this issue yield insights into, and cautions concerning, claims for, and demonstrations of, causality. The papers here indicate methods, challenges, outcomes, and benefits of studying causality.

The challenges of “methods” and “understanding” when investigating causality are legion. Witness, for example, in the perennial search for causality, its differences from association, prediction, explanation, inference, influence, correlation, accounting for, correspondence to, purposiveness, and a whole armoury of other words. Look at the dangers of working with mediating, confounding, and moderating variables, transitivity, or controlling out almost everything such that what remains is very little. Wrestle with underdetermination, overdetermination, supervenience, and the difficulties of mereology. Consider the challenges of probabilistic causation and Bayesian approaches, leavened by multilevel causal modelling. Add to these the context-dense, variable-rich, causally complex world of education, and the attraction of Pearl and MacKenzie’s (2018) “childlike simplicity” (p. 39) of a causal diagram evaporates in front of our eyes.

Little wonder it is, then, that authors studying causality in the field of education choose their words very carefully. This helps them to avoid charges of assumed or inferred, but unproven, causality, or of a failure to distinguish between: (a) reality-based and imagined design models; (b) correlation, association, and causation; (c) statistical prediction and real-world, applied prediction; (d) inference and actuality; and (e) regression analysis and causation; and so on.

Taking up Pearl and Mackenzie’s (2018) concerns, the articles in this issue address “methods” and “understanding” in considering causality. Added to these is the wording of causality, which attests to the caution that must be exercised in order to avoid erroneous claims of causality, and to establish the levels of certainty and security at which such causal claims can be made (note the italicised additions to words below, suggesting causality).

The first paper, by Marks, in its very title, throws down a gauntlet in asking “[i]s the relationship between socioeconomic status (SES) and student achievement causal? Considering student and parent abilities”. His “methods” involve: (a) regression analysis; (b) analyses of covariance; (c) correlations; (d) controls; and (e) modelling. His “understanding” and conclusions are telling:

… the common assumption that SES and its components have strong causal effects on student achievement is not tenable. … Therefore, the relationships between aspects of SES (e.g., parents’ education and occupation, family income and wealth) and achievement cannot be assumed to be causal, but to a considerable extent reflect their associations with parental abilities.

His wording is as careful and well chosen as the challenges that he raises.

This issue then moves to two papers on test-taking. Both use a range of “methods” to yield greater “understanding” of their findings and their implications for causality. In the first of these, Yildirim-Erbasli and Bulut investigate “[t]he impact of students’ test-taking effort on growth estimates in low-stakes educational assessments”, including “non-effortful test-taking”. Studying “growth” here refers to “academic growth, screening learning problems, and designing interventions for struggling students in kindergarten through to Grade 12 (K–12)”, which seeks to “evaluate student learning, define individualised learning goals for students, and tailor their instruction to meet these goals”. The authors use many “methods” of data analysis: (a) the normative threshold method and a modified threshold method in order to “detect non-effortful responses based on both rapid-guessing and slow-responding behaviours”; (b) an effort-moderated item response theory two-parameter model; (c) multiple regression; and (d) t tests.

From these “methods”, Yildirim-Erbasli and Bulut move to “understanding”, and they conclude that “testing organisations designing computer-based assessments for universal screening and progress monitoring should detect non-effortful responses based on both rapid guessing and slow responding before reporting test results to teachers and other school-based professionals”. Researchers and practitioners, they advise,

should carefully consider the testing contexts to determine how non-effortful testing behaviour could affect students’ academic growth estimates. … Furthermore, schools that regularly use low-stakes assessments for evaluating and monitoring student progress should review the assessment results carefully before implementing any instructional changes or interventions.

These are important findings, with significant implications in the causally complex, context-rich world of education. They resonate powerfully with Pearl and Mackenzie’s (2018) comments above, namely, that causal “understanding” yields something more than original data alone.

In the second paper on test-taking, Silm, Must, Täht, and Pedaste note that “[t]est-taking motivation (TTM) has been associated with test performance in low-stakes testing contexts”, and so their study applies this to high-stakes contexts (e.g., university entrance examinations). The “methods” used to test the association between test-taking motivation and test performance in high-stakes contexts are: (a) time-based and self-reported data on test-taking effort; (b) structural equation modelling to predict test performance; (c) the Student Opinion Scale to measure test-taking motivation.

Moving from “methods” to “understanding”, the authors report that test-taking motivation “can have a statistically significant association with test performance” in a high-stakes test. They also note the significance of context, in stating that

[i]t is possible that there are different high-stakes testing situations in which the predictive power of previous performance and gender on test performance is even larger, and the role of TTM [test-taking motivation] is smaller. But in these cases, another situational predictor – test-taking anxiety – may play a larger role.

Addressing the context-dense, variable-rich, causally complex world of education, as indicated above, they comment that

other possible predictors of performance should be considered, such as test anxiety, and interaction effects studied. … [T]he possible effect of TTM should be taken into account when conducting and interpreting results of tests in various contexts, whether they are considered low-stakes or high-stakes.

In a world where attributing causality is a dangerous occupation, the authors are suitably cautious in the expressions that they use and the implications and conclusions that they draw.

The next paper, by Uunk and Pratter, is overtly causal in its title – “Gender Differences in Higher Education in Germany: Are Women Under- or Overrepresented at University, and Why?” – and its opening questions are clearly causal:

Do women, due to life and job goals, less often enrol than men in traditional universities and more often enrol at the more practice- and profession-oriented universities of applied sciences? Or are women overrepresented at traditional universities due to prior educational choices and outcomes such as higher school grades and more frequent choice of non-technical fields of study?

To address these questions, their study uses data from “a nationally representative sample of 1st-year students from the National Educational Panel Study”, which adopts a multi-cohort sequence design with a very large sample (11,896 cases). Their “methods” include: (a) logistic regression models and odds ratios; (b) the Fairlie non-linear decomposition method to “assess to what extent a (potential) gender gap in institution choice is due to gender differences in exam grades, study field choices, job goals, and other background characteristics”; and (c) controls for age and parental education in the analysis.

In “understanding” the data analysis, the causality at work becomes clear: The job goals of men and women “hardly account for the gender gap in institution choice (5% in total)”, that “[t]his gender gap can almost entirely be attributed to educational factors, specifically women’s less frequent choice of engineering majors, and hardly by job goal preferences”, and that

when men and women choose similar study fields, no gender difference in the choice of higher education institution can be observed. … Women in Germany choose traditional university more often than men because women attend vocational school less often, because women obtain higher overall exam results, and above all because women choose non-STEM fields of study more often than men.

The authors add that

[t]he higher odds of women than men of entering traditional university rather than FH [Fachhochschulen: universities of applied sciences], which are primarily caused by a more frequent choice of women for non-STEM fields of study, may give women a slight wage advantage in their occupational career.

The final paper in this issue, by Dörrenbächer-Ulrich, Stark, and Perels, is a study of “[p]rofiles of teachers’ concerns about heterogeneity in classrooms” with regard to inclusive education, using the celebrated concerns-based adoption model (CBAM) and its partners: Stages of Concern (SoC) and Levels of Use (LoU); a method that is celebrating its diamond jubilee (50 years) in 2021, and the paper usefully combines it with Ajzen’s (1991) theory of planned behaviour.

At first blush, this might not appear to be a causal account, but the opening pages dispel this: “The degree to which teachers innovate their educational practice depends not only on teachers’ competences … but also on their attitudes … and on their well-being. … Thus, the successful implementation of an innovation is also driven by subjective dimensions”. (Of course, whether “depends” indicates causality or supervenience is a moot point.) In fact, the article is peppered with causal statements, for example, “research showed that (unfavourable) teacher attitudes can act as a barrier for the successful implementation of inclusive practices”; “it has been shown that positive attitudes regarding inclusive settings increase the intention to teach in inclusive classrooms”; and attitudes “have been shown to influence the implementation of inclusive education”; “teacher self-efficacy had a strong effect on teachers’ concerns regarding inclusion”; and “positive inclusive teaching experiences are a strong factor for increasing teaching self-efficacy with regard to inclusive settings”.

Using (a) the Stages of Concern and Levels of Use questionnaires and (b) discriminant analysis “methods”, the authors are careful in disclosing their “understanding” of what the results show, and their wording is suitably cautious: “ … discriminant analysis showed that all subjective dimensions contributed to a prediction of group membership with individual background factors being identified as predictors with the highest impact”; whether prediction is directly causal is an open question. Further, the authors note that “higher LoU are driven by higher SoC (impact concerns on students should lead to behavioural changes in the innovation to meet the concern)”. However, the authors comment that “[a]s our data are not longitudinal in nature, we cannot answer the question if concerns are driven by behaviour or vice versa”.

As these papers show, the attribution of causality is challenging in terms of “methods”, “understandings”, and wording. These challenges continue to the very end of this issue: Tan’s book review of Aubrey and Riley’s Understanding and Using Challenging Educational Theories, and, finally, Harindranathan’s book review of Kirschner and Hendrick’s How Learning Happens: Seminal Works in Educational Psychology and What They Mean in Practice constitute a parting shot in raising challenges in moving causally from “understanding” to practical application. How learning happens concerns causality; a fitting topic for education.