Visible learning and its enemies – the missing link

In 2017, NordSTEP published an interview with John Hattie by Hanne Knudsen. The interview raised a lot of interest, but also critical reactions. Particularly, one of the interview questions caused confusion. Here, Hanne Knudsen stated that in the introduction to the Danish translation of Visible Learning for Teachers by Niels Egelund and me, we should have written ‘that finally education has become evidence based. It has taken several hundred years for medicine to become evidence based, and now finally education is evidence based, and we can get rid of theory, meaning abstract thinking and speculative thinking’ (Knudsen, 2017, p. 259).

My first reaction was to become angry and vexed: We have never written that, and it would indeed be absurd to say that with evidence we can get rid of theory. I certainly did not like to be misrepresented as I was in this case. My second reaction was rather one of curiosity: How come we were so spontaneously misinterpreted? In a conversation that followed between Knudsen and me, also including John Hattie, it became apparent that Niels Egelund and I were misread. This raised another question – how can it be that Hattie’s work by some in the Danish context has been interpreted in such a way that it should represent a return to the most primitive version of positivism meaning ‘facts without theory’?

This actualizes two questions. One question is what ‘others’ have used John Hattie’s work for in a Danish context. Or rather: has a common (mis)understanding developed among Danish education researchers that can explain the misreading of Hattie, as well as of our initial text? If the answer is yes, this is a case of the development of interpretation communities (Fish, 1980). A particular, sometimes incorrect, interpretation gains momentum, e.g. because it supports political or ideological interests, and gradually it emerges from being an opinion to becoming a broadly accepted fact. A second question is: Which position do I actually represent? This is a case of enlightenment. Of course, I do not want to ‘get rid of theory’. The question is rather, what kind of theory lies behind the aim of Visible Learning to identify statistically significant key influences on student achievement.

In this short discussion paper, I provide three contributions to the discussion of Hattie’s Visible Learning: 1. Why has Hattie and his proponents been so radically misread? My answer is that a strong, ideological interpretation community has emerged that simply dislikes the idea of learning achievements. In this community, simplified ideas concerning Hattie and his proponents have developed. 2. Why has a conflict been created between quantitative and qualitative methods in education science? My answer is, that one explanation is that it is implied that ‘truth’ is a case of ontology, not a case of epistemology. My contribution to the epistemology of Visible Learning is that teaching and learning can be observed through different glasses, and that each point of observation provides a different picture. It is not a zero-sum game. 3. With a reference to Dewey, Hattie clearly states that Visible Learning does not provide the reader with what works recipes, but with hypotheses for intelligent problem solving (Hattie, 2009, p. 247), and that correlates must not be confused with causes (Hattie, 2009, p. 3). What is the theoretical reason for this? My answer is that teaching and learning are not causally related. The relation is a link, or a ‘structural coupling’ (Luhmann, 1997, pp. 92–134), between two closed, self-referential, complex and unpredictable reflection loops: The reflection loop of the learning student, and the reflection loop of the teacher as a reflective practitioner. This implies that one can identify statistical probabilities, not simple, causal mechanisms between interventions and achievements.

Inadequacies – why has John Hattie been misinterpreted?

As already mentioned, in a Danish context, John Hattie’s work has caused a lot of reactions, however mostly critical. Actually, a book has been published with the Danish title Hattie på dansk (Hattie in Danish) (Bjerre et al., 2017) including a number of strongly critical articles.

Some reactions have been purely emotional: We dislike, what we read, seems to be the implicit tone, simply because we dislike numbers and statistics. As an example, with an extreme formulation Steen Nepper Larsen wrote:

Proudly and stoutly as a devoted king of statistics Hattie presents his overwhelming 240 million data analyses, but a vigilant reader will notice that he is essentially a practitioner and a thinker who proclaims that each teacher must have an eye for the unique student (Larsen, 2015, p. 5)

Here, the important aspect is not the argument, but the highly rhetorical discourse represented by Larsen, which may be compared with Robin Alexander’s identification of four critical policy discourses: The discourses of dichotomy, derision, myth and meaninglessness. At least two of these discourses can be found in Larsen’s critique of Hattie. One is the discourse of dichotomy, which ‘reduces everything to mutual exclusives, to a choice between grossly over-simplified alternatives’ (Alexander, 2011, p. 274). In Larsen’s critique, the two over-simplified alternatives are statistics and uniqueness. The second is the discourse of derision: ‘If you don’t like it, first misrepresent then ridicule it, personalizing the attack where possible and appealing to the lowest common denominator of popular prejudice’ (Alexander, 2011, p. 274). There is no need to point out, that Visible Learning is both misrepresented, ridiculed and personalized in Larsen’s statement: Hattie is presented as a researcher presenting truisms wrapped in hot, statistical air.

It is important to emphasize that although this type of critique might be entertaining in colourful political debates, it has nothing to do in a critical academic discourse, which must be sober, transparent and aimed at refutation, not at creating false dichotomies and misrepresentations. Many years ago, as a young researcher, I was in the middle of ridiculing an academic opponent. Just before firing the final punch, an older colleague stopped me and advised me as follows: First you must summarize the position of your opponent. When he or she finally nods in agreement, you can start your criticism. The ideal is the ideal of refutation, and the consequence of not respecting this ideal has been clearly marked by Karl Popper: ‘Those among us who are unwilling to expose their ideas to the hazard of refutation do not take part in the scientific game’ (Popper, 1968, p. 280).

Unfortunately, the article by Steen Nepper Larsen is far from unique. In the already mentioned book Hattie på dansk (Bjerre et al., 2017) several contributions place themselves outside the scientific game. In his chapter ‘Know Thy Impact – blinde vinkler i John Hatties evidenscredo’ (‘Know Thy Impact – blind angles in John Hattie’s credo for evidence’) in Bjerre et al. Hattie in Danish Steen Nepper Larsen writes that ‘….his [Hattie’s] models [seem] to have been written with blindness’ (Bjerre et al., 2017, p. 103), and '…not only the German philosopher and sociologist Theodor Wiesengrund Adorno cries – from the depth of the grave – in despair and pain, this is also the case for all the critical theoreticians to follow…' (Bjerre et al., 2017, p. 107). In the same book, Thomas Aastrup Rømer without any kind of factual support states that Hattie’s ‘…concept of feedback (…) has a centralizing effect, which in the end has the potential to change the pedagogical activities of a country into one huge, hierarchical and computerized organism’ (Bjerre et al., 2017, p. 156).

Other, critical reactions have been based on superficial readings of Hattie’s publications. The interpretation seems to be that as the aim of Visible Learning is to identify statistical measures between interventions and learning outcomes in order to identify ‘what works best’ (Hattie, 2009, p. 18), the implicit idea must be that there exists a simple causal relation between intervention and outcomes. Thus, Hattie must be a ‘behaviorist’, which among many Danish education researchers is used as a derogatory term denoting that those who are alleged to be behaviorists are simple-minded. One example of this interpretation of Hattie can be found in Steen Nepper Larsen’s article in Hattie in Danish. Here, Hattie’s analyses are compared with analyses of simple causal systems such as ‘pool games’ or mechanical cleaning systems with the predictable conclusion, that Hattie reduces teaching to a mechanistic relationship between pedagogical interventions and learning outcomes (Bjerre et al., 2017, p. 104).

This summary of Hattie’s understanding of the relationship between pedagogical intervention and student achievement is, however, both misleading and outdated. We must make a distinction between classical behaviourism from the early 1900s and modern behaviourism, which is informed by cognitive and/or social science (Qvortrup, 2013). The latter version of behaviourism assumes that there is not a simple relationship between input and effect, but that human beings learn and act in social situations and/or according to eigen-dynamics and intentions. Thus, behaviourism has developed from operating with simple input-output schemes to operating with conditionalized loops. A human being’s or a psychic system’s reactions on external influence is conditioned by psychological factors and/or by factors in the human being’s social environment. It is on this background that e.g. modern understandings of motivation understood as expectations of self-efficacy (Bandura, 1977) or of feedback structures and mechanisms (Hattie & Timperley, 2008) have been developed. This is the reason why one cannot assume simple causality, but in relation to quantitative analyses must work with statistical probabilities expressed in e.g. correlation figures. This explains Hattie’s above-mentioned emphasis that '[c]orrelates (…) are not to be confused with the causes' (Hattie, 2009, p. 3). This makes it even more unreasonable to accuse Hattie for operating with simple causalities. The simple conclusion is that the underlying intention is not to understand Hattie, but – with the expression of Alexander – to ridicule him, appealing to the lowest common denominator of popular prejudice.

In summary, the above mentioned critical reactions are not interested in developing a research-based understanding of the relationship between teaching and learning with the critical, transparent dialogue as the basic tool and with an open-minded reading of research opponents’ contributions to the mutual building of new knowledge. On the contrary, in these reactions academic disputes are transformed into cultural wars between opponents with the aim of first disarming the opponent (e.g. by ridiculing him or her) and then beating the opponent to death by appealing to the lowest common denominator of prejudice.

These critical reactions have gained momentum in the broader public debate in Denmark. In 2014, a majority in the Danish Parliament launched an ambitious public school reform. One of the important elements of the reform was to replace subject objectives with learning objectives. The final aim of education should not be what teachers teach, but what students learn. In this change, there were explicit references to Hattie and to Visible Learning. No-one has stated, it should added, that Hattie says anything about correlations between using learning objectives and increasing learning achievements. It has only been a case concerning the importance of making student learning visible, both for the educator and the student. However, the reform caused lots of critical reactions, not least among teachers, partly because the reform was closely coupled with a legislative intervention in teachers’ labour agreements, partly because it challenged traditional views on the teacher role. Consequently, the critical reactions from researchers were used as an academic legitimation of the political and interest-based critique of the school reform. Actually, a strong cultural understanding between academics, politicians and union representatives emerged, with the common view that replacing subject objectives with learning objectives was the root of all evil. Within a short period of time, a wide-spread interpretation community among researchers and practitioners emerged, changing Hattie from being an educational heroe to becoming an enemy of education. From here, it is quite natural to present those who have introduced Hattie in Denmark (including myself) as stupid ‘anti-theorists’.

On this background, it cannot come as a surprise that the introduction by Niels Egelund and me by some have been read as a message of ‘getting rid of theory’. The line of argument seems to be that persons supporting John Hattie’s ideas must be positivists, positivists are simple-minded, and simple-minded people do not like theories, because theories are far too complicated.

Observing positions

What is my own position in this debate? Actually, my position should be divided into two dimensions: One is epistemological, the other is ontological.

Concerning the epistemological dimension, the point is that there is not a dichotomy between making meta-analyses concerning the relationship between pedagogical interventions and learning achievements and recognizing that every teaching situation is unique. Making an analysis is performing a systematic observation, and all observations are made from a specific position and through a particular lens of observation (von Foerster, 1984). Both the position and the lens influence the picture, one makes, and pictures made through the lens of meta-analyses of course are different from pictures made through the lens of e.g. qualitative observation studies of classroom interaction. Only if one mixes up epistemology and ontology, one would think that meta-analysts deny that every individual interaction between students and teachers is unique. The statistical answer to the uniqueness of social interactions is probabilities.

No single theory can claim to be true, because ‘truth’ is not a case of ontology, but a case of epistemology. As expressed by Niklas Luhmann, ‘(t)ruth is not a property of things or sentences or cognitions (…). Rather, the term denotes a medium of the emergency of unlikely communication’ (Luhmann, 1990, p. 173). On the contrary, different theories represent different views on a given empirical subject. ‘Scientific truth’ is a concept for a special communication code with highly refined truth criteria. This is the reason why today we normally do not talk about scientific truth, but about scientific validity. One can observe something through phenomenological glasses with one result, or one can observe it through statistical glasses with another result (Qvortrup, 2018). If these different approaches are of high research standards, both operate according to highly refined validity criteria. Thus, they do not exclude each other, they complement each other.

The epistemological argument for this approach is summarized in the Danish philosopher Ole Thyssen’s book The philosophical view (Thyssen, 2012), According to Thyssen, the history of philosophy is a history of different views. ‘Although observation is a simple matter, it has not been identical at all times. It will always be performed from a position, that we call a view’ (Thyssen, 2012, p. 725, my translation). Great philosophers are great, because they have introduced new views on the world, and because these views have been accepted and have gained consensus in broader social communities. In a modern, functionally differentiated society, which among others is characterized by the fact that the existence of and the navigation between different views or so-called ‘optics’ has become a common condition, also philosophy has abandoned the ambition to find that view, which is better – more true – than all other views.

This does not imply that ‘anything goes’. But it implies that we have to be concerned with developing and respecting common criteria for validity. The criteria of validity and reliability (does the result answer the original question? Are the observation tools reliable?) are the modern scientific community’s answer to the loss of an ontological concept of truth.

Professional judgment: the missing link of Visible Learning

After having addressed the epistemological aspect, I will turn to the ontological discussion: What is my own position in relation to interpreting the relationship between on one hand school and teacher interventions and on the other hand student achievements? Nobody assumes, as Nepper Larsen indicates, that there is a direct, causal connection, such as the one between billiard balls. It is my assumption that in order to understand the relationship between educational interventions and outcomes we have to insert a missing link, e.g. Professional judgement. This assumption is based on the theory of bounded rationality, developed by Herbert A. Simon in the 1960s and 1970s. In his book The Sciences of the Artificial it is a basic statement that professional tasks such as teaching students are not simple tasks that can be conceptualized in an input-output model, but that they are complex tasks. A common characteristic of complex professional tasks is that the complexity of the task is bigger than the practical capacity of the professional (Simon, 1969/1996).

This idea led to the understanding that modern practising professions do not just ‘do’ a job or ‘solve’ a task. They solve their tasks based on professional judgements. According to R. K. Merton, three dimensions can be identified in this practice: A ‘helping’, a ‘doing’ and a ‘knowing’ dimension. The ‘helping’ dimension is based on ethical criteria for doing the job. Compared with non-professionals, professionals have an explicit ethical perspective in their practice, while non-professionals have not. The ‘doing’ dimension is based on a methodologically explicit practice. Professionals can refer to methodologically explicit principles for their practice, while non-professionals cannot. Finally, the practice of modern professionals is based on a ‘knowing’ dimension: their practice is informed by research-based knowledge (Merton, 1982). Non-professionals do not have access to relevant research-based knowledge.

This implies that research knowledge or methods cannot be conceptualized as an input that can be ‘translated’ into an adequate professional action or method. Similarly, there is not a simple causal link between a specific challenge in the classroom and the teacher’s reaction to this challenge. Both the ex ante knowledge and a specific situation in the classroom must go through the complex and unpredictable reflection loop of the teacher. In particular, research knowledge is only one out of three ‘ingredients’, which together represent the basic elements for performing an action. The combination of ingredients and the transformation into professional action is the result of professional judgement.

How can this judgmental process be described? This was the question that Donald A. Schön tried to answer. He did so by transforming these considerations into a description of professional practice including not only the practice in itself, but also a single- and double-loop reflection process. Based on Donald Schön’s considerations one could say that performing professionally, e.g. teaching, always includes reflection-in-practice and reflection-on-practice: ‘Reflection-in-practice’ represents the ongoing process of self-evaluation of and within the current practice: Should I change or modify what I am doing in this very moment? ‘Reflection-on-practice’ represents what happens after or outside the practice, e.g. together with colleagues. Here, the current practice is reflected on with the aim of considering whether it should be methodologically changed, whether the practitioner should qualify him- or herself in new directions, etc. (Schön, 1983).

Similarly, the relationship between teaching and learning is not at simple causal input-output relationship. On the contrary, with his or her pedagogical interventions the teacher can affect the complex cognitive system of the student, but the student will not react to the input as a coffee machine (pushing a particular button each time results in black coffee). It is more adequate to say, that the pedagogical intervention will ‘irritate’ the internal cognitive system of the student to ask itself, how it should react on the external 'irritation'. Again, the external intervention of the teacher must go through the closed, self-referential, complex and unpredictable reflection loop of the student, before it results in a – for each single case unpredictable – reaction of the student.

In summary, the basic challenge of teaching is that the complexity of the task, to affect and stimulate the individual student and the community of students, is bigger than the practical capacity of the teacher. The answer to this challenge is that one must increase the teacher’s action capacity, knowing that teachers must always work under the condition of a deficit of rationality. The fundamental tool for answering the challenge of bounded rationality is to use feedback as an ongoing correction mechanism. This can be expressed otherwise: all teachers work under the condition of technology deficit, i.e. under the condition that the effect of outcome-oriented efforts never can be fully predicted (Luhmann & Schorr, 1982a, p. 11ff). Again, the conclusion is that teachers must enact professional judgement. Based on knowledge of what will most likely work best, teachers must choose interventions in relation to a specific situation, which continually have to be modified. The teacher must ‘use situation relative causality plans and primarily orient him- or herself towards variable factors, i.e. after events’ (Luhmann & Schorr, 1982b, p. 27, my translation).

This explains why Hattie emphasizes that Visible Learning provides ‘an explanatory story, not a “what works” recipe’ (Hattie, 2009, p. 3), i.e. that the aim of the book ‘is to develop an explanatory story about the key influences on student learning (…), not to build yet another “what works” recipe.’ (Hattie, 2009, p. 6). This also explains the emphasis in Visible Learning that ‘correlates (…) are not to be confused with the causes’ (Hattie, 2009, p. 3). This finally explains, why feedback plays such an important role in Hattie’s publications, stating that ‘the most powerful single influence enhancing achievement is feedback’ (Hattie, 2009, p. 12. Cf. also, 2012, pp. 115–137; Hattie & Timperley, 2008; Hattie & Yates, 2014, pp. 64–71). The aim of Visible Learning is to support teachers’ professional judgement. One of the contributions of educational research is, as accurate as possible, to identify, which interventions and conditions with the greatest probability will lead to learning and personal development. The aim is not to provide teachers with what works best recipes. The aim is to provide teachers with ‘hypotheses for intelligent problem solving’ (Hattie, 2009, p. 247).

Disclosure statement

No potential conflict of interest was reported by the author.