Knowledge capital theory: a critical analysis using Lakatos’ idea of research programmes

ABSTRACT

This paper introduces and uses Lakatos’ idea of research programmes to summarise and critically evaluate academic discourse towards knowledge capital theory. The analysis uses rational reconstruction to formulate the components of the hardcore and protective belt of knowledge capital theory. By critically surveying the literature, it challenges the validity of the claims made by Erik Hanushek, Ludger Woessmann and the OECD that there is a causal link between cognitive development and economic growth. It concludes by stating that knowledge capital theory in its current form is degenerate and should be considered a high-risk research programme and that more sophisticated theories are required to be developed to explain current phenomena.

KEYWORDS:

• Human capital theory
• knowledge capital theory
• OECD
• international large-scale assessments
• PISA

Introduction

The OECD and ILSAs

Since the early 2000s there has been a meteoric rise to prominence in academic discourse, international education policy discourse and traditional media of the Organisation for Economic Cooperation and Development’s (OECD) international large scale assessments (ILSA), primarily the Programme for International Student Assessment (PISA). Increasingly their other testing products are also gaining more widespread attention such as: the International Early Learning and Child Well-Being Study (IELS), Assessment of Learning Outcomes in Higher Education (AHELO) and the Programme for International Assessment of Adult Competencies (PIAAC) (Moss et al. 2016; Shahjahan and Morgan 2016). In 2010, the World Bank’s survey Skills Towards Employment and Productivity (STEP) was launched with a goal of being closely ‘linked to PIAAC’ (Educational Testing Services (ETS) 2020), which helps feed into the World Bank’s new Human Capital Index. While not being an ILSA, the Human Capital Index uses the data generated by the OECD’s ILSAs (World Bank 2018). These new measures have undoubtedly superseded other ILSAs which focused mainly on ascertaining the effective delivery of intended curricula, such as The Progress in International Reading Literacy Study (PIRLS), The Trends in International Mathematics and Science Study (TIMMS) and Literacy Assessment and Monitoring Programme (LAMP) for global recognition and policy influence.

The introduction of PISA for Schools, which allows for school-level measurements and gives teachers and leaders access to data to use OECD ‘best practice’ to inform their pedagogy and curriculum is problematic (Lewis 2017). Lewis (2017) argues that the OECD’s influence extends much deeper now than at the national policy level. This change is due to PISA for Schools, and there is a risk teachers will apply the same narrowly normative changes as their national policymakers have done (Carvalho and Costa 2015; Breakspear 2012).

The question of cognitive development and GDP Growth

The success of this new global education regime, led by the OECD and supported by the World Bank, derives from claims made by Erik Hanushek and Ludger Woessmann (hereafter H&W). H&W claim that gains in cognitive development of a population are closely correlated to future GDP per capita growth. H&W also claim that the strength of the evidence is so strong that the relationship between the two can now be considered causal (OECD 2015). The relationship has since been utilised as a means for predicting GDP gains and centring policy recommendations to target students’ cognitive development as measured by ILSA test scores.

H&W outlined their claims in three important publications (Hanushek and Woessmann 2008, 2015; OECD 2015). The first work in 2007 published by the World Bank presented their research that claimed a correlation between tests scores from 1964–2003 and GDP per capita growth for the period 1960–2000 across fifty major economies. Their second paper built upon and expanded upon the first. The third, which was published by the OECD, did not contribute any further data, but notably considered the supposed causation as if it were now merely an empirical fact.

The attractiveness of their model’s predictions are obvious to national and economic policymakers. At heart, those at the OECD are econometricians (Aloisi and Tymms 2017). The causal relationships advocated by H&W, therefore fall nicely into this narrative.

Improving scores would not only justify the ‘aggressive’ education reform, but that in the medium to longer-term the reforms will pay for themselves and much else besides (Komatsu and Rappleye 2017). These claims were seized upon and incorporated by the OECD to justify international assessment benchmarking and education reform. The assertions on causality between test scores and per capita GDP growth formed the basis of the third paper published by the OECD and authored by H&W. There are many positive political and economic dividends aggressive reforms would provide. It is not difficult to understand why governments, often democratically elected, have used these ideas as much of their basis to provide a narrative of ongoing school improvement leading to wider increases in competitiveness and strength of their economies.

The critical difficulty before H&W was that human capital theory is much more complicated than knowledge capital theory and considers many other factors as playing a significant role in supporting economic gains. Human capital theorists continued to be unsure if increasing education quality or expanding access contributed most to economic betterment (Komatsu and Rappleye 2017). The arguments over quality vs access were intractable for most of the period from the 1970s-1990s. Previously policymakers had to operate with a complex array of priorities. Policymakers needed to consider countries’ stage of development, changing contexts, means of measurement, feasibility etc. The vital distinction advocated by knowledge capital theorist such as H&W is: the quality of education is what matters most (OECD 2015). Once the position that what mattered the most was citizens’ cognitive development, then, arguments about the developmental stage and other priorities begin to recede (Komatsu and Rappleye 2017). PISA and other ILSAs provide the longitudinal and cross-national data needed to demonstrate many of the claims made by knowledge capital theory. The empirical work and the longitudinal sample of over 50 countries have become the foundation of a new logic for forming education policy (Rutkowski and Rutkowski 2016). What works for high-income countries is plausibly considered to work for low-income countries, given the right implementation. The discussions over priorities of what should be changed has given way to technical discussions regarding how to measure test scores and how to improve them.

Previously the concept of ‘universal basic skills’ had multiple interpretations. The OECD’s definition has begun to be considered the de facto definition. The OECD defines universal basic skills as those ‘basic skills’ measured by Level 1 PISA tasks (OECD 2015). Since a straightforward testing instrument had become available, it was coupled with H&W’s strong statistical relationship. Consequently, some quite extraordinary extrapolations have been made. For instance, by 2030, Ghana would increase current GDP by 944% (Schleicher and Qian 2015, 10). They go on to extrapolate that all lower-middle-income countries could expect a gain of 627% (Schleicher and Qian 2015, 55–56). The change required, render these numbers somewhat meaningless, but it serves as a foretaste of what could be achieved. Given the potentially astronomical GDP gains by improving Level 1 PISA tasks, the OECD’s necessary policy reform becomes a magic bullet for national policymakers.

There are concerns related to the legitimacy of: the mandate of the OECD (d’Agnese, Vasco 2015), methodological robustness of the testing instruments used in PISA (Zhao 2020; Goldstein 2018) and the rationale of the purpose of education underpinning the entire enterprise (Komatsu and Rappleye 2019).

In the next section, Lakatos’ (1978) notion of research programmes will frame current critical academic discourse. It will recommend that the knowledge capital research programme on which the OECD’s and World Bank’s ISLA’s are based appears to be in a degenerate state.

Lakatos’ notion of research programmes

The knowledge capital theory research programme

The assertion by H&W that increases in per capita GDP is causally related to improved cognitive development is challenged on a fundamental level by the Duhem-Quine thesis (DQ). DQ asserts that, ‘it is not possible to falsify single hypotheses because it is invariably conjunctions of hypotheses which are being tested.’ (Cross 1982, 320). Therefore, if a particular hypothesis is challenged with empirical evidence, the most we can say is that ‘… the conjunction of the particular hypotheses with a set of auxiliary hypotheses is false’ (p320). We cannot be sure that the anomalous empirical evidence is not explained by one or more of the auxiliary hypotheses. The foremost issue is that the participants in the debates about knowledge capital theory have primarily ignored the jointness of hypothesis testing. The central focus of knowledge capital theory has been mostly in single target hypotheses, such as, ‘better qualified teachers improve student outcomes’, which are considered in isolation from supporting auxiliary hypotheses. The assumption in this line of reasoning is that it is possible to falsify a single target hypothesis, such as falsification theory (Popper 1963), which DQ contends is wrong. We must therefore use supportive auxiliary hypotheses in conjunction to appraising any target hypothesis. The only truth that falsification establishes is the denial of the hypothesis. A denial does not necessarily imply any single ‘opposite’ hypothesis, but instead, entails multiple alternative conjunctions of hypotheses; some may, in fact, contain the target hypothesis.

The starting point for using Lakatos (1978) is to view the networks of hypotheses being tested as whole structures. The most important subgroup of hypotheses is called the hardcore. The collection of hardcore hypotheses defines a research programme. By agreement of researchers the hardcore is protected from being challenged empirically or from descriptive forms of arguments by a negative heuristic. The negative heuristic is how problems are not solved. The remainder of the hypotheses constituting a research programme are called the protective belt, which, as the name suggests, protects the hardcore from challenge. If hardcore assumptions are changed, it no longer describes the same research programme.

Underlying the research programme is a dynamic force named the positive heuristic, or, how problems are solved. The positive heuristic determines how the protective belt is adapted to protect the hardcore from empirical or logical challenges. Therefore, the research programme evolves rationally over time in response to the challenges made to the protective belt. A key tenant of Lakatos is that progressive research programmes can predict novel and more precise data or facts. Conversely, a degenerate programme shows a lack of growth; the belt of auxiliary hypothesis, supported by the positive heuristics do not lead to valid or novel predictions.

In contrast to Kuhn and Hawkins (1963) advocated that science followed long periods of ‘normal science’ to ‘paradigm change’. Earlier, Kuhn rejected the idea the paradigm change occurs without scientific reasons. Later, however, Kuhn argued for a more relativist interpretation of scientific change, and that scientist did not use rules to arrive at their decisions to choose the most successful paradigm. This view faced criticism, since, it seemed to suggest science was, in reality, irrational. Lakatos developed his notion of a progressive research programme to reconcile the rational positivist process of falsification of Popper and the more relativist Kuhn (Blaug 1975; Maxwell 2005; O’Donohue, Ferguson, and Naugle 2003). Importantly, when ‘paradigm shifts’ do occur, Lakatos argued it is as a result from a research programme transitioning from a degenerate state to a more progressive state. Consequently, the paradigm shift in Lakatosian research programmes are mostly rational, not irrational, as Kuhn suggests. It is for these reasons that the Lakatosian research programme is used to evaluate knowledge capital theory.

Lakatos used the term rational reconstruction. It refers to what the researchers in a research programme have been committed to historically and logically. A rational reconstruction of knowledge capital theory follows:

The hardcore (HC)

1. Increasing knowledge capital (as measured by student test scores) leads to increased GDP per capita growth.

2. Knowledge capital can be measured using testing instruments.

3. Purpose of education is for economic gain on the level of the agent (individual person) in an economic system.

4. Educational reform policies successfully increase knowledge capital.

Positive heuristics

1. Go forth and construct theories in how to increase knowledge capital.

2. Go forth and develop predictions based on the hardcore assumptions.

3. Develop comparative large scale testing instruments to measure knowledge capital growth.

4. Devise recommendations for national education policymakers.

Negative heuristics

1. Do not build theories in which the primary purpose of education is not for economic gain.

2. Do not construct theories in which schools do not affect student outcomes.

3. Do not construct theories in which the measuring of knowledge capital is not possible or is invalid.

4. Do not test hardcore propositions.

A critical summary of academic discourse. The hardcore (HC) of knowledge capital theory

Following a rational reconstruction of the knowledge capital research programme, the next section uses a ‘Point: Counterpoint’ style to survey the current academic debate concerning each of the hardcore’s four parts.

HC1: increasing knowledge capital (measured by student test scores) leads to increased GDP per capita growth

Temporal mismatch

Ramirez et al. (2006) and Kamens (2015) showed that the original analysis by H&W compared test scores and economic growth from the same period. They argued forcefully that the temporal comparisons by H&W are not appropriate. The period for the study of economic growth should follow that of the testing since it takes time for these students to enter the workforce. Both also showed by using similar data; the strong statistical link was no longer evident for test scores to impact future economic growth.

In response, H&W re-analysed their data and compared the testing period of 1964–1984 and the economic growth between 1985–2007 (OECD 2015). In contrast to Ramirez et al. (2006) and Kamens (2015), they were able to re-confirm and strengthen their argument about their original claims’ validity. The reasons why the three sets of researchers came up with differing results is a little unclear. It appears to revolve around different types of data and analysis used (Komatsu and Rappleye 2017). Kamens (2015) used test score data for a specific point in time, while H&W (OECD 2015) used long-term average test scores. H&W (OECD 2015) processed their data by interpolating and extrapolating the missing data, Ramirez et al. (2006) did not use a similar technique. Lastly, Sahlgren (2014) simply puts it down to Ramirez et al. (2006) not having access to the most up to date data and that they analysed too few countries for the trend to be apparent.

Extrapolating historical trends and making future projections

Klees (2016) and Stromquist (2016) have many criticisms of H&W’s work, but we will concentrate on those raising doubt regarding the internal validity of H&W’s claims for our purposes. Both authors take issue that H&W used the link between historical test scores and economic growth and subsequently extrapolated it for the future. Both Klees (2016) and Stromquist (2016) argued that it is not appropriate, as many factors affect economic growth, and H&W did not control for them. Klees (2016) cautions that policymakers should not take the projections given by H&W seriously. While a formal retort has still not been forthcoming from H&W, Komatsu and Rappleye (2017) suggest that H&W have in a way already pre-empted the criticism. They note that H&W argued that additional factors appeared to be inconsequential (OECD 2015, 91) and that they had already considered the relationship being a predictive one and concluded that it was (OECD 2015, 90). A formal reply by H&W has yet to appear in print, but this will likely be their response.

Validity of statistical claims made by H&W

Much deeper methodological concerns have been raised regarding the inappropriate criticism of the parameters used by Hanushek and Woessmann (2015), (Klees 2016; Komatsu and Rappleye 2017, 2019; Stromquist 2016). A methodological point has long been made of being cautious of the explanatory power given by the statistical significance (p) and the slope of the regression line for very large samples of data (Nuzzo 2014). What matters most is the explanatory power of test scores (Komatsu and Rappleye 2017). It is how much the variation seen in economic growth is explained by test scores which will produce a more robust analysis. The more robust analysis is given not by statistical significance (p), but by the determination coefficient (R²). Komatsu and Rappleye (2017) reviewed the same historical data and expanded on it to include the most recent PISA and PIAAC data (Rappleye and Komatsu 2019). They showed that when considering R², the apparent relationship between test scores and economic growth made by H&W becomes invalid. The work of Klees (2016), Ramirez et al. (2006) and Stromquist (2016) barely mention R², if at all, and so Komatsu and Rappleye (2017) contend the criticism is mostly misdirected. The focus on statistical significance allows H&W (OECD 2015) to quickly counter criticisms, since, given the large sample sizes, H&W are easily able to provide statistically significant results. A rigorous response by H&W regarding R² analysis by Komatsu and Rappleye (2017) would be beneficial to move the argument forward.

HC2: knowledge capital can be measured using testing instruments

Potential assessment bias

PISA assessments have faced criticism for potentially being biased and that this bias has impacted the performance of different groups of students (Zhao 2020). There are multiple areas which have come under scrutiny for potentially being biased; question formats, language, culture, constructs and text types (Hopfenbeck et al. 2018; Solheim and Kjersti 2018; Zhao 2020). PISA tests are 18% longer in German than in English; however, the assessment length stays the same (Shiel and Eivers 2009). There are linguistic divergences between Greek textbooks and PISA items, and as a result, this could help explain poor Greek performance in science (Hatzinikita, Dimopoulos, and Christidou 2008). Reading assessments are biased against boys in Nordic countries because the test items are ‘girl-friendly’ due to the type of writing required to document their reading comprehension (Solheim and Kjersti 2018).

Additionally, PISA is less comparable across Asian or Middle Eastern countries due to cultural and linguistic reasons (Grisay, Gonzalez, and Monseur 2009). In response, there have been some adjustments of late of the methodologies used to counter some of the above criticism. It is to this response by PISA we address next.

Item response model

There are concerns regarding the item response model used by PISA. The first was the lack of item-by-country interactions; without these, important and interesting differences could be ‘smoothed out’ which could jeopardise cross-national comparability (Goldstein 2018). Simply put, some testing items are treated as harder, for example, in Scotland than in England, which on a conceptual level seems difficult to justify, but not so given the biases discussed above. In 2015 PISA introduced item-by-country interactions. In a recent analysis of the 2015 PISA data set; there was no evidence to suggest that the inclusion of the item-by-country interactions provided a particular advantage or disadvantage for any of those countries in the sample (Jerrim et al. 2018).

Until 2015 the method for treating items not reached was to consider them as being incorrect. A longstanding concern was that those students from disadvantaged backgrounds were the most likely not to finish within the time limit; disproportionately affecting low achieving students (Bridgeman et al. 2004). Post 2015 PISA items not reached no longer contribute to proficiency scores. Jerrim et al. (2018) found once again that this change had almost no significant impact on the results.

Rasch model

A recurring criticism of PISA is that it uses the Rasch model, a well-known psychometric model named after the mathematician and statistician George Rasch (1901–1980). Kreiner and Christensen (2014) stated that PISA did not meet the Rasch model’s necessary criteria to be used appropriately. They argued that as a result of using an inappropriate model for the data, the resulting country rankings were not robust. Other researchers analysing the same data assert that the PISA data is more appropriately interpreted using a multidimensional model with two factors (Hopfenbeck et al. 2018). Jerrim et al. (2018) re-analysed the data and found no significant difference when using the Rasch model or supposedly more robust multilevel model with two factors. Jerrim et al., state that the vocal criticism of the methods used in analysing PISA data appear to be ‘overblown’ and the models used previously were ‘good enough’ (2018, 37). This contribution is a significant blow to those critics. Surprisingly, it has yet to feature prominently within recent reviews of the literature. For instance, the absence of any reference to Jerrim et al. (2018) in the recent work by Zhao (2020) who is a vocal critic, is puzzling. However, Jerrim et al. (2018) do caution that the robustness of the analysis may not be replicated when a more sophisticated three factor multi-level analysis is applied.

Sampling problems

Appropriately sampling populations so that they are genuinely representative is an area that has received close attention (Zhao 2020). Students with specific characteristics are excluded from the assessments as long as the total exclusion rate is below 5% (Hopfenbeck et al. 2018). However, during the 2012 cycle, for example, eight educations systems had an actual exclusion rate of above 5%, making robust comparisons more difficult (Rutkowski and Rutkowski 2016). The types of pupils who are excluded matters too.

Students with disabilities are omitted. Disabled pupils are, in reality, a vital indicator of the equity and quality an education system exhibits. Since students with disabilities and special educational needs can be excluded; there is little incentive for their inclusion in improvement initiatives, since they won’t contribute to rising PISA scores. This practice further marginalises those pupils with disabilities and special needs (Schuelka 2013).

Those students already marginalised for economic and societal reasons, such as the children of migrant workers in Shanghai who are excluded from the public school system (Chen and Feng 2013). These migrant students are not included in the sample, leading to considerable criticism of Shanghai’s ‘real’ performance (Loveless 2014). Those 15-year-old students not enrolled in schools are also not included because PISA is administered in schools. PISA aims to capture a measurement of the human capital development of those 15-year-olds; this means that those 15-year-olds who will also contribute to the workforce but not enrolled in school are not included.

Lastly, to students included, PISA does not differentiate between the starting ages of students in their respective school systems. Therefore, in some countries, students have had more exposure to their curriculum (Zhao 2020). Further, it is a test of 15-year-olds and so includes students at different cohort grade levels. There are significant concerns that these shifts of the proportions of 15-year-olds in different grade levels are a crucial factor in explaining between-cycle-variations (Aloisi and Tymms 2017). The OECD notes that in response to such concerns regarding exposure to the curriculum that unlike TIMMS or PEARLS, PISA does not assess curriculum focused skills and knowledge, however, exposure time to the curriculum would surely matter in PISA results (Hopfenbeck et al. 2018).

HC3: purpose of education is for economic gain on the level of the agent (individual person) in an economic system

Before considering knowledge capital, it is instructive to consider the role of education in the labour market from the perspectives of the different theories available, the summary of what follows uses the recent work of Shields and Sandoval Hernandez (Shields and Hernandez 2020) as a jumping-off point.

Human capital theory

Human capital theory has more or less become the de facto model for conceptualising the labour market and education (Shields and Hernandez 2020). The interplay between the labour market and knowledge was developed as a framework for understanding the variation in education and earnings of individuals (Mincer 1958). In its original form, the wages of a worker were seen as a return on investment (Becker 1994). Modern formulations of human capital theory consider the role of highly skilled labour and analyse the skills premium it can demand (Shields and Hernandez 2020). Knowledge capital theory at heart considers the cognitive and skills development of the worker as one of the factors in determining wage levels.

Other theories have emerged in addition to the classical theories of human capital (Shields and Hernandez 2020). Signalling refers to the idea of educational qualifications being used by employees to demonstrate their abilities using a trusted third party verification to potential employers (Spence 1973). The apparent status of the third party, such as an admission to a highly ranked university alone, can work to signal comparative advantage (Hoekstra 2009). The flipside to signalling is screening (Stiglitz 1975), whereby an employer sets ‘wage contracts’ for consideration for a particular salary, a specific educational qualification is required (Stiglitz and Weiss 1990; Stiglitz 1975). Both theories suggest that unlike human capital theory, part of the role of qualifications is to reduce uncertainty in the job market (Shields and Hernandez 2020). Those with the qualifications from a reputable institution are also likely to have higher cognition and skills levels. In all three theories above, education holds a unique role in labour markets; it is seen as both an absolute and a positional good (Matthews and Hirsch 1977). Absolute goods can be thought of as the size of a cake, with increasing the size of the pie for all, positional goods can be thought of as increasing the individual’s slice of the pie (Shields and Hernandez 2020).

Knowledge capital is an extremely narrow version of human capital theory; only the cognitive ability measured using test scores that matters (Rappleye and Komatsu 2019). Shields and Sandoval Hernandez (Shields and Hernandez 2020) used both qualifications, income and cognitive levels as measured by PIAAC to produce an international comparison of the roles cognitive skills and qualifications have on income. Interestingly, they found that broadly the level of cognitive skills is comparable with qualifications in their explanatory power for income; their results held for both the individual level and those as an outcome of the school system. However, they also found that there was a differing ratio of signalling to cognitive skills in different countries. The United States, in particular, appears to be a unique case where signalling of qualifications is rewarded disproportionately by the labour market over the actual cognitive skills workers possess.

The relationship between skills, signalling and so forth in national economies is now even more complicated due to globalisation. Previously, workers may have been able to accrue a ‘skills premium’ in the labour market. The increased mobility of workers from different countries suggests that this premium is now in question (Lauder, Brown, and Cheung 2018).

The forecasted impacts of automation further call into question specialist knowledge in certain high-risk industries (Hawksworth, Berriman, and Goel 2018; Kattan, Macdonald, and Harry 2018). Supply and demand of skills and qualifications is just one part of the picture. The composition of national economies, labour market institutions, the regulatory environment, historical and geographical contexts all play a role and as such the picture is much more complicated (Shields and Hernandez 2020). This is an interesting counterpoint to H&W’s assertion that cognitive development is the primary determinant of future GDP growth.

While there are indications that cognitive development does have an impact on individual earned income, the relationship between these skills and productive GDP development remains unclear. This further calls into question the use of knowledge capital rather than more nuanced theories to explain the role of education and the labour market.

The social compact promoted by the OECD from the viewpoint of the individual citizen of ‘… more learning leads to more earning’ is no longer tenable and has been fiercely opposed and criticised (Brown, Lauder, and Cheung 2020; Lauder 2015; Brown, Lauder, and Ashton 2010). In Bourdieu’s (2013) terms, the OECD uses a performative semiotic construction of globalisation concepts; it implies that only a neo-liberal kind of globalisation is valid and denies other accounts (Sellar and Lingard 2014). In an increasingly interdependent globalised economy, the pressure on systems is to be both economically prosperous and providing high-quality education. It is difficult to imagine that a developed economy would turn its back on this education model despite the inconclusive nature of the evidence that is purported to support it.

HC4: educational reform policies borrowed from more successful countries successfully increase knowledge capital

Policy borrowing and the importance of context

Different countries have often adopted policies on the OECD’s advice without a clear understanding or appreciation for local cultural, historical or socioeconomic considerations being taken into account (Salokangas and Kauko 2015). Finland is an illustrative example of a country which has leant many policies. Salokangas and Kauko (2015) explain that many of the Finnish comprehensive school model features were as a result of a longstanding political consensus which was agreed by the main political parties in the 1970s (Simola 2015). Therefore, education has not featured as a core political issue that has contributed to stability (Kauko et al. 2015). One of the central tenants of the Finnish model, is of a decentralised school system. The decentralisation, however, resulted from an economic depression in the 1990s and the need to make it easier for national-level decision-makers to delegate the detail of the budget cuts down to the local level. It was initially not a deliberate initiative to improve education quality (Simola 2015).

The political and cultural consensus in Finland has begun to show cracks. School choice initiatives have been trialled, without formal testing scores, comparative league tables, or school inspection reports. As a result, parents are exercising school choice based on families’ socioeconomic background that attend particular schools (Salokangas and Kauko 2015). In particular, in urban areas, this has led parents to conceptualise a ‘bad’ school from superficial criteria (Kosunen 2014). Many parents cite socially and ethnically mixed student populations as reasons to change school. However, the policy borrowing and lending from countries like Finland continues apace, even when prominent scholars from Finland advise against the de-contextualisation of policy and its subsequent importation into wildly different contexts (Sahlberg 2011).

The OECD professes not to encourage such policy borrowing. It states that it is up for the individual national governments to carefully consider which policies are applicable in their context (Schleicher 2018). In the same book, Schleicher speaks with pride on the phenomena of ‘PISA Shock’ and its effect on policymakers, see the case of Germany as an illustrative example (Gruber 2006; Ringarp 2016). In Sweden, PISA data was used selectively for party political purposes to legitimise their reform agenda and policy borrowing arguments. As a result, reforms are narrowed down to practice or teaching-oriented problems (Lundahl and Serder 2020). Who should countries choose to borrow from? In their overview of policy discourse Davis, Wilson, and Dalton (Davis, Wilson, and Dalton 2020) found that PISA performance was consistently sighted as a reason for policy borrowing from countries such as Finland and more recently East Asian cultures.

One example of a policy reform initiated by analysing top-performing countries was that of teacher recruitment. The policy recommended that schools recruit teachers from high school graduates’ top echelons because that is what high performing systems do (Barber and Mourshed 2007). These recommendations, however, resulted from erroneous observations (Zhao 2018). Research has shown that the top graduates tend to benefit the top-performing students in a system but at the expense of low performing students (Grönqvist and Vlachos 2016). Therefore, the policy initiative may raise PISA scores, but at what cost? (Zhao 2017).

The rise in policy borrowing from East Asian cultures has increased recently but is framed in a manner which continues the legacy of Orientalism (You 2019). Countries, like the UK, are attempting to replicate some of the success of countries in mathematics on PISA. One system they have looked at closely is that of as Shanghai’s (Chen and Feng 2013; d’Agnese, Vasco 2015). Policymakers have studied how mathematics is taught in England and China to try and replicate China’s consistently high scores (Miao and Reynolds 2017). This cross-national attraction eventually leads to policy borrowing (Davis, Wilson, and Dalton 2020). It is ironic that from the Chinese perspective, the strong performance on the PISA rankings has developed into a barrier for reform. The reform in China intends to lower the pre-eminence of the high stakes test-taking in university admissions (Ross and Wang 2010). This examination based culture has been a feature of China’s education system for hundreds of years, and reform has centred on increasing creativity and improving student wellbeing (Zhao 2014). But are policy reforms worth it? Do they lead to improvement in PISA scores?

Does policy change account for changes in ILSA scores?

An examination of some specific case studies where reforms have been cited for the reason for changes in PISA test scores will be addressed below to try to highlight some of the criticism against the idea that policy reform will lead to improved PISA scores.

In 2001, Qatar based their reforms on OECD recommendations (Guarino and Tanner 2012). The reforms included a significant intervention regarding the formation of publically funded independent schools (Zellman et al. 2009). After the reforms, Qatar increased its PISA score by up to 75 points, and in TIMMS 2007–2011 the change was over 100 points. Several points should be considered before apportioning the change to the scores to the reforms alone.

The first point is that the achievement gap remained unchanged between public schools’ performance, which included the new independently funded school and private schools. Private schools did better, even though they were not affected by the policy reforms (Cheema 2015). Secondly, the adjusted PISA scores showed that the gains between 2009–2012 are because of the student populations’ socioeconomic and demographic changes (OECD 2014). Lastly, the main driver of the improvement was the performance of immigrant students in Qatar. Immigrants are a group who routinely outperform local students in Qatar (Cheema 2014).

A case where reforms of the curriculum were more plausible was that of Ireland. In PISA 2009, they experienced a significant drop in reading and mathematics scores, and this was ascribed to, like Qatar, as a result of changes to the demography being sampled (Cosgrove and Cartwright 2014). Science scores however stayed level. As a result, the Educational Research Centre in Dublin suggested that earlier reforms to the primary and secondary curriculum could have mitigated the demography changes and kept the scores stable (Perkins et al. 2011). It is an interesting assertion, given that the policies implemented at the time were focused on reading and mathematics and not science. Even the evidence of implementing those curriculum reforms was best considered moderately successful (Varley, Murphy, and Veale 2008). Conversely, Ireland’s mathematics scores did not change when the curriculum was first revised in 2000, but scores did begin to change when examinations were modified to be more PISA-like (Perkins et al. 2013). A trend of teaching to the test that incidentally is not relegated to Ireland alone (Breakspear 2012).

A case where scores drastically decreased following educational reform is that of Argentina. Between 1994 and 1998, new curricula were introduced, following these reforms, a new framework to help ensure the system was more equitable was introduced in 2004 (UNESCO-IBE 2006). From 2000 to 2006, Argentina experienced a 45 point drop in reading scores (Aloisi and Tymms 2017). The narrow viewpoint of considering educational reform and student reform as an isolated system inevitably leads to oversimplification. During the same period under question, Argentina underwent a severe economic crisis leading to significant job losses, a sovereign debt default and social unrest. Is it unreasonable to consider that those students who spent a large proportion of their education during this tumultuous period would do worse in 2006 than those students in 2000? It is important to consider the social impact in part as an explanation rather than attributing it to catastrophic reading and literacy reform alone. Realistically, it is difficult for policymakers to justify keeping reforms which have appeared to fail so abjectly in improving performance, despite evidence that the reasons for failure a varied and complex.

The causes of between cycle’s fluctuations in PISA scores are essential to understand. While PISA should be commended for stability in the results, year on year errors might account for 20 points or more, being just a fifth of a standard deviation is an impressive feat for such a large scale assessment (Aloisi and Tymms 2017). Unfortunately at the same time, countries go through the PISA shocks and recriminations from falling down the tables. A particular example following a new scaling approach in 2015 may have affected several countries’ trend comparisons (OECD 2016). Korea was reported to have ‘lost’; 13 points between 2009 and 2015, wherein fact these are within the margin of error. These potentially non-significant changes are not portrayed as such in the media, and as a result, the pressure for policy change to find solutions increases (Harris and Jones 2017).

The effect of policy changes on PISA scores is critical in assessing whether policy borrowing based on strong school system performance on PISA tests is valid. When considering socioeconomic and demographic predictors, there was a correlation of p = 0.99 of a countries performance between two cycles of PISA (Aloisi and Tymms 2017). In other words, almost all of the variations seen in the scores could be attributed to changing proportions of grade levels accounting for the 15-year-olds and their socioeconomic status. In the same study, Aloisi and Tymms (2017, 206) showed that the impact of reforms on test scores amounted to annual effect size of around just p = 0.02, suggesting very little correlation. A cautionary note must be raised here, earlier in this paper attention was drawn to the proposed use of R² and not p to give a more robust analysis, no R² analysis was included to accompany p values in this instance.

In defence of PISA, they do publish adjusted scores based on demographic and other factors. However, these scores are not published prominently in the league tables for public consumption, but are relegated to an appendix. Further, while Aloisi and Tymms’s (2017) conclusions bring into question much of the evidence used by policymakers to justify policy borrowing and reform, importantly, it is a model that they have recently developed, and it suggests correlation; causation would need further research.

Conclusion

This paper considers the question: is the knowledge capital research programme in a degenerate state? The criticisms of the hardcore of the knowledge capital research programme would seem to suggest that it is. The hardcore is being consistently and effectively criticised, and it has yet to give novel predictions. The case of (Komatsu and Rappleye 2017; Rappleye and Komatsu 2019) in particular should be of considerable concern to proponents of knowledge capital theory, a criticism that has yet to be met with a response by H&W. Further, the work of Aloisi and Tymms (2017) on the impact of policy reform on PISA performance requires further attention.

The influence of H&W within the OECD and the World Bank is considerable. The role the two organisations play as trans-national policy actors, means the likelihood of a new research programme supplanting it appears remote. Lakatos’ research programme structures this analysis. Yet, the degenerate knowledge capital research programme appears to be following more along the lines of Kuhn; the question is not if a paradigm shift will happen, but a question of when and at whose insistence given the influential proponents of the current paradigm dominating academic discourse. What may replace it? The work of Brown, Lauder, and Cheung (2020) and their New Human Capital Theory deserves serious debate as a potential new paradigm.

A limitation of this paper is that it uses the knowledge capital research programme concerning ILSAs. The knowledge capital perspective nevertheless represents an extreme and limited interpretation of human capital. Using Lakatos’ research programme to analyse whether the more representative human capital research programme concerning ILSAs is in a degenerate state is the next logical progression of this paper. Implicit in both analysis’ is a suggestion that underlying both research programmes is a lack of theory in both the selection, operationalisation and explanation of the relationships between the variables it uses. An attempt to develop a theory of scientific method that supports alternative research programmes to take ILSAs forward would be constructive to the ongoing academic discourse.

Disclosure statement

No potential conflict of interest was reported by the author.