The contributions of scientific realism and critical realism to realist evaluation

ABSTRACT

Realist evaluation has gained prominence in the field of evaluation in recent years. Its theory-driven approach to explaining how and why programmes work or not makes it attractive to many novices, early career researchers, and organizations implementing various programmes globally and relevant to policymakers and programme implementers. While realist evaluation seeks to be pragmatic, adopting principles and methods that can be used to help focus an evaluation, its deep ontological and epistemological foundations make its application in real-life situations challenging. In this paper, we seek to unpack the key tenets scaffolding the practical application of realist evaluation. Although Pawson and Tilley foreground realist evaluation in applied scientific realism, we argue that an amalgam of scientific and critical realist principles underpins realist evaluation. We unpack these principles and illustrate how they fit into each other to provide a cogent theoretical foundation for realist evaluation.

KEYWORDS:

• Realist evaluation
• critical realism
• scientific realism
• retroductive theorizing
• mechanism

View correction statement:

Correction

Introduction

Pawson and Tilley (1997a) introduced realist evaluation through their seminal work Realistic Evaluation as a practical approach to realist-informed research. Since then, this proposed approach to evaluating complex interventions¹ has been used in many fields of evaluation sciences. Realist evaluation is a form of theory-driven evaluation characteristically set apart by its explicit explanatory nature and philosophical foundation underpinnings regarding underlying (social) structures and generative (social) mechanisms (Renmans et al. 2022). Although realist evaluation is strongly underpinned by a philosophical metatheory (Realism), like other evaluation approaches, it seeks to be pragmatic, adopting principles and methods that can be used to help focus an evaluation. Nevertheless, difficulties pinpointing realist evaluation’s philosophical and methodological foundations, especially among novice realist researchers, have been noted (Marchal et al. 2012). Therefore, identifying the underlying realist principles and translating their practical implications for realist evaluators is critical to improving its uptake and robust application.

Realist evaluation focuses on unravelling the programme theory guiding how and why a programme works or not (Mukumbang et al. 2016). The programme theory’s explanations should refer to unobservable entities (mechanisms) whose behaviour is responsible for the observation made after an intervention is implemented. Realist evaluation aims to explain the processes involved in introducing an intervention in a particular context (with certain characteristics and groups of actors operating within it), and the outcomes observed after the intervention implementation (Porter and O’Halloran 2012). It is, thus, particularly relevant to evaluations involving human services, social policies, health interventions, and Implementation Science outcomes (Jagosh 2020) and seeks to unpack ‘what works, for whom, in what respects, to what extent, in what contexts, and how?’ (Pawson and Tilley 2004).

Considering that scientific work and practice are embedded within a philosophy of science or meta-theory (Allana and Clark 2018), realist evaluation is also entrenched in realist ontological and epistemological foundations. Therefore, for the proper application of realist evaluation, there is a need to address the weaknesses and contradictions in the meta-theory guiding realist evaluation (Porter 2015a). Many researchers conducting realist evaluation, including the RAMESES II study for developing guidance and reporting standards for realist evaluation, situate realist evaluation in the general philosophical position of Realism (Greenhalgh et al. 2015). Scientific realism and critical realism are the two prominent forms of Realism. Pawson and Tilley (1997a) specifically situate realist evaluation in scientific realism. Because scienfic realism and critical find their roots in Realism they thus have a lot in common. Given the similarities shared between scientific realism and critical realism, critical realism’s contribution to realist evaluation remains a source of confusion and frustration among many Ph.D. students and early career researchers conducting realist evaluation. To this end, this paper aims to identify fundamental principles of scientific realism and critical realism underpinning realist evaluation and explain how they contribute to retroductive theorizing in realist evaluation.

Realist evaluation and scientific realism

Scientific realism² constitutes a family of doctrines about the aim of science (Sankey 2001). What most characterizes scientific realism as a form of Realism is its metaphysical doctrine about the external world. Pawson and Tilley (1997a) explicitly reported that their ‘contribution to evaluation is distinctive in that it is the first to rest on realist principles’ (55). They situated realist evaluation specifically in scientific realism and described it as applied scientific realism. Indeed, they have described their version of evaluation as a ‘scientific realist evaluation’ (Pawson and Tilley 1997a). In chapter three of their book Realistic Evaluation, Pawson and Tilley (1997a) outlined some fundamental principles of the scientific realist paradigm guiding their proposed realist evaluation.

To the novice practitioner interested in realist evaluation, Pawson and Tilley’s (1997a) explicit insinuation that realist evaluation is predominantly informed by scientific realism and Pawson’s (2013) eventual distancing from some of the works of Roy Bhaskar, would seem that scientific realism predominantly informs the practice of realist evaluation. More recently, Pawson’s choice of drawing from Bhaskar’s work in the sciences in the development of realist evaluation has been questioned (de Souza 2022). Pawson’s straying from Roy Bhaskar’s works seemingly stems from a concern that, Bhaskar’s outpourings increasingly adopted a normative and emancipatory nature. It was highlighted that Bhaskar’s work in the human sciences, instead, would be more suited to the social science where programme evaluation (Rossi, Lipsey, and Henry 2018), permits more to be explained and critiqued. Consequently, evaluation practitioners suggests that Bhaskar’s philosophical works did not delve into scientific realism. This is far from true as Pawson (2013) himself, acknowledges drawing from Bhaskar’s ‘A Realist Theory of Science’ and Archer’s ‘Realist Social Theory’. These two key figures were, and have been, heavily involved in developing what has become more popularly known as critical realism.

Critical realism

Critical Realism³ came to the fore during the paradigm wars of the 80s and 90s when contestations began surfacing between proponents of the different schools of philosophical thought, on which would be more practical to inform research practice (Gage 2016; Oakley 1999). The contestations were clarion calls to rethink the scope of what was counted as ‘acceptable’ and ‘scientific’ research practices. Crudely speaking, the Positivist understanding of natural science has historically maintained that the methodology adopted in the study of natural and social sciences are unified in adherence to positivist principles, and that the objects of study in both disciplines can be understood and examined in the same manner (Keat 1971) In North America and the United Kingdom, however, the anti-naturalism shifts initiated by social constructionists, supported a cleavage in the research methods adopted in the social sciences.

This cleavage, however, was not without problems. The essential thrust of the social constructionist viewpoint is that ‘One can only change the narratives by changing the discursive conventions by which they are created and interpreted’ (Harré 2006, 26). Disquiet occurred about how some forms of social constructionism worked to undermine certain longstanding understandings by (1) relativizing foundationalists’ claim to certainty and positivists’ claim to objective truth; (2) making possible the rejection of humanism on grounds that the politics embedded in language challenged the possibility of autonomy and unitary identities; and (3) rejecting the probability of any objectivism in the idea of representation (Alvesson 2002; Parker 1998b). Opponents of social constructionism were concerned that it would theoretically allow for extreme forms of relativism. Proponents, in comparison, were apprehensive about how to make judgments between the multiple perspectives available and where agency might come into play within the perspective (Burr in [Parker 1998a]).

Archer (2020) positioned her work on Realist Social Theory as complementing Bhaskar’s (1998) social ontological work in ‘The Possibility of Naturalism’. While the latter has identified admissible constituents and concepts to address the social world, Realist Social Theory (Archer 1995) proceeds to order and connect these constituents and concepts into a meaningful framework for explaining social change and reproduction (social morphogenesis). Where there has been more agreement, is the possible influence Margaret Archer’s work on Realist Social Theory can have on realist evaluation.

The moderating effect is that Critical Realism is evident in the non-Positivist naturalism position it adopted, and the important distinction it makes between judgmental and epistemic relativism. Instead of advocating the need for a cleavage in the approaches used to inquire into the nature of natural and social objects thereby allowing questions and contestations to be raised about what does or does not count as ‘evidence’, ‘scientific’, and/or ‘rigorous’, Critical Realism maintains that both natural and social objects can be studied scientifically. Notably, Bhaskar (1998) asserted that the naturalism proposed by Critical Realism differs from that implied in the positivist version of naturalism in that,

… the predicates that appear in the explanation of social phenomena will be different from those that appear in natural scientific explanations and the procedures used to establish them will in certain vital respects be different too (being contingent upon, and determined by, the properties of the objects under study); but the principles that govern their production will remain substantially the same … because social objects are irreducible to (and really emergent from) natural objects, and so possess qualitatively different features from them, they cannot be studied in the same way as them, [but] they can still be studied ‘scientifically’. (Bhaskar 1998, 22)

Therefore, several authors have reported that realist evaluation is based on critical realist principles (Costa and Magalhães 2019; Porter and O’Halloran 2012; Smeets et al. 2021; Tennant et al. 2020; Wilson and McCormack 2006). This can be attributed to the fact that some of the tenets reported by Pawson and Tilley to scaffold realist evaluation are drawn from critical realism. Moreso, Pawson and Tilley (1997a) themselves acknowledge this saying that ‘the roots of our perspective can be traced directly back to the influential realist tradition in the philosophy of science, as identified in the writing of Hesse (1974), Lakatos (1970), Bhaskar (1975) and Harre and Secord (1972)’ (55); authors who have been credited for developing scientific realism and critical realism. Although Pawson and Tilley (1997a), in their conceptualizing of realist evaluation, align more with those who tried to develop Realism as an empirical method than those who took the normative turn (de Souza 2014), they drew from principles governing Critical Realist thinking. These principles apply both to scientific realism and critical realism and have been identified by Bhaskar as necessary for a non-Positivist naturalism. He states that the essence of the natural and human scientific method respectively comprises the

… movement at any one level of inquiry from manifest phenomena to the structures that generate them. It shows that experimental and practical activity entails an analysis of causal laws as expressing the tendencies of things, not conjunctions of events (‘epistemology’); that scientific discovery and development entails that scientific inferences must be analogical and retroductive, not simply inductive and/or deductive (‘logic’); and that the process of knowledge-production necessitates a conceptual system based on the notion of powers (‘metaphysics’) … [for the social sciences] things are viewed as individuals possessing powers (and as agents as well as patients). And actions are the realization of their potentialities. Historical things are structured and differentiated (more or less unique) ensembles of tendencies, liabilities and powers; and historical events are their transformations. (Bhaskar 1998, 19)

Little wonder then, why, Porter (2015b) argues that ‘the differences between critical realism and realist evaluation are not as significant as Pawson contends’ (65) in his book entitled the philosophy of science.

Towards a cogent theoretical foundation for realist evaluation

An amalgam often informs scientific inquiries of principles and practice (Pawson and Tilley 1997b). Evangelopoulos (2013) reported that, while scientific realism focuses on the objectivity aspect of science – reality consists of an independent object whose knowledge is the ultimate objective of all scientific activities, critical realism focuses on the social dimension of science, which also captures the scientific activities as the historical product of the human society. Owing to their seemingly different fundamental goals, scientific realism and critical realism offer different but complementary perspectives to inform realist evaluation. Indeed, they share similar understandings relating to the existence of a mind-independent reality, the existence of the unseen, upward and downward causation, stratified reality, emergence, the embrace of multiple methodologies, and the importance of theory in science (Brekke et al. 2019). Consequently, many researchers pilfer various aspects of scientific realism and critical realism to inform their realist evaluation methodology. Porter (2017) confirms that Pawson developed the realist evaluation approach through erudite eclecticism.

While Pawson and Tilley ground realist evaluation in scientific realism, attempts have been made to examine the Critical Realist roots of realist evaluation (Porter 2015c). In agreement with Porter that realist evaluation requires a cogent theoretical foundation, we unpack the ontological and epistemological positions of scientific realism and critical realism regarding their contributions towards realist evaluation methodologies. Such explication will provide a basic understanding of realist evaluation to the non-realist audience and a gentle introduction to novice and early career researchers willing to employ realist evaluation methodologies or undertake a realist evaluation. Consequently, current philosophical discussions around scientific realism and critical realism will not be the focus of this paper. Instead, it will be about the relative merits of the two realist positions in evaluation research.

We focus on capturing those aspects of the two forms of Realism that are generally acceptable in the extant literature and unpack their methodological implications for realist evaluation. We have organized this discussion accordingly. First, we will discuss some central tenets of Realism and how they are accentuated in scientific realism and critical realism in a general manner and then explain how these tenets contribute to retroductive theorizing in realist evaluation. Then, we will conceptualize their contributions toward a cogent epistemological foundation of realist evaluation.

Mind-independent reality

Mind-independent reality means that the world’s existence, nature, and structure are independent of human thought, language, conceptual activity, and perceptual experience (Sankey 2021). It has properties and structures that can be known through empirical inquiry (Brekke et al. 2019). Therefore, mind-independence is a way of saying something is real, objective, and truly exists, not just the product of our imaginations or other cognitive processes (Khalidi 2016). For example, the generative mechanisms of the outcome of a healthy eating intervention on pre-diabetic individuals will remain what they are, irrespective of our perspective of the programme. The way the intervention will play out after its implementation and the generative mechanisms involved are ontologically distinct from the observed outcomes. They do not depend on the researcher or the way the researcher thinks, or tools and methods used – ontological objectivity. To this end, the goal of realist evaluation is to determine the nature of the intervention – how and why the intervention works or not, irrespective of how and why we think they work. Realist evaluation is, therefore, proposed as one practical activity among others, which allows us to take a closer look at the relationship between practice, meaning, concepts, and language and the underlying mechanisms and structures constituting reality vis-à-vis a particular phenomenon or intervention. Although language, and consequently conceptualization, stands out as one of our most essential instruments for scientific research, reality does not depend on them, and they do not determine what is real (Danermark, Ekström, and Karlsson 2019).

Semantic notion

The second principle of scientific realism relates to the relationships between theories and the world – semantic notion. The semantic notion of scientific Realism relates to the idea that reality (although mind-independent) can be effectively captured through scientific theories. Bhaskar (2013, 16) calls these the transitive objects of knowledge ‘the raw materials of science – the artificial objects fashioned into items of knowledge by the science of the day. They include the antecedently established facts and theories, paradigms and models, methods, and inquiry techniques available to a particular scientific school or worker’. Scientific realism does not simply accept that there is a mind-independent world, but they try to capture the truth about the nature of reality. Therefore, scientific theories are used to describe this mind-independent reality. Capturing reality should not be solely achieved through measurement, observations, intentions, information, social norms, or any other epistemically loaded or anthropocentric term. Instead, these anthropocentric measurement approaches should seek to unpack the underlying causal processes of our observations. Bhaskar and Lawson (2013) also demonstrates ‘how the intelligibility of experiments presupposes that reality is constituted not only by experiences and the course of actual events, but also by structures, powers, mechanisms and tendencies – by aspects of reality that underpin, generate or facilitate the actual phenomena that we may (or may not) experience, but are typically out of phase with them’ (Bhaskar and Lawson 2013, 5).

To elicit programme theories, therefore, the realist evaluator embarks on the process of retroduction, a form of retrospective theorizing, which entails moving from the observed outcomes of the intervention to reconstruct the conditions for the outcomes to occur by unearthing the underlying generative mechanisms and structures (Fletcher 2017; Meyer and Lunnay 2013; Mukumbang, Kabongo, and Eastwood 2021). Retroduction as applied in realist evaluation involves exploring the intervention’s effects and then working backward to unearth the generative mechanisms responsible for producing the results (Jagosh 2020). Retroductive theorizing in realism-informed inquiries requires the realist evaluator to remain faithful to the ontological and epistemological underpinnings of realism (Mukumbang 2021), which have been situated in scientific realism (Jagosh 2020) and critical realism (Downward and Mearman 2007).

Scientific Realism posits that our scientific theories are true or false based on the best scientific knowledge. We should consider our scientific theories true – the entities they postulate as real, not merely as ‘useful’ (Chernoff 2007). Theoretical explanations in scientific realism go beyond the observable world captured through anthropocentric approaches and other observational measurements to accommodate knowledge about the unobservable (Allzén 2021). The theoretical discourse of scientific realism thus focuses on both the observable and the unobservable entities. To justify the centrality of observable and unobservable objects to theoretical discourse, Scientific Realism typically invokes inference to the best explanation or theories (Allzén 2021). For these theories to be accurate or referential, the unobservable entities they propose must exist, a condition described as realist correspondence of truth. Realist correspondence of truth submits that a proposition is true if it corresponds to an objectively existing state of affairs, whether or not we believe that the state of affairs exists (Sankey 2021).

Epistemic notion

The epistemic notion of scientific realism suggests that the best and most credible scientific theories represent an approximation of reality (truth). These theories could emanate from our common sense, which our interactions with the physical world can inform through our senses (Sankey 2001). Therefore, the reality about everyday objects and our epistemic access to such objects provide the starting point for common sense or abductive realist components of the argument for scientific realism (Sankey 2001). Scientific realism takes common sense or abduction as a starting point toward obtaining more robust scientific theories through epistemic access using our senses. Nevertheless, these theories should only be taken at face value as they are only truth approximating, considering that they can be true or false – fallible (Brekke et al. 2019). In Chapter 3 of A realist Theory of Science, Bhaskar and Lawson explain that explanatory science,

seeks to account for … an experimentally produced event pattern in terms of a (set) of mechanism(s) most directly responsible. Producing this explanation will involve drawing upon existing cognitive material and operating under the control of something like a logic of analogy and metaphor, to construct a theory of mechanism that, if it were to work in the postulated way, could account for the phenomenon in question. The reality of the mechanism so retroduced is subsequently subjected to empirical scrutiny, and the empirical adequacy of the hypothesis maintained compared to that of competing explanations. (Bhaskar and Lawson 2013, 5)

The epistemic notion of scientific realism also addresses the difference between a theory being the most explanatory and predictive potential and its being (approximately) true. Following the notion of inference to the best explanation, Allzén (2021) proposes that ‘if a hypothesis H is ranked to be the best explanation among its competitors concerning the relevant background knowledge, we should infer that it is (approximately) true’ (Allzén 2021, 154). This opens to adopting abductive thinking because it is based on inference for the best explanation (Sankey 2001). To this end, realist evaluators rely on abductive thinking – interpreting and re-contextualizing observed actions and events to suggest the ‘best explanation’ of how a programme works or not (Downward and Mearman 2007). Therefore, scientific realism makes a modest claim that there is an independent and largely unobservable-by-means-of-the-senses world, which science tries to uncover using theories (Psillos 2000).

Following this initial abductive theorizing, retroductive thinking is adopted to investigate the hypothesized generative mechanisms to confirm or challenge the initial abductively generated programme theory. These retroductively obtained theories may be close to or considered an approximation of how and why the programme works (or not). Programme theories may not embody mechanisms in the truest ontological sense. Nevertheless, subsequent programme theory development and testing cycles may increase our knowledge of these generative mechanisms.

Levels of reality

Bhaskar (1975), in his conceptualization of science as a social activity, described Realism as having three levels: Real, Actual, and Empirical. Although scientific Realism does not explicitly classify reality as constituting of these three levels, it suggests that reality comprises of both observable and unobservable entities. Critical Realism, conversely, explicitlyillustrates where these observable and unobservable entities can be found. Bhaskar (1975) used the following analogy (Table 1) to describe observable and unobservable entitles, explicating our access to information on these levels.

Table 1. Three levels of reality (Bhaskar 1975).

	The domain of the Real	The domain of the Actual	The domain of the Empirical
Mechanism	√
Events	√	√
Experiences	√	√	√

According to Bhaskar’s (1975) classification, the Domain of the ‘Real’ contains both observable/measurable entities (experiences) and unobservable entities (causal entities such as mechanisms). While these causal entities cannot be observed directly, their effects can be captured and described using empirical methods, and their existence is theorized. Research activities are purported to capture some events easily, and these events reported as experiences are placed at the ‘Empirical’ level. While some of these events can be captured using our observation methods and empirical activities, not all these events get experienced so they go uncaptured even with our best observation methods (Brekke et al. 2019). Whether captured or not, events are generally considered to be situated at the level of the ‘Actual’ as they represent things that ‘actually’ happened. Therefore, real structures and causal mechanisms exist independently of actual patterns of events necessitating the need to undertake scientific activities to make sense of their operations.

When a social intervention such as an intervention to reduce gender-based violence is implemented, various social mechanisms (such as willingness to change, love, and emotional intelligence) and intervention mechanisms (such as understanding relationship dynamics) are triggered based on the interactions of different agents (different genders) acting within social structures such as cultural values and family dynamics. These structures and mechanisms are found in the realm of the Real, and the goal of the realist evaluator is to unearth them. When these interactions occur, events and nonevents occur. Improved understanding and cooperation among couples can lead to demi-regularities or observations such as reduced incidence of gender-based violence and other unnoticed events that can be captured in the ‘Actual’. The actors’ experiences after implementing the intervention can be captured empirically using various research methods such as interviews and surveys to explain the (un)changed pattern of behaviour, which occurs at the ‘Empirical’. The role of the realist researcher is to use the information captured at the empirical to excavate those mechanisms and context conditions that combine to explain how and why the interventions work or not.

Transitive and intransitive entities

The social activity of unravelling the underlying generative mechanisms and real structures generating the observed outcome(s) of intervention brings to the fore the notions of transitive and intransitive entities in critical realism. The transitive entities (entities amenable to alteration by human action) of sciences represent the outcomes of the activities of science through empirical activities. For example, the knowledge, models, and theories derived from research activities to understand how and why an intervention works constitute transitive objects. This means that as time goes on, our understanding of how and why an intervention works or not might change as better methods of capturing experiences and observations become available. The implication is that those programme theories elicited from a realist evaluation cycle are always provisional and partial. Still, they can be improved as more evidence becomes available (MESH 2017).

Intransitive entities (entities that human actions cannot change), on the contrary, represent laws and properties of the world independent of our knowledge of them and efforts to understand them. For instance, culture and beliefs exist independent of our knowledge of them. Still, they are just as ‘real’ as physical processes, and they have tangible effects even if they are not directly observable and need to be inferred from evidence (MESH 2017). This distinction between what critical Realism calls the transitive (the changing knowledge of things) and the intransitive (the relatively unchanging things which we attempt to know) has implications for realist evaluation – our efforts to understand how and why an intervention works or not can be finite. The programme theories obtained from a realist evaluation inquiry can inform another realist evaluation cycle to capture the existing mechanisms and structures irrespective of our knowledge of their existence. The notions of transitive and intransitive objects of Critical Realism highlight the intelligibility of scientific understanding, particularly the fallibility and transformation of human knowledge. For example, while our understanding of how and why an intervention works or not can change over time as more evidence becomes available through empirical investigations, the underlying structures and mechanisms that interact to produce the observed outcome most often remain unchanged – more become unveiled as time goes on or the nature of the intervention changes with time.

Emergence – the whole is greater than the sum of its parts

Another central element of critical Realism is emergence. Emergence occurs when a whole possesses one or more emergent properties (Elder-Vass 2010). Unobservable entities are structured, and these structures are nested within other structures, for example, family dynamics nested within cultural or religious practices. These entities are usually unobservable, and their operations depend on situational conditions created by complex interactions with other things (Brönnimann 2021). The occurrence of novel qualities from the interactions of these existing entities is described as emergence. Some of the causal properties of the existing entities emerge from structured relations between their constituent entities (Sorrell 2018). The novel qualities that emerge are not always the sum of the parts of the interacting entities, and they can also not be reduced to the entities from which they arise (Brekke et al. 2019). Therefore, ‘An emergent property is not possessed by any of the parts individually, and that would not be possessed by the full set of parts in the absence of a structuring set of relations between them’ (Elder-Vass 2010). Social interventions have emergent properties, that is, possessing causal entities, which may result in a particular outcome under certain conditions. Therefore, the realist evaluator’s role is to unveil the causal mechanisms and those conditions under which the causal entities of the intervention are triggered to produce the outcomes (intended or unintended).

Emergence is hierarchical, occurring at different levels of reality: molecular, chemical, organisms, groups, and society. Entities at a lower level (molecular) can create conditions for unfolding new entities at a higher level. Sayer (2000) illustrates this idea by highlighting that ‘social phenomena are emergent from biological phenomena, which are emergent from chemical and physical strata’ (13). While other entities may emerge from lower strata, they are not reducible to those strata and, therefore, still need to be researched in the strata in which they operate to explain their operation (Mukumbang 2021).

Open and closed systems

The notions of open and closed systems elucidate the concept of emergence. From an empirical point of view, critical realists characterize the social world as an open system (Fleetwood 2017). Critical Realism suggests that the world is an open system with a constellation of structures, mechanisms, and other entities responsible for the observable patterns of events. Bhaskar (1989) explained that human capacity and agency are omnipresent and unlimited in their ability to alter their environment. The agents’ interactions with social structures give rise to observable and unobservable patterns occurring as events (demi-regularities) in the ‘Actual’. The demi-regularities are usually consistent with regular habits and behaviours, but these events or behavioural patterns are never deterministic or stochastic event regularities (Fleetwood 2017). This is because, under countervailing conditions, other unintended or alternative behaviours could be observed – contrastive (Lawson 2001). Therefore, in critical Realism, an open system is about (ir)regularities in the flux of events and states, not systems (Fleetwood 2017).

Considering reality as an open system in critical Realism has implications for systems thinking in realist evaluation (de Souza 2022). Pawson et al. (2005), while drawing from Archer’s realist social theory, acknowledged that programmes are parts of open systems or embedded in multiple social systems. As such, social interventions are considered as ‘complex systems thrust amidst complex systems’ (Pawson 2006, 25). The adoption of systems thinking approaches in realist evaluation, thus, captures the notion of complexity by foregrounding relationships in wholes rather than by isolating parts (de Souza 2022), which is the premise of irregularities and demi-regularities.

Generative view of causation: agent-structure relations

Within open systems are the activities of agents (thoughts and actions taken by people) interacting with structures (organized social institutions and patterns of institutionalized relationships). The concept of social morphogenesis proposed by Archer (1995, 2020) deals with the relationship between structures and agents in critical social realist terms. According to Archer (1995), agents in the past organized and constructed the various structures – social, economic, cultural, and political systems – that make up our society. These social structures are causally effective, having generative powers that differ from agents’ causal powers (Elder-Vass 2008). The generative powers that the agents possess interact with those of the social structures to determine the agents’ actions, thus the notions of emergence and open systems (Bhaskar 2009). For example, multiple stakeholders are involved in any given intervention, each with their interpretation of the structures governing the interventions and ideas held within the intervention modalities. In Bhaskar’s (2009) agential movement, the agent’s actions contribute to reproducing and transforming the structure(s) concerned, consequently leading to social change.

Elder-Vass (2010) clarified that the causal powers of structures (and interventions in the case of realist evaluation) are only activated when agents act upon them. Therefore, the product of the causal influence of structures is highly contingent on the dispositional properties of the agents. Programme mechanisms are a sub-set of social mechanisms (structures) in that agents have consciously created them to alter the interpretations and actions of other agents (Porter 2015b). As such, the characteristics or modalities of interventions are only part of the relevant entities. The social processes involved in its implementation must be understood to describe how the outcomes are achieved (Porter and O’Halloran 2012). The implication is that the mechanisms introduced by an intervention are not the only ones in operation. To explain observed intervention outcomes, it is also necessary to consider the context in which the intervention is introduced (Porter 2015a). To this end, Pawson and Tilley (1997a) proposed the context + mechanism = outcome as a heuristic tool to formulate realist programme theories. The context here relates to ‘ … the relational and dynamic features that shape the mechanisms through which the intervention works, assuming that context operates in a dynamic, emergent way over time at multiple different levels of the social system’ (Greenhalgh and Manzano 2021, 583).

The methodological implication of the agent-structure relationship for realist evaluation is that critical realism accepts the categorical distinction between social structure and agency and their corresponding generative powers and potentials (Porter 2015b). Bhaskar explicitly distinguished the roles of agency and structure while establishing their relationship. Similarly, Archer found the relationship between agents and structures by exploring their roles in changing social reality. Bhaskar suggested that ‘agents have intentions and plans, which determine their actions while structures govern the reproduction and transformation of social activities’ (Bhaskar, 1989, 79). This suggests that mechanisms can be found at the level of the agent, structures, and the interaction between the two. Conversely, Pawson regards social mechanisms as a combination of agency and structure, focused on people’s reasoning (cognitive or emotional) and choices (Porter 2015b). According to Pawson and Tilley (1997), programmes alone do not cause changes but rather how agents respond to the resources, ideas, and practices that those programmes introduce (programme ‘mechanisms’) to create programme outcomes. Critical realism recognizes that structural and cultural situational mechanisms create enabling and restricting conditions. Using the critical realist lens to programme evaluation also suggests that social interventions are stimulative for specific actions and provide conditions that may be less connected with the agency’s intentions (Brönnimann 2021). Other outcomes are observed when social programmes or interventions are introduced in the context of these structural and cultural situational mechanisms.

Tenets of critical realism and scientific realism, and retroductive theorizing

Retroductive theorizing identifies and conceptualizes the underlying causal entities, structures, and mechanisms to explain social activities (Jagosh 2020; Mukumbang 2021). As applied in realist evaluation, retroductive theorizing is a retrospective approach to unearthing the generative (social) mechanisms and structures responsible for the outcomes observed when an intervention, programme, or policy is implemented in a particular setting.

The point here is to illustrate how the different epistemological principles of critical realism and scientific realism discussed in the section above contribute toward retroductive theorizing in realist evaluation. Figure 1 illustrates the visual contributions of the epistemological principles of critical realism and scientific realism, to realist evaluation, with retroductive theorizing serving the purpose of ‘re-vindicating ontology by taking stock of and transcending the social construction of facts and evidence’ (Jagosh 2020).

Figure 1. Contributions of the different realism tenets to retroductive theorizing and the nature of the generative theory.

Note: The blue arrows are tenets related to the contributions toward theory formulation. The black arrows are related to tenets describing the nature of the formulated theories.

In addition to illustrating the ontological and epistemological contributions of both forms of Realism to retroductive theorizing, we have added in Table 2 the methodological implications of these tenets for realist evaluation. Critical realist tenets predominantly contribute towards the social activities of eliciting the theories, while the scientific realist principles relate to the validity of the theories developed. This conception by Brönnimann (2021) illustrates the movement from Critical realist principles to Scientific realist principles within our conceptualization (Figure 1). (1) ‘Explication of Events’ by collecting information about observed empirical outcomes of the intervention – the principle of stratification of reality. (2) Based on the observed outcomes, identify structures, mechanisms, and their relevant context – open system and emergence principles. (3) Based on these conceptualizations, retroduce multiple viable mechanisms responsible for the observations – the generative view (principle of agent-agent – agent structure relationships). (4) Engage in empirical corroboration, which includes judgmental rationality (the critical realist concept relating to the possibility of making claims to knowledge and truth) to validate the hypothesized mechanisms – transitive and intransitive dimensions of reality leading to understanding the epistemic notion of Scientific Realism. Throughout this process, the application of triangulation and multi-methods are encouraged to accumulate richer data – the semantic notion of scientific realism. Validating or invalidating these hypothesized mechanisms against the original data and case reality aligns, thus supporting the thesis of the correspondence between theory and social reality and truth approximation represented by the epistemic notion of Scientific Realism. A detailed contribution of the Critical Realist and Scientific principle is described in Table 2.

Table 2. Realist tenet, ontological/epistemological basis, and implications for realist evaluation.

Key tenets	The ontological and epistemological basis	Implication for realist evaluation
Stratified ontology	Three levels of reality, the real, actual, and empirical, illustrate how observable and unobservable entities exist in reality.	Logical and analytical support for the existence of relevant mechanisms linking the structure to events. Actual events can be investigated through directed probing questions. Describe any observed events where people or groups acted to change the existing behaviour or adopt a new behaviour.
Open-systems perspective	The world is an open system with a constellation of structures, mechanisms, and other entities responsible for the observable patterns of events.	Identification of the relationships among the programme context, mechanisms, and outcomes Explication of changes to the programme or intervention outcomes.
Emergence	The occurrence of novel qualities from the interactions of existing entities. The effects are contextual – dependent on other structures with generative powers. Mechanisms at a lower level can create conditions for unfolding mechanisms at a higher level.	Identify plausible candidate causal mechanisms and how they interact for the programme outcomes to be observed. Description of the programme outcomes based on the interactions of the emergent properties.
Agent-agent – agent-structure relations	Critical Realism accepts the existence of independent structures that influence actors’ actions in a particular setting while acknowledging the role of these actors’ subjective knowledge and reasoning.	Questions need to investigate the mode of interactive mechanism people display. Identifying programmatic structures, relevant context, and generative mechanisms. What pre-existing social structures and culture existed to affect the implementation and adoption of the intervention or programme, e.g. organizational social structures The formulation of context-mechanism-outcome configurations.
Transitive and intransitive dimensions of reality	The transitive speaks to the changing knowledge of things (how things are understood), and the intransitive relates to the relatively unchanging things we attempt to know – structures and mechanisms.	Employ multiple approaches to support causal analysis based on various data types, sources, analytical methods, investigators, and theories.
Independent Reality	The world’s existence, nature, and structure are independent of human thought, language, conceptual activity, and perceptual experience	Multiple theoretical perspectives Variety of data sources and types
Semantic notion	Scientific theories offer a credible way of capturing mind-independent reality	Selection of context-mechanism-outcome configurations that offers the best explanation. Application of theoretical saturation to guide theory formulation – retrodiction.^a Applying judgment rationality to validate an explanatory theory.
Epistemic notion	The best and most credible scientific theories are well-confirmed and approximately true.	Multiple analytical and methodological techniques can be applied in a realist evaluation inquiry Methodological pluralism.

^a In retroduction we identify individual causal powers and the mechanisms that produce them, and in retrodiction we investigate what mix of causal powers interacted in what way to produce any particular event (Elder-Vass 2015, 81).

Discussion

At the epistemological level, Realism seeks to understand the extent to which scientific theories access and describe the world as it is. Following this general raison d’être of Realism, its variants have some common basic tenets. Based on our exploration of the principles guiding scientific realism and critical realism, we find that they are underpinned by the notions of ontological depth and mechanism-based theorizing (Mukumbang, Kabongo, and Eastwood 2021). As such, they share identical tenets relating to the existence of a mind-independent reality, the unseen, upward and downward generative causation, stratified reality, emergence, the embrace of multiple methodologies, and the importance of theory in science (Brekke et al. 2019). With realist evaluation being supposedly informed by scientific realism (Pawson and Tilley 1997b), it stands to logical understanding that it would also be considered that Critical realist principles also underpin realist evaluation. Nevertheless, the different forms of Realism provide deeper explication and focus on different epistemological and ontological stances underpinning Realist evaluation.

Many authors who have explored the relationship between scientific realism and critical realism contend that they are closely related (Chernoff 2007). Whether one agrees that scientific realism and critical realism are closely related, many authors recognize their contributions to realist evaluation. For instance, Jagosh (2020) illustrated that Bhaskar’s conceptualization informs the notion of ontological depth adopted in realist evaluation through the three ontological levels of reality for research activities, the Real, Actual, and Empirical, and the need for repeated scientific research efforts to capture the elusive generative mechanisms to approximate actual mechanisms. (Currie, Chiarella, and Buckley 2015) also suggested that the notion of ‘generative mechanism’, a fundamental tenet of realist evaluation, also emanates from Bhaskar’s explication of causality.

Our illustrations in Figure 1 and Table 1 show that the Critical realist tenets and their implications for realist evaluation lend them to contribute to retroductive theorizing predominantly. At the same time, scientific realism focuses on the nature and role of the developing theory. The tenets of critical realism encourage retroductive theorizing, which requires empirical engagement and corroboration to conceptualize and validate generative mechanisms and theories. The Critical realist tenets, especially those related to generative causation, open systems, agent-agent – agent-structure relations, and emergence, underpin the different research approaches, methods, sources of data, and analytic inferential thinking. The transitive nature of reality offers insights into the nature of mechanisms and structures.

Scientific realism, contrariwise, predominantly focuses on describing the nature of the realist theories and their role in ‘accurately’ describing reality. In contrast, the intransitive nature illuminates the nature of the developing or emerging theory (no final truth, and theories can change as time evolves). Table 1 provides important considerations while conducting a realist evaluation to ensure that the developing generative theory substantially describes how and why the programme works, for whom and under what conditions. This way, the realist evaluator should consider using multiple theoretical perspectives, theoretical saturation, judgmental rationality, and retrodiction. Therefore, scientific realism’s semantic notion focuses more on the objective nature of the programme theories elicited during the evaluation exercise. These theories are consequently examined at three levels: (i) an ontological sense of objectivity relating to the mind-independence of the natural world; (ii) a semantic form of objectivity relating to the nature of truth; and (iii) an epistemic notion of objectivity relating to methodological practices and justification underpinning their adoption.

Pawson and Tilley (1997a) laid the foundations for realist evaluation with the focus, among other things, on explaining ‘how and why for whom and under what conditions programmes and policies work’ as a contribution to the varied approaches and methods used in evaluation sciences. They suggested that programmes work or not (outcomes) if they introduce ideas, constraints, and opportunities (mechanisms) to relevant actors in the appropriate relational, social and cultural conditions (contexts). Therefore, mechanisms acting in contexts is the axiomatic base upon which all realist explanation builds. Consequently, Pawson and Tilley (1997a) conceptualized context-mechanism-outcome (CMO) configuration to express causal explanations in realist evaluation inquiries. While the CMO configuration, for obvious reasons, is the most adopted in realist evaluation, CMO configurations have been criticized as limited in their explanatory potential. It has been suggested that the ‘mechanism’ represented in the CMO configurations as a combination of reasoning and resources implies that the mechanisms can be components of structure and agency. Consequently, determining the actual source of the mechanism becomes challenging and can erroneously be attributed entirely to the impact of the intervention (Hinds and Dickson 2021). As such, other explanatory factors have been added in other realist evaluation inquiries; it is recommended that any configuration type obtained from the addition of these factors must adhere to the rule of generative causation (De Weger et al. 2020).

Following the critical realist generative view of causality, Porter (2015a) suggested that the realist evaluation configuration should include structural and agent mechanisms. (Mukumbang et al. 2017, 2018) support this suggestion stating that interventions cannot work unless taken up by the relevant actors, thus recommending the addition of ‘intervention’ and ‘actors’ to the original CMO configuration. To this end, Porter (2015a) proposes ‘to identify the mechanisms embedded in an intervention and its social context, and those designed to uncover the experiences, interpretations, and responses of the actors involved’ (244). Brönnimann (2021) also supports capturing the social mechanisms and actors’ perspectives embedded in the intervention. To this end, Mukumbang et al. (2017, 2018) proposed the intervention-context-actor-mechanism-outcome (ICAMO) configuration as a robust heuristic tool for exploring realist programme theories. Porter (2015a) proposed a more comprehensive formula for unpacking realist explanatory programme theories: Contextual Mechanisms + Programme Mechanisms + Agency = Outcome (CM + PM + A = O). His formula proposes that the evaluative process should consider the generation and testing of hypotheses about the mechanisms embedded in the extant social context (CM), the programme mechanisms (PM), and an examination of how agents interpret and respond to these mechanisms (A) leading to the observed outcome (O). In adopting this approach, Bhaskar warns that:

It will not generally be possible to specify how a mechanism operates independently of its context. Hence, we must not only relate mechanisms to explanatory or grounding structures, as in the theoretical natural sciences, but also to context or field of operation. This means that in the social field, in principle, we always need to think of a context-mechanism couple, C + M, and thus of the trio of context, mechanism, outcome (CMO), or more fully, the quartet composed of context, mechanism, structure, and outcome (CMSO). (Bhaskar 2016, 80)

Realist programme theories embody the knowledge of what works, for whom, and under what circumstances. Nevertheless, because mechanisms are elusive to capture empirically, our scientific efforts use proxy measures that approximate mechanisms existing in the realm of the ‘Real’. Pawson and Tilley (1997a) believe that knowledge is cumulative, and we can generate knowledge through successive attempts to intervene in programmes (practical knowledge). Added to this practical knowledge is the one obtained through research, providing policymakers with different ‘families’ of theories unveiled through different context-mechanism-outcome configurations (Costa and Magalhães 2019). This aligns with scientific realism’s assertion that evidence obtained using our best available research methods and tools should be considered valuable to describe the phenomenon. In other words, our knowledge of the objective world is conceptually mediated: facts (representing the objective nature of the world) are theory-dependent but not theory-determined (Danermark, Ekström, and Karlsson 2019).

Conclusion

The ontological foundation of realist evaluation remains confusing among many novice realist researchers. This paper aims to clarify the nature of the principles underpinning realist evaluation to improve the application of realist evaluation in practice. In this paper, we illustrated that both scientific realism and critical realism make different but substantial contributions regarding realist evaluation’s ontological and epistemological foundations. We illustrated that while Critical Realism and Scientific Realism are informed by Realism and share common tenets, they provide slightly different but complementary tenets that underpin realist evaluation.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article was originally published with errors, which have now been corrected in the online version. Please see Correction (http://dx.doi.org/10.1080/14767430.2023.2232661).

Additional information

Funding

This work was supported by the Sydney Local Health District , Sydney, New South Wales, Australia, and the Australian National Health and Medical Research Council Grant #1 [grant number APP1198477].

Notes on contributors

Ferdinand C. Mukumbang

Ferdinand C. Mukumbang is an Assistant Professor of Implementation sciences at the Global Health of Woman Adolescents and Children (Global WACh) research unit, Department of Global. He has a broad background in Public Health, with specific training and expertise in health systems and policy research and particular research methodology expertise in realist-informed research methodologies.

Denise E. De Souza

Denise E. De Souza is a postdoctoral fellow at Torrens University Australia. Her interests include conducting reviews and publishing on policy and program evaluations in Education and Public Health.

John G. Eastwood

John G. Eastwood is Clinical Director – Localities, Department of Prime Minister and Cabinet, New Zealand; Acting Chief Medical Officer and Public Health Physician, Southern District Health Board, New Zealand; Executive Clinical Advisor Clinical Services Integration and Population Health, Sydney Local Health District, New South Wales, Australia; Director Early Years Research Group, Ingham Institute of Applied Medical Research, UNSW, Australia; Director National Health and Medical Research Council Centre of Research Excellence for Integrated Health and Social Care; Co-Chair Sydney Institute for Women, Children and their Families; and Director Healthy Homes and Neighborhoods Integrated Care Initiative, Sydney, Australia. He is also an Adjunct Professor, School of Population Health, UNSW, Sydney, Australia; Clinical Professor University of Sydney, Australia; Honorary Clinical Professor Department of Preventive and Social Medicine, University of Otago, Dunedin, New Zealand, and Visiting Academic Sydney Children's Hospital Network.

Notes

1 We have used intervention in this manuscript to represent policies, programme, and interventions, which are designed to change the behaviour of a targeted group of people.

2 The term ‘scientific realism’ in small caps represents those within realist evaluation who tend to view their evaluative work as being rather exclusively associated with Bhaskar's and other realist work done in the sciences and ‘critical realism’ to those associating their work more with Bhaskar's and Archer's work in the human sciences, respectively.

3 For purposes of clarification, we use the term ‘Critical Realism’ here to refer to the philosophical movement that has come to be associated with the works of Roy Bhaskar.