Qualitative Comparative Analysis and the Study of Policy Processes

Abstract

Given the increasing popularity of Qualitative Comparative Analysis (QCA) for the study of public policy and policy processes, this article offers a review of two key issues: multiple configurational causality and temporality. On the one hand, the study of multiple configurational causal relations allows researchers to deal with the extremely complex set of elements that interact in policy processes. However, this task poses the challenge of balancing deductive and inductive logics in a research design. On the other hand, policy process theories often involve temporal arguments, but QCA does not easily deal with dynamic elements. This article discusses these challenges and proposes several ways to address them. It thereby illustrates the advantages and limitations of QCA for students of policy processes.

Keywords:

• Qualitative Comparative Analysis
• public policy
• policy process
• multiple configurational causality
• temporality

Introduction

Policy processes involve a large number of individual and collective actors that cooperate and struggle in various ways. These processes unfold over time within an environment shaped by socioeconomic conditions, institutions and culture at different levels of government, and their development is influenced by both predictable and unanticipated events such as elections, scientific discoveries and chronic or acute crises and disasters (Sabatier 2007b; Schlager and Weible 2013). In short, policy processes and their outcomes in the form of public policies are affected by a highly complex set of factors that interact and evolve over time (Sabatier 2007b).

The comparative analysis of policy processes can be an appropriate strategy to shed light on the complex configurations of factors that influence policy processes. The analytical technique of Qualitative Comparative Analysis (QCA, Ragin 1987) is specifically designed to detect how complex sets of conditions are related to the outcomes of interest. With its focus on a holistic understanding of cases and on the complex configurations of conditions related to an outcome, QCA is especially appropriate for backward-looking research designs (Scharpf 1997), which ask for the factors causing a given phenomenon, such as law-making outputs (Christmann and Danaci 2012), government spending (Vis 2011), coalition formation (Fischer 2015), policy change (Fischer 2014), or the success of policy projects (Verweij et al. 2013). Furthermore, QCA is particularly suitable for researchers who want to systematically compare a small to medium number of observations and obtain in-depth insight into the complexity of cases while retaining capacity for some level of generalization (Rihoux et al. 2011).¹ This number of observations is common in public policy research, whereby researchers typically focus on a small to medium number of policy processes (Engeli et al. 2015). In short, QCA explicitly takes into account causal complexity.

In this paper, we discuss two important challenges related to the study of causal complexity with special attention to policy processes, based on a review of the recent literature on QCA and policy analysis. The first challenge concerns multiple configurational causality. The assessment of the complex interactions among interrelated factors, as needed for the analysis of phenomena such as policy processes, calls for a partially inductive approach. Yet, on the one hand, an inductive use of QCA is not always possible or warranted. On the other hand, a deductive approach would require parsimonious theoretical hypotheses that are neither appropriate for the in-depth study of policy processes, nor do they correspond to the methodological focus on complexity. Thereby, the challenge consists of finding the right mix between a deductive and an inductive application of QCA. The second challenge deals with the issue of temporality. In that regard, researchers are confronted with the difficulty of reconciling the dynamic nature of policy processes, which are moving targets that unfold over time, with the static character of the QCA framework, which usually allows only for synchronic comparisons.

We discuss a number of potential ways of addressing both of these challenges. With respect to multiple configurational causality, we point to the importance of bridging potential gaps between theoretical expectations and the empirical analysis. We then discuss two ways researchers can find a balance between pure inductive and deductive approaches. With regard to the issue of temporality, we discuss five solutions, such as the introduction of sequences in the configurational analysis, the operationalization of non-commutative combinations of conditions, and the comparative assessment of different QCA at several points in time, as well as the limitations of these approaches.

By discussing these two points and related empirical applications, this paper aims to help students of policy processes to explicitly deal with these two important challenges. This is not to say that the two challenges are specific to public processes or to QCA: multiple configurational causality and temporality are critical issues in any social science research design, and our discussion thus might also be of use to researchers applying other methods than QCA. We do not cover the entire discussion on the methodological and ontological background of QCA, which is equally important to the development and the proper use of the method. For example, the more recent methodological literature controversially discusses issues of measurement error or the robustness of QCA results when changing the calibration or thresholds (Skaaning 2011; Hug 2013; Maggetti and Levi-Faur 2013; Fiss et al. 2014; Lucas and Szatrowski 2014, Thiem 2014; Vaisey 2014; Braumoeller 2015; Krogslund et al. 2015; Thiem et al. 2015). Furthermore, it is not the goal of the article to introduce new researchers to QCA, nor to explain its research approach step by step. Readers unfamiliar with the method should refer to Ragin (1987, 2008), Rihoux and Ragin (2009) or Schneider and Wagemann (2010, 2012).

The remainder of the paper is structured as follows. In the next section, we present the issue of multiple configurational causality and the challenge of finding a balance between induction and deduction. The third section tackles the issue of temporality. Both sections first introduce the general challenge in relation to QCA, then discuss it more specifically for the analysis of policy processes, and finally propose some ways to address the issue at stake. In the last section, we present our conclusive remarks and formulate some recommendations to researchers.

The Challenge Related to Multiple Configurational Causality

Policy process research involves a highly complex set of different factors operating at multiple levels that all influence the variables of interest (Scharpf 1997; Sabatier 2007b). For example, the occurrence of policy change can only be fully understood by taking into account external events, cooperation, power and conflict structures among coalitions, the institutional context of a country, and the interactions between these different factors (Fischer 2014). QCA is equally based on the premise that several causal conditions most often interact with each other in complicated ways in order to produce an outcome in the real social world. The method thus seems suitable for the study of the complexity of the policy process. This complexity – and the aspiration to fully understand a policy process and its output – calls for a partially inductive approach where an in-depth knowledge of the cases under study informs the mix of theoretical arguments the researcher might rely on to explain the cases. However, relying on existing theoretical expectations is equally useful. Indeed, QCA scholars emphasize the importance of a dialogue between theory and evidence based on cases throughout the analytical process ‒ i.e. combining both inductive and deductive logics. Finding a balance between induction and deduction for the study of policy processes with QCA is, however, a tricky issue. As mentioned by Collier (2014), the fact that QCA is both highly structured and based on formal logic, but at the same time qualitative and inductive, can be confusing. Below, we first explain how the issue of multiple configurational causality is conceived within the QCA approach. We then discuss the shortcomings of both the purely inductive and purely deductive approaches, before addressing possible solutions

QCA as a Tool to Study Multiple Configurational Causality

The QCA researcher deals with more complex causal relations than those usually assumed by co-variational reasoning² adopted by a large number of variable-oriented social science methods. Multiple configurational causality is an important aspect of complex causality and involves three elements (Ragin 1987, 2000; Braumöller 2003; Hall 2003; George and Bennett 2005; Bennett and Elman 2006; Rihoux and Ragin 2009).³ First, it suggests that a combination of different conditions, rather than a single condition, causes an outcome. Second, and as a logical consequence of the first point, researchers might observe multifinality or conjunctural causation; that is, a condition having different – or even opposite – effects on the outcome depending on the context. Some conditions might be irrelevant when they stand alone, but matter when combined with other conditions. Third, multiple configurational causality points to phenomena of equifinality, which means that different (combinations of) conditions can lead to the same outcome.

QCA aims to compare cases systematically while still preserving a strong focus on individual cases (Ragin 1987; Rihoux and Ragin 2009). The goal of most QCA is thus to explain all cases as comprehensively as possible by taking into account a set of explanatory factors and their complex combinations. On the contrary, variable-oriented methods focus on the net effects of one or more variables over a large number of cases. Such an approach allows researchers to generalize their findings more easily, but usually does not lead to encompassing, complex explanations for a given phenomenon. Basically, the case orientation of QCA fits well with the scope of policy process research, which aims to explain elements of the policy process as well as policy outputs and outcomes that lie at the end of a (hypothetically) complex causal chain (Scharpf 1997).

Multiple Configurational Causality and Inductive Logic

Case-oriented research basically calls for an inductive logic. However, a completely inductive approach has at least three shortcomings, briefly presented hereafter. First, it lacks theoretical guidance about which are relevant factors to be included in the analysis, and about the interesting configurations among these factors (Hug 2013). What is more, the absence of a clear theoretical framework hinders the assessment of the scope of a theory, which would be important for producing broader and more generalizable knowledge about an empirical phenomenon (Lijphart 1971; Ragin 1987). Second, acquiring in-depth case knowledge, arguably a pre-condition for the inductive elaboration of theoretical explanations, is potentially difficult. This is especially true when dealing with more than a handful of cases. Without any guidance with respect to the causal conditions to be selected, comparing more than a couple of cases in a systematic way is difficult, if not impossible. QCA, just as any other comparative method, is vulnerable to the “too few cases, too many variables” problem (Lijphart 1971). This means that some theoretical factors need to be prioritized over others. The literature suggests that such a selection of factors should, among other criteria, depend on the existing literature (Amenta and Poulsen 1994; Berg-Schlosser and Meur 2009; Yamasaki and Rihoux 2009). Third, as Hug (2013) argues, the inductive use of QCA may generate wrong inferences, especially when measurement error is present. The author suggests that potential measurement error of conditions or outcomes is a serious issue, specifically in social science applications. Based on simulations, the author shows that measurement error on a small number of cases can still have an important influence on the results. While one might argue that measurement error can be minimized when the researcher analyzes a relatively low number of cases for which s/he has in-depth case knowledge (George and Bennett 2005), measurement error certainly becomes an important issue as soon as the number of cases increases. Following the argumentation of Hug (2013), generating a theoretical argument on the basis of a small number of cases, which additionally might suffer from measurement error, can be greatly misleading.

Multiple Configurational Causality and Deductive Logic

Contrary to an inductive logic, a deductive approach is based on the idea that the researcher disposes of ready-made theoretical arguments that capture the complexity of policy processes. Yet adopting a purely deductive approach also presents some challenges. While QCA scholars (Schneider and Wagemann 2012) argue that researchers indeed most often have expectations about complex combinations of conditions producing an outcome, strong doubts exist about whether such expectations are easily deduced from existing theories on policy processes. Policy process theories rather materialize in the form of frameworks, and complex theoretical expectations on causal processes within these frameworks have long been underdeveloped (Sabatier 2007a).

For example, the Advocacy Coalition Framework (ACF, Sabatier 1987; Sabatier and Weible 2007) most importantly argues that actors with similar preferences tend to form coalitions in order to influence policy processes. However, until recently, the institutional context that acts as an opportunity structure for coalition formation, as well as the resources of actors, were both neglected as additional factors influencing the ability and willingness of actors to form coalitions (Sabatier and Weible 2007; Gupta 2012; Fischer 2014; Henry et al. 2014). Also, while early policy network theory partly overlooked the role of actors in the network (Dowding 1995), it is by now widely acknowledged that single actors may influence the configuration and outcome of a policy network in a given context (Adam and Kriesi 2007). Wolfe et al. (2013) mention that the interaction between information and institutions is key in understanding the influence of the media in policy agenda setting, and they lament the absence of communication efforts between media and policy studies. Lubell et al. (2010) hypothesize that the ways in which participation in a collaborative institution influences an actor’s behavior or attitudes is conditional on participation in other institutions, as participation in other institutions provides or absorbs resources. Further, Owens and Bressers (2013) focus on interactions between power, motivation and information of state actors and stakeholders in implementation projects.

All these examples point towards potential interactions between conditions at different levels of policy processes. Still, ready-made theoretical arguments that would guide the researcher examining these interactions are generally underdeveloped. And even while theoretical arguments on interactions may be gaining momentum, arguments about equifinality – implying different causal pathways to the same outcome – are certainly not prominent in policy process theories.

Obviously, there are good reasons for this absence of arguments implying multiple configurational causality. Generally speaking, the function of theories and frameworks is – by definition – to reduce the complexity and facilitate our interpretation of social reality (Scharpf 1997; Sabatier 2007b). It thus comes as no surprise that arguments involving multiple configurational causality are not very prominent in existing studies. Furthermore, the possibility of equifinality enables different, alternative theoretical explanations, usually borrowed from different frameworks, for the same phenomenon.

In light of this, a recent review on QCA and public policy research by Rihoux et al. (2011) is able to identify only a few applications that have fully exploited the configurational nature of QCA when formulating hypotheses. Relatedly, Emmenegger et al. (2013) find that only about half of the studies included in their review formulate complex expectations “of varying concreteness”. Other examples include Maggetti (2007), who formulates complex hypotheses for the explanation of the de facto independence of regulatory agencies. He expects conjunctural causation between some of the seven conditions, but does not systematically discuss all possible combinations of conditions. Blatter et al. (2009) explicitly formulate “most-likely” hypotheses with respect to configurations of conditions supposed to lead to given foreign policy strategies of European regions. They based their three alternative hypotheses on different theoretical frameworks, a choice which allows them to bring some order to the potentially high number of conditions. Still, Blatter et al. (2009) also expect to detect configurations they did not hypothesize. Fischer (2012) explicitly formulates hypotheses in terms of multiple configurational causation as well as necessity and sufficiency. In order to reduce the number of hypotheses, he eliminates the combinations of conditions that are unlikely to occur ‒ that is, he makes “easy” theoretical assumptions (for the treatment of logical remainders in QCA, see Rihoux and Ragin 2009). However, it is up for discussion whether the theoretical rationales behind the complex hypotheses are always convincing. Raab et al. (2015) formulate five hypotheses with respect to necessary conditions for the effectiveness of crime prevention networks. Furthermore, they expect that the combination of the five necessary conditions is sufficient for the outcome. While the theoretical reasons for assuming an influence of these five conditions on effectiveness are sound, it remains unclear why they should be necessary, and why the combination of necessary conditions should become automatically sufficient. Thus, as illustrated by these studies, taking an explicitly deductive stance and formulating exact expectations based on multiple configurational causality is difficult.⁴

Possible Ways to Address the Challenge Related to Multiple Configurational Causality

In what follows, we discuss two different, alternative stances with respect to the balance between inductive and deductive approaches. These correspond to two solutions that researchers can adopt to explicitly take into account their prior theoretical expectations.

Explicit Construction of Configurationally Complex Hypotheses

Despite the discussion above, researchers might want to take the deductive logic seriously by explicitly formulating hypotheses involving multiple configurational causality. As explained above, theories of the policy process usually lack this level of precision and do not contain ready-made causally complex hypotheses. Researchers thus need to construct configurationally complex hypotheses based on existing, simpler theoretical arguments. To make such an exercise more systematic and theory-guided, we suggest they look for combinations between theoretical elements along two dimensions.

First, they should take advantage of the different analytical levels that affect policy processes. Arguments about interactions between conditions often involve at least two levels, or more. On the micro-level, actors interact in policy networks, form coalitions with like-minded actors, or direct their messages towards the media in order to publicly defend their preferences. On the meso-level of policy sectors, any given issue belongs to a specific type of policy, displays a certain degree of technical complexity, or might be negotiated under the influence of international organizations. Finally, on the macro-level, national institutions, the type of interest groups system, or the party-political composition of parliament are supposed to play a role (Scharpf 1997; Lijphart 1999). Of course, if one compares cases within a country, for example across policy domains (Fischer 2012, 2014, 2015) or regional entities (Christmann 2010), there is no variation on the macro-level. Still, explicitly taking into account different levels allows introducing some order into the conditions potentially affecting policy processes, and enables thinking about multiple configurational causalities within existing theoretical frameworks.

Second, researchers can try to combine conditions that belong to different theoretical frameworks (Blatter et al. 2009). With such an approach, the goal would be to explicitly test the interdependencies among segments of different existing theories. This can be done by following a logic of equifinality ‒ that is, taking into account different theories as alternative explanations for an outcome. Alternatively, such an approach can explore the conjunctions between segments of different theories. Especially the second option, however, comes with certain risks. Social scientific concepts, for instance those related to institutions or ideas, might have different meanings within different intellectual traditions (Cairney 2013). If this is the case, the researcher will have a hard time interpreting the cases after the comparative part of the QCA, as conditions stem from different theoretical approaches. Some scholars therefore argue that different policy theories are lenses that must not be simply juxtaposed, and that each framework imposes discipline and consistency in the use of language and the development of theories (Zahariadis 2007; Schlager and Weible 2013). This second strategy of explicitly looking for multiple configurational causality between factors stemming from different theories should therefore be used with care.

Co-Variational Hypotheses and Abductive Logic

Given the challenges related to the formulation and testing of hypotheses involving multiple configurational causality, researchers can also adopt a position midway between deductive and inductive approaches. Adopting this second stance, the researcher formulates simple co-variational hypotheses and then inductively looks for more complex causal relations (Fischer 2014). The logic behind this proposition is abductive (Aliseda 2006): the researcher does not adopt a completely inductive approach, but wants to postulate simple hypotheses based on existing theory. While theory tells the researcher a relation should exist, the complexities of this relation are theoretically unspecified. In this case, the method allows researchers to test a hypothesis and then, additionally, to potentially develop and refine the theoretical argument based on the empirical findings. True, in this case, the theoretical hypotheses and the methodological characteristics of QCA are no longer exactly in line. This would be most problematic in reverse situations ‒ that is, when the method is not elaborate enough to test the complex causal relationship formulated in the hypothesis. Yet, in the case discussed here, the method can identify parsimonious relations between two phenomena, as suggested by the hypotheses, but additionally has the potential to disentangle more complex causal relationships.

As an example, let us assume that the researcher postulates that the openness of the policy process has an impact on the level of conflict among political actors, and finds that open venues are only leading to low conflict when the policy process is dealing with highly technical issues (Fischer 2015). The researcher can corroborate the hypothesis, discuss it in more detail, and then possibly propose a further, more detailed hypothesis. This way, an expectation is formulated that a given condition is part of the solution term without specifying exactly what role the condition plays within the solution term and with which other conditions it has to be combined. For example, both Fischer (2014, 2015) and Maggetti (2009) explicitly present separate and simple correlational hypotheses, but mention the possibility of finding more complex causal patterns. Maggetti (2009) justifies this approach with the partly inductive nature of QCA, whereas Fischer (2014) refers to the complexity of the phenomenon and the difficulty of formulating causally complex hypotheses.⁵

The Challenge Related to Temporality

The order of things, sequences, timing, long-term processes and temporal connections among events are crucial to understand political phenomena (Thelen 2000; Pierson and Skocpol 2002). Following Pierson (2000), “placing politics in time – systematically situating particular moments (including the present) in a temporal sequence of events and processes – can greatly enrich our understanding of complex social dynamics”. In comparative research, time is considered as an empirical dimension of variation, together with synchronic variations, along which to gather information and to make observations (Bartolini 1993). In some cases, temporal variations can be considered even more important than spatial variations for explaining differences between countries or regions, or differences among individuals of a target sample, groups or populations, especially for explaining phenomena such as political attitudes and behavior.

Temporality and QCA

Studying temporality through QCA presents a challenge that is different from that of multiple configurational causality discussed above – that challenge being of a more conceptual nature. Indeed, there is a methodological limitation inherent in incorporating the time dimension in a QCA-based analytical framework. Consequently, dealing with temporality is an enduring concern for QCA analysts. As a matter of fact, QCA is an analytical technique for cross-case comparisons that lacks a direct way to treat the time dimension. Truth tables, which are the cornerstone of QCA’s “analytical moment”, represent atemporal configurations of conditions. Correspondingly, QCA solution formulas are commutative. This means that the order by which conditions are connected through logical AND or OR does not matter: “A or B” equates to “B or A”, and “A and B” equates to “B and A”. As Schneider and Wagemann (2012) perceptively note, this insensitivity to the time dimension is particularly disturbing for a technique called “qualitative comparative analysis”, as a meticulous treatment of the time dimension is one of the trademarks of qualitative research.

Temporality and Public Policy

The problem is even more pressing for the study of public policies, whereby it is often necessary to make sense of the diachronic nature of policy processes. This is the case more so than for comparative politics, for example, where cross-sectional analyses are commonplace. Indeed, many important theories in public policy invoke the notion of time. To begin with, the time dimension is fundamental in policy cycles theory, which provides a framework to break down policy processes in a number of consecutive stages, such as agenda setting, policy formulation, decision making, policy implementation and policy evaluation (Lasswell 1951; Howlett and Ramesh 2003). Multiple streams theory incorporates the time dimension in a different way (Kingdon 1995; Zahariadis 2003). On the one hand, temporality is part of the explanandum, as this theory aims to explain agenda change ‒ that is, how and why some policy issues appear on or disappear from the governmental agenda. On the other hand, while this model rejects sequential and incremental models of policy making, timing is nonetheless a crucial part of the explanation as well, because momentum for change emerges when three “streams” – the policy problem, the solutions developed by policy communities and macro-political factors – converge within a “window of opportunity”. Furthermore, punctuated equilibrium theory as applied in political science follows a similar goal ‒ that is, to explain the dynamics of change in public policies (Baumgartner and Jones 1991; Jones and Baumgartner 2005). The core idea is that public policies, instead of evolving gradually and incrementally, are characterized by long periods of relative stability, punctuated by sudden and transformative episodes of intense change. Policy diffusion studies focus on interdependence as a driver of policy change, and, more precisely, on how adoption of policies, institutions, practices, norms or ideas in one jurisdiction affects the probability of an analogous adoption in another jurisdiction (Simmons et al. 2008; Gilardi 2012). In this case, time is the dimension along which diffusion processes unfold and become observable. Finally, scholars studying the path-dependency of policies look at critical junctures and long-term processes instead of paying attention only to slices of time or short-term phenomena (Pierson 2000). The underlying assumption is that the cost of switching from one policy alternative to another is growing over time. Specifically, adherents of this approach maintain not only that the probability of further steps along the same path increases with each move down that path, but also that early events are more important than later ones, and that temporal ordering could be causally relevant.

The Dangers of Not Taking Time into Account

Overlooking the time dimension may be detrimental to the empirical analysis. To begin with, the fact that an event or instance occurs at a certain point in time may be crucial to explaining a policy outcome. For instance, the inclusion of target groups in the early stages of policy processes ‒ that is, when drafting a new bill ‒ can lead to a different policy outcome than their inclusion at later stages, when implementing the policy program (Sabatier and Mazmanian 1979; Matland 1995; Knoepfel et al. 2007). In the former case, one would expect a positive effect on implementation, while in the latter scenario more resistance is expected. Second, when timing is part of the explanandum, such as in policy agenda theories, it is important to explain why some of the new policy issues become central concerns for the government at specific points in time, while others do not. Therefore, the time dimension is needed to make sense of positive outcomes and exclude irrelevant cases. Third, policy diffusion and path-dependent processes should be examined from a diachronic perspective. Indeed, when looking at snapshots only, the researcher has to make suppositions about the shape of the diffusion slope or about the trajectory of policy evolution, a step that entails the risk of optical illusions. For instance, a policy could be adopted by a handful of countries at t1 and then become widespread at t2, while another policy could be adopted by more countries at t1 but remain steady at t2.

Possible Ways to Address the Challenge of Temporality

The simplest solution to the challenge of temporality would be to consider QCA as a first analytical step and then implement ad hoc qualitative (Blatter and Haverland 2012; Schneider and Rohlfing 2013) or quantitative techniques (Lieberman 2005) to deal with the time dimension. However, the researcher may want to analyze the time dimension directly with QCA in order to apply this analytical framework and its tools directly to time-related conditions and to create a single, comprehensive model. This section discusses five solutions to this challenge: the use of time-related conditions; the inclusion of non-commutative sequences of conditions in the analysis of set relations; the aggregate operationalization of procedural variables; the temporal use of two-step QCA; and the comparison of separate analyses for each point in time. The advantages and limitations of each solution will be discussed in order to identify the scope for their applicability.

Time-Related Conditions

To begin with, it may be useful to add one condition (or more) to the analysis that is explicitly constructed to identify time-related factors. Examples are the degree of membership: in the set of “old” units of analysis; in the set of events that occurred after or before a certain date; in the set of phenomena that were very quick; or in the set of phenomena that lasted very long. The inclusion of temporality in crisp or fuzzy sets is also possible by introducing some quality of time into existing conditions. For example, we can add a condition to assess the membership of different countries in the set of “quick economic growth after democratization”. The advantages of this procedure lie in its simplicity and wide applicability. In empirical research, this solution is frequently applied, for instance in Ragin’s analysis of variations in pension systems (Ragin 1994), where a high score in a causal condition indicates a “young social security program”. However, temporality is operationalized only implicitly. Therefore, it is recommended especially when studying questions associated with time-related factors that are not necessarily diachronic in and of themselves, such as the timing of policies, which is examined in the case of the multiple streams theory. What is more, the researcher should bear in mind that the inclusion of additional conditions exacerbates the problem of limited diversity.

Non-Commutative Sequences of Conditions

Temporal QCA (TQCA) offers a way to make sense of temporality through the analysis of sequences, by introducing the idea of “non-commutativity” of historical configurations into standard QCA (Caren and Panofsky 2005). The traditional QCA approach, while highlighting the “multiple and conjunctural” nature of causation (see above), is unfit to capture the sequential unfolding of events producing historical paths of causation. TQCA involves the possibility of sequencing the causal conditions related to each case. To begin with, TQCA requires the researcher to know the sequence in which events unfold: the researcher must specify which factors “always come first”, “always come last”, “always come in a specific order” and whether the conditions could come in any order. This procedure will greatly expand the number of possible configurations, and therefore reinforces the problem of “limited diversity” (Ragin 2008). However, it is possible to place theoretical restrictions to limit the number of configurations and reduce the set to a manageable size, and find the most parsimonious solution to the QCA expressions representing the causal process. All in all, TQCA is a potentially powerful tool, but as such it is only applicable to crisp-set analysis and in empirical studies with a very limited number of conditions. Indeed, empirical studies using this technique are still very rare (Hak et al. 2013). In public policy research, TQCA seems particularly appropriate for accounting for mechanisms of policy diffusion, given its capacity of discriminating sequences that produce a positive outcome (for example, the adoption of a policy in any given jurisdiction) from causally inert sequences of necessary conditions.

Aggregate Operationalization of Procedural Variables

The third idea is simply to aggregate quantities that vary over time to incorporate a procedural element into calibrated conditions. For instance, if we want to account for the participation of an actor in the course of a given policy-making process, we could proceed as follows. First, we have to break the process in a number of discrete events, which are not necessarily sequential, such as (1) agenda setting; (2) explorations; (3) working out the draft; (4) consultation; (5) draft modification; (6) decision; (7) monitoring/implementation; (8) sanctioning/evaluation. Then we can observe the participation of the target actor in each one of these events, and finally aggregate the obtained scores in a single condition to be included in the QCA analysis. This condition operationalizes the time dimension because it gives weight to different stages of the policy process – the more events an actor takes part in, the higher its overall score will be. However, the time dimension is not explicitly modeled. Furthermore, the aggregation procedure may be cumbersome and therefore results could be difficult to interpret with respect to the causal relevance of temporality. Since information on the unfolding of events over time has been conflated into a single aggregated measure, it is crucial to use descriptive information about the policy process in question to gain knowledge on how time matters when interpreting the QCA results. More specifically, longitudinal descriptive evidence should be used to understand which events represent the most relevant elements of the causal chain leading to the outcome, in a way that is not dissimilar to the technique of process-tracing (Checkel 2006). As an example, Maggetti (2009) uses network analysis to derive a synthetic measure of the influence of independent regulatory agencies in selected policy processes. Afterward, this measure is treated as the outcome condition in a QCA analysis. A “thick description” of the policy process is used to identify the stages of the process that most affected the final result. All in all, this procedure is useful to make sense of phenomena related to policy processes, such as those studied with policy cycle theories.

Temporal Use of Two-Step QCA

Furthermore, two-step QCA is a technique that operationalizes the distinction between so-called remote and proximate conditions in a context of limited diversity. First, the researcher identifies contextual factors (remote conditions) that enable the occurrence of the outcome. Second, the remote conditions that offered explanatory power in the first analysis are then combined with specific triggers (proximate conditions) in a more precise analysis, in order to find out parsimonious necessary and sufficient combinations leading to the outcome. The goal is to reduce complexity so as to mitigate the problem of limited diversity and accurately model the causal structure of the argument (Schneider and Wagemann 2006). Schneider and Wagemann applied this technique to analyze the causes of the consolidation of democracy. In their example, remote conditions summarize sociocultural, economic and historical features of the countries under investigation, while proximate conditions refer to politico-institutional factors such as the executive format, the type of electoral law and the degree of party fragmentation. Remote and proximate conditions are usually conceived of in a causal sense.⁶ However, it is possible to assume that remote conditions may also correspond to those that are remote in time, while proximate conditions would correspond to the immediate trigger of an event. This approach could be appropriate to study policy punctuations, because it allows researchers to study the long-term and short-term conditions for (sudden) policy change.

Separate Analysis of Points in Time

The last technique involves the execution of a separate QCA analysis for each point in time followed by the comparative inspection of the solutions. Specifically, when data are available for a small to medium number of points in time, as for panel analysis, it is possible to examine each segment with a distinct analysis. Some conditions are likely to remain constant across the models (such as those related to the institutional framework), while others should vary (such as contextual factors and some actor-level variables). Afterwards, the comparative inspection of the solutions referring to each specific segment allows the researcher to identify regular patterns leading to the outcome of interest that might emerge over time. The examination of the similarities and differences across the solutions provides evidence about the time consistency of the explanatory model. In order to systematize the comparison of the solutions, it is possible to use the distinction between core and peripheral conditions introduced by Fiss (2011). In this case, core conditions would represent conditions that hold explanatory power over time, while peripheral conditions are those that are contingent on specific segments. This approach is relevant when the analytical goal is to find a set of explanatory factors that are stable for a certain period, or, on the contrary, to discriminate between periods that are characterized by different explanatory factors, such as in the case of the identification of critical junctures and path-dependent processes (Pierson 2000; Hacker 2004). The main restrictions are the availability of longitudinal data for all the conditions and the difficulty of examining a large number of points in time.

Conclusions

This article reviews and discusses two challenges – and ways to address them – related to the analysis of policy processes with Qualitative Comparative Analysis. Both are related to causal complexity, with a specific focus on multiple configurational causality and temporality. These challenges do not apply only to QCA or the analysis of policy processes, but given QCA’s explicit claim to be able to deal with causal complexity, which is an inherent feature of policy processes, discussing them in this context is particularly important.

Concerning multiple configurational causality, we argue that the inductive assessment of the complex interactions among interrelated elements would be appropriate for the analysis of phenomena such as policy processes, and yet the purely inductive use of QCA is not always possible or warranted. A deductive approach, on the contrary, raises questions about the correspondence between hypotheses, the method and the analytical goal (i.e. the in-depth study of policy processes). We discuss two possible stances that researchers can adopt to strike an appropriate balance between induction and deduction: On the one hand, the researcher can formulate their own theoretical argument involving complex interactions between two or more conditions stemming from different levels or from several theoretical frameworks. However, one should make sure that the theoretical explanation is compelling and well supported by logical arguments. On the other hand, one can formulate simple hypotheses based on existing theory, and then test for more complex configurations in a more inductive way. In this case, it seems important that researchers explicitly acknowledge the potential gap between theoretical expectations and the empirical analysis and recognize the trade-offs associated with inductive and deductive approaches.

With respect to temporality, the main challenge is that QCA lacks a direct way to incorporate the time dimension into the empirical analysis, despite the fact that a diachronic perspective is crucial to make sense of policy processes. We discuss a number of solutions, none of which are dispositive. They only apply in quite specific contexts. The use of an additional condition to discriminate time-related factors is recommended when studying research questions that are not diachronic in and of themselves, such as the timing of policies, which is examined by the multiple stream theory. Temporal QCA, which is based on the idea of “non-commutativity” of configurations, seems particularly appropriate for policy diffusion processes, given its capacity of identifying sequences that produce a positive outcome. The third idea is to aggregate quantities that vary over time to incorporate a procedural element into calibrated conditions. This procedure is useful to make sense of policy processes, such as those studied with policy cycle theories, but requires including a “thick” description of what happens in the course of the process under investigation. Furthermore, the distinction between so-called remote and proximate conditions adopted by two-step QCA could be applied both to conditions that are remote in time and to those that correspond to the immediate triggers of an event. This approach could be appropriate to study policy punctuations, because it allows researchers to study the long-term and short-term conditions for policy change. Finally, the comparative inspection of separate QCA executed at specific points in time allows the researcher to identify regular patterns that emerge over time. This solution mostly applies to cases where longitudinal data is available for all conditions in a small to medium number of segments.

While multiple configurational causality and temporality prove to be tricky issues when using QCA for the analysis of policy processes, our discussion of potential solutions shows that the existing challenges are not impossible to overcome. Yet we do not wish to, nor would it be advisable to, present one single best practice to deal with both issues. Researchers should choose their preferred solution, depending on their specific research question and study context, or they may be able to propose additional solutions to the challenges identified here. What seems important is that researchers recognize the existence of these challenges more explicitly in their empirical work, as well as the way they deal with any specific challenge.

Furthermore, we believe that this article also provides an important critical review of recent applications of QCA, as well as a discussion of current issues. These methodological issues are potentially problematic well beyond the study of policy processes and public policy. First, many other (social) phenomena have been successfully studied under the assumption of multiple configurational causality. They range from sociological or anthropological questions to questions related to business management and health care (Rihoux and Marx 2013). Applications in other domains could also benefit from our discussion on how to deal with the mix between inductive and deductive approaches. Second, and similarly, whereas temporality is a crucial feature of policy processes, many other (social) phenomena also evolve over time. Taking time into account is a challenge in any domain and with any method of analysis (Pierson 2000). As such, the issues and potential solutions we discuss have the potential to be of interest also to other, adjacent disciplines.

Additional information

Notes on contributors

Manuel Fischer

Manuel Fischer is a researcher at Eawag (Swiss Federal Institute for Aquatic Research and Technology) and a lecturer at the Institute of Political Science at the University of Bern. He is working on the characteristics and transformations of political decision making and political networks, with a focus on environmental and water governance. He teaches governance and energy policy. His latest book is Political Decision-Making in Switzerland: The Consensus Model under Pressure (Palgrave MacMillan 2015, co-written with Pascal Sciarini and Denise Traber).

Martino Maggetti

Martino Maggetti is associate professor at the Institute of Political, Historical and International Studies (IEPHI) of the University of Lausanne. His current research interests mainly focus on regulatory agencies, multi-level policy making, the domestic impact of soft rules, and transnational private governance. He teaches public policy, mixed methods, and the politics of regulation. His latest book is Comparative Politics: Theoretical and Methodological Challenges (Edward Elgar 2015, co-edited with Dietmar Braun).

Notes

1. Note that from a technical point of view, the number of cases is not a reason to use or not use QCA (Cooper 2005; Thiem 2014). However, an intimate knowledge of cases is important for the interpretation of results.

2. Co-variational reasoning is based on the simple co-variation or correlation between two variables. The main aim of these approaches is most often to detect the net effect of a single variable on an outcome (dependent) variable.

3. Another aspect of causal complexity if asymmetrical causality, including necessity and sufficiency of solutions.

4. An additional challenge with testing complex configurational hypotheses is related to the number of possible combinations of factors. Given that QCA solutions are complex, it is difficult to define when a hypothesis should be accepted and when not, given that the complex solution term will probably never exactly correspond to the complex hypothesis.

5. Adopting a middle ground between deduction and induction might also present a way to deal with the problem of measurement error raised by Hug (2013). The problem could be minimized when the researcher limits her/his analysis to (very) few cases to be studies in-depth, for which the researcher develops an intimate and fine-grained knowledge, so that measurement error can be largely avoided.

6. Proximate conditions are directly causing the outcome, whereas remote conditions have some influence on the outcome, but the influence is related to the outcome only through intermediary variables.