Studying expert influence: a methodological agenda

Abstract

Expert communities, advisory bodies and expert bureaucracies are ubiquitous in national and international governance. Yet, we have surprisingly little systematic empirical knowledge about how much influence these experts actually have on public policies. This is partly due to limited methodological innovation in studies of expertise and policy making and a lack of relevant methodological guidance. The article presents a new methodological agenda for studying expert influence. It outlines five methodological strategies for studying expert influence empirically: two existing approaches – process-tracing and surveys of attributed influence – and three novel strategies – quantitative analysis of preference attainment, text reuse analysis and citation analysis. The agenda is aimed at students of expert influence across a wide range of phenomena, including the influence of scientific experts on policy making, the policy impact of expert advisory bodies, and the sway of national and international expert bureaucracies.

Keywords:

• Influence
• expertise
• knowledge
• methods
• evidence-based policy making

Experts have become a regular presence in policy making across a range of important issues. Scientific councils advise governments about all aspects of the coronavirus response. Economic expert institutions like the International Monetary Fund (IMF) or the Organisation for Economic Co-Operation and Development (OECD) offer member states guidance about the design of economic policies, education systems and public administration reform. Climate scientists produce studies of global warming and policy recommendations for international bodies and national leaders.

Some see this as a sign of a growing technocracy or expertisation of decision making, where experts enjoy great authority in discussions about public policies and where decisions are increasingly based on knowledge and evidence (Habermas 2015). Others argue that politicians have little genuine interest in expert knowledge and frequently reject scientific advice or use experts as convenient cover for predetermined political decisions. Rather than scientization, we are witnessing ‘the death of expertise’ (Nichols 2017).

So how much influence do experts actually have on public policies? Although issues of power and influence are at the heart of the study of politics (March 1955) and experts are a key category of actors providing decision-makers with input (Haas 1992), this question remains poorly understood. We have little systematic empirical knowledge about the extent to which policy decisions are based on expert knowledge rather than on ideology or interests, how influence varies across expert bodies, or how the sway of experts differs across countries and policy areas.

The reasons for this gap are partly theoretical, as key literatures on expertise and policy making have avoided the notion of expert ‘influence’ (Christensen 2021). Work on knowledge utilisation has been interested in the different ways in which expert knowledge is used in policy making (Weiss 1979; Boswell 2008), but with unclear implications for the actual influence of experts. And the literature on evidence-based policy making has favoured the notion of experts ‘informing’ policy making by providing sound evidence derived from rigorous analyses (Head 2016), which grants experts a special status rather than analysing them alongside other actors who seek to influence policy.

However, the reasons for the dearth of systematic research on expert influence are also methodological. Whereas research on the policy influence of actors such as interest groups has seen major methodological advances (Dür 2008; Collins et al.2015), there has been little methodological innovation in studies of expert influence. For instance, work on the policy impact of epistemic communities has largely been confined to single case studies (Cross 2013). Moreover, existing literature offers very little guidance about possible methodological approaches and sources of data for investigating expert influence and the advantages and drawbacks of different approaches.

This article therefore offers a novel methodological agenda for studying expert influence. This agenda is aimed at students of expert influence across a wide range of phenomena, including the influence of scientific experts on policy making, the policy impact of expert advisory bodies, and the sway of national and international expert bureaucracies. The article conceptualises expert influence as the ability of expert actors – i.e. groups, bodies or organisations composed of individuals with specialised knowledge derived from academic training – to shape a policy decision in line with their knowledge-based preferences.

The article presents and discusses the relative merits of five methodological strategies for measuring expert influence. Existing research predominantly relies on two approaches: process-tracing and surveys of attributed influence. Yet, the increasing availability of documents relevant for studying expert influence, such as academic publications, advisory reports, international organisation publications and government white papers, has opened the way for alternative methodological approaches (cf. Wilkerson and Casas 2017). The article therefore outlines three novel methodological strategies for examining expert influence based on documents: quantitative analysis of preference attainment, text reuse analysis and citation analysis.

Conceptualising expert influence

Influence

Questions of power and influence are at the very centre of studies of politics, as they concern the fundamental question of who decides over public policy (March 1955). Existing scholarship has conceptualised and examined the policy influence of various types of actors, including interest groups (Dür 2008; Klüver 2009), politicians (Wilkerson et al.2015) and international organisation bureaucracies (Ege et al.2020; 2021). However, the influence of expert actors has received little systematic attention.

Discussing expert influence only makes sense if we make certain assumptions. First, it presupposes that experts have a degree of autonomy from other actors, such as politicians and interests groups. Experts to some extent need to be able to formulate and pursue independent preferences that do not simply mirror those of other actors. This autonomy does not have to be absolute. Certainly, the research agendas of scientists may partly be shaped by the need to secure government funding, and experts involved in policy making are usually subject to some degree of political control (Hesstvedt and Christensen 2021). Yet, this does not mean that experts are fully controlled by and thus beholden to other actors, as some scholarship in science and technology studies (STS) argues (see, e.g. discussion in Littoz-Monnet 2020). There are enough examples of expert actors who have pursued policy agendas that were not reducible to what other political actors wanted (e.g. Haas 1992; Chwieroth 2010; Christensen 2017).

Second, speaking of expert influence implies that experts do not simply inform policy making. The notion of influence steers clear of the assumption in rationalist work on evidence-based policy making that research evidence is neutral and that experts simply help decision-makers identify the best policies (see Newman 2017). Instead, it assumes that experts are one type of actor among others vying for influence in the political system (Haas 1992).

Dür generally defines policy influence as ‘an actor’s ability to shape a decision in line with her preferences’ (Dür 2008: 561). Adapting this to experts, this article defines expert influence as the ability of an expert actor to shape a policy decision in line with its knowledge-based preferences.

Expert actors

The first element of the definition is the expert actor. While understandings of what experts are vary (e.g. Grundmann 2017), we rely on a mainstream definition based on the following elements: First, experts possess specialised knowledge derived from academic training in a specific field and further developed through professional practice. Second, experts are present within a broad range of institutions, including universities and research institutes, advisory bodies, government ministries and agencies, and international organisation bureaucracies.

Importantly, our definition of expert influence is about experts as actors. It focuses on the influence of the actors and institutions that produce, carry and advocate expert knowledge, rather than on the influence of expert knowledge or ideas in themselves (see Christensen 2021). The process of influencing necessarily involves agency, since some individual, group or body needs to formulate a certain knowledge-based policy preference and bring it to the attention of decision-makers.

Expert actors may be individuals but are more often conceptualised as collective or organisational actors. Expert actors may take the form of groups of experts with shared policy preferences and capable of acting collectively, such as an ‘epistemic community’ (Haas 1992) or a group of experts belonging to a specific ‘profession’ (Fourcade 2006). We are not referring here to entire professions or research communities, but rather to smaller and more cohesive groups of experts, e.g. a group of legal experts working within a competition authority. Expert actors may also take the form of expert organisations, such as national or international expert bureaucracies, or expert bodies, such as expert commissions or advisory councils. Our understanding of expert actors encompasses all these different expert groups, organisations and bodies.

Preferences

The second element of the definition of expert influence concerns the policy preferences of expert actors. If influence is about achieving a preferred outcome, an actor’s preferences are central to the concept of influence (Ege et al.2020). From the perspective of expert influence, we are interested in the policy preferences of expert actors that are derived from their expert knowledge. The knowledge produced within expert disciplines is often seen as an important source of policy preferences, since it gives rise to specific worldviews, beliefs about cause-and-effect relationships and ideas about appropriate policy solutions (Haas 1992; Campbell 2002). This is also the assumption of this article. Yet, certainly, the preferences of experts may also be coloured by self-interest or personal political attitudes (Holst and Molander 2018). The scientific community and advisory institutions have mechanisms for mitigating these kinds of bias, such as conflict of interest rules and peer review. But ultimately, to what extent experts’ preferences reflect their knowledge rather than values or interests is an empirical question.

Policy decisions

The final element of the definition is the policy decision that experts seek to influence. The end result of expert influence is that the content of some public policy comes to reflect the preferences of expert actors. This may involve policy change, if a policy is altered based on expert advice, or policy stability, if expert actors successfully defend their preferred policy in the face of political or interest group pressure for change. Moreover, expert influence may concern one specific policy decision (Dür 2008) or multiple policy decisions over a longer period (Landry et al.2003). The focus in this article is squarely on policy formulation. But it is also conceivable to study the influence of experts on policy implementation, for instance when international expert bureaucracies influence how policies are implemented in member countries (Ege et al.2020).

One can object that this ‘decisionistic’ view of influence leaves out other ways in which experts shape policy making. There is an extensive literature on the different ways in which expertise can be used in policy making, including instrumental, symbolic and political-strategic uses (e.g. Weiss 1979; Boswell 2008). Yet, this typology has partly obscured the question of how much influence experts have, since the implications of the different types of knowledge utilisation for influence are unclear: For instance, appointing an expert commission for symbolic reasons may or may not lead to expert influence on policies. Scholars have also had serious difficulties distinguishing the different types of knowledge utilisation empirically (Christensen 2021). This article therefore reframes the discussion around the notion of expert influence, which is more amenable to operationalisation and empirical study. Importantly, this reframing does not exclude that knowledge can be used in multiple ways; rather, the instrumental, symbolic or political use of knowledge may be the mechanisms explaining why or why not expert preferences come to shape public policies.

Limits

By specifying the basic components of expert influence, this conceptualisation aims to lay the foundation for systematic empirical research on the sway of experts over public policy. Yet, inevitably, the theoretical assumptions made to render the concept operationalizable restrict the kind of phenomena that can be studied. Our concept does not capture all aspects of the role of expert knowledge in policy making. For instance, the assumption that experts possess some degree of autonomy means that the concept is ill suited to study the political role of knowledge providers whose independence is questionable, such as advocacy think tanks or for-hire consultancies. Moreover, since we assume that knowledge and ideas are held by specific actors, the concept does not well capture the more diffuse power of expert ideologies, such as the broad impact of neoliberalism in politics and society. And by focussing on policy decisions, the concept does not capture the effect of expert knowledge on the thinking of policy makers and publics if the shift in thinking does not lead to a shift in policy. Still, the concept offers a robust basis for analysing the influence of a wide range of different expert actors and institutions on public policies through multiple channels across different policy domains and governance levels.

Methodological approaches for studying expert influence

Turning from conceptual considerations to methodological issues, how can the influence of experts be studied empirically? This section discusses five methodological strategies for studying expert influence: two common strategies – process-tracing and surveys of attributed influence – and three new approaches – quantitative analysis of preference attainment, text reuse analysis and citation analysis.

Process-tracing

The first methodological strategy for studying expert influence is process-tracing, which entails tracing the process from the preferences of an expert actor to a final policy decision. Typically, it involves empirically tracing (a) the initial preferences of experts and those of other political actors, (b) the access of experts to decision-makers, (c) the attempts of experts to advocate these preferences to decision-makers, (d) the response of decision-makers to these influence attempts, and (e) the degree to which expert preferences are reflected in the final policy decision (cf. Dür 2008: 562). These steps are traced based on data such as policy documents and semi-structured interviews with participants in the policy-making process, including academic experts, bureaucrats and political decision-makers, which are analysed qualitatively.

Process-tracing is widely used in single case studies and small-n comparative studies of expert influence. It has been employed to examine the influence of economic experts in national policy making (e.g. Lindvall 2009), the impact of expert groups in EU policy making (Metz 2015), and the sway of international expert bureaucracies over national policies (e.g. Chwieroth 2010). For instance, a recent comparative study used interviews and documents to trace the preferences of economists about tax reform, their position within government bureaucracies and the fate of their efforts to advocate their policy preferences to political leaders (Christensen 2017).

The unique advantage of the process-tracing approach is that it allows us not only to examine whether expert preferences were reflected in a policy decision, but also to establish whether and how the two were causally linked by tracing the influence process (George and Bennett 2005). Process-tracing makes it possible to investigate how an expert policy proposal was chosen (or not chosen) among the policy options on the table, and to assess other possible explanations for a policy decision. It also allows us to examine multiple channels of influence and to get at the mechanisms underlying expert influence (see Ege et al.2021).

The major limit of process-tracing is of course that it is so labour intensive that studies are typically limited to a small number of cases: a handful of countries or organisations or a few policy processes. Since a small number of cases are not necessarily representative of the larger universe of cases, it is difficult to draw general conclusions about expert influence based on process-tracing studies. If we want to study expert influence across a larger number of cases, other approaches are needed.

Surveys of attributed influence

The second methodological strategy for tracing expert influence is based on the attributed influence method (March 1955; Dür 2008: 565). Rather than tracing the influence process, this method focuses on measuring perceptions of the influence of an expert actor, based on the actor’s own assessment or on the perceptions of others. It involves asking experts, decision-makers or other involved parties how much influence an expert community, body or organisation had on policy. For few cases, attributed influence can be assessed through qualitative interviews. This is usually one element of process-tracing studies. But attributed influence can also be assessed on a larger scale using surveys.

A number of studies have used surveys of government officials to assess the influence of different types of expertise on the work and decisions of government bureaucracies (e.g. Landry et al.2003; Head et al.2014). For instance, Head and colleagues asked more than 2000 Australian bureaucrats how much importance they attributed to different sources of knowledge in policy making and whether academic research had ultimately influenced policy changes in their policy domain. Recently, surveys have also been used to assess the influence of international expert bureaucracies. Herold et al. (2021) asked more than 350 heads of unit in national ministries in more than 100 countries to what extent they take the policy advice of different international bureaucracies into account.

Surveys of attributed influence allow us to systematically assess variation in the influence of expert actors across a large number of government departments or countries, or between multiple expert organisations (Herold et al.2021), which is not feasible with process-tracing. This approach is also well suited to examine expert influence on policy decisions in general within a department or policy area. It is less suited to examine expert influence on specific policy decisions, since typically only few individuals have first-hand knowledge about who influenced a specific decision. The attributed influence approach also suffers from other problems: it relies on perceptions of expert influence rather than on an objective measure of influence, and expert influence may be over-reported due to the social desirability of knowledge-based policy making. These biases also apply to qualitative interviews, but process-tracing studies can triangulate interview accounts with other evidence when assessing influence.

Thus, while process-tracing and surveys of attributed influence have been the dominant methods for studying expert influence, their limits highlight the need for new methodological approaches.

Quantitative analysis of preference attainment

Inspired by work on interest group influence (Dür 2008; Klüver 2009), one promising alternative approach to measuring expert influence is to examine the preference attainment of expert actors, i.e. to assess the degree to which final policy decisions match initial expert preferences. Assessing preference attainment qualitatively is usually an important element of the process-tracing approach (e.g. Metz 2015). But preference attainment may also be examined quantitatively across a large number of expert bodies, policy processes or countries.

Analysis of preference attainment involves determining the policy positions of expert actors on a given policy dimension (e.g. pro/anti-environmental regulation) and comparing these positions with the final policy output. This can be done based on documents containing information about the policy preferences of experts, e.g. advisory body reports or publications from expert bureaucracies, and documents with information about policy decisions, e.g. government white papers or press releases. Policy positions can be scored based on hand-coding or using quantitative text analysis, for instance by comparing the words used in a document with those used in a reference text for which the policy position is known (Klüver 2009).

This approach is frequently used to study the influence of interest groups (e.g. Klüver 2009). But it has so far not been employed in quantitative studies of expert influence. The possible applications are many: For instance, scholars could take reports from multiple expert advisory bodies (or multiple reports from a single body) and compare the policy positions expressed in these reports to the policy decisions adopted by a government or international organisation. Researchers could also compare the advice of expert agencies and government decisions on a given set of policy measures across a large number of countries, for instance to examine the influence of public health agencies on coronavirus measures.

The major advantage of the preference attainment approach is that it relies on an objective measure of influence rather than on perceptions (Dür 2008). However, a problem is that the correspondence between expert preferences and policy decisions is not necessarily the result of expert influence; a decision may well reflect the influence of other actors with similar preferences. This approach also presents considerable practical challenges (see Helboe Pedersen 2013). First, scholars need to identify the relevant policy decision or non-decision connected to a specific piece of expert advice, which is problematic since expert advice may be relevant for multiple policy decisions. Second, determining policy positions is not straightforward. While hand-coding policy positions is laborious, extracting policy positions based on quantitative text analysis requires documents that are comparable and address a single policy dimension.

Analysis of text reuse

Whereas the preference attainment approach assesses congruence between expert preferences and final decisions, another possible approach to studying expert influence is to examine textual congruence between expert advisory documents and policy decision documents. That is, to what extent do final decisions reuse (i.e. borrow, copy or plagiarize) text from expert documents? Documents expressing policy decisions usually incorporate text from various preceding documents, including documents about existing policy, government platforms, interest group memos and advisory reports. This text-borrowing tells us something about which input is incorporated in policy and which input is not (Wilkerson et al.2015). When a policy decision copies text from an expert advisory report, it may thus be seen as evidence of expert influence (cf. Collins et al.2015: 920). To investigate text reuse, documents are analysed using quantitative text analysis that works like plagiarism detection software: It can assess to what extent sequences of words in final policy documents are copied from a corpus of expert advisory documents (cf. Wilkerson and Casas 2017).

Text reuse analysis has been used to study the influence of interest groups (Collins et al.2015) and the influence of policy ideas on legislation (Wilkerson et al.2015). Yet, the approach also bears great promise for students of expert influence. For instance, scholars could examine the varying extent to which text from advisory commission reports is incorporated in government white papers or bills, or the varying extent to which policy recommendations from international expert bodies are copied in national policy documents.

The text reuse approach has some notable advantages. It relies on an objective measure of influence that can be studied quantitatively, but avoids the practical challenges of the preference attainment approach, such as identifying the specific decision connected to a piece of expert advice. In text reuse analysis, whether there is a link between expert input and final decisions is determined empirically based on an analysis of the entire corpus of documents expressing expert advice and policy decisions. This makes it possible to examine the influence of an expert actor across multiple policy decisions. Moreover, it is possible to pinpoint exactly whose input was incorporated in a policy decision and to rule out other suspected sources of influence.

However, there are also problems. First, copying text does not necessarily indicate influence. It may concern irrelevant procedural information or background description. Language may also be copied to criticise or reject a proposal (Collins et al.2015: 921). Second, expert actors may influence a policy decision without leaving a textual trace, either because the final policy decision expresses in different terms what experts have advised or if expert advice is presented orally or informally.

Citation analysis

The final alternative strategy for studying expert influence is citation analysis. Rather than assessing the substantive or textual correspondence between expert advice and final decisions, this approach examines relations between documents in the form of citations. Policy documents often cite their sources, either through academic-style referencing or by mentioning sources in the text. Which documents are cited in documents expressing final policy decisions provides information about what knowledge and arguments the decision builds on (Steiner-Khamsi et al. 2020). Citations in policy decisions to expert publications can thus be seen as an indicator of influence.

Citation analysis involves gathering and coding citations manually or in automated ways, and then analysing them using standard statistics or network analysis techniques. Various measures of influence can be constructed, including citation counts of how many policy decisions cite a given expert publication (analogous to the Google Scholar citation count for an article) and different network measures that show the centrality of an expert publication within a citation network (cf. Fowler et al.2007).

Citation measures are frequently used to study the impact of academic publications within the scientific community (e.g. Wuchty et al.2007) or the influence of cases in law (e.g. Fowler et al.2007). Yet, they are a relative novelty in research on expert influence. Steiner-Khamsi et al. (2020) use citation analysis to examine which evidence presented by expert commissions influenced government white papers on education policy. Christensen and Hesstvedt (2021) investigate the varying influence of 600+ expert commissions using the number of citations to commission reports in government white papers as a measure of influence. But citation analysis can be applied more broadly. For instance, scholars could analyse citations to publications from different international expert organisations in national policy decisions to measure the relative influence of these organisations and how it varies across countries and policy domains.

The advantages of citation analysis are similar to those of text reuse analysis: It uses an objective measure of influence, can be applied to a large corpus of readily available documents, and allows us to examine expert influence on multiple policy decisions and to compare the policy influence of different expert publications. Yet, citations are an indirect measure of influence: they do not tap into the substance of expert preferences and policy decisions. Moreover, a citation provides little information about the nature or strength of the relation between two documents. Policy decisions may cite expert publications because they are genuinely inspired by their recommendations or symbolically to give the impression that policies are evidence-based. And a policy may be profoundly influenced by some of the sources it cites and marginally influenced by others. Sometimes citations are even used to reject expert recommendations. However, aggregate measures like citation counts or network centrality measures are less sensitive to these problems than single citations.

Conclusion

Debates about the impact of experts on public policies have raged for decades. It is time that empirical research caught up. The article has presented a novel methodological agenda for studying the influence of expert actors such as scientists, advisory bodies and expert bureaucracies. This agenda urges scholars to go beyond the traditional approaches to studying expert influence and explore novel methodological avenues. In particular, it encourages students of expert influence to make creative use of the wealth of readily available documents from expert bodies, expert bureaucracies and governments, which lend themselves to various forms of text and citation analysis.

Both old and new methodological approaches have their strengths and weaknesses. Yet, a broader menu of approaches allows researchers to pick a strategy that is well suited to their specific research question and data. It also opens for methodological triangulation, which can strengthen our measures of expert influence (see Dür 2008). By raising this agenda, the article thus hopes to inspire significant empirical advances in research on the sway of experts over public policy.

Acknowledgements

I would like to thank Cathrine Holst, Stine Hesstvedt, Valérie Pattyn, the participants at the 2022 ECPR Joint Sessions workshop on ‘Governing with Evidence’ and the Institute of Public Administration Research Seminar at Leiden University, as well as the WEP reviewers for thoughtful comments and suggestions on the article.

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

Johan Christensen

Johan Christensen is Assistant Professor of Public Administration at Leiden University. His research centres on the role of experts and expert knowledge in public bureaucracies and policy making. His books include The Power of Economists within the State (Stanford University Press, 2017) and a forthcoming book on expertise, policy making and democracy (with Cathrine Holst and Anders Molander, Routledge, 2022).