The knowledge behind Brexit. A bibliographic analysis of ex-ante policy appraisals on Brexit in the United Kingdom and the European Union

ABSTRACT

In this article we map and explain the sources of knowledge cited on 85 Brexit impact appraisals, 46 of which were formal impact assessments ordered and published by the European Parliament and 39 ‘sectoral reports’ ordered by the UK Government and released by the House of Commons Exiting the EU Committee. All reports were published between the day after the UK referendum and the year after the start of the UK-EU negotiations. We conducted a citation analysis of 3537 references and tested author push and policy sector pull hypotheses with non-parametric tests. Our findings highlight the epistemic function of the professional referent groups to which authors belong. Authors tend to generate information and cite sources that are congruent with their ‘home group’ in the departmental unit where they work, or their larger professional group, even in urgent high-salient risk situations like Brexit. Differences between policy sectors do not strongly matter.

KEYWORDS:

• Bibliometric analysis
• Brexit
• impact assessment
• knowledge utilization
• social epistemology

Introduction

Brexit constitutes an unprecedented complex situation for policy makers in the European Union (EU) and the United Kingdom (UK) (Fossum, 2019; McConnell & Tormey, 2020). While its full consequences can hardly be predicted, analysts and decision makers on both sides of the Channel are challenged to anticipate them and take the necessary policy measures. However unprecedented, Brexit is not unique. There are numerous international problems that are urgent (Dunlop, 2014), such as for example the recent Coronavirus pandemic, the refugee crisis and climate change. They are all technically complex problems and characterized by uncertainty as to the appropriate political and policy responses. It is these ‘sorts of uncertainty’ that ‘give rise to demands for particular sorts of information’ (Haas, 1992, p. 4). Since we are not only living in ‘an age of assessment’ (Rayner, 2003, p. 163) and evidence based policy, but also in an era of ‘fake news’ (D’Ancona, 2017), it is of great political and social significance to examine the knowledge informing major policy decisions on such high salient risk problems.

The knowledge behind such decisions comes from many sources and takes many forms. The ex-ante assessment of legislation and policy programmes, is one of the most valuable tools governments have at their disposal in order to determine future consequences and thus plan best future policies (Dunlop & Radaelli, 2016; Hertin et al., 2008). Such impact assessments, or more broadly speaking ex-ante policy appraisals (Adelle et al., 2012) have strong potential to influence the policy agenda, and the eventual choice of policy measures. With this potential in mind, it is important to investigate what type of knowledge sources feed into impact studies, and whether the selection of such sources is influenced by any clear dynamics. Thanks to the political salience of the UK government sectoral reports in the first stage of negotiations with the EU, Brexit is not merely an ‘urgent’, but also a ‘critical case’ (Flyvberg, 2006) that allows us to observe the dynamics of knowledge use within such ex-ante policy appraisals. Given the involvement of multiple departments and authors in drafting impact studies across the Channel, as well as the multiplicity of policy fields involved, the Brexit ‘impact reports’ allow us to examine the interplay between author bias and policy sector pull when it comes to utilizing policy relevant information. Do knowledge sources differ depending on the type of actor holding the pen in drafting these strategic policy documents, or does the type of knowledge feeding into impact assessments depend on policy field characteristics?

These questions touch on an important dimension of the practice of ex-ante policy appraisals: the use of evidence within impact reports. Although there is a plethora of studies on the quality of evidence in policy appraisals (see systematic review by Adelle et al., 2012), the issue of the knowledge sources used to inform such assessments is relatively under-researched. Our aim here is to first of all cover this void. We map the sources of knowledge cited by the authors of 85 impact studies in a wide variety of policy fields that concern Brexit. We then explain variability in the use of such information sources. Thus, we do not analyse the quality of the assessments as such, but merely the quantity of the different types of knowledge sources cited. Our study is the first, though, to analyse different types of knowledge sources- beyond scientific knowledge- by different types of authors, other than public servants, drafting policy appraisals.

Method-wise, we join a growing strand of knowledge utilization literature applying citation analysis (e.g., Ban & Patenaude, 2019; Christensen, 2018; Desmarais & Hird, 2014; Vilkins & Grant, 2017). Our units of observation are 3537 unique citations. Our cases and level of analysis are the impact studies from which we extracted the references. We constructed a novel data set consisting of 46 Impact assessments issued by the European Parliament (EP), and 39 so-called ‘Sectoral Reports’ issued by the UK government and released by the Exiting the European Union Parliamentary Committee in the period between the day after the referendum on Brexit up until 1 year after the official start of the negotiations.

Theoretically, we apply a social epistemology lens to knowledge utilization (Vähämaa, 2013). We argue that authors who steer the pen are powerful actors who control knowledge and information, not on the basis of ‘guesses’, nor of ‘raw’ data, but because of their subjective perceptions of what constitutes credible knowledge (Haas, 1992; Vähämaa, 2013). Such perceptions are only rarely the product of author participation in a network of professional experts, an epistemic community à la Haas (1992). At their most basic level, they are the product of the epistemic functions of the main professional referent groups to which authors belong. Authors generate information and cite sources that are congruent with their main referent groups. Our approach, borrowed from social epistemology, brings in a new angle to the already quite extensive literature on the use of evaluations, knowledge, policy advice and policy evidence (e.g., Lindblom & Cohen, 1979; Weiss, 1979).

Theory

The dependent variable

Our study focuses on the sources of knowledge used by the authors of the Brexit impact reports, which were released in the aftermath of the UK referendum on Brexit and during the initial stage of EU-UK Brexit negotiations. Thus, the concept of interest here is knowledge utilization, also referred to as information utilization (Oh & Rich, 1996). The literature on knowledge utilization dates back to the 1970s (e.g., Caplan, 1979; Lindblom & Cohen, 1979; Weiss, 1979), and has known a strong revival with the evidence-based policy movement (e.g., Nutley et al., 2007). Although it has gone through different stages of development (Dunlop, 2014), emphasis is mainly put on the demand side of the knowledge market. Scholarship on the supply side is less empirically developed (see Howlett & Newman, 2010). As to the type of knowledge, the majority of studies focuses on social science research (e.g., Lindblom & Cohen, 1979; Weiss, 1979), and on the use of knowledge by public servants in particular (e.g., Howlett & Wellstead, 2011; Jennings & Hall, 2012; Landry et al., 2001; Newman et al., 2017). Studies on the use of other types of knowledge are limited. Moreover, attention to the authors who actually draft impact reports is rare. Our research addresses these gaps.

Investigating knowledge utilization is a notoriously difficult methodological undertaking. Notwithstanding the problem of defining knowledge, when can we say that a piece of knowledge has been used? The three-fold conceptualization of knowledge use – instrumental, conceptual and symbolic –-, which was pioneered by Carol Weiss (1980), still structures scholarly debates today (Dunlop, 2014). Given that conceptual and symbolic uses can be considered as catchall categories that limit their validity (Rich, 1991), other concepts and indices have been devised to measure knowledge utilization (Landry et al., 2001, p. 336). One of the most frequent alternatives sees knowledge utilization as a single step by step utilization ladder (Knott & Wildavsky, 1980). From bottom to top the steps in the ladder are knowledge transmission, followed by cognition, referencing, adaptational effort, influence and finally application (Knott & Wildavsky, 1980; Landry et al., 2001, p. 336; Amara et al., 2004).

In the present article our interest lies with all different types of evidence informing the Brexit impact reports and not solely with scientific knowledge. Thus, to paraphrase David Bloor (1976, pp. 2–3) knowledge in our study is whatever authors of impact reports take knowledge to be. Impact studies, by default rely on a diversity of evidence in terms of relevant scientific disciplines, study designs, and sources of information that they comprise. This evidence diversity reflects a knowledge base that is not only technocratic (scientific and/or bureaucratic), but also participative (stakeholders) (Cashmore, 2004). The exact types of knowledge that are prioritized differ across studies though.

With respect to use/utilization our focus is on referencing, as this is evidenced by the citations used in each impact study. Despite its documented drawbacks mainly when focusing on academic documents (e.g., MacRoberts & MacRoberts, 1989), citation analysis has a major strength. It enables a quantifiable and measurable approach to the phenomenon of information utilization. We acknowledge, however, that our methodological strategy captures only one side of the varied nature of knowledge utilization (Nutley et al., 2007; Weiss, 1979). Moreover, as already highlighted by Lindblom and Cohen (1979), it would be a mistake to conclude that a particular type of knowledge source that is not directly used (i.e., not cited in our case), has not had any influence at all.

Explaining sources of knowledge used

Policy appraisals, whether in the form of ‘systemic and mandatory’ impact assessments (Dunlop & Radaelli, 2016), or in the form of ad hoc and light touch impact reports (see below), are drafted by single authors, or teams of authors, who carry out a policy analysis by making use of a wide range of information. Their aim is to assess the effects of a proposed legislation, policy programme, or specific project (Adelle et al., 2012; Owens et al., 2004). The question then becomes, what type of information/knowledge source do such actors use in policy appraisal studies and why?

We answer this question by looking into the epistemic function of group membership (Fallis, 2007; Fuller, 2002; Vähämaa, 2013). Authors push information into the policy appraisal studies on the basis of their individual preferences and attitudes towards what constitutes valuable, credible and trustworthy knowledge. However, authors’ preferences for information use are not exogenous, but will be influenced by the groups of which they are members. First, the authors of policy appraisals are members of large-scale professional groups, such as for instance public servants; scientists; journalists; members of a stakeholder group etc. Second, they are also members of smaller and more local casual groups such as for instance the group of colleagues in their office (the ‘home group’), or other informal groupings with the members of which they are cognitively and emotionally involved (Tajfel, 1982). We know from social epistemology that all these groups function as epistemic communities (Fallis, 2007), broadly defined as ‘thought collectives’, meaning sociological groups with a common style of thinking (Fleck, 1939 [1979]). Such groups ‘act as a type of epistemic machinery’ (Bergin, 2001, p. 376), as referent groups that employ an ‘epistemic calculus’ (Vähämaa, 2013) with two purposes. First, a ‘veritistic’ one, according to which they maximize accurate beliefs, while rejecting as many false beliefs as possible (Fallis, 2007). Second, a social one. According to Vähämaa (2013, p. 6) more than ascertaining the truth, the epistemic purpose of such groups is ‘the functionality of the group itself, maintenance of group coherence and allocation of shared understanding among the group members’.

Authors of policy appraisals are by default members of two types of referent groups each with its own epistemic functions: the smaller scale, local referent group of the immediate professional environment –their co-authors on impact assessments (if applicable); their office; unit; departmental organization- and the larger scale group of the professional class to which they belong. They might also belong to broader networks of experts, such as an epistemic community as defined stricto senso by Haas (1992), or a scientific community as defined by Holzner and Marx (1979). Participation in such networks would have additional important implications for the knowledge they use in their appraisal reports. However, as this might not often be the case, it is beyond the scope of the present study to investigate this type of expert network membership.

Following this line of reasoning, we formulated two author push hypotheses. To begin with, the authors of the impact reports can be insiders/employees of the administrative unit that undertook the task of writing a policy appraisal report on behalf of executive politicians. For example a civil servant in the department of Trade in the UK, drafting an appraisal study on behalf of her minister. Such authors are aware of being members in a professional in-group of departmental colleagues, they share a set of values and are cognitively and emotionally involved in the group's work. Membership of this in-group serves two important epistemic functions. First it provides functional knowledge as to what constitutes a credible source of knowledge and second it allows authors to maintain a ‘personal affective state’ with their ‘home’ group. Thus, when writing an appraisal report, authors coming from this internal bureaucratic in-group will tend to generate information and cite sources that are congruent with their referent home group. As a result, we would expect that very often these ‘reliable sources’ would tend to come from inside the authors’ organization. By contrast, some authors might be writing a policy appraisal report on commission. A political organization, such as the EP, might have outsourced via one of its departments the writing of an impact assessment to an expert outside. Such external authors will not be bound by the internal in-group membership and they will be more inclined to utilize information which is not necessarily congruent with the bureaucracy that commissioned the report. We therefore expect:

H1: Authors who are insiders/employees of the administrative unit that undertook the task of producing a policy appraisal report will tend to cite more sources from their home organization

As professionals, authors are by default members of larger scale professional groups. The insiders of the ‘home’ departmental units mentioned above are by definition core government public servants. Policy appraisal authors, however, the externals mentioned above for instance, can also come from the broader public sector, for example from independent public research institutes, or statistical agencies, but also from outside the state, from science, think tanks, consultancies or even multiple professions when working in teams. On the basis of what we know from social epistemology (Fallis, 2007; Fuller, 2002; Goldman, 1999), not only local ‘home’ groups, but also larger and more formal professional groups have their own distinct ‘epistemic calculus’ and thus produce their own distinct epistemic standards as to what is considered credible and trustworthy knowledge, or the opposite. When looking for functional knowledge to complete their policy appraisal tasks, authors-members of such a larger professional group would seek information within this referent group, consistent with the observation that ‘like attracts like’. An academic will tend to draw more on academic sources, while a bureaucrat, even if external to the departmental unit that commissioned the study, will tend to look in the wider bureaucracy for information. Thus:

H2: Authors will tend to cite knowledge sources from organizations that are similar to the one they as authors are affiliated (like attracting like hypothesis)

Impact studies are usually organized at sectoral level (Dunlop & Radaelli, 2016). This means that authors are asked to evaluate future developments within a specific policy sector in which they are experts. Policy sectors are not referent groups, but ‘relatively stable and clearly demarcated issue arenas’, or ‘subsystems’ around which develop ‘relatively integrated policy communities’ thanks to the ‘joint effects of specialization, expertise and social interaction’ among the various participants such as for example bureaucrats, clientele groups and policy professionals (Freeman, 1985, pp. 483–484). As communities organize around specific issues, programmes, ministries and policies, similar policy sectors tend to exhibit convergence, whatever the national or system context. It has been argued that one area of within policy sector convergence is knowledge utilization. Different policy sectors are associated with different processes of information use (Oh, 1997, pp. 7–8). One reason for this is that they lower the costs of information (Freeman, 1985). Beyond, this transactional logic though policy sectors can also set epistemic standards, much like social referent groups do. This is achieved via two avenues. First, policy sectors can be dominated by ‘sectoral paradigms’ (Beland, 2005). These are ‘road maps’ that may range from a shared understanding of how politics and policy function to shared cause and effect assumptions. Second, the ‘logic of subsystem politics’ (Freeman, 1985) means that policy sectors can be dominated by specific configurations of actors and policy communities, which can project their epistemic standards on the sector. As a matter of fact, the evaluation field has largely developed along sectoral lines, with different configurations of actors dominating the discourse in particular fields, and particular ex ante techniques prevailing in specific sectors (Stockmann et al., 2020). Similar dynamics have been found in the policy advisory systems (PAS) literature which has shown that the configuration of advisory actors differs substantially across policy sectors (Halligan, 1995, p. 141). Some PAS are relatively strongly externalized and pluralized, implying also that in these settings a relatively diverse set of knowledge will circulate. Especially in policy fields in which policy appraisal studies are strongly institutionalized, such as environment or energy, authors can rely on a wide body of knowledge coming from a diverse set of actors. With this in mind, we speculate that:

H3: Reports in policy sectors, where the policy advisory system o is more pluralized, would cite a wider range of knowledge sources

Methodology

The study population

To test our theoretical expectations we assembled a novel dataset of 85 Brexit impact appraisal documents, out of which 46 are formal impact assessments ordered and released by the EP (European Parliament, 2018) and 39 are sectoral reports ordered by the UK government and released by the House of Commons’ so-called Exiting the EU Committee (House of Commons Library, 2018). The latter was created in 2016 to scrutinize the UK government's activity and legislation around Brexit, and the work of the Department for Exiting the European Union (House of Commons Library, 2018). Both the EU and the UK have institutionalized a fairly extensive IA system (Hertin et al., 2008; Radaelli & De Francesco, 2010). Research has also indicated that they do not differ significantly from one another, albeit the fact that the EU seems to be a bit ahead when it comes to the estimation of environmental and social effects (Fritsch et al., 2013). The EP can be said to be an emerging actor in the field of impact assessments, as also documented by the activity reports of the Directorate for Impact Assessment and European Added Value of the EP Research Service (EPRS) (EPRS, 2019).

All studies analysed are members of the ‘wider family of ex ante techniques’ (Owens et al., 2004, pp. 1943–1944), commonly referred to as policy appraisals, which ‘seek to inform decision-making practices by predicting and evaluating the consequences of various activities according to certain conventions’ (Adelle et al., 2012, p. 401).

We included all reports dating between the day after the 23rd June 2016, when the UK referendum on Brexit was held, and the 1st of May 2018, which marks exactly one year after the start of the Brexit negotiations. The cut-off date was chosen on pragmatic grounds, as, when the research was concluded, this was the most recent date for which Brexit reports were available.

Operationalization of the dependent variables. Citation analysis

The terms use and utilization of knowledge sources are used interchangeably to refer to the citation of multiple sources of information by the author(s) of each impact study. We analyzed the 85 reports using bibliometric/citation analysis as the main method. There was wide variation in the way of referencing. Not all documents included a list of references. Some used endnotes, others footnotes, or only had web links. The analysis singled out 3537 unique citations to multiple sources of information. We manually reconstructed all citation information per document. As a first step towards constructing our dependent variables, we coded each unique citation according to the type of knowledge source it represented. We distinguished between seven categories: academia; government and administration; independent government research institutes and statistical agencies; think tanks; consultants; societal stakeholders; media (see supplementary material- table 3). The coding proceeded in an iterative way, in which we triangulated between different researchers, to ensure validity and reliability and reducing potential biases.

Once all 3537 unique references were coded we were able to calculate the exact number, as well as the percentage of the knowledge source types that were cited in each impact study. Besides knowledge sources, we also collected data on a series of key variables at the level of the impact reports, including publication date; length in pages; organization that sponsored/ordered the drafting of the document; policy sector; type of organization in which the author is affiliated; location of the author vis a vis the sponsor organization.

Operationalization of the independent variables

Our key independent variables comprise measures of the three factors expected to influence the use of knowledge source on the Brexit impact appraisal reports. The author push factors are home group membership and membership of a larger professional group. The former is coded as a 0,1 categorical variable that measures the location of the author vis-à-vis the departmental unit responsible for producing the report. Authors who are employees of the specific in-house department that sponsored the impact report are coded as insiders/internal (0), while all others are coded as outsiders/external / (1). The second variable, membership of a larger professional group, is operationalized as a categorical variable discerning 6 groups of author professional affiliation: academics; civil servants in the core administration; civil servants in the broader public sector; think tanks; consultants; multiple authors. Finally, we operationalized policy sector as a categorical variable with 11 categories: 1 = fiscal and monetary; 2 = health; 3 = agriculture; 4 = social policy, employment and pensions; 5 = education; 6 = environment and energy; 7 = constitutional affairs; 8 = Real economy (business, trade, finance, industry); 9 = Foreign policy and defence; 10 = Constitutional issues; 11 = Migration and asylum. Given the difficulty in producing theoretically abstract policy sector categories we additionally used the policy sector operationalisations in the UK and EU Comparative Agendas Project codebooks (Alexandrova et al., 2015; Jennings & Bevan, 2010) (supplementary material S2).

Analysis

In order to test our hypotheses we used a series of non-parametric tests. In particular the Wilcoxon rank-sum and Mann–Whitney tests (2 independent conditions) to examine hypothesis one (home group membership). We used the Kruskal–Wallis test (several independent conditions) for hypotheses two (membership of a larger professional group) and three (policy sectors). Parametric tests such as multiple regression, logistic regression and MANOVA violated fundamental assumptions such as linearity, homoscedasticity and normality. Our sample was also not very big to rely on the corrective effects of the central limit theorem. Running a multivariate model with multiple dependent variables ended up with the problem of too many parameters per observation (17 parameters and 85 observations). MANOVA also violated Levene's Test of Equality of Error Variances. We attempted various corrections. Despite the various corrections, though, certain assumptions were still not met. We thus decided in the end to opt for non-parametric, ‘assumption-free tests’. Although, these tests are powerful in detecting effects when the data are not normally distributed, they do not allow us to build a model. Essentially they work as a series of one-way ANOVAs.

Thus, we created seven dependent variables each representing percentages of citations to a specific knowledge source within an impact report. We then proceeded with ranking the percentages for every dependent variable. A collateral benefit of ranking was that it solved the problem of having to deal with fractions as dependent variables. The ranking took an ascending order. We used the mean of two ranks to deal with tied ranks. The results for the Mann–Whitney and Wilcoxon tests for hypothesis one were quite straight forward as they only test two independent conditions. In the case of the Kruskal–Wallis test (several independent conditions), which we used for hypotheses two and three, we performed follow up analyses in order to investigate differences between pairs of categories (professional groups and also policy sectors). For this we used Dunn's post hoc test with the most conservative Bonferoni adjustment.

Results

Table 1 summarizes the results from the non-parametric tests. For each test between an independent and a dependent variable we report the relevant statistics (U = Mann–Whitney test and W = Wilcoxon test for IV 1; H = Kruskal–Wallis test for IVs 2 and 3). We also report the standardized scores (z) and the effect size (r), as well as whether the association between the independent and the dependent variables is statistically significant (p).

Table 1. Explaining variability in citations of different types of knowledge sources.

Independent variables	Dependent variables: 7 sources of knowledge
	DV1. Academic sources	DV2. Government & public administration sources	DV3. Independent government think tanks & statistical agencies sources	DV4. Think tank sources	DV5. Consultancy sources	DV6. Stakeholder sources	DV7. Media sources
IV1 Author home group membership	U = 1547.500, W = 2367.5 z = 6.11, r = 0.662, ***(p = 0.000)	U = 825, W = 1645.5, z = -.656, r =  −0.071, (p = 0.512)	U = 403.000, W = 1223.000, z = −4.420, r = −0.48, ***(p = 0.000)	U = 1411.00, W = 2231.000, z = −4.775, r = −0.517, ***(p = 0.000)	U = 511, W = 1331.000, z = −3.809, r = −0.413, ***(p = 0.000)	U = 452.500, W = 1272.5.000, z = −3998, r = −0.433, ***(p = 0.000)	U = 1356.000, W = 2176.000, z = 4.613, r = 0.5, ***(p = 0.000)
IV2 Author professional group membership	H(5) = 48.176, ***(p = 0.000)	H (5) = 7.948, p = 0.159	H (5) = 26.64, *** (p = 0.000)	H (5) = 27.827 ***(p = 0.000)	H (5) = 17.804, **(p = 0.03)	H (5) = 19.617, ***(p = 0.01).	H (5) = 21.437, ***(p = 0.01)
IV3 Policy sector	H (10) = 26.545 **(p = 0.03)	H (10) = 18.240, *(p = 0.051)	H (10) = 11.174, p = 0.274	H (10) = 19.530, **(p = 0.034)	H (10) = 21.47, **(p = 0.018)	H (10) = 21.437, **(p = 0.018)	H (10) = 17.015, *(p = 0.074)
Observations	85	85	85	85	85	85	85

Note 1: *p < 0.1; **p < 0.05; ***p < 0.01.

Note 2: U = Mann-Whitney test; W = Wilcoxon test; H = Kruskal-Wallis test.

To begin with, we tested our first hypothesis using the Mann–Whitney (U) and Wilcoxon (W) tests. The evidence is strong (Table 1, author home group membership) that differences in membership of the home group (internal/insider), or not (external/outsider) significantly affect the share of citations in six categories of sources of knowledge (academia; independent government research institutes; think tanks; consultants; stakeholders; media), except for sources from government and the public administration. Pairwise comparisons (see supplementary material S3.IV1) between the two author home group membership categories show how each category affects the citation shares across the seven types of knowledge sources. We observe that authors who are home-group members (internal/insiders) use more information from sources that are either close to their professional in-group, for example independent government institutes and statistical agencies, or from actors with whom their professional in-group most regularly engage with in the policy process: consultants and societal stakeholders. By contrast, non-home group members (external/outsiders) tend to use more sources from academia, think tanks and media sources. Differences in home-group membership do not affect the citation of sources from the core government and public administration, meaning that there is no insider/outsider bias when it comes to the use of such sources.

For the second hypothesis, author professional group membership, we used a Kruskal–Wallis (H) test (Table 1). Similar to what we found for the home-group bias hypothesis, the test showed that differences in author professional group membership significantly affect the share of citations for most types of knowledge sources. The only exception again is the citation of government and administration sources. As the test does not show which professional group differences matter, we conducted Dunn's pairwise tests with Bonferoni adjustments (supplementary material S3.IV2) in order to examine the precise way in which such differences affect the quantity of citations for each type of knowledge source. The pairwise comparisons revealed that a) academics tend to be positively biased towards citing academic work in comparison to civil servants; b) civil servants are more inclined to cite sources from independent government research institutes and statistical agencies, consultants and societal stakeholders when compared to academics; c) authors from think tanks tend to cite more sources from thinks tanks when compared to civil servants, but not when compared to other professional groups.

Finally, we tested the policy sector pull hypothesis. The Kruskal–Wallis (H) test (Table 1) shows that policy sector differences significantly affect the share of citations to academic, think tank, consultant and societal stakeholder sources, though at a p < 0.05 level. The test also shows that differences in policy sectors affect the share of citations to government and administration and media sources, though at an even lower statistical level (p < 0.1). Finally, policy sector differences do not affect the share of citations to independent government research institute sources. As the test does not tell us which policy sector differences are the ones that matter, we carried out Dunn's post hoc tests with Bonferoni adjustments for each pair of policy sectors (see supplementary material S3.IV3). The pairwise comparison showed that in the end only two policy sector reports differ significantly and only with respect to the share of citations to consultancy sources. The environment and energy sector impact reports cite significantly higher shares of consultancy sources when compared to studies in the area of constitutional issues, such as for example the effects of Brexit on the composition of the EP. The specific result does not provide strong evidence that policy sectors affect the quantity of knowledge cited from different knowledge sources. The two extra tests we ran for robustness corroborate our result.

Discussion and conclusion

In this article we mapped the different sources of knowledge cited in the 39 Brexit Sectoral Reports (SRs) commissioned by the British government and published by the House of Commons; and the 46 Impact Assessments commissioned and published by the European Parliament. The 85 reports were published in the period covering from the immediate aftermath of the Brexit referendum up to one year from the start of official EU-UK negotiations. Our aim was not to describe differences between the EU and the UK, nor to make normative judgements about the quality of information cited in the impact appraisals. Given the multiplicity of authors and policy sectors involved, the Brexit impact studies offer a unique opportunity to examine author biases and policy sector effects on the use of knowledge in ex-ante policy appraisals in the background of an urgent, high-salient risk and technically complex international problem with no easy political and policy responses.

One could have expected perhaps that in such an event as Brexit, the quest for ‘truthfulness’ and ‘verifiability’ would have pushed the use of knowledge towards similar source directions. On the contrary, our results highlight the relevance of an anthropomorphic perspective to knowledge use (Radaelli, 1997; p. 169-see also Dunlop, 2012) and the importance of social referent groups even in high-salient risk situations (Sjoberg, 2007). Authors of impact studies push information into their reports in a way that reveals biases and thus subjectivity. We explained this by using insights from the knowledge utilization and especially the social epistemology literatures (Fallis, 2007; Fuller, 2002; Goldman, 1999; Vähämaa, 2013). Authors are members of social referent groups (colleagues; profession; policy field) with important epistemic functions: a ‘veritistic’ and a social one. This means that the criterion behind knowledge use is not only credibility of information, as this can be approached by an elaborate cognitive process, but also functionality as authors try to function and benefit as members of a social group by following the ‘epistemic standards’ that the referent group sets (Vähämaa, 2013, p. 7).

The non-parametric tests we conducted provide strong evidence that authors tend to generate information and cite sources congruent with their professional referent groups, be they small and local – their home unit-, or larger and more open – their professional group. This is evident in that academics tend to cite more academic sources, civil servants more public sector sources and members of think tanks more think tank sources. However, beyond this ‘like attracts like’ logic, social referent groups also function as more complex ‘epistemic synthesizers’ (Vähämaa, 2013, p. 8). In their double quest for truthfulness and functionality different professional referent groups point to different knowledge source directions. There emerged two interesting divides here: the insider – outsider and the bureaucrat – academic one. Authors – insider bureaucrats seem to trust information coming from actors they most regularly engage with in the policy process: a) the broader independent government research and statistical agency sector, rather than academics;b) stakeholders with whom they often engage in participative forms of advice exchange, in comparison to think tanks who also provide research like types of evidence; and c)consultants to whom they traditionally outsource work to. Authors – outsiders to the ‘home’ group appear to use significantly more academic sources, more sources from think tanks and more sources from the media. Beyond social epistemology, the above results put previous experimental research into the credibility of different sources of evidence into perspective (Doberstein, 2017). While government policy analysts tend to trust more research from academics compared to think tanks, internal bureaucrats seem to give more credibility to in-house evidence from actors they are more familiar with.

The second and most fundamental divide concerns the use of information between authors with academic affiliations and bureaucrats. Authors from academia significantly differ to authors from government and public administration, as well as authors from independent government research institutes and statistical agencies. There seems to be a difference of culture around the use of scientific knowledge by academics and bureaucrats, which only concerns these two categories of actors and no one else. Our observation comes close to previous studies, which highlighted the lack of capacity of civil servants to use scientific evidence (Newman et al., 2017), or the existence of different interpretive frames of reference (Freiberg & Carson, 2010; Marston & Watts, 2003). Interestingly, the divide emerges also with respect to the use of sources from government research institutes and statistical agencies; sources from consultancy; and sources from stakeholders with academics using significantly less such sources than bureaucrats.

Rather surprisingly, we found robust evidence that differences in policy sectors do not affect citation shares whatever the cited knowledge source. This non-finding is important as it shows that policy sectors, at least with respect to cited knowledge sources in ex-ante policy appraisals, do not show significant within sector convergence. Thus, contrary to the widely acclaimed importance of policy sectors in a wide array of policy phenomena, in this particular one they are not empirically meaningful categories. The theoretical focus should perhaps be elsewhere, on specific issues, such as for instance climate change, rather than whole sectors (ie environment).

Even though 3537 references were examined, we acknowledge that the data set is relatively limited (n = 85). As a result we were only able to test individual hypotheses, without building a model, or controlling for other factors. Future studies will need to test these hypotheses more comprehensively. Moreover, we chose to analyse the number of unique references, irrespective of their actual frequency. Such approach does not do full justice to the importance of a particular source. Yet, merely focusing on the frequency that a certain source is quoted would be too heavily affected by citation cultures. A social network analysis could bring more clarity here. Also, knowing the relative importance of each knowledge source would be interesting. Future research ideally examines this in more depth, also in a longitudinal way. Given all discourse on post fact policy making, it would be useful to examine whether some epistemic groups changed their beliefs about what credible sources are.

These reflections notwithstanding, our study prompts the important question whether the ‘truth’ can be ‘objectively’ ascertained in ex-ante policy appraisal reports. Given the growing technical complexity of policy issues, a growing body of actors tends to consider themselves as experts (Dunlop, 2014). In view of this, a whole range of actors are asked to provide advice on the likely results of various courses of action through policy appraisal studies. As we showed in the case of Brexit, all these actors apply a certain ‘epistemic calculus’ (Vähämaa, 2013) in line with the shared belief and faith of their ‘thought collective’ (Fleck, 1936 [1979]). More recent actors that joined the policy advisory system, such as consultants and think tanks, do not escape these dynamics. Moreover, our findings indicate that the ‘politics of impact assessment’ (e.g., Walker, 2010) already starts at the level of the authors of the impact reports, before it even moves up the ladder to decision makers. While further research should verify this, we expect to find similar dynamics in other instances of complexity, especially since impact assessments usually proceed along the same procedural lines.

Finally, our study also poses questions of a normative kind. From the point of legitimacy, the process of information selection feeding into the policy process can be considered of equal importance as the actual policy decision. This especially applies to wicked problems, such as Brexit, with major social impact (Fossum, 2019; McConnell & Tormey, 2020). When governments decide to outsource the drafting of impact studies to outsiders, this can have clear implications on the sources of evidence that will likely be considered, which may in turn affect the type of advice feeding back to them. Similarly, when keeping the drafting of policy appraisals in house, this entails the risk, deliberate or not, that some types of sources are potentially overlooked. From the point of view of democratic pluralism and policy effectiveness, this can be worrisome, given that wicked problems ideally require a nuanced and multi-sided outlook.

Acknowledgements

The authors would like to thank the editors, Berthold Rittberger and Jeremy Richardson, four anonymous reviewers, as well as Johan Christensen, Fabrizio de Francesco, Jonathan Kamkhaji, and Stéphane Moyson for helpful comments and suggestions.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Notes on contributors

Valérie Pattyn

Valérie Pattyn is Assistant Professor at the Institute of Public Administration of Leiden University

Athanassios Gouglas

Athanassios Gouglas is Lecturer in Politics and Public Policy at the University of Exeter

Julianne De Leeuwe

Julianne de Leeuwe is policy advisor at Rotterdam University of Applied Sciences.