The differential impact of evidence types in policy decisions: comparing evidence priorities in employment and education ministries

ABSTRACT

Evidence-based policy-making has received increasing attention among governments in recent decades. However, comparative research examining patterns of evidence impact and administrative priorities for different forms of evidence remains scarce. In response, this article investigates the impact of different evidence types in two Danish ministries – the Ministry of Employment and the Ministry of Children and Education – characterized by similar analytical capacities but different strategies and criteria for prioritizing and using evidence. Applying a novel document matching method to analyze 1159 research publications and policy decisions from 2015–2021 about Danish active labor market policies and public school policies, the article shows how causal effect evidence has to a greater extent influenced policy decisions in the Ministry of Employment than in the Ministry of Children and Education. The article proposes that the observed variation between the policy domains can be attributed to ministerial evidence strategies, shaping administrative perceptions of the relevance and usefulness of different types of evidence. Based on the results, the article discusses the implications of prioritizing particular evidence types in ministries, considering variations between study designs and their appropriateness for different policy needs and purposes.

KEYWORDS:

• Evidence-based policy-making
• evidence impact
• evidence strategy
• ministries
• education policy
• employment policy
• Denmark

Introduction

Responding to widespread pressures for accountability and efficiency, using evidence to inform policy decisions has received increasing attention among governments (Head 2016; Liket 2017; Reed 2018; Sayer 2020). Relying on evidence in policy-making has been promoted by international organizations, such as the OECD (2017) and the European Commission (2015), while advances in modern technology and analysis have expanded the supply of research and data to decision-makers (Legrand 2012). In relation to this, the notion of “evidence-based policy-making” (EBPM) has been promoted and debated (Botterill and Hindmoor 2012; French 2019; Pawson 2002). EBPM proponents have borrowed heavily from “evidence-based medicine” to argue that evidence of causal effect should take centre stage in policy-making. Systematically consulting evidence about the effects of different policy options to make well-informed decisions has been promoted as contributing to efficient and responsible governance (Andersen and Smith 2022; Baron 2018; Cairney 2016; Hansen and Rieper 2010; Head 2016; Ingold and Monaghan 2016). However, the seemingly frictionless transfer of causal evidence into better policy decisions, as implied by the EBPM notion, has repeatedly been described as problematic. Scholars have emphasized how policy decisions are rarely “based” directly on evidence and problematized how the EBPM notion stipulates a narrow definition of “useful evidence”, favouring evidence of causal effect while marginalizing other forms of knowledge from policy discussions and decision-making processes (Ansell and Geyer 2017; Botterill and Hindmoor 2012; Daviter 2015; Oliver 2022; Sanderson 2006; Van der Arend 2014). Significant debates have addressed how to evaluate the relevance and usefulness of study designs for informing policy, what constitutes appropriate types of evidence for various policy needs, and the implications of adopting evidence evaluation criteria from medicine in other policy domains (Dahler-Larsen 2017; Nutley, Powell, and Davies 2013; Parkhurst 2017; Parkhurst and Abeysinghe 2016; Petticrew and Roberts 2003).

Theoretical contributions to the EBPM literature have elucidated different functions of evidence and forms of utilization (Beyer 1997; Nutley, Walter, and Davies 2007; Weiss 1979), and it has been proposed that different cultures of evidence use exist across policy domains and government institutions (Boswell 2015; Lorenc et al. 2014). Such cultures reflect different positions on what constitutes useful evidence and the role that different evidence types should play in policy-making. For example, studies have documented how the medical field adheres to a causal hierarchy of evidence, in which randomized controlled trials (RCTs) are considered the gold standard, and how policy-makers in economics tend to display rationalist and technocratic views on policy-making (Boaz et al. 2019; Christensen and Mandelkern 2022). Oliver (2022:, 80) explains how the common worldview in such policy domains is that policy is best made using evidence, RCTs and systematic reviews are the best types of evidence, and researchers should conduct more such studies to maximize their use by policy-makers. Contributions to the EBPM literature have shown how such perceptions concerning the relevance and usefulness of different evidence types have travelled to other policy domains, shaping how policy problems are addressed in government institutions (Andersen 2020; Baron 2018; Clarence 2002; Hansen and Rieper 2010; Oliver 2022). As such, EBPM has influenced administrative priorities and practices in areas like education, criminal justice, employment, and social policy.

A number of scholars have stressed that the diffusion of evidence evaluation criteria from medicine to other policy domains, through the notion of EBPM, is problematic because an unconditional preference for causal effect evidence in government institutions may marginalize certain perspectives from policy discussions and bias policy decisions towards specific kinds of solutions (Andersen 2020; Dahler-Larsen 2017; Parkhurst 2017; Sayer 2020; Triantafillou 2015). However, comparative research examining patterns of how different forms of evidence impact policy decisions across government institutions remains scarce, thereby limiting the empirical basis for discussing administrative evidence priorities and choices. Existing studies rely on a limited set of perceptual measures, including surveys of self-reported use of evidence (e.g. Head et al. 2014; Jennings and Hall 2012; Landry, Lamari, and Amara 2003) or qualitative case studies of how particular items of research have shaped policy processes (e.g. Hayden and Jenkins 2014; Van Toorn and Dowse 2016; Zarkin 2021). While these studies have provided important empirical insights, the applied measures entail different biases and limitations. Particularly, it has proved challenging for scholars to separate the priorities and values of policy-makers from whether and how various types of evidence inform policy decisions. There is therefore a need for alternative measures to capture variations between government institutions and policy choices concerning evidentiary priorities and needs.

In response, this article studies how different types of evidence have informed policy decisions in two government institutions in Denmark: the Ministry of Employment and the Ministry of Children and Education. The article does so by examining the nature and forms of evidence reflected in active labor market policies (ALMPs) and public school policies from 2016–2021, considering the evidence available to the two ministries as well as their strategies for prioritizing different evidence types in policy development and decision-making. The article is based on a quantitative content analysis of 1159 research publications and policy decisions and makes a threefold contribution: First, the article introduces a novel analytical approach to examine patterns of how different types of evidence is reflected in subsequent policy decisions to approximate levels of evidence impact. Second, the article empirically documents policy domain variations concerning the impact of different evidence types on policy decisions. Third, the article contributes to debates about EBPM by discussing the implications of prioritizing specific evidence types in ministries, considering variations between study designs and the appropriateness of different methods for different policy needs and purposes.

Conceptualising and measuring the impact of evidence in policy-making

The idea that knowledge rather than interests should inform policy-making is longstanding. In recent decades, this idea has gained popularity via the notion of EBPM, originating from medicine and endorsed by governments and international organizations to address the complex challenges facing societies today (Andersen and Smith 2022). Young et al. (2002) define EBPM as a prescription for the increased use of evidence in policy-making as well as the methodological character of this evidence. Baron (2018, 40) further specifies how EBPM encompasses “(1) the application of rigorous research methods, particularly randomized controlled trials (RCTs), to build credible evidence about ‘what works’ to improve the human condition; and (2) the use of such evidence to focus public and private resources on effective interventions.” The EBPM notion hence promotes the reliance on causal effect evidence derived from experimental methods to guide the decisions made by policy-makers and government institutions. Yet the EBPM notion has been criticized by several scholars for overrating the extent to which evidence of causal effect can or should inform policy decisions in democratic systems, considering the versatility of policy issues and needs (Ansell and Geyer 2017; Botterill and Hindmoor 2012). In light of this, the early aspirations of “basing” policies on evidence have been heavily qualified, with many scholars now applying the more modest expression “evidence-informed policy-making” (Boaz et al. 2019; Head 2016). Interestingly, however, scholars have documented how the notion of EBPM remains a widespread discourse among policy-makers, shaping their attitudes regarding the relevance and usefulness of different types of evidence (Oliver 2022). According to Parkhurst (2017), policy-makers tend to display a bias towards favouring quantitative, experimental evidence regardless of the policy issues at stake, which might affect the policy solutions adopted.

While the EBPM literature is long on concepts about the impact of evidence on policy decisions, it is short on measurements (Cozzens and Snoek 2010; Knudsen 2018). Empirical knowledge to underpin debates about EBPM and the implications of its assumptions is therefore missing. Part of this pertains to a lack of conceptual consistency and clarity in how terms like “evidence”, “utilisation” and “impact” are understood (Blum and Pattyn 2022; Boaz and Nutley 2019; Cairney 2016). For example, a challenge is the lack of distinction between policy-makers deciding to implement a policy that has been tested in a piece of research vis-a-vis policies being informed by or based on evidence. Given the complexity of how evidence is used and influences policy decisions, there have been recent calls for innovation in methods to capture the contribution of evidence to policy-making (Christensen 2023; Jørgensen 2023; Riley et al. 2018). Notably, scholars have emphasized the importance of considering questions of time (when in the policy process to identify impact) and attribution (assuring that a particular item of evidence and not something else has produced a change in policy) (Boaz and Nutley 2019). The article deals with these considerations in the following ways.

First, the article considers “evidence” as the written product of research, comprising both quantitative and qualitative studies, but not political know-how, expert advice and professional experience. This is similar to Head’s (2008) definition of scientific (research-based) knowledge; yet the present study is not limited to academic publications. Reports from public and private research organizations and think tanks are included too. Second, the study focuses on national-level “policy decisions”. These decisions comprise various policy instruments, including laws, policy programmes, economic incentives and penalties, public resource allocations, and information campaigns (Cairney 2012; Vedung 2007). One might object that such a “decisionistic” focus neglects the alternative ways evidence can shape policy processes. There is a wide literature describing how evidence can be used instrumentally, conceptually, and symbolically at various stages of policy-making (Beyer 1997; Boswell 2009; Knott and Wildavsky 1980; Nutley, Walter, and Davies 2007). Distinguishing between different forms of utilization and impact, however, has proved methodologically challenging. According to Christensen (2021; 2023), the concepts are somewhat overlapping, and separating them may obscure the distinction between the intentions behind using different forms of evidence and their impact. The implications of the different forms of utilization for evidence impact are thus unclear.

Recognizing this, and to enable wide-scale analysis of numerous research publications and policy decisions over time, this study uses a broad conception of “evidence impact” focusing on whether the conclusions and recommendations from research publications are reflected in subsequent policy decisions. This does not equal evidence utilization per se but it enables an approximation of evidence use across a large number of policy decisions, which would be laborious and time-consuming to document qualitatively. Drawing on Landry, Lamari, and Amara (2003), the analysis seeks to capture the “influence stage” of utilization, in which evidence appears to impact policy decisions. This impact is attributed to when there is an apparent match between the conclusions and recommendations of particular research publications and subsequent policy decisions. By sampling research publications, which are accessible and relevant to the ministries under study, employing systematic procedures for identifying evidence–policy matches, and by aggregating the matches across numerous policy decisions, the results are considered good indicators of impact (Jørgensen 2023; Knudsen 2018).

Administrative priorities regarding different types of evidence

Despite the enthusiasm for EBPM among governments, existing research on the topic provides somewhat ambiguous findings as to whether, and what types of, evidence influences policy decisions. Scholars have identified a range of facilitators and barriers to the influence of evidence on policy (Christensen and Mandelkern 2022; Nutley, Walter, and Davies 2007; Oliver et al. 2014). First, it is commonly recognized that the impact of evidence on policy decisions is positively associated with administrative capacities for producing, accumulating and considering research and data in the policy process (Howlett 2009, 2015; Jennings and Hall 2012; Newman, Cherney, and Head 2017). Second, scholars have documented how the timeliness, accessibility and relevance of research are important for whether it finds its way into policy decisions (Oliver et al. 2014; Reid et al. 2017). Third, an important body of work has focused on the characteristics of the evidence itself to explain why it (does not) inform policy-making. Here, “evidence quality” is considered an important factor for impact (Oliver et al. 2014). However, scholars has also shown that different perceptions exist regarding what constitutes relevant and useful evidence for policy-making (Nutley, Powell, and Davies 2013; Parkhurst and Abeysinghe 2016).

Inspired by practices from medicine, the early proponents of EBPM advocated a “hierarchy of evidence” in which RCTs and systematic reviews were perceived as providing more credible knowledge than research derived from observational or qualitative research approaches (Borgerson 2009; Høydal and Tøge 2021). Yet as EBPM has spread to other policy domains, this perception of “useful evidence” has been broadened to reflect a wider range of policy concerns and issues (Smith and Haux 2017). In light of this development, several scholars have proposed that the relevance and usefulness of different types of evidence should be judged based on their appropriateness for specific policy needs rather than adopting a fixed causal hierarchy with experimental study designs at the top (Nutley, Powell, and Davies 2013; Parkhurst and Abeysinghe 2016; Petticrew and Roberts 2003). Instead of ranking different evidence types based on their ability to document causal effects of policy choices, evidence should be appraised based on how well it serves particular policy needs (Parkhurst et al. 2021).

Administrative positions regarding the relevance and usefulness of different evidence types can be observed and might manifest in strategies, vision statements and work plans of government institutions. As argued by Parkhurst et al. (2021, 448), institutional logics shape perceptions of the appropriate forms and applications of evidence for policy needs. In this article, these logics are referred to as “evidence strategies” (understood as the criteria employed to accept or reject available evidence), guiding what forms of evidence are consulted in policy-making processes. According to Head (2016), two dominant positions exist among those who advocate for the increased use of evidence in policy-making. The first position adheres to prescriptions from evidence-based medicine, where the relevance and usefulness of evidence is judged based on a causal hierarchy, which favors experimental designs to programme evaluation, while cross-sectional and case-based, qualitative research is given a lower priority. In such a strategy, which is mostly found in policy areas associated with the natural sciences and economics, evidence serves narrow the purpose of documenting “what works” to solve policy problems most effectively and efficiently (Nutley, Powell, and Davies 2013). This could be characterized as a “hierarchical strategy” towards evidence use. The second position is more inclusive, maintaining that the objective good is contingent upon the policy issue at stake and accepting a wider range of research designs and knowledge types to inform policy-making. In this perspective, the relevance and usefulness of different evidence types should be evaluated based on how well they contribute to better deliberations about problems and solutions, as emphasized in the programmatic perspective from Parkhurst et al. (2021). This could be labelled an “inclusive strategy”.

Implications of adopting different evidence strategies

When evidence perceptions and practices from medicine travels to other policy domains and inspires administrative practices in areas like education, employment, or social policy, they may promote particular ways of understanding and addressing policy problems. Oliver (2022:, 78) explains how EBPM prescriptions from medicine “shape policy appetites” for evidence use among policy workers, motivating an unconditional preference for causal effect evidence to address policy issues, notwithstanding the policy issue in question. Parkhurst (2017) argues that such fixed preferences for specific types of evidence can foster “issue biases”, meaning that some concerns and perspectives are systematically marginalized because of the characteristics of the evidence types favored to inform policy processes. Accordingly, calls for basing policy on specific forms of evidence risk motivating that policy decisions are based on what has been counted, not necessarily what counts (Parkhurst 2017, 54). Hence, a number of scholars have advocated for replacing fixed hierarchies with a focus on evidence appropriateness for different policy needs (Nutley, Powell, and Davies 2013; Parkhurst and Abeysinghe 2016; Petticrew and Roberts 2003). Such a view does not imply that all study designs are equal, but acknowledges that, for example, RCTs are useful to answer causal effect concerns while qualitative research is appropriate to concerns like explanation or process understanding. Government institutions should thus not apply fixed hierarchies to evaluate evidence types across all concerns.

While these arguments are reasonable, they are often normatively founded, and there is a lack of comparative empirical research documenting the priorities and criteria regarding evidence adopted by government institutions and whether policy decisions are in fact more influenced by some evidence types than others. The few studies that exist provide somewhat contradictory results on these matters: While some scholars have documented policy-maker preferences for quantitative and experimental study designs (Amara, Ouimet, and Landry 2004; Ingold and Monaghan 2016; Lingard 2011; Monaghan and Ingold 2018), others find that causal hierarchies play a minor role among policy-makers (Petticrew et al. 2004), that the utilization of different evidence types depends on the purpose of their use (Høydal and Tøge 2021), or that other factors than the type of evidence are more important determinants of evidence impact (Landry, Lamari, and Amara 2003). More research is therefore needed examining the impact of different evidence types across government institutions and policy domains.

Case selection and expectations

To study variations between government institutions regarding the impact of different evidence types, the Ministry of Employment and the Ministry of Children and Education in Denmark are compared, focusing on policy decisions relating to active labor market policies and public school policies from 2016‒2021.¹ The cases are interesting for comparison for three main reasons:

First, the two ministries share a number of similarities. The ministries are comparable (Lijphart 1971, 687) in the sense that both are characterized by high analytical capacities for accumulating and considering research and data as well as affiliated agencies and research organizations that regularly provide research-based evidence. The compilation and dissemination of research is also a distinctively marked budget heading in the annual reports of both ministries (STAR 2022; STUK 2022). The ministries are thus characterized by having close links to research producers and by having a highly skilled workforce – two factors considered conducive to evidence use in government institutions (Howlett 2009; Jennings and Hall 2012; Oliver et al. 2014; Van der Arend 2014).

Second, the employment and education domains are relatively similar concerning governance structures and policy conflict dynamics. Both policy domains operate within a framework of centralized regulation and municipal implementation of policies (Bredgaard and Rasmussen 2022; OECD 2020). The policy domains also deal with similar areas of tension in the sense that both domains are occupied with a dual task of rendering the system more effective (reducing unemployment; increasing student performance) while ensuring good conditions for the citizens in the system (social support; well-being) (Andersen 2020; BUVM 2023). This has fostered debates regarding work-first versus human capital approaches in the employment domain and discussions about central learning goals and monitoring vis-à-vis professional autonomy in education. The present study does not offer a comprehensive analysis of the extent to which different types of evidence are required by the two ministries in relation to policy priorities and goals in the studied period – a limitation which should be explored in future studies. However, both ministries deal with policy programmes and solutions in which many types of evidence could be useful for informing a single policy question, such as how to get different citizen groups into employment, how to increase student performance in schools, etc.

Third, there are significant differences between the two ministries regarding their strategy towards evidence use, i.e. the criteria employed to accept or reject available evidence. The Ministry of Employment has been following a formal evidence strategy since 2012, which is closely inspired by the EBPM notion with a focus on basing policies on causal effect evidence and a strong preference for experimental designs (STAR 2023a). By contrast, the Ministry of Children and Education is characterized by a more inclusive and less explicit strategy, appraising many forms of evidence and aiming to let the “best available evidence” inform policy decisions broadly (STUK 2023).

By comparing two ministries with similar background characteristics, but different strategies regarding the prioritization of specific evidence types, the case selection seeks to capture the role of ministerial evidence strategies for whether different types of evidence influence policy decisions. Due to variations in the ministerial strategies adopted, the article expects that decisions in the two policy domains will be influenced by different forms of evidence: Because the Ministry of Employment has been following a strategy that aims at basing policies on causal effect evidence, the article expects to observe a higher impact of causal effect evidence on policy decisions regarding active labor market polices. Contrarily, the article expects a broader prioritization of different evidence types in the Ministry of Children and Education and a less clear focus on causal effect evidence in public school policy decisions. The section below describes how these expectations were examined.

Document matching analysis

The article applies a document-matching approach to study the impact of evidence (Jørgensen 2023; Knudsen 2018). The analysis involved a sequential process of collecting documents, applying a coding scheme and analyzing the codes using different statistical tools. The analysis sought to identify matches between conclusions and recommendations from research publications and policy decisions, capturing “preference attainment” (Christensen 2023, 8) or the “influence stage” of evidence utilization (Landry, Lamari, and Amara 2003). The data collection and analysis were carried out in 2022.

Collecting documents

Research publications and policy decisions concerning ALMP and public school policy from 2015‒2021 formed the units of analysis. In early 2022, 588 research publications and 571 policy decisions were retrieved. Collecting documents required important selection criteria. To accommodate issues with timeliness, accessibility and relevancy of research (Oliver et al. 2014), the collection of research publications was based on a purposive sampling strategy (Krippendorf 2013). Nutley, Walter, and Davies (2007, 62) emphasize that policy-makers predominantly utilize knowledge from research-, government, and specialist organizations, and Van der Arend (2014, 618) has shown how policy officials generally express difficulty locating relevant evidence if not provided by acquainted organizations. The document collection therefore aimed to capture the knowledge horizon of the two ministries under study. A similar strategy was applied by Knudsen (2018), which inspired the analysis. The document collection targeted publications that were timely (i.e. issued within a six-year timeframe), on topic and issued by Danish research-producing organizations referred to on the ministry websites or appearing on public hearing lists in relation to recent employment and education reforms. The documents were thus retrieved from the ministry websites (349) or stakeholder websites (266).

The collection of policy decisions was limited to law documents and decisions issued on ministry website news feeds. Only decisions mentioning “active labor market” or “public school” were included. The study exclusively focused on national-level policy decisions made by political electives and ministries. Such policy decisions comprise different policy instruments, including public spending levels, economic incentives or penalties, laws, orders, regulations, policy programmes, and information measures (Cairney 2012; Vedung 2007). The decisions were hence identified in laws, amending acts, orders, agreements and notices about policy initiatives and programme funding. Duplicates of policy decisions were removed and technical changes in law documents (e.g. due to decisions taken by other authorities) were discarded. COVID-19 decisions enforced by the Danish Epidemic Act were also removed. If a policy document included several decisions, they were separated into discrete units. The 571 documents thus comprised all relevant national-level decisions.

The selection criteria may have implications for results. The strict delimitation of research publications possibly ignores evidentiary sources that could have informed policy decisions in the studied cases. For example, policy-makers may learn from international sources of evidence (Legrand 2012) or from interest groups (Flöthe 2020). Yet defining the boundaries for which international actors to include and identifying important actors from other countries is challenging. Second, restricted access to international journals or other knowledge outlets may limit availability for Danish policy-makers. It would have been possible to include publications from interest groups; yet not all of these publications would satisfy the definition of “evidence” employed in the study.

The selection criteria might affect the observed levels of evidence impact. Since the collection exclusively targeted research figuring within the ministries' “knowledge horizons”, the observed impact levels might be higher than in other existing studies investigating the role of evidence in policy–making. Future studies using this matching method would benefit from including more different types of knowledge to capture the role of research vis-à-vis other types of information or explore the kinds of international sources influencing national-level policy decisions Figure 1.

Figure 1. Document collection process.

Applying the coding scheme

The applied coding scheme contained three types of codes: Document attributes, thematic codes, and evidence‒policy matches (see Appendix A). The document attribute codes were used to extract descriptive information from the documents. The research publication attributes included their publication date, provider type, provider name and whether the study was commissioned by the ministry. To investigate ministerial priorities regarding the use of different types of evidence, the documents were further coded using a classification scheme of different evidence types with five categories based on study design. The scheme was inspired by existing literature (Nutley, Powell, and Davies 2013; Petticrew and Roberts 2003) but was also inductively inferred from the search for research publications (i.e. the available types of evidence in the two policy domains). The classification scheme is shown in Table 1.

Table 1. Classification of evidence types.

Evidence types	Examples
Systematic reviews	Systematic reviews, meta-analyses
RCTs/quasi-experimental studies	RCTs, pseudo-RCTs, difference-in-difference designs, regression-discontinuity designs
Observational-analytic studies	Cohort studies, time series studies, case-control studies, standard regressions, factor analysis
Observational-descriptive studies	Cross sectional studies, case series, descriptive analyses
Qualitative studies	Case studies, qualitative interviews, focus groups, field observations, document analyses

The policy decision attributes included their release date and whether the decision document included a direct reference to a research publication. Finally, the thematic codes were inductively inferred from systematically browsing the documents, noting down topics and condensing them into larger categories. Ten topics were recorded for each policy domain and validated by cross-checking descriptions on the ministry websites (BUVM 2023; STAR 2023b). The thematic codes supported the identification of matches between research publications and policy decisions.

The evidence‒policy matches were used to analyze the extent to which conclusions and recommendations from particular research publications align with subsequent policy decisions concerning the same topic (Jørgensen 2023; Knudsen 2018). The matching was based on whether (1) or not (0) a decision and a research publication addressed the same topic and target group and applied similar wording. A reference to a research publication in a decision was also coded as a match (31 instances). While direct references provide a stronger proof of impact, they are assigned the same score as textual correspondences for the sake of simplicity. Contextual factors were considered when coding matches, including whether the research provider had been consulted in previous, equivalent policy decisions and whether the research publication has been commissioned. The example below illustrates the coding process: A report from the EVA (2015) issued in 2015 showed that 25% of 8th-grade students were not ready for upper-secondary education. The report was coded as commissioned evidence, applying observational-descriptive methods, issued by a public research institution. In June 2017, the Ministry of Children and Education introduced a skills development programme for 8th-grade students not ready for upper-secondary education. The topic was coded as future education. The excerpt below derives from the programme announcement:

About one-quarter of all 8th-grade students are not ready for upper-secondary education, meaning that they have poor prerequisites for finishing public school with a result that can help them into upper-secondary education. A programme focusing on short courses for 8th-grade students who are not ready for upper-secondary education will contribute to putting matters right (BUVM 2017).

Although the research report is not mentioned directly in the programme announcement, it appeared very likely that its conclusions motivated the ministry to initiate the programme for 8th-grade students. A match between the research report and the policy decision was therefore recorded. The same process was carried out for all the 1159 research publications and policy decisions in the dataset.

Analysing the results

After coding the 1159 documents, the data was exported into R (version 4.2.2) for statistical analysis. The analysis included descriptive statistics and tests for differences between policy domains and different evidence types using Pearson χ²-tests. Linear probability models were further estimated to examine variation between policy domains and evidence types (main parameters of interest), controlling for potential confounding factors, including time, provider type, the number of methods per research publication, and whether the research publications were commissioned. Aggregate levels of evidence impact were measured as the share of research publications matching one or more subsequent policy decisions. The measure produces a value between 0 and 1, accounting for how often available and relevant research publications have been reflected in subsequent policy decisions.

Findings

First, a descriptive analysis of the evidence types available to the two ministries and whether they are reflected in subsequent policy decisions is performed. The reflection of evidence in ALMP and public school policy decisions is used as an indicator of ministry use of evidence. Second, linear probability models are used to compare impact levels between policy domains and evidence types.

Available types of evidence and how they are reflected in policy decisions

Table 2 displays descriptive information about the 588 research publications in the dataset, representing the evidence types available to the ministries regarding ALMP and public school policy. Following the document collection criteria, the research publications were considered both accessible to and relevant for the ministries under study. Table 2 shows how 246 publications on ALMP and 342 publications on public school policy have been available from 2015–2021. There is no substantial policy domain variation concerning the study designs applied in the research publications, besides a larger share of systematic reviews in the employment domain (16.7%) than in education (7.3%). The systematic reviews on active labor market measures mainly derive from the Ministry of Employment’s evidence bank “Job Effects”, which accumulate causal effect evidence about labor market measures (STAR 2023c), while most RCTs and quasi-experimental studies are published by consultancy firms, universities or the Danish Centre for Social Science Research (VIVE). VIVE has also published most of the experimental studies in the education domain. VIVE is a Danish semi-public research organization, which has played a central role in evaluating major employment and education reforms (e.g. Nielsen et al. 2020; Pedersen et al. 2019; Thuesen, Bille, and Pedersen 2017).

Table 2. Available evidence in the two policy domains

	Policy domain		Total
	ALMP	Public school policy
Evidence types
Systematic reviews	41 (16.7%)	25 (7.3%)	66 (11.2%)
RCTs/quasi-experimental studies	43 (17.5%)	63 (18.4%)	106 (18%)
Observational-analytic studies	29 (11.8%)	53 (15.5%)	82 (13.9%)
Observational-descriptive studies	89 (36.2%)	123 (36%)	212 (36.1%)
Qualitative studies	44 (17.9%)	78 (22.8%)	122 (20.7%)
Total	246 (100%)	342 (100%)	588 (100%)
Provider types
Ministerial unit	80 (32.5%)	20 (5.8%)	100 (17%)
Private consultancy firm	104 (42.3%)	39 (11.4%)	143 (24.3%)
Public research institution	37 (15%)	189 (55.3%)	226 (38.4%)
Think tank	20 (8.1%)	20 (5.8%)	40 (6.8%)
University unit	5 (2%)	74 (21.6%)	79 (13.4%)
Total	246 (100%)	342 (100%)	588 (100%)
Commissioned research
No	62 (25.2%)	148 (43.3%)	210 (35.7%)
Yes	184 (74.8%)	194 (56.7%)	378 (64.3%)
Total	246 (100%)	342 (100%)	588 (100%)

Table 2 displays how 74.8% of the research publications about ALMP have been commissioned by the Ministry of Employment. In the education domain, the Ministry of Children and Education has commissioned a slight majority of the research publications (56.7%). In general, an extensive amount of evidence has been commissioned, which indicates substantial interest in the ministries to accumulate research-based knowledge, as similarly emphasized in the existing literature on the Danish context (e.g. Christensen, Gornitzka, and Holst 2017; Rambøll 2015).

Table 3 displays the distribution of matched and unmatched research publications across policy domains and different evidence types (see the corresponding results for each policy domain in Appendices B–C). Although significant only at a 10% level (p = 0.054), the results show a noticeable difference between the distribution of matched and unmatched research publications in the two policy domains: A larger share of the available research publications matches a subsequent ALMP decision (29.3%) compared to public school policy decisions (21.9%). Looking across the two policy domains, the share of research publications matching a subsequent policy decision is 25%. While the lack of studies employing similar measures of evidence impact makes it difficult to compare the results directly to previous research, one-fourth of available research publications being reflected in a subsequent policy decision represents a substantial impact of evidence to policy. Yet it is essential to note that the dataset only contains research publications sampled “close” to the ministries and that 64.3% were commissioned. Relatively high levels of evidence impact were thus expected.

Table 3. Distribution of matched and unmatched research publications.

	Evidence-policy match		Total
	No	Yes
Policy domain
Public school policy	174 (70.7%)	72 (29.3%)	246 (100%)
ALMP	267 (78.1%)	75 (21.9%)	342 (100%)
Total	441 (75%)	147 (25%)	588 (100%)
χ^2 = 3.727 · df = 1 · p = 0.054
Evidence types
Systematic reviews	40 (60.6%)	26 (39.4%)	66 (100%)
RCTs/quasi-experimental studies	82 (77.4%)	24 (22.6%)	106 (100%)
Observational-analytic studies	56 (68.3%)	26 (31.7%)	82 (100%)
Observational-descriptive studies	164 (77.4%)	48 (22.6%)	212 (100%)
Qualitative studies	99 (81.1%)	23 (18.9%)	122 (100%)
Total	441 (75%)	147 (25%)	588 (100%)
χ^2= 12.663 · df = 4 · p = 0.013

Regarding evidence types, Table 3 shows a significant difference (p = 0.01) between the distributions of matched and unmatched research publications. This is due to a larger share of systematic reviews and observational-analytic studies with a match compared to observational-descriptive studies and qualitative studies. Hence, there is a general tendency towards prioritizing quantitative studies to inform policy decisions and to some extent also evidence of causal effect. The match shares, however, are not significantly higher for RCTs/quasi-experimental studies than for observational-analytic studies or observational-descriptive studies. Nevertheless, interesting contrasts between the two policy domains and ministries appear when examining them separately, as described below.

Impact probabilities across policy domains and evidence types

This section presents findings from linear probability models using evidence‒policy matches as the dependent variable; policy domain and evidence types as the main parameters of interest; and fixed effects for provider types, whether the research publications were commissioned, the number of methods per research publication and publication years (see Appendix D for corresponding results using logistic regression). Table 4 shows descriptive statistics about the dataset. Contrary to the tables presented in the above section, this table shows the total number of matches and non-matches. Research publications matching more than one decision thus appear as multiple observations, thereby resulting in a total of 865 observations across the two policy domains.

Table 4. Descriptive statistics for all variables.

	N	Pct.
Observations	865	100
Dependent variable
Evidence – policy match: Yes (ref. no)	355	41.0
Independent variables
ALMP (ref. public school policy)	415	48.0
Systematic reviews	151	17.5
RCTs/quasi-experimental studies	131	15.1
Observational-analytic studies	135	15.6
Observational-descriptive studies	277	32.0
Qualitative studies	171	19.8
Control variables
Provider type: University unit	94	10.9
Provider type: Ministerial unit	201	23.2
Provider type: Private consultancy firm	198	22.9
Provider type: Public research institution	320	37.0
Provider type: Think tank	52	6.0
Commissioned: Yes (ref. no)	611	70.6
Methods per publication (mean)	1.61
Publication year: 2015	111	12.8
Publication year: 2016	155	17.9
Publication year: 2017	153	17.7
Publication year: 2018	133	15.4
Publication year: 2019	125	14.5
Publication year: 2020	85	9.8
Publication year: 2021	103	11.9

Table 5 displays correlations between evidence‒policy matches, policy domains and evidence types (see the full regression table in Appendix E). Model 1 shows how research publications about ALMP have a significantly higher probability of matching with a subsequent policy decision (0.158, SE = 0.039, P < 0.001) compared to research publications relating to public school policy. Model 2 displays how systematic reviews (0.330, SE = 0.053, P < 0.001) and observational-analytic studies (0.190, SE = 0.055, P = 0.001) have significantly higher probabilities of matching with a policy decision compared to qualitative studies. The results are consistent when changing the reference category to observational-descriptive studies. However, the results do not indicate a clear-cut relationship between evidence types and match probabilities in the sense that evidence of causal effect is consistently more prioritized than other types of evidence. For example, RCTs/quasi-experimental studies do not exhibit significantly higher probabilities than other evidence types. The findings from Models 1 and 2 remain significant after including both predictor variables in the same model (Model 3).

Table 5. Linear probability models with matches as the dependent variable.

Predictors	Model 1		Model 2		Model 3
	Estimates	SE	Estimates	SE	Estimates	SE
ALMP	0.158 ***	0.039			0.100 *	0.042
Systematic reviews			0.330 ***	0.053	0.292 ***	0.056
RCTs/quasi-experimental studies			0.089	0.054	0.069	0.054
Observational-analytic studies			0.190 ***	0.055	0.185 ***	0.054
Observational-descriptive studies			0.037	0.045	0.035	0.045
Observations	865		865		865
Controls	Yes		Yes		Yes
R^2 / R^2 adjusted	0.104 / 0.091		0.136 / 0.119		0.142 / 0.125

Notes: OLS regression. Match as dependent variable. All models control for publication year, the number of methods per publication, provider types and whether the publication has been commissioned. Reference levels: Public school policy; Qualitative studies. * p < 0.05 ** p < 0.01 *** p < 0.001.

Overall, the results elucidate how the reflection of research publications in policy decisions has been higher for ALMP decisions than for public school policy decisions, and that systematic reviews and observational-analytic studies, respectively, are the most prioritized evidence types to inform policy decisions in the two ministries, although they do not constitute the largest share of available evidence in the studied policy domains. This indicates a stronger orientation towards evidence broadly in the Ministry of Employment than in the Ministry of Children and Education. To further study variations between the two ministries, interactions between policy domains and evidence types were added to the regression model, as displayed in Table 6 (see the full regression table in Appendix F).

Table 6. Linear probability model with interactions.

Predictors	Match model
	Estimates	SE
ALMP	0.152	0.094
Systematic reviews	−0.011	0.062
RCTs/quasi-experimental studies	0.240 ***	0.066
Observational-analytic studies	0.146 **	0.055
Observational-descriptive studies	0.190 **	0.072
Interaction: ALMP*Systematic reviews	0.151	0.121
Interaction: ALMP*RCTs/quasi-experimental studies	0.125	0.105
Interaction: ALMP*Observational-analytic studies	−0.184	0.113
Interaction: ALMP*Observational-descriptive studies	−0.298 ***	0.089
Observations	865
Controls	Yes
R^2 / R^2 adjusted	0.169 / 0.148

Notes: OLS regression. Match as the dependent variable. Controls for publication year, the number of methods per publication, provider types, and whether the publication has been commissioned. Reference levels: Public school policy; Qualitative studies; Interaction: ALMP*Qualitative studies. * p < 0.05 ** p < 0.01 *** p < 0.001.

Based on the interaction model above, Figure 2 displays the marginal match probabilities across policy domains and evidence types (see Appendix G for documentation). The figure illustrates how systematic reviews (0.71) and RCTs/quasi-experimental studies (0.53) have substantially higher match probabilities for ALMP decisions compared to public school policy decisions (0.35 and 0.19). The marginal probabilities for observational-analytic studies and observational-descriptive studies are similar in the two domains, while qualitative studies have higher match probabilities (0.44) in ALMP decisions than in public school policy decisions (0.12). These results confirm the expectation presented earlier; that evidence prioritization in the Ministry of Employment is more oriented towards evidence of causal effect compared to the Ministry of Children and Education. The formal evidence strategy in the Ministry of Employment, which is inspired by the EBPM notion and strongly values experimental designs for measuring causal effects, seems to have greatly influenced the prioritization of evidence to inform ALMPs in the studied period. Contrarily, the impact of different types of evidence in the Ministry of Children and Education seems less determined by strict methodological priorities towards evidence of causal effect, thus reflecting a more inclusive position towards different evidence types. However, the results do not indicate that the more inclusive strategy in the Ministry of Children and Education has fostered a stronger reliance on observational-descriptive or qualitative studies to inform policy decisions compared to the Ministry of Employment.

Figure 2. Match probabilities for policy domains and evidence types.

Discussion and conclusion

According to EBPM proponents, better and more effective policies are achievable if policy decisions are based on evidence of causal effect. This has inspired several governments to promote the reliance on such evidence in public policy-making. Yet scholars have repeatedly documented that basing policies on evidence is more complicated than assumed by the EBPM notion, rendering the influence of evidence highly uneven across policy domains and government institutions (Amara, Ouimet, and Landry 2004; Botterill and Hindmoor 2012; Greenberg, Michalopoulos, and Robins 2006; Jennings and Hall 2012; Landry, Lamari, and Amara 2003). Moreover, scholars have criticized the strong prioritization of causal effect evidence implied in the EBPM notion, arguing that democratic and responsible evidence use should draw on different research methods based on their appropriateness for policy needs rather than an unconditional preference for causal effect evidence, marginalizing other forms of evidence from consideration (Davies and Nutley 2002; Parkhurst 2017; Parkhurst and Abeysinghe 2016; Parsons 2002). However, empirical research underpinning these debates is limited. Systematic examinations of the impact of different types of evidence across policy domains and policy decisions are rare in the literature.

The present article has provided empirical knowledge to these debates by examining how different types of evidence are reflected in policy decisions in two Danish ministries through the analysis of 1159 research publications and policy decisions from 2015–2021. A document matching method was developed to enable a wide-scale analysis of evidence impact patterns, responding to biases and limitations in existing approaches to study evidence impact (Christensen 2023). The analysis results indicate how evidence is a central asset for the Ministry of Employment and the Ministry of Children and Education: A large amount of research publications related to ALMP (N = 246) and public school policy (N = 342) has been available to the ministries from 2015–2021, while 64.3% of this research have been commissioned by the ministries. The broad availability of research reflects the characterization of Denmark as a leading knowledge economy, as suggested by Christensen, Gornitzka, and Holst (2017). The results reveal how 25% of the 588 research publications in the dataset are reflected in a subsequent decision, and the share is even higher for the Ministry of Employment (29.3%). While the results are not directly comparable to existing studies, which mainly rely on survey and interview methods (Christensen 2023), one-quarter of research finding its way into policy decisions represents a substantial contribution of evidence to policy. A likely explanation is that the studied ministries have high analytical capacities, including skilled workforce, vast resources for research activities, as well as affiliated agencies and research organizations that regularly provide research and data.

Comparing two similar cases, the article expected to observe variations between the policy domains concerning the prioritization of different evidence types in policy decisions, shaped by differences in ministerial evidence strategies. The article expected that the evidence strategy adopted by the Ministry of Employment in 2012, which closely adheres to the EBPM notion, would motivate a stronger prioritization of causal effect evidence in policy decisions, while the more inclusive approach in the Ministry of Children and Education would place more equal weight on different evidence types. The results somewhat confirm these expectations. Marginal match probabilities across policy domains and evidence types indicate that causal effect evidence is more strongly prioritized in the Ministry of Employment compared to the Ministry of Children and Education; the match probabilities for systematic reviews and RCTs/quasi-experimental studies are significantly higher in ALMP decisions than in public school policy decisions. As similarly observed by Andersen (2020), the adoption of an evidence strategy by the Ministry of Employment has motivated a strong orientation towards research designs that document the causal effects of policies. The results further confirm that the reflection of evidence in policy decisions in the Ministry of Children and Education is less systematically oriented towards the experimental study designs; yet the inclusive approach has not led to significantly higher match probabilities for other evidence types compared to the Ministry of Employment. Inspired by a study of evidence use and methodological preferences among Norwegian policy workers conducted by Høydal and Tøge (2021), the reason might be that, for example, descriptive and qualitative studies are more often used for enlightenment than for guiding policy decisions directly.

The findings of the present study are informative for discussions about EBPM and evidential priorities in government institutions. For example, the evidence strategy adopted by the Ministry of Employment appears to have fostered a strong reliance on causal effect evidence in ALMP decisions. As Parkhurst (2017) has shown in relation to other policy contexts, this might be problematic, since an uncritical favouring of such evidence can generate issue bias, marginalizing certain perspectives and solutions from consideration during policy development and policy discussions. Accordingly, strong administrative preferences for causal effect evidence might privilege scientific ideals (e.g. laws, causality, and generalisability) at the expense of professional and experiential knowledge about meaning, process understanding, etc., which is essential to understand and address the complexity of many policy issues. For example, activation measures in the unemployment system can have different consequences for socio-economically advantaged and disadvantaged citizens (Dahler-Larsen 2017). As Sayer (2020:, 250) argues, focusing exclusively on the causal effects of policies may distract policy-makers from more complicated questions regarding public problems and solutions.

In the case of Danish ALMPs, an uncritical orientation towards causal effects may imply that decisions focusing on human capital development (skills enhancement, education) are systematically marginalized, as they have potential retainment effects and do not immediately decrease unemployment levels. By contrast, work-first approaches (rules, sanctions, benefit reductions) are ceteris paribus promoted, as they incentivise people to leave the system and thus reduce the number of people on unemployment benefits. While the latter might increase overall employment figures, the former is essential to support socially disadvantaged people in finding employment. Hence, a recent research report has shown how, despite declining unemployment rates in Denmark, economic costs in job centres have remained high as evidence-based activation measures have been ineffective in moving socially vulnerable and long-term unemployed citizens into employment (Amilon et al. 2022).

In Danish public school policy, the prioritization of evidence of causal effect appears less substantial. A reason might be that policy and practice development in the education domain has historically been strongly influenced by professional knowledge and autonomy. However, the results also show that observational-analytic studies appear highly prioritized in public school policy decisions, while recent ministry initiatives indicate a growing interest in causal effect measurements. For example, the Ministry of Children and Education has recently introduced a management tool, which uses causal effect evidence to underpin public budgeting decisions and compare the economic effects of policy options (REFUD 2023). Another recent initiative in the Ministry of Children and Education have sought to strengthen quantitative education research (EduQuant 2023). The implications of this increased prioritization of casual effect evidence are important. Krejsler (2013) explains how EBPM, which has mainly been encouraged by governments and international agencies, resonates poorly with existing discourses among professionals in the education field. An increasing focus on quantitative measures of causal effect might clash with professional views on what constitutes quality in education. As teachers in Denmark are well-organized and their support is decisive for the implementation of policies, the Ministry of Children and Education might not be able to adopt an evidence strategy equivalent to the Ministry of Employment, as it could trigger substantial contestation. Instead, a more inclusive approach oriented towards different types of evidence has so far been adopted.

In summary, the article has provided new empirical knowledge on the impact of different types of evidence in government ministries. Developing and testing novel ways of measuring how different evidence types are reflected in policy decisions and documenting administrative priorities for different study designs is important for underpinning scholarly debates about methodological preferences and appropriate evidence for policy-making. The findings are further enlightening for policy-makers, as they document the extent to which policy decisions reflect available evidence. Since 64.3% of the research publications analyzed in the study have been commissioned by the ministries, it is critical to know whether these research publications actually find their way into policies.

The study has some limitations. First, the comparison of research publications and policy decisions only provides an approximation of whether different types of evidence are used in and impacts policy as well as a simplified account of policy processes. This allowed for the analysis of a very high number of policy decisions to produce generalizable results about the impact of evidence in the two policy domains. However, the study provides limited information regarding how the research publications were interpreted by policy-makers, the function they played in the policy process or why some research publications were not used. Combining the results with qualitative interviews and surveys could offer insights into the reasons for relying on different sources of evidence (Cherney et al. 2015). Second, the study only considered research produced by actors “close” to the ministries, while other studies have documented a broader range of evidentiary sources used by policy-makers (Head 2008; Jennings and Hall 2012). For instance, it could have been relevant to include knowledge from interest organizations and trade unions, which play a key role in Danish politics. International evidence, such as research publications from neighbouring countries or international organizations, could equally be interesting. Future studies could also focus on examining other policy arenas, such as how different evidence types influences parliamentary debates, political communication or policy implementation processes. Finally, future studies should go more into exploring the features of the issues involved in different policy cases to evaluate the relative appropriateness of different evidence types for different policy needs. This would imply considering what is the “right” proportion of different evidence types vis-à-vis the nature and balance of decisions made by ministries. Such studies are important to advance the discussion about EBPM and evidence priorities in government institutions.

Acknowledgements

I would like to acknowledge Associate Professor Jesper Dahl Kelstrup, Associate Professor Kim Sass Mikkelsen, and Professor Catherine Lyall for their constructive feedback as well as Student Assistant Ida Marie Nyland Jensen for her contribution to the coding of documents.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Independent Research Fund Denmark (DFF). Grant number: 0133-00183B. DFF has funded the project “Evidence-based policy-making in ministries. Explaining the effects of evidence standards on knowledge utilization”, to which this article contributes.

Notes on contributors

Jonas Videbæk Jørgensen

Jonas Videbæk Jørgensen has a Ph.D. degree from the Department of Social Sciences and Business, Roskilde University, Denmark. He has a master’s degree in Sociology from the University of Copenhagen. His Ph.D. project investigates how ministries in Denmark accumulate, consider and utilise evidence in relation to employment and education policies.

Notes

1 The timeframe is expanded to include 2015 for research publications to capture research possibly influencing policy decisions and debates from 2016 onwards.

Appendices

Appendix A. Coding scheme

Table

Research publication attributes
Provider name
Provider type	- Private consultancy firm - Public research institution - University unit	- Ministerial unit - Think tank
Publication month	MMYY
Publication format	- Research report - Research note - Journal article	- Evaluation - Other
Research method	- Systematic review - RCT - Quasi-experimental method - Observational-analytic studies - Observational-descriptive studies	- Interview - Field study - Document analysis - Desk research - Other
Evidence types	- Systematic reviews - RCTs/Quasi-experimental studies - Observational-analytic studies	- Observational-descriptive studies - Qualitative studies
Number of methods	No. of methods per research publications
Commissioned	- Yes	- No
Policy decision attributes
Decision date	DDMMYY
Policy instrument	- Regulation - Economic means	- Information
Type of decision	- Law - Amending act - Order - Political agreement	- Campaign - Project funds - Pilot programme - Other
Thematic categories
Public school policy	- Exams and tests - Student well-being - Future education - Disadvantaged students - School administration	- Subjects, classes, and hours - Learning - School staff - COVID-19 - Other initiatives
Active labour market policy	- Interviews - Business-oriented programs - Courses and projects - Training and education - Sanctions, rules, and benefits - Organisation of ALMP	- Sickness and reduced working capacity - Vulnerable and disabled citizens - COVID-19 - Other initiatives
Evidence-policy matches
Match	- 1 (Yes)	- 0 (No)
Direct reference	- 1 (Yes)	- 0 (No)

Appendix B. Ministry of Employment: Matched and unmatched research publications

Table

	Evidence-policy match
	No	Yes	Total
Evidence types
Systematic reviews	22 (53.7%)	19 (46.3%)	41 (100%)
RCTs/Quasi-experimental studies	27 (62.8%)	16 (37.2%)	43 (100%)
Observational-analytic studies	19 (65.5%)	10 (34.5%)	29 (100%)
Observational-descriptive studies	73 (82%)	16 (18%)	89 (100%)
Qualitative studies	33 (75%)	11 (25%)	44 (100%)
Total	174 (70.7%)	72 (29.3%)	246 (100%)

χ2 = 13.331 · df = 4 · p = 0.010.

Appendix C. Ministry of Children and Education: Matched and unmatched research publications

Table

	Evidence-policy match		Total
	No	Yes
Evidence types
Systematic reviews	23 (71.9%)	9 (28.1%)	32 (100%)
RCTs/Quasi-experimental studies	55 (87.3%)	8 (12.7%)	63 (100%)
Observational-analytic studies	37 (69.8%)	16 (30.2%)	53 (100%)
Observational-descriptive studies	87 (74.4%)	30 (25.6%)	117 (100%)
Qualitative studies	65 (84.4%)	12 (15.6%)	77 (100%)
Total	267 (78.1%)	75 (21.9%)	342 (100%)

χ2 = 8.717 · df = 4 · p = 0.069.

Appendix D. Regression results from the logistic model

Table

Predictors	Logistic model
	Odds ratios	SE
Intercept	0.361 **	0.140
ALMP	1.579 *	0.304
Systematic reviews	3.852 ***	1.056
RCTs/Quasi-experimental studies	1.399	0.385
Observational-analytic studies	2.334 **	0.615
Observational-descriptive studies	1.161	0.267
Provider: Ministerial unit	1.823	0.607
Provider: Private consultancy firm	1.217	0.422
Provider: Public research institution	1.200	0.350
Provider: Think tank	0.892	0.388
Number of methods per publication	0.888	0.090
Commissioned: Yes	1.806 **	0.359
Publication year: 2016	1.016	0.273
Publication year: 2017	0.762	0.204
Publication year: 2018	0.679	0.189
Publication year: 2019	0.363 ***	0.106
Publication year: 2020	0.363 **	0.124
Publication year: 2021	0.278 ***	0.087
Observations	865
R^2 Tjur	0.142

Notes: Logistic regression model. Match as dependent variable. Reference levels: Qualitative studies; Provider: University unit; Policy area: Public school policy; Commissioned: No; Publication year: 2015. * p < 0.05 ** p < 0.01 *** p < 0.001.

Appendix E. Full OLS regression table for Model 3.

Table

Predictors	Model 3
	Estimates	SE
Intercept	0.291 ***	0.079
ALMP	0.100 *	0.042
Systematic reviews	0.292 ***	0.056
RCTs/Quasi-experimental studies	0.069	0.054
Observational-analytic studies	0.185 ***	0.054
Observational-descriptive studies	0.035	0.045
Provider: Ministerial unit	0.126	0.068
Provider: Private consultancy firm	0.034	0.073
Provider: Public research institution	0.037	0.056
Provider: Think tank	−0.022	0.082
Number of methods per publication	−0.026	0.022
Commissioned: Yes	0.119 **	0.039
Publication year: 2016	−0.001	0.062
Publication year: 2017	−0.064	0.063
Publication year: 2018	−0.091	0.063
Publication year: 2019	−0.218 ***	0.064
Publication year: 2020	−0.211 **	0.068
Publication year: 2021	−0.273 ***	0.061
Observations	865
R^2 / R^2 adjusted	0.142 / 0.125

Notes: OLS regression model. Match as dependent variable. Reference levels: Public school policy; Qualitative studies; Provider: University unit; Commissioned: No; Publication year: 2015. * p < 0.05 ** p < 0.01 *** p < 0.001.

Appendix F. Full OLS regression table with interactions

Table

Predictors	Match model
	Estimates	SE
Intercept	0.304 ***	0.083
ALMP	0.152	0.094
Systematic reviews	−0.011	0.062
RCTs/Quasi-experimental studies	0.240 ***	0.066
Observational-analytic studies	0.146 **	0.055
Observational-descriptive studies	0.190 **	0.072
Provider: Ministerial unit	0.059	0.068
Provider: Private consultancy firm	0.000	0.072
Provider: Public research institution	−0.020	0.056
Provider: Think tank	−0.077	0.078
Number of methods per publication	−0.023	0.022
Commissioned: Yes	0.154 ***	0.038
Publication year: 2016	−0.046	0.064
Publication year: 2017	−0.095	0.063
Publication year: 2018	−0.124 *	0.063
Publication year: 2019	−0.231 ***	0.064
Publication year: 2020	−0.230 ***	0.066
Publication year: 2021	−0.276 ***	0.061
Interaction: ALMP*Systematic reviews	0.151	0.121
Interaction: ALMP*RCTs/Quasi-experimental studies	0.125	0.105
Interaction: ALMP*Observational-analytic studies	−0.184	0.113
Interaction: ALMP*Observational-descriptive studies	−0.298 ***	0.089
Observations	865
R^2 / R^2 adjusted	0.142 / 0.125

Notes: OLS regression model. Match as dependent variable. Reference levels: Public school policy; Qualitative studies; Provider: University unit; Commissioned: No; Publication year: 2015.

* p < 0.05 ** p < 0.01 *** p < 0.001.

Appendix G. Marginal match probabilities for evidence types

Table

Evidence types	Policy domain	Marginal mean	SE	Lower conf.	Upper conf.
Systematic reviews	ALMP	0.710	0.061	0.590	0.831
	Public school policy	0.348	0.088	0.176	0.520
RCTs/Quasi-experimental studies	ALMP	0.531	0.064	0.406	0.657
	Public school policy	0.185	0.061	0.065	0.306
Observational-analytic studies	ALMP	0.481	0.077	0.330	0.633
	Public school policy	0.441	0.052	0.338	0.543
Observational-descriptive studies	ALMP	0.259	0.056	0.150	0.369
	Public school policy	0.337	0.041	0.257	0.418
Qualitative studies	ALMP	0.436	0.061	0.316	0.556
	Public school policy	0.124	0.051	0.024	0.224