A realistic approach to policy formulation: the adapted EMMIE framework

ABSTRACT

Policy formulation is a crucial stage of the policy cycle, where social problems and demands are addressed, and transformed into government policies. This stage is complex and is one of the least analytically developed stages of the policy making process. In this article, we propose an adaptation of the EMMIE framework (created to review and rate the quality of evidence on crime reduction initiatives) as a practical means of encouraging an evidence based, systematic way of formulating policies. We argue that the five components of EMMIE (i.e. Effect, Mechanisms, Moderators, Implementation and Economics) provide useful dimensions that policy makers can apply to understand, plan and formulate successful policies. We suggest the application of the adapted EMMIE framework can improve policy formulation and in turn increase the likelihood of effective policy implementation and evaluation.

KEYWORDS:

• Public policy
• policy cycle
• EMMIE framework
• realistic evaluation
• crime

Introduction

Policy making is a core activity of any government. Despite improvements made to policy development by decades of research, civil servants, politicians and academics continue to express concerns about the way policy is made (Hallsworth and Rutter 2011; Hoornbeek and Peters 2017). The policy cycle consists of a process that conventionally includes five stages: agenda setting, policy formulation, adoption, implementation and evaluation (Howlett, & Ramesh 2003). Policy formulation is of vital importance to the success of the policy, yet it is one of the least analytically developed stages of the policy cycle process. Drawing from crime prevention policy, we propose and adapt the EMMIE framework (Johnson, Tilley, and Bowers 2015) as a realistic tool that seeks to structure formulation of policies of all types and to inform the adoption, implementation and evaluation stages of the policy cycle.

The realist approach is a form of theory-based evaluation that is used to better understand complex social interventions and programmes (Pawson 2002, 2004; Hewitt, Sims, and Harris 2012). Realistic evaluation aims to strengthen the explanatory power of evaluation studies and improve evidence-based policy making (Tilley 2016). The EMMIE framework was originally developed to summarize the evidence from systematic reviews of specific crime reduction policy initiatives (e.g. the evidence for neighbourhood watch initiatives in crime reduction) and to rate the quality of that evidence in a manner useful to policy makers and practitioners (Johnson, Tilley, and Bowers 2015). Subsequent work has proposed that this framework also has relevance for examining the formulation of policy more widely, which when implemented can be subsequently evaluated (Thornton et al. 2019). In this article, we develop this line of thought.

The EMMIE framework involves an assessment of Effect, Mechanisms, Moderators (or contexts), Implementation and Economic costs of a policy. Although developed in relation to systematic reviews of crime reduction programmes, here we adapt the EMMIE framework to be used as a systematic guide for initial policy formulation. The EMMIE framework helps to identify what policy makers should consider when designing policies in an evidence-based manner (Johnson, Tilley, and Bowers 2015), including how and why certain policies might work, under which circumstances and at what cost (Tilley 2016). We hypothesize that the use of the EMMIE framework can lead to a better understanding of how to design evidence-based policies in many areas of policy making, which in turn can facilitate implementation, evaluation and the accumulation of good evidence to inform future policies. Since the background of the authors is crime science and criminology, and the origins of EMMIE relate to crime prevention policy, several examples of security policies are used in the article to illustrate the adapted application of the EMMIE framework for policy formulation. We also include examples that relate to other policy areas to illustrate the wider appeal of the EMMIE framework for policy formulation.

The structure of the article is as follows: First, we review the concept of realistic evaluation and how it was used to inform the design of the EMMIE framework. We then review the policy cycle process and in particular the policy formulation stage. Finally, we present a revised version of the EMMIE framework, and discuss its use in providing a systematic basis for policy formulation and improving policy making. Conclusions are provided in the final section.

Realistic evaluation

Realistic evaluation was developed by Pawson and Tilley (1997) inter alia to overcome the shortcomings of meta-analyses and narrative reviews, and offer more and better information for policy makers. A realistic evaluation offers a generic approach to evaluating complex social interventions (Pawson et al. 2004). According to Tilley (2016), it is a “logic of inquiry” that generates distinctive research strategies and designs and offers researchers a logic through which to evaluate interventions and programmes. Instead of using the conventional method of programme evaluation that aims to determine if a programme works or not by describing the size of a programme’s effect (in most cases through controlled trials or systematic reviews), the realistic approach works on the premise that programmes only work for certain people in certain circumstances (Hewitt, Sims, and Harris 2012; Pawson and Tilley 2004). The central task of a realist evaluation is to explain causation and identify patterns through the interaction of different agents (James 2021). Simply put, realistic evaluation seeks to understand and explain patterns of success and failure (Pawson et al. 2004) and resolve unexplained variations in programme effectiveness in different places and at different times (Clark et al. 2005).

Although traditional methods of evaluation focus on measuring and reporting on programme effectiveness, the realist approach considers other features beyond simple impact estimates. Realistic evaluation does not generate a “pass/fail” verdict of entire families of interventions (Pawson et al. 2004). Rather, it involves holistically evaluating programme design and implementation, and prioritizing a search for underlying mechanisms that generate outcomes in particular contexts (Astbury 2013). Realistic evaluation pragmatically draws on whichever methods are best suited to unpack the “black box” of a complex intervention to develop programme theory and generate lessons that can be applied in new policy making (Hewitt, Sims, and Harris 2012). This type of evaluation flexibly uses qualitative or quantitative methods or both, and has been used to evaluate single case studies, large scale interventions and whole system changes (Greenhalgh et al. 2009). The realist approach is relatively new, and to date has been most used for evaluating health programmes (Pommier, Guével, and Jourdan 2010; Bonell et al. 2012; Marchal, Dedzo, and Kegels 2010) and in the field of criminology (Hewitt, Sims, and Harris 2012) to explain why crime has declined in recent years in western countries (Farrell, Tseloni, and Tilley 2016) and to evaluate public safety interventions, such as a media campaign implemented in London to tackle underreporting of unwanted sexual behaviour on public transport (Solymosi, Cella, and Newton 2018).

Realist studies explicitly focus on the causal ‘mechanisms’ through which interventions bring about their effects, the ‘contexts’ or conditions needed for treatments to activate potential causal mechanisms, and the ‘outcomes’ realized by the activation of causal mechanisms in the conditions in which they are introduced. (Johnson, Tilley, and Bowers 2015, 461)

This is summarized as a C–M-O configuration, where C represents context, M represents mechanisms and O represents outcomes. Realist evaluation thus develops and tests CMO configuration conjectures empirically (Sullivan 2009) by providing a structure to think “outside the box” (Pawson and Tilley 1997).

The end goal of the realistic approach is to produce a tested theory about what works for whom in what circumstances and in what respects, concentrating on identifying and diagnosing failures of programme design and programme implementation (Tilley 2016). This approach is particularly useful for policy makers in some developing countries, where the quality of public institutions varies extensively from country to country, affecting programme design and implementation (Chainey, Croci, and Rodriguez Forero 2021). The role of government institutions is of particular relevance (in all settings, rather than just developing countries) not only for overall policy design but particularly concerning the implementation and the contexts where policies are applied. Globally, the quality of institutional capacity and competence varies, which in policy implementation terms means that policies that might be successful in one setting may not be successful in others because of the manner in which the authorities implemented the policy and the extent to which the policy theory is applicable to different settings. In short, the aim of our study was to better understand which specific characteristics are considered during policy formulation and propose a systematic method of policy development.

The EMMIE framework

The EMMIE framework, first proposed by Johnson, Tilley, and Bowers (2015), is an evaluation tool inspired by the realistic approach to summarize the evidence presented in systematic reviews and to rate the quality of that evidence. Initially developed with a focus towards crime reduction, the central purpose of EMMIE is to act as a framework to effectively communicate research evidence to those that design policies (Thornton et al. 2019). Although originally developed for evaluating the content of systematic reviews, the EMMIE framework is also suggested (with pertinent adaptation) as a tool that can be applied to primary studies (Thornton et al. 2019).

EMMIE was devised to encapsulate the types of evidence needed to inform decision-making concerning the design and implementation of crime prevention programmes (Sidebottom et al. 2018). Importantly, part of the motivation for proposing EMMIE was to highlight that knowledge of “what works” (or has been found to work) is likely to be insufficient to achieve the sort of evidence-informed decision-making that policy makers need. The EMMIE framework teases out the CMO configuration by articulating the Effects (E), the Mechanisms (M), the Moderators (M), the Implementation (I) and the Economics (E) of an intervention.

Effect refers to the effect size of a policy or intervention and the confidence that should be placed on this estimate. Effects comprise the intended and unintended consequences of programmes that result from the activation of different mechanisms in different contexts. This component identifies the winners and losers amongst subjects, and pros and cons in programme delivery. Mechanisms explain the logic of the intervention and the theoretical principles on which the programme is based so that it is possible to articulate why a programme might be successful. Mechanisms focus on explaining the underlying processes that describe how the intervention produces its effects (Pawson 2002). Moderators (sometimes also referred to as contexts) are the conditions that need to be in place for a policy or intervention to activate the mechanisms necessary to produce the intended effects and are relevant to understanding why policies work in specific circumstances and/or for specific people. Importantly, moderators vary within programmes, meaning that different mechanisms will be triggered at different times or for different programme participants, creating different outcomes. Therefore, the realist researcher seeks out all the programme’s outcomes, both anticipated and unexpected (Hewitt, Sims, and Harris 2012). Decision-makers need to know how a policy can be put in place and what may facilitate or impede its implementation (Laycock and Tilley 1995). A review of implementation includes consideration of whether the programme was implemented as intended and what obstacles may have inhibited proper implementation. Fundamentally, it is not possible to test the effectiveness and evaluate an intervention if the intervention failed to be implemented as intended (Summerfelt 2003). The final part of EMMIE refers to the economics or the costs of the intervention. Resources are always limited, so policymakers need to determine how best to disburse those available to them. Ideally, policymakers will also try to anticipate unintended and indirect costs of a programme. We examine these parts of EMMIE in more detail in a later section.

In addition to summarizing the evidence in relation to each of the EMMIE dimensions in any particular systematic review, Johnson, Tilley, and Bowers (2015) created the EMMIE-Q scoring table, which assesses the strength of the evidence – the extent to which the results might be considered valid. Regarding systematic reviews, the first consideration is the extent to which the reviews themselves were appropriately and systematically identified. Examples that illustrate the use of the EMMIE-Q scoring process have, to date, related to crime reduction interventions, such as scoring the evidence on whether alley gates are effective at reducing crime (Sidebottom et al. 2018) and the effectiveness of security tags in reducing thefts from shops (Sidebottom and Tilley 2018). Table 1, which is taken from the Johnson, Tilley, and Bowers (2015), sets out the scoring framework for each component of EMMIE. In the context of the current article, we are concerned with how policymakers can consider all components of EMMIE in formulating their policies, and scoring the quality of their policies using this framework. We suggest a revision of Table 1 later in this article, but first, we review the policy cycle and its formulation.

Table 1. EMMIE evidence and five-point scales for assessing quality on each dimension.

EMMIE component	EMMIE-E (evidence itself)	EMMIE-Q scoring
Effect	Effect size	0. Insufficient consideration of validity elements* listed below:
	Moderator analysis
		1: Sufficient consideration of one element of validity
		2: Sufficient consideration of two elements of validity
		3: Sufficient consideration of three or four elements of validity
		4: Sufficient consideration of five or six elements of validity (including all of those marked with an “*”)
	Measurement/consideration of unanticipated effects
		Elements of Validity*:
		A transparent and well-designed search strategy*
		High statistical conclusion validity (at least four of the following are necessary for a study to be considered sufficient)*
		Sufficient assessment of the risk of bias (at least two necessary for sufficient consideration)*
		Attention to the validity of the constructs, with only comparable outcomes combined and/or exploration of the implications of combining outcome constructs*
		Assessment of the influence of study design (e.g. separate overall effect sizes for experimental and quasi-experimental design)
		Assessment of the influence of unanticipated outcomes or spin-offs on the size of the effect (e.g. quantification of displacement or diffusion of benefit)
Mechanisms/mediators	Map of possible mechanisms/logic maps	0. No reference to theory; simple black box
	A priori mediator or mechanism-based moderator analysis
		1: Broad statement of assumed programme theory stated (mechanisms and/or processes)
	Post hoc mediator or mechanism-based moderator analysis
	Assessment/statements of most likely mechanisms and any contextual conditions (these can be narratives)	2: Detailed articulation of theory, based on interrogation of relevant literature and/or elicited from practice.
		3: Formalization of theory and derivation of precise predictions from it
		4: Test, corroboration, falsification and refinement of theories, using data assembled for the purpose.
Moderators/contexts	A priori context-based moderator analysis/subgroup analysis (analysis testing the differences that context makes to outcome; theoretically driven)	0: No reference to condition contexts or moderators that may be significant for activation of mediators or mechanisms
		1: Ad hoc description of possible relevant moderators or contexts
	Post hoc context-based moderator analysis/subgroup analysis (analysis testing the difference context makes to outcome; conducted due to data availability/not theoretically driven/ not mentioned prior to analysis)
		2: Tests of the effects of moderators or mechanisms defined post hoc using variables that are at hand
		3: Theory-based pre-specification of expected moderators and mediators relevant to the activation of mediators or mechanisms
	Statements qualifying contextual variations (these can be narratives)
		4. Collection and analysis of relevant data relating to the pre-specified expected moderators and contexts.
Implementation	A list/statement of key components necessary for implementation of reviewed interventions	0: No account of implementation or implementation challenges
		1: Ad hoc comments on implementation
	A list/statement of key components deemed necessary for replication elsewhere	2: Systematic efforts to document implementation issues
		3: Detailed evidence-based account of expected levels of fidelity to programme, policy or treatment plans
		4: Complete evidence-based account of expected levels of fidelity to programme, expected obstacles and specification of elements necessary for replication elsewhere
Economic costs	Quantification of inputs to the intervention	0: No mention of costs (and/or benefits)
		1: Only direct or explicit costs (and/or benefits) estimated
	Quantification intervention outputs
		2: Direct or explicit and indirect and implicit costs (and/or benefits) estimated
	Quantification of intensity (e.g. spend per head)
		3: Marginal or total or opportunity costs (and/or benefits) estimated
	Estimate of cost of implementation
		4: Marginal or total or opportunity costs (and/or benefits) by bearer (or recipient) estimated
	Estimate of cost of implementation by subgroup
	Estimate of cost-effectiveness per unit output
	Estimate of cost-effectiveness by subgroup
	Estimate of cost–benefit
	Estimate of cost–benefit by subgroup

Source: Johnson, Tilley, and Bowers (2015) Table 1, page 465.

The policy cycle and policy formulation

Policies are commonly understood as actions that contain goals and the means to achieve them, however well or poorly identified, justified and formulated (Howlett and Cashore 2014). The primary agent of public policy making is the government, rather than the private sector, non-profit organizations, interest groups or other social groups. Different government institutions and public authorities have to choose between different sets of tools, goals, sectors and policies over others, making policy formulation and design particularly problematic (Béland 2007).

In turn, the policy cycle, originally proposed by Lasswell (1956), is a process that explains how policy should be drafted, implemented and assessed. Researchers have developed different models of the policy cycle; however, five stages are commonly identified: Agenda setting; policy formulation; adoption, implementation and; evaluation (Howlett & Ramesh 2003; Capano and Pritoni 2020). The crucial step of agenda-setting is the movement of an issue from its recognition to the formal political agenda (Jann and Wegrich 2007). As such, agenda setting is the process through which a policy and the problem it is intended to address evolve to become a “public problem” to be somehow dealt with, thereby becoming the focus of debate and potential controversy in the media and in politics (Savard and Banville 2012). Policy formulation is the stage at which the public administration examines and assesses the various policy options that might solve a presenting problem. Adoption is the stage at which a particular policy is agreed in principle at the government level, with implementation following. At the evaluation stage the policy is evaluated to determine whether its implementation and effects are aligned with the objectives that were explicitly or implicitly set out (Benoit 2013). Evaluation is not restricted to a particular stage of the policy cycle and may be applied at different stages of the process to determine the expected or unexpected consequences of the policy. The evaluation stage, as the final stage of the policy cycle, marks the point at which the policy is either extended, terminated or redesigned. Even though all five stages are important, the literature considers agenda-setting, policy formulation and implementation as crucial (Savard and Banville 2012).

Several researchers have, however, criticized the policy cycle model on several grounds. For example, the model’s linearity gives the impression that each stage in the cycle occurs in a precise manner, one stage after the other. Empirical reality does not fit with this policy cycle classification; stages do not evolve in clear-cut sequences and instead are constantly interconnected in an ongoing process (Jann and Wegrich 2007). Others have pointed to the model's inability to explain how policies advance from one stage to another (Savard and Banville 2012). Furthermore, the cycle does not consider the interaction between different policies and their parallel implementation. Overall, the policy cycle framework may lead to an oversimplified understanding of how government functions. However, the model is considered to be effective as a basic framework to understand the complex field of policy studies and policy making (Capano and Pritoni 2020), and became the basis for further research in this field (Perl 2020). Additionally, from the public administration perspective, the policy cycle helps understand which actors are relevant in which stages of the cycle, as well as providing a helpful and practical way to describe and understand the process of policy making (Jann and Wegrich 2007).

In this article, we propose an adaptation of the EMMIE framework as a tool to develop the stage of policy formulation. During this stage, problems, proposals and demands are transformed into government policies (Jann and Wegrich 2007). Policy formulation involves identifying possible solutions to policy problems by weighting their pros and cons, and deciding which should be accepted and rejected (Howlett & Ramesh 2003; Perl 2020). Policy formulation questions, “What is the plan for dealing with the problem?” (Sidney 2017, 79), and investigates what are the goals and priorities that the policy is trying to accomplish. As such, this stage is a complex process and is one of the least analytically developed and fundamental stages of the policy making process model (Howlett and Mukherjee 2017; Howlett 2018). Studies of policy formulation aim to inform practices within governments by introducing more rational and evidence-based tools, such as budget statements and cost–benefit analysis. We propose the EMMIE framework as a means of encouraging an evidence based, systematic way of formulating policies within the policy cycle (Figure 1). The five components of EMMIE provide valuable dimensions that policymakers can potentially use to better understand and plan their policies and to potentially increase the chances of the policies’ success. By applying the framework at the policy formulation stage, practitioners could increase the likelihood of policies being effective, reduce risks of implementation failure and increase the likelihood of good policy evaluation.

Figure 1. The EMMIE framework and the policy cycle.

The EMMIE framework also can be used as guidance in other stages of the policy cycle. For example, the framework can be used to detail the requirements for successful implementation, (i.e. an implementation plan that includes human, economic and/or technical requirements) which should assist practitioners. Similarly, processes should be established during the policy formulation stage based on the EMMIE framework, which should inform and facilitate policy evaluation. This can include specifying the application of statistically appropriate tools for evaluation, and reviewing how the components of the framework were implemented and which components failed to be applied throughout the policy making process.

The adapted EMMIE framework for policy formulation

In this article, we propose an adaptation of the EMMIE framework as a tool for policy formulation and as a guide for practitioners. The EMMIE components are considered crucial to policy formulation and the development of an evidence-based that is useful to further policy development. By improving policy formulation in this way, we would expect an improvement in policy outcome.

The EMMIE framework was originally designed to assess the evidence from systematic reviews on five dimensions and to give an indication of the quality of the evidence on these five dimensions. We adapt these key components of the EMMIE framework for policy formulation. The concepts remain the same, however, they are presented as a checklist of key variables that policy makers should consider when designing policy, and before adoption and implementation to increase policy effectiveness and contribute to the evidence-base.

Effect

The effect refers to the aims of the policy – what is its expected effect? If a policy is to be sensibly evaluated at the final stage of the policy process, then the expected effect of the policy needs to be clearly stated. For example, a government decides to implement a policy that requires children to continue in full-time education between the ages of 16 and 18 years, rather than the previous policy which was only until the age of 16 years (such as with the UK Education and Skills Act 2008¹ and the 2017 Austrian Compulsory Training Act – Education and Training up to 18²). The aim of the policy is, therefore, to increase participation in the education of 16- to 18-year-olds. Alternatively, it might be to improve the educational attainment of young adults when they reach 18 years of age or more long-term improve the number of people in high earning careers and if so, this should be stated. Or it might be a combination of all three aims, in which case this also needs to be made clear.

A statement of the policy objectives should be accompanied by the targets and key indicators for the measurement of those objectives. Measuring the effect is intrinsically related to the evaluation stage, which is fundamental for understanding the impact of the policy and the extent to which objectives were achieved. There are a number of positive reasons to evaluate policies, most importantly, it is the only way to know their real effects. This requires a methodologically valid evaluation that attempts to isolate the impact of the policy from all other elements that may affect the outcome. The evaluation of policies relies on the development of a well-designed and detailed evaluation strategy, with a clear set of objectives, target populations and areas and expected outcomes (UNODC 2010).

The application of evaluation methods is still relatively new (Bozeman and Sarewitz 2011), and with regards to crime reduction policies is considered to be one of the most neglected aspects of the policy cycle (UNODC 2010). For example, in Latin America, only a small proportion of security policies have been subject to impact evaluations that show which policies have been successful and to what extent (Chinchilla and Vorndran 2018; Cano, Rojido, and Borges 2020). This includes a failure to define measurable performance indicators (with baselines and targets) for security policies in the region (IDB 2010). Evaluation also increases financial accountability and evidence-based understanding of what works. Thus, policy evaluation can be used as a management tool to assist decision making about the continuity, interruption or adaptation of policies. In addition, any serious attempt to improve policy making needs to examine what has already been attempted, be able to demonstrate which aspects of a policy contributed to its outcomes, and which aspects seemed less effective, or generated unexpected results. In turn, this allows policy improvement and the ability to better replicate the policy elsewhere. An example of the use of the findings from an evaluation in the field of crime reduction is Operation Ceasefire in Boston, United States (Braga, Hureau, and Papachristos 2014; Kennedy, Braga, and Piehl 2001). This operation was successful in decreasing gun-related homicides among young people (Braga and Pierce 2005) and has since been replicated, with appropriate modification, into the policy design of violence reduction in several other countries (CVG 2021).

To reach a proper evaluation, it is not sufficient to state during the policy formulation stage the importance of evaluation. Ideally, policy makers would help evaluators by providing a transparent and well-designed evaluation strategy. At its most basic, the policy should mention possible targets and key indicators for the measurement of policy objectives and include the estimated costs of such evaluations. Adequate funding for evaluation also needs to be appropriately ring-fenced, with some studies indicating that five to 20 percent of the overall budget of programmes associated with the policy should be allocated to evaluations (Cano, Rojido, and Borges 2020; UNODC 2010). To ensure independence, evaluators of policies and programmes should be chosen from academia, think tanks and consultancies that have expertise in the policy’s topic matter and expertise in evaluation rather than the institution responsible for the policy evaluating the policy’s effect. Another benefit of including other parties in policy evaluation is that it encourages the institution responsible for the policy to be transparent about the policy and transparent in the publishing of disaggregated data relating to the indicators by which the policy’s impact was being measured. An example of transparency in the publishing of data relating to a policy is data that is published by the transport authority for London, England about serious injuries and deaths of cyclists from accidents with motor vehicles.³ The publishing of these data allows independent agencies to scrutinize these data and determine if the policy to improve cycling infrastructure in London is reducing the number of serious injuries and deaths of cyclists.⁴

Mechanisms

Mechanisms explain the logic of the intervention – how is the intervention expected to work in delivering the expected effect? Implicitly or explicitly, the mechanism refers to a theory about how the intervention is expected to achieve the desired outcome. As such, mechanisms are crucial to understanding social events (James 2021). Thinking in terms of mechanisms can also lead to identifying the causal chains within an intervention that leads to the desired outcome, with each link in the chain offering a means to measure intermediate outcomes. For example, in the case of a policy to improve street lighting the theory might be that to reduce crime (the desired outcome), improved street lighting increases visibility and results in more offenders being identified and caught. Alternatively, the improvement in street lighting results in more people on the street because they feel safer, and by doing so creating safety amongst themselves because of their larger number. Interventions usually consist of multiple mechanisms, as illustrated in the street lighting example, and when articulated in the manner described above identify the data that can be used to evaluate the impact of the intervention. For example, if improvements in street lighting were expected to result in more offenders being identified and caught, analysis of the changes in arrest data would indicate if this was the mechanism that was responsible for a decrease in crime. Alternatively, data on the number of pedestrians on the street before and after the implementation of street lighting improvements would indicate if more people were on the street during hours of darkness and if this was associated with any observed decreases in crime.

It is important that the mechanisms are stated at the policy formulation stage because it is essentially the theories to which these mechanisms relate that are tested in any subsequent policy evaluation. In addition to identifying the data that are necessary to test the impact of the policy, stating the mechanisms at the policy formulation stage can lead to identifying any data that need to be collected before policy implementation commences. Referring again to the street lighting example, to test if more people were on the streets during hours of darkness in the areas where street lighting had been improved would require a measure of the people on these streets during hours of darkness before the street lighting improvements were made. An understanding of the mechanisms also can lead to proposing the key indicators to use when determining the impact of the policy in the evaluation stage (e.g. in relation to improvements in street lighting, indicators relating to changes in crime, changes in the number of offenders arrested, changes in the number of people on the street during hours of darkness and changes in the fear of crime).

It is not uncommon that in policy formulation processes, logical or empirical relationships between policy components as solutions to problems are incorrect or ignored (Howlett 2018). For example, in an attempt to decrease crime committed by juvenile delinquents, many state and city governments in the United States have adopted Scared Straight programmes involving organized visits to prison facilities by juvenile delinquents. Evaluations of Scared Straight have found these programmes to be ineffective (Petrosino et al. 2013), with their failure associated with the inability in identifying the mechanisms behind how the programme is expected to result in deterring future offending by those who participate in the prison visits (Petrosino, Turpin-Petrosino, and Finckenauer 2000). Similarly, policies that were ineffective in reducing violent crime in Chile, El Salvador, Guyana, Honduras, Jamaica and Uruguay were found to lack details about the origins of the crime problems and failed to explain “the logical mechanism by which the proposed activities would result in the attainment of their specific or general objectives” (IDB 2010, 26). This inability to identify the mechanisms associated with a policy is illustrative of how the requirement to identify mechanisms during policy formulation can help policymakers to avoid mistakes in policy design and policy effectiveness.

Identifying mechanisms requires policymakers to understand the causes of problems and the theory of how and why a policy may achieve its desired goals (Sidney 2017). It is not enough to propose a broad statement of theoretical principles, but rather a detailed articulation of the mechanisms, and ideally precise predictions of what is expected to happen are required. By doing so, an articulation of the mechanisms can minimize any unexpected backfire or null effects when policies are adopted and instead are likely to lead to increasing the chances of a policy’s success.

When formulating new policies, attention also needs to be paid to what went before. Research on the effectiveness of crime prevention interventions, for example, has increased substantially in recent years, and the realistic approach explained in this article has contributed to that endeavour. But it is not the only one. Others, such as the Campbell Collaboration (an international social science research network that produces policy-relevant evidence syntheses and policy briefs)⁵ have evaluated prevention programmes that have met a series of scientific standards (such as the use of randomized control trials). Policy makers should use the rich sources of existing evidence to identify whether what they are proposing has been attempted elsewhere and how these other attempts worked, so that they can identify how the mechanisms behind the policy they are proposing are expected to work. In situations where previous policies have not articulated their mechanisms or in situations of policy innovation, policymakers would benefit from consulting colleagues, other stakeholders and experts from academia and think tanks to assist them in identifying how a policy is expected to work in delivering its desired effect.

The process of identifying mechanisms during policy formulation can also assist policymakers in critiquing whether a policy’s desired outcomes are realistic. The inclusion of a stage in the policy formulation process that requires policymakers to explain how the policy will work to achieve its desired outcomes can result in the identification of flaws in how the policy may work. Additionally, the requirement to identify mechanisms during policy formulation can help determine whether the mechanisms are likely to work in all the settings where they are to be applied and if there are challenges with how the mechanisms associated with a policy are implemented into practical interventions.

Moderators (contexts)

Moderators refer to the conditions that are necessary if the policy is to be effective. In practice, moderators refer to the broad context of the location within which the policy is to be implemented including the people that might be involved, the physical places and the time (i.e. what works at one point in history may not work at another). Whether or not a policy is effective is, therefore, context dependent. For example, improved street lighting might reduce crime if it were introduced in poorly lit areas of a city, but it may make no difference to day-time crime rates, or to crime at night in a middle-class area of a city that already had adequate lighting levels. Policy makers, therefore, not only need to think about the mechanisms but also the conditions that are necessary to ensure that the mechanisms “fire” in the expected manner.

Understanding exactly how mechanisms are constrained by sets of existing contextual factors is crucial for making correct policy design decisions in specific policy making contexts (Howlett 2014). For example, many policies for crime prevention were initially developed and evaluated in high income countries with considerable resources, but the experience from these countries is often not replicable or appropriate in less developed settings (UNODC 2010). It is not uncommon for government authorities of low- and middle-income countries to try to replicate policies that were successful in industrialized western countries. That in itself is not wrong, however, trying to reproduce that policy in a different place without understanding how the context may affect it needs consideration. For example, zero-tolerance policing, conceived in the United States, is a strategy that involves relentless order maintenance and aggressive law enforcement against minor crimes and incivilities. This policing approach has been repeatedly implemented in Latin American countries, and in particular in El Salvador, Honduras and Guatemala (Imbusch, Misse, and Carrión 2011) with limited to no success (Dammert 2019; Swanson 2013; Cruz 2010). Further, it has been argued that the policy created negative outcomes such as increasing police violence and police discrimination against the poor (Ungar 2009; Swanson 2013), and enhancing gang cohesion and strengthening gang organization (Bruneau, Dammert, and Skinner 2011). Thus, implementing zero-tolerance policing without understanding the institutional capabilities, cultural realities (such as police violence and institutional corruption) and proper oversight and accountability are errors often made by policymakers. Such replications of policies are often not successful because of the different settings and because the success of the policy is largely dependent on how well it is adapted to local needs. In short, policies should be adapted to the specific local problem and contexts. The mechanisms of the policy may be the same, but the context may alter the success or failure of that policy.

Ideally, understanding the specific moderators (place, time and people) requires collecting quantitative and qualitative information from a range of sectors, rather than relying solely on information from a single source e.g. recorded crime data from the police. By understanding the moderators, policymakers can anticipate how the mechanism might be affected and what modification might be needed before implementing the policy.

Implementation

The implementation process is a core and crucial stage of policy making (Jann and Wegrich 2007). Implementation refers to the actions and actors necessary to successfully install and maintain a programme. According to researchers, this is when a decision is carried out through the application of government directions and is confronted with reality (Savard and Banville 2012).

Essentially, this stage requires policymakers to be clear on how the policy intervention is to be implemented and what they foresee as possible obstacles. The literature refers to this situation as the implementation gap, which is defined as the difference between the planned outcomes of policy and the outcomes of the implementation process (Newton 2001). To avoid this gap, implementation processes need to be incorporated into action plans at the policy formulation stage, rather than being added as an afterthought. For example, in the field of health, there are numerous examples of well justified attempts to improve health care information systems that have failed because of poor implementation (Berg 2001; Giuse and Kuhn 2003). This included the requirement for new information systems to be housed within relatively powerful technological infrastructures which differed markedly from the reality of poor technology infrastructures in most hospitals where the improvements in health care information systems was implemented (Heeks 2006).

In another example, a study of the Scottish Community Engagement Trial (ScotCET) which was an attempt to test the procedural justice antecedents of trust, confidence and police legitimacy within a different policing context, yielded a set of unexpected negative results. The study showed that communication breakdowns during the ScotCET implementation led to misunderstandings of its aims and objectives, and of the requirements placed on officers (MacQueen and Bradford 2017). To avoid these types of failures, every step of the policy implementation process needs to be detailed during the policy formulation process.

Additionally, it is not possible to test the effectiveness of a policy if the policy fails to be implemented as intended (Summerfelt 2003). This means there is a need to monitor implementation processes. This is commonly referred to as “process evaluation”. Detailing the implementation process at the policy formulation stage facilitates this type of evaluation and helps policymakers to systematically plan and document implementation issues and produce a detailed evidence-based account of expected levels of fidelity to policies so that they could be replicated (with due adaptation) elsewhere. Also, policy implementation is likely to involve many agencies, rather than a single agency, to be involved in the implementation of the policy. To do so may require the agency that is responsible for leading the adoption of the policy to incentivise and persuade these other agencies that their involvement is important to the success of the policy (Scott 2005). If agencies responsible for policy implementation are not sufficiently prepared to adopt the policy or do not have sufficient resources, then the policy may fail – not because the theory was incorrect but because of implementation failure. For example, the Honduras “Peace and Citizen Coexistence Project for the Municipalities of the Sula Valley” failed to implement several institutional strengthening measures because the municipalities did not have the appropriate equipment or skills (IDB 2010). As such, a requirement that is placed on governments when seeking agencies to adopt policies is to provide support in training and the building of capacity before policies are adopted. For example, in the United Kingdom, the College of Policing established the What Works Centre for Crime Reduction⁶ to provide robust and comprehensive evidence to assist practitioner decision-making and thus support implementation.

Economics

The economics refers to the cost of the policy implementation. Although it is a straightforward concept, it is uncommon for a policy proposal to estimate its full costs. If done, reference is generally made only to the direct costs of the policy but not the implicit costs (implicit costs are a type of opportunity cost – the benefit that a government misses out on by choosing one policy versus another). A step further would be for policymakers to think of the opportunity cost of implementing a policy. The development of policies takes place within a larger governance context in which sets of institutions and actors are seeking to further their objectives and goals (Howlett 2014). Policies are subject to political realities, and governments must constantly juggle with different priorities which are competing for scarce public resources (Welsh and Farrington 2012). Since governments have limited resources, implementing a policy means that other policies will not be implemented. There is a direct risk when choosing to implement one policy over another to address an issue. If the mechanisms, moderators and implementation issues are not well thought through, the favoured policy might generate both a high economic cost and also a social cost for not implementing a different policy in that specific time and place.

Ideally policymakers need to provide estimates of costs, marginal costs and opportunity costs when designing a policy. They also need to take account of the benefits of the proposed policy, and where possible cost these. A cost–benefit analysis is obviously the most appropriate aim but is seldom realized (Tompson et al. 2020). Another option is to compare the cost of a new policy against existing ones. An example of this is the electronic monitoring of sexual offenders, with several studies showing that electronic monitoring is cheaper than imprisonment but more expensive than policies oriented towards parole and probation offender supervision (Belur et al. 2020). Determining the economic costs of a policy is particularly helpful when deciding upon the initial viability of a policy, but also for determining whether it can be afforded in other areas.

Applying the adapted EMMIE framework and EMMIE-Q scoring to policy formulation

Table 2 shows the adapted EMMIE framework for policy formulation. For the EMMIE-Q scoring, each component has a mark that ranges from 0 to 4, with 4 being the maximum. The table can be thought of as a checklist for policymakers. Ideally, policymakers would include as many of the components (and variables within each component) during the policy formulation process.

Table 2. Adapted EMMIE evidence and four-point scales for assessing quality on each dimension of policy formulation.

EMMIE component	EMMIE-E (evidence itself)	EMMIE-Q scoring
Effect	A clear statement of the policy objectives	0: No reference to objectives, targets or key indicators
	Possible targets and key indicators for measurement of objectives
		1: Broad statement of objectives and/or key indicators
	Intent to implement an evaluation strategy including the policy costs
		2: A clear statement of objectives, key indicators suggested and statement of expected impact
	Statements of evaluation costs
		3: Broad statement of evaluation with some general specification on type of evaluation and indicators used (including estimated evaluation costs e.g. Evaluation Budget Matrix and expected impact)
		4: A transparent and well-designed evaluation strategy that includes at least two factors* to determine its impact
		*Factors to determine impact:
		− High statistical validity (e.g. policy plan allows for the calculation of indicators’ effect sizes using an appropriate methodology such as: Randomized Control Trial, Regression Point Displacement Design, Regression Discontinuity, Propensity Score Matching, Time Series Analysis or Maryland scale)
		− Sufficient assessment of the risk of bias (e.g. consideration of the influence of statistical outliers)
		− Assessment of the influence of study design (e.g. separate overall effect sizes for experimental and quasi-experimental design)
		− Assessment of the influence of unanticipated outcomes or spin-offs on the size of the effect (e.g. quantification of displacement or diffusion of benefit)
Mechanisms/Mediators	Mechanisms, based on relevant literature, are outlined through which the policy is meant to achieve its effect.	0: No reference to theory; simple black box
		1: Broad statement of assumed policy theory stated (mechanisms and/or processes)
		2: Detailed articulation of theory, based on interrogation of relevant literature and/or elicited from practice.
		3: Formalization of theory and derivation of precise predictions from it
		4: Test, corroboration, falsification and refinement of theories, using data assembled for the purpose is possible.
Moderators/contexts	Analysis of possible context-based moderators (theoretically driven)	0: No reference to condition contexts or moderators
	Discussion of the context within which the policy is expected to work and why, relating this to the proposed mechanisms	1: Ad hoc description of possible relevant moderators or contexts
		2: Comments on the possible effects of moderators or mechanisms defined ad hoc using variables that are at hand
		3: Theory-based pre-specification of expected moderators relevant to the activation of mechanisms
		4. Collection and analysis of relevant data relating to the pre-specified expected moderators and contexts is possible
Implementation	A list/statement of key components/resources necessary for implementation	0: No account of implementation or implementation challenges
		1: Ad hoc comments on how the policy will be implementation and/or comments on probable implementation issues
	A timeframe for implementation
	Statements on possible implementation obstacles and challenges
		2: A clear implementation plan that includes human, economic and/or technical requirements for appropriate implementation
	A list/statement of key components deemed necessary for replication elsewhere
		3: A plan to produce a detailed evidence-based account of expected levels of fidelity to the policy and to systematically document implementation issues
		4: A plan to produce a complete evidence-based account of expected levels of fidelity to policy, expected obstacles and specification of elements necessary for replication elsewhere
Economic costs	Estimates of direct and indirect cost of implementation	0: No mention of costs of policy implementation
	Estimation of expected savings of the policy	1: Only direct or explicit costs of policy implementation estimated
		2: Direct or explicit and indirect and implicit costs of policy implementation estimated
	Quantification of intensity (e.g. spend per head) where appropriate
		3: Marginal or total or opportunity costs of policy implementation estimated
	Estimate of cost-effectiveness per unit output	4: Marginal or total or opportunity costs of (policy implementation) by bearer (or recipient) estimated
	Estimate of cost–benefit analysis

Source: Adapted from Johnson, Tilley, and Bowers (2015).

Space limitations in the current article prevent us from including a detailed example of a specific policy and which by doing so, such as with a specific example of a crime reduction policy, may limit the potential wider use of the adapted EMMIE framework to the many areas of social, economic, cultural and environmental policy that the adapted EMMIE framework may potentially be applied. Instead, we offer hypothetical examples to illustrate the application of the EMMIE-Q scoring process for policy formulation.

A fairly well formulated policy would include a broad statement of evaluation with some general specification on the type of evaluation and indicators used (scoring 3 for effect), formalization of theory and derivation of precise predictions of the impact of the policy (scoring a 3 for mechanisms), comments on the possible effects of moderators defined ad hoc using variables that are at hand (scoring a 2 for moderators), a plan to produce a complete evidence-based account of expected levels of fidelity of the policy, expected obstacles and specification of elements necessary for replication elsewhere (scoring 4 for implementation) and direct or explicit and indirect and implicit costs of policy implementation (scoring 2 for economics). This policy would score a total of 14 out of 20 possible marks. Conversely, a badly formulated policy would have no reference to objectives, targets, or key indicators (scoring 0 for effect), a broad statement of assumed programme theory (scoring 1 for mechanisms), an ad hoc description of possible relevant moderators or contexts (scoring 1 for moderators), no account of implementation or implementation challenges (scoring 0 for implementation) and only mention of direct or explicit costs of policy implementation (scoring 1 for economics). This policy would score a total of 3 out of 20. Although total scoring is a useful guide, what is more important is for policymakers to consider all the EMMIE components during policy formulation. For instance, scoring a high number on four of the components but not completing one component could drastically hinder a policy’s formulation. Also, although a higher score overall (e.g. of 14) would indicate a better consideration about the policy that is being formulated than a policy with a lower score (e.g. of 7), the difference in scores would not necessarily indicate (with use of these examples) that the higher scoring policy was twice as good as the lower scoring policy. A high EMMIE-Q score is also not an indication of the policy’s success but is rather a reflection of more thorough, detailed and nuanced thinking about the policy that would enable a better and more useful evaluation and measure of the policy’s success. In turn, an effective evaluation of a policy would help its adopters to identify ways the policy could be improved and contribute to the evidence-base from which others could learn to improve their policy formulation.

Conclusions

The aim of a realist approach is to analyse complex social interventions and to identify key components that explain why, for whom and under which circumstances policies work. The EMMIE framework is a practical tool that helps facilitate that task. With its origins from examining the evidence from systematic reviews of crime reduction programmes, in the current article, we adapt the EMMIE framework and introduce its use as a systematic guide for the formulation of policies of all types. We argue that by considering the EMMIE components when designing new policies, policymakers can increase their chances of successful policy development. By doing so, this would improve the evidence-base knowledge on the types of policies that work and facilitate the transfer of policies from one context to another. By applying the EMMIE framework at the policy formulation stage, practitioners are more likely to reduce the risks of policy implementation failure and increase the likelihood of being able to effectively evaluate their policies.

The EMMIE framework for policy formulation offers a schematic simplification of the rather complex world of public policy development, however, this is also an advantage. Each component of the EMMIE framework makes it possible to identify key elements of the development of a public policy and to modify policies as appropriate. Our adaptation of the EMMIE framework as a generic tool for assisting policy formulation offers the potential for more effectively guiding policymakers through this important stage of the policy cycle. We encourage policymakers to follow this systematic and evidence-based way of thinking for public policy development.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Notes on contributors

Gonzalo Croci

Gonzalo Croci is a Ph.D. candidate at the Jill Dando Institute of Security and Crime Science at UCL and a associate professor at Universidad ORT Uruguay. His research focuses on citizen security in Latin America, in particular researching the variables that affect homicides and the policies used to deter them. Gonzalo holds an MSc in Latin American Studies from the University of Oxford and an MPP from the Hertie School of Governance. He has consulted for several institutions such as the IISS, DCAF, GI-TOC, among others.

Gloria Laycock

Gloria Laycock has a BSc and Ph.D. in psychology from UCL. She established and headed the Home Office Police Research Group and was the founding Director of the UCL Jill Dando Institute. She has carried out research and development in prisons, policing and crime prevention and has acted as a consultant and trainer on policing matters around the world. She is currently Professor Emeritus of Crime Science at UCL and was Director of research supporting the What Works Centre for Crime Reduction until 2016. She was awarded an OBE in the Queen’s Birthday Honours 2008 for services to crime policy.

Spencer Chainey

Spencer Chainey is Director of UCL’s Jill Dando Institute Latin America and Caribbean Unit. His work places him at the interface between academic research and the practical world of policing and public safety. This involves applying academic knowledge and police/public safety practice to research; supporting the implementation of activities that improve crime reduction, criminal investigation, citizen security and countering organized crime; and furthering the education and professional development of current and future generations of practitioners and researchers. Spencer has advised numerous governments on their strategies and interventions for decreasing crime, including Jamaica, Brazil, Argentina, Uruguay, Mexico, Denmark and the UK.

Notes

1 https://www.legislation.gov.uk/ukpga/2008/25/section/174

2 https://www.ris.bka.gv.at/GeltendeFassung.wxe?Abfrage=Bundesnormen&Gesetzesnummer=20009604

3 Available at https://tfl.gov.uk/corporate/publications-and-reports/road-safety

4 For an example, see https://www.bikebiz.com/data-shows-decline-in-deaths-and-serious-injuries-on-londons-roads-in-2020/

5 Available at https://www.campbellcollaboration.org/

6 Available at https://whatworks.college.police.uk/