Grasping the nettle? Considering the contemporary challenges of risk assessment

Abstract

The process of risk regulation is crucial across a range of institutions, sectors and industries. Regulatory bodies worldwide are confronted with a plethora of challenges in managing risks and uncertainties. The precise sources of these challenges are diverse, but are commonly associated with the degree of confidence in predicting and quantifying risks. When the level of confidence is high, regulators tend to specify the outputs and take quantitatively informed preventive measures. However, when levels of confidence are lower, regulators may favour an inflection toward more qualitative considerations of risks to inform resilience building and absorption of adverse consequences. As part of an ongoing research project designed to explore the potentialities of developing a holistic framework for risk assessment which blends qualitative and quantitative methods, this article maps out the key challenges involved in evaluation and decision-making within risk regulatory bodies. In defining the problems and issues faced both by organizations in general and practitioners involved in everyday assessment and management of risk, we have developed a heuristic designed to assist in understanding, categorising and evaluating risk. It is anticipated that the development of knowledge in this area can contribute toward progressive process modifications, improved decision-making at senior management level, and enhance risk management practices amongst regulatory agencies. The project involved semi-structured interviews with practitioners working in risk regulatory bodies from the UK, Germany, France, Belgium, the Netherlands and New Zealand. In coalescing the findings of empirical studies, the sources of these challenges were discussed as being related to rational, technical and expert factors. The main areas of analysis focused on in this article revolve around the process of evaluation, organisational strategies, structural factors and expert perceptions.

KEYWORDS:

• Risk assessment
• risk regulation
• expert risk perceptions
• trust

Introduction

Whilst it may seem expectable that the higher the magnitude and scale of a given risk the more carefully the risks are considered in the assessment process, history and the present prove that this is not always the case. A litany of institutional failures in assessing and regulating risks indicate that underestimations of threat levels and inept regulatory measures can have calamitous human and economic consequences: from the COVID-19 pandemic, to the Fukushima nuclear plant disaster and the Boeing 737 max airplane crashes. Errors in risk assessment reproduce wide-ranging effects, can cause damage to the reputation of public authorities but also impact on levels of trust in regulators amongst the public (see Khodadadyan et al. 2018).

Given the above, assessment of risks is a major issue for regulatory bodies. The academic literature indicates that challenges are not only related to the sensitivity of risk assessments but also are connected to the: quality and scope of information (Mythen and Wardman 2016; Breyer 2009; European Commission 2000); allocation of resources and anticipatory resilience strategies (Mythen 2018; Linkov, Larkin, and Lambert 2015; Baldwin, Cave, and Lodge 2012); implementation of certain risk assessment tools and techniques (Boardman et al. 2017; Baldwin, Cave, and Lodge 2012; Arrow et al. 1996); assignment of liability (Baldwin, Cave, and Lodge 2012; Braithwaite and Fisse 1988); absence of appropriate knowledge exchange in practice (Heyman and Brown 2013; Lloyd-Bostock and Hutter 2008; Tait and Levidow 1992); lack of consensus on the acceptability level of risk; perception and categorisationof risks; overestimation or underestimation of risks; level of transparency and the involvement of the public in risk assessment and decision-making processes.

To reflect on contemporary risk assessment and management practices and to explore the possibilities of fusion between natural and social science methods and modes, this article focuses on the key challenges in assessment and risk decision-making processes. This article falls into four main sections, followed by a conclusion. Following on from the introduction, in the first section we discuss salient risk assessment and risk management challenges in regulatory bodies. We go on to identify and examine various processes and issues which generate dilemmas within regulatory bodies. The research methodology deployed in the project subsequently discussed is elaborated, leading into data analysis and critical, reflective discussion. The results of the research study indicate that the major challenges faced by public regulatory bodies can be indexed to three primary areas: process, organisational strategy and expert perception.

The challenges of risk assessment for regulatory bodies

Managing risk requires systematic and bureaucratic efforts to capture, assess and evaluate the probability and impact of uncertain events. The process can only be successfully accomplished through the constitution of effective and efficient regulatory strategies that facilitate optimum decision-making (Haines 2013). Findings from studies into risk management practices indicate that public regulatory bodies are recurrently confronted with processual challenges. Augmenting these challenges, regulators are increasingly required to enhance performance and provide greater efficiency, and, in times of austerity, reduce the spending of public funds (Curristine, Lonti, and Joumard 2007). In addition, challenges in risk assessment are frequently related to the quality and scope of risk information, particularly under conditions of uncertainty (see Mythen and Wardman 2016). Historical data shows that inadequate information in risk assessment has caused significant financial losses and also facilitated public relations disasters. As Breyer (2009) comments, inadequate and incomplete communication of risk information by regulators can weaken public trust in regulatory organisations and prevent the maturation of potentially progressive, practical regulations.

Of course, the selection of modes of risk assessment within regulatory bodies will vary according to organisational objectives, sector priorities and available resources. Baldwin, Cave, and Lodge (2012) underscore the role of accurate information in effective resource allocation within the broader risk management process. In particular, Baldwin, Cave, and Lodge (2012) highlight regulators’ concerns about deploying resources to develop or anticipatory and resilience based approaches. Pre-emptive risk anticipation strategies emerge in circumstances in which early risk identification may prevent the production of risks of large magnitude. Strategies designed to build resilience focus primarily on alleviating the effects of hazardous events such as informing the public about risks and relating mitigation plans (Linkov, Larkin, and Lambert 2015). The challenge for all public bodies is to find a point at which intervention should occur. In fact, managing risks requires the implementation of strategies to reduce the production of risks along with mitigation of the negative effects of hazardous events through adopting warning measures and procedures and safety systems as well as contingency plans. Therefore, it is necessary to develop proactive safety critical methods to improve the foundations of managing risks (e.g. dynamiting the avalanche slope), alongside more routine practical mechanisms to reduce the negative impact of unwanted events (e.g. evacuating people from the probable avalanche path). However, the tension between the concepts of anticipation and resilience to be considered by regulators and risk managers remains an ongoing challenge, as consistently highlighted in various literatures (see Linkov, Larkin, and Lambert 2015; Foster 2012; Turner and Pidgeon 1997; Wildavsky 1985).

Much of the discussion in risk research circles concerning anticipatory risk strategies revolves around application of the ‘precautionary principle’ (Mythen 2018; Baldwin, Cave, and Lodge 2012). The precautionary principle indicates that unconfirmed but suspected damaging risks to the environment and human health and welfare should be treated as ‘real’ and mitigation strategies should be rolled out accordingly. The German precautionary principle known as ‘vorsorgeprinzip’ has been incorporated into EU law, in the same way as Principle 15 of the 1992 Rio Declaration counsels governments to use the precautionary principle and apply financial measures to reduce regulatory costs in the absence of scientific data or when facing potential threats of irreparable damage. Likewise, adhering to the precautionary principle is necessitated by the European Commission in circumstances in which scientific evidence is uncertain, inconclusive or incomplete, as well as in situations in which preliminary scientific evidence indicates that reasonable concern is warranted (see Heyvaert 2006; European Commission 2000).

Most of the debates around the use of the precautionary principle focus on the necessity of making interventions that may possibly be non-rectifiable or irreversible (Baldwin, Cave, and Lodge 2012). Delivering innovative ideas to manage risks is inherently risky by its very nature (Brown and Osborne 2013). Failure in the short or long term may occur and this may render pre-emptive intervention unsustainable. The other problem with the precautionary principle is its potential to be abused by those holding economic and political power (Wardman and Mythen 2016; Majone 2002). Baldwin, Cave, and Lodge (2012) cite instances of absence of ‘scientific certainty’ as an opportunity for manipulation of public concerns by power brokers harbouring vested interests. Other factors that militate against the application of the precautionary principle include high costs and the resources required to implement anticipatory strategies in managing risks. Wildavsky (1988) explained that using the precautionary principle and associated safety rules may prohibit progress of particular products or practices and this may ultimately prove obstructive to development. It might lead to acceptance of more risks particularly after consumption of materials or products have already been approved. Thus, Wildavsky (1988) advocates the notion of ‘trial and error’ in tandem with experimentation with resilience strategies oriented toward significant improvements in the recovery of systems. The key point of contention around the application of the precautionary principle is not only connected to the variable application of methodology, but also, more foundationally, to how the principle itself is defined (see Frederickson and LaPorte 2002). Notwithstanding this definitional quagmire, certain risk tools and techniques - such as cost benefit analysis, cost effectiveness analysis and multi-criteria decision analysis - have proven utility within the risk assessment process. While such tools and techniques offer potentialities for progression, they have themselves been subjected to critique (Boardman et al. 2017; Arrow et al. 1996). It has long been observed that a pronounced inclination toward quantitative methods of risk assessment have led to some oversight of the limitations of such modes of analysis. These limitations include shortcomings in terms of post hoc evaluation of effectiveness, administrative inconsistencies in application, underlying value assumptions, public acceptability of findings and problems of equity.

Baldwin, Cave, and Lodge (2012) state that private organisations’ employment of various quantitative techniques geared toward population level statistics fall short in addressing discrete individual concerns about risks that members of the public may have. A prime example is the role of insurance companies and their interplay with regulatory and legal bodies in structuring responses to risks when insurance mechanisms form liability criterion which may be considered to be morally problematic. When regulatory bodies expect private organisations to address risk in the form of enforcement, the imposed obligation is subject to criticism. Katzman (1988) believed such public concerns should not be solely managed by private organisations. Scholars on ‘high reliability organisations’ imply that regulatory strategies are necessary to make the ground ready for mindful organisational culture and motivations to allocate resources in understanding and reducing man-made risks. It is therefore a requirement for organisations to be adaptable and resourceful in the prior construction of back-up facilities in the case of failure in the main system or concomitant databases. On the other hand, literatures on organisations and technologies explain data redundancy as a cause of failure – and indeed a generator of risk itself – since it increases complexity to pre-extant complex structural procedures in organisations. Further issues to consider are related to the assignment of liability and subsequent allocation of responsibility for risk. This raises the question as to whether the primary focus should be on responsibilizing one individual for adverse and harmful occurrences, or whether it is sensible to extend the parameters of law to prohibit charges of corporate and collective responsibility (Baldwin, Cave, and Lodge 2012; Braithwaite and Fisse 1988). Whichever looking glass one adopts, it is incontrovertibly the case that risks are better managed and controlled in open and dialogic environments, in which communication channels are unrestricted and horizontal. This observation notwithstanding, palpable challenges remain in terms of the creation of a solid foundation for appropriate application of knowledge around risk management, based on organisations’ strategies and design. A number of scholars (Heyman and Brown 2013; Lloyd-Bostock and Hutter 2008; Tait and Levidow 1992) have explored the creation of such principles and knowledge exchange practices, but the limitations imposed by practical restraints for organisations remain and these are more acute in conditions of austerity and dwindling resources.

The other remaining regulatory challenge is related to public acceptability of risks and what might constitute an optimum level of public participation in decision-making in terms of the management of such risks. Unsurprisingly, there is no consensus on this topic. One school of thought highlights the importance of accountability and emphasises the value of risk management approaches that are based on scientific evidence. This view explains the need to seek advice from wider communities of experts’ in decision-making when scientific evidence is not conclusive. However, opponents of this view have drawn attention to the accuracy of decision made by small teams of knowledgeable experts and the possibility of ‘group-think’ occurring. The errors reproduced by a small circles approach in the early stages of managing the COVID-19 outbreak in the UK are one pertinent example of this. The consensus view amongst epidemiologists is that the lives of thousands of British citizens could have been saved in the UK had the Government acted more swiftly and decisively in locking down and introducing social distancing measures (see Sridhar 2020). Casting back to the logic of the precautionary principle, it is clear that countries such as Germany, South Korea and New Zealand where precautionary measures were rapidly and decisively adopted suffered far fewer cases of infection and far lower casualty rates. In order to avoid the perils of groupthink, bringing in a plurality of viewpoints and opinions constitutes an obvious way forward in relation to high consequence risks (see Gorz 2010; Sunstein 2002). Nevertheless, it needs to be acknowledged that decision-makers within public risk regulatory bodies do not have a crystal ball, meaning that lapses and errors are inevitable. To this end, maintaining an institutional memory of successes and failures is vital to institutional learning and the building of a safer environment.

So, how to square the circle by the challenges presented above? Is the creation of an adaptable, holistic model for assessing risk possible? Indeed, is it even desirable? Whilst answers to questions of this magnitude extend beyond the ambit of this article, we do wish to pursue the more limited ambition of drawing together relevant literature on the major challenges faced by risk sensitive public bodies and connecting this to the findings of a small scale comparative study into practices of risk assessment within these agencies. In so doing, it is our broader intention to point up some areas which may be ripe for further investigation within the interdisciplinary field of risk studies.

Regulating risk: identifying the problems and issues

The academic and policy related literature suggests that there are complex and multiple challenges in the risk assessment and decision-making processes among risk regulatory bodies. In coalescing the findings of empirical studies, the sources of these challenges not only relate to organisational factors associated with the evaluation of probability and the impact of an occurring event. Rather, they also connect to rational, technical and expert factors. As an heuristic device, in the below we provide a summary of some of the most salient contributions to the literature according to these three factors.

While the table above is intended as indicative rather than exhaustive, it shows the various weights placed on the three factors - rational, technical, expert - in defining the challenges of risk assessment within relevant academic literatures.

For Baldwin, Cave, and Lodge (2012) the most significant challenge regarding the assessment of risks among regulatory bodies relates to the failure to use an appropriate risk assessment metric, coupled with the mismeasurement of criteria. For their part, Viscusi and Hamilton (1999) and Rothstein, Huber, and Gaskell (2006) suggest that political pressures which are reflective of people’s attitudes can shape and influence the ‘rational’ approach of regulators. Thus, regulators may commit irrecoverable errors within the process of risk assessment due to stakeholder, public or wider political pressures. Of course, regulatory experts are not immune to external forces and their decisions are not always objective. Indeed, on occasion the decisions of experts may reflect a range of risk biases (Noll and Krier 1990) that depart from predictions forecasted by a standard benefit-cost framework (Slovic 2010). Within rational approaches, the decision-making process adopted by regulators is influenced by the character of the organisation exposed to risk and the degree of risk inspection within the organisation (Kunreuther and Easterling 1990).

Regulators across the globe have developed myriad strategies and policies to perform technical risk assessments (Taylor and Yetley 2008). The technical approach was introduced and established as a critical process for ‘risk-free’ environments (Taylor and Yetley 2008). The landmark NRC report (National Research Council 1983) emphasized the usefulness of modes of scientifically informed technical risk assessment in relation to regulatory processes adopted by the US Federal Government. The primary purpose of technical risk assessment is to provide a scientific basis for making reasoned judgements regarding decisions that impact on public health. Following the technical model, decision-makers typically use historical data to calculate and measure probability and impact of risks (Slovic et al. 2004). However, this process is inherently predictive and cannot produce certainty about what constitutes a safe level of risk for the population. In diverse regulatory frameworks, the use of different risk criteria, measures and metrics will produce different outcomes.

Within approaches focussed primarily on experts, the major challenges identified were mainly initiated from linear and iterative approaches (Clemen and Winkler 1999). Regulatory bodies invest trust in expert judgments arising out of impact assessment and deploy this information as a means of forecasting, analysing and responding to risks. In this vein, Cooke (1991) reviewed several expert approach assessments in various spheres such as aerospace, economic, technology, nuclear engineering, military intelligence and environmental risk from toxic chemicals. On the basis of these reviews, Cooke (1991) provides an account of the influence that experts themselves have on linear assessment and explains why regulatory bodies need to create iterative multi-expert assessment to avoid subjective or biased decisions being made. An iterative multi-expert assessment process supports assessors in obtaining as much information as possible. Nonetheless, the key issue – ‘resolution of conflicting information or opinions’ (Wu, Apostolakis, and Okrent 1990) - still remains. Following an iterative process, experts’ probability distributions are combined to provide agglomerated information for risk analysts to consider and develop (Beroggi and Waliace 2000). This involves procedures for combining probability distributions which are often compartmentalised as mathematical aggregation methods or behavioural approaches. The challenges involved in the procedures are associated with both mathematical and behavioural methods. Lin and Cheng (2009) have argued that mathematical approaches – such as Cooke’s classical model or Bayesian aggregation models – may be adversely affected by miscalculation or ignorance of the expert assessors’ dependency structure. In contrast, challenges in behavioural approaches are seen to arise largely from human interpretation errors in obtaining agreement around levels of risk.

As articulated above, various empirical studies have identified a range of challenges inherent to the risk assessment and consequent decision-making processes among risk regulatory bodies. Combined they suggest that the challenges are not only about technical factors. Rather, they also include a series of organisational, individual, environmental and social factors. In order to further debates about enhanced risk management practices, we wish to move on now to discuss some of the preliminary findings from a qualitative study designed to capture some of the salient challenges faced by public regulatory bodies in the risk assessment process.

Research methods and study outline

This research reported on below constitutes part of a mixed method study designed to identify some of the key challenges in risk assessment within public regulatory bodies in the UK, Germany, France, Belgium, the Netherlands and New Zealand. The findings related are currently being reviewed and extended for the purposes of designing a holistic risk assessment framework that might be deployed in future by risk practitioners. The project was designed as a sequential exploratory combined methods study (see Cameron 2009) and included qualitative and quantitative methods. A mixed method was employed because neither quantitative nor qualitative methods were deemed singularly adequate to capture and assess the challenges, barriers and depth of understanding, given the complexities involved in the area under investigation (Ivankova, Creswell, and Stick 2006). In the remainder of the article, we wish to discuss the qualitative aspects of the study, based on panoramic literature reviewing and qualitative interviews with practitioners.

Based on the extant pertinent literature (see Table 1), a compendium of risk assessment factors were considered in order to identify the key challenges faced by risk assessors and analysts in public regulatory bodies. This exercise provided the opportunity to refine and augment the content of the literature review and, moreover, to probe some of the key issues faced by practitioners on the ground. The exercise also added breadth to the research and formed the basis for the content of semi-structured interviews. Qualitative semi-structured interviews were conducted during the first data collection phase of study. All of the participants were current risk regulator professionals working within public agencies. To ensure the quality of the information collected, participants were provided with an outline of topics to be discussed in advance. Each participant performed one of the following professional roles: senior manager, risk assessor/analyst, decision-maker.

Table 1. Source of challenges in the risk assessment process.

Author/s	Findings/Challenges	Source of Challenge
		Rational	Technical	Expert
NRC Risk assessment in the federal government (1983)	Evaluates past efforts to develop and use risk assessment guidelines, reviews the experience of regulatory agencies with different administrative arrangements for risk assessment, and evaluates various proposals to modify procedures.	✓	✓
Kasperson et al. (1988)	A conceptual framework that seeks to link systematically the technical assessment of risk with psychological, sociological and cultural perspectives of risk perception and risk related behaviour.		✓
Kunreuther and Easterling (1990)	Proposes a two-period expected utility model to explain preferences for benefit packages. The empirical analysis examines the predictive power of this model.	✓
Wu, Apostolakis, and Okrent (1990)	Examines the degree to which non-probabilistic models can be applied to system analysis in terms of capacity to combine knowledge.			✓
Noll and Krier (1990)	Considers implications of the cognitive theory for regulatory policies designed to control risks to life, health and the environment.	✓
Clemen R. T. and Winkler R. L. (1999)	Suggests that an overall aggregation process could involve both mathematical and behavioural aspects. No single process is defined as optimum for all circumstances.			✓
Viscusi and Hamilton (1999)	Analyses decisions made by federal and state regulators at hazardous waste sites addressed by the Superfund program to determine how their decisions diverge from those predicted by expected utility theory and benefit cost analysis.	✓
Beroggi and Waliace (2000)	Evaluates decision models proposed in the literature for individual risk managers to account for situations where multiple risk managers are involved.			✓
Slovic et al. (2004)	Posits that rational and the experiential systems operate in parallel through symbiosis. Focuses on studies that have demonstrated that analytic reasoning cannot be effective unless it is guided by emotion and affect.	✓	✓
Rothstein, Huber, and Gaskell (2006)	Proposes that pressures towards greater coherence, transparency and accountability in risk assessment can create institutional challenges by exposing the inherent limitations of regulation.	✓
Taylor and Yetley A. (2008)	Considers approaches that rely on the systematic scientific assessment of risk to determine the levels of intake below which no harm may occur.		✓
Lin and Cheng (2009)	Indicates that aggregation models significantly outperform the best expert approach, signalling the need for inputs from multiple experts.		✓	✓
Slovic (2010)	Explores the concept of ‘risk as feelings’ and examines the interaction of feeling and cognition in influencing perceptions of risk.	✓

The semi-structured interviews consisted of seven questions, divided into two sections. Section one solicited general information about the participants. Section two identified and discussed the key challenges in different risk assessment approaches within related professional impact areas. Through the interviews, the sources and ownership of key challenges were discussed in detail. Prior to the main study, a pilot study with risk assessment practitioners was conducted to test and verify the appropriateness of the semi-structured interview questions. The research questions were amended and refined based on the pilot study’s feedback. The pilot study involved four participants and the final study involved 36 professionals from public risk regulatory bodies who contributed during the period from 9th May 2018 to 17th November 2018.

A non-probability sampling approach was adopted in order to seek assistance from experts expressing an interest in the research topic. As Saunders, Lewis, and Thornhill (2009) note, unlike quota and probability samples, there are no rules for sample size in a non-probability sampling approach. Rather, the actual size depends, among other things, on available resources and the logic behind the sample selection. This argument is supported by Patton (2002), who maintains that the validity and understanding that the researcher will gain from analysing data via this type of sampling method serves to boost data collection and analysis skills. In this study, the sample size was determined on the concept of saturation in a qualitative method. After 36 interviews, it is highly probable that no new information or themes would be observed that would significantly extend the list of key challenges currently faced by risk regulatory bodies.

Qualitative interviews: data analysis

Transcripts from all of the face-to-face and on-line interviews were processed and analysed using NVivo 12. This enabled us to highlight and extract salient contemporary challenges of risk assessment being grappled with by public facing agencies. During the first stage, the word documents of transcriptions were imported into NVivo, generating a file dubbed ‘interviews’ in the source field. The cross-interview approach was considered to group responses from different interviews (Patton 2002). Further, the organised data from the approach were analysed through the content analysis. Referring to Holsti (1969), content analysis is a standard practice in social science in creating implications constructed upon particular characteristics of messages. A coding system helps to squeeze large quantities of words of transcript into smaller amounts of classifications (Allen and Reser 1990) determining the presence of convinced notions or words. In fact, content analysis was in assistance of this study in measuring and analysing the presence, perceptions and interconnections of sentences and words as well as developing interpretations concerning the messages in transcripts.

Coding process is the primary step of content analysis. In which, collected data have to be coded into groups or subjects constructed upon valid and effective interpretation. Groups and subjects are principally established on a range of points like words, word senses, sentences, expressions, and subjects. Further, the analysis of the coded and classified data is possible through one of the content analysis’ methods. There are two fundamental methods for content analysis known as conceptual and relational analysis. Conceptual analysis concerns of concepts review and analysis comprises measuring and totalling presence of the concept; though, the rational analysis includes the identification procedure of concepts existing in a set of transcript (Palmquist et al. 1997).

In this study, the conceptual analysis was adopted for the qualitative part of this study because of the chosen interview questions and selected samples. Therefore, the transcripts from interviews were collected, coded and were summarised into possible content classifications of words or phrases. Such classifications assist the researcher to concentrate on any particular problem-solving codes, words or configurations relative to the research objectives.

A total of 129 passages across the interviews were highlighted and deemed to be related to the subject area. The seven main high level themes were identified from the interviews. A thematic analysis has been produced to demonstrate the parent child relationship between the subjects and the lower level nodes by creating a parent child relationship. The key identified themes related to challenges in risk assessment for public regulatory bodies were related to: process, organisational strategy and expert perceptions.

Results and discussion

The first key challenge identified was the adaptation of the risk assessment process. The results from the data analysis indicated that there is a need for an adjustable risk assessment process among regulators. 33 out of 36 interviewees explained that this difficulty exists due to their inability to recognise to what extent the assessment of risks is needed to be conducted within dedicated resources to meet standards and public criteria. One of the participants in the health and safety sector stated, “there is a gap between what is written in guidelines and the actual practice of risk assessment (how different impacts are evaluated and treated in practice). So you'll find yourself in situations where you are trying to conduct an analysis while you know it is not the way which is discussed in guidelines. To give you an example, the assessment of unemployment impacts that might arise as a result of restricting a chemical substance – because company cannot use the substance any more, therefore has to dismiss its workers. Now, how do you assess the impact of the generated unemployment or even the employment that may be created in another company? Non of the regulation guidelines explain any of these factors, so it's the effort of analysts to adjust risk assessment process in practice within the existing resources and standards”. Participants mentioned the scope and extent of the process is highly related to the organisational characteristics, existence of standards and public requirements and preferences. This observation accents the difficulty of measuring and prioritising the vast range of probabilities with key standards and requirements with respect to available resources. Another participant from the environmental safety sector indicated that “we need to follow many guidelines and standards for environmental risk assessment (e.g. Green Leaves3, COMAH, etc.) along with a large numbers of environmental economic guidelines; however, analysts practically need to be expert in the fields of environment, economics and the consultancy to be able to use all these guidelines. Does that really work to be expert in all of these fields and be able to work through all of theses guidelines?” It was further amplified by the fact that the risk assessment process is continually evolving and updating to keep beneficiaries satisfied. Interviewees indicated that a fit-for-purpose risk-assessment framework is needed to adapt to a labile environment, new legal/regulatory standards and policy advances. A number of interviewees also emphasised the other rational aspect of the risk assessment establishment in their organisations. The view was expressed that for a group of experts to avoid bias measurements and calculations a fit-for-purpose risk assessment process should be carried out and that this itself should connect to recent adaptions to policies and regulations.

Within the risk assessment process, the research participants were also concerned about the influential parameters. They stated that the importance of identifying the most important parameters in the sense that they affect assessment outcomes most. The interviews revealed that actors within many regulatory bodies simply select the parameters that they feel would be most important. This approach can generally lead to problematic results because the importance of parameters may relate to distinct factors. Based on the EFSA (2014) report, the parameters depends on two factors. First, how strongly the identified parameters feature in the assessment model, and, second, the level of the parameters’ uncertainty. A number of analysts indicated that they principally “ignore low priority parameters in risk identification stage and mainly focus on medium to high-level priorities”. This shows how refusal in taking low priority parameters with high-level sensitivity indexing could lead to inaccurate results in risk assessment practices.

The second most important challenge in assessing risks is related to the organisational risk strategy. Thirty-two experts within the study indicated that their risk assessment results were influenced by misinterpretation of the objectives in assessment strategies. They stated that their strategic plans were primarily affected by misconfiguration of risk maps to identify and prioritise the full range of significant risks. In some cases, the absence of a detailed risk map was perceived to have disrupted the achievement of the regulator’s strategy through pre-active risk assessment, and failure to allocate roles, responsibilities and accountabilities for assessors and decision-makers. Several interviewees also raised the prospect of regulations and standards involved in risk assessment in obtaining assessor commitment to the principles of risk control being damaged by the misconfiguration of objectives in risk assessment strategies. Moreover, a number of participants also discussed the ways in detailed risk maps in risk assessment could facilitate compliance with the best practice in corporate governance. In addition, the research study identified that some regulators had no knowledge management plans in their risk assessment strategies.

Expert judgement was identified as the third key challenge in the risk assessment process. Risk regulators’ decisions within environmental and social management, are often based on either expert judgement or on complicated quantitative models. Interviewees stated that their judgements and models typically focus on a small subset of processes rather than actual values as such, and there is sometimes minimum efforts in understanding and quantifying risks in complex systems. Interviewees generally perceived that quantifying models for environmental, ecological, and social risk assessments are limited and that only one tool (Bayesian networks) presently offered a pragmatic and scientifically credible approach in modelling complex systems. Participants indicated that either experts’ knowledge or quantitative data is used to parameterise variables in Bayesian practices. The practices rarely include both forms of information, because the process can be confusing and time-consuming. In both approaches, concerns were expressed that poor uncertainty assessment results might be obtained based on insufficient expert knowledge or data gaps. One interviewee from EFSA highlighted challenges relating to expert judgement within the expert knowledge elicitation (EKE) approach in the European Food Safety Authority. The EKE process is based on experts’ knowledge and critical decisions, and includes facts, data, sources, requirements, preferences, utilities, probabilities and estimates. Selecting the experts for the EKE process who know the answers and variables in risk assessment was cited as a key problem. To reach an accurate judgement, experts must have specific knowledge and expertise which is recognised as generic knowledge in a field. A number of participants criticised the availability of experts in new and emerging risks or hazards. They discussed that knowledge or data relating to some individual parameters in the EKE approach are, at best, sparse and sometimes non-existent. A participant from the food safety sector stated “it is not easy for us to access people with the right skills and expertise to assess all risks associated with food. Lack of experts directly influences the organisational approach selecting methods of risk assessment”

In the interviews participants also raised issues around the potential impact of psychological factors in shaping expert assessments, citing instances where such biases – coupled to incomplete or inadequate judgements – may occur. Responses from experts were frequently structured in relation to answering questions which were naturally obtained from short-cut heuristics. This type of judgement potentially increases cognitive biases and may result in systematic errors in assessments. In addition, the interviewees mentioned more practical challenges impacting the risk assessment cycle, such as time and cost in expert judgement approaches. Limited access to and availability of temporal or financial resources may affect levels of expert involvement and methods of assessments. For instance, the resource cost of face-to-face meetings with experts is different from online or unsupervised assessments. The managing of the obtained knowledge from experts was also raised as another challenge of risk assessment. Respondents highlighted that it is important to achieve a single consensus probability distribution for each uncertain quantity in risk assessment. If experts provide individual judgements on probability distributions then the problem of how best to collate and combine all of the information came to the surface in discussions.

Concluding remarks: collecting up the problems and issues

What is readily apparent in the discussion above, is that overcoming risk-based regulation challenges requires a high degree of honesty and transparency amongst all stakeholders, particularly in conditions of high uncertainty or in situations in which the probability and the consequences of events remain unknown or unknowable. The creation and progression of institutional mechanisms for developing greater transparency in indeterminate contexts has previously been highlighted by other scholars (see Löfstedt and Wardman 2016; Wardman and Mythen 2016) and is worthy of further investigation. Constructions of such mechanisms is informed by and runs in tandem with engaging purposively with enduring and complex risk-based regulatory challenges, including divergent perceptions of risk, proclivity towards anticipation or resilience and public acceptability of risk to name a few.

The chief ambition of this article has been to define and explore the major challenges in risk assessment faced by public organisations involved in risk regulation. In order to inform our analysis, we have drawn on state of the art literature and the results of an empirical research study. We posit that the key challenges faced by regulatory bodies are related to the three main areas: process, organisational strategy and expertise (see Figure 1). The need to adopt an appropriate risk assessment framework to meet the requirements of regulatory bodies’ remains an ongoing and unfinished venture. The adoption of bespoke fit-for-purpose processes are required within such organisations and this crucially involves selecting appropriate variables, setting apt assessment criteria and identifying influential parameters and outliers that may impact. With reference to organisational strategy, the participants in the study identified the interpretation of objectives, pro-active approaches, allocation of accountability, awareness of new standards and regulations and establishing best practices as the most important sub-challenges for risk assessors. In relation to experts, lack of expertise in specific fields, robustness of quantitative simulation, incompleteness of data, psychological influences, cognitive biases, resource related challenges and consensus probability were identified as prominent issues to address.

Figure 1. Summary of key challenge areas in risk assessment for public regulatory bodies.

The findings of this article are designed to aid further understanding of some of the quandaries of risk assessment and to elucidate the decision-making predicaments prevalent within public risk regulatory agencies. Our approach has been informed by scoping the academic literature and scrutinising the findings from a rolling study which included professionals from a wide range of industries, including health and safety, food and nutrition and environmental management.

Our primary intention has been to excavate the key problems that underscore risk assessment and to explore alternative modes of measuring and evaluating risk. Aside from contributing to the development of academic knowledge within risk studies, we consider that our findings also have resonances worthy of discussion for public regulatory agencies. In defining the problems and issues faced both by organizations in general and practitioners involved in everyday assessment and management of risk, we have developed a heuristic designed to assist in understanding, categorising and evaluating risk. It is anticipated that the development of further knowledge in this area can contribute toward not only progressive process modifications, but also improved decision-making at senior management level. This itself may have the positive domino effect of enhancing sustainability, resilience and efficiency within risk regulatory bodies.

Acknowledgements

The authors would like to express especial thanks and deepest appreciation for the support and funding received from the EPSRC and ESRC Centre for Doctoral Training on Quantification and Management of Risk & Uncertainty in Complex Systems & Environments, within the University of Liverpool and the industrial partners involved in the project, the Health and Safety Executive (HSE) and the Food Standards Agency (FSA).

Disclosure statement

No potential conflict of interest was reported by the authors.