Critical review of methods for risk ranking of food-related hazards, based on risks for human health

ABSTRACT

This study aimed to critically review methods for ranking risks related to food safety and dietary hazards on the basis of their anticipated human health impacts. A literature review was performed to identify and characterize methods for risk ranking from the fields of food, environmental science and socio-economic sciences. The review used a predefined search protocol, and covered the bibliographic databases Scopus, CAB Abstracts, Web of Sciences, and PubMed over the period 1993–2013.

All references deemed relevant, on the basis of predefined evaluation criteria, were included in the review, and the risk ranking method characterized. The methods were then clustered—based on their characteristics—into eleven method categories. These categories included: risk assessment, comparative risk assessment, risk ratio method, scoring method, cost of illness, health adjusted life years (HALY), multi-criteria decision analysis, risk matrix, flow charts/decision trees, stated preference techniques and expert synthesis. Method categories were described by their characteristics, weaknesses and strengths, data resources, and fields of applications.

It was concluded there is no single best method for risk ranking. The method to be used should be selected on the basis of risk manager/assessor requirements, data availability, and the characteristics of the method. Recommendations for future use and application are provided.

KEYWORDS:

• Risk prioritization
• risk ranking
• food safety
• nutritional hazards
• health impact

Introduction

Ranking of health risks related to food safety and nutrition is generally recognized as the basis for risk-based priority setting and resource allocation. It permits governmental and regulatory organizations to allocate their resources efficiently to the most significant public health problems (Van Kreijl et al., 2006). Within the area of food, risk is defined as the analysis and prioritization of the combined probability of food contamination, consumer exposure and the size of the anticipated public health impact of specific chemical, microbiological and/or nutritional hazards related to food. It is the combination of the probability that a hazard may occur in a food product and the effect of exposure to the hazard on human health (Codex Alimentarius, 2001). Risk ranking has been applied to food safety monitoring programs and has shown to increase the efficiency of monitoring and to decrease inspection costs, both in practice and from theoretical calculations (Presi et al., 2008; Baptista et al., 2012; Reist et al., 2012).

To date, various risk ranking methods are available that prioritize food safety risks (Van Asselt et al., 2012). Methods vary from qualitative, through semi-quantitative, to quantitative methods (Cope et al., 2010; Van Asselt et al., 2012). Most methods are based on the ‘technical’ concept of risk being a function of presence of the hazard and severity of its impact on human health. However, some methods also involve other metrics, which may be considered in decision making, e.g., consumer perceptions of risk. In order to determine which methods are most suitable for ranking food-related risks, it is important to follow a structured, objective and transparent approach to identifying and evaluating the available methods (van Asselt et al., 2013).

The aim of the current study was to review available methods for ranking risks associated with food on the basis of anticipated health impact, to characterize the methods and to provide recommendations for their use.

Material and methods

Protocol for literature review

A literature review was conducted which aimed to identify risk ranking methodologies that can be used to prioritize food-related hazards, on the basis of the size of anticipated health impact. Hazards are defined as those agents that can be present in food and can negatively affect human health (Codex Alimentarius, 2001). Hazards included in this study were nutritional, chemical and microbiological hazards. The review covered methods from the fields of natural/life (food) science, socio-economic sciences and food safety governance, published during the period 1993-2013. Risk ranking methods from fields outside food science (i.e. environmental sciences and socio-economic methods) were also included to evaluate their appropriateness for application in food science. The literature review followed the principles of a systematic literature review as described by EFSA (2010). A protocol for the structured literature review was defined a priori, including search strings and criteria for evaluation of the literature references (Annex 1).

Literature review

Review methodology

1. Scientific articles were identified using the following bibliographic databases: Web of Science, Scopus, PubMed, and CAB Abstracts. In addition, the general search engine Google was used to search for reports, (the “grey literature”), from relevant international and national organizations, authorities, and agencies (e.g., EFSA, EMA, WHO/FAO, FDA, Health Canada, OECD). The literature search focused on papers and reports published in English.

2. The set of search strings was applied leading to an initial set of search results. All retrieved references were stored in an Endnote database. Duplicates, a result of using four different bibliographic databases, were removed.

3. The references resulting from the initial set of search results were screened for their relevance to the study objectives by applying the evaluation criteria. A two-tier approach was used. In tier 1, the applicability of each reference to the review objective was determined by examining the title, abstracts and key-words of each reference. Based on this evaluation, the references were allocated to one of three categories and placed in the corresponding category of the Endnote database:

   	–	Relevant for this study: the reference was included;
   	–	Possibly relevant for this study: uncertain if the reference was relevant for the study;
   	–	Not relevant for this study: the reference was determined to be out of scope.

An inter-observer check was conducted with a randomly selected subset (10%) of both selected and excluded references.

• d. In tier 2, the full text of the references that were in the Relevant and Possibly relevant groups of the Endnote database were retrieved. By reading the full texts, the papers/reports were evaluated for their relevance to the field of interest and their quality using the evaluation criteria. When deemed relevant, the reference was retained or moved to the group Relevant in the Endnote database. When deemed not relevant, the reference was moved to the group Not relevant in the Endnote database. Also at this stage, an inter-observer check was conducted; certain (randomly chosen) literature references were evaluated by two experts from the team (from different disciplines) in order to gain insights into the variation between the evaluation results of two different experts.

• e. Citations used in the reports/references of the final Endnote database were screened for additional relevant references, published after 1993 (snowball citation), and steps c) and d) were applied to them.

Evaluation of references

For each reference stored in the Relevant category of the Endnote database, the risk ranking method and its characteristics were evaluated in depth. A summary of the information obtained was stored in an excel sheet, using a unique row for each reference. The format of the excel sheet was defined beforehand, starting from the template developed by EFSA's BIOHAZ panel (EFSA, 2012b), but with some modification to increase relevance to the objectives of the current study. Separate columns were utilized for information about the reference (author names, title, abstract, journal, volume and page numbers), and for storing the results from the critical evaluation of the risk ranking methods including: the type of tool (short description); field of application (microbiological, chemical, and/or nutritional hazards); what was ranked (e.g., specific food products); specific application area (e.g., pesticides); metrics, i.e., the type of method, with different sub-columns for each method category; model structure (quantitative, semi-quantitative or qualitative); data requirements that describe the model variables (e.g., human population data, or microbial numbers); method of data collection, describing how the necessary data were collected and which data sources were used, and finally data integration, describing how data were integrated in the application described in the reference. Based on this evaluation, the references and the evaluated methods were categorized into different groups of methods. The method categories were then described according to the following characteristics: scope, application area, approach, strengths and weaknesses, and perspective for use by risk managers. At this stage, reviews on risk ranking methods and other relevant literature were also consulted.

Results

Literature search

At tier 1, application of the search strings and removal of duplicates led to the retrieval of the following numbers of references (Table 1): 6021 for chemical/toxicological hazards; 2932 for microbiological hazards; 1049 for nutritional hazards; 112 references using health adjusted live years method; and 3358 references using socio-economic methodology. The latter two method groups were considered since they could potentially include each of the three types of hazards (microbiological, chemical and/or nutritional hazards). The total numbers of references appearing in tier 2 are somewhat higher than in tier 1 due to snowballing citations. In total 253 references were judged to be relevant.

Table 1. Results of the literature search in the two-tier approach.

	Tier 1: Title, abstract, keywords			Tier 2: Full text
Type hazard/field	Not relevant	Maybe relevant	Relevant	Not relevant	Relevant
Chemical hazards	5769	79	173	5943	101
Microbiological hazards	2601	74	257	2844	110
Nutritional hazards	979	58	12	1045	4
Health adjusted live years	90	13	9	98	18
Socio-economic methods	3296	47	15	3366	20

Description of risk ranking methods

Based on the evaluation of the methods described in the relevant references, the risk ranking methods were classified, according to methodology, into the following categories: 1) Risk Assessment (RA), 2) Comparative risk assessment (CRA), 3) Risk ratio method, 4) Scoring method, 5) Risk matrix, 6) Flow charts (including decision trees and influence diagrams), 7) Cost of illness (CoI), 8) Health adjusted life years (HALY), 9) Multi criteria decision analysis (MCDA), 10) Stated preference methods, and 11) Expert judgment. Table 2 shows the numbers of references that presented a particular method category, per type of hazard. All methods included both presence of the hazard and its severity. Method categories differed in the way in which these two factors were evaluated and combined to come to an estimate of the risk. In some instances, a combination of methods was applied, in which case the study was classified to its main category.

Table 2. Number of references per method categories for risk ranking of the food and/or nutritional hazards.

Type hazard	Risk assessment	Comparative risk assessment	Ratio	Scoring	Cost of illness	HALY	Stated preference^1	MCDA^1	Risk matrix	Flow chart/decision trees	Expert synthesis
Chemical	19	0	31^2	19^3	1^2	9^3^4	1^2	13	12	13	0
Microbiological	72	0	6^2	5^3	9^2	19^3	6^2	4	4	7	14
Nutritional	4	3	1	0	0	1^4	0	1	0	2	0
Other	0	0	0	0	0	0	1	1	0	0	1
Sum	95	3	38	24	10	29	8	19	16	22	15

¹ WTP: Willingness to Pay; HALY; health adjusted live years, MCDA: Multi Criteria Decision Analyses;

² One reference described both chemical and microbiological hazards;

³ Three references described both chemical and microbiological hazards;

⁴ One reference described both chemical and nutritional hazards.

RA was by far the most frequently applied method. This method was applied to both chemical and microbiological hazards. For each of the chemical and microbiological hazards, about one third of all tier 1 references described the application of a RA to a particular hazard. However, as the procedure for each of the chemical and microbiological RA is comparable, only references describing guidelines for performing a RA were included. Risk ratio, scoring, risk matrices and flow charts were mostly applied to chemical hazards, whereas CoI, HALY, and expert judgments were mostly used for ranking microbiological hazards (Table 2). Ranking methods for nutritional hazards were fewer, and were mostly based on RA, CRA and expert judgment (Table 2). CRA, CoI, and stated preferences were the methods that were applied least frequently, with CRA used in three studies about nutritional hazards, and the latter two methods primarily applied to microbiological hazards. A few studies have considered both chemical and microbiological hazards in their ranking, applying methods for CoI and HALY. Summaries of each method and characteristics are presented in the following sections and in Table 3.

Table 3. Characteristics of risk ranking methods related to food safety.

Characteristic	Risk assessment	Comparative risk assessment	Ratio (exposure/effect)	Scoring method	Cost of illness	HALY^1	WTP^1	MCDA^1	Risk Matrix	Flow charts/decision trees	Expert synthesis
Amount of resources (time, money)	High	High	Moderate	Moderate	Moderate	Moderate	High	High	Low	Low	Moderate/Low
Level of output	Quantitative	Quantitative	Semi-quantitative	Semi-quantitative	(Semi-) quantitative	(Semi-) quantitative	(Semi-) quantitative	Semi-quantitative	Qualitative/semi-quantitative	Qualitative	Qualitative
Easy to explain to stakeholders (laymen)?	No	No	Yes	Yes	Yes	No	No	Yes	Yes	No	Yes
Inclusion stakeholder perception	Not possible	Not possible	Not possible	Possible	Not possible	Not possible	Possible	Possible	Not possible	Possible	Possible
Inclusion uncertainty	Possible	Possible	Possible	Possible	Possible	Possible	Possible	Possible	Not possible	Not possible	Possible
Inclusion weights for the risk ranking criteria	Not possible	Not possible	Not possible	Possible	Not possible	Not possible	Not possible	Possible	Not possible	Not possible	Possible
Inclusion human incidences	Possible	Possible	Not possible	Not possible	Possible	Possible	Possible	Possible	Not possible	Possible	Possible
Inclusion economic impact	Not possible	Not possible	Not possible	Not possible	Possible	Not possible	Possible	Possible	Not possible	Possible	Possible
Common method of communication (in addition to reports)	Graphs/Tables	Graphs/Tables	Tables	Tables	Graphs/Tables	Graphs/Tables	Graphs/Tables	Graphs/Tables	Graphs	Decision Tree	Tables
Essential data needed
Human incidence data needed?	No	Yes	No	No	Yes	Yes	Yes	No	No	No	No
Dose-response data needed?	Yes	Yes	No	No	No	No	No	No	No	No	No
Occurrence data (concentration, prevalence, dose) needed?	Yes	Yes	Yes	Yes	No	No	No	No	No	No	No
Food consumption data needed?	Yes	Yes	Yes	Yes	No	No	No	No	No	No	No
Growth models needed (only applicable for microbiological hazards)?	Yes	Yes	Yes	Yes	No	No	No	No	No	No	No
Toxicological reference values (ADI, TDI etc) needed (only applicable for chemical hazards)?	Yes	Yes	Yes	Yes	No	No	No	No	No	No	No

¹ WTP: Willingness to Pay; HALY; health adjusted live years, MCDA: Multi Criteria Decision Analysis

Risk assessment

Scope: A RA for a chemical or microbiological hazard aims to estimate the risk for human health associated with the presence of the hazard in one or more food products, and total food consumption. Numerous risk assessments have been applied to chemical and microbiological hazards in food. WHO (WHO, 2009) and Codex Alimentarius (2014) have provided guidelines regarding the principles and methods for the risk assessment of chemical contaminants and pathogens in foods. Although the application of the RA methodology is tailored to the hazard type, the principles for performing a risk assessment for both types of hazards are identical, consisting of the following four steps: hazard identification, exposure assessment, hazard characterization, and risk characterization.

Application area: Risk assessment is usually applied for one identified (chemical or microbiological) hazard occurring in a specific food commodity and for a predefined population, with the purpose of characterizing the associated health risk. Apart from this, an important reason for conducting a RA is to evaluate the impact of control measures to reduce the risk. If the results of different RA are compared (e.g. for different hazards or different foods), the RA can be used for risk ranking.

Approach: Various RA approaches for chemical and microbiological hazards in food were identified, applying different combinations of deterministic, probabilistic (or stochastic), qualitative, semi-quantitative, and quantitative modeling. Furthermore, different approaches were used for the exposure assessment and the hazard characterization steps. EFSA (2011) published an overview of procedures for current RA methods for dietary exposure of different chemical substances. The need for development of harmonized approaches, and future exploration of cumulative exposure assessments, is identified. In 2012, EFSA published its experiences gained with Quantitative Microbiological Risk Assessment (QMRA) studies (EFSA, 2012a).

Strengths and weaknesses: In RA, all available scientific and technical information and data, as well as variability and uncertainties are systematically organized and analyzed. It is a well-structured method, providing insights into what is known and what is not known. In particular, RA offers the opportunity to address uncertainties in a transparent way, e.g., via sensitivity analyses and/or modeling and simulation runs. It could be the most precise method to estimate risks, including the relevant uncertainties. However, a RA for one chemical or microbiological hazard usually requires a lot of time, data and knowledge. Ranking risks related to various hazards in food using outcomes of individual RAs will take even more resources and RAs are often hampered by a lack of quantitative data. Lack of data, selection of models to fit to the data, and assumptions that need to be made give rise to uncertainties in the outcomes. Recently, several tools for relative risk assessment for pathogens of pathogen-food combinations have been published. Examples of such tools applying quantitative methods are the swift QMRA tool (Evers and Chardon, 2010) and iRISK, which is a relative risk assessment system for evaluating and ranking food-hazard pairs (Chen et al., 2013, see http://https://irisk.foodrisk.org/). An example of a semi-quantitative approach is Risk Ranger (Ross and Sumner, 2002) developed by Food Safety Centre (2010).

Perspective for use by risk manager: Applied optimally, RA should disseminate key information regarding risk from exposure to food hazards to policy makers, decision makers and the public. RA are very useful for providing insights into gaps in knowledge and issues associated with high levels of uncertainty. However, they may not be suitable for risk ranking given the large amounts of data, knowledge and resources needed.

Comparative risk assessment

Scope: A Comparative Risk Assessment (CRA) analysis can estimate the number of deaths that would be prevented in a given period if current distributions of risk factor exposure were changed to a hypothetical alternative distribution (Danaei et al., 2009; Micha et al., 2012). In these papers, CRA is restricted to comparisons of deaths and it is, therefore, not comparable to a risk assessment or a relative risk assessment.

Application area: Three applications of CRA have been found; each of them studied the impact of dietary factors on disease mortality. Danaei et al. (2009) performed a CRA analysis for establishing the preventable causes of death associated with dietary, lifestyle and metabolic risk factors in the United States. Micha et al. (2012) used a CRA framework to develop methods for assessing the global impact of specific dietary factors on chronic disease mortality. Lim and co-workers (2012) investigated burden of disease and injury attributable to 67 risk factors (including chemical hazards and nutritional imbalances) in 21 regions through application of a systematic analysis for the Global Burden of Disease Study 2010. Although a CRA analysis as described below was not performed by Lim et al. (2012), several elements of a CRA analysis were included.

Approach: A CRA analysis is measured in population attributable fractions (PAFs), which describe the total effects of a risk factor (direct/indirect) by reflecting the proportional reduction in deaths for each disease causally associated with the exposure that would occur if the usual exposure distribution had been reduced to the optimal minimum-risk exposure distribution. Input needed to determine the PAF include: a) effect size (relative risk estimate) of the causal diet-disease relationship, b) optimal or theoretical minimum-risk exposure distribution, c) dietary risk factor exposure distribution in the population and, d) total number of disease-specific deaths (plus non-fatal events, when available) in the population. Data sources for obtaining these inputs include epidemiological studies, systematic reviews, meta-analysis, nationally representative nutrition surveys and mortality databases.

Strengths and weaknesses: A CRA analysis is a systematic assessment of unbiased data collected in national and international surveys as well as the peer reviewed literature. It allows for consistent, comparable and quantitative assessment of the global impact of risk factors on disease by sex- and age-specific groups. A CRA analysis requires knowledge and resources (manpower, money, data), which makes it expensive to perform. Unbiased data are also needed, e.g., to establish exposure distributions or causal diet-disease relationships, which may often not be easily accessible or available. The weights of different diseases are not considered. Uncertainties associated with a CRA analysis can be high because of data limitations.

Perspectives for use by risk manager: A CRA analysis offers a global assessment of the impact of dietary factors on disease mortality, which is very valuable for priority setting and policy making. However, with large and overlapping uncertainty ranges for the different risk factors, ranking of modifiable dietary risk factors may be difficult.

Risk ratio method

Scope: Risk ratios or quotients refer to a quantitative method in which estimates of exposure are divided by estimates of effect. For this purpose, data are needed regarding the amounts of the hazard consumed (either the dose or the concentration) as well as a measure for the effect of the hazards that are studied.

Application: The risk ratio method has usually been applied to rapidly screen the risk of a range of chemical compounds in order to rank them. Most studies applied the method to rank pesticides, although five studies focused on microbiological hazards, and one study applied the method to rank both chemical and microbiological hazards.

Approach: For chemical contaminants, some references derive a Hazard Index, in which the Estimated Daily Intake (EDI) is divided by the Acceptable Daily Intake (ADI), Tolerable Daily Intake (TDI) or the acute Reference Dose (RfD) (Calliera et al., 2006; Sinclair et al., 2006; Oldenkamp et al., 2013). The Margin of Exposure (MoE) approach is another method in which exposure and effect are compared by dividing the NOAEL (No Observed Adverse Effect Level) or the BMD (Bench Mark Dose) by the EDI (Rietjens et al., 2008; Madsen et al., 2009; Bang et al., 2012). The Hazard Index should be as low as possible, whereas the MoE should be as large as possible to obtain a low risk for human health. In general, the risk of pesticide residues for human health is ranked using the Hazard Index (e.g., Sinclair et al., 2006; Travisi et al., 2006; Whiteside et al., 2008; Labite and Cummins, 2012), whereas the risk of carcinogenic compounds is primarily ranked using MoE (Dybing et al., 2008; Lachenmeier et al., 2012). Applications of the method to microbiological hazards used different criteria, such as costs and effective dose.

Strengths and weaknesses: This method is easy to understand, and can be applied once concentration data and toxicological reference values are available; it only needs an estimate for both amounts of the hazardous material consumed and the effect of the hazard on human health. For emerging chemical hazards, e.g., nanomaterials, toxicological reference values are usually not available.

Perspectives for use by risk manager: The method can give a quick answer on the risk of food safety hazards for human health, and can be applied to both chemical and microbiological hazards.

Scoring method

Scope: This method is based on semi-quantitative scoring of both exposure and effect of the hazard on human health, followed by their multiplication (or—in one reference—addition).

Application: Scoring methods provide a simple risk ranking method to characterize chemical hazards for subsequent categorization into particular groups (Greim and Reuter, 2001; Bietlot and Kolakowski, 2012; Aylward et al., 2013; Bu et al., 2013; Taxell et al., 2013; van Asselt et al., 2013).

Approach: When a scoring method is applied, both exposure and severity (or effect) endpoints are considered. However, endpoints for exposure and effect can vary. Various endpoints have been used to estimate exposure, such as chemical transformation properties (degradability, half-life), mobility/distribution (such as bioaccumulation factors (BAF) or bioconcentration factors (BCF)), release, frequency of detection, and dose administered/concentrations. There is currently no scientific consensus on which endpoints to include and how to set criteria for classifying these endpoints. Consequently, selection of appropriate endpoints for a specific study is one of the steps in ranking risks according to a scoring method. Examples of endpoints for effect on human health might include acute toxicity, carcinogenicity, or reproductive toxicity, and can be based on LD50, MOAEL, BMDL10 etc. Once criteria are set, endpoints are classified semi-quantitatively, e.g., using scores from 1 to 3 or from 1 to 5, as applied in, for example Penrose et al. (1994).

After this classification system for endpoints has been established, data sources need to be found in order to assign scores for exposure and effect. These sources can be based on literature, available data and/or expert opinion. Scores subsequently need to be aggregated, which is mainly done by multiplying exposure and effect (see, e.g., Gamo et al., 2003; Juraske et al., 2007; van Asselt et al., 2013), although one study added the scores (Penrose et al., 1994). Some references also employ a weighing system to weigh the various endpoints included in the assessment (Penrose et al., 1994; Valcke et al., 2005; Juraske et al., 2007; Dabrowski et al., 2014). A general framework for risk ranking that includes the choice of endpoints, weighing endpoints and aggregating the scores into a final risk score is depicted in Figure 1.

Figure 1. Framework for risk ranking of chemicals, adapted from Bu et al. (2013).

Strengths and weaknesses: This semi-quantitative method is easy to conduct once scores have been assigned to the model variables. Furthermore, it allows the inclusion of stakeholder perceptions in assigning the scorings and the importance (to each stakeholder) of each model variable is reflected by the weighting allocated to it. The assigned weights should then be clearly documented to guarantee a transparent approach.

Perspectives for use by risk manager: Stakeholders can use this method to obtain a clear overview of prioritized risks in relation to food safety hazards. The method has been used as input to the establishment of national monitoring programmes (VRC, 2010).

Risk matrices

Scope: Just like the scoring methods, risk matrices also make use of scoring both exposure and effect endpoints. The difference between scoring methods and risk matrices is that, in the latter, the exposure and effect endpoints are not aggregated by multiplication or addition, but are depicted in a risk ranking matrix with effect on the one axis and exposure on the other.

Application: This method is usually applied to chemical or microbiological hazards for which limited quantitative data are available. This method has, for example, been applied for ranking the risks of nanomaterials (Zalk et al., 2009; Sorensen et al., 2010; O'Brien and Cummins, 2011).

Approach: Both the likelihood of occurrence and the consequences of the hazard for human health are scored into one of several classes; see Figure 2 for an example. Classes that could be used for likelihood of occurrence are: almost certain, likely, possible, unlikely and rare. Classes that could be used for the consequences are: insignificant, minor, moderate, major and severe. The division into these classes is subjective. Then, risk classes are assigned to the combinations of Likelihood and Consequences, e.g., being L (low), M (moderate), H (high), and E (extreme), as shown in Figure 2. Risk classification may also be based on scores. Zalk et al. (2009), for example, classified nanomaterials based on scores for probability and severity, and the results were depicted in a risk matrix. The results can also be visualized using spider web plots, as conducted by, e.g., Ranke and Jastorff (2000), who classified various endpoints using scores from 1 to 4, and compared plots for the various compounds to obtain an indication of the most risky ones.

Figure 2. Example of risk matrix.

Strengths and weaknesses: The risk matrix method is qualitative or semi-quantitative, and thus less accurate than methods based on concentration data and dose-response relationships or toxicological reference values. It provides a visualization for both presence of the hazard and its effects, giving direct insights into the way these two elements contribute to the overall risk of a hazard. For example, a hazard may present a high risk due to a high exposure, although its severity is low. Alternatively, due to its high toxicity, it may present a high risk rank despite low exposure. Matrices will give more information to the risk manager compared to other methods that produce a list of hazards according to the overall risk alone. However, the division between different categories for presence of the hazard (e.g. low, medium high occurrence) and its effects (e.g. low, medium, high toxicity) is subjective and, thus, other results are obtained when with other divisions.

Perspectives for use by risk manager: In case stakeholders prefer a graphical representation of the risks, this method can be used to visualize both the effect and the exposure of a hazard. This facilitates discussions amongst stakeholders regarding the risks of various hazards.

Flow charts

Scope: Flow charts or decision trees are based on a set of clearly defined questions or criteria. By following these,, the hazards can be classified into different categories (e.g., high, medium or low) with respect to their risk for human health.

Application: Flow charts or decision trees can be used for various purposes. In general these methods are used to obtain a qualitative indication about the risks associated with hazards. Haase et al. (2012), for example, established a decision tree for nanoparticles to determine whether a full risk assessment is required or not. EFSA described guidelines for classifying chemical hazards as negligible, low, medium, and high risks (EFSA, 2012c, 2012d).

Approach: A flow chart is generally based on several questions that need to be answered in order to arrive at a certain risk class. Questions can be based on the likelihood that specific chemicals or microbiological hazards are present in the study object; evidence of occurrence or incorrect practice in the food chain, the toxicological profile, and the outcome of national monitoring programmes (EFSA, 2012c, 2012d). Eisenberg and McKone (1998) used a Classification and Regression Tree Algorithm (CART) to specify the chemical and environmental properties and Monte Carlo simulations to estimate human exposure. Schmidt et al. (2011) utilized a decision support system (DSS) to rank genetically modified organisms (GMOs), based on a decision tree and rules, indicators and baselines, and thresholds (such as the LD50) (Schmidt et al., 2011). DSS may also be combined with multi-criteria decision analysis (MCDA). Critto (2007), for example, utilized a DSS system to evaluate ecological observations and ecotoxicological tests for contaminated sites and then incorporated MCDA and expert judgments into the ranking. This approach might also be used for ranking food safety risks.

Strengths and weaknesses: Flow charts/decision trees present a straightforward method with clear questions for which only qualitative information is needed, although quantitative information can be used where available. The method can, thus, be used for a quick screening of food safety hazards, in order that the most relevant ones may subsequently be investigated in more detail. However, this method strongly depends on expert input and it is, therefore, essential to perform a rigorous expert elicitation study. Furthermore, this type of method is vulnerable to being less transparent than other methods, as it is not always clear why hazards end up being classified as a high, medium or low risk. Therefore, for each hazard classified based on a decision tree or flow chart, the underlying reasons for the answers should be clearly documented in order to obtain a transparent classification.

Perspectives for use by risk manager: It is important to set up the right questions for inclusion in a flow chart/decision tree based on expert judgment and scientific evidence, which may be challenging to achieve. However, once a decision tree has been drafted, it is easily applicable for stakeholders to classify hazards into high, medium and low risks.

Cost of illness method

Scope: The underlying research objective of the Cost of Illness (CoI) approach is distinct from those of the methodologies described so far. CoI studies acquire data for conducting economic analysis in order to obtain a ranking in terms of how society might allocates scarce resources when addressing food-related hazards. The procedure involves calculating the directs costs to society related to disease and death due to chemical, microbial and/or nutritional hazards. It can be applied wherever there are quantitative data relating to the impact of disease (severity and duration; mortality) and sufficient cost data for calculating resultant treatment costs and loss of income. Subject to data availability, it is possible to compare large numbers of food risks.

Application area: This approach can be applied for comparing diseases (Gadiel, 2010), for food-disease combinations (Batz et al., 2011), and for supply chain analysis of a single food-disease combination (Miller et al., 2005).

Approach: The starting point of this quantitative method is the construction of a separate disease outcome tree (or equivalent) for each illness under consideration. This will show the numbers (and proportions) of the affected population who experiences each type of impact, defined as the disease severity class. A critical point is whether it is restricted to acute effects, or whether long-term effects (sequelae and deaths) are also included. This will be particularly important for diseases for which some affected individuals will experience life-long disease, or where medical problems may be latent for a period (e.g., toxoplasmosis).

If possible, the disease outcome tree is populated directly from existing data sources. However, data for disease incidence and attribution to a specific food source is often incomplete. The problems with inadequate or missing data are sometimes overcome by expert elicitation of (ranges of) parameter values (e.g., Golan et al., 2005; Batz et al., 2012). To address uncertainty caused by inadequate data, sensitivity analysis (e.g., Batz et al., 2011) or frequency distributions can be used in Monte Carlo or stochastic simulation models (Kemmeren et al., 2006; Lake et al., 2010). The costs incurred at each state are calculated, often including the categories of direct health costs, indirect health costs, and indirect non-health costs.

CoI studies generally make use of discounting by which the value of earnings and payments incurred in the future are expressed in terms of their present value. They are expressed as a given amount of money invested today at a given interest rate (or discount rate) (Crutchfield et al., 1999). By definition, discounting does not apply to the costs of health effects whose duration is shorter than one year, whereas other end-points, such as life-long disabilities, are strongly affected by discounting. Hence, the effect of discounting will differ per hazard (Kemmeren et al., 2006) and the rate of interest selected.

Strengths and weaknesses: The CoI method employs readily available and reliable data (Buzby et al., 1996) and the calculations are transparent and relatively simple. The same disease incidence data are used in HALY calculations so it is relatively efficient to produce both sets of rankings at the same time and they are, to some extent, complementary. A combined risk ranking can also be produced. A CoI ranking diverges from most measures of disease severity or social welfare (Golan et al., 2005) because CoI estimates are restricted to market goods. Therefore, apart from medical costs, the measures excludes non-workers, and do not address perceived quality of life including factors such as pain and stress (Golan et al., 2005). A further important weakness relates to the lack of accurate public health and attribution data, which is the biggest cause of uncertainty in CoI estimates. The results are dependent on the assumptions made inter alia about medical outcomes and the prevailing labor market.

Perspectives for use by risk manager: CoI is a well-tried technique with well-understood limitations relating to missing data, and failure of the approach to adequately include non-working members of society and quality of life impacts. Large numbers of risks can be ranked. The process appears highly transparent, but it should be remembered that the cost coefficients and incidence data may be derived from inadequate data, so sensitivity analysis is advisable. Due to non-standardization of technique (e.g. different components, and assumptions), comparability between studies is awkward.

Health adjusted life years (Burden of Disease)

Scope: “Health adjusted life years (HALY)” are nonmonetary health indices, where the actual health of an individual is compared with a perfect health situation (usually on a scale from 0 to 1) and this score is then multiplied by the duration of that health state. A descriptive summary of the various HALYs is presented by Mangen et al. (2014).

Application area: HALY measures may be applied when the ranking of hazards is to consider the level of human disease or loss of productive capacity for the exposed population, i.e., the burden of disease. HALY estimates such as disability adjusted life years (DALYs) or quality-adjusted life years (QALYs) may be used as the only parameter for risk ranking, but are often included as one of several parameters in a risk ranking model. The DALY method was developed at the WHO, and the Global Burden of Disease (GBD) study is the most often referenced source of disability weights for specific disease outcomes (ww.who.int/healthinfo/global_burden_disease/metrics_daly/en/). The HALY approach has been applied to rank different pathogens and chemical contaminants in the same food category, different hazard-food category combinations, or summarized and ranked for different food categories. Estimates of DALYs or QALYs have also been used to rank waterborne contaminants in lakes or water supplies as well as for ranking human risk factors in general.

Approach: Data are required for estimating the number of cases with the most relevant types of acute illnesses, chronic sequelae and mortality (also termed health outcomes) arising from exposure to the hazards under consideration. Different types of hazards (chemical, microbiological or nutritional) require different types of data and modeling approaches (Crettaz et al., 2002; Hofstetter, 2002; Pennington et al., 2002; Mangen et al., 2010; Mangen et al., 2014), but after the final DALY/QALY calculations have been made, the risks estimates should be readily comparable. DALY/QALY estimates may also be included in several of the other risk ranking methods, such as RA (Howard et al. (2007); Newsome et al. (2009)), CRA (Lim et al. (2012)), MCDA (Ruzante et al. (2010)), risk matrixes, flow charts/decision trees or in expert syntheses.

Strengths and weaknesses: HALY methodologies readily allow comparisons between very different types of hazards, not only food-related hazards but all types of human risk behavior over time and geographical regions as presented by the Global Burden of Disease Study 2010 (Lim et al., 2012) and ECDCs initiative for developing methodologies for measuring current and future burden of communicable diseases (Mangen et al., 2014).

DALYs and QALYs are semi-quantitative estimates based on disability scoring, and their accuracy is highly dependent on the quality of input data and risk assessment models used for estimating the incidences of relevant health outcomes. In the applied studies, the methods for estimating the incidences of relevant health outcomes varied widely. The estimated DALY or QALY values seem to be relatively precise quantitative estimates, and there is a risk of over-interpretation of the relative differences, if the level of uncertainty is not addressed. A general methodological weakness is inadequate evidence to estimate the incidences of chronic disability, especially in cases with few or no symptoms during the acute phase of a disease. Another methodological weakness is that the concept of DALYs assumes a continuum from good health to disease, disability, and death which is independent of time—a concept not universally accepted. Also, stakeholders have difficulty to understand the concept and what is meant by it.

Perspectives for use by risk manager: Tools are readily available for calculating DALYs for a range of infectious diseases including foodborne zoonoses in the EU (BCoDE tool from ECDC). If RA or models for estimation of reported cases are available, the resources needed to estimate DALYs are moderate. However, development of RA models to estimate the number of diseased individuals can in some instances be very time-consuming.

DALY or QALY estimates can be viewed as an economic measure of human productive capacity, enabling ranking of the “societal production losses” related to the included hazards. If HALY estimates from different studies are to be used in risk ranking, then differences in the methodology employed and the comparability of the studies must be considered. For monitoring purposes, risk ranking models estimating HALYs can be constructed so that yearly input of surveillance and population data can be entered, as done for the food borne pathogens in the Netherlands (Bouwknegt et al., 2013).

Multi-criteria decision analysis (MCDA)

Scope: MCDA is an approach which has the potential to evaluate multiple—often conflicting—criteria in decision making. It allows for comparison of different risks on common basis, by simultaneous consideration of technical information, uncertainty and different stakeholder preferences, both quantitative and qualitative data, and the integration of large amounts of complex information. MCDA helps structuring and solving problems, such to enable making more informed and better decisions. In the context of risk ranking, important criteria utilized in food safety can be identified through a process of expert or lay consultation, which may include not only public health impacts but also perception, costs—an in case of interventions—also weight of evidence, and practicality associated with the interventions. Application area: MCDA can be applied to any range of problems, which can be defined in terms of a common set of criteria. As the scientifically “best” solution may be inadequate in terms of acceptability to society, utilize resources which or not available, or be sub-optimal in terms of allocating resources, stakeholder methods are sometimes used to capture the preferences of consumers, citizens and/or experts. MCDA which combines expert judgment across a range of relevant criteria appears to be the second most popular method for relative risk ranking of microbiological hazards, after RA.

Approach: MCDA is a semi-quantitative method in which a range of different criteria are identified against which each problem is assessed. Participants, either experts, stakeholders or lay people (Fazil et al., 2008), can be supplied with technical information in relation to each risk criterion to assist their deliberations. The selection of preference functions and weights are an integral and core part of the MCDA methodology and must be selected when conducting a risk ranking. An example is provided by Ruzante et al. (2010) who utilized the method to develop a prioritization framework for foodborne risks that considered not only public health impacts but also market impact, consumer risk acceptance and perception, and social sensitivity. Another well-known example of a MCDA method for ranking pathogen-produce combinations is the Pathogen-Produce Pair Attribution Risk Ranking Tool (P³ARRT) developed by FDA (Anderson et al., 2011), which is available free (http://foodrisk.org/exclusives/rrt). Fazil et al. (2008) applied MCDA for the ranking of food safety interventions, considering amongst others cost, effectiveness, and weight of evidence. MCDA methods and applications vary in their complexity; they may even allow for probabilistic modeling and sensitivity analyses. Recently, alternative methods for performing a MCDA have been developed and employed, e.g., by Havelaar et al. (2010), in order to minimize the biases linked with experts' direct weighting of the MCDA criteria.

Strengths and weaknesses: MCDA allows consideration of stakeholder perceptions by using the weights and preference functions they assign to the various criteria in the analysis. Furthermore, economic impact or other criteria that are deemed relevant can be included, in addition to human health criteria. This makes the method broadly applicable, allowing risk assessors/managers to determine the impact of various criteria on the overall risk ranking of hazards. This method, therefore, allows inclusion of subjective elements that may also be important for risk managers to include in their decision making processes, depending on the aim of the ranking exercise. Alternative scenarios using weights and preference functions for various input factors can be compared. However, MCDA outcomes are more difficult to communicate compared to more straightforward methods such as risk matrices or scoring methods, as various criteria are included, which are weighted and prioritized differently. Furthermore, this method needs expert or stakeholder input in order to derive the weights and preference functions for the criteria. Therefore this method has weaknesses that are linked to the elicitation of information from experts (see below), i.e., the need for having rigorous, auditable methods to identify experts; high demand for resources (as training of experts in these methods and specialized risk analysts and modelers may be needed); the need to consider how to elicit experts' own uncertainties regarding their views, opinions, judgments; and—last but not least—the need to consider possible ways to combine individual opinions without masking variability in the experts' views.

Perspectives for use by risk manager: This systematic method is very valuable in cases where stakeholder perceptions are required to be included in the risk ranking, as weights and preference functions can be assigned to the various model variables. This method also allows the inclusion of factors other than effect and exposure endpoints, e.g. from the social-economic field, or in terms of policy development, which makes it a very versatile tool. The application of MCDA will provide a single number for ranking. However, the underlying calculations can be difficult for the non-expert to understand for those without expertise in the methodology.

Stated preference methods

Scope: Stated preference methods could be used to elicit the preferences of individuals (citizens and households) for reducing the risk from a range of food-related diseases. When aggregated they show society's preferences for risk reduction. These methods take into account the concerns and perceptions of society and, consequently, the ranking produced may be different from that produced by experts on technical grounds alone.

Application area: There is a relatively long history of the use of stated preference techniques for valuing non-market goods in the analysis of environmental problems. So far, their application in ranking food safety risks is limited and largely confined to valuing individual disease reduction measures or comparing alternative risk management options within single food-disease problem, see e.g., Mørkbak & Nordström (2009) and Miller et al. (2005). Golan et al. (2005) concluded that, at present, there is not a coherent set of guidelines for conducting such studies, making comparability between studies difficult. In theory, these methods could be used to rank diseases, disease-food combinations, or stages in supply chains. However, it is a complicated technique to use, which might explain the lack of use for ranking more than a small number of alternatives.

Approach: Using stated preference methods, a simulated market is constructed and monetary values are derived from hypothetical questions. The methods include stated preference techniques (contingent valuation and discrete choice experiments) and averting behavior or preventative expenditure, which is the cost of preventing illness. In contrast to the CoI approach, stated preference methods include the value individuals place on other factors for which no markets exist such as, for instance, (not) experiencing pain. Stated preference methods are also able to include the value of lost health in people who are not in the labor force (e.g. retired) who are excluded from CoI calculations.

One of the stated preference methods, willingness to pay (WTP) rests on the observation that people make trade-offs between health and other goods and services. The approach elicits the resources an individual is willing to give up for a reduction in the probability of encountering a hazard that will compromise their health (Golan et al., 2005). As an example, Mørkbak and Nordström (2009) conducted a choice experiment to elicit WTP for campylobacter-free chicken as compared to the alternatives, non-labeled chicken and outdoor-reared chicken; in other words, the WTP for higher food safety compared to the current level. This approach defines the choices which individuals make in terms of the levels of key attributes (such as high/low price, probability of illness, etc.) which are associated with each of the goods being compared.

Strengths and weaknesses: WTP is generally viewed as the most complete and correct economic welfare measure of the benefits of food safety policies. This is because, like CoI, WTP includes the cost of treatment and lost productivity but also (unlike CoI) changes in consumer welfare, such as pain, distress and inconvenience (Hoffmann, 2010). Both individual and societal WTP can be calculated. A useful feature is that stated preferences may be linked to participant profile revealing which societal groups (e.g., by age, background) ranks a particular risk most highly (see Haninger and Hammitt (2011) for an example). The aggregated value of benefits (or societal WTP) of food safety (e.g., reduced risks) can be compared with the costs for achieving them since both costs and benefits are expressed in monetary units.

However, WTP is a difficult technique to apply, and is prone to errors and bias unless conducted meticulously. Experience so far has been in comparing only 2 to 4 alternative risks. It may be possible to elicit mean WTP for a larger number of risks, but the scope of choice experiments may be limited by the capacity of participants to choose between a large number of choice sets encompassing many attributes. Moreover, WTP reflects the ability to pay, and implicitly assumes that the existing distribution of resources in society is acceptable (Golan et al., 2005). However, because WTP studies can produce results segmented by sub-population, they may draw attention to unequal distributional impacts which should be considered in policy making.

Perspectives for use by risk manager. These techniques provide a means to incorporate societal preferences in ranking and decision making. However, experience in the food safety field as yet is only modest, and there is scope to develop techniques still further.

Expert judgment

Scope: Expert judgment-based methods elicit rankings from citizens, stakeholders or other experts, and have the potential to produce a systematic and transparent ranking of risks.

Application area: Three principal applications of judgment-based risk ranking were identified: (a) achieving a ranking when there are data gaps, (b) reconciling the diverse information streams and considerations encountered in multi-attribute problems, and (c) incorporating societal values, e.g., Moffet (1996). The inclusion of public perceptions, priorities and values may result in a different ranking being reached to that derived from using scientific experts alone. This might reflect public concerns such as whether the distribution of costs and benefits is equitable, the characteristics of the people likely to be affected (e.g. children or elderly people), whether exposure to the risk is voluntary or involuntary, and whether there is ‘dread’ or fear of a catastrophic impact (DeKay et al., 2005).

Approaches: A variety of methods is available, for application in workshops or in surveys, which may be characterized by the flows of information which take place between the participants and the research team (Rowe and Frewer, 2005). There may be a one-way flow of information from experts (or other stakeholders) to researchers, which aims to capture participants' existing knowledge and experience. Alternatively, there may be a two-way flow, whereby participants are provided with detailed scientific and socio-economic information on which to base their deliberations and ranking, which is finally communicated to the researchers. Formal semi-quantitative techniques exist to combine divergent data sources, e.g., MCDA and the Carnegie-Mellon approach. In MCDA, the judgment of stakeholders is used to allocate weights and potentially also on the way to weight the different criteria and in establishing the preferences to the different attributes whereas the Carnegie-Mellon approach produces risk rankings. Approaches also vary according to whether they involve experts or lay people, the amount of technical information about risks and impacts that is provided to assist study participants, whether the approach is qualitative or semi-quantitative, and whether or not the process involves deliberation among participants. Four approaches were identified:

–	Expert elicitation, defined as a set of formal research methods used to characterize uncertainty about scientific knowledge and to provide alternative parameter estimates when there are meaningful gaps in available data (Batz et al., 2012). Commonly used approaches are workshops and the Classical Delphi method (Van der Fels-Klerx et al., 2002).
–	Survey based on existing knowledge of lay or expert participants (i.e. minimal technical communication during the study), as applied by, e.g., Schwarzinger et al. (2010) and Harrington (1994).
–	Ranking achieved through deliberation only, or deliberation with supporting technical information (e.g. focus group or workshop). Although the ranking process may be restricted to a panel of experts considering scientific data only (e.g. FAO/WHO, 2008), there is also the possibility to involve lay people and thus capture societal values.
–	Carnegie-Mellon approach which was specifically developed as a standardized procedure by which several risks could be ranked, and involves the elicitation of the explicit preferences of lay groups (DeKay et al., 2005). The basic procedure requires expert technical inputs to define and categorize the risks to be ranked, to select attributes by which the risks are characterized, and to prepare risk summary sheets to assist deliberations on each risk (Florig et al., 2001).
–	Ranking of risks is performed by lay people (not experts) in a workshop setting according to their levels of concern about the risks, having considered the information provided on the risk summary sheets. If used, weights for each attribute are obtained from each participant and reflect social value judgments. The procedure used for weighting is much simpler than that typically used in MCDA (DeKay et al., 2005).

Strengths and weaknesses: Judgment-based methods provide additional information to that of technical assessments, e.g., when a problem is poorly understood, or technical data are incomplete. The outputs commonly include a narrative component which can make explicit the interpretations and assumptions which underlie the final ranking, as well as identifying the difficulties and uncertainties which determine its limitations. They also provide a means of engaging the general public in evaluative and decision-making processes and of incorporating societal preferences for different alternatives. However, judgment-based methods require a very careful design if they are to provide valid outcomes. Biases are introduced by a number of means including: inappropriate selection of the participants; the framing of the problem(s) for consideration; the way the process is conducted such that the whole range of opinions may not be elicited and recorded, and the content of the technical information that is presented to participants (e.g. bias, comprehensibility, acknowledgment of its limitations). Due to this need for meticulous preparation the method is often resource intensive. Furthermore, a qualitative analysis of data (if required) makes heavy time demands both in the transcription of audio recordings and their subsequent (thematic) analysis.

Perspectives for use by risk manager: Unless judgment-based methods are planned and executed well there is a danger that they will be biased and unreliable. Depending on the specific method, the output may be a simple ranking, but could also be a lengthy narrative which, though having explanatory power, requires lengthy consideration. These methods can provide input in cases where crucial data are missing, and a decision needs to be made. Also, they could provide a means of incorporating societal values into risk ranking.

Discussion and conclusions

A literature review has been performed on methodologies for ranking risks related to chemical, microbiological and nutritional hazards in food, on the basis of their anticipated effects on human health. The results showed that a range of risk ranking methodologies has been applied depending on the purpose of the specific study. They have been grouped into eleven main categories, determined primarily by the type(s) of hazard that can be ranked, data needs, and uncertainty. Some methods allow ranking of different hazards types (chemical, microbiological), whereas others allow ranking only within one hazard category.

Four of the eleven method groups can be applied to all three types of hazards (microbiological, chemical and nutritional), either alone or in combination, these being MCDA, risk matrices, stated preferences techniques, and expert synthesis. For microbiological hazards, there is a close relationship between exposure and resulting levels of illness and death, which allows CoI and DALY/HALY calculations to be made. With chemical contamination of food, there is no such direct relationship between the contamination and resulting diseases/deaths in the population, since effects on human health are long-term and, hence, the cause-effect relationship is difficult to establish. Consequently, these methods are not often applied to chemical food contamination, although an exception is the study by Kemmeren et al. (2006) who calculated DALYs for chemical contaminants, using assumptions on the relations between chemical food contamination and disease outcomes. Although health effects of nutritional hazards are often evident only in the longer term, recent improved availability of insights from long-term epidemiological studies on the cause-relationships between nutritional hazard and disease outcomes sometimes allow COI and DALY/HALY be applied to nutritional hazards. Risk assessment methodology can be applied to chemical hazards and microbiological hazards, when it is known as quantitative microbiological risk assessment (QMRA). Although the same procedure is followed, the calculations and the information required are quite different. Both RA types aim to calculate human exposure to a particular food safety hazard— the chemical contaminant and the pathogen, respectively—through food consumption. The main difference is that MRA calculates the pathogenic contamination of food at time of consumption and numbers of people getting ill from consuming that food, whereas chemical RA calculate the exposure of the contaminant by food at the time of consumption and evaluate if this exposure is below or above the Tolerable Daily Intake (ADI), or similar. For ranking several chemical contaminants in food at once, methods typically applied are the risk ratio method and the scoring method. These methods either multiply or divide a parameter for occurrence of the chemical (e.g. concentration) and the severity of the hazard (e.g. TDI).

MCDA was mostly applied to rank microbiological hazards, but could also be applied for ranking chemical hazards, or both. However, when applied to ranking two or even three types of hazards (if nutritional hazards are included), great care must be taken in designing the MCDA so that a common set of parameters are identified which are relevant to all hazard groups.

For some methods, such as risk matrix and risk ratio, essential data needs appear to be smaller than with other methods, like RA, CRA and MCDA. However, it is more that these former methods could also be applied when less information is available, although ideally larger amounts would be available. This is in contrast to the latter methods that have a large demand of quantitative data and can only be applied when these data are available. When new, additional data become available, this should be processed by the method selected in order to update risk ranking results. Automatic or easy updating of results is an issue that was hardly touched upon in the risk ranking method application found in literature, but this issue merits further investigation. In addition, automatic or easy updating of results could also be used for the scenario analyses or sensitivity analyses of results. It requires an IT application of data, stored in datasheets or databases, linked to model calculations expressed in scripts. Methods most suitable for such an automatic update are RA, risk ratio, risk scoring, risk matrices, COI, HALY, and MCDA. It is more difficult to apply with CRA, WTP and expert synthesis. For WTP and expert synthesis, the context in which participants make their choices will be altered (e.g. changes in relative prices or perceived risk), and hence primary data will need to be collected again with the method designed to reflect the altered context.

Methods that apply quantitative approaches demand more data and result in more precise outcomes with a better description of the uncertainties, assuming that data quality is high. Qualitative methods can be used when data are scarce, e.g., when emerging hazards, such as botanicals, are to be ranked. They also have the advantage of generating rich descriptive material, by which insights into the reasoning behind the opinions (or ranking decisions) of participants can be obtained. In the cases of limited data availability, the appropriate methods are risk matrix, flow charts/decision trees with an emphasis on input from experts, or a ranking based solely on expert synthesis of available quantitative and qualitative information. In the cases of the latter, use qualitative inputs, the outcomes will also be less precise.

In general, quantitative methods taking into account uncertainty and variability require more time and resource than qualitative methods. However, most methods that are used for qualitative situations can also be used semi-quantitatively or quantitatively. And in the latter case, they would also require an equal amount of time and resource. For instance, risk matrices and expert judgments can be used in a simple application using qualitative input or asking the expert to provide their qualitative opinion, respectively. When performed more quantitatively also expert judgment and risk matrices are also resource intensive.

In principle, all methods can account for uncertainty and variability in the input data used, acknowledging this information is more precise and quantitatively defined with the quantitative methods. RA and CRA, both of which can accommodate uncertainty and variability in the input data, appear to be very useful methods for providing quantitative results, provided their substantial data requirements are met.. Semi-quantitative and qualitative methods could also allow for inclusion of uncertainty. Two methods do not have the capacity to consider uncertainty in terms of outcomes, these being risk matrix and flow/decision charts.

Risk ranking can be based on a narrow range of parameters, e.g., measurements of exposure and effect on human health, such as risk ratio or the scoring method, or can include wider issues such as economic impacts and societal preferences. Most methods are demanding of time and other resources, e.g., for primary data collection, although some predefined tools for risk ranking are openly available. MCDA is typically applied when, besides exposure and effect, other metrics need to be considered, such as the consumers' perception of risk associated with different hazards. The strength of this method is in this wider applicability and the involvement of stakeholder groups to assess preference functions and weights. It is often applied in a multi-stakeholder situation. WTP is typically applied when consumer perception on food safety is to be included in the risk ranking.

The results of risk rankings should be interpreted carefully as relatively small differences in methodology can result in changes in final rankings. There is a need for transparency regarding the method used and its application and adequate explanation so users can understand the rationale which has been used to derive the numbers.

An important element of all risk ranking activities is communication of the outputs to interested end-users, including the general public. A question arises as to how such communication processes are developed from the outputs of these different risk ranking methodologies in forms which are both understandable and relevant to different interested end-user communities, and there is no comparative analysis currently available. Including risk perceptions may, for example, increase the relevance of the outputs to the general public, but the extent to which such communication is trusted compared to the communication of outputs from risk ranking methodologies where this has not been the case requires further research, as does the development of a more general communication strategy regarding risk ranking practices and allocation of resources to associated risk mitigation activities.

In conclusion, this study showed there is a wide range of methods that can be used for ranking food related hazards, based on their impact on human health. It has demonstrated that there is no single best risk ranking method. Each of the method categories has its own strengths and weaknesses. The most suitable methods should be selected based on the risk manager's requirements and needs, as well as available resources, the risk ranking task at hand, data availability and the characteristics of the methods. To this end, close communication between risk managers and risk assessors is needed to identify to the most suitable method for risk ranking. Uncertainties associated with data input need to be clearly stated. To date, this is not part of the standard procedure of most methods. This overview is valuable for industrial and governmental risk managers, and risk assessors for selecting the most appropriate methods for risk ranking of food and diet related hazards on the basis of human health impact. The overview will facilitate this decision process and allow for a structured and transparent selection of the most appropriate risk ranking method.

Acknowledgments

The authors gratefully acknowledge the grant received from EFSA for performing this study (EFSA/SCOM/01/2013) as well as critical reflections from Arianna Chiusolo (EFSA) and her colleagues on the study results.

Annex 1.

Literature search protocol

a) Search strategy and search strings

The search strategy consisted of three major steps, each designed to search titles and subject headings. Combinations of search strings were used, starting with a broad screening for methods for risk ranking and prioritisation in the field of food related issues (step 1), then narrowing down the methods relating to size of anticipated impact on human health (step 2), and finally focusing on chemical hazards, biological hazards, nutritional components, or social issues related to food (step 3). The strategy steps and final search strings are as follows:

Step 1: Captured titles/subject headings that studied methods and tools for risk ranking and prioritization related to food issues. This step included the following search strings:

TOPIC = (risk*ⁱ OR hazard*) AND

TITLE = (categor* OR rank* OR method* OR nomogram* OR matric* OR decision* OR priori* OR analys* OR mc*a OR multi-criteri* OR assessment*) AND

TOPIC = (food* OR agri* or agro*OR environ*) AND

Step 2: Captured titles/subject headings that investigated risk ranking and prioritisation methods on the basis of anticipated health impact. This step included the following search terms:

TOPIC = (disease* OR human health* OR *tox* OR illness* OR cost* OR sever* OR adi* OR tidI* OR epidemiol* OR BoD OR wtp OR incidence OR prevalence)

TOPIC = (“socio* impact” OR “econ* impact” OR WTP OR cost* OR WTA)

Step 3: Captured titles/subject headings that investigated specific application fields of biological hazards, chemical hazards, nutritional components in food, or social science issues related to food hazards, from consumer and governance perspectives. This step included the following search strings:

TITLE = (zoonos* OR microb* OR gen* OR pathogen* OR qmra OR “antimicrobial resistance” OR parasite* OR virus* OR bacteria* OR micro*rgan* OR prion* OR TSE* OR QRA) AND

NOT = benefit*

OR:

TITLE = (nano* OR chemic* OR antibiotic* OR dioxin* OR “heavy metal*” OR carc* OR pesticid* OR “plant protection product*” OR hormon* OR mycotoxin* OR phytotoxin* or phycotoxin* or marine biotoxin* OR Biocid* OR *contam* OR *pollutant* OR Melamin* OR Acrylamid* OR PCB* OR Residu* OR Endocr* OR Mutag* OR Botanic* GMO* OR “Genetic* modif*” OR “Novel protein*” OR Allerg* OR Insecticid* OR Acaricid* OR Herbicid* OR Fungicid* OR “plant growth regulat*” OR POP OR POPs OR Persistent* OR *accumul*) AND

NOT = benefit*

OR

TITLE = (*nutri* OR *diet* OR bioavail* OR *supplement* OR “Novel protein*” OR Fortification* OR “Novel food*” OR Allerg*) AND

NOT (toxic* OR microbial* OR chemic* OR socio* OR benefit*)

DALY/QALY concept:

TOPIC = (daly* OR qaly* OR haly* OR HRQL* OR HALE) AND

NOT = benefit*

OR

TOPIC = (“focus group*” OR survey* OR interview* OR public* OR “expert analys*” OR *attitud* OR *percep* OR Willingness* OR *Soci* OR Determ* OR Cultur* OR Tradition* OR Typic* OR Consumer* OR Ethic* OR accept* or opinion* or view* or behaviour* or behavior* or employ* or communicat* or dialog* or engage* or particip* or gover* or legal* or law* or regul*) AND

NOT: religious* or halal* OR benefit*

b) Evaluation criteria

The references judged to be relevant for the study objectives were evaluated for eligibility and quality of the described research. References were included when:

1.	Reference was relevant for the objective of the literature review; ○ References discussing prioritisation/ranking methods for human health risks and/or, ○ References describing risk prioritization/ranking methods applied for environmental/ecological risks and/or, ○ References to risk prioritization, risk analysis, risk assessment methods and/or risk modelling included in abstract and/or, ○ Any relevance of the work for application to human health, including references on drinking water and/or, ○ Abstract indicates socio-economic research methodology is employed.
2.	Reference came from international peer-reviewed journals;
3.	Methods in the reference were well described (semi-)quantitative or qualitative, user-friendly, transparent, structured, and objective;
4.	Methods in the reference were applicable in wider decision making schemes/frameworks;
5.	In case of reports, they should originate from well-known, highly-respected governmental bodies or research organisations.

Criteria for excluding references were:

–	References discussing only parts of a method (only exposure or only human health effects), such as references dealing with presence of chemical hazards, analytical methods, and/or references about toxicity studies. These are all parts of a risk assessment and/or,
–	References addressing non-human related aquaculture and non-human related animal health.