Evaluation of Sampling-Based Methods for Sensitivity Analysis: Case Study for the E. coli Food Safety Process Risk Model

ABSTRACT

This article evaluates selected sensitivity analysis methods applicable to risk assessment models with two-dimensional probabilistic frameworks, using a microbial food safety process risk model as a test-bed. Six sampling-based sensitivity analysis methods were evaluated including Pearson and Spearman correlation, sample and rank linear regression, and sample and rank stepwise regression. In a two-dimensional risk model, the identification of key controllable inputs that can be priorities for risk management can be confounded by uncertainty. However, despite uncertainty, results show that key inputs can be distinguished from those that are unimportant, and inputs can be grouped into categories of similar levels of importance. All selected methods are capable of identifying unimportant inputs, which is helpful in that efforts to collect data to improve the assessment or to focus risk management strategies can be prioritized elsewhere. Rank-based methods provided more robust insights with respect to the key sources of variability in that they produced narrower ranges of uncertainty for sensitivity results and more clear distinctions when comparing the importance of inputs or groups of inputs. Regression-based methods have advantages over correlation approaches because they can be configured to provide insight regarding interactions and nonlinearities in the model.

Key Words:

• sensitivity analysis
• microbial food safety process risk models
• two-dimensional probabilistic framework
• variability
• uncertainty

ACKNOWLEDGMENTS

The work reported here is based on work supported by Cooperative Agreement No. 58-0111-0-005 between the U.S. Department of Agriculture, Office of the Chief Economist, Office of the Risk Assessment and Cost Benefit Analysis (USDA/OCE/ORACBA) and the Department of Civil, Construction, and Environmental Engineering at North Carolina State University. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the authors and do not necessarily reflect the views of the U.S. Department of Agriculture.

Notes

^a Values are based on the co-mingled analysis of variability and uncertainty.

^b Lot is defined as the total number of cattle necessary to fill one combo bin.

^c Refers to reduction in organisms attributable to this processing.

^d I = Initial number of organisms on contaminated carcasses introduced during dehiding, modeled as a cumulative frequency distribution based on FSIS data. Beta distribution is used to define the corresponding cumulative frequency at each value of the distribution as: CF _i = BetaINV(α, β) where,

i	=	Cumulative rank of data associated with a number of organisms
CF i	=	Cumulative frequency at value i of the empirical distribution
BetaINV	=	Inverse of a beta distribution
α	=	Parameter α of the beta distribution: ∑ k = 1 i n k
β	=	Parameter β of the beta distribution: n T − ∑ k = 1 i n k + 1
n k	=	Number of available data at the kth value of the number of organisms
n T	=	Total number of available survey data

^a The abbreviations used for inputs in this table are the same as those defined in Table 1.

^b The percentage of the uncertainty realizations that produced statistically significant correlation coefficient (p value less than 0.05).

^c Arithmetic mean of ranks for 100 uncertainty realizations. Inputs were ranked based on the magnitude of correlation coefficients.

^d Group 1: most important input (i.e, rank 1); Group 2: secondary importance inputs (i.e, ranks between 2 and 5); Group 3: minor importance inputs (i.e, ranks between 6 and 8); Group 4: unimportant inputs (i.e, ranks between 9 and 12).

^e See Figure 2 for associated probabilities.

^a The abbreviations used for inputs in this table are the same as those defined in Table 1.

^b The percentage of the uncertainty realizations that produce statistically significant regression coefficient (p value less than 0.05).

^c Arithmetic mean of ranks for 100 uncertainty realizations. Regression terms were ranked based on the magnitude of relative partial sum of squares (Eq. 3).

^d For each input, MLIG represents the most likely range of ranks in uncertainty realizations. Group 1: most important input (i.e, rank 1); Group 2: secondary importance inputs (i.e, ranks between 2 and 5); Group 3: minor importance inputs (i.e, ranks between 6 and 8); Group 4: unimportant inputs (i.e, ranks between 9 and 16).

^e See Figure 4 for associated probabilities.

^a The abbreviations used for inputs in this table are the same as those defined in Table 1.

^b The percentage of the uncertainty realizations that produce statistically significant regression term (p value less than 0.05).

^c Arithmetic mean of ranks for 100 uncertainty realizations. Regression terms were ranked based on the magnitude of relative partial sum of squares (Eq. 3).

^d Group 1: most important input (i.e, rank 1); Group 2: secondary importance inputs (i.e, ranks between 2 and 5); Group 3: minor importance inputs (i.e, ranks between 6 and 8); Group 4: unimportant inputs (i.e, ranks between 9 and 16).

^e Probability that each regression term was assigned to the MLIG in 100 uncertainty realizations.

^a The abbreviations used for inputs in this table are the same as those defined in Table 1.

^b The percentage of uncertainty realizations that the regression term was selected in the stepwise regression analysis.

^c Arithmetic mean of ranks for 100 uncertainty realizations. In each uncertainty realization, regression terms were ranked based on the order in which they were selected in the regression model.

^d Group 1: most important input (i.e, rank 1); Group 2: secondary importance inputs (i.e, ranks between 2 and 5); Group 3: minor importance inputs (i.e, ranks between 6 and 8); Group 4: unimportant inputs (i.e, ranks between 9 and 16).

^a PCA: Pearson Correlation Analysis; SCA: Spearman Correlation Analysis; SRA: Sample Regression Analysis; RRA: Rank Regression Analysis; SSRA: Sample Stepwise Regression Analysis; RSRA: Rank Stepwise Regression Analysis.

^b Requires a priori knowledge of the functional form of interaction terms.

^c Rank transformation can mitigate the problem of nonlinearity in monotonic models.

^d Nonlinear terms can be included in the regression model in order to account for nonlinearity in the model.

^e SCA is not sensitive to the presence of a threshold in the model output. However, SCA cannot identify the threshold.

^f Change-point regression analysis can be used to identify the threshold in the model output.