The Impact of Different Performance Measures on Model Selection for Fraser River Sockeye Salmon

Abstract

Uncertainties prevalent in fisheries systems result in deviations between management targets and observed outcomes. As an example of attempting to deal with such uncertainty, fishery managers of sockeye salmon Oncorhynchus nerka from the Fraser River, British Columbia, use environmentally based management adjustment (MA) models to forecast indices of in-river loss of adults as they migrate upstream to spawn. Losses forecasted by MA models are directly incorporated into estimates of total allowable catch, resulting in harvest reductions that aim to increase the probability of achieving spawning escapement targets. However, the relative forecasting success of different MA models has not been rigorously assessed. Therefore, we used a suite of forecasting and hindcasting metrics to rank the performance of numerous MA models. We found that the rank of each model varied across sockeye salmon stock aggregates (i.e., run timing groups) and depended on the performance measures chosen for evaluation. Although model selection in fisheries research is often determined solely by model-fitting criteria, such as R ² and Akaike's information criterion (corrected for small-sample bias), in our case the models with the largest mean R ² value, the smallest mean corrected Akaike's information criterion, or both often ranked poorly for measures of model forecast performance (i.e., mean raw error, mean absolute error, and root mean square error). Although no single model performed best across all run timing groups, failure to apply an MA produced the worst outcome (for 3 of the 4 run timing groups) or second-worst outcome (for the fourth group). We provide a framework for model selection based on the relative importance of different model selection criteria and their associated performance measures. We urge scientists and managers to work closely together to develop appropriate metrics for assessing model performance and for objectively selecting forecast models that will best meet management objectives.

Received April 29, 2010; accepted December 6, 2010

ACKNOWLEDGMENTS

Thanks to Duncan Knowler for feedback on this manuscript. Insightful suggestions and feedback on preliminary analyses were provided by Steve Macdonald (DFO), Ian Guthrie (PSC), and Mike Lapointe (PSC). Support for this work was provided by the PSC Southern Boundary Restoration and Enhancement Fund, the DFO Fraser River Environmental Watch Program, and the Natural Sciences and Engineering Research Council of Canada (the latter provided to R.M.P.).

¹Present address: Rubenstein School of Environment and Natural Resources, University of Vermont, 105 Carrigan Drive, Burlington, Vermont 05405, USA.