Comment on Gard (2009): Comparison of spawning habitat predictions of PHABSIM and River2D models

Gard (2009) compared the performance of two instream flow models, PHABSIM and River2D, that combine one- or two-dimensional (1D or 2D) hydraulic models with simple biological models called habitat suitability criteria (HSC) to estimate an index of habitat called weighted usable area (WUA). PHABSIM and related models such as River2D are widely used for setting instream flow standards, so tests of the models are badly needed (Williams 2001). Although Gard (2009) collected useful data for testing the models, he failed to make good use of them, or to present the data in a form that would allow others to do so. Here, I summarize Gard's (2009) analysis, critique it, and suggest how the data could be better used. I also suggest a statistically and biologically superior alternative to the HSC used in both the 1D and 2D instream flow models. I do not discuss the basic adequacy of such models for their intended function, which can be questioned (e.g. Rosenfeld 2003, Anderson et al. 2006).

Gard (2009) applied the models to 14 sites on 3 rivers in the Central Valley of California; 1 on the Merced River, 5 on the American River, and 8 on the upper Sacramento River. Sites ranged from 0.3 to 10.4 channel widths in length. Both models divide the study areas into ‘cells’ and predict the depth, velocity and substrate size for each, but the size and shape of the cells vary between the models and among the sites, and the 1D model could not be used in parts of the sites with transverse flows or other complex flow patterns. HSC for spawning steelhead and three runs of Chinook salmon (fall, late fall and winter) were developed for depth, velocity, and substrate size, mainly from redds in the study rivers, and the locations of Chinook redds were determined with surveying equipment or GPS readings. The HSC take values between 0 and 1, depending on the value of the relevant habitat variable. The same HSC were used in both models and were combined by simple multiplication to produce a ‘composite suitability index’ (CSI), that also takes values from 0 to 1, for each modelled cell in the study sites at each modelled discharge. WUA is the sum of the areas of the cells multiplied by their respective CSIs, normalized by the channel length. Running each model for a range of discharge at each site produces curves of WUA over discharge. Gard (2009) used Mann–Whitney U tests to determine whether, for Chinook, there were statistically significant differences between the CSI, at the appropriate discharge, of cells that did or did not include a redd site. For both species, he tested whether there were significant differences between the sets of WUA curves produced by the two models for the sites and runs using Kolmogorov–Smirnov tests.

Gard (2009) found that with both the 1D and 2D models, the CSI of used cells differed significantly from the CSI of unused cells for fall and winter Chinook, but not for late fall Chinook, for which the sample size was small. He also found only one significant difference among the 55 pairs of WUA curves generated by the models. On that basis, he found that PHABSIM could ‘relatively accurately predict the CSI of redd locations’, and that the study ‘found little difference between the PHABSIM and River2D in flow–habitat relationships’, but that River2D could model parts of the river with transverse flows or other hydraulic features that precluded use of the 1D model.

Testing whether the CSI of used cells is significantly greater than that of unused cells essentially is testing whether PHABSIM or related models do better than a random guess. This is a weak test, as has been pointed out previously (Williams 1997, Williams et al. 1999), and confuses statistical significance with biological or practical significance. In general, it is more useful to consider the magnitude of differences (effect size) and the precision of estimates, rather than statistical significance per se (Stewart-Oaten 1996). The rationale for using the Kolmogorov–Smirnov test on WUA curves from different models is not obvious, and Gard (2009) does not provide it. For two samples, the test uses the maximum difference between their empirical distribution functions, which does reflect differences in the shapes of the curves, but assigning probabilities to this seems questionable (e.g. what are the sample sizes?). Regardless, whether the WUA curves generated by the two models differ in a statistical sense does not matter as much as whether they differ in a practical sense. As Gard (2009) points out in his results section, two of the curves that were similar in terms of the test statistic that he used ‘could result in different flow management decisions’, which seems a practical difference.

As Gard (2009) notes, PHABSIM and related models combine biological and hydraulic models, so errors can arise from either. Accordingly, the parts of the models should be tested separately, as well as together (Williams 2001). Gard has data on depth, velocity, and substrate from over 200 Chinook redds on the American and Merced Rivers, with which the hydraulic and spatial aspects of the models could be tested by comparing these measured data with the values predicted by the 1D and 2D models (uncertain GPS locations make the Sacramento redd data less useful for this purpose). Such tests, and comparing the CSIs calculated from the measured data with the model predictions, would be much more informative than simply comparing the model predictions with each other, or with random guesses. Box plots could be used to present the comparisons effectively. Unfortunately, Gard (2009) presented a comparison of model predictions and measured data only for velocity with River2D, giving mean absolute errors for measured velocities less than 0.91 ms⁻¹ and percentage errors for higher velocities. For the six sites on the Merced and American Rivers, the mean errors for the lower velocities ranged from 0.17 to 0.63 ms⁻¹, so the errors are not trivial.

The HSC used in PHABSIM and similar models lack a clear conceptual basis. For example, the velocity HSC shown in Figure 2 in Gard (2009) seems to be fit to data, but the depth HSC seems to be fit to both data and belief about spawning in deep water. More importantly, the HSC for depth, velocity, and substrate typically are multiplied together to produce a CSI, as was done for this study, and this implies that the influence of each is equal and independent. Logistic regression is an alternative that avoids these assumptions, and has been used on microhabitat data on salmonid redds by Knapp and Preisler (1999), who also used box plots to display comparisons of used and unused cells. Tools for model selection such as the Akaiki information criterion (Burnham and Anderson 1998) can be used for choosing the most appropriate set of variables. Moreover, the use of logistic regression has a much clearer conceptual basis, as a resource selection function; these return values that are proportional to the probability that the resource or habitat will be used or otherwise selected. The statistical and biological assumptions involved with resource selection functions have been made explicit (Manly et al. 2002), and the literature provides useful guidance for testing resource selection models, as well as presence–absence models that are analogous to classifying habitat as suitable or unsuitable with PHABSIM (e.g. Olden et al. 2002, Vaughan and Ormerod 2005, Freeman and Moisen 2008).

For the combined hydraulic/biological models studied by Gard (2009), comparing the distributions of CSI for used and unused cells with box plots would be more informative than reporting medians and p-values. Another good approach would be the following: for each model and site, order all cells by CSI and divide them into ranks by CSI. Then, plot the percentage of all cells in each rank that are used over the mid-point of its range. Thus, with 10 ranks, the percentage of cells with CSI > 0.9 that are used would be plotted over 0.95, the percentage of cells with CSI > 0.8 and ≤ 0.9 that are used would be plotted over 0.85, etc. Confidence intervals for the plots could be estimated by bootstrapping. If the CSI is a kind of resource selection function, then the plot should approximate a straight line. This line should lie on the diagonal if the percentage of occupied cells is scaled by the constant of proportionality between the value returned by the function and the probability that the cell will be used. If instead the CSI is an index of suitability rather than of probability of use, then the percentage of used cells should increase sharply at higher CSI values (Freeman and Moisen 2008), since fish should select the most suitable spawning habitat available (Fretwell 1972). If neither of these conditions obtains, then either the hydraulic model has not performed well, or the utility of the index should be questioned.

Gard has done valuable work by collecting the data with which PHABSIM and River2D can be tested, but better methods for testing the data are readily available (e.g. Freeman and Moisen 2008). He could make a valuable contribution by applying these methods to his data.

Acknowledgements

This work is funded by the Public Interest Energy Research of the California Energy Commission through the Instream Flow Assessment Program of the Center for Aquatic Biology and Aquaculture of the University of California, Davis. Suggestions by Peter Moyle improved the text.