Advancing traditional strategies for testing hydrological model fitness in a changing climate

Figures & data

Figure 1. The selected catchments in Sweden and their properties: (a) locations of the catchments and distributions of their key topographic properties (area, slope, elevation and latitude); (b) climate of the selected catchments, and the distributions of mean annual precipitation, percentage of precipitation in the form of snowfall, mean annual temperature and the aridity index (ratio of mean precipitation to mean potential evapotranspiration); (c) degree of regulation (DOR, percentage of mean annual runoff from the drainage area of the reservoir) in the catchments and distributions of shares of prevailing land-use types, and water surfaces and glaciers; and (d) mean annual runoff in the catchments and distributions of the centre of timing (COT, in days of water year; Kromos et al. 2016), runoff coefficients, and annual maxima and seven-day minima. The hydroclimatic variables presented in the figure were obtained over water years 1962–2020.

Figure 2. Illustration of relationships between performance indicator values and the absolute values of the Wilcoxon rank sum (WRS) test statistic. Larger values of performance indicators imply higher model efficiency, while larger absolute values of the test statistics imply rejection of the WRS test null hypothesis, i.e. the signatures obtained from the observed and simulated flows have statistically different medians. Panel (a) illustrates a performance indicator that can be considered informative about model ability to simulate the distribution of a signature, while panel (b) illustrates a non-informative indicator.

Figure 3. Model performance across the 50 catchments in the calibration (left), evaluation (mid) and full record periods (right panels). Model performance is quantified in terms of the objective function (OF, Equation 1(1) $OF=0.8KGE+0.2KG{E}_{1/\sqrt{Q}}$(1) ; panels a–c), and Kling-Gupta efficiency coefficients obtained from daily flows (KGE; panels d–f) and log-transformed flows (KGE[1/√Q]; panels g–i).

Figure 4. Model performance in the 50 catchments in the calibration, evaluation and full record periods, and performance of the 50 best parameter sets in the Assembro catchment in the calibration period. Boxes indicate 25^th, 50^th and 75^th percentiles, while the whiskers stretch to the point nearest to 1.5 times the interquartile range from the boxes. Outliers and negative values are omitted from the figures for clarity. Abbreviations for the performance indicators are explained in Table 2.

Table 1. Hydrological signatures (flow statistics, and runoff timings and durations) used for model evaluation.

Hydrological signature	Description	Reference
Mean annual flow, Qmean	Mean flows in a water year.
Mean spring flow, Qspring	Series of mean flows in the spring (1 March through 31 May) over the simulation period.	(Chen et al. 2017)
1-, 5- and 30-day maximum annual flows, Qmax,d for d = 1, 5 and 30	Series of annual maxima obtained from daily flows averaged over 5 and 30 days in each water year of the simulation period.	(Dankers et al. 2014, Vis et al. 2015)
1-,3-, 7-, 10-, 20-, 30- and 90 day minimum flows, Qmin,d for d = 1, 3, 7, 10, 20 and 30	Series of minimum flows averaged over a given number of days (d) obtained in each water year of the simulation period.	(Richter et al. 1996, Olden and Poff 2003, Garcia et al. 2017)
10^th and 90^th flow percentiles in wet seasons, Qwet,10p and Qwet,90p	Series of specific flow percentiles obtained in each water year of the simulation period. The wet season is defined as the period from 1 April through 30 September.	(Yarnell et al. 2020)
10^th and 90^th flow percentiles in dry seasons, Qdry,10p and Qdry,90p	Series of specific flow percentiles obtained in each water year of the simulation period. The dry season is defined as the period from 1 October through 31 March.	(Yarnell et al. 2020)
Timing of the centre of mass of annual flow, COM	Timing is computed from daily flows Qi and for each year in a simulation period:$COM=\sum i{Q}_i{t}_i\sum i{Q}_i$where ti represents the ordinal day of a water year.	(Mendoza et al. 2015, Kormos et al. 2016)
Spring onset (spring “pulse day”), SPD	Spring onset is the ordinal number of the day in which the negative difference between the streamflow mass curve and the mean streamflow mass curve is the greatest. Spring onset series is obtained from values in each water year of a simulation period.	(Cunderlik and Ouarda 2009)
High-flow frequency, HFF	Series of mean number of days in a water year with flows greater than 5 times the mean observed flow in the simulation period. In the literature, flows greater than nine times the mean observed flow are used for high-flow frequency computations. Since the catchments considered in this paper exhibited relatively low flow variability, this threshold was reduced to 5.	(Westerberg and McMillan 2015, Krysanova et al. 2017)
Low-flow frequency, LFF	Series of mean number of days in a water year with flows smaller than 20% of the mean observed flow in the simulation period.	(Nicolle et al. 2014, Westerberg and McMillan 2015, Krysanova et al. 2017)
Timing of the maximum annual flow, TQmax	Ordinal days in which maximum annual flow occurred, obtained in each water year of the simulation period.	(Richter et al. 1996)
Timing of the minimum annual flow, TQmin	Ordinal days in which minimum annual flow occurred, obtained in each water year of the simulation period. If there are several consecutive days with the same minimum flows, the mean timing of these days in a water year is adopted.	(Vis et al. 2015, Parajka et al. 2016)

Table 2. Performance indicators used for model evaluation.

Notation	Description of performance indicator and references	Equation
KGE	Kling-Gupta efficiency coefficient is computed from: daily flows, KGE (Gupta et al. 2009); reciprocal of root-transformed daily flows, KGE1/√Q, to put more emphasis on low flows (Santos et al. 2018); daily flows in a representative year, KGEwy, obtained by averaging daily flows on a specific calendar day over the entire simulation period (Schaefli et al. 2014)	$KGE=1-\sqrt{{\left(r-1\right)}^2+{\left(\alpha -1\right)}^2+{\left(\beta -1\right)}^2}$$r=\sum Ni=1\left(Q_{obs,i}-{\overline{Q}}_{obs}\right)\left(Q_{sim,i}-{\overline{Q}}_{sim}\right)\sum Ni=1{\left(Q_{obs,i}-{\overline{Q}}_{obs}\right)}^2{\left(Q_{sim,i}-{\overline{Q}}_{sim}\right)}^2$$\alpha =\frac{{\hat{S}}_{Q_{sim}}}{{\hat{S}}_{Q_{obs}}},\beta =\frac{{\bar{Q}}_{sim}}{{\bar{Q}}_{obs}}$
NPKGE	Non-parametric formulation of KGE indicator (Pool et al. 2018). It is computed as KGE, with Spearman instead of Pearson correlation coefficient, and with the ratio of standard deviations estimated from FDCs.	$NPKGE=1-\sqrt{{\left(r_{Spearman}-1\right)}^2+{\left({\alpha }_{NP}-1\right)}^2+{\left(\beta -1\right)}^2}$${\alpha }_{NP}=1-\frac{1}{2}\sum Ni=1\left|\frac{FD{C}_{sim,i}}{N{\overline{Q}}_{sim}}-\frac{FD{C}_{obs,i}}{N{\overline{Q}}_{obs}}\right|$
NSE	Nash-Sutcliffe efficiency coefficients (Nash and Sutcliffe 1970) are computed from daily (NSE) and log-transformed daily flows (NSElogQ).	$NSE=\sum Ni=1{\left(Q_{obs,i}-Q_{sim,i}\right)}^2\sum Ni=1{\left(Q_{obs,i}-{\overline{Q}}_{obs}\right)}^2$
LME	Liu mean efficiency represents a modification of KGE computed from daily flows (Liu 2020).	$LME=1-\sqrt{\left[{\left(k_1-1\right)}^2+{\left(\beta -1\right)}^2\right]}$$k_1=r\frac{{\hat{S}}_{Q_{sim}}}{{\hat{S}}_{Q_{obs}}}=\alpha r$
R^2	Coefficient of determination (Krause et al. 2005).	$\sum Ni=1\left(Q_{obs,i}-{\overline{Q}}_{obs}\right)\left(Q_{sim,i}-{\overline{Q}}_{sim}\right)\sum Ni=1{\left(Q_{obs,i}-{\overline{Q}}_{obs}\right)}^2\sum Ni=1{\left(Q_{sim,i}-{\overline{Q}}_{sim}\right)}^2$
VE	Volumetric efficiency (Criss and Winston 2008).	$VE=1-\sum Ni=1\left|Q_{sim,i}-Q_{obs,i}\right|\sum Ni=1Q_{obs,i}$
IA	Index of agreement (Krause et al. 2005).	$IA=1-\sum Ni=1{\left(Q_{obs,i}-Q_{sim,i}\right)}^2\sum Ni=1{\left(\left|Q_{obs,i}-{\overline{Q}}_{obs}\right|-\left|Q_{sim,i}-{\overline{Q}}_{obs}\right|\right)}^2$
LE	Lindström efficiency coefficient (Seibert and Vis 2010).	$LE=NSE-0.1\sum Ni=1\left|Q_{obs,i}-Q_{sim,i}\right|\sum Ni=1Q_{obs,i}$
KGEFDC	KGE computed from: entire flow duration curve (KGEFDC) (e.g.Todorović et al. 2019); and different flow duration curve (FDC) segments. The FDC segments are set after recommendations by Pfannerstill et al. (2014). Segments are obtained from flows that are exceeded given the percentage of time of the simulation period.	FDC segments considered: extremely high flows exceeded up to 5% of the time (KGE0–5), high flows: exceeded 5–20% of the time (KGE5–20), mean flows, exceeded 20–70% of the time (KGE20–70), low flows, exceeded 70–95% of the time (KGE70–95), very low flows, exceeded 95–100% of the time (KGE95–100), overall low flows, exceeded 70–100% of the time (KGE70–100).

* Q_obs – observed daily flows; Q_sim – simulated daily flows; ${\bar{Q}}_{\mathrm{o}\mathrm{b}\mathrm{s}}$ – mean observed flow over the simulation period; ${\bar{Q}}_{sim}$ – mean simulated flow over the simulation period; N – length of a simulation period (in days); ŝ – standard deviation; FDC – flow duration curve.

Figure 5. Model performance in reproducing distributions of the selected hydrological signatures (Table 1). Cell values are percentages of successful simulations, i.e. percentage of catchments in the calibration (CAL), evaluation (EVAL) or full record period (FRP), or percentage of the behavioural parameter sets in the Assembro catchment in the calibration period (GLUE) that well reproduced the distributions of the signatures. The last column in the heat map shows the average performance of the simulation across all signatures.

Figure 6. Percentage of catchments in which significant trends were detected in the selected hydrological signatures (Table 1) according to the results of the Mann-Kendall test in the calibration (CAL), evaluation (EVAL) and full record periods (FRP). The signatures were computed from observed flows.

Figure 7. Model performance in reproducing trends in the selected hydrological signatures (Table 1). Cell values in the heat map are percentages of successful simulations, i.e. percentage of catchments in the calibration (CAL), evaluation (EVAL) or full record period (FRP), or percentage of the behavioural parameter sets in the Assembro catchment in the calibration period (GLUE) that successfully reproduced trends (or the lack of trends) in hydrological signatures. The last column in the heat map shows average performance across all signatures.

Figure 8. Informativeness of performance indicators (Table 2) about model ability to reproduce distributions of the hydrological signatures (Table 1) in different simulations. Values of 1 in the heat maps indicate that the two groups of simulations (i.e. simulations in which the distributions were well reproduced versus the remaining ones) have significantly different indicator values, suggesting that a specific indicator can be informative about model performance in reproducing the distributions. Values of −1 mean that the two groups of simulations result in similar values of an indicator, suggesting that the indicator is not informative about model ability to reproduce the distributions. A value of 2 (−2) suggests that the distributions of a signature were reproduced in all (none) of the catchments or by the behavioural parameter sets (GLUE), respectively.

Figure 9. Pearson correlation coefficients between the values of performance indicators (Table 2) and absolute values of the Wilcoxon rank sum test statistic, z, obtained in different simulations. Strong negative correlations suggest that a specific indicator can be informative about the model ability to reproduce the distribution of a hydrological signature (Table 1), as opposed to weak negative or positive correlations.

Figure 10. Informativeness of performance indicators (Table 2) about model ability to reproduce trends in series of hydrological signatures (Table 1) in different simulations. Values of 1 in the heat maps indicate that the two groups of simulations (i.e. simulations in which the trends were well reproduced versus the remaining ones) have different indicator values, which can suggest that a specific indicator is not informative about model performance in reproducing the trends. Values of −1 mean that the two groups of simulations results in similar values of a specific indicator, suggesting that a specific PI is not informative about model ability to reproduce the trends. A value of 2 (−2) suggests that the trends of a signature were reproduced in all (none) of the simulations or by the behavioural parameter sets (GLUE), respectively.

Kormos, P.R., et al., 2016. Trends and sensitivities of low streamflow extremes to discharge timing and magnitude in Pacific Northwest mountain streams. Water Resources Research, 52 (7), 4990–5007. doi:10.1002/2015WR018125

 Web of Science ®Google Scholar

Chen, J., et al., 2017. Impacts of weighting climate models for hydro-meteorological climate change studies. Journal of Hydrology, 549, 534–546. doi:10.1016/j.jhydrol.2017.04.025

 Web of Science ®Google Scholar

Dankers, R., et al., 2014. First look at changes in flood hazard in the inter-sectoral impact model intercomparison project ensemble. Proceedings of the National Academy of Sciences, 111 (9), 3257–3261. doi:10.1073/pnas.1302078110

 PubMed Web of Science ®Google Scholar

Vis, M., et al., 2015. Model calibration criteria for estimating ecological flow characteristics. Water, 7 (12), 2358–2381. doi:10.3390/w7052358

 Web of Science ®Google Scholar

Richter, B.D., et al., 1996. A method for assessing hydrologic alteration within ecosystems. Conservation Biology, 10 (4), 1163–1174. Available from: http://links.jstor.org/sici?sici=OSSS-8892%28199608%2910%3A4%3C1163%3AAMFAHA%3E2.0.C0%3B2-E.

 Web of Science ®Google Scholar

Olden, J.D. and Poff, N.L., 2003. Redundancy and the choice of hydrologic indices for characterizing streamflow regimes. River Research and Applications, 19 (2), 101–121. doi:10.1002/rra.700

 Web of Science ®Google Scholar

Garcia, F., Folton, N., and Oudin, L., 2017. Which objective function to calibrate rainfall–runoff models for low-flow index simulations? Hydrological Sciences Journal, 62 (7), 1149–1166. doi:10.1080/02626667.2017.1308511

 Web of Science ®Google Scholar

Yarnell, S.M., et al., 2020. A functional flows approach to selecting ecologically relevant flow metrics for environmental flow applications. River Research and Applications, 36 (2), 318–324. doi:10.1002/rra.3575

 Web of Science ®Google Scholar

Mendoza, P.A., et al., 2015. Effects of hydrologic model choice and calibration on the portrayal of climate change impacts. Journal of Hydrometeorology, 16 (2), 762–780. doi:10.1175/JHM-D-14-0104.1

 Web of Science ®Google Scholar

Cunderlik, J.M. and Ouarda, T.B.M.J., 2009. Trends in the timing and magnitude of floods in Canada. Journal of Hydrology, 375 (3–4), 471–480. doi:10.1016/j.jhydrol.2009.06.050

 Web of Science ®Google Scholar

Westerberg, I.K. and McMillan, H.K., 2015. Uncertainty in hydrological signatures. Hydrology and Earth System Sciences, 19 (9), 3951–3968. doi:10.5194/hess-19-3951-2015

 Web of Science ®Google Scholar

Krysanova, V., et al., 2017. Intercomparison of regional-scale hydrological models and climate change impacts projected for 12 large river basins worldwide—a synthesis. Environmental Research Letters, 12 (10), 105002. doi:10.1088/1748-9326/aa8359

 Web of Science ®Google Scholar

Nicolle, P., et al., 2014. Benchmarking hydrological models for low-flow simulation and forecasting on French catchments. Hydrology and Earth System Sciences, 18 (8), 2829–2857. doi:10.5194/hess-18-2829-2014

 Web of Science ®Google Scholar

Parajka, J., et al., 2016. Uncertainty contributions to low flow projections in Austria. Hydrology and Earth System Sciences, 20 (5), 2085–2101. doi:10.5194/hess-20-2085-2016

 Web of Science ®Google Scholar

Gupta, H.V., et al., 2009. Decomposition of the mean squared error and NSE performance criteria: implications for improving hydrological modelling. Journal of Hydrology, 377 (1–2), 80–91. doi:10.1016/j.jhydrol.2009.08.003

 Web of Science ®Google Scholar

Santos, L., Thirel, G., and Perrin, C., June 2018. Technical note: pitfalls in using log-transformed flows within the KGE criterion. Hydrology and Earth System Sciences Discussions, 1–14. doi:10.5194/hess-2018-298

 Web of Science ®Google Scholar

Schaefli, B., et al., 2014. SEHR-ECHO v1.0: a spatially explicit hydrologic response model for ecohydrologic applications. Geoscientific Model Development, 7 (6), 2733–2746. doi:10.5194/gmd-7-2733-2014

 Web of Science ®Google Scholar

Pool, S., Vis, M., and Seibert, J., 2018. Evaluating model performance: towards a non-parametric variant of the Kling-Gupta efficiency. Hydrological Sciences Journal, 63 (13–14), 1941–1953. doi:10.1080/02626667.2018.1552002

 Web of Science ®Google Scholar

Nash, J.E. and Sutcliffe, J.V., 1970. River flow forecasting through conceptual models, Part I - A discussion of principles. Journal of Hydrology, 10 (3), 282–290. doi:10.1016/0022-1694(70)90255-6

 Google Scholar

Liu, D., August 2020. A rational performance criterion for hydrological model. Journal of Hydrology, 590, 125488. doi:10.1016/j.jhydrol.2020.125488

 Web of Science ®Google Scholar

Krause, P., Boyle, D.P., and Bäse, F., January 2005. Comparison of different efficiency criteria for hydrological model assessment. Advances in Geosciences, 5, 89–97. doi:10.5194/adgeo-5-89-2005

 Google Scholar

Criss, R.E. and Winston, W.E., 2008. Do Nash values have value ? Discussion and alternate proposals. Hydrological Processes, 22 (14), 2723–2725. doi:10.1002/hyp

 Web of Science ®Google Scholar

Seibert, J. and Vis, M.J.P., 2010. HBV-lightHELP. Available from: http://www.geo.uzh.ch/en/units/h2k/services/hbv-model

 Google Scholar

Todorović, A., et al., 2019. The 3DNet-Catch hydrologic model: development and evaluation. Journal of Hydrology, 568, 26–45. doi:10.1016/j.jhydrol.2018.10.040

 Web of Science ®Google Scholar

Pfannerstill, M., Guse, B., and Fohrer, N., 2014. Smart low flow signature metrics for an improved overall performance evaluation of hydrological models. Journal of Hydrology, 510, 447–458. doi:10.1016/j.jhydrol.2013.12.044

 Web of Science ®Google Scholar

Data availability statement

Hydro-climatic data can be retrieved from the Swedish Meteorological and Hydrological Institute (SMHI) database (http://vattenwebb.smhi.se/). Geospatial data for the streamflow stations can be downloaded from the SMHI SVAR database (https://www.smhi.se/publikationer/svar-svenskt-vattenarkiv-1.17833).