Why and when it is useful to publish and share inconclusive results and failures: reply to “Reporting negative results to stimulate experimental hydrology”^*

ABSTRACT

We thank van Emmerik et al. for their discussion of our opinion paper. We do not fully agree that publication of negative results will be very effective in promoting experimental hydrology. Instead, we think that experimental hydrology is considered to be more risky than modelling studies because of difficulties in publishing inconclusive results and the potential for failure.

KEYWORDS:

• negative result
• unexpected results
• failures
• experimental design

Editor A. Castellarin Associate editor not assigned

We thank van Emmerik et al. (2018) for contributing to this discussion. As stated in our opinion paper (Blume et al. 2017), we agree that improvements in hydrological understanding are unlikely to come from pure modelling exercises and that the value of experimental work needs to be acknowledged. However, we fear that a focus on negative results might not be the best way to reach this goal.

We do not think that experimental hydrology suffers from negative results. If we rephrase the definition of negative results of van Emmerik et al. (2018): “expected or wanted result was not observed despite careful experimental design, planning and execution”, the expression could be “an unexpected result”. Unexpected results are often very interesting and insightful, and as such are not negative at all. In this context, many published studies describe “negative results” without calling them negative and we often learn more from unexpected results than from expected ones. Pfister and Kirchner (2017) state “It is essential for future progress in hydrology that [this] exploratory research continues to be supported.” We agree with that because exploratory studies often highlight unexpected behaviour and lead to new hypotheses. If we then set out to test the hypotheses, we should design experiments or monitoring set-ups with the aim of rejecting them. This would be the best possible outcome, as proving a hypothesis is hardly ever possible and rejecting a hypothesis is more insightful than not rejecting it.

We think that hydrologists may consider experimental work to be risky because of struggles with (a) inconclusive results, (b) things that go wrong despite careful planning and execution, and (c) things that go wrong because of a lack of careful planning or an inadequate experimental design. Some of the studies resulting in what are considered inconclusive results might still warrant publication. One can learn a lot about hydrological processes if a trench does not capture any subsurface stormflow, if the expected difference in streamflow or tracer response for two catchments is not observed, or if trees do not show any sign of water stress even though the soil is very dry. However, unfortunately, referees then often ask authors to test additional hypotheses to explain the data, which frequently requires new measurements or additional experiments. This makes it hard to publish field studies for which the expected response was not observed and the reasons for the lack of the expected response remain inconclusive.

Many experimental studies go wrong because the data were lost or not collected, either because of actual equipment failure or due to unforeseen circumstances (e.g. a landslide took away all equipment, a bear destroyed the raingauge, cables were damaged by rodents, etc.). Modellers are more immune to the risk of failure after long-term investments as they usually do not need to let time pass and wait for sufficiently long time series or for a certain event or season before they can analyse their data. However, these risks can often be reduced with careful planning, frequent field visits, redundant measurements and frequent quality checks of the incoming data, etc. Failures of the third type (things going wrong because of a lack of carefulness or inadequate experimental design) sometimes only become apparent at a late stage of the study when a lot of time and resources have already been invested. As time and equipment are valuable resources, it is painful to invest in them and then find that they are wasted. This can be avoided by learning from others and by careful planning of the projects. Courses on field measurements, field experiments and experimental design reduce the chances for these mistakes. Unfortunately, most undergraduate and graduate curricula in hydrology programmes do not include (m)any courses on experimental design. In this respect, we can learn a lot from other fields of environmental sciences where experimental design courses are a much larger part of the curriculum. None of the above described risks can, however, be mitigated by publishing negative results in scientific papers. We do agree with van Emmerik et al. (2018) that failures and the conditions leading to them (be it sloppiness, technical failures or events that could not be expected) should be shared in notes or posts on forums, such as the Experimental Hydrology Wiki (http://www.experimental-hydrology.net), or in presentations in special conference sessions on things that go wrong, to help other researchers avoid these same mistakes because this has the potential to mitigate future failures and thus to advance our science. However, only in some cases do failures warrant publication in research articles.

Disclosure statement

No potential conflict of interest was reported by the authors.