Reporting negative results to stimulate experimental hydrology: discussion of “The role of experimental work in hydrological sciences – insights from a community survey”^*

ABSTRACT

Experimental work in hydrology is in decline. Based on a community survey, Blume et al. showed that the hydrological community associates experimental work with greater risks. One of the main issues with experimental work is the higher chance of negative results (defined here as when the expected or wanted result was not observed despite careful experimental design, planning and execution), resulting in a longer and more difficult publishing process. Reporting on negative results would avoid putting time and resources into repeating experiments that lead to negative results, and give experimental hydrologists the scientific recognition they deserve. With this commentary, we propose four potential solutions to encourage reporting on negative results, which might contribute to a stimulation of experimental hydrology.

KEYWORDS:

• experimental hydrology
• negative results
• field hydrology

Editor A. Castellarin Associate editor not assigned

Anyone who has never made a mistake has never tried anything new. – Albert Einstein

Experimental work – field hydrology, developing/deploying new sensors, laboratory experiments – in hydrology is in decline (Sidle 2006, Burt and McDonnell 2015, Vidon 2015). As identified in the Opinion paper by Blume et al. (2017), the hydrological community associates experimental work with greater risks. Compared to pure modelling work, experimental work generally has higher project costs, and chances of negative results (defined here as when the expected or wanted result was not observed despite careful experimental design, planning and execution) are higher. Furthermore, publishing experimental work is considered to be more difficult (i) due to the long time it usually takes to obtain publishable results and (ii) because experimental results are often considered to be case studies (Blume et al. 2017). Hence, researchers become more and more reserved in dedicating themselves to experimental work with potentially lower scientific output (e.g. h-index, impact factors), on which their success is mainly evaluated in our current academic system. A considerable amount of experimental work never sees the light of publication due to negative results (Granqvist 2015). The current “publish or perish” culture of our scientific system leads to a publication bias towards positive results (Curry 2015). However, experimental failure in geosciences is imperative for scientific advancement. Interestingly, we typically do not share negative findings, not even in informal settings. In the past, many philosophers, including Popper (1963) and Chalmers (1973), have emphasized that science can only advance by learning from mistakes. Recent literature in various fields states the many benefits and values of publishing negative results and calls upon the scientific community to nurture their dissemination (e.g. Andréassian et al. 2010, Schooler 2011, Matosin et al. 2014, Boorman et al. 2015, Granqvist 2015). If we valued all results more equally by also publishing negative results, we could not only avoid spending time and resources in repeating experiments that lead to negative results, but also give experimental hydrologists the scientific recognition they deserve. This, in turn, would motivate more researchers to invest their time and resources in experimental hydrology again. With this commentary, we would like to highlight negative findings stemming from sound science as valuable components of hydrological sciences and to encourage publishing experimental work including negative results to ultimately foster experimental hydrology.

Hydrology is an outdoor science (Burt and McDonnell 2015) and experimental work is imperative for hydrological understanding, as emphasized by Blume et al. (2017). Field data are crucial for studying hydrological systems, revealing underlying mechanisms, and testing hypotheses (Montanari et al. 2013, Peters-Lidard et al. 2017). Experience with experimental work allows for a better understanding of data limitations (Van Emmerik 2015, Vidon 2015), aids in improving perceptual models, and helps to explain irregularities and data gaps. Unfortunately, in the last 50 years, the number of papers in, for example, Water Resources Research that include experimental work dropped from 50 to 10% (Burt and McDonnell 2015). A keyword search on Scopus showed that, from all papers found with the keyword “hydrology”, the ratio between those also including the keyword “experimental” versus “modelling” clearly decreased. In the 1970s, five times as many papers included “experimental” than “modelling”. In the 2010s, twice as many papers included “modelling” compared to “experimental”. We believe that, without a revival of (published) experimental work, hydrological sciences might turn into a communal modelling exercise, without much progress towards hydrological understanding.

In our opinion, the decreasing share of papers that include experimental work is related to the current academic system. Blume et al. (2017) conducted a survey to enquire about the views of the hydrological community on the role of experimental work in hydrological sciences. They showed that the majority of the 336 respondents agree with the statements that experimental work involves higher risks for failure but also that it is imperative for hydrological understanding. Apparently, experimental work is appreciated by the community, and yet it is in decline. A main reason might be science’s obsession with metrics (Benedictus et al. 2016). Scientists, early career scientists in particular, are under historically high pressure to publish (Powell 2016, p. 427). Their funding, and therefore future, is mainly judged on their publication record (Benedictus et al. 2016). With that in mind, how can we expect scientists to choose between becoming a potential trailblazer with the increased risk of unpublishable results, and aiming for a potentially safer, non-experimental, path?

To stimulate experimental hydrology, we believe that the scientific community should encourage the reporting of negative results stemming from experimental work. Sharing negative results is essential for research progress, since it prevents other scientists wasting time and effort by stepping into the same pitfalls again, and contributes to better design of future experimental studies. However, we do emphasize that it will always remain essential to maintain the high standard of scientific publishing. Secondly, the increased possibility for scientific output is beneficial for a researcher’s metrics. Thirdly, this will help to reduce the publication bias towards positive results in science. In science, many things (initially) fail, but eventually contribute to scientific advancement. Unfortunately, inconclusive or negative results are often not considered impactful enough to be published. Instead of seeing negative results as “failure”, we should see their value for the community and encourage their publication. Finally, we believe that by sharing and reporting on negative results, collaborative experimental efforts will be encouraged within the hydrological community. Initiatives, such as the IAHS Measurements and Observations in the XXI Century (MOXXI; Tauro et al. 2018), identify common objectives and challenges, and aim to resolve them from different perspectives.

The call for publishing negative results is not new (e.g. Andréassian et al. 2010, Schooler 2011, Matosin et al. 2014, Boorman et al. 2015, Granqvist 2015, PLOS collections 2015, 2017, Nature Editorial 2016). However, highlighting the value of reporting negative results is not straightforward. With the following ideas we aim to encourage our community to promote and acknowledge the concept of reporting on negative results more actively:

1. Conference sessions: At the European Geosciences Union (EGU) General Assembly, a session on sharing negative results was successfully organized twice (Müller et al. 2017) and is also planned for upcoming years. This provides a platform for sharing lessons learned, and changes the perception that science and scientific output is perfect or straightforward, which is especially important for early career scientists. Through these sessions, valuable insights can be shared with relatively little effort. Presenting at conferences does not directly influence scientific output metrics, but it does contribute to encouraging scientists to report on negative results.

2. Blog posts: Reporting on failures does not necessarily need to go through peer-reviewed journals. The Young Hydrologic Society (YHS), an organization run for and by early career scientists, is launching a blog section dedicated to reporting on negative results. The Experimental Hydrology Wiki (2017) also offers the possibility to share, learn and discuss existing and new methods in experimental hydrology. In addition to the YHS blog and the Experimental Hydrology Wiki, other outlets could be facilitated through influential blogs, such as those from the American Geophysical Union (AGU), the EGU, and the International Association of Hydrological Sciences (IAHS). Blogs offer an accessible, low-key platform to share science, especially when reporting on negative results. Through repositories such as Zenodo, blog posts can also get DOIs, and thus be easily cited and start to contribute to the author’s metrics. Also, blog posts contribute to the researcher’s profile within the hydrological community. Consequently, this will broaden awareness of the lack of negative findings in the peer-reviewed literature, which may facilitate a faster overcoming of the reluctance to report on negative results.

3. Special issues or manuscript types: An effort to start reporting on negative results in hydrology was made through a special issue on negative results from experimental work published in Hydrological Sciences Journal (see Andréassian et al. 2010). This was the outcome of a workshop in which participants were invited to present their experience with unpublished negative results. It was concluded that discussing and reporting on these negative results can only lead to progress. Also, several journals, such as Hydrology and Earth System Sciences, offer the opportunity to publish “articles on high-level scientific failures”, aiming to report on unexpected and instructive scientific failures. Unfortunately, to date no article of this type has been published. An explanation for this might be the reluctance of scientists to share their negative results due to the embarrassment of publishing something that might be perceived as failure. Alternatively, journals could encourage, and even provide templates, to include negative results in the supplementary materials of accepted articles.

4. Pre-publishing of experimental set-ups: In other scientific fields, such as social psychology, pre-publishing of experimental work (i.e. publishing the methods before the experimental work is done) is becoming the new norm. This guarantees that experimental work that does not confirm the hypothesis, and thus is considered less impactful, still gets published (Nosek and Lakens 2014, Van’t Veer and Giner-Sorolla 2016). In geosciences, this would translate to submitting a paper on the planned experimental work, before it is actually done. Such a paper, without results and discussion, will go through regular peer review, allowing reviewers to make suggestions on how to improve the experimental design. Some journals, such as Frontiers in Earth Science – Hydrosphere, already allow submissions of papers that focus only on methods, and these can be either fieldwork set-ups or computational experiments. The next step would include the acceptance of the methods paper, in which the journal commits to publishing the results regardless of the outcome. Even if the results are different from what was expected, either the hypothesis is rejected or the fieldwork actually failed. Such a publishing structure would encourage and facilitate more open and unbiased science.

We acknowledge that publishing on negative results also comes with pitfalls. First, additional contributions might lead to an unnecessary increase in papers. Second, in some cases it is difficult to draw the line between bad luck and bad science. To overcome these pitfalls, it is essential not to lower the standards of academic publishing. As long as the science is sound, a well-written paper is a valuable addition to the academic literature. Rejecting a hypothesis is as insightful as confirming one, if possible. And showing that a certain experimental set-up gives no, or unexpected, results, is as valuable as when measurements do confirm a priori expectations.

Although some infrastructure for publishing negative results in hydrology already exists, scientists still have to decide to start using it. We cannot force a cultural shift in hydrology, but we can call on everyone in the community to take their responsibility. As researchers, we have a responsibility to share all acquired knowledge including negative results. Only by starting to report on negative results can the current publishing norms be changed. An important role should be played by new generation scientists. Specifically, early career scientists benefit (from a future career point of view) from better acceptance of publishing negative results. In addition, changing the mindset of a community does not happen overnight, but the new generation can show that it is possible.

Experimental work has been very important for the advancement of hydrology. The pressure imposed through the current academic system unfortunately leads to fewer hydrologists dedicating themselves to experimental work, mainly because of the higher risk of negative results. By encouraging the community to report on negative results originating from well-planned and well-executed science, opportunities for scientific output increase, the pressure to publish only high-impact studies decreases and valuable lessons learned are shared to improve future work. For these reasons, we believe that reporting on negative results can encourage experimental hydrology, as long as scientific standards are not lowered. In the end, the only real failure is leaving (negative) results originating from good science unshared.

Acknowledgements

We thank Scott Allen and Wouter Berghuijs for the fruitful discussions that led to significant improvement of this manuscript. We also thank Harsh Beria for his input. We thank Theresa Blume, Ilja van Meerveld and Markus Weiler for the community survey and the subsequent paper. Finally, we thank Flavia Tauro, Theresa Blume and one anonymous reviewer for the constructive reviews of this paper, which helped to further improve it.

Disclosure statement

No potential conflict of interest was reported by the authors.