Reproducible Research Practices and Barriers to Reproducible Research in Geography: Insights from a Survey

References

• Aguinis, H., and A. M. Solarino. 2019. Transparency and replicability in qualitative research: The case of interviews with elite informants. Strategic Management Journal 40 (8):1291–1315. doi: 10.1002/smj.3015.
   Web of Science ®Google Scholar
• American Association for Public Opinion Research (AAPOR). 2023. 2023 standard definitions: Final dispositions of case codes and outcome rates for surveys. 10th ed. Alexandria, VA: AAPOR. https://www.aapor.org/AAPOR_Main/media/publications/Standard-Definitions20169theditionfinal.pdf.
   Google Scholar
• Anderson, M. S., B. C. Martinson, and R. De Vries. 2007. Normative dissonance in science: Results from a national survey of U.S. scientists. Journal of Empirical Research on Human Research Ethics: JERHRE 2 (4):3–14. doi: 10.1525/jer.2007.2.4.3.
   PubMed Web of Science ®Google Scholar
• Baker, M. 2016. 1,500 scientists lift the lid on reproducibility. Nature 533 (7604):452–54. doi: 10.1038/533452A.
   PubMed Web of Science ®Google Scholar
• Balz, T., and F. Rocca. 2020. Reproducibility and replicability in SAR remote sensing. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13:3834–43. doi: 10.1109/JSTARS.2020.3005912.
   Google Scholar
• Barba, L. A. 2018. Terminologies for reproducible research. arXiv Preprint arXiv:1802.03311. doi: 10.48550/arXiv.1802.03311.
   Google Scholar
• Boulbes, D. R., T. Costello, K. Baggerly, F. Fan, R. Wang, R. Bhattacharya, X. Ye, and L. M. Ellis. 2018. A survey on data reproducibility and the effect of publication process on the ethical reporting of laboratory research. Clinical Cancer Research: An Official Journal of the American Association for Cancer Research 24 (14):3447–55. doi: 10.1158/1078-0432.CCR-18-0227.
   PubMed Web of Science ®Google Scholar
• Brunsdon, C. 2016. Quantitative methods I: Reproducible research and quantitative geography. Progress in Human Geography 40 (5):687–96. doi: 10.1177/0309132515599625.
   Web of Science ®Google Scholar
• Byrne, M. 2017. Making progress toward open data: Reflections on data sharing at PLoS ONE. https://everyone.plos.org/2017/05/08/making-progress-toward-open-data/.
   Google Scholar
• Camerer, C. F., A. Dreber, E. Forsell, T.-H. Ho, J. Huber, M. Johannesson, M. Kirchler, J. Almenberg, A. Altmejd, T. Chan, et al. 2016. Evaluating replicability of laboratory experiments in economics. Science 351 (6280):1433–36. doi: 10.1126/science.aaf0918.
   PubMed Web of Science ®Google Scholar
• Camerer, C. F., A. Dreber, F. Holzmeister, T.-H. Ho, J. Huber, M. Johannesson, M. Kirchler, G. Nave, B. A. Nosek, T. Pfeiffer, et al. 2018. Evaluating the replicability of social science experiments in nature and science between 2010 and 2015. Nature Human Behaviour 2 (9):637–44. doi: 10.1038/s41562-018-0399-z.
   PubMed Web of Science ®Google Scholar
• Chang, A. C., and P. Li. 2015. Is economics research replicable? Sixty published papers from thirteen journals say “usually not.” Finance and Economics Discussion Series 2015 (83):1–26. https://www.federalreserve.gov/econres/feds/is-economics-research-replicable-sixty-published-papers-from-thirteen-journals-say-quotusually-notquot.htm. doi: 10.17016/FEDS.2015.083.
   Google Scholar
• Clarivate. 2023. Web of Science journal citation reports. Accessed January 15, 2022. https://clarivate.com.
   Google Scholar
• Collberg, C., T. Proebsting, G. Moraila, A. Shankaran, Z. Shi, and A. M. Warren. 2014. Measuring reproducibility in computer systems research. Technical Report 37, Department of Computer Science, University of Arizona. http://reproducibility.cs.arizona.edu/v1/tr.pdf
   Google Scholar
• Dillman, D. A., J. D. Smyth, and L. M. Christian. 2014. Internet, phone, mail, and mixed-mode surveys: The tailored design method. Hoboken, NJ: Wiley.
   Google Scholar
• Earp, B. D., and D. Trafimow. 2015. Replication, falsification, and the crisis of confidence in social psychology. Frontiers in Psychology 6:621. doi: 10.3389/fpsyg.2015.00621.
   PubMed Web of Science ®Google Scholar
• Fanelli, D. 2009. How many scientists fabricate and falsify research? A systematic review and meta-analysis of survey data. PLoS ONE 4 (5): e5738. doi: 10.1371/journal.pone.0005738.
   PubMed Web of Science ®Google Scholar
• Fraser, H., T. Parker, S. Nakagawa, A. Barnett, and F. Fidler. 2018. Questionable research practices in ecology and evolution. PLoS ONE 13 (7):e0200303. doi: 10.1371/journal.pone.0200303.
   PubMed Web of Science ®Google Scholar
• Goodchild, M. F., A. S. Fotheringham, P. Kedron, and W. Li. 2021. Introduction: Forum on reproducibility and replicability in geography. Annals of the American Association of Geographers 111 (5):1271–74. doi: 10.1080/24694452.2020.1806030.
   Web of Science ®Google Scholar
• Gundersen, O. E., and S. Kjensmo. 2018. State of the art: Reproducibility in artificial intelligence. Proceedings of the AAAI Conference on Artificial Intelligence 32 (1):1644–51. doi: 10.1609/aaai.v32i1.11503.
   Google Scholar
• Holler, J., P. Kedron, D. An-Pham, D. Burt, and J. Zhou. 2023. Reproduction of Chakraborty 2021 distribution of COVID-19 and intra-categorical analysis of people with disabilities. OSF. osf.io/s5mtq. doi: 10.17605/OSF.IO/S5MTQ.
   Google Scholar
• Holmes, A. G. D. 2020. Researcher positionality—A consideration of its influence and place in qualitative research—A new researcher guide. Shanlax International Journal of Education 8 (2):1–9. doi: 10.34293/education.v8i2.1477.
   Google Scholar
• Iqbal, S. A., J. D. Wallach, M. J. Khoury, S. D. Schully, and J. P. Ioannidis. 2016. Reproducible research practices and transparency across the biomedical literature. PLoS Biology 14 (1):e1002333. doi: 10.1371/journal.pbio.1002333.
   PubMed Web of Science ®Google Scholar
• Kedron, P., S. Bardin, T. D. Hoffman, M. Sachdeva, M. Quick, and J. Holler. 2022. A replication of DiMaggio et al. (2020) in Phoenix, AZ. Annals of Epidemiology 74:8–14. doi: 10.1016/j.annepidem.2022.05.005.
   PubMed Web of Science ®Google Scholar
• Kedron, P., S. Bardin, J. Holler, J. Gilman, B. Grady, M. Seeley, X. Wang, and W. Yang. 2023. A framework for moving beyond computational reproducibility: Lessons from three reproductions of geographical analyses of COVID-19. Geographical Analysis. doi: 10.1111/gean.12370.
   Google Scholar
• Kedron, P., and J. Holler. 2022. Replication and the search for the laws in the geographic sciences. Annals of GIS 28 (1):45–56. doi: 10.1080/19475683.2022.2027011.
   Web of Science ®Google Scholar
• Kedron, P., J. Holler, and S. Bardin. 2023. A survey of reproducibility in geographic research. OSF. osf.io/5yeq8 doi: 10.17605/OSF.IO/5YEQ8.
   Google Scholar
• Kedron, P., J. Holler, S. Bardin, and Z. Hilgendorf. 2022. Reproducibility, replicability, and open science practices in the geographical sciences. OSF. doi: 10.17605/OSF.IO/C5A2R.
   Google Scholar
• Kedron, P., W. Li, S. Fotheringham, and M. Goodchild. 2021. Reproducibility and replicability: Opportunities and challenges for geospatial research. International Journal of Geographical Information Science 35 (3):427–45. doi: 10.1080/13658816.2020.1802032.
   Web of Science ®Google Scholar
• Konkol, M., C. Kray, and M. Pfeiffer. 2019. Computational reproducibility in geoscientific papers: Insights from a series of studies with geoscientists and a reproduction study. International Journal of Geographical Information Science 33 (2):408–29. doi: 10.1080/13658816.2018.1508687.
   Web of Science ®Google Scholar
• National Academies of Science, Engineering, and Medicine (NASEM). 2019. Reproducibility and replicability in science. Washington, DC: National Academies Press. doi: 10.17226/25303.
   Google Scholar
• Nosek, B. A., J. R. Spies, and M. Motyl. 2012. Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science: A Journal of the Association for Psychological Science 7 (6):615–31. doi: 10.1177/1745691612459058.
   PubMedGoogle Scholar
• Nüst, D., C. Granell, B. Hofer, M. Konkol, F. O. Ostermann, R. Sileryte, and V. Cerutti. 2018. Reproducible research and GIScience: An evaluation using agile conference papers. PeerJ 6:e5072. doi: 10.7287/peerj.preprints.26561v1.
   PubMed Web of Science ®Google Scholar
• Nüst, D., and M. Hinz. 2019. containerit: Generating dockerfiles for reproducible research with R. Journal of Open Source Software 4 (40):1603. doi: 10.21105/joss.01603.
   Google Scholar
• Nüst, D., F. O. Ostermann, R. Sileryte, B. Hofer, C. Granell, M. Teperek, A. Graser, K. Broman, K. Hettne, and M. Konkol. 2023. Reproducible publications at agile conferences. OSF. osf.io/phmce. doi: 10.17605/OSF.IO/PHMCE.
   Google Scholar
• Open Science Collaboration. 2015. Estimating the reproducibility of psychological science. Science 349 (6251):aac4716. doi: 10.1126/science.aac4716.
   PubMed Web of Science ®Google Scholar
• Ostermann, F. O., and C. Granell. 2017. Advancing science with VGI: Reproducibility and replicability of recent studies using VGI. Transactions in GIS 21 (2):224–37. doi: 10.1111/tgis.12195.
   Web of Science ®Google Scholar
• Ostermann, F. O., D. Nüst, C. Granell, B. Hofer, and M. Konkol. 2021. Reproducible research and GIScience: An evaluation using GIScience conference papers. In 11th international conference on geographic information science (GISience 2021). Vol. 208, Part 2, , ed. K. Janowicz and J. A. Verstegen, 2:1–2:16. Dagstuhl, Germany: Schloss Dagstuhl – Leibniz-Zentrum für Informatik. doi: 10.4230/LIPIcs.GIScience.2021.II.2.
   Google Scholar
• Paez, A. 2021. Reproducibility of research during COVID-19: Examining the case of population density and the basic reproductive rate from the perspective of spatial analysis. Geographical Analysis 54 (4):860–80. doi: 10.1111/gean.12307.
   Web of Science ®Google Scholar
• Plesser, H. E. 2017. Reproducibility vs. replicability: A brief history of a confused terminology. Frontiers in Neuroinformatics 11:76. doi: 10.3389/fninf.2017.00076.
   PubMed Web of Science ®Google Scholar
• Pratt, G. 2009. “Positionality.” In The Dictionary of Human Geography, ed. D. Gregory, R. Johnston, G. Pratt, M. Watts, and S. Whatmore, 556–57. Malden, MA: Wiley-Blackwell.
   Google Scholar
• Qin, D. 2016. Positionality. The Wiley Blackwell Encyclopedia of Gender and Sexuality Studies, ed. N. A. Naples. London: John Wiley.
   Google Scholar
• Raghupathi, W., V. Raghupathi, and J. Ren. 2022. Reproducibility in computing research: An empirical study. IEEE Access 10:29207–23. doi: 10.1109/ACCESS.2022.3158675.
   Web of Science ®Google Scholar
• Ranstam, J., M. Buyse, S. L. George, S. Evans, N. L. Geller, B. Scherrer, E. Lesaffre, G. Murray, L. Edler, J. L. Hutton, et al. 2000. Fraud in medical research: An international survey of biostatisticians. Controlled Clinical Trials 21 (5):415–27. doi: 10.1016/s0197-2456(00)00069-6.
   PubMedGoogle Scholar
• Roberts, K., A. Dowell, and J.-B. Nie. 2019. Attempting rigour and replicability in thematic analysis of qualitative research data: A case study of codebook development. BMC Medical Research Methodology 19 (1):66. doi: 10.1186/s12874-019-0707-y.
   PubMed Web of Science ®Google Scholar
• Schmidt, S. 2009. Shall we really do it again? The powerful concept of replication is neglected in the social sciences. Review of General Psychology 13 (2):90–100. doi: 10.1037/a0015108.
   Web of Science ®Google Scholar
• Scopus. 2023. Home page. Accessed January 20, 2022. http://www.scopus.com.
   Google Scholar
• Singleton, A. D., S. Spielman, and C. Brunsdon. 2016. Establishing a framework for open geographic information science. International Journal of Geographical Information Science 30 (8):1507–21. doi: 10.1080/13658816.2015.1137579.
   Web of Science ®Google Scholar
• Stodden, V., M. S. Krafczyk, and A. Bhaskar. 2018. Enabling the verification of computational results: An empirical evaluation of computational reproducibility. In Proceedings of the first international workshop on practical reproducible evaluation of computer systems, 1–5. London and New York: Association for Computing Machinery. doi: 10.1145/3214239.3214242.
   Google Scholar
• Stodden, V., M. McNutt, D. H. Bailey, E. Deelman, Y. Gil, B. Hanson, M. A. Heroux, J. P. A. Ioannidis, and M. Taufer. 2016. Enhancing reproducibility for computational methods. Science 354 (6317):1240–41. doi: 10.1126/science.aah6168.
   PubMed Web of Science ®Google Scholar
• Sui, D., and P. Kedron. 2021. Reproducibility and replicability in the context of the contested identities of geography. Annals of the American Association of Geographers 111 (5):1275–83. doi: 10.1080/24694452.2020.1806024.
   Web of Science ®Google Scholar
• Wainwright, J. 2021. Is critical human geography research replicable? Annals of the American Association of Geographers 111 (5):1284–90. doi: 10.1080/24694452.2020.1806025.
   Web of Science ®Google Scholar
• Wilson, J. P., K. Butler, S. Gao, Y. Hu, W. Li, and D. J. Wright. 2021. A five-star guide for achieving replicability and reproducibility when working with GIS software and algorithms. Annals of the American Association of Geographers 111 (5):1311–17. doi: 10.1080/24694452.2020.1806026.
   Web of Science ®Google Scholar
• Yin, D., Y. Liu, H. Hu, J. Terstriep, X. Hong, A. Padmanabhan, and S. Wang. 2019. CyberGIS-Jupyter for reproducible and scalable geospatial analytics. Concurrency and Computation: Practice and Experience 31 (11):e5040. doi: 10.1002/cpe.5040.
   Web of Science ®Google Scholar