Producing consistent visually interpreted land cover reference data: learning from feedback

References

• Boud, David, and Elizabeth Molloy. 2013. “Rethinking Models of Feedback for Learning: The Challenge of Design.” Assessment & Evaluation in Higher Education 38 (6): 698–712. doi:10.1080/02602938.2012.691462.
   Web of Science ®Google Scholar
• Breiman, Leo. 2001. “Random Forest.” Machine Learning 45: 5–32. doi:10.1023/A:1010933404324.
   Web of Science ®Google Scholar
• Breiman, Leo. 2002. Manual on Setting up, Using, and Understanding Random Forests V3.1. Statistics Department University of California Berkeley. https://www.stat.berkeley.edu/~breiman/Using_random_forests_V3.1.pdf.
   Google Scholar
• CEOS. 2019. “CEOS Working Group on Calibration and Validation: The Land Product Validation Subgroup.” https://lpvs.gsfc.nasa.gov/.
   Google Scholar
• CGLS. 2019. ‘Copernicus Global Land Service.’ https://land.copernicus.eu/global/index.html.
   Google Scholar
• Comber, Alexis, Linda See, Steffen Fritz, Marijn Van der Velde, Christoph Perger, and Giles Foody. 2013. “Using Control Data to Determine the Reliability of Volunteered Geographic Information About Land Cover.” International Journal of Applied Earth Observation and Geoinformation 23 (1): 37–48. doi:10.1016/j.jag.2012.11.002.
   Google Scholar
• da Silva, S. M., S. C. M. Rodrigues, M. A. S. Bissaco, T. Scardovelli, S. R. M. S. Boschi, M. A. Marques, M. F. Santos, and A. P. Silva. 2019. “A Novel Online Training Platform for Medical Image Interpretation.” In World Congress on Medical Physics and Biomedical Engineering 2018. IFMBE Proceedings (Vol. 68), edited by L. Lhotska, L. Sukupova, I. Lacković, and G. Ibbott. Singapore: Springer Singapore. doi:10.1007/978-981-10-9035-6_153.
   Google Scholar
• Hothorn, Torsten, Peter Buehlmann, Sandrine Dudoit, Annette Molinaro, and Mark Van Der Laan. 2006. “Survival Ensembles.” Biostatistics 7 (3): 355–373. doi: 10.1093/biostatistics/kxj011
   PubMed Web of Science ®Google Scholar
• Jia, Xiaowei, Ankush Khandelwal, James Gerber, Kimberly Carlson, Paul West, and Vipin Kumar. 2016. “Learning Large-Scale Plantation Mapping from Imperfect Annotators.” In 2016 IEEE International Conference on Big Data (Big Data), 1192–1201. IEEE. doi:10.1109/BigData.2016.7840723.
   Google Scholar
• Lallé, Sébastien, Cristina Conati, and Giuseppe Carenini. 2016. “Prediction of Individual Learning Curves Across Information Visualizations.” User Modeling and User-Adapted Interaction 26 (4): 307–345. doi:10.1007/s11257-016-9179-5.
   Web of Science ®Google Scholar
• Lesiv, Myroslava, Linda See, Juan Laso Bayas, Tobias Sturn, Dmitry Schepaschenko, Mathias Karner, Inian Moorthy, Ian McCallum, and Steffen Fritz. 2018. “Characterizing the Spatial and Temporal Availability of Very High Resolution Satellite Imagery in Google Earth and Microsoft Bing Maps as a Source of Reference Data.” Land 7 (4): 118. doi:10.3390/land7040118.
   Web of Science ®Google Scholar
• Liaw, Andy, and Matthew Wiener. 2002. “Classification and Regression by Random Forest.” R News 2 (3): 18–22. https://cran.r-project.org/doc/Rnews/.
   Google Scholar
• Lillesand, Thomas, Ralph W Kiefer, and Jonathan Chipman. 2008. Remote Sensing and Image Interpretation. 6th ed. Hoboken, NJ: John Wiley & Sons.
   Google Scholar
• McGill, Robert, John W Tukey, and Wayne A Larsen. 1978. “Variations of Box Plots.” The American Statistician 32 (1): 12. doi:10.2307/2683468.
   Web of Science ®Google Scholar
• McRoberts, Ronald E., Stephen V. Stehman, Greg C. Liknes, Erik Næsset, Christophe Sannier, and Brian F. Walters. 2018. “The Effects of Imperfect Reference Data on Remote Sensing-Assisted Estimators of Land Cover Class Proportions.” ISPRS Journal of Photogrammetry and Remote Sensing 142 (February): 292–300. doi:10.1016/j.isprsjprs.2018.06.002.
   Google Scholar
• Mincer, Jacob. 1974. “Schooling, Experience, and Earnings.” Human Behavior & Social Institutions, no. 2.
   Google Scholar
• Narciss, Susanne. 2008. “Feedback Strategies for Interactive Learning Tasks.” In Handbook of Research on Educational Communications and Technology, edited by J.M. Spector, M.D. Merrill, J. Van Merriënboer, and M.P. Driscoll, 125–143. Mahwah, NJ: Erlbaum.
   Google Scholar
• Olofsson, Pontus, Giles M. Foody, Martin Herold, Stephen V. Stehman, Curtis E. Woodcock, and Michael A. Wulder. 2014. “Good Practices for Estimating Area and Assessing Accuracy of Land Change.” Remote Sensing of Environment 148 (May): 42–57. doi:10.1016/j.rse.2014.02.015.
   Google Scholar
• Olofsson, Pontus, Stephen V. Stehman, Curtis E. Woodcock, Damien Sulla-Menashe, Adam M. Sibley, Jared D. Newell, Mark A. Friedl, and Martin Herold. 2012. “A Global Land-Cover Validation Data Set, Part I: Fundamental Design Principles.” International Journal of Remote Sensing 33 (18): 5768–5788. doi:10.1080/01431161.2012.674230.
   Web of Science ®Google Scholar
• Peel, Murray C., Brian L. Finlayson, and Thomas A. McMahon. 2007. “Updated World Map of the Köppen-Geiger Climate Classification.” Hydrology and Earth System Sciences Discussions 4 (2): 439–473. doi: 10.5194/hessd-4-439-2007
   Google Scholar
• Pengra, Bruce W., Stephen V. Stehman, Josephine A. Horton, Daryn J. Dockter, Todd A. Schroeder, Zhiqiang Yang, Warren B. Cohen, Sean P. Healey, and Thomas R. Loveland. 2019. “Quality Control and Assessment of Interpreter Consistency of Annual Land Cover Reference Data in an Operational National Monitoring Program.” Remote Sensing of Environment 238. doi:10.1016/j.rse.2019.111261.
   Web of Science ®Google Scholar
• Powell, R. L., N. Matzke, C. de Souza, M. Clark, I. Numata, L. L. Hess, and D. A. Roberts. 2004. “Sources of Error in Accuracy Assessment of Thematic Land-Cover Maps in the Brazilian Amazon.” Remote Sensing of Environment 90 (2): 221–234. doi:10.1016/j.rse.2003.12.007.
   Web of Science ®Google Scholar
• R Core Team. 2017. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing. http://www.r-project.org/.
   Google Scholar
• Schank, Roger C., Tamara R. Berman, and Kimberli A. Macpherson. 1999. “Learning by Doing.” In Instructional-Design Theories and Models: A New Paradigm of Instructional Theory, edited by C. M. Reigeluth, 161–181. Mahway, NJ: Erlbaum.
   Google Scholar
• See, Linda, Steffen Fritz, Christoph Perger, Christian Schill, Ian McCallum, Dmitry Schepaschenko, Martina Duerauer, et al. 2015. “Harnessing the Power of Volunteers, the Internet and Google Earth to Collect and Validate Global Spatial Information Using Geo-Wiki.” Technological Forecasting and Social Change 98 (September): 324–335. doi:10.1016/j.techfore.2015.03.002.
   Google Scholar
• Sievert, Carson. 2018. “Plotly for R.” https://plotly-book.cpsievert.me.
   Google Scholar
• Speelman, Craig P., and Kim Kirsner. 2005. Beyond the Learning Curve: The Construction of Mind. Oxford: Oxford University Press.
   Google Scholar
• Strobl, Carolin, Anne-Laure Boulesteix, Thomas Kneib, Thomas Augustin, and Achim Zeileis. 2008. “Conditional Variable Importance for Random Forests.” BMC Bioinformatics 9 (1): 307. http://www.biomedcentral.com/1471-2105/9/307. doi: 10.1186/1471-2105-9-307
   PubMed Web of Science ®Google Scholar
• Strobl, Carolin, Anne-Laure Boulesteix, Achim Zeileis, and Torsten Hothorn. 2007. “Bias in Random Forest Variable Importance Measures: Illustrations, Sources and a Solution.” BMC Bioinformatics 8 (1): 25. doi:10.1186/1471-2105-8-25.
   PubMed Web of Science ®Google Scholar
• Tarko, Agnieszka, Sytze de Bruin, and Arnold K. Bregt. 2018. “Comparison of Manual and Automated Shadow Detection on Satellite Imagery for Agricultural Land Delineation.” International Journal of Applied Earth Observation and Geoinformation 73 (April): 493–502. doi:10.1016/j.jag.2018.07.020.
   Google Scholar
• Tsendbazar, N.-E., M. Herold, S. de Bruin, M. Lesiv, S. Fritz, R. Van De Kerchove, M. Buchhorn, M. Duerauer, Z. Szantoi, and J.-F. Pekel. 2018. “Developing and Applying a Multi-Purpose Land Cover Validation Dataset for Africa.” Remote Sensing of Environment 219 (March): 298–309. doi:10.1016/j.rse.2018.10.025.
   Google Scholar
• Tsendbazar, N.-E., M. Herold, A. Tarko, L. Li, and M. Lesiv. 2019. “Copernicus Global Land Operations ‘Vegetation and Energy’, ‘CGLOPS-1’, Validation Report, Moderate Dynamic Land Cover Collection 100 m, Version 2.” https://land.copernicus.eu/global/sites/cgls.vito.be/files/products/CGLOPS1_VR_LC100m-V2.0_I1.00.pdf.
   Google Scholar
• Tuia, Devis, and Jordi Munoz-Mari. 2012. “Putting the User into the Active Learning Loop: Towards Realistic but Efficient Photointerpretation.” In 2012 IEEE International Geoscience and Remote Sensing Symposium, 75–78. IEEE. doi:10.1109/IGARSS.2012.6351633.
   Google Scholar
• Van Coillie, Frieke M.B., Soetkin Gardin, Frederik Anseel, Wouter Duyck, Lieven P.C. Verbeke, and Robert R. De Wulf. 2014. “Variability of Operator Performance in Remote-Sensing Image Interpretation: The Importance of Human and External Factors.” International Journal of Remote Sensing 35 (2): 754–778. doi:10.1080/01431161.2013.873152.
   Web of Science ®Google Scholar
• Wei, Taiyun, and Viliam Simko. 2017. “R Package ‘Corrplot’: Visualization of a Correlation Matrix.” https://github.com/taiyun/corrplot.
   Google Scholar
• Zhao, Yuanyuan, Le Yu Duole Feng, Linda See, Steffen Fritz, Christoph Perger, and Peng Gong. 2017. “Assessing and Improving the Reliability of Volunteered Land Cover Reference Data.” Remote Sensing 9 (10): 1034. doi:10.3390/rs9101034.
   Web of Science ®Google Scholar
• Zhao, Yuanyuan, Le Yu Peng Gong, Luanyun Hu, Xueyan Li, Congcong Li, Haiying Zhang, et al. 2014. “Towards a Common Validation Sample Set for Global Land-Cover Mapping.” International Journal of Remote Sensing 35 (13): 4795–4814. doi:10.1080/01431161.2014.930202.
   Web of Science ®Google Scholar