Variability of operator performance in remote-sensing image interpretation: the importance of human and external factors

Abstract

This study tackles a common, yet underrated problem in remote-sensing image analysis: the fact that human interpretation is highly variable among different operators. Despite current technological advancements, human perception and interpretation are still vital components of the map-making process. Consequently, human errors can considerably bias both mapping and modelling results. In our study, we present a web-based tool to quantify operator variability and to identify the human and external factors affecting this variability. Human operators were given a series of images and were asked to hand-digitize different point, line, and polygon objects. The quantification of performance variability was achieved using both thematic and positional accuracy measures. Subsequently, a series of questions related to demographics, experience, and personality were asked, and the answers were also quantified. Correlation and regression analysis was then used to explain the variability in operator performance. From our study, we conclude that: (1) humans were seldom perfect in visual interpretation; (2) some geographic objects were more complex to accurately digitize than others; (3) there was a high degree of variability among image interpreters when hand-digitizing the same objects; and (4) operator performance was mainly determined by demographic, non-cognitive, and cognitive personality factors, whereas external and technical factors influenced operator performance to a lesser extent. Finally, the results also indicated a gradual decline in performance over time, mimicking classical mental fatigue effects.

Acknowledgements

AGIV is gratefully acknowledged for providing the images and reference data. The members of the WAVARS project steering committee provided useful comments. We also thank the anonymous reviewers for their valuable suggestions and the time they have invested in their review.

Funding

The Belgian Federal Science Policy Office (BELSPO) provided funding through the Research Programme for Earth Observation Stereo II [grant number SR/02/121].