Crowdsourced geospatial data quality: challenges and future directions

Introduction

A decade ago, Volunteered Geographical Information (VGI) was identified as a new source of information that would blur the traditional boundary between producers and the consumers of data (Goodchild 2007). This form of information has been recognised by multiple names, including crowdsourced geospatial data (Heipke 2010) and user-generated geographic content (Fast and Rinner 2014), to name but a few. Many applications and services benefit from user-generated content contributed by a wide range of users through crowdsourcing projects. VGI has made it possible for a much wider group of contributors to create and share geographical information. Despite the success and popularity of many VGI projects, such as OpenStreetMap (OSM), researchers continue questioning the reliability and fitness for use of crowdsourced data (Haklay et al. 2010, Koukoletsos et al. 2012, Arsanjani et al. 2015, Foody et al. 2015, Salk et al. 2015, Basiri et al. 2016a, Senaratne et al. 2017).

The belief that VGI is contributed by the public, some contributors with little experience and expertise of geospatial data, might have contributed to the perception of the unreliability of this data source. Such issues have impeded the adoption of crowdsourced geospatial data in several projects. While one can argue the importance of the individuals and their levels of expertise based on the concepts of ‘the wisdom of the crowd’ and the collective decision, some questioned the representativeness, i.e. the structure of the crowd and ‘power of the elites’, in many crowdsourcing projects (Leszczynski and Elwood 2015, Ballatore and De Sabbata 2018).

This special section of the International Journal of Geographical Information Science looks at the challenges and future directions of crowdsourced geospatial data with particular attention to the issues stemming from data quality and biases of VGI. This editorial highlights how these issues are discussed and addressed by the articles of this special issue and how the papers highlight emerging technologies, concepts, platforms, debates, and methodologies and techniques within VGI and suggest future research directions. This special issue gathered papers on the topics of crowdsourced geospatial data quality (Ballatore and Arsanjani 2018), thematic uncertainty and consistency across data sources (Hervey and Kuhn 2018), spatial biases (Millar et al., 2018), trust issues within VGI (Severinsen et al. 2019), and contributors behaviour and interactions (Truong et al. 2018).

Crowdsourced data quality challenges

VGI data quality issues

Crowdsourced geographic data quality has been the main core part of many research and studies; Fonte et al. (2015), Senaratne et al. (2017), Fonte et al. (2017), Basiri et al. (2016a), Antoniou and Skopeliti (2015), Goodchild and Li (2012) and several other studies reviewed VGI’s quality assessment and assurance methods. There are several ways to classify the quality assessment methods but the following categories are commonly mentioned in literature with different titles; (a) comparing data against “authoritative” spatial data (Koukoletsos et al. 2012, Dorn et al. 2015) (b) user’s and/or machine learnt rules and patterns for checking the entries (Neis et al. 2012, Jilani et al. 2013; Ali and Schmid 2014, Basiri et al. 2016a, 2016b, Leibovici et al. 2017) (c) gatekeeping and weighting users’ entries (e.g. with respect to the their experiences, expertise, proximity, number of their entries, history and changesets) (McGreavy et al. 2017, Ciampaglia et al. 2018). Having a better understanding of the quality of VGI may help the adoption of crowdsourced geospatial data in some projects as the perception of unreliability may impede the adoption. To address the issues on trust, transparency and reliability Truong et al. (2018) looked at the contributors behaviour and their interactions. They qualified the behaviour of contributors to OpenStreetMap (OSM) through a multigraph approach to reproduce contributor’s interactions in a more comprehensive way. Ballatore and Arsanjani (2018) looked at the origin and development of Wikimapia and discussed some aspect of Wikimapia, including the project’s intellectual property and strategies for quality management. Hervey and Kuhn (2018) explored uncertainty with locational data obtained from social networks. They presented a taxonomy of things that can be located from social network posts and a means to describe them to users. Severinsen et al. (2019) present a formulaic model to addresses VGI quality issues by quantifying trust in VGI. Their ‘VGTrust’ model assessed information about a data author, and the spatial and temporal trust associated with the data they create in order to produce an overall VGTrust rating metric.

VGI biases

While quality issues of crowd-sourced data have been studied widely, the identification and estimation of biases in crowd-sourced projects have not received the same attention. This is due mainly to the lack of availability of the (geo-) demographic data of the contributors, which is either unrecorded (e.g. OpenStreetMap) or inaccessible due to commercial interest (e.g. in the now defunct Google MapMaker). Therefore understanding the impacts of demographic biases on crowdsourced maps is challenged by a lack of data on these data (Mullen et al. 2015, Haklay 2016, Basiri et al. 2018, Gardner and Mooney; 2018, Gardner et al. 2018). Millar et al. (2018) looked at the biases and studies the lack of citizen science monitoring programs. The study focuses on natural and demographic biases related to the location, accessibility, size and general attractiveness of lakes in Ontario.

Any VGI project is biased in one or more ways (Basiri et al. 2018). At the first glance, it seems that all the data contributed through VGI projects are ‘voluntary response samples’, which are always biased as they only include people who have chosen to volunteer (DeMaio 1980). Whereas a random sample would need to include people whether or not they choose to volunteer (Goyder 1986). Thus inferences from a voluntary response sample are not as credible as conclusions are based on a random sample of the entire population. While crowdsourcing projects are technically open to the whole population, and of course, anyone should be able to contribute, recent studies (Mullen et al. 2015, Yang et al. 2016, Zhu et al. 2017, Gardner et al. 2018) have shown that even the most popular crowdsourced projects, such as OSM, are biased by the contribution patterns of its contributors, i.e. that a small percentage of the community contribute the greatest proportion of activity (the ‘long tail effect’ or 90–9-1 rule (Haklay 2016)). Ballatore and Arsanjani (2018) studied the popularity of the project using behavioural data from Google Trends and compared the geography of Interest in Wikimapia with OpenStreetMap, from a temporal and spatial perspective. And found while OpenStreetMap attracts more interest in high-income countries, Wikimapia emerges as relatively more popular in low- and middle-income countries, countering the received notion of VGI as a Global North phenomenon. Therefore, we might questions the use of the terms ‘crowd’ and ‘public’ used in many crowdsourcing and public participatory projects by virtue of this skewed pattern of participation. This excludes the projects which may require a relatively higher experience level, access to some resources, or may limit participation to a specific geography or particular time interval due to the nature of the project (Morschheuser et al. 2018).

In addition to voluntary response bias, the volunteers, as individuals, can have different aspects and levels of quality of judgement and decision making (Hammond 2000). Their decisions, opinions, and preferences could be significantly represented and/or influence their contribution (e.g. data). Although there are some arguments based on the concepts of ‘the wisdom of the crowd’ trying to undermine or counter- balance the impacts of the individuals’ biases on the collective decision, there are two challenges to this notion: Firstly, representativeness, i.e. the structure of the crowd and ‘power of the elites’, in many crowdsourcing projects have been questioned (Comber et al. 2016), (See et al. 2013). For example, both Elwood (2010) and Leszczynski and Elwood (2015) have problematized participation biases in VGI on the grounds of a failure of crowdsourced mapping projects to represent the interests of the wider public, specifically those of women. Similarly, Ballatore and De Sabbata (2018) have explored the extent to which VGI are representative of the wider population of the geospatial units they represent.

The issues of representation could, therefore, be an issue in terms of biases, however, some believe that the super active contributors are experts and so it is better to leave some decisions in their hands. While Giles (2005) and Rajagopalan et al. (2011) showed that collective decision-making can be more accurate than experts’ comments, accuracy does not necessarily show all the aspects of quality and might not be even loosely correlated with potential bias. In terms of biases Greenstein and Zhu (2017) found the knowledge produced by the crowd are not necessarily less biased than the knowledge produced by experts. Ciampaglia et al. (2018) confirmed this by using Wikipedia contents, however, they found both biases and data quality could be moderated if substantial revisions and supervisions (of the gatekeepers) were implemented.

The second challenge to the notion of the ‘wisdom of the crowd’, is the process of many VGI projects which is not based on a collective decision but instead on crowd ‘participation’. The difference is relatively implicit but highly important; the participants do not vote for/against every single decision or entry. The collection of individual decisions does not necessarily mean the collective decision making. Therefore the wisdom of the crowd may not be relevant to such projects as the individual bias can remain at micro-level. As the crowd makes decisions individually in a participatory project, the results of an individual’s contributions could be biased. Therefore for these projects the case of ‘given enough eyeballs, all bugs are shallow’ (Raymond 1998) is no longer valid as there is not enough revision/votes for each piece of information contributed by volunteers. Ciampaglia et al. (2018) found that crowd-sourced content can also produce a large sample with a great variety of biased opinions.

Future directions

The papers within this special issue have looked at some of the challenges and issues of crowdsourced geographic data, including data quality, biases, and trust issues. They have also provided some solutions to either address or have a better understanding of the implications of these issues. It seems that research focus of research on VGI has been moving towards: the structure of the ‘crowd’ and volunteers (geo-)demographic biases, the impact of having such biases in different VGI projects, research on how to promote diversity of contributors communities, addressing the issues of transparency and trust while protecting the privacy of the contributors, and working on intellectual property of crowdsourced data and projects. It seems that future research looks beyond VGI as just a way to create maps but as a complex and more democratic, reproducible and open but reliable system engaging society, promoting diversity, collaborations, and wider engagement.

Disclosure statement

No potential conflict of interest was reported by the authors.