Anomaly detection for volunteered geographic information: a case study of Safecast data

ABSTRACT

Volunteered Geographic Information (VGI), defined as geographic information contributed voluntarily by individuals, has grown exponentially with the aid of ubiquitous GPS-enabled technologies. VGI projects have generated a large amount of geographic data, providing a new data source for scientific research. However, many scientists are concerned about the quality of VGI data for research, given the lack of rigorous and systematic quality control procedures. This study contributes to the improvement of quality control procedures by proposing a Cross-Volunteer Referencing Anomaly Detection (CVRAD) method to filter anomalous data, using the crowdsourced Safecast radiation data as a case study. The anomaly detection method is validated using two data sets: (1) an official radiation survey data set collected by the KURAMA car-borne system, (2) a set of anomalous Safecast measurements filtered by Safecast moderators. The validation results show that the proposed CVRAD method outperformed the 1.5 IQR benchmark method in minimizing the overall measurement error and detecting abnormal imports of Safecast measurements, thus demonstrating the effectiveness of the proposed method in improving the overall accuracy of crowdsourced radiation measurements.

KEYWORDS:

• Anomaly detection
• Volunteered Geographic Information (VGI)
• data quality
• participatory sensing
• radiation

1. Introduction

The increasing availability of GPS-enabled smart devices and Web 2.0 technologies has boosted the voluntary collection, organization, and sharing of geographic information from the public. This type of geographic information is commonly referred to as Volunteered Geographic Information (VGI) (Goodchild 2007). The availability of large volumes of VGI data has drawn considerable attention from researchers, particularly in circumstances where the massive quantity of data can compensate for the lack of expert quality (Trumbull et al. 2000, Bonney et al. 2014). But, researchers are presented with particular challenges in using the VGI data for scientific investigation (Sui et al. 2013). Among all the challenges a significant barrier of utilizing VGI data is the varied data quality, particularly uncertainty about errors or malicious data entries in the contributions (Goodchild and Li 2012, Bordogna et al. 2016, Basiri et al. 2019).

Apart from tackling the inherent difficulties in implementing a systematic order to gather a large quantity of data, some VGI projects are trying to improve their quality of data as well. A common strategy has been to organize physical or online workshops for volunteers to equip them with the necessary skill set to operate the data collection devices (Safecast 2017, Bird Academy 2018). This strategy not only prepares the volunteers for the project but also ensures a higher quality of data by increasing the level of understanding of the observed phenomenon (Goodchild 2007, Antoniou and Skopeliti 2017). However, a large number of VGI projects do not require any prerequisite experience or skills from the volunteers to contribute to the data collection process (Haklay 2013). Additionally, these VGI projects tend to adopt straightforward data collection protocols to maximize the number of participants, thereby making the data collected prone to observational error and bias (Minghini et al. 2017). Furthermore, several VGI-based projects have suffered from the lack of authoritative reference data to validate the data gathered (Antoniou and Skopeliti 2015, Haklay et al. 2014). These challenges have motivated an increasing number of researchers to assess the data quality of various VGI projects to enrich our understanding of the current state of the quality of VGI, develop innovative methods to address the quality issues, and efficiently harness the immense amount of VGI data (Senaratne et al. 2017, Moreri et al. 2018, Severinsen et al. 2019).

Senaratne et al. (2017) conducted an extensive review of the state-of-the-art quality assessment methods for VGI. According to their survey, the majority of the existing VGI evaluation methods follow the three approaches proposed by Goodchild and Li (2012): the crowd-sourcing approach, the social approach, and the geographic approach. The crowd-sourcing approach refers to the ability of a group to validate and correct errors that one individual might make, and the ability of the crowd to converge on the truth. The social approach relies on trusted individuals to serve as gatekeepers. The geographic approach makes use of the existing body of geographic knowledge to validate the volunteered geographic information. While most of the reviewed papers focused on the quality assessment of map-based, text-based, and image-based VGI, little attention has been given to participatory sensing projects. Participatory sensing is a type of VGI that uses mobile sensor networks formed by humans, vehicles, or animals for data collection (Pon et al. 2005, Burke et al. 2006). In the past decade, a growing number of participatory sensing applications have been deployed successfully in various domains, including environmental monitoring (Rana et al. 2010, Méndez et al. 2011, Hasenfratz et al. 2012, Brown et al. 2016), transportation monitoring (Eriksson et al. 2008, Mohan et al. 2008), and urban sensing (Miluzzo et al. 2011, Al-Husain et al. 2013). The omission of participatory sensing VGI projects in the 56 studies reviewed by Senaratne et al. (2017) clearly shows a lack of attention in geography outlets to addressing the quality issues of participatory sensing data.

Nevertheless, participatory sensing data provide an unprecedented opportunity to the science community to understand the dynamic phenomena of the environment (Pon et al. 2005, Abdelzaher et al. 2007). Through a pervasive mobile sensor network formed by citizens, millions of environmental measurements are collected in near real-time. In addition to these measurements, data collected in participatory sensing often come with GPS coordinates and timestamps. The majority of the studies on measuring, controlling, and improving the data quality of participatory sensing projects have been conducted in the fields of pervasive computing and sensor networks (Ganeriwal et al. 2008, Huang et al. 2010, Mashhadi and Capra 2011). Detecting anomalous data entries is the focus of multiple studies due to the broad application of anomaly detection methods in practice (Golle et al. 2004, Talasila et al. 2010, Barnwal et al. 2016).

An anomaly or outlier¹ is ‘an observation that deviates so much from other observations as to arouse suspicion that it was generated by a different mechanism’ (Hawkins 1980, p. 1). From a probability distribution perspective, outlier detection methods are directed to finding data points that deviate from a standard distribution. An example of using a distribution-based model to detect outliers for participatory sensing data is presented by Barnwal et al. (2016). The authors used a Bayesian probabilistic model to detect false traffic reports in Waze, a GPS navigation app, and automatically assign a confidence score to the crowdsourced reports. In addition to the distribution-based methods, other approaches include a spatial proximity approach – validating a submitted report using nearby reports (Talasila et al. 2010), and consistency checking – validating the spatial or logical consistency of data to flag malfunctioning devices (Golle et al. 2004).

However, these methods are designed for discrete observations to detect malicious nodes. Many mobile environmental sensing tasks have continuous measurements along traveling paths that are sampled at a high temporal resolution. These continuous observations capture a gradual change of an environmental phenomenon, such as the concentration of air pollutants or radioactive materials. Meanwhile, continuous measurements pose new challenges in detecting anomalies since the data are oversampled along the paths while also having large gaps from one path to another. This study presents an anomaly detection method for participatory sensing data, with Safecast data as a case study. Inspired by the crowd-sourcing approach and geographic approach proposed by Goodchild and Li (2012), the key idea of the anomaly detection method is that data measured in close spatial and temporal proximity are similar, therefore can be used to cross-validate each other and identify errors in the measurements at a fine spatiotemporal granularity.

The rest of the paper is organized as follows: Section 2 explains the case study data set and validation data sets. Section 3 describes the methodology developed in the study. Section 4 presents the validation process and the results. Section 5 concludes the paper and summarizes guidelines for implementing the proposed anomaly detection method.

2. Data

2.1. Case study data: Safecast data

The citizen science project Safecast was launched shortly after the 2011 Fukushima Nuclear Accident with the goal to encourage citizens to collect radiation measurements and publicly share the data online using a mobile radiation detector called bGeigie (Brown et al. 2016). The project attracted hundreds of volunteers across Japan and eventually evolved into a global campaign. By August 2020, the radiation measurements collected through the Safecast project have surpassed 150 million data points.² In this study, data from 2011 to 2016 were used to make it comparable with the reference data set, a car-borne survey data collected by the Kyoto University RAdiation MApping system (KURAMA). The number of Safecast volunteers and total radiation measurements per year are shown in Figure 1.

Figure 1. Comparison between the entire Safecast global data set versus data collected in Japan from 2011 to 2016: (a) the number of Safecast volunteers, (b) the number of Safecast measurements

Although the number of volunteers in countries other than Japan has grown consistently from 2011 to 2016, as the birthplace of Safecast, Japan still has the most substantial Safecast radiation data set (Figure 1(b)). In addition to the Safecast project, many government agencies, professional organizations, and individuals launched radiation surveys in Japan after the nuclear accident. These independent efforts made it possible to validate the anomaly detection method proposed in this study. Given those factors, the case study used Safecast measurements collected in Japan³ from 23 April 2011 to 8 November 2016 to evaluate the anomaly detection method, with more than 57 million radiation measurements in total.

Currently, the most used Safecast radiation detector unit is the bGeigie Nano kit, developed in late 2012 (Safecast 2013). The device is a geo-tagged mobile sensor for ionizing radiation with internet data sharing capacity. The bGeigie Nano measures alpha, beta, and gamma radiation. According to the Safecast organization’s online report, the device has been ‘put through stringent calibration test and has a quality control standard as minimum ± 15% accuracy for gamma radiation from the Barium-137m daughter of Caesium-137, which is the primary source of gamma radiation from the Fukushima-Daiichi accident’ (Safecast 2015). Given the stable performance of the bGeigie Nano kit (Walsh et al. 2019), it is believed that ‘human error is usually a greater source of measurement inaccuracy than minor variances in hardware’ (Safecast 2015).

The bGeigie Nano is designed for mounting on the outside of a car window but can also be used on other modes of transportation or for static/spot radiation detection. In logging mode, the detector takes radiation measurements every 5 s. In addition, it also records the GPS coordinates and timestamps at the time of measure. Users can submit their mobile radiation measurements to the Safecast online mapping system⁴ to share the data publicly.

The majority of Safecast volunteers use bGeigie Nano as a mobile radiation detector rather than a fixed radiation monitor and carry their bGeigie Nanos with them while traveling. These sequential radiation records uploaded by each volunteer therefore also documented the travel trajectories of volunteers which provide valuable information to understand their measurement activities. Utilizing the trajectory information, the study proposes an anomaly detection method to improve the accuracy of the radiation measurements.

Since the anomaly detection method is designed for mobile sensing applications, static sensors are therefore removed from the Safecast data set during the analysis. In this study, static sensors are defined as devices that moved less than 100 m in the course of use. One hundred devices were recognized as static sensors and removed from the analysis, which corresponds to 1.46% of total devices. The analysis of travel speed shows that 80.5% of Safecast measurements in the data set were collected while driving. Hence, an official car-borne radiological survey data set collected through the KURAMA system was used for validation to reduce the discrepancy between different measuring systems.

2.2. Validation data 1: KURAMA car-borne survey data

The official car-borne radiological survey data set published by the Japan Atomic Energy Agency (JAEA) was a result of a series of surveys on radioactive substances conducted by the Ministry of Education, Culture, Sports, Science and Technology (MEXT), and the Secretariat of the Nuclear Regulation Authority. The survey used the KURAMA gamma-ray survey system with a GPS tracking function (Tanigaki 2014). Early versions of the KURAMA system use calibrated NaI scintillation survey meters or ionization chamber-type survey meters for air radiation dose rate monitoring. Since 2012 the upgraded KURAMA-II system uses CsI scintillation detectors instead.

The survey was conducted periodically from 2011 to 2016, 13 times in total. Since values measured in winter tend to be smaller than the actual values due to the shielding effect of snow (Nursal et al. 2016, JAEA 2019), six surveys conducted in summer were used as the primary validation data set to reduce errors caused by the survey process. One survey conducted in winter, December 2011, was also included to test the effect of snow. In total, seven surveys were used in the study for validation. The original survey IDs, survey dates, and the number of records for each survey are listed in Table 1. The radiation level is measured in microsieverts per hour ($\mu \mathrm{S}\mathrm{v}/\mathrm{h}$). The conversion between $\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ and counts per minute (cpm) used by the Safecast data follows the equation $1\mathrm{u}\mathrm{S}\mathrm{v}/\mathrm{h}=334\mathrm{c}\mathrm{p}\mathrm{m}$ (Walsh et al. 2019).

Table 1. KURAMA survey data used in the validation

Survey ID	Survey Dates	Number of Records
1^st survey	6 June 2011–13 June 2011	61,736
2^nd survey	5 December 2011–28 December 2011	177,768
4^th survey	20 August 2012–12 October 2012	695,446
6^th survey	12 June 2013–8 August 2013	558,467
8^th survey	23 June 2014–8 August 2014	497,946
10^th survey	29 June 2015–4 August 2015	361,047
12^th survey	27 June 2016–5 August 2016	288,480

The published values of the measurement points have been averaged into 100 m by 100 m grids by MEXT to reduce statistical variation (JAEA 2019). Each record represents the average air dose rate measured in a grid during the survey period. Decay correction is not considered in this data set since the decrease of air dose rate caused by the decay of Caesium during the survey period is only about 1–2%, which is smaller than the errors of the measuring instruments (JAEA 2019).

2.3. Validation data 2: rejected measurements by the Safecast moderation process

Besides the official car-borne radiation survey data collected through the KURAMA system, a data set containing anomalous bGeigie measurements filtered by Safecast moderators was also used to validate the anomaly detection method. These measurements were rejected by a rule-based automatic checking process and a manual verification process by Safecast moderators.⁵ The auto-approval process currently has five rules to check the validity of the measurements⁶: (1) no zero radiation values, (2) radiation value is less than or equal to 120 cpm for non-cosmic radiation measurements, (3) GPS validity reaches a minimum threshold, (4) the bGeigie device makes frequent contributions and has more than 10 approved measurement imports in the past, and (5) the bGeigie device has no rejected measurements in the past year. Failure to pass one of the rules will flag the measurement import as an anomaly and put the data into the manual verification process waiting for approvals of moderators.

3. Methodology

The methodology developed in this study consists of three modules: a data preprocessing module, an anomaly detection module, and a validation module. The data preprocessing module includes a spatiotemporal clustering step and a trip segmentation step. The anomaly detection module uses two methods to identify and filter anomalies in Safecast data, a Cross-Volunteer Referencing Anomaly Detection (CVRAD) method developed in this study and a benchmark method 1.5 Interquartile Range (1.5 IQR) (Tukey 1977). Lastly, in the validation module, the filtered Safecast data are compared with the KURAMA car-borne official survey data and the filtered anomalous measurements by Safecast moderators to evaluate the performance of the CVRAD method. A workflow of the methodology is presented in Figure 2.

Figure 2. A flowchart of the methodology

3.1. Data preprocessing

3.1.1. Spatiotemporal clustering

To leverage the crowd-sourcing approach, we first need to cluster data with similar measurements so that the measurements within one cluster could cross-validate each other. In this section, Safecast data measured in close spatial and temporal proximity were clustered into space-time cubes. To find proper spatial and temporal aggregation units that minimize the variability of Safecast data within each space-time cube, a space-time variogram was used to calculate the variations in the Safecast measurements at various spatial distances and temporal intervals (Pebesma 2004, Cressie and Wikle 2015). Fifteen spatial bins with a bin width of 20 m and 13 temporal bins with a bin width of one week were used in generating the variogram, which calculates the pair-wise semivariance at a high spatiotemporal resolution. Since the computational time of the space-time variogram grows exponentially as the data size increases, two strategies were adopted to reduce the overall computational time: (1) Only the 2011 Safecast data within the region of the first KURAMA survey were used in the calculation, which approximates 1.03 million Safecast measurements. The 2011 Safecast data set was used because it has the highest level of variation. Therefore, an optimal clustering unit calculated using the 2011 data applies to data collected in later years, where the variation gradually decreases over time due to the decay effect of radioactive materials. (2) Parallel computing techniques were employed to reduce the computational time of generating the space-time variogram. The Safecast data were separated into batches of 1 km $\times$ 1 km grids for parallel computing. The 1 km unit is much larger than the target maximum spatial distance of 300 m ($20\mathrm{m}/\mathrm{b}\mathrm{i}\mathrm{n}\times 15\mathrm{b}\mathrm{i}\mathrm{n}\mathrm{s}$), therefore the error margin is negligible when splitting the data into 1 km $\times$ 1 km grids. The calculated grid-level variograms were further aggregated to estimate the global variogram (Figure 3).

Figure 3. Space-time variogram map for 2011 Safecast radiation measurements (cpm). Each interval on the x-axis represents 20 m and each interval on the y-axis represents one week

Each cell in the space-time variogram represents the semivariance of data measured in a spatially and temporally lagged window. The dark blue colour indicates a low variation and the bright yellow colour indicates a high variation. The variogram shows that within a 120 m distance and 3 weeks, the variability of the data is relatively small and stable. As the spatial distance or the time lag increases, the semivariance does not constantly increase as what would be expected from a typical variogram. This is because the variability in radiation values is not only influenced by the distance apart but also the topology. The radioactive materials can be blocked by mountain ranges during the dispersion and transmission process, causing the accumulation of radioactive materials on one side of a mountain range (Yasunari et al. 2011). Besides, the Safecast measurements are not evenly sampled in space and time since they are collected in an uncoordinated manner by volunteers, which further increases the variability in the pair-wise semivariance. For example, abnormally high values are observed at four-week and seven-week intervals which are caused by periodic measurements by certain Safecast users near the Fukushima region with elevated radiation levels. Since the semivariance is relatively small within 120 m distance and the KURAMA validation data set has a spatial resolution of 100 m $\times$ 100 m, the Safecast measurements were aggregated into the same 100 m $\times$ 100 m grids for comparison (Figure 4).

Figure 4. Spatial clustering of Safecast measurements (black points) by 100 m $\times$ 100 m grids used in the KURAMA car-borne survey data

After the spatial clustering, data within each spatial grid were segmented further into temporal intervals using the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) method (Ester et al. 1996). The DBSCAN algorithm is used in this study to account for the different sampling densities of Safecast data over time. The algorithm searches for the closest neighbouring measurements for each data point. If the time difference between two data points is less than a temporal window, denoted as epsilon, these two data points form a group. The algorithm runs iteratively for each Safecast measurement and separates all data points into groups. A group of points is considered a cluster if the number of points is above a threshold value. In the case study, the threshold was set as 10 to ensure sufficient points for the anomaly detection process. The optimal value of the temporal window epsilon was selected using the Davies-Bouldin index (Davies and Bouldin 1979).

Since the semivariance of Safecast measurements within 3-week interval is relatively small, we set the upper temporal interval as 3 weeks which approximates an epsilon of 10 days. To examine the performance of the DBSCAN method with different epsilons, values ranging from 3 to 10 were tested using the Davies-Bouldin Index. This index quantifies the average similarity between each cluster $C_i,i=1,\dots ,k$ and its most similar cluster $C_j$. The similarity measure $R_{ij}$ between two clusters $C_i$ and $C_j$ are calculated using the equation $R_{ij}=\frac{s_i+s_j}{d_{ij}}$, where $s_i$ is the average distance between each point in cluster $C_i$ and the centroid of that cluster, $d_{ij}$ is the distance between centroids of clusters $C_i$ and $C_j$. In this study, all clusters are aligned chronologically, thus adjacent clusters are considered most similar to each other. The Davies-Bouldin index is defined as $DBI=\frac{1}{k}\sum _{i=1}^k\underset{i\ne j}{max}{R}_{ij}$. A lower value of DBI indicates a lower similarity between clusters, thus a better cluster partition. The result of the Davies-Bouldin measure shows that an epsilon value of 7 days produced the best partition of DBSCAN clusters (Figure 5), which is chosen as the epsilon of the DBSCAN method for the following analysis. After running the spatiotemporal clustering, similar Safecast data were grouped into space-time cubes. A statistic of the number of volunteers per space-time cube shows that many space-time cubes have a few volunteers, and as the number of volunteers increases, the number of space-time cubes drops quickly (Figure 6). This pattern is consistent with the long-tail distribution observed in other VGI projects (Ma et al. 2015).

Figure 5. Result of the Davies-Bouldin Index for DBSCAN clustering method with different epsilons

Figure 6. The number of space-time cubes with different numbers of volunteers

3.1.2. Trip detection of Safecast volunteers

For each volunteer, the stream of data collection is usually not continuous. Gaps exist in the data collection due to the volunteer’s inactive participation. During this inactive period, the accuracy of the same device could change due to changes in device settings, unexpected malfunction of the unit, incorrect operations, or malicious manipulations. Given the potential discrepancy in device accuracy, it is important to detect the start and the end points of each measurement activity. Since most of the data that have been collected are during times of travel, in this paper, measurement activities are separated by trips.

A trip was marked by start and end stops. Since the radiation measurements were sampled at the rate of every 5 s, a start or end stop will have a noticeably longer temporal gap than 5 s to its adjacent records. Due to occasional loss of GPS signals, the gaps between consecutive measurements can be longer than 5 s sometimes, but the majority of the temporal gaps fall in the range of 5 s to 1 min in the data set. A two-minute gap is considered an apparent gap between continuous measurements and was therefore chosen in the case study to separate trips of a volunteer, i.e. measurement activities. This two-minute threshold separates trips at very high granularity. For other similar tracking-based environmental sensing data, the threshold should be adjusted to fit the temporal sampling rate of the input data.

3.2. Anomaly detection method

3.2.1. CVRAD method

Many researchers have argued that with a large volume of observations contributed by multiple independent sources, VGI could potentially provide reliable data quality (Haklay 2010, Elwood et al. 2012, Goodchild and Li 2012). One assumption behind the argument is that errors could be corrected through cross-validation between data contributors with enough independent observers or contributors. Based on this assumption, an anomaly detection method called CVRAD is proposed to filter outliers for mobile sensing VGI projects.

The CVRAD method first finds the local value range of each measurement activity and the global value range of all measurements in a space-time cube by extracting the lowest and highest values respectively. For instance, the global range is $0.1-0.5\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ for data shown in Figure 7(a). All local highest and lowest values are used to cut the global range into small segments. Those dividing lines are indicated by horizontal white dash lines in Figure 7(a). Within each segment of values, the number of independent volunteers is considered a reliability score of the data measured in that segment. If a segment has a reliability score that is lower than a threshold (denoted by $\gamma$), all measurements within the segment will be considered as outliers. $\gamma =3$ means that less than three independent volunteers will be considered as outliers. Figure 7(a) presents an example of $\gamma =3$, where segments with a value above 0.43 $\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ or below 0.19 $\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ are contributed by less than three volunteers and marked as outliers. In addition, the reliability score of the non-outlier segments is represented by a colour ramp from red to orange, with the red colour indicating a high reliability and the orange colour indicating a low reliability.

Figure 7. (a) Illustration of the CVRAD method, (b) illustration of the benchmark 1.5 IQR anomaly detection method

The threshold $\gamma$ value has an important influence on the performance of the CVRAD method. A higher value of $\gamma$ will remove more data points than a lower value of $\gamma$, therefore achieving a higher level of accuracy. However, removing more data points will inevitably cause greater information loss. Meanwhile, to avoid labelling outliers when there are few volunteers in the space-time cube, a space-time cube has to have at least $\gamma$ number of volunteers to be considered in the outlier filtering process. For space-time cubes that do not satisfy this requirement, data points in those space-time cubes are assigned an ‘unknown’ label and will not enter the anomaly detection process. If none of the measurements in a space-time cube have a sufficient reliability score, it means these measurements do not have enough overlaps to generate a trustworthy range. As a result, these measurements will not be considered outliers but assigned an ‘unknown’ label instead. To understand the influence of the threshold on the filtering result, $\gamma$ values ranging from 2 to 7 were tested, and the result is shown in the Validation and Results section.

3.2.2. The 1.5 IQR method

The benchmark method used to validate the proposed method is 1.5 IQR, which filters outliers based on the distribution of the data in each space-time cube. Outliers are defined as radiation values that lie outside of a range that is 1.5 times the difference between the first quartile and the third quartile of the measurements. The 1.5 IQR is calculated using the formula $1.5IQR=1.5\times \left(Q_3-Q_1\right)$, where ${\mathrm{Q}}_3$ is the third quartile of the data set and ${\mathrm{Q}}_1$ is the first quartile of the data set. A data point is considered as an outlier by this method if its value falls outside of the range $\left[{\mathrm{Q}}_1-1.5\mathrm{I}\mathrm{Q}\mathrm{R},\hspace{0.17em}{\mathrm{Q}}_3+1.5\mathrm{I}\mathrm{Q}\mathrm{R}\right]$. Figure 7(b) shows an example of the 1.5 IQR outlier filtering results.

Compared to the 1.5 IQR approach, the CVRAD method weights more on the concurrency of different data contributors. For example, in Figure 7 data points that fall into the value range 0.38–0.43 $\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ were filtered as outliers according to the 1.5 IQR criterion. However, this segment of values is not considered outliers by the CVRAD method, since four out of the total five volunteers contributed observations to this value range, which makes 0.38–0.43 $\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ a valid range under the threshold of $\gamma =3$.

4. Validation and results

The study evaluated the effectiveness of the CVRAD method through two validations. The first validation tests whether the Safecast measurements are closer to the official KURAMA radiation survey data after being corrected by the CVRAD method compared to the 1.5 IQR method. The second validation tests if the CVRAD method can successfully detect abnormal values using measurements rejected by Safecast moderators.

4.1. Validation by the KURAMA radiation survey data

In the first validation, the CVRAD method and the 1.5 IQR method were compared on their performance to filter Safecast anomalies. The Safecast data set was joined with the KURAMA survey data based on the same spatial grids and temporal extent. Spatial grids with anomalies identified by both methods were selected and the number of spatial grids is shown in Table 2. The difference between the Safecast measurements and the official KURAMA survey data is quantified by the mean absolute error (MAE). A decrease in MAE indicates an increase in the accuracy of Safecast data. The MAE of all grids is used to compare the accuracy of the Safecast data corrected by the CVRAD method, the Safecast data corrected by the 1.5 IQR method, and the uncorrected Safecast data (Figure 8). The CVRAD method was tested with six $\gamma$ values, $\gamma =2,3,4,5,6,\mathrm{o}\mathrm{r}7$.

Figure 8. The mean absolute error between the Safecast data and the official KURAMA survey data. The figure compares the mean absolute error between the Safecast data corrected by the CVRAD method (orange line), Safecast data corrected by the 1.5 IQR method (blue line), and the original non-corrected Safecast data (black line)

Table 2. Number of grids corrected by both CVRAD and 1.5 IQR methods

$\gamma$ Value	Number of Spatial Grids	Average Air Dose Rate by KURAMA Car-Borne Survey Data ($\mu \mathbf{S}\mathbf{v}/\mathbf{h}$)
2	978	0.408
3	351	0.151
4	133	0.138
5	67	0.137
6	39	0.138
7	25	0.142

The comparison shows that when the gamma value is less than three, the MAE from the CVRAD method is larger than the 1.5 IQR method and uncorrected Safecast data (Figure 8). In contrast, when the gamma value is greater than or equal to three, the CVRAD starts to show its advantage and yields the smallest MAE compared to the 1.5 IQR method and the uncorrected Safecast. However, the decrease in MAE (i.e. increase in accuracy) resulting from the CVRAD method is marginal. The same pattern is also observed for the standard deviation of error where the CVRAD method slightly outperforms the 1.5 IQR method when the gamma value is greater than or equal to three. The marginal improvement by the CVRAD method when the gamma value is greater than or equal to three is partially because a larger gamma value requires a spatial grid to have more volunteers to enter the filtering process. Grids that satisfy this requirement are mostly in populated places and have low radiation values. For example, when the gamma value is three, the mean air dose rate of the 351 spatial grids is 0.151 $\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ (Table 2). Thus, when the small radiation values are corrected by the anomaly detection methods, the decrease in errors becomes even smaller. Besides, the Safecast data have been pre-filtered by Safecast moderators which rejected the most impactful anomalies that would introduce large errors.

When the gamma value equals two, many grids with relatively high radiation values enter the anomaly detection process and increase the variability of radiation values. This resulted in an MAE value of 0.124 $\mu \mathrm{S}\mathrm{v}/\mathrm{h}$ for uncorrected Safecast data at $\gamma =2$ which is more than twice the value of MAE for $\gamma \ge 3$. The relationship between gamma values and the mean air dose rate of the grids can be seen in Table 2.

4.2. Validation by the rejected Safecast measurements

In the second validation, the rejected Safecast measurements are used to test if the CVRAD method can detect unusually high radiation values on the raw measurements. In this validation, the preprocessing step of the workflow is modified to an online approach to allow the detection of anomalies in real-time. After receiving a new data import, the pipeline will identify the GPS point with the highest radiation value and use this point as the centre to retrieve historical data within a spatiotemporal radius. These retrieved data are used as the reference data to check if any anomalies exist in the data import.

Since the CVRAD method is not designed to check the GPS validity and relies on the GPS location to retrieve neighbouring measurements, only rejected measurements with valid GPS data accuracy and abnormally high radiation values were used, resulting in a total of 60 rejected measurement imports. For each rejected measurement import, neighbouring accepted Safecast data within a search radius of 7 days and 100 m are retrieved using the Safecast API. If no neighbouring data are found within this range, the spatial radius is expanded to 500 m to increase the likelihood of getting neighbouring reference data. To implement the CVRAD method, the retrieved data must come from at least one other volunteer apart from the one that contributed the rejected measurements.

The data retrieval process showed that only two rejected measurement imports had valid neighbouring data returned from the Safecast API. These two rejected measurement imports are combined with their respective neighbouring data as two data groups to validate the effectiveness of both CVRAD and 1.5 IQR methods. The result shows that the CVRAD method successfully identified the rejected measurements in both data groups, whereas the 1.5 IQR method failed to flag any anomalies (Figures 9 and 10). This difference in results can be explained by the mechanism of the two anomaly detection methods. The 1.5 IQR method filters outliers based on the distribution of data. If many measurements are anomalies, the median value will lean towards these anomalous values, thus the 1.5 IQR method will fail to identify anomalies. In contrast, the CVRAD method is based on the majority vote of unique value ranges and is not influenced by the number of contributions from a particular volunteer. Compared to a constant threshold (120 cpm) currently used by Safecast moderators to filter high radiation values, the CVRAD method can adjust the threshold automatically based on the data.

Figure 9. A comparison between the CVRAD method (γ = 2) and the 1.5 IQR method on detecting the first rejected measurement import. The rejected measurement import is contributed by volunteer B and marked in solid triangles. Users are anonymized and represented as letters to protect Safecast user’s privacy

Figure 10. A comparison between the CVRAD method (γ = 2) and the 1.5 IQR method on detecting the second rejected measurement import. The rejected measurement import is contributed by volunteer C and represented in solid dots. Users are anonymized and represented as letters to protect Safecast user’s privacy

5. Conclusion

The study presents an anomaly detection method, CVRAD, to improve the overall accuracy of participatory environmental sensing data. The CVRAD method was implemented on a crowdsourced radiation data set collected through the Safecast project. Successful validation of the method was performed using both official KURAMA car-borne survey data and filtered measurements by Safecast moderators. The validation results demonstrate that the anomaly detection method can successfully identify abnormal values and reduce errors in the VGI data when sufficient volunteers are present; thereby increasing the overall accuracy of the data.

The proposed anomaly detection method is an intrinsic evaluation method that is independent of an external reference data set. Although reference data sets are used to validate the feasibility of the anomaly detection method, the implementation of the anomaly detection method itself does not require an external data set. The proposed method enables the validation of VGI data sets when an authoritative counterpart data set is not available or inconsistent for comparison.

The application of the proposed anomaly detection method can be extended to a broader range of participatory environmental sensing projects, such as air pollution monitoring, temperature sensing, and noise monitoring. The method best fits data with two characteristics: (1) the data are a continuous numerical variable and (2) the data have independent concurrent observations. For environmental measurements with continuous GPS tracking data, the concurrency of measuring activities of independent volunteers can be derived using GPS records coupled with the spatiotemporal clustering and trip segmentation process presented in this paper.

Researchers who plan to use the CVRAD method need to consider two factors: (1) threshold value, $\gamma$, used to define the minimum number of volunteers required for non-outliers, and (2) spatiotemporal unit used to cluster data.

First, the $\gamma$ parameter of the cross-volunteer referencing method needs to be adjusted according to the specific requirements of a study. A high $\gamma$ value indicates a low tolerance for outliers and will filter out more outliers than a low $\gamma$ value. In addition, the $\gamma$ value also controls the number of data groups to be examined by the anomaly detection method (in this study, the data are grouped by space-time cubes). A low $\gamma$ value will include more data groups than a high $\gamma$ value. In this case study, the $\gamma$ value needs to be greater than or equal to three for the CVRAD method to outperform the distribution-based 1.5 IQR method at an aggregated level. As the $\gamma$ value increases, the reliability of corrected data also increases. However, a high $\gamma$ value requires VGI projects to have a large group of participants and concurrent measurements at the same time and location. A future research direction to improve the CVRAD method is to develop an automatic selection of the $\gamma$ parameter based on the characteristics of the input data.

Second, the spatial and temporal units used for aggregating the data must account for the local heterogeneity of the crowdsourced environmental data. In cases where limited domain knowledge is available, a space-time variogram can be used to test the variability of the data. Since environmental sensing data are often directly associated with the land use type, future research could cluster mobile environmental sensing data based on the underlying spatial context, for example, clustering sensing data by road segments.

Participatory environmental sensing projects often collect valuable data at a significantly higher granularity and frequency than traditional surveying methods. However, many challenges still exist in ensuring the quality of crowdsourced environmental data which calls for innovative methods to automatically detect anomalous data entries and improve data accuracy. This paper contributes to this direction by developing a CVRAD method that successfully detected anomalies in crowdsourced radiation measurements. This study aims to draw researchers’ attention to improving the data quality of participatory environmental sensing data.

Data and codes availability statement

The Safecast data set used in the case study is freely downloadable from the Safecast official website: https://blog.safecast.org/data/. The KURAMA care-borne survey data set used for validation is openly available on the JAEA official website: https://emdb.jaea.go.jp/emdb/en/portals/b1010202/. The rejected Safecast measurements used in validation can be retrieved from the Safecast API: https://api.safecast.org/en-US/bgeigie_imports?utf8=%E2%9C%93&by_user_name=&by_user_id=&uploaded_before=&uploaded_after=&q=&status=rejected&rejected_by=&by_status=&commit=Filter. The code developed to support the findings of the paper is available in figshare.com with the identifier: https://doi.org/10.6084/m9.figshare.11660046.v1.

Acknowledgements

I would like to thank the editors, anonymous reviewers, Dr Alan MacEachren, and Dr Tony Smith for their valuable and constructive comments. I would also like to thank all staff and contributors of the Safecast project that made the data set publicly available.

Additional information

Notes on contributors

Yanan Xin

Yanan Xin is a postdoctoral researcher at the Mobility Information Engineering Lab, Institute of Cartography and Geoinformation, ETH Zurich. Her research interests are computational mobility analysis, mobility-based anomaly detection, interpretable machine learning, and spatial statistics.

Notes

1. Anomaly and outlier are used interchangeably in this paper.

2. 150, 000, 000 Data Points, the Safecast Blog. URL: https://safecast.org/2020/09/150000000-data-points/

3. The geographic coordinates of the bounding box used for Japan: the minimum longitude is 126.45073, the maximum longitude is 145.58229, the minimum latitude is 26.18840, and the maximum latitude is 45.52232.

4. The Safecast API. URL: https://api.safecast.org/.

5. Notes about the moderation process, Safecast. https://safecast.org/faq/notes-about-the-moderation-process/

6. Code of the Safecast automatic rule-based anomaly checking. URL: https://github.com/Safecast/safecastapi/blob/4f6d4d1c2ff0346319cba24927b2d47d26be7047/app/models/bgeigie_import.rb#L410-L421