VGI and crowdsourced data credibility analysis using spam email detection techniques

ABSTRACT

Volunteered geographic information (VGI) can be considered a subset of crowdsourced data (CSD) and its popularity has recently increased in a number of application areas. Disaster management is one of its key application areas in which the benefits of VGI and CSD are potentially very high. However, quality issues such as credibility, reliability and relevance are limiting many of the advantages of utilising CSD. Credibility issues arise as CSD come from a variety of heterogeneous sources including both professionals and untrained citizens. VGI and CSD are also highly unstructured and the quality and metadata are often undocumented. In the 2011 Australian floods, the general public and disaster management administrators used the Ushahidi Crowd-mapping platform to extensively communicate flood-related information including hazards, evacuations, emergency services, road closures and property damage. This study assessed the credibility of the Australian Broadcasting Corporation’s Ushahidi CrowdMap dataset using a Naïve Bayesian network approach based on models commonly used in spam email detection systems. The results of the study reveal that the spam email detection approach is potentially useful for CSD credibility detection with an accuracy of over 90% using a forced classification methodology.

KEYWORDS:

• VGI
• crowdsourced data
• credibility
• Bayesian networks
• spam emails

1. Introduction

Volunteered geographic information (VGI) (Goodchild 2007), with its geographic context, is considered a subset of crowdsourced data (CSD) (Howe 2006; Goodchild and Glennon 2010; Heipke 2010; Koswatte, McDougall, and Liu 2016). In recent times, there has been an increased interest in the use of CSD for both research and commercial applications. VGI production and use have also become simpler than ever before with technological developments in mobile communication, positioning technologies, smart phone applications and other infrastructure developments which support easy to use mobile applications. However, data quality issues such as credibility, relevance, reliability, data structures, incomplete location information, missing metadata and validity continue to limit its usage and potential benefits (Flanagin and Metzger 2008; De Longueville, Ostlander, and Keskitalo 2010; Koswatte, McDougall, and Liu 2016). Therefore, researchers are now seeking new approaches for improving and managing the quality of VGI and CSD in order to increase the utilisation of this data.

VGI quality can be described in terms of quality measures and quality indicators (Antoniou and Skopeliti 2015). The quality measures of spatial data have largely focused on quantitative measures such as completeness, logical consistency, positional accuracy, temporal accuracy and thematic accuracy whilst the quality indicators are often more difficult to measure and refer to areas such as purpose, usage, trustworthiness, content quality, credibility and relevance (Senaratne et al. 2016). However, in CSD it may not always be appropriate to trust the information provided by the volunteers as their experience and expertise varies dramatically and assessing the credibility of the provider may be impractical. In particular, the volunteers in a disaster situation are often extremely heterogeneous and their input only occurs during a short period. Hence, it is difficult to profile these contributors, unlike many users of Twitter which may have a long history of activity. Therefore, a key challenge is to assess the credibility of the provided data in order to utilise it for future decision-making.

A popular approach to assess credibility in spam email detection is to numerically estimate the ‘degree on belief’ (Robinson 2003) by analysing the email content using natural language processing and machine learning techniques. Natural language processing is a commonly used term to describe the use of computing techniques to analyse and understand natural language and speech. These approaches have been successfully applied to the detection of spam in Twitter messages (Wang 2010). The objective of this research is to investigate and test the use of spam email detection processes for credibility detection of crowdsourced disaster data.

The data for this research were collected through the Uhsahidi CrowdMap platform (https://www.ushahidi.com) which has been successfully used in a range of disasters including the 2011 Australian floods, the Christchurch earthquake and the 2011 tsunami in Japan. The Ushahidi platform was initially developed to easily capture crowd input via cell phones or emails (Bahree 2008; Longueville et al. 2010) and was utilised to report the election violence in Kenya. Over time, its popularity has increased and the platform has been successfully deployed in a number of disasters around the world.

This paper discusses the use of a Naïve Bayesian network (BN)-based model to detect the credibility of CSD using a similar approach to spam email detection. The paper is structured as follows: Section 2 discusses the background of CSD credibility detection and the use of Naïve BNs for spam email detection. Section 3 explores the methods used in the study. Section 4 details the results of the study and discusses their implications. Finally, Section 5 provides some concluding remarks and some future suggestions for research.

2. CSD credibility

Hovland, Janis, and Kelley (1953) defined credibility as ‘the believability of a source or message’ which comprises primarily of two dimensions, trustworthiness and expertise. However, as identified by Flanagin and Metzger (2008), the dimensions of trust and expertise can also be considered as being subjectively perceived, as the study of credibility is highly interdisciplinary and the definition of credibility varies according to the field of study. Whilst the scientific community views credibility as an objective property of information quality, the communication and social psychology researchers treat credibility more as a perceptual variable (Fogg and Tseng 1999; Flanagin and Metzger 2008). According to Fogg and Tseng (1999), credibility is defined as ‘a perceived quality made up of multiple dimensions such as trustworthiness and expertise’ or simply as believability.

Credibility analysis approaches and the methods will vary depending on the context. Studies conducted by Bishr and Kuhn (2007), Noy, Griffith, and Musen (2008), Janowicz et al. (2010), Sadeghi-Niaraki et al. (2010), and Shvaiko and Euzenat (2013) have identified the importance and usefulness of spatial semantics and ontologies in assessing the quality of CSD. Most approaches tackle CSD quality by qualifying contributors and contributions (Brando and Bucher 2010). Various authors have investigated the classification of users based on their purpose (Coleman, Georgiadou, and Labonte 2009), their geographic location (Goodchild 2009) and trust as a reputational model (Bishr and Kuhn 2007). Quality based on contributions has mostly been validated using rating systems (Brando and Bucher 2010; Elwood 2008) or using a reference dataset (Haklay 2010; Goodchild and Li 2012). Longueville et al. (2010) proposed an approach which consisted of a workflow that used prior information about the phenomenon. The key to their approach was to extract valid information from CSD using cross validation, cluster processing and ranking. A similar but extended approach for the automated assessment of the quality of CSD was proposed by Ostermann and Spinsanti (2011).

Given the variability of contributors of CSD during a disaster event, and the complexities in qualifying the expertise or experience of contributors, it was decided that a content analysis approach would provide the greatest likelihood of success for this research.

2.1. Statistical approaches for CSD credibility detection in disaster management

Disaster-related CSD is quite different in the sense of its lifetime and contributors. Data are often collected over a very short period of time with many different contributors during the event. Recent research conducted by Hung, Kalantari, and Rajabifard (2016) identified the possibility of using statistical methods to assess the credibility of VGI. They used the 2011 Australian flood VGI dataset as the training data and the 2013 Brisbane floods data as the testing dataset. Their approach was to use binary logistic regression modelling to achieve an overall accuracy of 90.5% for a training model and 80.4% accuracy for the testing dataset. They highlighted the potential of using statistical approaches for efficiently analysing the CSD credibility and for rapid decision-making in the disaster management sector even without real-time or near real-time information.

Kim (2013) developed a framework to assess the credibility of a VGI dataset from the 2010 Haiti earthquake based on a BN model. The outcomes of this earthquake damage assessment study were compared with the results from official sources. The author reported that ‘the experiments have not only demonstrated microscopic effects on the individual data, but also showed the macroscopic variations of the overall damage patterns by the credibility model’. Both of these models were identified as being more suitable for post-disaster management purposes.

In filter-based classification processes, it is important to simplify the message content using transformations including tokenising, stemming and lemmatising (Figure 1) which may improve the classification accuracy and performance (Guzella and Caminhas 2009). This research followed a similar approach by incorporating natural language processing techniques and enhancing a ‘bag of words’ model with tokenising (extracting words), stemming (removing derivational affixes), lemmatising (remove inflectional endings and returning the base or dictionary form of the word) and removing stop-words (common words in English).

Figure 1. Main steps involved in filter-based email classification (Guzella and Caminhas 2009).

Credibility can be calculated and rated into different levels which may be useful for disaster management staff. However, in critical events such as disaster management, a binary form of credibility representation would be simpler and less confusing for the general public (Ostermann and Spinsanti 2011). This research has adopted a similar binary approach by classifying the credibility using a ‘credible/unreliable’ labelling. The term ‘unreliable’ is used to describe those messages or reports that were not classified as ‘credible’.

2.2. Why use spam email detection as an approach for CSD credibility detection?

Spam email is considered as ‘unsolicited bulk email’ in its shortest definition (Blanzieri and Bryl 2008). Spam emails cost industries billions of dollars annually through the misuse of computing resources and the additional time required by users to sort emails. Spam emails can often carry computer viruses and also violate users’ privacy (Blanzieri and Bryl 2008). Compared to the spam emails, CSD has some similarities and differences. Firstly, CSD also has a mixture of content that varies in credibility and the CSD events often generate large volumes of data. Emails, including spam emails, often have a specified structure (sender, body text and header), however, CSD often lacks structure. Finally, the aim of the filtering data to identify legitimate or credible content is similar in both cases.

Spam email detection (Pantel and Lin 1998; Cranor and LaMacchia 1998; Metsis, Androutsopoulos, and Paliouras 2006; Robinson 2003; Lopes et al. 2011), junk-email detection (Sahami et al. 1998) or anti-spam filtering (Androutsopoulos et al. 2000; Schneider 2003) research has a long history which grew from the commercialisation of the internet in mid-1990s (Cranor and LaMacchia 1998). Researchers have explored various approaches with content-based filters or Bayesian filters being the most popular anti-spam systems (Lopes et al. 2011). Wang (2010) tested a Bayesian classifier for spam detection in Twitter and confirmed that Bayesian classifiers performed highly in terms of weighted recall and precision, and outperformed the decision tree, neural network, support vector machines and k-nearest neighbour’s classifications.

Castillo, Mendoza, and Poblete (2011) analysed the news worthiness of tweets using a supervised classifier whilst Kang, O’Donovan, and Höllerer (2012) analysed the ‘credible individual tweets or users’ based on three models (social model, content model and hybrid model) using Bayesian and other classifiers. These studies support the use of a modified Bayesian approach for assessing the credibility of crowd sourced data.

2.3. A naïve BN-based model for CSD credibility detection

The BNs were initially identified as powerful tools for knowledge representations and inference. With the advent of Naïve BNs, which are simple BNs that assume all attributes are independent, the classification power of BNs were expanded (Cheng and Greiner 1999). The credibility CSD detection engine proposed in this research was developed using a Naïve Bayesian-based spam detection model.

A credibility detection function can be defined as: where m is a message to be classified, θ is a vector of parameters, t_credible and t_unreliable are tags to be assigned based on the threshold T to the messages.

The vector of parameters θ is the result of training the classifier on a pre-collected dataset: where are previously collected messages, are the corresponding labels, and Θ is the training function.

As Guzella and Caminhas (2009) defined; if a given message is represented by which belongs to class , the probability that a message is classified as c and represented by can be written as:where is overall probability that any given message is classified as c, is the a priori probability of a random message represented by , and are the probabilities that a message is classified as spam or legitimate, respectively, and are overall probabilities that any given message is classified as spam or legitimate, respectively.

The naïve classifier assumes that all feature in are conditionally independent to every other feature and the probability can be defined considering N number of messages as:

So, Equation (1) becomeswith , given bywhere function f depends on the representation of the message. The probability is determined based on the occurrence of term t_i in the training dataset .

3. Methods

During the 2011 Australian floods, the Australian Broadcasting Corporation (ABC) (www.abc.net.au) developed a customised version of the Ushahidi CrowdMap to report/map disaster communications (Koswatte, McDougall, and Liu 2016). These data comprised primarily of text-based content that was submitted by volunteers during the flood event. The data included input from a heterogeneous range of volunteers who submitted reports during a relatively short period of time (approximately 7 days) via various channels including a mobile app, a website, SMS messages, emails, phone calls and Twitter.

3.1. CSD credibility detecting algorithm based on spam email detection approach

An algorithm for the CSD credibility detection based on the Naïve BN was developed for the analysis. The Java programming language was used for coding the system within the NetBeans Integrated Development Environment (https://netbeans.org/). The pseudo code of the algorithm consisted of two phases including training and testing, and is listed below.

Phase 1: Start training

Select Classifier and Training Dataset

   for each Message m_i in Training Dataset D_tr do

      for each Word in the Corpus do

      Calculate the Credible and Unreliable word Probabilities and store in Hash Table

      end for

   end for

End training

Phase 2: Start classification

Select Classifier, Testing Dataset and Hash Table

   for each Message m_i in the Training Dataset D_tr do

      for each Word in the Corpus do

      Calculate the Word Probability for being Credible and Unreliable

      Update Hash Table

      end for

      Calculate combined Probability for the Message

      if combined Probability > Threshold

      Label Message as Credible

      else

      Label Message as Unreliable

      end if

   end for

End classification

The probability threshold was determined after the initial testing and was set at the .9 probability level.

Figure 2 illustrates the key steps in CSD credibility detection approach based on the Naïve BN and the classical ‘bag of words’ model popular in email spam detection.

Figure 2. CSD credibility detection workflow.

The ABC’s 2011 Australian Flood Crisis Map dataset (Ushahidi CrowdMap) was used as the input CSD. The dataset was initially pre-processed using the steps explained in Figure 2. After the data pre-processing, the system was trained using a training sample dataset.

Within the ABC’s Ushahidi CrowdMap, there were approximately 700 reports during the period of 9–15th of January 2011 which often included information about the location where the report had originated. After the initial duplicates were removed, there were 663 unique Ushahidi CrowdMap reports remaining. The duplicates of the dataset were removed using the ‘Remove duplicates’ tool of the MS Excel software.

For training and testing purposes, approximately 20% of the total reports (143 reports) were randomly selected from this Ushahidi CrowdMap dataset. Eighty percent of these reports (110 reports) were then selected as training data and remaining 20% selected as the testing data (33 reports). The remainder of the full dataset (520 reports) was then used for the credibility detection analysis.

The whole dataset was initially pre-processed to prepare for the training, testing and credibility detection. The training dataset was classified through a manual decision process which identified messages that where either credible or unreliable based on the credibility of terms within the message. The classification was undertaken by a reviewer who had local and expert knowledge of the disaster area. The system was then trained and tested using the testing dataset under two different environments namely, unforced and forced conditions, to test the accuracy and performance improvements.

In the unforced training, the data processing of the test data followed the normal pre-processing steps and was then used directly for refining the training of the system with no human intervention undertaken. The results of this unforced training provided a report on the level of possible false positives (FPs) in the classification. A high level of FPs is indicative of a possible bias in the classification process and is often referred to as Bayesian poisoning (Graham-Cumming 2006). The purpose of the forced training was then to review the FPs and other classified data to improve the quality of the classification process and hence re-train the system. The forced training required human intervention to improve the training of the system and therefore some terms which had artificially increased the credibility of the messages were identified and removed. This enabled the training of the system to be further refined and to more effectively distinguish the credible or unreliable messages. The forced training process consisted of the following stages:

• The location terms were removed/disabled from both the credible and unreliable messages.

• Highly credible terms such as evacuation centre, road close, police, hospital, etc. were removed from messages that were identified as unreliable to give more weight to similar terms in the credible messages and to avoid Bayesian poisoning.

• Removing remaining messages which could cause a high FP rate and therefore avoid Bayesian poisoning.

When location terms appeared frequently in messages, these terms tended to increase the probability of the message being credible when in reality this was not the case. This impacted both the credible and unreliable messages. This impact was reduced by removing all the location terms in both credible and unreliable training sample messages. The Queensland Place Names Gazetteer was used as the basis for removing location terms as it provided a list of registered geographic locations and places. All incoming message terms were cross checked against the gazetteer list and discarded if found. Due to the large range and complexity of local or vernacular place names, these were not identified and would therefore be ignored by the gazetteer.

The full message structure from the Ushahidi reports included information on message number, incident title, incident date, location, description, category, latitude and longitude. For example:

101, Road closure due to flooding, 9/01/2011 20:00, Esk-kilcoy Rd, Fast running water over the road at the bottom of the decent below lookout, Roads Affected, -27.060215, 152.553593.

Some of the message descriptions were very brief in the Ushahidi CrowdMap data. The content of these messages were further reduced when some of the pre-processing activities were undertaken including the removal of numbers, units, time, dates, hashtags, Twitter user accounts and URLs. If the number of characters of these messages were less than 30 characters, the data columns ‘Incident Title’ and ‘Description’ were combined (see Table 1) to make the descriptions more comprehensive and meaningful.

Table 1. Example of the combination results of the Incident title and Description of the Ushahidi CrowdMap message fields.

Incident title	Description	Combined message
Road Closed-Manly Rd between new Cleveland Rd and Castlerea St, Manly	Road closed due to flooding	Road Closed-Manly Rd between new Cleveland Rd and Castlerea St, Manly road closed due to flooding

In some cases, this combination did not provide a meaningful result and did not satisfy the above condition. Therefore, the ‘Location’ column was also combined in these situations (see Table 2) to improve the message meaning. However, a small number of messages had to be discarded as they did not succeed in any of the above operations.

Table 2. Example of the combination result of the Incident title, Description and Location of the Ushahidi CrowdMap message fields.

Incident title	Description	Location	Combined message
Roads Affected	Not passable	Gailey Rd, St Lucia	Roads Affected Not passable Gailey Rd, St Lucia

The following example shows how the original Ushahidi CrowdMap message was processed after tokenisation, stemming, lemmatisation and stop-word removal before being used for training, testing and credibility detection.

Original Ushahidi Crowdmap message:

Access to Stanthorpe town is severely restricted and all residents along Quart Pot Creek have been ordered to evacuate.

Tokenised, stemmed and lemmatised message:

access to Stanthorpe town be severely restrict and all resident along Quart Pot Creek have be order to evacuate.

Stop word removed message:

access stanthorpe town severely restrict resident along quart pot creek order evacuate.

4. Results and discussion

4.1. Results of initial training and testing using different sized training data

The system was initially trained using two different sized training datasets to assess any variations in the outcomes based on the size of the training dataset. The first training dataset consisted of 35 messages of which there were 25 credible messages and 10 messages identified as unreliable. The second training set was a larger training sample and consisted of 77 messages with 53 credible messages and 24 messages identified as unreliable.

A dataset of 33 messages was then tested using both the smaller and larger training datasets to train the system under both forced and unforced conditions. The test dataset was also manually pre-classified to identify credible and unreliable messages in order to confirm the accuracy and performance during the testing. Table 3 shows example messages of correctly and misclassified results. Tables 4–7 show the classification results for the four test environments. Test 1 utilised the smaller training dataset (35 messages) with the 33 test messages under unforced training conditions. Test 2 utilised the smaller training dataset (35 messages) with the 33 test messages under forced training conditions. Test 3 utilised the larger training dataset (77 messages) with the 33 test messages under unforced training conditions. Finally, Test 4 utilised the larger training dataset (77 messages) with the 33 test messages under forced training conditions.

Table 3. Examples of correctly and incorrectly classified messages.

Correctly classified		Misclassified
Credible	Unreliable	Actually credible – classified unreliable	Actually unreliable – classified credible
Gold Coast – Springbrook Road is closed between Belmont Park Dv and Pine Creek Rd due to flooding. Also closed at the Austineville Rd turn off	Fast running water over the road at the bottom of the decent below lookout	The Ipswich Show Grounds has been declared the evacuation centre. For more information go to the Queensland Police Service Facebook page	Lots of road closures due to flooding, our back fence was partially pulled down by the flooded Coonowrin creek overnight
An emergency evacuation centre was set up at the Gatton Shire Hall in North Street Gatton. Residents who were evacuated from their homes in Grantham, Helidon and the low-lying parts of Gatton	Evacuation – Toogoolawah from river rising My sis has been told to evacuate by the SES as the water has reached her fence	Flash flooding at Goodna. Picture of Leslie Park off Bertha Street Goodna where the floodwater is rising	I live in Dubai and own a house in Ellena Street Paddington BNE, can you please advise if this house was affected?
Queensland Police Service: The D’Aguilar Highway at Kilcoy is now closed in both directions. Police remind motorists not to attempt to cross flooded roads or causeways	Thanks local baker keep spirit keep bake provide bread other side town picture nothing	There are two evacuation centres in Dalby. South State School, Bunya and Owen Steets, in Dalby. You can also evacuate to the Dalby Agriculture College in Nicholson Street	Creek (now river) still a long way over and flowing fast – no traffic passable

Table 4. Test 1 – unforced training results using the small training sample (35 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	24 (TP)	1 (FN)	25
Actually unreliable	7 (FP)	1 (TN)	8
Total	31	2	33

Table 5. Test 2 – forced training results using small training sample (35 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	23 (TP)	2 (FN)	25
Actually unreliable	0 (FP)	8 (TN)	8
Total	23	10	33

Table 6. Test 3 – unforced training results using the larger training sample (77 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	21 (TP)	4 (FN)	25
Actually unreliable	3 (FP)	5 (TN)	8
Total	24	9	33

Table 7. Test 4 – forced training results using the larger training sample (77 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	23 (TP)	2 (FN)	25
Actually unreliable	0 (FP)	8 (TN)	8
Total	23	10	33

The terms true positive (TP), true negative (TN), FP and false negative (FN) were used to compare the results of the classification. A TP result correctly predicts a ‘Credible’ outcome when it is ‘Credible’, a TN result correctly predicts an ‘Unreliable’ outcome when it is ‘Unreliable’, a FP result falsely predicts a ‘Credible’ outcome when it should be ‘Unreliable’ and finally, a FN result falsely predicts an ‘Unreliable’ outcome when it should be ‘Credible’.

Table 4 results indicate that the system correctly classified 24 out of 25 credible messages during unforced training, but only 1 out of the 8 messages identified as unreliable was correctly classified. This outcome resulted in a high number of FPs for the unforced training which indicated that further training was required.

When the system utilised the same training dataset but ran under forced training conditions the results as expected varied (Table 5). Of the 25 credible messages 23 messages were correctly classified and only 2 messages incorrectly classified. These results only varied slightly from the unforced training outcomes in regard to detecting credible messages correctly. However, there was a significant improvement in the correct detection of unreliable messages with all messages being correctly classified during this test. Overall, the results were considered acceptable with a high classification accuracy for both the credible and unreliable messages classification and hence validated the forced training conditions.

Next, the size of the training sample was increased from 35 messages to 77 messages and then the unforced and forced training was repeated on the same test dataset. The results of unforced training are shown in Table 6 and identify that for the credible message classification, 21 out of 25 messages were correctly classified which was a small decrease in accuracy compared to the previous result (Table 4). However, the classification accuracy of unreliable messages improved from one correctly classified message to five correctly classified messages out of the eight to be classified.

Finally, Table 7 shows the results of the classification using the larger training dataset under forced training conditions. The results of the testing are identical to the forced training using the smaller training dataset with 23 out of 25 credible messages correctly classified and all 8 unreliable messages were also correctly classified. This indicated that the forced training conditions were consistent and were not impacted by the changed training sample size.

A number of measures such as accuracy, precision, sensitivity and the F1-score provided an indication of each classification’s effectiveness. The accuracy, which is the ratio of correctly predicted observations, was calculated by the formula (TP + TN)/(TP + TN + FP + FN). The precision or positive predictive value (PPV) is the ratio of correct positive observations. The PPV was calculated by TP/(TP + FP). The F1-score (F1) is used to measure classification performance using the weighted recall and precision, where the recall is the percentage of relevant instances that are retrieved and was calculated by 2 * TP/(2 * TP + FP + FN). The sensitivity or true positive rate was calculated by TP/(TP + FN).

The classification quality for the four tests are summarised in Table 8. The accuracy and precision was higher for the forced training outcomes for both training sample sizes and indicates the importance of the forced training. It can also be seen that the classification accuracy and precision increased slightly for the unforced training outcomes when the larger training sample size was utilised. However, the precision and accuracy outcomes for the forced training were similar and indicate that there may be a lesser dependency on the size of the training dataset when force training is utilised. The F1-score did not change with the sample size but the measures indicate that the forced training again performed better than the unforced training scenarios. Finally, the classification sensitivity remained constant for the forced training for both training sample sizes but dropped slightly with the larger training sample size for the unforced training test outcomes.

Table 8. Quality of the CSD classification.

Test scenario	Accuracy	Precision	F1-score	Sensitivity
Test 1 – unforced Using the small training sample (35 messages) and 33 test messages	76	77	86	96
Test-2 – forced Using the larger training sample (77 messages) and 33 test messages	94	100	96	92
Test 3 – unforced Using the small training sample (35 messages) and 33 test messages	79	88	86	84
Test 4 – forced Using the larger training sample (77 messages) and 33 test messages	94	100	96	92

4.2. Results of the full Ushahidi CrowdMap data CSD analysis

After the training testing of the system was completed to an acceptable classification quality, the full Ushahidi CrowdMap sample of remaining 433 messages was analysed for credibility. As Figure 3(a) indicates, 54% (234 out of 433) of the messages were identified as credible using an unforced training classification. However, when the system was run under forced conditions, 77% (334 out of 433) of the messages were identified as credible (Figure 3(b)). This was a more confident value than the previous result as the accuracy and precision of the credibility detection was higher.

Figure 3. Assessed credibility of 2011 Australian flood’s Ushahidi CrowdMap data.

5. Conclusion

The CSD message credibility detection is a challenging task due to the high degree of variability of the data, the lack of a consistent data structure, the variability of the data providers and the limited metadata available. This study identified that Bayesian spam email detection approaches can be applied successfully to the challenge of classifying the credibility of CSD. However, the training approaches and the size of the training dataset can influence the quality and performance of the training outcomes.

Due to the variability of the data, it is recommended that forced training is undertaken to achieve the highest accuracy and performance. In particular, the forced training provided a higher level of confidence in eliminating the number of FP outcomes which were the incorrect classification of messages. The size of the training dataset was found to be less critical when a forced training approach was utilised with the results of the classification outcomes being similar for both the smaller and larger training datasets. However, if the system training was unforced, a larger training dataset is recommended.

Although this study focused on the issue of credibility, it should be recognised that the relevance of that dataset is another critical dimension in the quality assessment of the crowd sourced datasets. It is often not enough to just have a credible source of information as it is also important that the information is relevant to the purpose of the operational activity. For example, in the case of a flood disaster, the relevant information should relate to useful and relevant data regarding the support of the flood operations or emergency services. It is therefore important that future studies analyse both the credibility and the relevance of the crowd sourced datasets.

Disclosure statement

No potential conflict of interest was reported by the authors.

ORCID

Saman Koswatte http://orcid.org/0000-0002-3484-854X

Kevin McDougall http://orcid.org/0000-0001-6088-1004

Xiaoye Liu http://orcid.org/0000-0002-0487-9193

Additional information

Funding

Authors wish to acknowledge the Australian Government for providing support for the research work through the Research Training Program (RTP) and Monique Potts, ABC – Australia for providing the 2011 Australian Flood’s Ushahidi Crowdmap data.