VGI and crowdsourced data credibility analysis using spam email detection techniques

Figures & data

Figure 1. Main steps involved in filter-based email classification (Guzella and Caminhas 2009).

Figure 2. CSD credibility detection workflow.

Table 1. Example of the combination results of the Incident title and Description of the Ushahidi CrowdMap message fields.

Incident title	Description	Combined message
Road Closed-Manly Rd between new Cleveland Rd and Castlerea St, Manly	Road closed due to flooding	Road Closed-Manly Rd between new Cleveland Rd and Castlerea St, Manly road closed due to flooding

Table 2. Example of the combination result of the Incident title, Description and Location of the Ushahidi CrowdMap message fields.

Incident title	Description	Location	Combined message
Roads Affected	Not passable	Gailey Rd, St Lucia	Roads Affected Not passable Gailey Rd, St Lucia

Table 3. Examples of correctly and incorrectly classified messages.

Correctly classified		Misclassified
Credible	Unreliable	Actually credible – classified unreliable	Actually unreliable – classified credible
Gold Coast – Springbrook Road is closed between Belmont Park Dv and Pine Creek Rd due to flooding. Also closed at the Austineville Rd turn off	Fast running water over the road at the bottom of the decent below lookout	The Ipswich Show Grounds has been declared the evacuation centre. For more information go to the Queensland Police Service Facebook page	Lots of road closures due to flooding, our back fence was partially pulled down by the flooded Coonowrin creek overnight
An emergency evacuation centre was set up at the Gatton Shire Hall in North Street Gatton. Residents who were evacuated from their homes in Grantham, Helidon and the low-lying parts of Gatton	Evacuation – Toogoolawah from river rising My sis has been told to evacuate by the SES as the water has reached her fence	Flash flooding at Goodna. Picture of Leslie Park off Bertha Street Goodna where the floodwater is rising	I live in Dubai and own a house in Ellena Street Paddington BNE, can you please advise if this house was affected?
Queensland Police Service: The D’Aguilar Highway at Kilcoy is now closed in both directions. Police remind motorists not to attempt to cross flooded roads or causeways	Thanks local baker keep spirit keep bake provide bread other side town picture nothing	There are two evacuation centres in Dalby. South State School, Bunya and Owen Steets, in Dalby. You can also evacuate to the Dalby Agriculture College in Nicholson Street	Creek (now river) still a long way over and flowing fast – no traffic passable

Table 4. Test 1 – unforced training results using the small training sample (35 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	24 (TP)	1 (FN)	25
Actually unreliable	7 (FP)	1 (TN)	8
Total	31	2	33

Table 5. Test 2 – forced training results using small training sample (35 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	23 (TP)	2 (FN)	25
Actually unreliable	0 (FP)	8 (TN)	8
Total	23	10	33

Table 6. Test 3 – unforced training results using the larger training sample (77 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	21 (TP)	4 (FN)	25
Actually unreliable	3 (FP)	5 (TN)	8
Total	24	9	33

Table 7. Test 4 – forced training results using the larger training sample (77 messages) and 33 test messages.

	Classified as credible	Classified as unreliable	Total
Actually credible	23 (TP)	2 (FN)	25
Actually unreliable	0 (FP)	8 (TN)	8
Total	23	10	33

Table 8. Quality of the CSD classification.

Test scenario	Accuracy	Precision	F1-score	Sensitivity
Test 1 – unforced Using the small training sample (35 messages) and 33 test messages	76	77	86	96
Test-2 – forced Using the larger training sample (77 messages) and 33 test messages	94	100	96	92
Test 3 – unforced Using the small training sample (35 messages) and 33 test messages	79	88	86	84
Test 4 – forced Using the larger training sample (77 messages) and 33 test messages	94	100	96	92

Figure 3. Assessed credibility of 2011 Australian flood’s Ushahidi CrowdMap data.

Guzella, Thiago S., and Walmir M. Caminhas. 2009. “A Review of Machine Learning Approaches to Spam Filtering.” Expert Systems with Applications 36 (7): 10206–10222. doi: 10.1016/j.eswa.2009.02.037

 Web of Science ®Google Scholar