A visual–textual fused approach to automated tagging of flood-related tweets during a flood event

Figures & data

Figure 1. Hurricane Harvey and the study area of Houston, TX.

Figure 2. The workflow of the proposed approach. The methodology is composed of three steps: CNN design and training, word sensitivity test and duplication test.

Figure 3. CNN architecture: the number before ‘@’ denotes the depth of a layer.

Table 1. Parameter settings.

Notation	Parameter description	Value
Feeding phrase
${ϵ}_z$	Zooming factor	$<0.2$
${\theta }_r$	Rotation factor	$<{40}^{\circ }$
$\tau$	Batch size	32
$M$	Input (resampled) image size $\left(M\ast M\right)$	128
N	Number of channels	3
Training phrase
$F_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}$	Spatial extent of convolutional filters $(F_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}\ast F_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}})$	3
$S_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}$	Convoluting stride$(S_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}}\ast S_{\mathrm{c}\mathrm{o}\mathrm{n}\mathrm{v}})$	2
$F_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}$	Spatial extent of pooling filters ($F_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}\ast F_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}$)	2
$S_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}$	Pooling stride $(S_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}}\ast S_{\mathrm{p}\mathrm{o}\mathrm{o}\mathrm{l}})$	2
$D_r$	Dropout rate of a D-layer	0.15
P	Zero padding	True^a
$\mathrm{e}$	Maximum epochs	200
$l_r^i$	Initial learning rate	0.00002
$l_r^e$	Progressive learning rate per epoch	$l_r^i/\mathrm{e}\mathrm{p}\mathrm{o}\mathrm{c}\mathrm{h}$^b
$N_{\mathrm{f}\mathrm{c}}^1$	Number of neurons in first FC	128
$N_{\mathrm{f}\mathrm{c}}^2$	Number of neurons in second FC	2

^aTrue zero padding denotes the application of zero padding in CNN training phrase.

^b $l_r^e$ is a progressive variable that updates itself on each epoch.

Figure 4. Training accuracy and loss of designed CNN architecture on flooding and non-flooding dataset supported by GTX 1050 GPU and CUDA.

Figure 5. CNN performance with different ${\mathit{T}}_{\mathit{f}}$: (a) number of pictures classified as positive (flooding), verified as positive (TP) and verified as negative (FP) given different $T_f$ and (b) trend of precision and recall given different $T_f$.

Table 2. CNN labeling result.

	Classified as positive			Classified as negative			Evaluation (positive)
$T_f$	Total	TPs	FPs (Type I)	Total	TNs	FNs (Type II)	Precision (%)	Recall (%)
0.5	2545	1301	1244	36,447	36,373	74	51.1	94.6
0.55	1828	1119	709	37,164	36,908	256	61.2	81.4
0.6	1364	964	400	37,628	37,217	411	70.7	70.1
0.65	997	810	187	37,995	37,430	565	81.2	58.9
0.7	755	663	92	38,237	37,525	712	87.8	48.2
0.75	539	500	39	38,453	37,578	875	92.8	36.3
0.8	349	338	11	38,643	37,606	1037	96.8	24.6
0.85	197	193	4	38,795	37,613	1182	98.0	14.0
0.9	74	73	1	38,918	37,616	1302	98.6	05.3
0.95	8	8	0	38,984	37,617	1367	100.0	00.6

Figure 6. Temporal distribution of selected flood-sensitive words (${\mathit{S}}_{\mathit{f}}>15$). The X-axis and Y-axis in each subfigure denote time (from 15th August to 15th September in 2017) and number of tweets (count) respectively. Number of tweets for each word has been scaled for better visualization and comparison.

Table 3. Positive label (TPs and FPs) statistics in CNN only and CNN refined by sensitive words.

	CNN only			CNN refined by sensitive words
$T_f$	Total extraction	TPs	FPs	Total extraction	TPs	FPs
0 . 5	2545	1301	1244	1455	1272	183
0.55	1828	1119	709	1273	1141	132
0.6	1364	964	400	972	883	89
0.65	997	810	187	812	739	73
0.7	755	663	92	658	608	50
0.75	539	500	39	513	472	41
0.8	349	338	11	321	294	27
0.85	197	193	4	185	170	15
0.9	74	73	1	67	65	2
0.95	8	8	0	8	7	1

Table 4. Precision, recall and $F_1$ score in CNN only and CNN refined by sensitive words.

	CNN only			CNN refined by sensitive words
$T_f$	Precision	Recall	$F_1$	Precision	Recall	$F_1$
0 . 5	51.1%	94.6%	0.664	87.4%	92.5%	0.899
0.55	61.2%	81.4%	0.699	89.6%	83.0%	0.862
0.6	70.7%	70.1%	0.704	90.8%	64.2%	0.752
0.65	81.2%	58.9%	0.683	91.0%	53.7%	0.676
0.7	87.8%	48.2%	0.622	92.4%	44.2%	0.598
0.75	92.8%	36.4%	0.522	92.0%	34.3%	0.500
0.8	96.8%	24.6%	0.392	91.6%	21.4%	0.347
0.85	98.0%	14.0%	0.246	91.9%	12.4%	0.218
0.9	98.6%	5.3%	0.101	97.0%	4.7%	0.902
0.95	100.0%	0.6%	0.012	87.5%	0.5%	0.101

Figure 7. Examples of selected originals and their duplications presented in their timeline. Only the originals (the first in timeline) were kept in the final selection.

Figure 8. Selected examples of final tweets geotagging.