Identifying disaster related social media for rapid response: a visual-textual fused CNN architecture

Figures & data

Figure 1. Conceptual workflow.

Figure 2. Transfer learned model from Inception-v3; the original Inception-v3 architecture graph (Main body) is adopted from Serengil (2018).

Table 1. Layer functionality summary.

Layer type	Functionality
Convolution	Learning features by applying a convolution operation to the input: $x_{ij}^l=\sum _{a=0}^{m-1}\sum _{b=0}^{m-1}{w}_{ab}{x}_{\left(i+a\right)\left(j+b\right)}^{l-1}$ where a $m\times m$ filter $w$ is applied to a previous layer $x^{l-1},$convoluting it down to the current layer $x^l$.
AvePool	An extreme type of dimensionality reduction where a $h\times \omega \times d$ feature map is further reduced to 1$\times 1\times d$ by averaging values in $R^{h\times \omega }$.
MaxPool	Reducing the spatial size of the representation by selecting the max value as representative: $x_{ij}^l=\mathrm{M}\mathrm{a}{\mathrm{x}}_{\left(i+e\right),\left(j+f\right)\in R^p}{x}_{\left(i+e\right)\left(j+f\right)}^{l-1}$ where $R^p$ denotes the pooling filter domain.
Concat	A concatenating function that merge shorter vectors to form a single long vector: $X=x_1\oplus x_1\oplus x_2\oplus \cdots \oplus x_n$ where $X$ denotes concatenated vector and $x_i$ denotes short vectors.
Dropout	A regulation function that prevents the co-adaption of hidden units by randomly setting a proportion to 0 in forward propagation, thus significantly reducing the chance of network being overfitted (Srivastava et al. 2014). I.e. replace $y=W\cdot z+b$ with $y=W\cdot \left(z\circ r\right)+b$ where ‘$\circ$’ represents element-wise multiplication and $W$, $z$, $b$ and $r$ to be weight matrix, feature vector, bias term and ‘masking’ vector, respectively (Kim 2014)
FC	Flattening high dimensional features by connecting every neuron in the prior layer to next layer.
Softmax	A function that classifies mutually exclusive labels and makes predicted probabilities to each label add up to 1. A binary Softmax function in this case can be simplified to a logistic function: $y={\displaystyle \frac{1}{1+e^{-\left(W\cdot \left(z\circ r\right)+b\right)}}}$

Figure 3. Word embedded CNN architecture.

Figure 4. Fusion of visual and textual information.

Figure 5. Research area for two flooding cases with their geotagged tweets; (a) Continental U.S; (b) South Carolina flood in 2015 with 934,896 geotagged tweets from October 2 to October 9; (c) Houston flood in 2017 with 501,516 geotagged tweets from August 25 to September 1.

Table 2. Visual training set.

Sources	Method	number of pictures
Flood pictures (positive)
Google	Pictures with keyword ‘flooding’	500
Baidu	Pictures with keyword ‘flooding’ and ‘Hongshui’ (‘flooding’ translated to Chinese Pinyi)	500
Flickr	Pictures tagged by ‘flooding’	500
Twitter & Instagram	Pictures from tweets with keyword ‘flood*’ or hashtag ‘flood*’ from Dec 1st 2015 to Dec 1st 2016 in U.S and manually verified.	4000
Positive total		5500
Non-flood pictures (negative)
Twitter & Instagram	Pictures extracted from tweets in U.S from Dec 1st 2015 to Dec 1st 2016 and manually verified.	5500
Negative total		5500

Table 3. Textual training set.

Subsets	Source	Sample size	Duration	Purpose
Word embedding training set (embedding model)	Twitter	13,830,023	1 January 2017 – 31 Decembwe 2017	To train the embedding model, Word2Vec, to learn high-dimensional word representations, providing 300-dimension vector for each word before feeding sentences to the neural network.
Textual feature training set		5706	2 October 2015 to 9 October 2015 (SC Flood)	To train the textual CNN to learn the meaning of textual information of a post, providing textual features (1024-dimension vector) that well characterize the given textual information

Figure 6. Model performance of the visual CNN: (a) the ROC curves for all 5 folds; (b) the training and validation accuracy curves for fold 3 (best fold) in two stages.

Table 4. Visual CNN performance.

	Fold 1	Fold 2	Fold 3	Fold 4	Fold 5	Average	SD
Accuracy	93.32%	92.98%	94.50%	91.96%	91.95%	92.94%	0.95%
AUC	0.983	0.987	0.989	0.978	0.971	0.982	0.006

Table 5. Word2Vec training results (top 5 neighboring words with their cosine-similarity distances).

Example words	Top 1	Top 2	Top 3	Top 4	Top 5
bad	terrible (0.635)	horrible (0.610)	shitty (0.553)	awful (0.524)	crappy (0.469)
good	great (0.639)	decent (0.563)	nice (0.489)	amazing (0.469)	interesting (0.461)
flood	hurricane (0.596)	damage (0.581)	evacuate (0.560)	storm (0.534)	underwater (0.496)
hurricane	storm (0.702)	Harvey (0.642)	flood (0.596)	tornado (0.477)	disaster (0.463)
father – son + daughter	mother (0.561)	grandmother (0.533)	wife (0.531)	woman (0.404)	beloved (0.403)
earthquake – quake + water	flood (0.523)	pouring (0.492)	underwater (0.444)	rain (0.418)	evacuated (0.413)
flood – water + wind	thunderstorm (0.512)	storm (0.498)	tornado (0.484)	rainfall (0.464)	gust (0.447)

Notes: Cosine-similarity distance is calculated as${\sum }_{i=1}^n{A}_i{B}_i/\left(\sqrt{{\sum }_{i=1}^n{A}_i^2}\sqrt{{\sum }_{i=1}^n{A}_i^2}\right)$ where $A_i$ and $B_i$ represent components of vector $\mathrm{A}$ and $\mathrm{B}$. The idea of a word vector $\delta$ similar to a vector resulting from a linear function: $\alpha -\text{β}+\gamma$ can be interpreted as word vector $\delta -\gamma$ similar to vector $\alpha -\text{β}$.

Figure 7. (a) Textual CNN ROC curve for all 5 folds; (b) Training accuracy curve for fold 1 during 200 epochs; (b) Training loss curve for fold 1 during 200 epochs.

Table 6. Textual CNN performance.

	Fold 1	Fold 2	Fold 3	Fold 4	Fold 5	Average	SD
Accuracy	94.17%	93.13%	92.54%	93.64%	93.79%	93.45%	0.63%
AUC	0.970	0.963	0.960	0.967	0.966	0.965	0.004

Table 7. Visual-textual fused classification accuracy.

Method	Fold 1	Fold 2	Fold 3	Fold 4	Fold 5	Average	SD
LogR	96.3%	95.5%	97.1%	97.6%	96.2%	96.5%	0.73%
DT	91.5%	89.9%	92.5%	91.0%	90.6%	91.1%	0.87%
RF	93.1%	93.2%	93.7%	94.7%	92.9%	93.5%	0.64%
NB:
Gaussian	88.1%	87.0%	88.5%	90.3%	89.1%	88.6%	1.09%
Multinomial	76.4%	75.3%	77.6%	79.2%	76.7%	77.0%	1.31%
Bernoulli	67.8%	68.1%	70.7%	69.6%	68.3%	68.9%	1.09%
DA:
LDA	86.3%	87.2%	86.6%	88.5%	88.3%	87.4%	0.88%
QDA	72.2%	73.6%	75.2%	73.9%	73.7%	73.7%	0.95%
SVM:
Linear	96.2%	95.6%	96.2%	96.9%	96.4%	96.3%	0.42%
Polynomial	74.1%	62.2%	75.4%	66.5%	70.2%	69.7%	4.87%
RBF	94.7%	93.7%	95.0%	94.9%	93.9%	94.4%	0.54%
Sigmoid	94.5%	93.4%	94.3%	94.4%	93.7%	94.1%	0.43%

Notes: LogR, DT, RF, NB and DA denote Logistic Regression, Decision Tree, Random Forest, Naïve Bayes and Discriminant Analysis, respectively. LDA and QDA denote Linear Discriminant Analysis and Quadratic discriminant analysis, respectively. Linear, Polynomial, RBF (Radial Basis Function) and Sigmoid are kernel functions utilized in SVM classifier. Classification method with an average accuracy of over 90% for all 5 folds is highlighted in bold.

Table 8. Visual-textual fused classification compared with textual only.

Method	Accuracy
	Textual only	Visual-textual fused
LogR	82.6%	95.2%
DT	77.6%	89.7%
RF	80.5%	92.1%
SVM (Linear)	82.9%	92.7%
SVM (RBF)	79.5%	90.2%
SVM (Sigmoid)	79.2%	93.6%

Figure 8. ROC curves of the six algorithms using visual-textual fused vector ($Z_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}$) and using textual vector ($Z_{\mathrm{t}\mathrm{e}\mathrm{x}\mathrm{t}\mathrm{u}\mathrm{a}\mathrm{l}}$) alone; (a) LogR; (b) DT; (c) RF; (d) SVM (Linear); (e) SVM (RBF); (f) SVM (Sigmoid).

Figure 9. Eight Examples of classification results. $P_{\mathrm{f}\mathrm{l}\mathrm{o}\mathrm{o}\mathrm{d}}$ denotes the probability of a post being flood relevant. Blurring was applied to faces appeared in the photo.

Serengil, S. L. 2018. “Transfer Learning in Keras Using Inception V3.” March 12. Accessed June 12, 2018. https://sefiks.com/2017/12/10/transfer-learning-in-keras-using-inception-v3/.

 Google Scholar

Srivastava, N., G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. “Dropout: A Simple Way to Prevent Neural Networks from Overfitting.” The Journal of Machine Learning Research 15 (1): 1929–1958.

 Google Scholar

Kim, Y. 2014. “Convolutional Neural Networks for Sentence Classification.” arXiv preprint arXiv:1408.5882.

 Google Scholar