Detection of Hate Speech using BERT and Hate Speech Word Embedding with Deep Model

Figures & data

Figure 1. Block diagram of the experiments.

Table 1. Datasets description.

Dataset	Original labels	Number of hate	Number of non-hate	Total
Davidson-ICWSM (Davidson et al. 2017)	Hate speech, offensive, and neither	20620	4163	24783
Waseem-EMNLP (Waseem 2016)	Racism, sexism, both, and neither	1059	5850	6909
Waseem-NAACL (Waseem and Hovy 2016)	Sexist, racist, and neither	5406	11501	16910
Balanced Combined	Racism, sexism, and neither, both	16260	16260	32520

Table 2. Details description of embedding models.

Methods	Dimension	Trained on data of size	Pretrained on Platform
GoogleNews-vectors-negative^a (Mikolov et al. 2013)	300	3 billion words	Google News
glove.6B. 300d^b (Pennington, Socher, and Manning 2014)	300	6B tokens, 400K vocab	Wikipedia 2014
glove.twitter.27B.200d ^b (Pennington, Socher, and Manning 2014)	200	2B tweets, 27B tokens, 1.2M vocab	Twitter
W2V-Hate (Gupta and Waseem 2017)	300	1 billion documents	Twitter
Hate speech Word2Vec (HSW2V)	300	1,048,563 sentences, 116955 vocab, uncased	Twitter, hate websites

a. https://code.google.com/archive/p/word2vec/

b. https://nlp.stanford.edu/projects/glove/

Table 3. Results of bidirectional LSTM-based deep model on the datasets.

Source	Machine Learning Approach	Methods	Dataset	P	R	f1-score (hate)	f1-score (non-hate)	f1-score	AUC
Gupta and Waseem (2017)	LR	Hate W2V(300)	Davidson ICWSM	0.91	0.91	-	-	0.9120	0.8400
			Waseem EMNLP	0.84	0.86	-	-	0.8440	0.6380
			Waseem NAACL	0.76	0.77	-	-	0.7500	0.6790
Our proposed deep model	BiLSTM Deep model	GoogleNews- vectors- negative300	Davidson ICWSM	0.94	0.94	0.9679	0.8457	0.9473	0.9132
			Waseem EMNLP	0.91	0.91	0.6737	0.9484	0.9033	0.7697
			Waseem NAACL	0.80	0.80	0.6805	0.8542	0.7990	0.7654
			Combined balanced	0.94	0.94	0.9365	0.9376	0.9371	0.9370
		GloVe.6B.300d	Davidson ICWSM	0.94	0.94	0.9646	0.8310	0.9421	0.9073
			Waseem EMNLP	0.90	0.91	0.6631	0.9463	0.9028	0.7792
			Waseem NAACL	0.80	0.80	0.6794	0.8569	0.8002	0.7634
			Combined balanced	0.94	0.94	0.9414	0.9413	0.9414	0.9414
		GloVe.Twitter. 27B.200d	Davidson ICWSM	0.95	0.95	0.9676	0.8347	0.9453	0.8927
			Waseem EMNLP	0.91	0.90	0.6917	0.9399	0.9018	0.8324
			Waseem NAACL	0.80	0.80	0.6911	0.8503	0.7994	0.7738
			Combined balanced	0.94	0.94	0.9381	0.9394	0.9388	0.9388
		HSW2V(300)	Davidson ICWSM	0.95	0.95	0.9703	0.8551	0.9509	0.9162
			Waseem EMNLP	0.91	0.91	0.6928	0.9469	0.9079	0.8094
			Waseem NAACL	0.81	0.81	0.7055	0.8634	0.8129	0.7831
			Combined balanced	0.94	0.94	0.9428	0.9439	0.9434	0.9434

Table 4. BERT for sequence classification hate speech experiment results (base-large).

Methods	Datasets	P	R	f1-score (Hate)	f1-score (non- Hate)	f1-score	AUC
BERT Base	Davidson- ICWSM	0.96	0.96	0.98	0.89	0.962	0.9309
	Waseem- EMNLP	0.92	0.92	0.7654	0.9541	0.9216	0.8455
	Waseem- NAACL	0.85	0.85	0.7612	0.8881	0.8472	0.8227
	Combined Balanced	0.95	0.95	0.9543	0.9552	0.9547	0.9547
BERT Large	Davidson- ICWSM	0.96	0.96	0.9788	0.8924	0.9646	0.9345
	Waseem- EMNLP	0.91	0.91	0.6939	0.9458	0.9103	0.8371
	Waseem -NAACL	0.85	0.85	0.7643	0.8937	0.8521	0.823
	Combined Balanced	0.96	0.96	0.962	0.9625	0.9623	0.9623

Table 5. Word similarity of misspelled hate word fc*.

The word	Word2Vec	Glove	HSW2V
fc* (missplled)	hahah,lmfao, questlove_@, u_ppl,hahahahahahaha, JeremyShockey_@, ummmmm, ROTFLMAO, Awright, freaken	h?ndbold, nordsj?lland, k?benhavn,br?ndby, hik, parken, s?nderjyske, genēlerbirli?i, 1972/73, iwr, al-hilal	fu*, lmfaoo, shii, fucc, dese, lmaooooo, dats, noooo, lmfaoooo, ay

Figure 2. BERT Base vocabulary search for misspelling and hate term.

Table 6. Confusion matrix of HSW2 V and BERT.

Datasets/model	Davidson-ICWSM	Waseem-EMNLP	Waseem-NAACL	Balanced Dataset
HSW2V(300), Bidirectional LSTM deep model
BERT Large

Figure 3. Result of applying LIME on document actual label=1 and predicted=1.

Figure 4. Result of applying LIME on document actual label=0 and predicted=0.

Figure 5. Result of applying LIME on document actual label=0 and predicted=1.

Figure 6. Result of applying LIME on document actual label=1 and predicted=0.

Davidson, T., D. Warmsley, M. Macy, and I. Weber. 2017. “Automated hate speech detection and the problem of offensive language.” In Eleventh international aaai conference on web and social media, Canada.

 Google Scholar

Waseem, Z. 2016. Are you a racist or am i seeing things? annotator influence on hate speech detection on twitter. In Proceedings of the first workshop on NLP and computational social science, Austin, 138–42.

 Google Scholar

Waseem, Z., and D. Hovy. 2016. Hateful symbols or hateful people? predictive features for hate speech detection on twitter. In Proceedings of the NAACL student research workshop, San Diego, California, 88–93.

 Google Scholar

Mikolov, T., K. Chen, G. Corrado, and J. Dean. 2013. Efficient estimation of word representations in vector space. arXiv preprint arXiv:13013781 11:608–611.

 Google Scholar

Pennington, J., R. Socher, and C. D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP’14), 1532–43.

 Google Scholar

Gupta, S., and Z. Waseem. 2017. A comparative study of embeddings methods for hate speech detection from tweets. Association for Computing Machinery.

 Google Scholar