Robust Rumor Detection based on Multi-Defense Model Ensemble

Figures & data

Figure 1. Rumor detection framework based on multi-defense model ensemble.

Table 1. Adversarial Text Examples.

Attack Methods	Original Text	Adversarial Text
TextFooler	North korea may be preparing for a bullet launch.	North korea may be develop for a rocket launch.
PWWS(Wordnet)	North korea may be preparing for a bullet launch.	North korea may be preparing for a projectile launch.
PWWS(Hownet)	North korea may be preparing for a bullet launch.	North korea may be preparing for a cartridge launch.

Table 2. Results of different models on the Twitter15 dataset and its adversarial text.

Models		Original data	TextFooler			PWWS(Wordnet)			PWWS(Hownet)
		Acc	Acc	Suc	Que	Acc	Suc	Que	Acc	Suc	Que
$f^{O\mathrm{r}\mathrm{i}\mathrm{g}\mathrm{i}\mathrm{n}\mathrm{a}\mathrm{l}}$	CSI	0.712	0.207	0.79	36.8	0.166	0.80	102.5	0.192	0.82	269.1
	dEFEND	0.771	0.281	0.72	56.9	0.202	0.86	131.8	0.198	0.79	341.7
	GLAN	0.827	0.374	0.59	53.4	0.259	0.82	213.8	0.287	0.71	322.6
	Bert	0.824	0.396	0.48	65.5	0.274	0.81	296.4	0.292	0.77	406.0
$f^{D\mathrm{a}\mathrm{t}\mathrm{a}}$	CSI	0.709	0.337	0.74	66.4	0.522	0.64	284.9	0.543	0.61	404.5
	dEFEND	0.717	0.402	0.63	80.4	0.693	0.60	298.5	0.690	0.56	474.8
	GLAN	0.821	0.519	0.53	86.3	0.734	0.57	384.7	0.795	0.47	487.4
	Bert	0.824	0.522	0.45	103.9	0.791	0.52	446.7	0.788	0.49	579.2
$f^{RS}$	CSI	0.704	0.421	0.68	72.1	0.427	0.68	197.3	0.467	0.67	329.4
	dEFEND	0.765	0.497	0.57	87.7	0.519	0.62	294.5	0.518	0.65	382.2
	GLAN	0.824	0.613	0.44	98.0	0.633	0.60	337.4	0.620	0.59	395.8
	Bert	0.822	0.624	0.37	99.2	0.674	0.54	385.3	0.623	0.60	476.9
$f^{PGD}$	CSI	0.719	0.334	0.74	57.9	0.250	0.71	178.5	0.239	0.74	309.3
	dEFEND	0.773	0.412	0.62	61.2	0.362	0.70	204.7	0.351	0.74	365.8
	GLAN	0.831	0.470	0.48	74.2	0.417	0.69	278.4	0.447	0.63	335.9
	Bert	0.827	0.488	0.39	81.8	0.494	0.72	332.3	0.462	0.68	465.3

Figure 2. Robust indicators improvement value of each rumor detection model under adversarial conditions.

Table 3. Experimental results after CSI apply the ensemble strategy.

	Original data	TextFooler		PWWS(Wordnet)		PWWS(Hownet)
	Acc	Acc	Suc	Acc	Suc	Acc	Suc
$f^{Original}$	0.712	0.207	0.79	0.166	0.80	0.192	0.82
$f^{Data}$	0.709	0.337	0.74	0.522	0.64	0.543	0.61
$f^{RS}$	0.704	0.421	0.68	0.427	0.68	0.467	0.67
$f^{PGD}$	0.719	0.334	0.74	0.25	0.71	0.239	0.74
loggits-summed	0.718	0.425	0.68	0.479	0.62	0.457	0.67
majority-vote	0.730	0.478	0.62	0.536	0.59	0.548	0.56

Table 4. Experimental results after dEFEND applies the ensemble strategy.

	Original data	TextFooler		PWWS(Wordnet)		PWWS(Hownet)
	Acc	Acc	Suc	Acc	Suc	Acc	Suc
$f^{Original}$	0.771	0.281	0.72	0.202	0.86	0.198	0.79
$f^{Data}$	0.717	0.402	0.63	0.693	0.60	0.69	0.56
$f^{RS}$	0.765	0.497	0.57	0.519	0.62	0.518	0.65
$f^{PGD}$	0.773	0.412	0.62	0.362	0.7	0.351	0.74
loggits-summed	0.765	0.464	0.58	0.697	0.60	0.622	0.59
majority-vote	0.792	0.510	0.53	0.712	0.57	0.702	0.58

Table 5. Experimental results after GLAN are applied to the ensemble strategy.

	Original data	TextFooler		PWWS(Wordnet)		PWWS(Hownet)
	Acc	Acc	Suc	Acc	Suc	Acc	Suc
$f^{Original}$	0.827	0.374	0.59	0.259	0.82	0.287	0.71
$f^{Data}$	0.821	0.519	0.53	0.734	0.57	0.795	0.47
$f^{RS}$	0.824	0.613	0.44	0.633	0.6	0.62	0.59
$f^{PGD}$	0.831	0.47	0.48	0.417	0.69	0.447	0.63
loggits-summed	0.789	0.616	0.44	0.747	0.33	0.764	0.30
majority-vote	0.845	0.665	0.42	0.777	0.30	0.783	0.30

Table 6. Experimental results after Bert applies the ensemble strategy.

	Original data	TextFooler		PWWS(Wordnet)		PWWS(Hownet)
	Acc	Acc	Suc	Acc	Suc	Acc	Suc
$f^{Original}$	0.824	0.396	0.48	0.274	0.81	0.292	0.77
$f^{Data}$	0.824	0.522	0.45	0.791	0.52	0.788	0.49
$f^{RS}$	0.822	0.624	0.37	0.674	0.54	0.623	0.6
$f^{PGD}$	0.827	0.488	0.39	0.494	0.72	0.462	0.68
loggits-summed	0.827	0.619	0.37	0.780	0.30	0.790	0.29
majority-vote	0.863	0.683	0.35	0.793	0.29	0.824	0.27