Saliency Detection Using a Bio-inspired Spiking Neural Network Driven by Local and Global Saliency

Figures & data

Figure 1. Comparison with prior based saliency detection methods (a)original image (b)groundtruth (GT) (c)GR (Yang, Zhang, and Lu 2013) (d)MR (Yang et al. 2013) (e)LGSD-DCPCNN.

Figure 2. Process of proposed algorithm.

Table 1. Features used for generating global saliency map.

Index Component	Feature	Dimension
Features related to Location
0	Normalized x coordinate mean	1
1	Normalized y coordinate mean	1
Features related to color values
2–4	Mean CIELab values	3
5–7	Mean RGB values	3
8	Mean Hue value	1
9	Mean Saturation value	1
Features related to Color Histogram
10	RGB-Histogram	1
11	CIELab-Histogram	1
12	Hue-Histogram	1
13	Saturation-Histogram	1
Features related to Textures
14–21	The Histogram of Gradient	8
22–80	The LBP Histogram	59

Table 2. Features used in local saliency map generation.

Features	Dimension
Superpixels spatial distance from K-nearest foreground superpixels	K
Superpixels spatial distance from K-nearest background superpixels	K
Superpixels color distance from K-nearest foreground superpixels	8 K
Superpixels color distance from K-nearest background superpixels	8 K
Superpixels texture distance from K-nearest foreground superpixels	10 K
Superpixels texture distance from K-nearest background superpixels	10 K

Table 3. Salient object detection datasets.

Sr. No.	Dataset	No.of images	Description
1	SOD (Movahedi and Elder 2010)	300	Many images include multiple prominent items with little color contrast to the background
2	ECSSD (Shi et al. 2015)	1000	The dataset contains multiple salient objects and also having a complex background, which makes the dataset more challenging for the salient object detection task
3	DUT-OMRON (Yang et al. 2013)	5168	Images of high quality with multiple significant objects, while the backgrounds are relatively cluttered
4	HKU-IS (Li and Yu 2015)	4447	Majority of the images are low contrast with several salient objects

Table 4. Evaluation metrics for salient object detection.

Sr. No.	Evaluation Metrics	Mathematical Expression
1	Precision (Davis and Goadrich 2006)	$Precision=\frac{{\sum }_{x=1}^w{\sum }_{y=1}^{h}Sma{p}_{x,y}^{FT}\cdot G{T}_{x,y}}{{\sum }_{x=1}^w{\sum }_{y=1}^{h}Sma{p}_{x,y}^{FT}}$
2	Recall (Davis and Goadrich 2006)	$Recall=\frac{{\sum }_{x=1}^w{\sum }_{y=1}^{h}Sma{p}_{x,y}^{FT}\cdot G{T}_{x,y}}{{\sum }_{x=1}^w{\sum }_{y=1}^{h}G{T}_{x,y}}$
3	F-measure (Achanta et al. 2009)	$F_m=\frac{(1+{\xi }^2)\cdot Precision\cdot Recall}{({\xi }^2\cdot Precision)+Recall}$ where ${\xi }^2=0.3$
4	MAE (Borji et al. 2015)	$MAE=\frac{1}{w\times h}{\sum }_{x=1}^w{\sum }_{y=1}^h|Sma{p}_{x,y}^{FN}-G{T}_{x,y}|$
5	TPR (Davis and Goadrich 2006)	$TPR=\frac{|Sma{p}^{FT}\cap GT|}{|GT|}$
6	FPR (Davis and Goadrich 2006)	$FPR=\frac{|Sma{p}^{FT}\cap GT|}{|Sma{p}^{FT}\cap GT|+|\bar{S}ma{p}^{FT}\cap \bar{G}T|}$

Figure 3. Qualitative analysis of the proposed algorithm on DUTOMRON dataset compared to other salient object detection techniques (a)Original Image (b)Groundtruth (c)CA (d)GR (e)SEG (f)MR (g)MC (h)LPS (i)RR (j)LGF (k)DPSG (l)NCUT (m)RCRR (n)SMD (o)LGSD-DCPCNN.

Figure 4. Qualitative analysis of the proposed algorithm on ECSSD dataset compared to other salient object detection techniques (a)Original Image (b)Groundtruth (c)CA (d)GR (e)SEG (f)MR (g)MC (h)LPS (i)RR (j)LGF (k)DPSG (l)NCUT (m)RCRR (n)SMD (o)LGSD-DCPCNN.

Figure 5. Qualitative analysis of the proposed algorithm on HKUIS dataset compared to other salient object detection techniques (a)Original Image (b)Groundtruth (c)CA (d)GR (e)SEG (f)MR (g)MC (h)LPS (i)RR (j)LGF (k)DPSG (l)NCUT (m)RCRR (n)SMD (o)LGSD-DCPCNN.

Figure 6. Qualitative analysis of the proposed algorithm on SOD dataset compared to other salient object detection techniques (a)Original Image (b)Groundtruth (c)CA (d)GR (e)SEG (f)MR (g)MC (h)LPS (i)RR (j)LGF (k)DPSG (l)NCUT (m)RCRR (n)SMD (o)LGSD-DCPCNN.

Figure 7. Quantitative analysis of the proposed algorithm based on the PR curve on four benchmark saliency detection datasets (a)DUT-OMRON (Yang et al. 2013) (b)ECSSD (Shi et al. 2015) (c)HKUIS (Li and Yu 2015) (d)SOD (Movahedi and Elder 2010).

Figure 8. Quantitative analysis of the proposed algorithm based on the ROC curve on four benchmark saliency detection datasets (a)DUT-OMRON (Yang et al. 2013) (b)ECSSD (Shi et al. 2015) (c)HKUIS (Li and Yu 2015) (d)SOD (Movahedi and Elder 2010).

Table 5. Quantitative analysis of the proposed algorithm with other saliency detection techniques based on the MAE, F-measure (Fm) and AUC values on four benchmark saliency detection datasets. (Red, green, and blue highlight the three leading models, respectively.) MAE less is better, AUC and Fm more is better.

Datasets	DUT-OMRON			HKUIS			ECSSD			SOD
Evaluation metric	AUC	Fmeas	MAE	AUC	Fmeas	MAE	AUC	Fmeas	MAE	AUC	Fmeas	MAE
CA	0.771	0.355	0.254	0.77	0.46	0.274	0.726	0.431	0.309	0.717	0.449	0.313
GR	0.804	0.486	0.259	0.802	0.564	0.256	0.784	0.512	0.283	0.719	0.442	0.318
SEG	0.778	0.394	0.337	0.774	0.464	0.323	0.746	0.394	0.341	0.681	0.316	0.36
MR	0.781	0.48	0.187	0.787	0.517	0.178	0.79	0.691	0.189	0.714	0.57	0.259
MC	0.82	0.533	0.186	0.831	0.678	0.184	0.81	0.699	0.202	0.75	0.59	0.261
LPS	0.767	0.539	0.187	0.752	0.619	0.164	0.782	0.644	0.196	0.703	0.526	0.261
RR	0.782	0.529	0.185	0.796	0.665	0.172	0.796	0.697	0.184	0.713	0.575	0.26
LGF	0.811	0.532	0.178	0.819	0.669	0.177	0.803	0.708	0.196	0.733	0.592	0.256
DPSG	0.796	0.559	0.177	0.808	0.694	0.165	0.795	0.701	0.181	0.71	0.571	0.251
NCUT	0.798	0.568	0.174	0.799	0.679	0.167	0.797	0.69	0.186	0.721	0.577	0.248
RCRR	0.78	0.527	0.182	0.796	0.664	0.171	0.794	0.696	0.184	0.717	0.577	0.257
SMD	0.809	0.538	0.166	0.823	0.691	0.156	0.811	0.729	0.173	0.737	0.61	0.235
FCB	0.668	0.480	0.149	–	–	–	0.702	0.607	0.173	0.641	0.520	0.244
LSP	0.803	0.546	0.133	–	–	–	0.808	0.761	0.152	0.732	0.661	0.226
BSDL	0.668	–	0.145	–	–	–	0.811	–	0.098	0.741	–	0.198
CDHL	0.806	0.709	0.153	–	–	–	0.824	0.581	0.143	–	–	–
TMR	–	0.592	0.159	–	–	–	–	0.708	0.143	–	–	–
LGSD-DCPCNN	0.832	0.621	0.142	0.839	0.702	0.164	0.828	0.71	0.15	0.753	0.65	0.23

Table 6. Quantitative analysis of the proposed algorithm with deep learning based saliency detection techniques based on the MAE and AUC values on four benchmark saliency detection datasets. (Bold values show the best outcome, AUC higher is better and MAE lower is better)

Datasets	DUT-OMRON		HKUIS		ECSSD		SOD
Evaluation metric	AUC	MAE	AUC	MAE	AUC	MAE	AUC	MAE
KSR	0.825	0.131	0.824	0.12	0.823	0.134	0.739	0.199
RSD-R	0.812	0.178	0.867	0.155	0.845	0.172	0.753	0.226
PAGR	0.813	0.071	0.867	0.047	0.849	0.061	0.744	0.147
SSNet	0.773	0.056	0.819	0.041	0.813	0.046	0.692	0.118
EGNet-R	–	–	–	–	0.865	0.037	0.782	0.099
HRSOD-DH	0.782	0.065	0.851	0.042	0.839	0.052	0.716	0.139
SCRN	0.864	0.056	0.883	0.034	0.868	0.037	0.781	0.107
LGSD-DCPCNN	0.832	0.142	0.839	0.164	0.828	0.15	0.753	0.23

Figure 9. Qualitative analysis of the proposed algorithm compared to other deep learning based salient object detection techniques (a)Original Image (b)Groundtruth (c)KSR (d)HRSOD-DH (e)PAGR (f)EGNet-R (g)RSD-R (h)LGSD-DCPCNN.

Figure 10. Comparative results using F-measure curve on the HKUIS dataset for features selection of global saliency map generation.

Figure 11. The proposed method’s visual output when baseline saliency maps produce poor results. (a)original image (b)GT (c)global saliency map (d)LGSD-DCPCNN.

Yang, C., L. Zhang, and H. Lu. 2013. Graph-regularized saliency detection with convex-hull- based center prior. IEEE Signal Processing Letters 20 (7):637–40. doi:10.1109/LSP.2013.2260737.

 Web of Science ®Google Scholar

Yang, C., L. Zhang, H. Lu, X. Ruan, and M.-H. Yang (2013). Saliency detection via graph- based manifold ranking. In Proceedings of the ieee conference on computer vision and pattern recognition, Portland, OR, USA (pp. 3166–73).

 Google Scholar

Movahedi, V., and J. H. Elder (2010). Design and perceptual validation of performance measures for salient object segmentation. In 2010 ieee computer society conference on computer vision and pattern recognition-workshop, San Francisco, CAs (pp. 49–56).

 Google Scholar

Shi, J., Q. Yan, L. Xu, and J. Jia. 2015. Hierarchical image saliency detection on extended cssd. IEEE Transactions on Pattern Analysis and Machine Intelligence 38 (4):717–29. doi:10.1109/TPAMI.2015.2465960.

 Web of Science ®Google Scholar

Li, G., and Y. Yu (2015). Visual saliency based on multiscale deep features. In Proceedings of the ieee conference on computer vision and pattern recognition, Boston, MA, USA (pp. 5455–63).

 Google Scholar

Davis, J., and M. Goadrich (2006). The relationship between precision-recall and roc curves. In Proceedings of the 23rd international conference on machine learning, Pittsburgh Pennsylvania, USA (pp. 233–40).

 Google Scholar

Achanta, R., S. Hemami, F. Estrada, and S. Susstrunk (2009). Frequency-tuned salient region detection. In 2009 ieee conference on computer vision and pattern recognition, Miami, Florida (pp. 1597–2928).

 Google Scholar

Borji, A., M.-M. Cheng, H. Jiang, and J. Li. 2015. Salient object detection: A benchmark. IEEE Transactions on Image Processing 24 (12):5706–22. doi:10.1109/TIP.2015.2487833.

 PubMed Web of Science ®Google Scholar