An unsupervised semantic segmentation method that combines the ImSE-Net model with SLICm superpixel optimization

Figures & data

Figure 1. Illustration of the workflow for the proposed model. The pre-segmentation phase integrates the ImSE-Net model with SLICm superpixel optimization, generating a preliminary semantic segmentation result denoted by the red dashed line above. The UGLS algorithm for further refinement classification results is indicated by the lower green dashed frame. (a) Original image; (b) segmentation result using ImSE-Net; (c) contour refinement based on SLICm; (d) segmentation result combining ImSE-Net and SLICm; (e) final segmentation result.

Figure 2. Illustration of the workflow for the proposed ImSE-Net method. FCN-32 uses ResNet-101 as the backbone, in combination with the proposed ImSE-Net model from Conv2_x to Conv5_x. This method groups pixels from the downsampling layer to create a allocation matrix assigning predicted target values to each pixel group. Unlike existing neural network methods, this method does not explicitly employ superpixel segmentation for downsampling. Hence, the method can be integrated into existing architectures without altering their feedforward paths.

Table 1. Detailed process for the ImSE-Net.

Algorithm 1. ImSE-Net segmentation model
Step 1: Input an image $X=\left[x_1,x_2,\dots ,x_n\right]$and generate a prediction map $y^{\left(32\right)}=\mathrm{\varnothing }\left(x^{\left(32\right)}\right)$;
Step 2: Downsampling based on superpixels in the FCN-32 architecture: Pixels are grouped as superpixel seeds to generate multiple clusters $\left\{C^{\left(s\right)}\right\}$; Determining the cluster $A^{\left(s\right)\ast }=\arg \underset{A^{\left(s\right)}\in {\left\{0,1\right\}}^{U\times V}}{max}\sum _{\mathrm{ij}}{A}_{\mathrm{ij}}^{\left(s\right)}{S}_{\mathrm{ij}}^{\left(s\right)}$, $s.t.,\sum _j{A}_{\mathrm{ij}}^{\left(s\right)}=1$, to which each pixel i belongs;
Step 3: Decoding low-resolution feature images to high-resolution ${\tilde{x}}^{\left(s\right)}=A^{\left(s\right)\ast }{x}^{\left(2s\right)}$;
Step 4: Recursive decoding with bilinear upsampling $y^{\left(1\right)}=\prod _{s^{\widehat{{}^{\mathrm{\prime }}}}}=\left\{16,8,4,2,1\right\}{A}^{\left(s^{\widehat{{}^{\mathrm{\prime }}}}\right)}\ast y^{\left(32\right)}$;
Step 5: Output the ImSE-Net segmentation result Y.

Table 2. Detailed process for superpixel refinement.

Algorithm 2. SLICm Superpixel refinement model
Step 1: Input image $I$and ImSE-Net segmentation result Y;
Step 2: Define the N kinds of semantic classifications $C=\left\{C_1,C_2,\dots ,C_N\right\}$, and the image I Segment result based on SLIC $S=\left\{S_1,S_2,\dots ,S_n\right\}$, and there are $m_i$ pixels in the superpixels $S_i$;
Step 3: Find the semantic subclass of $p_i$ in C from Y, $1\le i\le n,1\le j\le m_i,p_i$ denotes the j-th pixels in $m_i.$ If $p_i$refers to $C_k,$let $M_k=$ $M_k+1,$where $M_k$ means the total number of pixels referring to $C_k.$ Until find the $C_{\max }$ which have the most corresponding pixels in the superpixels $S_i$ and assign $C_{\max }$ to this superpixel.
Step 4: Obtain the final superpixel refinement semantic segmentation result $\tilde{Y}$.

Figure 3. Illustration of the global hidden pseudo-positives and local hidden pseudo-positive selection process. GHPsp: unlabeled image samples $\underset{i}{\overset{\mathrm{\prime }}{x}}$; feature extractor F; task-agnostic reference pool $Q^{\mathrm{ag}}$; anchor features $f_i$; the segmentation head S produces corresponding segmentation features $s_i=S\left(f_i\right)$; an index set of $P_i^{\mathrm{ag}}$ for each i-th anchor feature $f_i$; the momentum segmentation head $S^{\mathrm{\prime }}$ produces corresponding segmentation features$s_i^{\mathrm{\prime }}=S^{\mathrm{\prime }}\left(f_i\right)$; task-specific reference pool $Q^{\mathrm{sg}}$; an index set of $P_i^{\mathrm{sp}}$ by comparing $s_i^{\mathrm{\prime }}$ and $Q^{\mathrm{sg}}$; the projection head Z produces a projection anchor vector $z_i$; GHPsP contrastive loss for i-th patch in unsupervised semantic segmentation with multiple positives $L_{\mathrm{ag}}^{\mathrm{cont}}$ and $L_{\mathrm{sp}}^{\mathrm{cont}}$; LHPsp: an index set $M_i^{\mathrm{nei}}$ that contains the i-th anchor and its neighboring patches; it uses the average attention score value of ${\tilde{T}}_i$ as the threshold for selecting LHPsP $M_i^{\mathrm{loc}}$ among$M_i^{\mathrm{nei}}$ and then takes the above-average portion as the final experiment selection; the calculated gradient (G) for mixed feature $s_i^{\mathrm{mix}}$combines the neighboring positive features $G_i$ with the corresponding attention scores $T_i$ proportionally; the LHPsP objective functions $Z\left(s_i^{\mathrm{mix}}\right)$produces a projected mixed vector $z_i^{\mathrm{mix}}$; the global cost is Lc, and the local loss is L_r.

Table 3. Detailed process for unsupervised semantic segmentation UGLS.

Algorithm 3. Unsupervised semantic segmentation model
Step 1: Input a small sample set $\stackrel{\widehat{{}^{\mathrm{\prime }}}}{X}={\left\{\underset{i}{\overset{\widehat{{}^{\mathrm{\prime }}}}{X}}\right\}}_{i=1}^E$ containing E unlabeled images;
Step 2: Build a global matching set GHPsP; Collect task-agnostic reference pools ${\mathrm{Q}}^{\mathrm{ag}}$ and ${\mathrm{Q}}^{\mathrm{sg}}$ according to Eq. (11) and Eq. (12), respectively; During training, ensure that $Q^{\mathrm{ag}}$ is frozen, while $Q^{\mathrm{sg}}$ and the momentum segmentation head S’ is synchronously updated;
Step 3: Construct the local matching set LHPsP; Calculate the threshold for local hidden positives $M_i^{\mathrm{loc}}$ among$M_i^{\mathrm{nei}}$ according to Eq. (14); Mix neighboring positive features $G_i$ and corresponding attention scores $T_i$ in proportion to obtain the projected mixed vector$z_i^{\mathrm{mix}}$;
Step 4: Compute the loss function and complete all clustering;
Step 5: Output the final segmentation result.

Table 4. Grouping information about datasets.

Grouping	Serial number	Source	Size (pixels)	Resolution (m)
First	I1	COCO-stuff	481 × 321	Unknown
	I2	COCO-stuff	480 × 640	Unknown
	I3	COCO-stuff	480 × 640	Unknown
	I4	COCO-stuff	632 × 491	Unknown
Second	I5	JL-1	513 × 513	0.75
	I6	BJ-2	513 × 513	0.80
	I7	JL-1	513 × 513	0.75
	I8	JL-1	513 × 513	0.75
Third	I9	JL-1	513 × 513	0.75
	I10	BJ-2	513 × 513	0.80
	I11	JL-1	513 × 513	0.75
	I12	JL-1	513 × 513	0.75

Table 5. Test set task categories.

Test set	Categories
COCO-stuff	Person, tree, shadow, building, animal, sky, chair, ground, low veg.
JL-1	Low veg., ground, building, road
BJ-2	Water, tree, low veg., ground, building, greenhouse, road

Table 6. Quantitative evaluation indicators.

Evaluation index
$\mathrm{Precision}={\displaystyle \frac{p_{\mathrm{ii}}}{{\sum }_{j=1}^k{p}_{\mathrm{ji}}},\mathrm{Recall}={\displaystyle \frac{p_{\mathrm{ii}}}{{\sum }_{i=1}^k{p}_{\mathrm{ij}}},{\begin{array}{c}\\ F\end{array}}_1\mathrm{\_socre}=2\times {\displaystyle \frac{\mathrm{Precision}\ast \mathrm{Recall}}{\mathrm{Precision}+\mathrm{Recall}},}}}$	(22)
$\mathrm{OA}={\displaystyle \frac{{\sum }_{i=1}^k{p}_{\mathrm{ii}}}{{\sum }_{i=1}^k{\sum }_{j=1}^k{p}_{\mathrm{ij}}},}$	(23)
$\mathrm{MIoU}={\displaystyle \frac{1}{k}\sum _{i=1}^k{\displaystyle \frac{p_{\mathrm{ii}}}{{\sum }_{j=1}^k{p}_{\mathrm{ij}}+{\sum }_{j=1}^k{p}_{\mathrm{ji}-}{p}_{\mathrm{ii}}},}}$	(24)
$\mathrm{ACC}={\displaystyle \frac{{\sum }_{i=1}^{N}1\left\{y_i=\max \left(k_i\right)\right\}}{N},}$	(25)
$\mathrm{NMI}={\displaystyle \frac{H\left(C\right)+H\left(Y\right)}{H\left(C,Y\right)}}$	(26)

Table 7. Description of the unsupervised semantic segmentation algorithms.

K-means+	SLIC*	AFCF*	SOAWS*	SLSP-Net*	ImSENet + SLICm	PiCIE	STEGO	Ours
Pedregosa et al. 2011	Achanta et al. 2012	Lei et al. 2020	Jia et al. 2020	Yang et al. 2022		Cho et al. 2021	Hamilton et al. 2022

Figure 4. Example results with the COCO-stuff test set.

Figure 5. Example results on the build test set.

Figure 6. Example results on the mixed region test set.

Table 8. Per-class result on the COCO-stuff test set.

Method	F1-score(%)
	Person	Tree	Low veg.	Ground	Shadow	Building	Animal	Sky	Chair
K-means+	53.23	54.36	53.21	65.32	57.36	57.45	55.36	55.36	57.26
SLIC*	51.36	50.36	51.36	65.36	56.56	58.32	54.36	56.36	60.36
AFCF*	62.35	61.36	60.26	73.21	67.36	59.36	58.36	61.36	66.36
SOAWS*	65.58	63.56	61.36	74.36	69.36	66.34	63.21	67.36	68.53
SLSP-Net*	73.26	72.53	68.63	72.36	72.35	65.36	65.45	67.24	69.36
ImSE-Net+SLICm	73.23	73.26	68.75	71.36	70.88	69.35	65.22	68.02	70.29
PiCIE	70.32	70.38	64.32	70.63	68.36	65.78	65.86	65.36	64.23
STEGO	76.72	76.53	68.36	71.25	69.36	66.36	68.36	65.36	70.32
Ours	78.76	79.36	69.36	75.36	76.59	70.56	69.54	70.16	74.33

Table 9. Evaluation scores (%) of different baseline methods based on the COCO-stuff test set.

Method	OA	MIoU	ACC	NMI
K-means+	60.35	20.59	63.69	65.89
SLIC*	60.39	21.77	68.56	67.36
AFCF*	65.32	23.23	74.56	73.26
SOAWS*	70.36	25.12	73.86	74.36
SLSP-Net*	75.53	28.33	76.36	76.31
ImSE-Net+SLICm	75.36	32.56	75.21	75.16
PiCIE	74.18	28.42	68.36	72.25
STEGO	75.36	33.53	75.36	76.36
Ours	77.59	37.23	77.89	78.21

Table 10. Comparison between baseline and state-of-the-art methods for experiment 2. The best values are highlighted in bold.

Method	F1-score				OA	MIoU	ACC	NMI
	Low veg.	Ground	Building	Road
K-means+	58.36	60.36	58.35	57.68	59.87	19.82	59.36	60.36
SLIC*	59.32	61.25	59.36	59.89	57.36	20.56	57.36	66.47
AFCF*	63.36	62.58	67.36	64.21	64.36	22.32	69.56	71.56
SOAWS*	63.83	64.32	65.12	65.39	65.36	24.31	71.69	69.88
SLSP-Net*	71.36	70.58	70.32	70.36	71.56	27.83	72.59	74.36
ImSE-Net+SLICm	71.86	69.89	70.36	71.33	70.98	31.42	74.35	73.26
PiCIE	59.23	61.23	60.12	65.54	66.65	28.33	67.58	68.36
STEGO	72.36	71.36	71.25	68.98	74.83	30.23	74.36	71.36
Ours	75.85	71.36	72.36	72.65	75.38	35.26	75.47	75.58

Table 11. Per-class result for experiment 3.

Method	F1-score
	Water	Tree	Low veg.	Ground	Building	Green-house	Road
K-means+	58.83	57.36	57.32	55.36	53.25	51.99	52.65
SLIC*	61.36	58.36	59.12	60.32	57.56	58.69	60.36
AFCF*	62.53	65.89	67.36	65.32	59.88	73.32	65.99
SOAWS*	65.36	66.36	67.56	73.32	62.56	69.84	63.84
SLSP-Net*	67.59	65.32	68.36	70.36	63.32	71.36	64.25
ImSE-Net+SLICm	68.56	67.23	67.53	69.66	63.22	75.36	63.25
PiCIE	58.36	58.36	60.32	61.24	58.69	58.84	60.36
STEGO	70.18	65.69	68.34	71.36	66.59	68.34	67.68
Ours	73.32	68.36	70.35	72.26	68.26	71.32	70.55

Table 12. Comparison between baseline and state-of-the-art methods for experiment 3.

Method	OA	MIoU	ACC	NMI
K-means+	55.98	19.32	56.99	58.63
SLIC*	55.36	20.19	64.86	59.63
AFCF*	61.36	20.23	65.32	67.36
SOAWS*	64.21	22.98	68.87	65.89
SLSP-Net*	71.36	23.23	67.36	66.79
ImSE-Net+SLICm	70.36	26.63	69.11	66.89
PiCIE	59.32	23.36	65.89	65.36
STEGO	71.21	28.23	69.86	68.32
Ours	71.63	32.53	71.89	70.38

Table 13. Experimental results with various backbone network combinations.

Metric	Im + ResNet-18	Im +ResNet-34	Im +ResNet-50	Im +ResNet-101
OA	62.32	63.22	63.65	63.98
MIoU	27.83	28.19	28.98	29.36
ACC	64.86	65.32	65.77	65.86
NMI	61.63	62.32	62.53	63.01

Table 14. Evaluation results of the hierarchical levels.

	Backbone	Stride convolution layer				OA	MIoU	ACC	NMI
		Conv_2	Conv3_x	Conv4_x	Conv5_x
(a)	ResNet-18				√	58.28	24.22	59.44	57.67
(b)	ResNet-18			√	√	62.32	27.83	64.86	61.63
(c)	ResNet-18		√	√	√	63.55	29.28	65.01	62.88
(d)	ResNet-18	√	√	√	√	60.12	26.36	62.14	60.31
(e)	ResNet-101				√	59.30	25.18	61.33	58.65
(f)	ResNet-101			√	√	63.98	29.36	65.86	63.01
(g)	ResNet-101		√	√	√	62.15	27.58	64.52	61.24
(h)	ResNet-101	√	√	√	√	65.74	31.89	66.93	63.47

Table 15. Influence of main components on semantic segmentation results.

	Method	GHPsP		CA	LHPsP	OA	MIoU	ACC	NMI
		TAs	TSs
(a)	IS+ TAs+ CT	√		√		70.14	32.36	70.44	70.61
(b)	IS+ TAs+ CT+ LHPsP	√		√	√	72.25	33.59	72.66	72.83
(c)	IS+ GHPsP+ CT	√	√	√		73.50	34.22	73.54	73.75
(d)	IS+ GHsP+ LHPsP	√	√		√	71.23	32.63	71.93	71.52
(e)	IS+UGLS (Ours)	√	√	√	√	75.36	35.49	75.44	75.76

Table 16. Influence of different stages on semantic segmentation results.

	Different stages			OA	MIoU	ACC	NMI	Time / (s)
	stage 1	stage 2	Ours
(a)	√			65.74	31.89	66.93	63.47	4526
(b)		√		73.23	33.26	73.86	72.36	5079
(c)			√	75.36	35.49	75.44	75.76	9605

Pedregosa, Fabian, Gael Varoquaux, Alexandre Gramfort, Vincent Michel, and Bertrand Thirion. 2011. “Scikit-Learn: Machine Learning in Python.” Journal of Machine Learning Research 12 (2011): 2825–2830. https://doi.org/10.48550/arXiv.1201.0490.

 Google Scholar

Achanta, Radhakrishna, Appu Shaji, Kevin Smith, Aurelie Lucchi, and Pascal Fua. 2012. “SLIC Superpixels Compared to State-of-the-Art Superpixel Methods.” IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (11): 2274–2281. https://doi.org/10.1109/TPAMI.2012.120.

 PubMed Web of Science ®Google Scholar

Lei, Tao, Peng Liu, Xiaohong Jia, Xuande Zhang, Hongying Meng, and Asoke K. Nandi. 2020. “Automatic Fuzzy Clustering Framework for Image Segmentation.” IEEE Transactions on Fuzzy Systems 28 (9): 2078–2092. https://doi.org/10.1109/TFUZZ.2019.2930030.

 Web of Science ®Google Scholar

Jia, Xiaohong, Tao Lei, Peng Liu, Dinghua Xue, Hongying Meng, and Asoke K. Nandi. 2020. “Fast and Automatic Image Segmentation Using Superpixel-Based Graph Clustering.” IEEE Access 8: 211526–211539. https://doi.org/10.1109/ACCESS.2020.3039742.

 Web of Science ®Google Scholar

Yang, Zenan, Haipeng Niu, Liang Huang, Xiaoxuan Wang, and Liangxin Fan. 2022. “Automatic Segmentation Algorithm for High-Spatial-Resolution Remote Sensing Images Based on Self-Learning Super-Pixel Convolutional Network.” International Journal of Digital Earth 15 (1): 1101–1124. https://doi.org/10.1080/17538947.2022.2083247.

 Web of Science ®Google Scholar

Cho, Jang Hyun, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. 2021. “PiCIE: Unsupervised Semantic Segmentation Using Invariance and Equivariance in Clustering.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16794–16804.

 Google Scholar

Hamilton, Mark, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. 2022. “Unsupervised Semantic Segmentation by Distilling Feature Correspondences.” Proceedings of the International Conference on Learning Representations.

 Google Scholar

Data availability statement

The code used in this study are available by contacting the corresponding author.