Boundary-enhanced dual-stream network for semantic segmentation of high-resolution remote sensing images

Figures & data

Figure 1. Diagram of common semantic segmentation methods combined with edge detection.

Figure 2. Overall architecture of the proposed BEDSN.

Figure 3. Diagram of MSFM.

Table 1. Channel depth of each convolution block.

Layer ID	L1_1	L2_l	L2_2	L3_1	L3_2	L4_1	L4–2
Encoder	1×1,64	1×1,192	3×3,256	1×1,64	5×5,128	1×1,32	7×7,64
Decoder	1×1,64	1×1,256	3×3,128	1×1,64	5×5,32	1×1,32	7×7,32

Figure 4. Diagram of HACM. Coverage maps of local receptive field for the three atrous convolutions with different dilation rates are presented from left to right.

Figure 5. Diagram of MIFM.

Figure 6. Experimental data distribution maps.

Figure 7. Diagram of overlap clipping inference. Only the first horizontal traversal is presented for better visual effects.

Table 2. Results of different methods on Vaihingen dataset.

Methods	Building	Imp-suf	Low-veg	Tree	Car	OA	Kappa	AF	MIoU	Parameters	GFLOPs	Test time
FCN-8s	87.46/77.71	86.13/75.64	75.44/60.57	85.33/74.41	73.53/58.15	83.69	78.36	81.58	69.30	136.83M	57.29	9.13s
UNet	93.88/88.46	91.41/84.18	81.17/68.30	87.91/78.44	84.45/73.08	88.83	85.19	87.76	78.49	31.04M	109.31	15.36s
SegNet	92.77/86.52	90.05/81.90	80.08/66.78	87.32/77.50	69.97/53.81	87.65	83.62	84.04	73.30	29.46M	81.41	11.09s
PSPNet	92.37/85.83	90.44/82.56	81.47/68.73	88.39/79.20	70.35/54.47	88.20	84.34	84.64	74.16	46.71M	46.15	15.35s
DeepLabv3+	93.55/87.87	91.05/83.57	81.40/68.64	88.03/78.62	80.46/67.30	88.71	85.03	86.90	77.20	59.37M	146.70	10.89s
ERN	93.77/88.27	91.02/83.52	81.33/68.54	87.93/78.45	85.34/74.43	88.76	85.09	87.88	78.64	6.63M	57.57	12.49s
SCAttNet	92.14/88.27	89.62/83.52	81.65/69.00	88.30/79.05	74.42/59.26	87.85	83.88	85.23	74.78	29.46M	82.67	11.10s
ResUNet-a	93.02/86.95	89.56/81.09	81.77/69.16	88.00/78.57	77.08/62.70	88.53	84.79	86.21	76.23	48.77M	93.29	16.46s
GAMNet	91.20/83.82	89.56/81.09	78.88/65.13	87.30/77.47	79.01/65.30	86.88	82.61	85.19	74.56	387.4M	245.23	19.83s
LANet	90.67/82.93	89.31/80.68	78.15/64.14	87.09/77.13	80.86/67.87	87.03	82.08	85.22	74.55	23.79M	8.29	3.71s
MDANet	91.27/83.94	89.46/80.92	78.35/64.40	87.26/77.39	80.82/67.82	87.33	82.46	85.43	74.90	31.90M	11.06	8.89s
BEDSN	94.71/89.95	92.30/85.71	83.71/71.99	89.24/80.56	86.31/75.92	90.14	86.93	89.25	80.82	21.27M	83.89	15.53s

For brevity, Imp-surf and Low-veg denote the impervious surface and the low vegetation classes, respectively. Note that evaluations for per class are presented in the form of F1%)/IoU (%), optimal values are presented in bold font, and suboptimal values are presented in blue font. The test time is calculated on the basis of area 10 of Vaihingen dataset.

Figure 8. Results for complete tiles of the Vaihingen dataset: areas 27 (top) and 31(bottom). (a) original, (b) GT, (c) FCN–8s, (d) UNet, (e) SegNet, (f) PSPNet, (g) DeepLabv3+, (h) ERN, (i) SCAttNet, (j) ResUNet-a, (k) GAMNet, (l) LANet, (m) MDANet, (n) proposed BEDSN.

Figure 9. Results for local areas of the Vaihingen dataset. (a) original, (b) GT, (c) FCN–8s, (d) UNet, (e) SegNet, (f) PSPNet, (g) DeepLabv3+, (h) ERN, (i) SCAttNet, (j) ResUNet–a, (k) GAMNet, (l) LANet, (m) MDANet, (n) proposed BEDSN.

Table 3. Results of different methods on Potsdam dataset.

Methods	Building	Imp-suf	Low-veg	Tree	Car	Clutter	OA	Kappa	AF	MIoU
FCN-8s	84.78/73.58	83.28/71.35	78.78/64.99	77.62/63.43	87.38/77.59	27.80/16.14	79.13	72.81	73.27	61.18
UNet	93.82/88.35	90.14/82.05	84.22/72.74	85.65/74.90	93.02/86.95	35.69/21.72	86.91	82.86	80.42	71.12
SegNet	94.93/90.34	91.44/84.22	85.26/74.30	86.82/76.72	94.46/89.50	47.10/30.80	88.45	84.88	83.33	74.32
PSPNet	92.53/86.10	90.10/81.98	83.77/72.07	83.53/71.71	93.43/87.66	40.25/25.20	86.24	81.96	80.60	70.79
DeepLabv3+	94.65/89.85	90.63/82.87	83.76/72.06	83.55/71.43	93.53/87.84	40.26/25.20	87.48	83.51	81.03	71.54
ERN	93.44/87.69	89.82/81.52	84.23/72.76	83.33/75.41	93.53/88.13	34.95/21.18	86.46	82.31	80.32	71.07
SCAttNet	95.26/90.96	91.26/83.93	83.79/72.10	85.12/74.10	92.66/86.32	46.63/30.41	88.00	84.26	82.46	72.97
ResUNet-a	92.86/86.68	89.00/80.19	83.00/70.95	82.28/69.89	91.73/84.72	36.20/22.10	85.84	81.37	79.18	69.09
GAMNet	92.41/85.89	87.11/77.16	79.60/66.12	80.53/67.40	93.04/86.99	30.02/17.66	83.22	78.14	77.12	66.87
LANet	92.68/86.36	88.90/80.02	81.50/68.78	83.64/71.88	94.62/89.79	34.77/21.04	85.37	80.85	79.35	69.65
MDANet	92.86/86.67	89.11/80.35	81.42/68.67	83.86/72.20	94.57/89.69	33.96/20.45	85.55	81.08	79.29	69.67
BEDSN	95.64/91.64	91.83/84.90	85.85/75.20	86.72/76.55	95.03/90.53	49.42/32.82	89.17	85.78	84.08	75.27

Figure 10. Results for complete tiles of the Potsdam dataset: areas 5_13 (top) and 5_14 (bottom). (a) original, (b) GT, (c) FCN–8s, (d) UNet, (e) SegNet, (f) PSPNet, (g) DeepLabv3+, (h) ERN, (i) SCAttNet, (j) ResUNet–a, (k) GAMNet, (l) LANet, (m) MDANet, (n) proposed BEDSN.

Figure 11. Results of local areas on the Potsdam dataset. (a) original, (b) GT, (c) FCN–8s, (d) UNet, (e) SegNet, (f) PSPNet, (g) DeepLabv3+, (h) ERN, (i) SCAttNet, (j) ResUNet–a, (k) GAMNet, (l) LANet, (m) MDANet, (n) proposed BEDSN.

Figure 12. Results of boundary blur on the Potsdam dataset: a part of area 7_13 (a patch identification number of Potsdam dataset). (a) original, (b) GT, (c) edge, (d) dilated edge, (e) FCN–8s, (f) UNet, (g) SegNet, (h) PSPNet, (i) DeepLabv3+, (j) ERN, (k) SCAttNet, (l) ResUNet–a, (m) GAMNet, (n) LANet, (o) MDANet, (p) proposed BEDSN.

Table 4. Results of boundary metrics on Vaihingen dataset with dilated edge labels.

Methods	TP	FP	FN	TN	recall	precision	F1 score
FCN-8s	1376326	4159010	5459587	79205142	20.13%	24.86%	22.25%
UNet	1316870	1991482	5519043	81372670	19.26%	39.80%	25.96%
SegNet	1216539	2583025	5619374	80781127	17.80%	32.02%	22.88%
PSPNet	1075445	2190203	5760468	81173949	15.73%	32.93%	21.29%
DeepLabv3+	1253405	2164946	5582508	81199206	18.34%	36.67%	24.45%
ERN	1407411	2451553	5428502	80912599	20.59%	36.47%	26.32%
SCAttNet	1198432	2373216	5637481	80990936	17.53%	33.55%	23.03%
ResUNet-a	1246352	2347454	5589561	81016698	18.23%	34.68%	23.90%
GAMNet	1307215	2501804	5528698	80862348	19.12%	34.32%	24.56%
LANet	1283609	2229294	5552304	81134858	18.78%	36.54%	24.81%
MDANet	1280607	2142706	5555306	81221446	18.73%	37.41%	24.96%
BEDSN	1375792	1934438	5460121	81429714	20.13%	41.56%	27.12%

Table 5. Results of boundary metrics on Potsdam dataset with dilated edge labels.

Methods	TP	FP	FN	TN	recall	precision	F1 score
FCN-8s	5224168	28939699	19742606	450093527	20.92%	15.29%	17.67%
UNet	4599604	10052678	20367170	468980548	18.42%	31.39%	23.22%
SegNet	4918198	11396487	20048576	467636739	19.70%	30.15%	23.83%
PSPNet	3671640	9467582	21295134	469565644	14.71%	27.94%	19.27%
DeepLabv3+	3904419	8563245	21062355	470469981	15.64%	31.32%	20.86%
ERN	5031544	12841064	19935230	466192162	20.15%	28.15%	23.49%
SCAttNet	4881454	18016908	20085320	461016318	19.55%	21.32%	20.40%
ResUNet-a	4744242	18556613	20222532	460476613	19.00%	20.36%	19.66%
GAMNet	4926969	15746290	20039805	463286936	19.73%	23.83%	21.59%
LANet	4353693	9980000	20613081	469053226	17.44%	30.37%	22.16%
MDANet	4356780	9321809	20609994	469711417	17.45%	31.85%	22.55%
BEDSN	4759754	8799015	20207020	470234211	19.06%	35.10%	24.71%

Table 6. Results of ablation experiments on Vaihingen dataset.

Methods	A	B	C	D	E	F	Building	Imp-suf	Low-veg	Tree	Car	OA	Kappa	AF	MIoU	Parameters	GFLOPs
UNet							93.88/88.46	91.41/84.18	81.17/68.30	87.91/78.44	84.45/73.08	88.83	85.19	87.76	78.49	31.04M	109.31
Net1	√						94.00/88.68	91.63/84.55	81.54/68.83	88.02/78.60	85.79/75.10	89.05	85.49	88.19	79.15	46.39M	133.47
Net2	√	√					94.26/89.14	91.83/84.90	82.51/70.23	88.52/79.41	86.11/75.60	89.48	86.06	88.65	79.86	46.44M	133.97
Net3	√	√	√				94.25/89.11	92.04/85.26	82.40/70.07	88.61/79.54	86.93/76.89	89.55	86.16	88.85	80.18	27.56M	112.41
Net4	√	√	√	√			94.62/89.80	92.01/85.36	82.47/70.17	88.59/79.52	87.07/77.10	89.71	86.36	88.97	80.39	30.94M	104.35
Net5	√	√	√	√	√		94.61/89.77	91.95/85.10	83.61/71.84	89.04/80.24	87.13/77.19	89.88	86.58	89.27	80.83	31.29M	104.40
BEDSN	√	√	√	√	√	√	94.71/89.95	92.30/85.71	83.71/71.99	89.24/80.56	86.31/75.92	90.14	86.93	89.25	80.82	21.27M	83.89

Note: “A,” “B,” and “C” denote depth dimension expansion, edge detection branch stream, and hybrid atrous block, respectively; while “D,” “E,” and “F” represent deconvolution, MIFM, and MSFM, respectively. Prominent positive and negative effects are presented in green and red fonts, respectively. Note that evaluations for per class are presented in the form of F1%)/IoU (%).

Figure 13. Results for the improved effects of SSBS and EDBS on Vaihingen (top) and Potsdam (below) datasets. (a) Image, (b) GT, (c) UNet, (d) SSBS, (e) BEDSN-P, (f) BEDSN, (g) Boundary results of BEDSN.

Table 7. Experiments on arrangement of SSBS, EDBS and efficiency.

Vaihingen Dataset

Methods

Building

Imp-suf

Low-veg

Tree

Car

Clutter

OA

Kappa

AF

MIoU

Parameters

GFLOPs

UNet

93.88/88.46

91.41/84.18

81.17/68.30

87.91/78.44

84.45/73.08

–

88.83

85.19

87.76

78.49

31.04M

109.31

PSPNet

92.37/85.83

90.44/82.56

81.47/68.73

88.39/79.20

70.35/54.47

–

88.2

84.34

84.64

74.16

46.71M

46.15

ERN

93.77/88.27

91.02/83.52

81.33/68.54

87.93/78.45

85.34/74.43

–

88.76

85.09

87.88

78.64

6.63M

57.57

LANet

90.67/82.93

89.31/80.68

78.15/64.14

87.09/77.13

80.86/67.87

–

87.03

82.08

85.22

74.55

23.79M

8.29

MDANet

91.27/83.94

89.46/80.92

78.35/64.40

87.26/77.39

80.82/67.82

–

87.33

82.46

85.43

74.90

31.90M

11.06

BEDSN

94.71/89.95

92.30/85.71

83.71/71.99

89.24/80.56

86.31/75.92

–

90.14

86.93

89.25

80.82

21.27M

83.89

SSBS

94.39/89.37

91.77/84.80

83.30/71.38

88.78/79.82

85.97/75.39

–

89.62

86.23

88.84

80.15

21.24M

83.41

BEDSN-P

94.28/89.18

91.77/84.80

83.07/71.04

88.91/80.02

86.41/76.97

–

89.63

86.24

89.00

80.40

32.79M

113.81

LBEDSN

94.45/89.47

91.72/84.71

83.36/71.46

88.99/80.16

85.12/74.10

–

89.67

86.31

88.73

79.98

5.33M

21.21

Potsdam Dataset

Methods

Building

Imp-suf

Low-veg

Tree

Car

Clutter

OA

Kappa

AF

MIoU

Parameters

GFLOPs

UNet

93.82/88.35

90.14/82.05

84.22/72.74

85.65/74.90

93.02/86.95

35.69/21.72

86.91

82.86

80.42

71.12

31.04M

109.31

PSPNet

92.53/86.10

90.10/81.98

83.77/72.07

83.53/71.71

93.43/87.66

40.25/25.2o

86.24

81.96

80.60

70.79

46.71M

46.15

ERN

93.44/87.69

89.82/81.52

84.23/72.76

83.33/75.41

93.53/88.13

34.95/21.18

86.46

82.31

80.32

71.07

6.63M

57.57

LANet

92.68/86.36

88.90/80.02

81.50/68.78

83.64/71.88

94.62/89.79

34.77/21.04

85.37

80.85

79.35

69.65

23.79M

8.29

MDANet

92.86/86.67

89.11/80.35

81.42/68.67

83.86/72.20

94.57/89.69

33.96/20.45

85.55

81.08

79.29

69.67

31.90M

11.06

BEDSN

95.64/91.64

91.83/84.90

85.85/75.20

86.72/76.55

95.03/90.53

49.42/32.82

89.17

85.78

84.08

75.27

21.27M

83.89

SSBS

95.24/90.91

91.05/83.56

85.38/74.49

86.13/75.63

94.43/89.45

42.31/26.83

88.49

84.85

82.42

73.48

21.24M

83.41

BEDSN-P

94.99/90.45

90.86/83.26

85.11/74.08

86.39/76.05

94.26/89.15

42.24/26.77

89.17

84.60

82.31

73.29

32.79M

113.81

LBEDSN

94.94/90.36

91.09/83.64

84.91/73.78

86.16/75.69

93.45/87.71

43.43/27.75

88.17

84.47

82.33

73.16

5.33M

21.21

Figure 14. Semantic segmentation results of different λ and loss functions. Optimal indicators are marked with red triangles.

Figure 15. Results of the joint loss function and Full–CE. (a) Image, (b) GT, (c)–(d) Segmentation and boundary predictions of the joint loss function (λ = 0.5), (e)–(f) Segmentation and boundary predictions of Full–CE.

Table 8. Experiments on different batchsizes with learning rate of 3 × 10⁻⁴.

Batchsize	Building	Imp-suf	Low-veg	Tree	Car	OA	Kappa	AF	MIoU
2	–/–	–/–	–/–	–/–	–/–	–	–	–	–
4	94.67/89.88	91.82/84.88	83.32/71.41	89.14/80.41	86.72/76.55	90.47	86.55	89.13	80.63
8	94.70/89.93	91.94/85.08	83.93/72.31	89.17/80.45	87.89/78.39	90.59	86.75	89.52	81.23
12	94.75/90.02	92.18/85.49	83.59/71.80	89.10/80.34	87.08/77.12	90.59	86.72	89.34	80.95

Table 9. Experiments on different learning rates with the batchsize of 8.

Learning rate	Building	Imp-suf	Low-veg	Tree	Car	OA	Kappa	AF	MIoU
0.3	92.97/86.86	89.93/81.70	79.20/65.57	87.75/78.17	78.94/65.20	88.22	83.73	85.76	75.50
0.03	94.62/89.79	92.36/85.80	81.57/68.88	88.11/78.74	86.63/76.42	89.90	86.06	88.66	79.93
0.003	94.62/89.79	92.14/85.42	82.79/70.63	88.74/79.76	86.19/75.73	90.19	86.54	88.90	80.27
0.0003	94.70/89.93	91.94/85.08	83.93/72.31	89.17/80.45	87.89/78.39	90.59	86.75	89.52	81.23
0.00003	93.51/87.81	90.62/82.86	81.54/68.84	88.10/78.74	82.75/70.57	89.17	84.80	87.30	77.76

Data availability statement

The data that support the findings of this study are openly available in ISPRS at https://www.isprs.org/education/benchmarks/UrbanSemLab/Default.aspx. Other data that support the findings of this study including codes and results are available from the first author, [Xinghua Li, [email protected]], upon reasonable request.