Dual-det : a fast detector for oriented object detection in aerial images

Figures & data

Figure 1. A pipeline of the proposed method has been shown here. A group of feature maps with $\frac{1}{4}$ the size of the input is obtained after the feature extractor, and the light attention learner is used to highlight the foreground and finally output five sets of heatmaps. This method is called DASR. The final result can be decoded after the SSG loss to correct the angle and diagonal length based on the short side of the object

Figure 2. The network architecture of Dual-Det. A deformable convolution(DCN) is applied between downsampling and upsampling, which is shown as the Orange feature map in the figure. The light yellow heatmap is the output involved in the loss calculation. Conv Block is a small convolution module with detailed structure in .Section 3.4

Figure 3. Illustration of the proposed regression strategy for an oriented object. We utilize $(x,y,s,r,{\theta }_1,{\theta }_2)$ to represent the box $P$

Figure 4. Comparison of the errors between dual-angle prediction and single-angle prediction. The red box $P$ is the ground truth, $P^{\prime }$ is the error box and the green area is the intersection of the two boxes. See Section 3.2 for details

Table 1. Different results for short-side-based, long-side-based and the baseline

method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbour	SP	HC	mAP
Short-based	90.30	67.99	34.18	57.61	47.40	78.46	84.48	90.84	61.41	81.56	58.99	72.76	56.99	65.06	13.41	64.10
Long-based	89.56	71.99	24.56	47.63	46.04	70.33	69.34	90.74	62.30	77.86	58.66	60.21	52.01	65.46	28.75	61.03
Baseline	90.10	74.24	26.89	56.26	51.14	74.98	78.90	90.86	63.06	78.65	64.70	69.37	55.92	68.93	35.58	65.30

Figure 5. The detailed structure of light attention learning module

Table 2. Ablation studies of each component

Baseline	DASR	SSG	Light Attention Learner	mAP
✓	–	–	–	65.30
✓	✓	–	–	67.91
✓	✓	✓	–	68.07
✓	✓	✓	✓	69.32

Table 3. Ablation studies of each component (the details of each category are given in Table 2). LAL is the acronym for the $LightAttentionLearner$

method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbour	SP	HC	mAP
Baseline	90.10	74.24	26.89	56.26	51.14	74.98	78.90	90.86	63.06	78.65	64.70	69.37	55.92	68.93	35.58	65.30
+ DASR	90.26	72.31	33.36	54.52	58.53	80.11	79.64	90.77	60.66	79.13	68.86	70.17	65.91	66.33	48.13	67.91
+ SSG	90.20	73.53	32.05	56.44	57.43	79.84	79.41	90.83	65.43	78.88	66.14	70.59	64.74	65.09	50.39	68.07
+ LAL	90.13	74.57	38.21	55.69	61.99	79.65	87.10	90.64	64.39	79.78	67.73	69.04	64.48	67.51	49.94	69.39

Figure 6. Comparison of using the DASR (right) and baseline (left) in the harbour and large vehicle categories

Table 4. Comparison of different settings for the SSG loss

Settings	with label $d$	with label $s$	without label
mAP	64.36	66.52	68.07

Table 5. Comparison of the speed and memory of each component

method	train speed	test speed	parameters
Baseline	0.378s	0.017s	173.7MB
+ DASR	0.467s	0.017s	174.2MB
+ SSG	0.472s	0.017s	174.2MB
+ LAL	0.498s	0.018s	174.6MB

Figure 7. Visualization of detection results from Dual-Det in DOTA

Table 6. Comparison with state-of-the-art detectors on DOTA

method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbour	SP	HC	mAP
Two-stage
FR-O	79.42	77.13	17.7	64.05	35.3	38.02	37.16	89.41	69.64	59.28	50.3	52.91	47.89	47.4	46.3	54.13
RRPN	80.94	65.75	35.34	67.44	59.92	50.91	55.81	90.67	66.92	72.39	55.06	52.23	55.14	53.35	48.22	61.01
R2CNN	88.52	71.2	31.66	59.3	51.85	56.19	57.25	90.81	72.84	67.38	56.69	52.84	53.08	51.94	53.58	60.67
R-DFPN	80.92	65.82	33.77	58.94	55.77	50.94	54.78	90.33	66.34	68.66	48.73	51.76	55.1	51.32	35.88	57.94
ICN	81.36	74.3	47.7	70.32	64.89	67.82	69.98	90.76	79.06	78.2	53.64	62.9	67.02	64.17	50.23	68.16
RoITransformer	88.64	78.52	43.44	75.92	68.81	73.68	83.59	90.74	77.27	81.46	58.39	53.54	62.83	58.93	47.67	69.56
SCRDet	89.98	80.65	52.09	68.36	68.36	60.32	72.41	90.85	87.94	86.86	65.02	66.68	66.25	68.24	65.21	72.61
One-stage
YOLOv2	39.57	20.29	36.58	23.42	8.85	2.09	4.82	44.34	38.35	34.65	16.02	37.62	47.23	25.5	7.45	21.39
IENet	80.20	64.54	39.82	32.07	49.71	65.01	52.58	81.45	44.66	78.51	46.54	56.73	64.40	64.24	36.75	57.14
$R^3$Det-Res101	89.54	81.99	48.46	62.52	70.48	74.29	77.54	90.80	81.39	83.54	61.97	59.82	65.44	67.46	60.05	71.69
$O^2$-DNet	89.31	82.14	47.33	61.21	71.32	74.03	78.62	90.76	82.23	81.36	60.93	60.17	58.21	66.98	61.03	71.04
Dual-Det	89.80	83.32	37.83	65.01	73.78	75.44	79.13	90.83	79.70	84.78	56.63	61.76	66.19	72.87	62.97	72.00
DualDet-FT	89.91	85.33	44.25	63.68	78.65	76.00	86.48	90.86	80.04	85.79	56.41	62.63	67.22	74.30	62.76	73.62

Table 7. Comparisons of the accuracy and speed with the state-of-the-art methods on HRSC2016. * means that the VOC12 AP metric is used

method	Backbone	Input Size	mAP	FPS
R2CNN (Jiang et al. 2017)	VGG16	800$\times$800	73.07	2
RRPN (Ma et al. 2018)	ResNet101	800$\times$800	79.08	3.5
$R^2$PN(Zhang et al. 2018)	VGG16	–	79.6	–
RRD (Liao et al. 2018)	VGG16	384$\times$384	84.3	–
RoITransformer(Ding et al. 2019)	ResNet101	512$\times$800	86.2	6
$R^3$Det(Yang et al. 2019a)	ResNet101	800$\times$800	89.26	12
CenterMap-Net*(Wang et al. 2021)	ResNet50	–	92.8	–
Dual-Det	Resdcn18	800$\times$800	90.04	62.5
Dual-Det*	Resdcn18	800$\times$800	95.15	62.5
Dual-Det	Resdcn18	1024$\times$1024	90.23	46
Dual-Det*	Resdcn18	1024$\times$1024	96.4	46

Table 8. Comparison with the state-of-the-art detectors on UCAS-AOD (Zhu et al. 2015). * means that the VOC12 AP metric is used

method	plane	car	mAP
RRPN (Ma et al. 2018)	88.04	74.36	81.20
R2CNN (Jiang et al. 2017)	89.76	78.89	84.32
R-DFPN (Yang et al. 2018b)	88.91	81.27	85.09
$O^2$-DNet (Wei et al. 2020)	93.21	86.72	89.96
Dual-Det	95.64	86.02	90.83
Dual-Det*	97.34	89.70	93.52

Table

BC	basketball-court
BD	baseball-diamond
DASR	dual-angle with short-side and ratio
DCN	deformable convolution
FPS	Frames Per Second
GTF	ground-track-field
HBB	Horizontal Bounding Box
HC	helicopter
IoU	Intersection over Union
LV	large-vehicle
mAP	mean Average Precision
NMS	Non-Maximum Suppression
OBB	Oriented Bounding Box
RA	roundabout
RoI	regions of interest
RPN	region proposal network
SBF	soccer-ball-field
SSG	short-side guided
SV	small-vehicle
ST	storage-tank
SP	swimming-pool
TC	tennis-court

Jiang, Y., X. Zhu, X. Wang, S. Yang, L. Wei, H. Wang, F. Pei, and Z. Luo. 2017. “R2cnn: Rotational Region Cnn for Orientation Robust Scene Text Detection.” In arXiv Preprint arXiv:1706.09579.

 Google Scholar

Ma, J., W. Shao, H. Ye, L. Wang, H. Wang, Y. Zheng, and X. Xue. 2018. “Arbitrary-oriented Scene Text Detection via Rotation Proposals.” IEEE Transactions on Multimedia 20 (11): 3111–3122. doi:https://doi.org/10.1109/TMM.2018.2818020.

 Web of Science ®Google Scholar

Zhang, Z., W. Guo, S. Zhu, and W. Yu. 2018. “Toward Arbitrary-oriented Ship Detection with Rotated Region Proposal and Discrimination Networks.” IEEE Geoscience and Remote Sensing Letters 15 (11): 1745–1749. doi:https://doi.org/10.1109/LGRS.2018.2856921.

 Web of Science ®Google Scholar

Liao, M., Z. Zhu, B. Shi, G.-S. Xia, and X. Bai. 2018. “Rotation-Sensitive Regression for Oriented Scene Text Detection.” In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 5909–5918.

 Google Scholar

Ding, J., N. Xue, Y. Long, G.-S. Xia, and L. Qikai 2019. “Learning RoI Transformer for Oriented Object Detection in Aerial Images.” In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2844–2853.

 Google Scholar

Yang, X., Q. Liu, J. Yan, and L. Ang. 2019a. “R3DET: Refined Single-stage Detector with Feature Refinement for Rotating Object.” In arXiv Preprint arXiv:1908.05612.

 Google Scholar

Wang, J., W. Yang, L. Heng-Chao, H. Zhang, and G.-S. Xia. 2021. “Learning Center Probability Map for Detecting Objects in Aerial Images.” IEEE Transactions on Geoscience and Remote Sensing 59 (5): 4307–4323. doi:https://doi.org/10.1109/TGRS.2020.3010051.

 Web of Science ®Google Scholar

Zhu, H., X. Chen, W. Dai, F. Kun, Y. Qixiang, and J. Jiao. 2015. “Orientation Robust Object Detection in Aerial Images Using Deep Convolutional Neural Network.” In 2015 IEEE International Conference on Image Processing (ICIP), Quebec City, QC, Canada, 3735–3739.

 Google Scholar

Yang, X., H. Sun, K. Fu, J. Yang, X. Sun, M. Yan, and Z. Guo. 2018b. “Automatic Ship Detection in Remote Sensing Images from Google Earth of Complex Scenes Based on Multiscale Rotation Dense Feature Pyramid Networks.” Remote Sensing 10 (1): 132. doi:https://doi.org/10.3390/rs10010132.

 Web of Science ®Google Scholar

Wei, H., Y. Zhang, Z. Chang, L. Hao, H. Wang, and X. Sun. 2020. “Oriented Objects as Pairs of Middle Lines.” ISPRS Journal of Photogrammetry and Remote Sensing 169: 268–279. doi:https://doi.org/10.1016/j.isprsjprs.2020.09.022.

 Web of Science ®Google Scholar