Dual-det : a fast detector for oriented object detection in aerial images

ABSTRACT

Fast and accurate object detection in aerial images remains a challenging task. Usually, to better describe an object, oriented bounding boxes (OBBs) can better fit objects. Due to high background complexity and large object scale variation, single-angle anchor-based two-stage detectors are widely adopted, which offer better accuracy. However, the single-angle prediction has a small error tolerance for objects with a large aspect ratio, and the hyperparameters of the anchor-based network are difficult to adjust, and the number of hyperparameters is extremely large. Furthermore, the two-stage detection inference speed is slow, and it is difficult to achieve real-time detection. In this paper, we propose Dual-Det, a keypoint-based oriented object detector. We firstly propose a dual-angle with a short-side and ratio regression strategy (DASR), which uses the object centre and the length and angles of two diagonals to represent an object. A short side guided (SSG) loss is further added to guide the direction of the diagonal regression box. To improve the detection performance for dense and tiny objects, a lightweight supervised pixel attention learner is finally proposed. The experiment results show that Dual-Det achieves 90.23$\mathrm{\%}$ mAP at 46FPS on HRSC2016, 90.83$\mathrm{\%}$ mAP at 46FPS on UCAS-AOD and 72.00$\mathrm{\%}$ mAP at 0.018 s per image in the inference phase on DOTA. The code will be open source on https://github.com/gqy4166000/ijrs_dasr.

1. Introduction

With the development of remote sensing technology in recent years, the public remote sensing data are available for open access. However, most of these datasets are labelled similarly to natural image datasets, which use horizontal bounding boxes (HBBs) to depict objects (Wu et al. 2019; Wu et al. 2020). Due to the characteristics of the object distribution in remote sensing images, this kind of label method and regression method easily generates too large of an overlap between two adjacent objects. Because Non-Maximum Suppression (NMS) will suppress each object and lead to serious numbers of missed detections. Besides, HBBs will mix a large amount of background in the detection area, which is not conducive to the improvement of the detection quality. The labelling method of oriented bounding boxes (OBBs) can overcome these problems. OBBs introduce direction information to make labels closer to real objects. In the case of dense objects, it reduces the overlap between boxes. Therefore, oriented object detection is more necessary in the field of remote sensing object detection.

Figure 1. A pipeline of the proposed method has been shown here. A group of feature maps with $\frac{1}{4}$ the size of the input is obtained after the feature extractor, and the light attention learner is used to highlight the foreground and finally output five sets of heatmaps. This method is called DASR. The final result can be decoded after the SSG loss to correct the angle and diagonal length based on the short side of the object

For the choice of basic framework, most of the currently successful detectors are based on the RCNN framework(Cheng, Zhou, and Han 2016; Deng et al. 2017; Long et al. 2017). This framework use a region proposal network (RPN) to regress a large number of candidate regions based on anchors and then perform detailed classification and further regression correction of each region. However, due to the introduction of orientation information, the number of anchors required increases exponentially, so this kind of framework is undoubtedly cumbersome and inefficient. Therefore, the emerging anchor-free framework, which is comparable with RCNN framework in accuracy, is more suitable for remote sensing oriented object detection. For the choice of how to accurately locate the target direction, part of the work regards the problem as a key point regression problem. They locate objects by directly predicting the coordinates of endpoints, such as ($e_1$, $e_2$, $e_3$, $e_4$) or ($e_1$, $e_2$, $h$)(Jiang et al. 2017), where $e_i=\{(x_i,y_i)|i=1,2,3,4or1,2\}$. To be specific, there are two solutions adopting this method. One is to use a fully connected structurethe to output the coordinates of each region proposal in the two-stage network (Liao et al. 2018; Xia et al. 2018; Yang et al. 2019a). The advantage is that a region corresponds to a set of coordinates. However, how to arrange and combine the coordinates is not considered. The calculated loss will fluctuate greatly, despite the target is the same. The other solution is applied to the point-based network(Zhou, Zhuo, and Philipp 2019). It predicts all endpoints by outputting a binary heatmap and combining the endpoints belonging to the same object through a certain logic. But different objects have different difficulties in predicting the endpoints, it is difficult to find the appropriate logic. Thus, the $\theta$-based method is proposed. This method constructs oriented bounding boxes by adding angle parameter $\theta$(Ding et al. 2019; Liu et al. 2017; Ma et al. 2018), and its form is ($x$, $y$, $w$, $h$, $\theta$); However, the predicted angle is very unstable due to the periodicity. Another part of the work is inspired by the segmentation field and uses binary segmentation maps to locate targets. However, due to the large number of dense small objects in remote sensing images, determining the boundaries between objects is still a challenging problem.

In this paper, we propose a fast and efficient point-based oriented object detector called Dual-Det. To be more specific, we introduce a dual-angle with short-side and ratio regression strategy (DASR), which can represented oriented objects using ($x$, $y$, $s$, $r$, ${\theta }_1$, ${\theta }_2$) instead of using ($x$, $y$, $w$, $h$, $\theta$). This strategy improves the performance of the regression with rare time costs, especially objects such as harbours, vehicles, etc., which have a large aspect ratio. Besides, we design a short-side guided loss (SSG) to further guide the regression direction of the diagonal regression boxes. Finally, a light attention learner is designed to reduce the false detection of small and dense objects. We evaluate and test the performance of Dual-Det on three public datasets, (Xia et al. 2018), HRSC2016 (Liu et al. 2016b) and UCAS-AOD (Zhu et al. 2015), and compare its performance with those of the state-of-the-art detectors. Dual-Det achieves excellent detection performance under the premise of rapid detection. In summary, our contribution has three points:

• We present a novel oriented objects representation strategy called DASR. This strategy improve the performance of the regression with rare time cost, especially objects with large aspect ratios.

• We propose a novel loss (SSG loss) to guide the DASR. The loss deepens the coupling between the two angles and the diagonal length during the iteration through the short-side, and fixes some extreme cases of DASR.

• We propose a lights attention learner to improve the performance on small and dense objects with rare time cost, and achieve competitive results with a fast speed.

2. Related work

2.1. Overview

The emergence of deep learning technology makes the application field of remote sensing image develop rapidly. For instances, Hong et al. (2021b) proposes a general multimodal deep learning framework to provide a baseline solution for pixel-level RS image classification tasks using multimodal data. miniGCN(Hong et al. 2021a) greatly reduces the computational complexity of traditional graph convolutional network through ingenious design, and proposes the trainable minibatch strategy to jointly use graph convolutional networks and convolutional neural networks. And in the object detection field, the emergence of R-CNNs(Girshick et al. 2014; Girshick 2015; Lin et al. 2017a; Dai et al. 2016) has greatly improved the overall object detection accuracy. Compared with deep learning technology, the method (such as HOG(Dalal and Triggs 2005)) of manually selecting features and then classifying them by using classifiers is clumsy and inaccurate. At present, most object (Ding et al. 2019; Azimi et al. 2019; Wu et al. 2019; Yang et al. 2018, 2018a; Liu and Mattyus 2015) in the remote sensing field is also based on the R-CNN framework, especially the representative Faster-RCNN (Ren et al. 2017) detector.

2.2. Anchor-based oriented object detectors

Some early oriented object detectors draw on the idea of Faster-RCNN, change the regression method of the second step, and obtain rotation information based on the HRPs (Jiang et al. 2017). However, this strategy may result in a large amount of background being included, that is, a misalignment problem is generated. Many works(Liu et al. 2017; Ma et al. 2018) try to solve this problem by predicting the rotation region proposals, and letting the two stages learn the direction information at the same time. Rotation region proposal networks(Ma et al. 2018) and other methods have achieved promising detection results by using a Rotation Region-of-Interest pooling layer which is based on rotated anchors, but this method also causes a dramatic increase in the number of anchors. With an input size of 1024 $\times$ 1024, the total number of anchors will exceed 1 million, which greatly increases the calculation parameters and hyperparameter selection complexity. The RoI-Transformer(Ding et al. 2019) avoids the problem of the explosion in the number of anchors by adding the RRoI learner module on the basis of HRPs. SCRDet (Yang et al. 2019b) uses a multi-dimensional attention network to suppress the noise impact. However, these two-stage networks based on the RCNN framework all have high computational complexity, especially in remote sensing datasets, where the computational costs are much higher than that of natural images. Among the images obtained by cropping the DOTA dataset with a subsize of 1024 $\times$ 1024, the number of objects in a single picture may reach 1,000 $\sim$ 2,000. Therefore, thousands of RoIs will be input into the second stage to ensure that missed detection does not occur. TextBoxes++(Liao, Shi, and Bai 2018) based on the SSD(Liu et al. 2016a) network improvement directly regresses the 8 vertices of an oriented bounding box. The RRD (Liao et al., 2018) further improves the performance of SSD by using oriented response networks to extract rotation-sensitive text features and then adding maximum pooling to the classification branch to extract features that are not sensitive to classification. However, the accuracy of the one-stage detector is generally inferior to that of the two-stage network.

2.3. Anchor-free oriented object detectors

The early exploration of anchor-free networks is YOLO(Redmon et al. 2016). However, in addition to YOLOv1, both YOLOv2 (Redmon and Farhadi 2017) and v3 (Redmon and Farhadi 2018) use anchor to improve accuracy. FCOS(Tian et al. 2019) propose a one-stage full convolution detector with anchor-free and proposal-free, which can be comparable to the two-stage detector in the accuracy. Inspired by the algorithm, Both AF-RPN(Zhong, Sun, and Huo 2019) and APE (Zhou, Wu, and Du 2019) utilize anchor-free RPN networks, which use shrunk segmentation labels instead of anchors. Among these methods, APE has achieved top-level detection results on the ODAI dataset, but due to the use of the Cascade-RCNN(Cai and Vasconcelos 2018) structure, it still needs a large time cost.

Figure 2. The network architecture of Dual-Det. A deformable convolution(DCN) is applied between downsampling and upsampling, which is shown as the Orange feature map in the figure. The light yellow heatmap is the output involved in the loss calculation. Conv Block is a small convolution module with detailed structure in .Section 3.4

2.4. Keypoint-based oriented object detectors

The emergence of CornerNet(Law and Deng 2018) opens the door for detecting objects based on key points. CornerNet uses corner pooling to detect a diagonal point outside an object to represent an object, but this difficult to apply in oriented object detection. ExtremeNet(Zhou, Zhuo, and Philipp 2019) can detect irregular objects by detecting the four extreme points, but computational load is even much larger than that of the normal two-stage network. CenterNet (Zhou, Wang, and Philipp 2019) with high real-time performance simultaneously treats an object as a centre point and directly regresses the length and width. $O^2$-DET (Wei et al. 2019) treats the rotating target as pairs of middle lines and predicts the intersection point. This paper considers the above aspects and proposes a novel method for oriented object detection in aerial images.

3. Proposed method

3.1. Overview

We first present the proposed keypoint-based method framework as shown in Figure 1. Unlike traditional object detection tasks, we use the location of the centre point to determine the distribution of objects. For a $w\times h$ input image, we predict the heatmap $\hat{K}\in [0,1{]}^{\frac{w}{T}\times \frac{h}{T}\times C}$ to predict the centre point, where $C$ is the number of object categories, and $T$ is the downsampling factor. For the centre point $m$ of each object, the position corresponding to the output heatmap is $\hat{m}=⌊\frac{m}{T}⌋$, and the discretization error can be computed as $O=\frac{m}{T}-\hat{m}$. Heatmap ${\hat{O}}^{\frac{w}{T}\times \frac{h}{T}\times 2}$ is used to compensate for the discretization error caused by downsampling, and the factor $T=4$, which is the same that in as CenterNet. The label setting and prediction method of the centre point followed those of CenterNet. The difference is that we use heatmap $\hat{A}\in [0,\pi {]}^{\frac{w}{T}\times \frac{h}{T}\times 2}$ to predict the angles of two diagonal lines. Heatmap ${\hat{S}}^{\frac{w}{T}\times \frac{h}{T}\times 1}$ is responsible for predicting the length of the short side, and heatmap $\hat{R}\in [0,1{]}^{\frac{w}{T}\times \frac{h}{T}\times 1}$ regresses the ratio of the short side and diagonal lines. The detector outputs a total of $(C+2+2+1+1)\times H^{\frac{w}{T}\times \frac{h}{T}}$ heatmaps. The architecture is shown in Figure 2. We choose ResNet18 (He et al. 2016) as the backbone and use the deformable convolution (DCN) (Dai et al. 2017) to improve the elasticity of the receptive field of the shallow network.

Figure 3. Illustration of the proposed regression strategy for an oriented object. We utilize $(x,y,s,r,{\theta }_1,{\theta }_2)$ to represent the box $P$

3.2. DASR regression strategy

Our proposed method relies on two diagonal angles of the oriented object, as shown in Figure 3. We find that when using a single-angle prediction to describe the direction, especially an object with a large aspect ratio such as a bridge or harbour, if the prediction of the other parameters is completely accurate, a small error $\theta$ in the angle still greatly reduces the Intersection over Union (IoU) between the prediction box and ground truth box as shown in Figure 4(a). The use of a dual-angle prediction can be divided into four cases as shown in Figure 4(b) $\sim$(e), where the red rectangle $P$ is the ground truth and the blue rectangle $P^{\prime }$ is the prediction box. $(l_1,l_2)$ and $(l_1^{\prime },l_2^{\prime })$ are the diagonals of $P$ and $P^{\prime }$, respectively. Case (b): $l_1^{\prime }$ deviates from $l_1$ clockwise by $\theta$ degrees, and the other angle is accurate. Case (c): $l_1^{\prime }$ deviates from $l_1$ clockwise by $\theta$ degrees, and $l_2^{\prime }$ deviates from $l_2$ counterclockwise by $\theta$ degrees. Case (d): $l_1^{\prime }$ deviates from $l_1$ counterclockwise by $\theta$ degrees, and the other angle is accurate. Case (e): $l_1^{\prime }$ deviates from the angle of $l_1$ counterclockwise by $\theta$ degrees, and $l_2^{\prime }$ deviates from the angle of $l_2$ clockwise by $\theta$ degrees. If $l_1^{\prime }$ deviates from the angle of $l_1$ counterclockwise by $\theta$ degrees, and $l_2^{\prime }$ deviates from $l_2$ counterclockwise by $\theta$ degrees, it is the same as figure (a); therefore, it is not listed separately. In Cases of (b) and (c), there can still be a great intersection (green area) between $P$ and $P^{\prime }$ which is higher than that of Case (a); and Cases (d) and (e) are even worse than that of (a). However, because of the prediction of the double angles to represent, the angle losses of Case (c) and Case (e) will be double the calculation losses, which can be easily corrected during the iteration. To solve Case (d), we propose the SSG loss to compensate in Section 3.3.

Figure 4. Comparison of the errors between dual-angle prediction and single-angle prediction. The red box $P$ is the ground truth, $P^{\prime }$ is the error box and the green area is the intersection of the two boxes. See Section 3.2 for details

An intuitive illustration of the proposed representation is depicted in Figure 3. For a given oriented object $P$ (orange dotted yellow rectangle), the angles between the two diagonal lines (blue rays) and the X coordinate axis are ${\theta }_1$ and ${\theta }_2$, respectively. The coordinates $(x,y)$ are the centre of the object, and the solid red line is the short side of the target. We use $(x,y,s,r,{\theta }_1,{\theta }_2)$ to represent the target $P$, where $r$ is defined as follows:

(1) $r=\frac{s}{d}$(1)

where $s$ is the short side length of the object and $d$ is the diagonal length. Let $e_i,i\in 1,2,3,4$ respectively be the right, bottom, left and top endpoints of box $P$. We can decode the coordinates of the four vertices through the above variables as follows:

(2) $d=\left(\begin{array}{cc}\frac{s}{|tan(({\theta }_1+{\theta }_2)/2)|}& |{\theta }_1-{\theta }_2|\le \frac{\pi }{2}\\ s\times |tan(({\theta }_1+{\theta }_2)/2)|& |{\theta }_1-{\theta }_2|>\frac{\pi }{2}\end{array}\right.$(2)

(3) $\begin{array}{c}{e}_{1,4}=(x+\frac{d}{2}sin({\theta }_{1,2}-\frac{\pi }{2}),y-\frac{d}{2}sin({\theta }_{1,2}-\frac{\pi }{2}))\\ e_{2,3}=(x-\frac{d}{2}sin({\theta }_{2,1}-\frac{\pi }{2}),y+\frac{d}{2}sin({\theta }_{2,1}-\frac{\pi }{2}))\end{array}$(3)

3.3. SSG loss

In order to fix Case (d) and further improve the stability of the regression, we designed the SSG loss based on rectangular geometry. We found that in edge regression process, the accuracy of the short side is higher than that of the long side. We verify our supposition on the validation set of the DOTA(Xia et al. 2018) dataset. We use CenterNet (Zhou, Wang, and Philipp 2019), which is extended for oriented object detection with a single-angle output as the baseline; that is, the object is represented by $(x,y,w,h,\theta )$. Let $r=min(w,h)/max(w,h)$, and regress the object based on $(x,y,s,r,\theta )$ and $(x,y,l,r,\theta )$, respectively. where $s=min(w,h)$ and $l=max(w,h)$. The experimental results are shown in Table 1. The prediction result based on the short side method is much higher than that based on the long edge, especially for the harbour and bridge categories, and it is also higher than the baseline. We think that the receptive field of each point on heatmap can more easily to completely cover the short edge of object, especially for the bridge and harbour categories.

Table 1. Different results for short-side-based, long-side-based and the baseline

method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbour	SP	HC	mAP
Short-based	90.30	67.99	34.18	57.61	47.40	78.46	84.48	90.84	61.41	81.56	58.99	72.76	56.99	65.06	13.41	64.10
Long-based	89.56	71.99	24.56	47.63	46.04	70.33	69.34	90.74	62.30	77.86	58.66	60.21	52.01	65.46	28.75	61.03
Baseline	90.10	74.24	26.89	56.26	51.14	74.98	78.90	90.86	63.06	78.65	64.70	69.37	55.92	68.93	35.58	65.30

According to the geometric relationship of the prediction box in Figure 3, we find that the angle between the two diagonal lines ${\theta }_1$ and, ${\theta }_2$ and the length of the diagonal lines should meet the following conditions:

(4) $s^{\prime }=\left(\begin{array}{cc}d\times sin(\pi /2-{\theta }_{in})& {\theta }_{in}>\frac{\pi }{4}\\ d\times sin({\theta }_{in})& {\theta }_{in}\le \frac{\pi }{4}\end{array}\right.$(4)

where ${\theta }_{in}=\frac{|{\theta }_1-{\theta }_2|}{2}$. If $d,{\theta }_1$ and ${\theta }_2$ are all accurate, $s$ should be infinitely close to $s^{\prime }(i.e.,s=s^{\prime })$. Therefore, we designed the loss based on the short side to guide the process as follows:

(5) $L_{ssg}=Smoot{h}_{L_1}(s-s^{\prime })$(5)

According to Eq 1(1) $r=\frac{s}{d}$(1) and Eq 4(4) $s^{\prime }=\left(\begin{array}{cc}d\times sin(\pi /2-{\theta }_{in})& {\theta }_{in}>\frac{\pi }{4}\\ d\times sin({\theta }_{in})& {\theta }_{in}\le \frac{\pi }{4}\end{array}\right.$(4) , the common constant $s$ is eliminated, and Eq 5(5) $L_{ssg}=Smoot{h}_{L_1}(s-s^{\prime })$(5) is simplified as:

(6) $L_{ssg}=\left(\begin{array}{cc}Smoot{h}_{L_1}(1-\frac{1}{r}sin(\frac{\pi }{2}-{\theta }_{in}))& {\theta }_{in}>\frac{\pi }{4}\\ Smoot{h}_{L_1}(1-\frac{1}{r}sin({\theta }_{in}))& {\theta }_{in}\le \frac{\pi }{4}\end{array}\right.$(6)

Figure 5. The detailed structure of light attention learning module

3.4. Light attention learner

In order to further improve the accuracy on small and dense objects, we propose a light attention learning module as shown in Figure 2, and the detailed structure of light attention learning module has been shown in Figure 5, which is a kind of supervised module. We obtain the Attention Map $\hat{Y}\in [0,1{]}^{\frac{w}{T}\times \frac{h}{T}\times 1}$ from the Feature Map of the last layer of the backbone network through a simple convolution structure called Conv Block (mentioned in Figure 2), where the structure consists of a 3$\times$3 convolution, a ReLU activation function, and another 1$\times$1 convolution. The label of the module is a 0–1 binary image (Label Map in Figure 5) generated by the truth value. Among them, the pixel value in the object truth box area of all categories is set to 1 (the white area in Figure 5), and the non-object area is set to 0 (the black area in the Figure 5). Since Gaussian kernel is not used to generate labels, there are only two categories of 0 and 1 in the Label Map, so Focal Loss(Lin et al. 2017b) is used as the loss function:

(7) $L_{Att}=\frac{-1}{N}\sum \left(\begin{array}{cc}{(1-\hat{Y})}^{\alpha }log(\hat{Y})& \mathrm{i}\mathrm{f}Y=1\\ \begin{array}{c}{\hat{Y}}^{\alpha }log(1-\hat{Y})\end{array}& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\right.$(7)

Finally, the Attention Map predicted by this module will be element-wise addition with the input feature map before the output head ($\oplus$ in Figure 2) to reinforce the features within the area containing the object. All positions on the Attention Map are involved in the loss calculation. The light attention learner has a simple structure and only contributes very little time cost to the inference.

3.5. Loss function

In Section 3.1, our proposed method has $(C+2+2+1+1)$ outputs, and each set of the outputs corresponds to a loss. The loss for centre-point heatmap $L_k$ and offset heatmap $L_o$ follow CenterNet

(8) $L_k=\frac{-1}{N}\sum _{xyc}\left(\begin{array}{cc}{(1-{\hat{K}}_{xyc})}^{\alpha }log({\hat{K}}_{xyc})& \mathrm{i}\mathrm{f}{K}_{xyc}=1\\ \begin{array}{c}{(1-K_{xyc})}^{\beta }{({\hat{K}}_{xyc})}^{\alpha }\\ log(1-{\hat{K}}_{xyc})\end{array}& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}\end{array}\right.$(8)

(9) $L_{off}=\frac{1}{N}\sum _m\left|{\hat{O}}_m-O_m\right|.$(9)

where ${\hat{K}}_{xyc}$ is the corresponding prediction at the coordinate $(x,y)$ of any position on the $C^{th}$ heatmap, $\alpha$ and $\beta$ are the hyperparameters of the focal loss(Lin et al. 2017b) and their values do not change, and $N$ is the number of objects in the input images. We use an L1 loss at the angles of two diagonal lines and the short side, which is similar to Eq 9(9) $L_{off}=\frac{1}{N}\sum _m\left|{\hat{O}}_m-O_m\right|.$(9) :

(10) $L_a=\frac{1}{N}\sum _m\left|{\hat{A}}_m-A_m\right|$(10)

(11) $L_s=\frac{1}{N}\sum _m\left|{\hat{S}}_m-S_m\right|$(11)

where $A_m$ and $S_m$ are the ground truth of the angle and short side, respectively. The ratio loss $L_r=L_{CE}(\hat{R},R)$ is the cross-entropy between $\hat{R}$ and ground truth $R$. The overall training objective is

(12) $L=L_k+L_{Att}+L_{off}+{\lambda }_1{L}_a+{\lambda }_2{L}_s+L_r+L_{ssg}$(12)

We set the hyperparameters ${\lambda }_1=10$ and ${\lambda }_2=0.5$ in all of our experiments to simply the balancing of each part.

4. Experiments

4.1. Datasets

For the experiments, we choose three datasets, DOTA, HRSC2016 and UCAS-AOD, for the oriented object detection in aerial images.

• DOTA. DOTA is a challenging and large remote sensing dataset for both horizontal and oriented objects. DOTA V1.0 contains 15 categories and 2806 images while V1.5 adds one category and adds numerous labels for small objects. The image size in this dataset is large, and the number of instances in a single image is very high. As with other works, we use the offical tools provided by the DOTA to crop the original image in the experiment, and verify the results on the server that is provided by DOTA. In our work, only oriented labels are used. We use both the training and validation sets for training and the testing set for testing. We crop a series of $1024\times 1024$ patches from the original images with a stride of 824. For those categories with a small number of samples, we do not perform any augmentation to avoid the effect of an imbalance between different categories. Finally, compared with the RoI Transformer (37,373 patches) and the official baseline implements (150,342 patches), we only use 19,219 patches. The short names for each category are as follows: baseball diamond (BD), ground track field (GTF), small vehicle (SV), large vehicle (LV), tennis court (TC), basketball court (BC), storage tank (ST), soccer-ball field (SBF), roundabout (RA), swimming pool (SP), and helicopter (HC). The FR-O and RoI Transformers are both official detectors provided by DOTA.

• HRSC2016. HRSC2016 is also an oriented object detection dataset of aerial images, but it is much smaller than DOTA. It contains remote sensing images of various ships, and the size of the images is below $1500\times 1500$. Therefore, we directly resize all the images of the training and validation sets to ($1024\times 1024$) for training without any other tricks. We use the train-val data for training and testing on the test data set, as in other works ding2018learning.

• UCAS-AOD. The UCAS-AOD is a dataset widely used in oriented object detection in aerial images. It contains 2 categories and 1510 aerial images. There are 1000 plane images containing 7482 objects and 510 car images containing 7114 objects. In our experiments, we choose the oriented labels as research data and randomly select 1110 images for training and 400 images for testing, as in (Ding et al. 2019).

4.2. Implementation details

All the experiments are implemented using PyTorch 1.0 and 4 $\times$ 16GB NVIDIA Tesla V100 GPUs. To increase the detection speed as much as possible, we adopt ResNet-18 as the backbone in all the experiments and compare its results with the results of other detectors based on ResNet-101 or more complex backbones. The network is trained by the Adam (Kingma and Jimmy 2014) optimizer, and the learning rate is set to $1.25\times {10}^{-4}$ at the beginning of training with a batch size of 64. The learning rate is decreased by $10\times$ at epochs the 200 and 300, respectively. Random rotation (from $-\frac{\pi }{9}$ to $\frac{\pi }{9}$), scaling (from 0.5 to 1.5), and colour augmentation are used as data augmentation methods in all training processes. The specific parameters of the test phase will be described in detail in the relevant sections.

4.3. Ablation studies

To further illustrate the effectiveness of each part of our proposed work, we conduct a series of experiments on the DOTA validation dataset. We use CenterNet, which is extended for oriented object detection, as the baseline. The training parameters and settings are described in Section 4.2 and Section 4.1, respectively, and the modules are gradually added in the experiments. All the results are shown in Table 2, and our proposed method improved the mAP by a total of $4\mathrm{\%}$. We discuss the details in the following.

Table 2. Ablation studies of each component

Baseline	DASR	SSG	Light Attention Learner	mAP
✓	–	–	–	65.30
✓	✓	–	–	67.91
✓	✓	✓	–	68.07
✓	✓	✓	✓	69.32

4.3.1. DASR Strategy

The DASR regression strategy provides a very significant improvement since it outperforms the baseline by 2.61$\mathrm{\%}$. Furthermore, the performance of some single categories, especially the $harbor$, $small$ $vehicle$, and $bridge$ categories, have very conspicuous improvements, which are 6.48$\mathrm{\%}$, 7.39$\mathrm{\%}$, and 10$\mathrm{\%}$, respectively. The details results of each category as shown in Table 3. For some categories with smaller aspect ratios, such as $plane$, $storage$-$tank$, and $roundabout$, our approach has barely improved; For some extremely large objects, such as $baseball$-$diamond$ and $ground$-$track$-$field$, the accuracy of DASR strategy decreases. We believe that the reason is that it is relatively difficult to predict the direction of the object with large body size, which makes the prediction of angle unstable and more likely to appear as shown in Figure 4(d). After SSG loss and light attention learner are used, the detection accuracy of these categories rises to a certain extent, which further verifies the effectiveness of the proposed method. An intuitive result is shown in Figure 6. Single-angle prediction often occurs when inaccurate angle prediction causes an object with a large length and width ratio to deviate greatly from the ground truth, as shown in Figure 6 (left). DASR prediction stabilizes the unstable prediction box and is closer to the ground truth box as shown in Figure 6 (right).

Table 3. Ablation studies of each component (the details of each category are given in Table 2). LAL is the acronym for the $LightAttentionLearner$

method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbour	SP	HC	mAP
Baseline	90.10	74.24	26.89	56.26	51.14	74.98	78.90	90.86	63.06	78.65	64.70	69.37	55.92	68.93	35.58	65.30
+ DASR	90.26	72.31	33.36	54.52	58.53	80.11	79.64	90.77	60.66	79.13	68.86	70.17	65.91	66.33	48.13	67.91
+ SSG	90.20	73.53	32.05	56.44	57.43	79.84	79.41	90.83	65.43	78.88	66.14	70.59	64.74	65.09	50.39	68.07
+ LAL	90.13	74.57	38.21	55.69	61.99	79.65	87.10	90.64	64.39	79.78	67.73	69.04	64.48	67.51	49.94	69.39

Figure 6. Comparison of using the DASR (right) and baseline (left) in the harbour and large vehicle categories

4.3.2. Short-side guided loss

The purpose of the SSG loss is to guide the network to avoid $Case$ $(d)$ mentioned in Figure 4. We apply different settings in the experiments on the SSG loss, as shown in Table 4. Our SSG loss is a self-supervised loss, so we try to replace the parameters in the loss with the truth value. Where $with$ $label$ $d$ refers to replacing $d$ in the Eq 4(4) $s^{\prime }=\left(\begin{array}{cc}d\times sin(\pi /2-{\theta }_{in})& {\theta }_{in}>\frac{\pi }{4}\\ d\times sin({\theta }_{in})& {\theta }_{in}\le \frac{\pi }{4}\end{array}\right.$(4) ; $with$ $Label$ $s$ refers to replacing $s$ in the Eq 5(5) $L_{ssg}=Smoot{h}_{L_1}(s-s^{\prime })$(5) ; $without$ $label$ refers to the original loss of not making parameter substitution. It can be seen that the participation of labels in the loss calculation will decrease the mAP. The reason is that excessively strong restrictions will lead the network to forcibly maintain the geometric characteristics between parameters, making DASR invalid. Therefore, we just use it to fine-tune the network. We add the SSG loss after 200 epochs training based on DASR and continue training to 300 epochs. Table 2 shows a $0.16\mathrm{\%}$ improvement in the mAP.

Table 4. Comparison of different settings for the SSG loss

Settings	with label $d$	with label $s$	without label
mAP	64.36	66.52	68.07

4.3.3. Light attention learner

A light attention learner is a simple structure used to further improve the accuracy of small or dense object detections. The total mAP increases by 1.25$\mathrm{\%}$, as shown in Table 2, and the main improvement comes from $bridge$ (6.17$\mathrm{\%}$), $ship$ (7.69$\mathrm{\%}$), $small$ $vehicle$ (4.56$\mathrm{\%}$) categories. We also compared the performance improvement of the addition and multiplication operations between the attention mask and the feature map the baseline, which improved by 1.25$\mathrm{\%}$ and 0.48$\mathrm{\%}$, respectively. The multiplication operation greatly suppresses the background information, which significantly improves small object detections, but there is a certain drop for large and middle object detections.

4.3.4. Complexity analysis

We conduct complexity analysis experiments for each part of the study on DOTA dataset (Table 5). You can see that the individual parts have little impact on the memory and speed of the network. During the training phase, DASR and light attention learner with additional branches put a major burden on training speed and memory. SSG Loss does not add additional parameters in the structure, so it does not affect the memory occupation, but the loss of extra calculation makes the training speed slightly slower. In terms of the test speed, each part has no significant influence on the final speed, and the gap is in the microsecond level. Therefore, errors such as data loading and data post-processing will also have a great impact on the test speed.

Table 5. Comparison of the speed and memory of each component

method	train speed	test speed	parameters
Baseline	0.378s	0.017s	173.7MB
+ DASR	0.467s	0.017s	174.2MB
+ SSG	0.472s	0.017s	174.2MB
+ LAL	0.498s	0.018s	174.6MB

4.4. Comparisons with State-of-the-art

We compared the performance of Dual-Det with those of state-of-the-art algorithms on three datasets: DOTA(Xia et al. 2018), HRSC2016 (Liu et al. 2016b) and UCAS- AOD (Zhu et al. 2015).

Figure 7. Visualization of detection results from Dual-Det in DOTA

4.4.1. Results on DOTA

As seen in Table 6, we compare the performance of our method with those of state-of-the-art detectors, which include both one-stage and two-stage algorithms. Compared to one-stage and two-stage methods, our detector has better performance in seven categories including $BD$, $Ship$, $Harbor$, $SP$, $SV$, $TC$ and $LV$ and worse performance in the six categories of $GTF$, $Bridge$, $BC$, $SBF$, $HC$ and $RA$. However, compared to the performaces of one-stage detectors, our method achieves the best performances on the $GTF$, $ST$, $RA$ and $HC$ categories. For the large-scale object categories ($GTF$, $Bridge$, $BC$, and $SBF$), the one-stage detector has obvious disadvantages compared to the two-stage networks. Although our method does not overcome those shortcomings, it still has competitive performance compared to the performances of one-stage detectors. For the $RA$ category, we observe that the recall in those categories is higher, but the accuracy is lower. We think that the reason is that the shallow network is not conducive to the classification of confusing categories. In single-scale testing, Dual-Det can reach 72.00$\mathrm{\%}$ mAP in 0.018s (per image) in the inference phase, which is much faster than other one-stage and two-stage algorithms, and it is even 4 times faster than the fastest algorithm. In addition, the accuracy is also ahead of most of the others without the FPN structure in which only the output of the last feature map is used. DualDet-FT is the result of our flip test. During the test, we concatenate the image with horizontal flips and the original image as the input and take the average value of $\hat{K}$, $\hat{S}$, and $\hat{R}$ of the prediction result as the final prediction value. The other outputs keep the original image prediction value unchanged. We finally achieved 73.62$\mathrm{\%}$ mAP in 0.026s (per image) in the inference phase, which is only a 44.4$\mathrm{\%}$ decrease in the detection speed, but a 1.62$\mathrm{\%}$ increase in the mAP. The Visualization results for each category on the DOTA dataset were shown in Figure 7.

Table 6. Comparison with state-of-the-art detectors on DOTA

method	Plane	BD	Bridge	GTF	SV	LV	Ship	TC	BC	ST	SBF	RA	Harbour	SP	HC	mAP
Two-stage
FR-O	79.42	77.13	17.7	64.05	35.3	38.02	37.16	89.41	69.64	59.28	50.3	52.91	47.89	47.4	46.3	54.13
RRPN	80.94	65.75	35.34	67.44	59.92	50.91	55.81	90.67	66.92	72.39	55.06	52.23	55.14	53.35	48.22	61.01
R2CNN	88.52	71.2	31.66	59.3	51.85	56.19	57.25	90.81	72.84	67.38	56.69	52.84	53.08	51.94	53.58	60.67
R-DFPN	80.92	65.82	33.77	58.94	55.77	50.94	54.78	90.33	66.34	68.66	48.73	51.76	55.1	51.32	35.88	57.94
ICN	81.36	74.3	47.7	70.32	64.89	67.82	69.98	90.76	79.06	78.2	53.64	62.9	67.02	64.17	50.23	68.16
RoITransformer	88.64	78.52	43.44	75.92	68.81	73.68	83.59	90.74	77.27	81.46	58.39	53.54	62.83	58.93	47.67	69.56
SCRDet	89.98	80.65	52.09	68.36	68.36	60.32	72.41	90.85	87.94	86.86	65.02	66.68	66.25	68.24	65.21	72.61
One-stage
YOLOv2	39.57	20.29	36.58	23.42	8.85	2.09	4.82	44.34	38.35	34.65	16.02	37.62	47.23	25.5	7.45	21.39
IENet	80.20	64.54	39.82	32.07	49.71	65.01	52.58	81.45	44.66	78.51	46.54	56.73	64.40	64.24	36.75	57.14
$R^3$Det-Res101	89.54	81.99	48.46	62.52	70.48	74.29	77.54	90.80	81.39	83.54	61.97	59.82	65.44	67.46	60.05	71.69
$O^2$-DNet	89.31	82.14	47.33	61.21	71.32	74.03	78.62	90.76	82.23	81.36	60.93	60.17	58.21	66.98	61.03	71.04
Dual-Det	89.80	83.32	37.83	65.01	73.78	75.44	79.13	90.83	79.70	84.78	56.63	61.76	66.19	72.87	62.97	72.00
DualDet-FT	89.91	85.33	44.25	63.68	78.65	76.00	86.48	90.86	80.04	85.79	56.41	62.63	67.22	74.30	62.76	73.62

4.4.2. Results on HRSC2016

We compare the accuracy and speed of our method with those of other state-of-the-art detectors on the HRSC2016 test set in Table 7, Resdcn18 is the Figure 2 architecture that uses ResNet18 as the backbone. In the experiments, no other tricks are used except that the upper limit of the centre point is set as $k=10$. The results show that although our method uses a low-complexity backbone, it still achieves the best detection performance based on the VOC07 AP metric, which is 0.97$\mathrm{\%}$ mAP higher than $R^3$Det(Yang et al. 2019a) (the input size is 1024 $\times$ 1024). For fairness, we also reduce the input size to 800 $\times$ 800, and the results show that this change reduces the mAP by only a small amount and the methods still outperforms other detectors. Besides, we also calculated mAP based on the VOC12 AP metric like CenterMap-Net (Wang et al. 2021). It can be seen that no matter what the input size is, the performance of our low-complexity network is better than the other methods. In terms of detection speed, the low-complexity backbone gives our network a great advantage in speed, which is also proved by the results. Therefore, in many occasions requiring real-time performance, our method also has great practicability.

Table 7. Comparisons of the accuracy and speed with the state-of-the-art methods on HRSC2016. * means that the VOC12 AP metric is used

method	Backbone	Input Size	mAP	FPS
R2CNN (Jiang et al. 2017)	VGG16	800$\times$800	73.07	2
RRPN (Ma et al. 2018)	ResNet101	800$\times$800	79.08	3.5
$R^2$PN(Zhang et al. 2018)	VGG16	–	79.6	–
RRD (Liao et al. 2018)	VGG16	384$\times$384	84.3	–
RoITransformer(Ding et al. 2019)	ResNet101	512$\times$800	86.2	6
$R^3$Det(Yang et al. 2019a)	ResNet101	800$\times$800	89.26	12
CenterMap-Net*(Wang et al. 2021)	ResNet50	–	92.8	–
Dual-Det	Resdcn18	800$\times$800	90.04	62.5
Dual-Det*	Resdcn18	800$\times$800	95.15	62.5
Dual-Det	Resdcn18	1024$\times$1024	90.23	46
Dual-Det*	Resdcn18	1024$\times$1024	96.4	46

4.4.3. Results on UCAS-AOD

As seen in Table 8, we compare the performance of our method with the performance of other state-of-the-art detectors. We set the upper limit of the centre point as $k=40$, and other settings are the same as for HRSC2016. We used the VOC07 and 12 AP metric to calculate mAP. The results show that our detector achieve competitive performance on this dataset on both metric. Besides, with this dataset, the inference speed can also reach 46FPS.

Table 8. Comparison with the state-of-the-art detectors on UCAS-AOD (Zhu et al. 2015). * means that the VOC12 AP metric is used

method	plane	car	mAP
RRPN (Ma et al. 2018)	88.04	74.36	81.20
R2CNN (Jiang et al. 2017)	89.76	78.89	84.32
R-DFPN (Yang et al. 2018b)	88.91	81.27	85.09
$O^2$-DNet (Wei et al. 2020)	93.21	86.72	89.96
Dual-Det	95.64	86.02	90.83
Dual-Det*	97.34	89.70	93.52

5. Conclusion

In this paper, we propose a one-stage object detection framework, Dual-Det, which is a real-time keypoint-based detector for oriented objects in aerial images. The DASR redesigns the regression method based on the prediction of the dual-angle and short side of a box. Furthermore, the SSG loss guides the DASR to more accurate results. Additionally, the attention learner strategy helps to locate small and dense objects. All these methods achieve considerable performance improvements. Our network runs at 46FPS with 90.23$\mathrm{\%}$ mAP on the HRSC2016 dataset and at 46FPS (Frames Per Second) with 90.83$\mathrm{\%}$ mAP on the UCAS-AOD dataset. We also achieve the relatively good speed-accuracy trade-off on the DOTA dataset with 72.00$\mathrm{\%}$mAP in 0.018 s (per image) and 73.62$\mathrm{\%}$ mAP in 0.026 s (per image) in inference phase. Due to the low complexity of the backbone, Dual-Det has a fast detection speed, but its accuracy is still excellent. Experiments on three datasets show that the proposed method can achieve the state-of-the-art performance. Besides, we also would like to welcome other researchers optimize Dual-Det performance with a more complex backbone in the future.

6. List of acronyms

Table

BC	basketball-court
BD	baseball-diamond
DASR	dual-angle with short-side and ratio
DCN	deformable convolution
FPS	Frames Per Second
GTF	ground-track-field
HBB	Horizontal Bounding Box
HC	helicopter
IoU	Intersection over Union
LV	large-vehicle
mAP	mean Average Precision
NMS	Non-Maximum Suppression
OBB	Oriented Bounding Box
RA	roundabout
RoI	regions of interest
RPN	region proposal network
SBF	soccer-ball-field
SSG	short-side guided
SV	small-vehicle
ST	storage-tank
SP	swimming-pool
TC	tennis-court

Disclosure statement

No potential conflict of interest was reported by the author(s).