High-resolution satellite video single object tracking based on thicksiam framework

Figures & data

Figure 1. In high-resolution satellite videos, ground targets show attributes such SS, PO, PoV, PTBD, SD, and PGFI. These attributes are the challenges in current satellite video SOT task and also make natural scene-based trackers inapplicable to satellite videos.

Figure 2. The overall tracking workflow of ThickSiam. It formally includes TRBS-Net for extracting robust semantic features to obtain the initial tracking results and a RKF module for simultaneously correcting the trajectory and size of the targets. The results of TRBS-Net and RKF modules are combined by an N-frame-convergence mechanism to achieve final tracking results.

Figure 3. The structure comparisons of original residual block and the proposed TRB. Based on the original residual block, the modifications of TRM include doubling the number of channels in bottleneck and cropping out the limbic elements attached to the feature map.

Figure 4. The structure comparisons of original down-sampling residual block and the proposed TMRB. Based on the original down-sampling residual block, the modifications of the TMRB include doubling the number of channels in bottleneck, cropping out the outermost features, modifying the stride in the above two convolutional modules to 1, and adding a maxpooling layer to achieve down-sampling of the feature map.

Table 1. The detailed information of feature map in each stage of TRBS-Net. $s$ is the abbreviation of $stride$. $conv1$ and $bn1$, respectively, represent the convolutional layer and batch normalization in $stage1$.

		template branch	search branch
stage 1 ($s$=2)	conv1	64$\times$64$\times$64	128$\times$128$\times$64
	bn1	64$\times$64$\times$64	128$\times$128$\times$64
	ReLU	64$\times$64$\times$64	128$\times$128$\times$64
	crop	60$\times$60$\times$64	124$\times$124$\times$64
stage 2 ($s$=4)	maxpooling	30$\times$30$\times$64	62$\times$62$\times$64
	TRB	28$\times$28$\times$256	60$\times$60$\times$256
	TRB	26$\times$26$\times$256	58$\times$58$\times$256
	TRB	24$\times$24$\times$256	56$\times$56$\times$256
stage 3 ($s$=8)	TMRB	11$\times$11$\times$512	27$\times$27$\times$512
	TRB	9$\times$9$\times$512	25$\times$25$\times$512
	TRB	7$\times$7$\times$512	23$\times$23$\times$512
	TRB	5$\times$5$\times$512	21$\times$21$\times$512

Figure 5. The constructed exemplar-search training pairs. The annotation selected from DIOR (Li et al. 2020) object detection dataset is expanded outward by 1/2 of the sum of width and height, and scaled according to the sizes of the exemplar image and search image of the TRBS-Net.

Figure 6. The constructed testing dataset used in the experiments. There are twelve objects in eight satellite scenarios, and the targets consist of airplanes, ships, trains, and vehicle.

Table 2. The detailed descriptions of the constructed testing dataset. $px$ is the abbreviation of $pixel$. Attributes refer to the difficulties in tracking this object.

	Target	Target Size (px)	Frame Size (px)	Resolution (m)	Total Frames	FPS	Source	Location	Attributes
video 1	airplane-1	48 $\times$ 35	3840 $\times$ 2000	0.92	568	25	Jilin-1	Atlanta	SS, PoV, PTBD, SD, PGFI
video 1	airplane-2	61 $\times$ 54	3840 $\times$ 2000	0.92	737	25	Jilin-1	Atlanta	SS, SD
video 1	airplane-3	49 $\times$ 32	3840 $\times$ 2000	0.92	737	25	Jilin-1	Atlanta	SS, PoV, SD
video 2	airplane-4	43 $\times$ 47	3316 $\times$ 3024	0.92	246	10	Jilin-1	Helsinki	SS, SD, PGFI
video 3	airplane-5	35 $\times$ 25	3840 $\times$ 2000	0.92	375	25	Jilin-1	Guadalajara	SS, PoV, SD, PGFI
video 4	ship-1	14 $\times$ 15	2570 $\times$ 2160	0.92	500	25	Jilin-1	San Francisco	SS, PTBD, SD
video 5	ship-2	31 $\times$ 23	4096 $\times$ 3072	0.92	300	10	Jilin-1	Hong Kong	SS, PTBD, SD
video 5	ship-3	28 $\times$ 15	4096 $\times$ 3072	0.92	300	10	Jilin-1	Hong Kong	SS, PTBD, SD
video 5	ship-4	27 $\times$ 34	4096 $\times$ 3072	0.92	300	10	Jilin-1	Hong Kong	SS, PTBD, SD
video 6	train-1	29 $\times$ 79	3840 $\times$ 2160	1	362	30	ISS	Vancouver	PTBD, SD, PGFI
video 7	train-2	154 $\times$ 63	3840 $\times$ 2000	0.92	625	25	Jilin-1	Tian Jin	PTBD, SD
video 8	vehicle	10 $\times$ 30	2880 $\times$ 2000	0.92	500	25	Jilin-1	Naples	SS, PO, SD

Table 3. The experimental results of the ThickSiam framework with different training mechanisms. The baseline method was stacked by original residual block and down-sampling residual block according to the structure of the TRBS-Net.

Methods	Training Datasets	P	S	FPS
baseline (a)	COCO	0.922	0.683	63.166
	DIOR	0.938	0.688
	COCO+DIOR	0.934	0.699
baseline+KF (b)	COCO	0.927	0.687	61.936
	DIOR	0.943	0.696
	COCO+DIOR	0.947	0.708
baseline+RKF (c)	COCO	0.923	0.702	61.568
	DIOR	0.940	0.714
	COCO+DIOR	0.949	0.722
TRBS-Net (d)	COCO	0.939	0.703	57.315
	DIOR	0.943	0.701
	COCO+DIOR	0.959	0.721
TRBS-Net+KF (e)	COCO	0.941	0.701	57.014
	DIOR	0.950	0.687
	COCO+DIOR	0.967	0.722
TRBS-Net+RKF (f)	COCO	0.937	0.711	56.849
	DIOR	0.962	0.737
	COCO+DIOR	0.973	0.745

Table 4. Comparative experiment results of different $N$ values in the $N$-frame convergence mechanism. $N$ selected the number divisible by 10 from 10 to 100.

Methods	Training Datasets	Without RKF		N=10		N=20		N=30		N=40		N=50		N=60		N=70		N=80		N=90		N=100
		P	S	P	S	P	S	P	S	P	S	P	S	P	S	P	S	P	S	P	S	P	S
baseline (a)	COCO	0.922	0.683	0.921	0.701	0.921	0.704	0.937	0.708	0.911	0.700	0.915	0.701	0.915	0.702	0.923	0.702	0.936	0.709	0.923	0.703	0.937	0.708
	DIOR	0.938	0.688	0.939	0.714	0.944	0.714	0.937	0.712	0.941	0.712	0.943	0.712	0.941	0.713	0.940	0.714	0.948	0.720	0.949	0.715	0.945	0.714
	COCO+DIOR	0.934	0.699	0.937	0.717	0.940	0.717	0.932	0.716	0.935	0.719	0.948	0.722	0.954	0.722	0.949	0.722	0.941	0.718	0.945	0.718	0.950	0.720
TRBS-Net (b)	COCO	0.939	0.703	0.938	0.716	0.937	0.712	0.919	0.694	0.938	0.713	0.917	0.696	0.938	0.711	0.937	0.711	0.937	0.711	0.935	0.711	0.916	0.698
	DIOR	0.943	0.701	0.936	0.737	0.922	0.728	0.941	0.736	0.942	0.736	0.935	0.735	0.921	0.727	0.962	0.737	0.960	0.739	0.936	0.736	0.866	0.686
	COCO+DIOR	0.959	0.721	0.992	0.752	0.954	0.732	0.981	0.748	0.983	0.747	0.991	0.755	0.962	0.738	0.973	0.745	0.961	0.737	0.962	0.737	0.984	0.749

Table 5. Comparisons with the state-of-the-art trackers on our constructed testing dataset.

Trackers	Methods	Features	Backbones	CUDA	P	S	FPS
MOSSE (2010) (Bolme et al. 2010)	CF-based	Grayscale Intensity	-	-	0.745	0.480	4.964
CSK (2012) (Henriques et al. 2012)	CF-based	Grayscale Intensity	-	-	0.755	0.512	5.478
KCF (2014) (Henriques et al. 2014)	CF-based	HOG	-	-	0.851	0.634	18.210
CN (2014) (Danelljan et al. 2014b)	CF-based	Color Table	-	-	0.859	0.609	8.763
DSST (2014) (Danelljan et al. 2014a)	CF-based	HOG	-	-	0.782	0.596	9.720
Staple (2016) (Bertinetto et al. 2016a)	CF-based	HOG+Color Histogram	-	-	0.776	0.58	10.887
SiamFC (2016) (Bertinetto et al. 2016b)	DL-based	CNN Features	AlexNet	✓	0.902	0.663	127.174
DCFNet (2017) (Wang et al. 2017)	CF-based	CNN Features	conv1 from VGG	✓	0.833	0.644	12.400
ECO (2017) (Danelljan et al. 2017)	CF-based	CNN Features	ResNet18 with vgg-m conv1 layer	-	0.856	0.645	3.998
STRCF (2018) (Li et al. 2018)	CF-based	HOG+Color Table	-	-	0.795	0.557	7.498
ATOM (2019) (Danelljan et al. 2019)	DL-based	CNN Features	ResNet18	✓	0.852	0.622	10.771
DiMP (2019) (Bhat et al. 2019)	DL-based	CNN Features	ResNet18	✓	0.717	0.545	12.697
DiMP (2019) (Bhat et al. 2019)	DL-based	CNN Features	ResNet50	✓	0.747	0.597	11.239
SiamFC+ (2019) (Zhang and Peng 2019)	DL-based	CNN Features	ResNet22	✓	0.839	0.652	59.333
SiamRPN+ (2019) (Zhang and Peng 2019)	DL-based	CNN Features	ResNet22	✓	0.878	0.618	114.867
SiamRPN++ (2019) (Li et al. 2019)	DL-based	CNN Features	AlexNet	✓	0.883	0.656	144.783
SiamRPN++ (2019) (Li et al. 2019)	DL-based	CNN Features	ResNet50	✓	0.828	0.655	31.617
SiamFC++ (2020) (Xu et al. 2020)	DL-based	CNN Features	AlexNet	✓	0.925	0.699	139.828
ID-DSN (2021) (Zhu et al. 2021)	DL-based	CNN Features	ResNet50	✓	0.933	0.718	31.167
ThickSiam (ours, TRBS-Net)	DL-based	CNN Features	TRB+TMRB	✓	0.959	0.721	57.315
ThickSiam (ours, TRBS-Net+RKF)	DL-based	CNN Features	TRB+TMRB	✓	0.991	0.755	56.849

Figure 7. The visualized tracking results of ThickSiam, SiamFC++ (AlexNet) and SiamFC (AlexNet) trackers with corresponding GT. The yellow $\mathrm{\#}$ at the top right of the image represented the video frame number. The yellow, red, blue, and green bounding boxes represented the results of GT, ThickSiam (ours, TRBS-Net+RKF), SiamFC++ (AlexNet), and SiamFC (AlexNet), respectively.

Figure 8. The tracking results of the ThickSiam tracker in six typical scenarios containing all attributes including SS, PO, PoV, PTBD, SD, and PGFI.

Figure 9. The field of view of “Jilin-1” satellite video. The specified area in the red box on the left was enlarged and displayed in the upper right corner. Cars on the bridge were further magnified for visual display. These targets usually had only a few or dozens of pixels, and the ultra-small size made their apparent features weak, and the boundary with the background was not clearly visible.

Li, K., G. Wan, G. Cheng, L. Meng, and J. Han. 2020. “Object Detection in Optical Remote Sensing Images: A Survey and a New Benchmark.” Isprs Journal of Photogrammetry and Remote Sensing 159: 296–307. doi:10.1016/j.isprsjprs.2019.11.023.

 Web of Science ®Google Scholar

Bolme, D. S., J. R. Beveridge, B. A. Draper, and Y. M. Lui. 2010. Visual Object Tracking Using Adaptive Correlation Filters. 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 2544–2550. doi:10.1109/CVPR.2010.5539960.

 Google Scholar

Henriques, J. F., R. Caseiro, P. Martins, and J. Batista. 2012. Exploiting the Circulant Structure of Tracking-By-Detection with Kernels. Proceedings of the european conference on computer vision, Florence, Italy.

 Google Scholar

Henriques, J. F., R. Caseiro, P. Martins, and J. Batista. 2014. “High-Speed Tracking with Kernelized Correlation Filters.” IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (3): 583–596. doi:10.1109/TPAMI.2014.2345390.

 Web of Science ®Google Scholar

Danelljan, M., F. S. Khan, M. Felsberg, and J. Van De Weijer. 2014b. Adaptive Color Attributes for Real-Time Visual Tracking. 2014 IEEE conference on computer vision and pattern recognition, pp. 1090–1097. doi:10.1109/CVPR.2014.143.

 Google Scholar

Danelljan, M., G. Häger, F. S. Khan, and M. Felsberg. 2014a. Accurate Scale Estimation for Robust Visual Tracking. British machine vision conference, Nottingham, September 1-5, 2014. Bmva Press.

 Google Scholar

Bertinetto, L., J. Valmadre, S. Golodetz, O. Miksik, and P. H. S. Torr. 2016a. Staple: Complementary Learners for Real-Time Tracking. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, Nevada, pp. 1401–19. doi:10.1109/CVPR.2016.156.

 Google Scholar

Bertinetto, L., J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr. 2016b. Fully-Convolutional Siamese Networks for Object Tracking. In: G. Hua and H. Jégou (edited by), Computer vision - eccv 2016 workshops - Amsterdam, The Netherlands, October 8-10 and 15-16, 2016, Proceedings, Part II, pp. 850–865. doi:10.1007/978-3-319-48881-3\_56.

 Google Scholar

Wang, Q., J. Gao, J. Xing, M. Zhang, and W. Hu. 2017. Dcfnet: Discriminant Correlation Filters Network for Visual Tracking. https://github.com/foolwood/DCFNet_pytorch.

 Google Scholar

Danelljan, M., G. Bhat, F. S. Khan, and M. Felsberg. 2017. Eco: Efficient Convolution Operators for Tracking. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6931–6939. doi:10.1109/CVPR.2017.733.

 Google Scholar

Li, F., C. Tian, W. Zuo, L. Zhang, and M. H. Yang. 2018. Learning Spatial-Temporal Regularized Correlation Filters for Visual Tracking. 2018 IEEE/CVF conference on computer vision and pattern recognition, pp. 4904–4913. doi:10.1109/CVPR.2018.00515.

 Google Scholar

Danelljan, M., G. Bhat, F. S. Khan, and M. Felsberg. 2019. Atom: Accurate Tracking by Overlap Maximization. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4655–4664. doi:10.1109/CVPR.2019. 00479.

 Google Scholar

Bhat, G., M. Danelljan, L. Van Gool, and R. Timofte. 2019. Learning Discriminative Model Prediction for Tracking. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 6181–6190. doi:10.1109/ICCV.2019.00628.

 Google Scholar

Zhang, Z., and H. Peng. 2019. Deeper and Wider Siamese Networks for Real-Time Visual Tracking. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4586–4595. doi:10.1109/CVPR.2019.00472.

 Google Scholar

Li, B., W. Wu, Q. Wang, F. Zhang, J. Xing, and J. Yan. 2019. Siamrpn++: Evolution of Siamese Visual Tracking with Very Deep Networks. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4277–4286. doi:10.1109/CVPR.2019.00441.

 Google Scholar

Xu, Y., Z. Wang, Z. Li, Y. Yuan, and G. Yu. 2020. Siamfc++: Towards Robust and Accurate Visual Tracking with Target Estimation Guidelines. Proceedings of the AAAI conference on artificial intelligence, Hilton New York Midtown, New York, New York, USA, pp. 12549–12556.

 Google Scholar

Zhu, K., X. Zhang, G. Chen, X. Tan, P. Liao, H. Wu, X. Cui, Y. Zuo, and Z. Lv. 2021. “Single Object Tracking in Satellite Videos: Deep Siamese Network Incorporating an Interframe Difference Centroid Inertia Motion Model.” Remote Sensing 13 (7): 1298. doi:10.3390/rs13071298.

 Web of Science ®Google Scholar

Data availability statement

The testing data in this study are available at https://github.com/CVEO/ThickSiam.