A news picture geo-localization pipeline based on deep learning and street view images

Figures & data

Figure 1. Data and study area. (a) The location distribution of street view images. (b) Street view distribution near the SOGO Department Store and example of street view panorama. (c) The process of generating perspective images from the panorama.

Figure 2. Example of query images.

Figure 3. Google Landmarks Dataset.

Figure 4. The architecture of pipeline for news picture geo-localization.

Figure 5. The network architecture for retrieval in the training and feature extraction steps.

Figure 6. Loss of feature extraction network.

Figure 7. The tuple input of the network and contrastive loss.

Figure 8. The KDP process of the example image.

Table 1. Results of the proposed pipeline.

$Acc@d,k$	1	2	3	5	10	15	20
50 m	50.62%	55.56%	55.56%	56.79%	58.02%	58.02%	58.02%
100 m	59.26%	65.43%	66.67%	67.90%	70.37%	70.37%	70.37%
150 m	61.73%	67.90%	70.37%	71.60%	74.07%	74.07%	74.07%
200 m	67.90%	74.07%	77.78%	81.48%	85.19%	85.19%	85.19%
300 m	72.84%	80.25%	83.95%	88.89%	93.83%	95.06%	97.53%

Figure 9. The subexperimental area and dataset.

Table 2. The subexperimental results of this paper pipeline.

$Acc@d,k$	1	2	3	5	10	15	20
50 m	50.62%	58.02%	64.20%	70.37%	74.07%	77.78%	77.78%
100 m	58.02%	64.20%	66.67%	76.54%	82.72%	85.19%	86.42%
150 m	60.49%	69.14%	74.07%	82.72%	91.36%	92.59%	93.83%
200 m	65.43%	76.54%	81.48%	87.65%	96.30%	97.53%	98.77%
300 m	74.07%	83.95%	90.12%	95.06%	97.53%	97.53%	98.77%

Figure 10. The improvement of ED between both methods.

Table 3. The subexperimental results of the GFS method.

$Acc@d,k$	1	2	3	5	10	15	20
50 m	33.33%	38.27%	46.91%	55.56%	69.14%	75.31%	76.54%
100 m	46.91%	59.26%	64.20%	69.14%	80.25%	82.72%	85.19%
150 m	53.09%	64.20%	69.14%	74.07%	88.89%	92.59%	93.83%
200 m	60.49%	70.37%	74.07%	79.01%	91.36%	95.06%	95.06%
300 m	69.14%	82.72%	85.19%	90.12%	97.53%	98.77%	98.77%

Table 4. The subexperimental results of the GeM method.

$Acc@d,k$	1	2	3	5	10	15	20
50 m	9.88%	16.05%	18.52%	25.93%	28.40%	32.10%	34.57%
100 m	14.81%	23.46%	28.40%	35.80%	45.68%	50.62%	53.09%
150 m	18.52%	27.16%	32.10%	43.21%	49.38%	55.56%	60.49%
200 m	19.75%	29.63%	34.57%	49.38%	56.79%	61.73%	65.43%
300 m	20.99%	35.80%	41.98%	56.79%	62.96%	66.67%	74.07%

Table 5. The subexperimental results of the NetVLAD method.

$Acc@d,k$	1	2	3	5	10	15	20
50 m	7.41%	11.11%	11.11%	13.58%	19.75%	24.69%	29.63%
100 m	11.11%	17.28%	24.69%	27.16%	33.33%	34.57%	41.98%
150 m	17.28%	19.75%	29.63%	32.10%	39.51%	41.98%	48.15%
200 m	18.52%	23.46%	34.57%	38.27%	48.15%	51.85%	59.26%
300 m	20.99%	28.40%	37.04%	40.74%	53.09%	59.26%	66.67%

Table 6. The statistics of subexperimental ED results of comparison.

Statistics of ED	Median(m)	Average(m)	Standard deviation (m)
This paper	47.77	309.68	529.64
GFS	130.04	379.93	559.01
GeM	991.83	992.83	739.46
NetVLAD	1173.61	1099.78	822.67

Figure 11. The example of attention layer visualization of news images on different scales.

Table 7. Results of the retrieval method.

$Acc@d,k$	1	2	3	5	10	15	20
50 m	48.15%	55.56%	59.26%	64.20%	71.60%	74.07%	74.07%
100 m	55.56%	58.02%	62.96%	70.37%	79.01%	83.95%	83.95%
150 m	56.79%	60.49%	65.43%	74.07%	81.48%	86.42%	86.42%
200 m	60.49%	65.43%	69.14%	80.25%	88.89%	92.59%	93.83%
300 m	64.20%	71.60%	74.07%	85.19%	90.12%	92.59%	95.06%

Figure 12. Results in the different kernel density radii.

Figure 13. The Top 1 positive result samples of query images.

Figure 14. The distribution of KDP Top 1 ED and corresponding image retrieval Top 1 example results.

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.