Global road extraction using a pseudo-label guided framework: from benchmark dataset to cross-region semi-supervised learning

ABSTRACT

Recent advancements in satellite remote sensing technology and computer vision have enabled rapid extraction of road networks from massive, Very High-Resolution (VHR) satellite imagery. However, current road extraction methods face the following limitations: 1) Insufficient availability of accurate and diverse training datasets for global-scale road extraction; 2) Costly and time-consuming manual labeling of millions of road samples; and 3) Limited generalization ability of deep learning models across diverse global contexts, resulting in better performance for regions well-represented in the training dataset, but worse performance when faced with domain gaps. To address these challenges, a semi-supervised framework was developed in this study, which includes a global-scale benchmark dataset – termed GlobalRoadSet (GRSet) – and a pseudo-label guided semi-supervised road extraction network – termed GlobalRoadNet^SF (GRNet^SF). The GRSet dataset was constructed using high-resolution satellite imagery and open-source crowdsourced OpenStreetMap (OSM) data. It comprises 47,210 samples collected from 121 capital cities across six populated continents. The GRNet^SF trains the network by generating pseudo-labels for unlabeled images, combined with a few labeled samples from the target region. To enhance the quality of the pseudo-labels, strong data augmentation perturbation and auxiliary feature perturbation techniques are employed to ensure model prediction consistency. The proposed GRNet^SF_GRSet framework was implemented in over 30 cities worldwide, where most of the Intersection-over-Union (IoU) values increased by more than 10%. This outcome confirms its strong generalization ability.

KEYWORDS:

• Road extraction
• global-scale
• remote sensing
• deep learning
• semi-supervised learning

1. Introduction

Rapid road expansion is currently underway and is expected to continue this century (Kleinschroth et al. 2019; Laurance et al. 2014), which is promoted by a multitude of factors, including urbanization and increased demand for travel and transportation. Accurate and up-to-date mapping and extraction of road networks are critical tasks to maintain the function of urbanicity and socioeconomic development, particularly in realizing smart transport and intelligent traffic management. Although there are some open-source datasets mapping roads globally, such as OpenStreetMap (OSM), they are typically outdated and spatially incomplete, with uneven accuracies. Moreover, the efficiency and accuracy of crowdsourced road mapping efforts cannot meet the requirement of real-world urban applications. Therefore, spatially explicit and complete global-scale road networks are urgently needed to improve our understanding of road network patterns, structure, and expansion across spatial-temporal scales and support a wide range of urban implications (Kleinschroth et al. 2019; Meijer et al. 2018).

Recently, with the rapid development of earth observation technology (D. Li et al. 2024), many advanced methods for road extraction from Very High Resolution (VHR) remote sensing imagery have been developed. Road extraction methods have developed from traditional methods (Shi, Miao, and Debayle 2013) based on manual features to deep learning methods (Mei et al. 2021) based on self-learning of features. Deep learning methods have become the mainstream methods for road extraction due to their high degree of automation and their ability to process massive remote sensing images (Z. Li et al. 2023; X. X. Zhu et al. 2017). However, supervised deep learning methods require numerous training samples for optimal performance. Thanks to the contributions of the scientific community, four typical road datasets are publicly available: Massachusetts (Mnih 2013), DeepGlobe (Demir et al. 2018), SpaceNet (Van Etten, Lindenbaum, and Bacastow 2018), and RoadTracer (Bastani et al. 2018).

Based on these four public datasets, some representative studies have been conducted. For example, Mnih (2013) employed a multi-layer neural network for road extraction and released the Massachusetts road dataset. This was the first large-scale open-source dataset for road extraction, which was widely used in the following years (Saito, Yamashita, and Aoki 2016; M. Zhou et al. 2020). In the DeepGlobe Road Extraction Challenge, the D-LinkNet (Zhou et al. 2018) model won first place, which was built with LinkNet (Chaurasia and Culurciello 2017) and dilated convolution. For the SpaceNet Road Detection Challenge, the winning approach (Van Etten, Lindenbaum, and Bacastow 2018) used an ensemble of deep learning segmentation models. Differing from the previous segmentation-based methods, Bastani et al. (2018) used Convolutional Neural Network (CNN)-guided iterative search to directly output the road graph. Undoubtedly, all these datasets and deep learning methods have promoted the progress of road extraction technology. However, these road datasets cover only a few countries, with limited data volumes and data diversity, and as deep learning models are heavily biased toward the data distribution of the training samples, the model performance will decline sharply when the data distributions of the training images and test images are different (Wang et al. 2022). Developing more diverse datasets will help improve model performance, but it requires expensive pixel-level annotation. Fortunately, differing from the ground objects in natural images, roads in remote sensing images already have many historical OSM road centerlines, which can be used as noisy supervised information.

As an alternative approach, many studies have attempted to improve the generalization ability of models by designing more advanced algorithms. For example, to improve the model performance on a target domain containing insufficient annotated data, domain adaptation techniques (Csurka 2017; Patel et al. 2015) emerged and have been successfully applied in many practical applications (Ghifary et al. 2016). Among these methods, Unsupervised Domain Adaptation (UDA) based on adversarial learning (Goodfellow et al. 2014) is the most widely used, which only needs labeled source domain samples and unlabeled target domain images to achieve domain adaptation. For instance, Lu et al. (2021a) proposed the Global-lOcal Adversarial Learning (GOAL) approach, which can adaptively reweight the adversarial loss according to the road recognition difficulty; and Wang et al. (2022) presented a feature transfer network to achieve cross-domain road extraction. But the UDA technique does not utilize target region samples at all, resulting in a limited model performance.

Similar to the UDA method, Semi-Supervised Learning (SSL) (H. Chen et al. 2022; X. Zhu and Goldberg 2009) is also a popular way to overcome the dependence on huge labeled datasets, and involves leveraging numerous unlabeled target images alongside a few labeled target samples. Consistency regularization and the pseudo-labeling learning framework predominantly drive the SSL methods, enabling effective learning on unlabeled data (Sohn et al. 2020). Consistency regularization refers to enforcing the model to maintain consistent predictions for inputs under different perturbations, thereby achieving supervised learning. Pseudo-labeling refers to incorporating high-confidence predictions of unlabeled images by the model into the training dataset. For instance, Cross-Consistency Training (CCT) (Ouali, Hudelot, and Tami 2020) and Mean Teacher (MT) (Tarvainen and Valpola 2017) were proposed for semi-supervised semantic segmentation based on consistency regularization. Chen et al. (2023) employed rotation consistency constraints and adversarial learning to fully exploit the potential information from low-confidence pixels in pseudo labels for road extraction. Li et al. (2022) proposed a multi-stage pseudo-label iteration framework for the land-cover mapping task of the 2022 IEEE GRSS Data Fusion Contest, which fuses the predictions of four models to generate pseudo-labels.

Based on the above analysis, although some road datasets and road extraction methods have been developed, the generalization ability of road extraction models is still limited. To alleviate this problem, a pseudo-label guided global-scale road extraction framework is introduced in this article. The contributions of this article can be summarized as follows:

1. A robust global-scale road benchmark dataset – GlobalRoadSet (GRSet) – was built in this study. Developing more diverse datasets can greatly improve model performance and help us understand how models perform in different regions of the world. However, manually labeling millions of road samples is labor-intensive. Thus, we leveraged massive VHR satellite imagery and crowdsourced OSM data to build the GRSet dataset, containing 47,210 samples from 121 capital cities across six continents in Europe, Africa, Asia, South America, Oceania, and North America, with a total area of 49,503 km².

2. A pseudo-label guided global-scale road extraction method – GRNet^SF—is proposed. In order to ensure that the model can adapt from GRSet to new imagery across different geographical regions worldwide, GRNet^SF is first pre-trained on the GRSet dataset, to provide initialization parameters for the road extraction network. The pre-trained GRNet^SF then produces pseudo-labels for the unlabeled images of the target region, along with the few labeled samples, to retrain the network. To improve the quality of the pseudo-labels and avoid the model overfitting on inaccurate pseudo-label annotated samples, strong data augmentations perturbation and auxiliary feature perturbation are adopted to constrain the consistency of the model predictions, which is based on the assumption that the model predictions should be similar when different perturbations are made to the same image.

3. The GRSet and GRNet^SF were applied to the global-scale validation set, covering more than 30 cities in developed and developing countries across four continents: Europe, Africa, Asia, and North America. The global-scale validation set is made up of public road test sets and self-annotated large-scale samples. The experimental results show that GRSet and GRNet^SF have strong robustness and a superior generalization ability, and can obtain the most complete and continuous roads, most of which improve the Intersection-over-Union (IoU) by more than 10%.

2. Related work

This section introduces the current research status from two aspects: the typical road datasets and the semi-supervised learning method applied in the remote sensing image processing field.

2.1. Representative road datasets

The most widely used road datasets are the Massachusetts (Mnih 2013), DeepGlobe (Demir et al. 2018), SpaceNet (Van Etten, Lindenbaum, and Bacastow 2018), and RoadTracer (Bastani et al. 2018) datasets. Table 1 summarizes the resolution, width, data volume, and coverage area of the four public road datasets and the GRSet. Among them, the DeepGlobe dataset is precisely annotated with varying road widths. The remaining datasets are generated with OSM road centerlines with an equal width, and thus their road widths are fixed, and the road boundaries are not completely accurate. Figure 1 gives examples of the five road datasets. As can be seen, the road types in the Massachusetts and SpaceNet datasets are relatively simple, most of which are cement or asphalt roads. The RoadTracer and DeepGlobe road datasets are more diverse, with DeepGlobe in particular containing many rural dirt roads. Overall, GRSet is the most diverse.

Figure 1. Examples from the five road datasets (512 × 512 pixels).

Table 1. Details of the five road datasets.

Road Datasets	Resolution (m/pixel)	Width (m)	Size (pixels)	Images (train/test)	Countries	Cities	Types	Area (km^2)
Massachusetts	1	7	1500 × 1500	1108/49	1	>1	Urban	2603.25
RoadTracer	0.6	6	1024 × 1024	2880/1920	6	40	Urban	1811.94
SpaceNet	0.3	2	1300 × 1300	2213/567	4	4	Urban	422.84
DeepGlobe	0.5	–	1024 × 1024	4696/1530	3	>3	Urban/rural	1632.11
GRSet	1	5	1024 × 1024	47,210/-	121	121	Urban/rural	49,503.27

2.2. Semi-supervised learning

Semi-supervised learning provides an effective means to improve model generalization performance using unlabeled data and it is dominated by pseudo-labeling and consistency regularization.

Pseudo-labeling is aimed at generating high-confidence pseudo-labels for unlabeled data using the model itself (Lee 2013). For the input unlabeled data $u_b$, the predicted probability is $\mathit{p}$, the largest class probability is $\max \left(\mathit{p}\right)$, and the pseudo-label is $\hat{p}=\mathrm{argmax}\left(\mathit{p}\right)$, which refers to the “hard” one-hot label. The pseudo-labeling loss can then be expressed as:

(1) $\frac{1}{\mathit{\mu B}}\sum _{\mathit{b}=1}^{\mathit{\mu B}}1\left(\mathit{max}\left(\mathit{p}\right)\ge \mathit{\tau }\right)\mathit{H}\left(\hat{p},\mathit{p}\right)$(1)

where $\mathit{\tau }\in \left(0,1\right)$ is the threshold and $1\left(\max \left(\mathit{p}\right)\ge \mathit{\tau }\right)$ is an indicator function, which is used to filter out noisy unlabeled data with low prediction confidence; $\mathit{H}$ is the cross entropy used to make the prediction $\mathit{p}$ close to the pseudo-label $\hat{p}$; B is the batch size of labeled data; $\mathit{\mu }$ is the ratio of unlabeled data to labeled data; and $\mathit{b}=1\dots \mathit{\mu B}$.

Consistency Regularization is a powerful technique that has been widely used in SSL algorithms (Sajjadi, Javanmardi, and Tasdizen 2016). To sum up the core idea, it enforces the consistency of the model predictions through various perturbations, for instance, input perturbation (Kim, Jang, and Park 2020), feature perturbation (Ouali, Hudelot, and Tami 2020) and network perturbation (Ke et al. 2019). Taking input interference as an example, the network output should be invariant when fed the same image with different data augmentations, such as the weak and strong data augmentations. The weak data augmentation usually includes random flip, crop and shift. For segmentation tasks, geometric transformation is not applicable due to the pursuit of pixel-level accuracy, and color jittering is generally used as strong data augmentation (T. Chen et al. 2020; Zou et al. 2020). To calculate the consistency loss, the L2-loss is commonly used, which can be expressed by Equation (2)(2) $\sum _{\mathit{b}=1}^{\mathit{\mu B}}||p_1\left(\mathit{y}|{\alpha }_1\left(u_b\right)\right)-p_2\left(\mathit{y}|{\alpha }_2\left(u_b\right)\right){||}_2^2$(2) :

(2) $\sum _{\mathit{b}=1}^{\mathit{\mu B}}||p_1\left(\mathit{y}|{\alpha }_1\left(u_b\right)\right)-p_2\left(\mathit{y}|{\alpha }_2\left(u_b\right)\right){||}_2^2$(2)

where $u_b$ refers to unlabeled data; ${\alpha }_1$ and ${\alpha }_2$ represent different perturbations; $p_1$ and $p_2$ are the output probabilities.

3. GRSet dataset generation

The GRSet dataset was generated using VHR satellite imagery and OSM road data without any manual annotation. To ensure the quality of the OSM road data, only capital cities with abundant OSM data were selected, and only six categories of main roads were considered, to minimize the inclusion of narrow roads and inaccurate OSM data. The specific methodological steps were as follows:

• Step 1: Satellite images with a spatial resolution of 1 m per pixel were acquired from the Google Earth Platform, and the OSM road centerline vector data were obtained using global city names as keywords. The capital cities were categorized based on administrative boundaries, encompassing both urban and rural regions. Notably, use of the Google Earth images must respect the “Google Earth” terms of use, and these images are exclusively intended for academic purposes, with any commercial usage strictly prohibited.

• Step 2: The six main types of roads were retained based on the properties of the vector data, i.e. motorway, trunk, primary, secondary, tertiary, and residential. Referring to Van Etten et al. (2018), the study retained the six main road types while disregarding other less significant road types to mitigate the impact of their inaccuracies.

• Step 3: A 5-meter width label was assigned to each edge of the road vector through rasterization. It is important to emphasize that the annotation is rarely entirely accurate, and the principle of rasterization aims to maximize the number of positive samples.

• Step 4: The images and labels were aligned based on their corresponding geographic coordinates, and subsequently, the images were cropped to a size of 1024 × 1024 pixels. Tiles that contained fewer than five roads were excluded from the dataset, as such tiles tend to exhibit a relatively high error rate.

The GRSet dataset was constructed based on the aforementioned steps and is publicly accessible at: https://github.com/xiaoyan07/GRNet_GRSet. As summarized in Table 1, GRSet is a geographically diverse road dataset with a substantial data volume, playing a critical role in enhancing the accuracy and generalizability of deep networks. Furthermore, the GRSet dataset is highly complex and diverse, encompassing roads with varying radiation levels, types, materials, and scales, as depicted in Figure 2. However, GRSet is not a perfect dataset, as it includes certain instances of noise, such as missing roads and misregistration, as illustrated in Figure 3.

Figure 2. Distribution of the geographical locations of the GRSet dataset.

Figure 3. Annotation examples of the GRSet.

4. The pseudo-label guided global-scale road extraction framework

This section provides a comprehensive description of the proposed framework. As depicted in Figure 4, the framework initially employs the GRSet dataset to pre-train the encoder and the main decoder (namely LinkNet50), which uses ResNet50 as the feature extractor. The sigmoid classifier is utilized to generate a one-layer prediction map. Specifically, LinkNet50 is the baseline model that won first prize in the DeepGlobe Road Extraction Challenge (L. Zhou, Zhang, and Ming 2018). In the second step, the encoder and the main decoder are initialized using the pre-trained model parameters from GRSet, and the auxiliary decoder is initialized using PyTorch’s default initialization (He et al. 2015), where linear layers are initialized with kaiming_uniform, and the weights and biases of convolutional layer are respectively initialized with kaiming_uniform and uniform. The network first produces pseudo-labels for the unlabeled target region images, and then is restrained with the few accurately labeled samples and the pseudo-label annotated samples from the target region. Nevertheless, the quality of the generated pseudo-labels cannot be guaranteed, potentially leading to overfitting of the model on incorrect samples. To alleviate this problem, different input data perturbations and auxiliary decoder feature perturbations are employed to enforce the consistency of model predictions, thereby mitigating the risk of overfitting to inaccurate pseudo-labels.

Figure 4. The flowchart of the proposed pseudo-label guided global-scale road extraction framework, which is composed of two parts: (a) GRSet generation, and (b) pseudo-label guided global-scale road extraction network.

In essence, the fundamental concept of the second step is to achieve invariance in the model predictions when presented with the same image but subjected to various data and feature perturbations. The data augmentation perturbations encompass weak and strong approaches. The weak data augmentation comprises random horizontal flip, vertical flip, and rotation. And color jitter serves as the approach for strong data augmentation perturbations, encompassing adjustments in brightness, contrast, saturation, and hue. Figure 5 presents visualizations of both weak and strong data augmentations. In this study, the parameters for brightness, contrast, saturation, and hue were uniformly set to 0.5. Consequently, the brightness, contrast, and saturation of the image were randomly adjusted within a range of 50–150% relative to the original image’s levels, whereas the hue factor was randomly altered within the range of −0.5 to 0.5. The network main outputs for the weakly augmented data and strongly augmented data should share the same label. Regarding feature perturbations, we introduce a straightforward channel dropout technique with a 50% probability (implemented as nn.Dropout 2d (0.5) in PyTorch) into the auxiliary decoder. This dropout is applied to the extracted features of weakly perturbed images. The network main output and auxiliary output for the weakly perturbed images should share the same label. This study employs the consistency loss to enforce the consistency of the model predictions.

Figure 5. Visualizations of the weak and strong data augmentations.

Specifically, for the labeled samples input $x_l$, they are transformed with weak data augmentation and propagated forward through the encoder and the main decoder (LinkNet50) to obtain the model prediction $p_l$, and the supervised loss between the predictions $p_l$ and the ground truth $y_l$ is calculated through Equation (3)(3) $L_s=\frac{1}{B}\sum _{\mathit{b}=1}^B\left(L_{\mathit{BCE}}\left(p_l,y_l\right)+L_{\mathit{DCL}}\left(p_l,y_l\right)\right)$(3) :

(3) $L_s=\frac{1}{B}\sum _{\mathit{b}=1}^B\left(L_{\mathit{BCE}}\left(p_l,y_l\right)+L_{\mathit{DCL}}\left(p_l,y_l\right)\right)$(3)

where $L_{\mathit{BCE}}$ and $L_{\mathit{DCL}}$ respectively represent the binary cross-entropy (BCE) loss and dice coefficient loss, which are widely used for the road segmentation task.

For the unlabeled images input $x_u$, in the branch generating the pseudo-labels, $x_u$is transformed with weak data augmentation, and as Equation (4)(4) $\hat{p}=\left\{\begin{array}{c}0,\left(\mathit{if}{p}_{\mathit{wu}}\in \left[0,0.5\right)\right)\\ 1,\left(\mathit{if}{p}_{\mathit{wu}}\in \left[0.5,1\right]\right)\end{array}\right.$(4) shows, the model main output $p_{\mathit{wu}}$ is used to produce the pseudo-label $\hat{p}$ according to the threshold $\mathit{\tau }$, where $\mathit{\tau }$ is set to 0.5, which is a default setting for binary segmentation tasks. In the data perturbation branch, $x_u$ is transformed with weak data augmentation, which is followed by strong data augmentation, and the model main output is $p_{\mathit{su}}$, and then the consistency loss of data perturbation $L_u^d$ is calculated between $p_{\mathit{su}}$ and $\hat{p}$ through Equation (5)(5) $L_u^d=\frac{1}{\mathit{\mu B}}\sum _{\mathit{b}=1}^{\mathit{\mu B}}(1\left(p_{\mathit{wu}}\ge {\tau }_1\right)|1\left(p_{\mathit{wu}}\le {\tau }_2\right))(L_{\mathit{BCE}}\left(p_{\mathit{su}},\hat{p}\right)+L_{\mathit{DCL}}\left(p_{\mathit{su}},\hat{p}\right))$(5) :

(4) $\hat{p}=\left\{\begin{array}{c}0,\left(\mathit{if}{p}_{\mathit{wu}}\in \left[0,0.5\right)\right)\\ 1,\left(\mathit{if}{p}_{\mathit{wu}}\in \left[0.5,1\right]\right)\end{array}\right.$(4)

(5) $L_u^d=\frac{1}{\mathit{\mu B}}\sum _{\mathit{b}=1}^{\mathit{\mu B}}(1\left(p_{\mathit{wu}}\ge {\tau }_1\right)|1\left(p_{\mathit{wu}}\le {\tau }_2\right))(L_{\mathit{BCE}}\left(p_{\mathit{su}},\hat{p}\right)+L_{\mathit{DCL}}\left(p_{\mathit{su}},\hat{p}\right))$(5)

where ${\tau }_1$ is set to 0.9 and ${\tau }_2$ is set to 0.1 to filter out any noisy predictions with low confidence (the real label of road is 1, and 0 is for background).

In the feature perturbation branch, $x_u$ is transformed with weak data augmentation, and propagated forward through the encoder and the auxiliary decoder to obtain the auxiliary output $p_{\mathit{fu}}$. Then, the consistency loss of feature perturbation $L_u^f$ is calculated between $p_{\mathit{fu}}$ and $\hat{p}$ through Equation (6)(6) $L_u^f=\frac{1}{\mathit{\mu B}}\sum _{\mathit{b}=1}^{\mathit{\mu B}}(1\left(p_{\mathit{wu}}\ge {\tau }_1\right)|1\left(p_{\mathit{wu}}\le {\tau }_2\right))(L_{\mathit{BCE}}\left(p_{\mathit{fu}},\hat{p}\right)+L_{\mathit{DCL}}\left(p_{\mathit{fu}},\hat{p}\right))$(6) :

(6) $L_u^f=\frac{1}{\mathit{\mu B}}\sum _{\mathit{b}=1}^{\mathit{\mu B}}(1\left(p_{\mathit{wu}}\ge {\tau }_1\right)|1\left(p_{\mathit{wu}}\le {\tau }_2\right))(L_{\mathit{BCE}}\left(p_{\mathit{fu}},\hat{p}\right)+L_{\mathit{DCL}}\left(p_{\mathit{fu}},\hat{p}\right))$(6)

The whole optimization function is then $\mathit{L}=L_s+L_u^d+L_u^f$.

To more clearly illustrate the training procedure for the pseudo-label guided global-local road extraction framework, the details are provided in Algorithm 1.

Table

Algorithm 1 Training procedure for the pseudo-label guided global-local road extraction framework.
Model: Encoder, Main decoder, Auxiliary decoder; Model parameters: $\mathit{\rho }$
Weak data augmentation: $\mathit{W}$, Strong data augmentation: $\mathit{S}$
Step1: use GRSet to pre-train Encoder and Main decoder (LinkNet50).
Input: OSM-annotated GRSet ${\left\{X,Y\right\}}_{i=1}^N$.
for j = 1 to epochs do
Get the mini-batch training samples: $\mathit{x},\mathit{y}.$
LinkNet50 forward: $\mathit{p}$= LinkNet50($\mathit{W}$($\mathit{x}$))
Calculate the segmentation loss:$L_s\left(\mathit{p},\mathit{y}\right)$ and update $\mathit{\rho }$.
end
obtain the pre-trained LinkNet50
Step2: update GRNet^SF parameters using target region samples.
Input: target region labeled samples ${\left\{X_l,Y_l\right\}}_{\mathit{i}=1}^{N_l}$ and unlabeled images ${\left\{X_u\right\}}_{\mathit{i}=1}^{N_u}$
for k = 1 to iterations do
Get the mini-batch training samples: $x_l,y_l$, and unlabeled images $x_u$
LinkNet50 forward: $p_l$= LinkNet50($\mathit{W}$($x_l$)), $\hat{p}$ = threshold (LinkNet50($\mathit{W}$($x_u$))), $p_{\mathit{su}}$ = LinkNet50($\mathit{S}$($x_u$)), $p_{\mathit{fu}}$ = Auxiliary decoder (Encoder ($\mathit{W}$($x_u$)))
Calculate the loss:$L_s\left(p_l,y_l\right)$ + $L_u^d\left(p_{\mathit{su}},\hat{p}\right)$ + $L_u^f\left(p_{\mathit{fu}},\hat{p}\right)$ and update $\mathit{\rho }$.
end

5. Experiments and analyses

5.1. Experimental settings

Global-scale validation areas and data: the global-scale validation set was made up of the four test sets from the Massachusetts, RoadTracer, SpaceNet, and DeepGlobe road datasets, and the self-annotated test sets from the Boston, Birmingham, Helsinki, Kigali, Coast Province, Beijing, Shanghai, Wuhan, and LoveDA road datasets (Wang et al. 2021). The LoveDA dataset contains 2522 samples for training and 1669 samples for validation. Figure 6 displays the geographic distribution of the global-scale validation set, covering four continents – Europe, Africa, Asia, and North America – involving both developed and developing countries. The 1669 LoveDA test images cover the Chinese cities of Nanjing, Changzhou, and Wuhan, with 992 images from rural areas and 677 images from urban areas. The RoadTracer test set contains data from Boston and Paris, and the SpaceNet test set contains data from Paris and Shanghai. Therefore, the global- scale validation set covers more than 30 cities, with different spatial resolutions and different types of roads, and the details are given in Table 2.

Figure 6. Geographical locations of the validation sets. The validation set covers the four continents of Europe, Africa, Asia, and North America.

Table 2. The details of the validation set of global road networks.

Road dataset	Acquisition time	Resolution (m/pixel)	Width (m)	Size (pixels)	Images	Cities	Types	Area (km^2)
Massachusetts test set	2013	1	7	1500×1500	49	>1	urban	110.25
RoadTracer test set	2018	0.6	6	1024×1024	1920	15	urban	724.78
SpaceNet test set	2018	0.3	2	1300×1300	567	4	urban	86.24
DeepGlobe test set	2018	0.5	–	1024×1024	1530	>3	urban/rural	401.08
Boston	2019	0.4411	–	23,104×20,480	1	1	urban	92.06
Birmingham	2019	0.3640	–	22,272×22,464	1	1	urban	66.29
Helsinki	2018	0.5	–	36,864×23,552	1	1	urban	217.06
Kigali	2021	1.2	–	20,480×20,480	1	1	rural	603.98
Coast Province	2021	0.6	–	20,480×20,480	1	1	rural	150.99
Beijing	2019	0.5	–	20,480×20,480	1	1	urban	104.86
Shanghai	2019	0.5	–	16,768×12,544	1	1	urban	52.58
Wuhan	2019	0.5	–	18,272×15,040	1	1	urban	68.70
LoveDA road test set	2016	0.3	–	1024×1024	1669	3	urban/rural	157.51
Total	–				5743	>30	–	2836.38

Implementation details: All the experiments were conducted on the PyTorch (Paszke et al. 2019) framework using an NVIDIA Titan RTX GPU accelerator with 24-GB memory. For the first step, random crops of 768 × 768 pixels with horizontal, vertical, and diagonal flipping were used. The training epochs were set to 12. The initial learning rate was set to 2e − 4, and was divided by 5 at epochs $\left\{7,9,11\right\}$. The segmentation loss was the sum of the Binary Cross Entropy (BCE) loss and dice coefficient loss. In addition, to avoid the interference caused by inaccurate road boundaries, when taking the GRSet dataset as the training set, the regions within 8 m (8 pixels) from the left and right road boundaries were not involved in the model optimization. To ensure the balance of positive and negative samples, we randomly sample a number of negative samples close to the number of positive samples (roads) from the background to optimize the model together, as shown in Figure 7.

Figure 7. Illustration of sampling of GRSet.

For the second step, the few labeled samples and unlabeled images from the target region were used to train the model for 40,000 steps, with both weak data augmentation and strong data augmentation. The model was trained using the Stochastic Gradient Descent (SGD) optimizer with the initial learning rate of 1e − 3, the weight decay was 5e − 4, and the crop size $\mathit{s}$ was set to 512. For the hyperparameter, τ was set to 0.9 and the initial value of $\mathit{\mu }$ was set to 2, and thus the batch size was set to 4 for labeled samples and 8 for unlabeled images.

For model evaluation, four metrics are applied to calculate the road extraction accuracy: Precision (P), Recall (R), F1-score, and IoU.

Comparison methods: to demonstrate the superiority of GRSet over the other road datasets, four advanced road extraction models: LinkNet (Chaurasia and Culurciello 2017), D-LinkNet (L. Zhou, Zhang, and Ming 2018), FarSeg (Zheng et al. 2020) and GAMSNet (Lu, Zhong, Zheng, and Zhang 2021) were selected. Among them, LinkNet and D-LinkNet showed great advantages in the DeepGlobe Road Extraction Challenge. FarSeg is a remote sensing object segmentation network by foreground-scene relation modeling. GAMSNet improves the road recognition rate by using a global awareness operation to capture long-range information. All of these models used an ImageNet pre-trained ResNet50 network as the backbone, and are denoted as LinkNet50_IN, D-LinkNet_IN, FarSeg_IN, and GAMSNet_IN, respectively, consistent with the rest of this paper.

To evaluate the generalization ability of GRSet and GRNet^SF, 14 comparison algorithms were used, with LinkNet50 used as the baseline model. As listed in Table 3, the comparison algorithms included UDA methods, semantic segmentation methods, and SSL methods, with different initialization approaches. For example, LinkNet50, LinkNet50_IN, and LinkNet50_GRSet respectively represent the LinkNet50 with PyTorch’s default initialization (P) (He et al. 2015), ImageNet pre-trained initialization (I) (Russakovsky et al. 2015), and GRSet pre-trained initialization (G).

Table 3. Initialization details of the comparison methods.

Type	Method	Model initialization parameters
		P	I	G
Unsupervised domain adaptation	OSN (Tsai et al. 2018)		✓
	MaxSquare (M. Chen, Xue, and Cai 2019)		✓
	GOAL (Lu, Zhong, Zheng, and Wang 2021)		✓
Semantic segmentation	LinkNet50	✓
	LinkNet50_IN		✓
	LinkNet50_GRSet			✓
Semi-supervised learning	MT (Tarvainen and Valpola 2017)			✓
	CCT (Ouali, Hudelot, and Tami 2020)			✓
	GRNet^S	✓
	GRNet^S_IN		✓
	GRNet^S_GRSet			✓
	GRNet^SF	✓
	GRNet^SF_IN		✓
	GRNet^SF_GRSet			✓

GRNetS means using only strong data augmentation perturbation, and GRNetSF means using strong data augmentation perturbation and feature perturbation.

5.2. Experiment 1: the impact of different road training sets on model performance

This section aims to analysis the impact of different road training sets on model performance, and Figure 8 presents quantitative IoU comparisons among the five road datasets. Blue numbers indicate test results obtained when the training and test sets were sourced from the same dataset, while red numbers indicate the optimal test results achieved when training models on GRSet. The analysis of Figure 8 reveals the following findings:

1. Although the models trained on the Massachusetts, RoadTracer, SpaceNet, and DeepGlobe datasets obtain the best results when trained on their own training sets (blue numbers), they show a poor generalization ability when performing cross-region road extraction. Notably, the models trained on DeepGlobe demonstrate obvious advantages in terms of generalization capability. For the DeepGlobe→DeepGlobe and DeepGlobe→LoveDA tasks, D-LinkNet_IN (0.6662 and 0.3369), FarSeg_IN (0.6665 and 0.3613), and GAMSNet_IN (0.6826 and 0.3797) show a stable superiority, compared to the baseline, i.e. LinkNet50_IN (0.6630 and 0.3308). This is because the DeepGlobe dataset exhibits greater diversity, encompassing accurately annotated road pixels. However, the road labels in the Massachusetts, RoadTracer, SpaceNet, and GRSet datasets are derived from OSM road centerlines with uniform width, leading to a significant amount of contextual noise.

2. GRSet significantly enhances the model’s generalization ability and yields the most comprehensive and contiguous road extraction results. Figure 9 visualizes the results of LinkNet50_IN on the Birmingham image, providing a visual comparison of the effects of different road datasets. The models trained on the Massachusetts and SpaceNet training sets struggle to identify road areas, whereas those trained on the RoadTracer and DeepGlobe training sets yield a relatively comprehensive road network due to the greater diversity of these two training sets compared to Massachusetts and SpaceNet.

3. For the GRSet→DeepGlobe and GRSet→LoveDA tasks, FarSeg_IN (0.2047 and 0.2428) and GAMSNet_IN (0.2348 and 0.2668) obtain a lower accuracy compared to LinkNet50_IN (0.3392 and 0.4184) and D-LinkNet_IN (0.3930 and 0.4238). Since FarSeg_IN and GAMSNet_IN primarily concentrate on modeling the relationship between road objects and contextual information. Thus, the inclusion of noisy contextual information in GRSet leads to significantly lower accuracy. Additionally, the DeepGlobe and LoveDA road datasets encompass numerous narrow rural roads that are particularly susceptible to the interference caused by such noisy contextual information.

Figure 8. Quantitative IoU comparison of models trained on the different road datasets.

Figure 9. The visual outputs for the Birmingham image produced by LinkNet50_IN with different training sets.

5.3. Experiment 2: validation of the proposed GRNet^SF_GRSet framework

This section aims to verify the effectiveness of the proposed GRNet^SF_GRSet framework, and the experiments are conducted on the SpaceNet and DeepGlobe datasets. For the semi-supervised training samples setting, 50, 100, and 200 labeled samples and the remaining unlabeled images were selected from the training set to train the model. Figure 10 gives the quantitative IoU accuracy, and the oracle setting was designed to test the upper limit of the accuracy, where the training set and test set were from the same dataset, and it is consistent with Figure 8. From Figure 10, there are three noticeable observations: 1) The UDA methods, namely OSN (0.3360 and 0.3832), MaxSquare (0.3347 and 0.4064), and GOAL (0.3309 and 0.4162), exhibit unsatisfactory performances primarily due to the noisy nature of the GRSet dataset, which contains missing and misregistered roads as depicted in Figure 3. Hence, GRSet is not suitable for standalone use as a training set; instead, it is better suited for joint learning with precisely labeled samples from the target region. By leveraging these accurate samples, the semi-supervised GRNet^SF_GRSet and GRNet^S_GRSet can be guided toward improved accuracy. 2) In general, the performance of pre-trained models on GRSet surpasses that of the ImageNet pre-trained models. Since the GRSet dataset is specifically designed for road extraction, it can be effectively utilized for extracting roads in the target region through fine-tuning. 3) GRNet^SF_GRSet (0.6006 and 0.6134) and GRNet^S_GRSet (0.5854 and 0.6034) significantly enhances the model’s generalization ability, compared to other SSL methods like MT (0.5710 and 0.5907) and CCT (0.5698 and 0.5958), and supervised method: LinkNet50_GRSet (0.5601 and 0.5860), and its performance improves gradually as the number of labeled training samples increases. Notably, the accuracy of GRNet^SF_GRSet is closest to the optimal performance achieved by the oracle setting (0.6220 and 0.6630), demonstrating the effectiveness of the proposed semi-supervised framework.

Figure 10. The histogram illustrates the quantitative results obtained on the SpaceNet and DeepGlobe dataset.

Figure 11 presents the visualization results of LinkNet50, LinkNet50_GRSet, GRNet^S_GRSet and GRNet^SF_GRSet on the 50/2163 training set division, along with the visualization results of the oracle setting. The visualization results align with the quantitative results, demonstrating that employing a GRSet pre-trained model can significantly enhance model performance. GRNet^S and GRNet^SF further refine the results, approaching the optimal performance of the oracle setting. In summary, the outputs of GRNet^SF_GRSet exhibit the highest level of completeness and closest resemblance to the labeled data, even when utilizing only 50 annotated samples from the target region.

Figure 11. The visualization results of different road datasets, where the first and second rows respectively represent the results obtained on the SpaceNet and DeepGlobe datasets.

Figure 11 presents the visualization results of LinkNet50, LinkNet50_GRSet, GRNet^S_GRSet and GRNet^SF_GRSet on the 50/2163 training set division, along with the visualization results of the oracle setting. The visualization results align with the quantitative results, demonstrating that employing a GRSet pre-trained model can significantly enhance model performance. GRNet^S and GRNet^SF further refine the results, approaching the optimal performance of the oracle setting. In summary, the outputs of GRNet^SF_GRSet exhibit the highest level of completeness and closest resemblance to the labeled data, even when utilizing only 50 annotated samples from the target region.

5.4. Experiment 3: global-scale road extraction applications

Due to space limitations, this section solely presents the qualitative and quantitative results of LinkNet50, GRNet^S_GRSet and GRNet^SF_GRSet for global-scale road extraction. Figure 12 lists the quantitative results for the large-scale images, revealing that GRNet^SF_GRSet exhibits an IoU performance improvement of over 10% compared to LinkNet50, with the exception of the Kigali and Coast Province images. Figure 13 depicts the cross-region visualization results, revealing that LinkNet50 frequently generates fragmented roads due to the interference of trees, buildings, or surrounding objects with similar textures. In such case, GRNet^S_GRSet and GRNet^SF_GRSet can obtain continuous and smooth roads, confirming the superiority of the GRSet and the proposed framework. In some details, GRNet^SF_GRSet is superior to GRNet^S_GRSet, as shown in Figure 13 (d), (h). Conversely, the Kigali and Coast Province images predominantly consist of mountain dirt roads, as depicted in Figure 13 (d), (e), which represent a road type of lesser prevalence in the GRSet dataset, thereby diminishing the discernible advantages of GRNet^SF_GRSet in these specific areas.

Figure 12. The Histogram illustrates the quantitative results of the large-scale images.

Figure 13. The local visualization results of the large-scale images.

Moreover, the performance of the GRNet^SF_GRSet framework in urban and rural areas and in countries with different levels of development is analyzed and discussed below.

1) Performance of GRNet^SF_GRSet in urban and rural areas: the LoveDA validation set, containing 1669 samples, of which 992 are from rural areas and 677 from urban areas, covering an area of 157.51 km², was used to evaluate the model performance in urban and rural areas. For the semi-supervised training set, it contains 50 annotated samples (urban: 25, rural: 25) and 2472 unannotated images (urban: 1131, rural: 1341). Figure 14 lists the quantitative results obtained on the LoveDA validation set, which gives the urban, rural, and overall accuracy, respectively. It can be found that: 1) The accuracy for urban areas is always higher than that for rural areas, because rural roads are narrow and recognition is more challenging. 2) Pre-training can greatly improve the model performance, and pre-training with GRSet gains a greater improvement than pre-training with ImageNet in urban areas and overall accuracy. In rural areas, GRNet^SF_GRSet is inferior to GRNet^SF_IN, and GRNet^S_GRSet is inferior to GRNet^S_IN. One reason for this is that the GRSet dataset retains the six types of roads which mainly exist in urban areas, and another reason may be that the labels of the rural roads are not entirely accurate and cannot strictly reflect the accuracy of the model, as shown in Figure 15. 3) In the case of a small number of samples, GRNet^SF_GRSet can significantly improve the model performance, especially for rural roads. For instance, from LinkNet50 to GRNet^SF_GRSet, the IoU performance is improved by 18.93% and 26.64% in urban and rural areas, respectively. This is because, in the GRNet^SF_GRSet method, 2472 target region images are applied to generate pseudo-labels and participate in the model optimization, which significantly improves the model performance in the target region. In contrast, the UDA methods of OSN, MaxSquare, and GOAL perform poorly, and even worse than the baseline LinkNet50, which uses only 50 samples of the target region. This confirms that target region samples are crucial for model performance improvement, and the semi-supervised method provides the possibility for global-scale road extraction application.

Figure 14. The Histogram illustrates the quantitative results obtained on the LoveDA dataset.

Figure 15. The visual urban and rural results obtained on the LoveDA road validation set.

Figure 15 shows the visualization results of LinkNet50, LinkNet50_GRSet, GRNet^S_GRSet and GRNet^SF_GRSet for urban and rural areas. The first urban image reflects the typical building style of a developing country such as China, which is dense and tall. As the magnified urban image shows, although there are no labels for the residential roads between buildings, the GRSet dataset retains residential road types, so that LinkNet50_GRSet, GRNet^S_GRSet and GRNet^SF_GRSet produce residential roads, and the prediction results of GRNet^SF_GRSet are smoother and more continuous, and are the most consistent with the real image. Especially at intersections, GRNet^SF_GRSet can successfully connect residential roads to main roads. In the rural image, there are mainly narrow roads in fields, for which the texture is very similar to that of the surrounding fields. In this case, as the magnified rural image shows, LinkNet50 produces badly fragmented roads, whereas the LinkNet50_GRSet results are improved, but there are still a lot of false positives. Meanwhile, the predictions of GRNet^S_GRSet and GRNet^SF_GRSet fit perfectly with the real image, even though the labels were not fully labeled. As shown in the red circle, the LinkNet50_GRSet predicted roads are broken due to the shadows caused by the surrounding trees, while GRNet^S_GRSet and GRNet^SF_GRSet obtain continuous roads by using the SSL technique. All the above results confirm the effectiveness of the proposed pseudo-label guided global-scale road extraction framework.

2) Performance of the GRNet^SF_GRSet framework in countries with different levels of development: The LoveDA dataset represents the typical style of a developing country, with both urban asphalt and rural dirt roads, which are easily obscured by the surrounding tall buildings and trees. For developed countries, the Birmingham image, which covers an area of 66.29 km², was chosen for the verification. The corresponding training set contains 3,000 samples of 1024 × 1024 pixels, the labels were generated by freely available OSM road centerlines, and the road width is fixed to 7 m, referring to (Lu, Zhong, and Zhang 2022). Thus, there are 50 labeled samples and 2,950 unlabeled images in the semi-supervised training set. The quantitative results are shown in Figures 16 and 17 gives the visual prediction results. Differing from developing countries, developed countries mainly feature asphalt roads, which are easier to identify, so that the IoU of GRNet^SF_GRSet without pre-training reaches 0.5992. In addition, buildings in developed countries are mostly low and roads are often obscured by the surrounding trees, as the magnified images show. In this case, LinkNet50 obtains many fragmented roads, while the predicted roads of LinkNet50_GRSet are relatively complete, but not perfect in some details. Meanwhile, the road outputs of GRNet^S_GRSet and GRNet^SF_GRSet are complete and continuous, and the results most closely resemble the labels, which verifies the effectiveness of the proposed framework.

Figure 16. The histogram illustrates the quantitative results obtained on the Birmingham image.

Figure 17. The visual results obtained on the Birmingham image.

Generally speaking, roads in developed countries are easier to recognize due to the reduced building shelter and more complete road facilities. Roads in developing countries are generally more difficult to identify because the buildings in urban areas are typically dense and tall, and can heavily obscure roads, while the rural roads are typically narrow. This situation is also consistent with the existing OSM road data. Developed countries typically feature completed urbanization, and their OSM road data are relatively complete and accurate, while there are still many roads in underdeveloped countries that need to be added to the existing OSM data, especially rural roads.

6. Sensitivity analysis

In the proposed GRNet^SF, $\mathit{\mu }$ is the ratio of unlabeled to labeled data. In this section, we describe how parameter $\mathit{\mu }$ was analyzed based on the LoveDA and Birmingham data. Figure 18 respectively gives the IoU performance variation of GRNet^S, GRNet^SF, GRNet^S_IN, GRNet^SF_IN, GRNet^S_GRSet and GRNet^SF_GRSet when $\mathit{\mu }$ was set to 1, 2, and 3. From the trendline of the IoU, it can be seen that the value of $\mathit{\mu }$ has a greater influence on the LoveDA data, but is relatively stable on the Birmingham data. Moreover, the IoU does not increase with the increase of $\mathit{\mu }$, but maintains a stable advantage when $\mathit{\mu }$ = 2. Thus, in this study, $\mathit{\mu }$ was set to 2, and the model input at each step consisted of four labeled samples and eight unlabeled images.

Figure 18. The IoU variation with different values of $\mathit{\mu }$.

7. Conclusions

The primary objective of this study was to enhance the generalization capability of a deep road extraction network. To alleviate this problem, we introduced a pseudo-label guided global-scale road extraction framework called GRNet^SF_GRSet. The GRNet^SF_GRSet framework was constructed based on two key components. Firstly, a robust global-scale road benchmark dataset called GRSet was built, comprising 47,210 samples from 121 capital cities across six continents. This dataset provides valuable insights into the model’s performance across various regions worldwide. Secondly, a pseudo-label guided global-scale road extraction network named GRNet^SF was developed to enable the model’s adaptation to new cross-region VHR satellite imagery globally. Experimental evaluations conducted in over 30 cities across four continents revealed that GRSet surpasses existing road datasets in terms of generalization capability. Additionally, the semi-supervised GRNet^SF approach can further enhance the model’s generalization capability by employing strong data augmentation perturbations and feature perturbations strategy. We further explored and analyzed the performance of the proposed GRNet^SF_GRSet framework across countries with varying levels of development, encompassing both urban and rural areas. Our findings reveal that identifying rural roads poses notably greater challenges due to their narrowness and the surrounding interference, particularly in developing nations. Therefore, future research could focus more on the recognition of roads in developing countries and rural areas.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available at https://github.com/xiaoyan07/GRNet_GRSet from the author [X. Lu] and the corresponding author [Y. Zhong] ([email protected]), upon reasonable request.

Additional information

Funding

This work was supported by National Key Research and Development Program of China [grant number 2022YFB3903404], and National Natural Science Foundation of China [grant number 42325105].

Notes on contributors

Xiaoyan Lu

Xiaoyan Lu is an IEEE member and a postdoctoral fellow at The Hong Kong Polytechnic University. She received the PhD degree from Wuhan University, in 2022. Her research interests include intelligent interpretation and application of remote sensing images and medical images. She was the recipient of the 2021 John I. Davidson President’s Award (First Place).

Yanfei Zhong

Yanfei Zhong is an IEEE senior member and a full professor with the State Key Laboratory of Information Engineering in Surveying, Mapping and Remote Sensing (LIESMARS), Wuhan University. He organized the Intelligent Data Extraction, Analysis and Applications of Remote Sensing (RSIDEA) research group. His research interests include hyperspectral remote sensing information processing, high-resolution remote sensing image understanding, and geoscience interpretation for multisource remote sensing data and applications. Dr. Zhong is a Fellow of the Institution of Engineering and Technology (IET). He is currently serving as an Associate Editor for the IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, and the International Journal of Remote Sensing.

Zhuo Zheng

Zhuo Zheng is a postdoctoral fellow at Stanford University. He received the PhD degree from Wuhan University, in 2023. His major research interests remote sensing visual perception and Earth vision, especially multimodal and multitemporal remote sensing image analysis.

JunJue Wang

JunJue Wang is a PhD candidate at Wuhan University. His major research interests are high-resolution remote sensing imagery semantic segmentation and computer vision.

Dingyuan Chen

Dingyuan Chen is a PhD candidate at Wuhan University. His research interests include building extraction with multisource remote sensing imagery and deep learning-based object detection.

Yu Su

Yu Su received the PhD degree from Wuhan University, in 2023. Her major research interests include urban scene understanding for high-resolution remote sensing imagery and land use mapping based on multisource geographic data.