Automatic extraction and 3D modeling of real road scenes using UAV imagery and deep learning semantic segmentation

ABSTRACT

The extraction of roads from UAV images is challenged by lighting, noise, occlusions, and similar non-road objects, making high-quality road extraction difficult. To addressing these issues, this study proposes an enhanced U-Net network to automate the extraction and 3D modeling of real road scenes using UAV imagery. Initially, a cascaded atrous spatial pyramid module was integrated into the encoder to capitalize on global context information, thereby refining the fuzzy segmentation outcomes. Subsequently, a module for augmenting road feature extraction was added within the channel, and a spatial attention mechanism was introduced in the decoder to enhance edge clarity. Experimental results demonstrated that this model captures more road information compared to mainstream networks and effectively incorporates topological structure perception for road extraction in complex scenarios, thus improving road connectivity. The model achieved an F1 score and mean Intersection over Union (mIoU) of 85.6% and 81.2%, respectively, on UAV images of road scenes – marking improvements of 3.9% and 3.4% over the traditional U-Net model, thereby exhibiting superior automatic road extraction capabilities. Ultimately, the model facilitated refined modeling and visual analysis of road scenes, achieving high overall accuracy and detailed local restoration of the actual scene.

KEYWORDS:

• Real road scenes
• UAV imagery
• semantic segmentation
• improved U-Net
• 3D reconstruction

1. Introduction

Traffic management, disaster management, and public health all rely on road information (Guan et al. 2016; Liu, Yang, and Gu 2023). Meanwhile, as unmanned aerial vehicles (UAVs) become more widely used (Fan et al. 2020; Altuntabak and Ata 2022), UAV remote sensing (RS) plays a vital role in fine-tuning surface attachment measurements such as roads and three-dimensional reconstruction of urban buildings (Liu et al. 2024). Currently, UAV images are being used to extract geospatial road information. Traditional methods for extracting roads from high-resolution images are time-consuming and labor-intensive, and the study of automatic extraction method will effectively improve the efficiency of road acquisition and update. The UAV is small in size, easy to carry and low in ground support requirements, which greatly reduces the operation and maintenance costs of the UAV (Wang et al. 2014; Hastaoğlu, Göğsu, and Yavuz 2022). Compared with road information acquisition methods such as vehicle-mounted cameras, the road information acquisition method based on UAV images has obvious advantages, and effectively solves the problems of small information acquisition area, low image resolution and poor mobility (Samad et al. 2013; Akay and Ozcan 2021). However, it is very difficult to analyze a large amount of road RS-data obtained by UAV using traditional methods (Olson and Anderson 2021; Qin et al. 2022; Wang et al. 2023), including spectrographic analysis (Gomez 2002), threshold segmentation (Singh and Garg 2013), knowledge model (Shen et al. 2008), etc. Due to the frequent necessity for manual intervention, these methods entail a substantial workload, proving both time-consuming and labor-intensive. Consequently, they fall short of achieving automated road extraction, particularly when compared with traditional methodologies. This reliance on manual processes significantly hampers efficiency and scalability in road mapping applications (Xu et al. 2018).

In recent years, after many scholars adopt convolutional neural networks (CNNs) to realize image recognition (Wang et al. 2022; Li et al. 2021), deep learning (DL) methods have gained more and more applications in RS image’s information extraction (Zhang et al. 2016; Liu et al. 2021). Minh and Hinton first applied DL method to extract road edges, using restricted Boltzmann machines to detect road in high-resolution RS images and achieved good results (Mnih and Hinton 2010; Liu et al. 2023). Li et al. (2016) predicted the probability graph of pixels belonging to road regions in RS images using CNNs, and then smoothed it by using linear integral convolution algorithm to retain the road edge information. Wang et al. (Wang et al. 2020) proposed a new coordinate convolution module to acquire multi-level spatial data. These path extraction methodologies predominantly concentrate on developing deeper network architectures or enhancing functional capabilities to optimize classifier construction. The global context awareness module stands as a pivotal element within these models, enabling the extraction of semantic information from the global context. This module facilitates the integration of this information to produce higher-level feature maps, thereby enriching the model's ability to discern and interpret complex scenes.

U-Net is currently one of the most effective approaches to semantic segmentation (Oktay et al. 2018). U-Net model uses two convolution layers in the encoder part for feature extraction and an average pooling layer for subsampling. Compared with this method, the ResNet model deepens the number of layers of the training model by adding a shortcut mechanism, which effectively avoids gradient disappearance and accelerates network convergence (He et al. 2016). In the process of using RS images to extract road context information, there have been studies that propose multi-scale and multi-level features for combining pixel category and spatial information (Chen et al. 2022; Liu et al. 2023). For example, PSPNet used the spatial pyramid module to gather multi-level features as context information (Zhao et al. 2017). Later, some studies used extended convolution to enlarge the receiving field to obtain context information. DeepLab v2 combined extended convolution with different expansion rates to propose an extended convolution module, which can capture a large range of multi-scale context information without increasing the computational burden (Chen et al. 2018). This method, however, does not adequately consider the overall geometry of the road, nor does it fully utilize contextual information. Throughout the down-sampling process, there is a progressive decline in image resolution, resulting in the loss of critical spatial information which proves challenging to recover. Consequently, this loss adversely affects the clarity of the extracted road edges, diminishing the precision of the segmentation outcome. By introducing the space void pyramid module into U-Net model, many scholars could acquire more global context information by integrating image feature information of different sizes (Fan et al. 2022; Liu, Su, and Lv 2022; Bayramoğlu and Melis 2023). Some studies also put forward an improved U-Net algorithm with improved attention mechanisms to promote the feature extraction of road edges and improve segmentation accuracy (Han et al. 2021; Su et al. 2022). Road extraction research can benefit from the above improved ideas.

In addition, cross-entropy loss function is often adopted for DL-based models (Tao et al. 2019). As a result, the loss caused in the process of obtaining information about curves such as roads and complex topologies is local, and the same penalty is applied to all errors, without considering the influence of geometric topological factors (Neven et al. 2018). However, road information in RS images often has a tiny pixel share and a strong topological structure. Therefore, this loss function is not optimal for road segmentation. During model training, the feature map encoded by the encoder will contain more feature information (He et al. 2019), including boundary, texture and spatial structure. However, the pooling operations employed during the encoding process inevitably led to a loss of semantic information and the generation of additional background noise. Conversely, feature maps that have not undergone encoding retain a greater amount of semantic information, which could potentially enhance the overall quality of the image analysis (Liu et al. 2019). The combination of un-coded and encoded feature images is very helpful for feature localization and noise suppression.

Besides, road model reconstruction using multi-angle images collected by UAVs has been widely used (Knyaz et al. 2020; Kalacska et al. 2020). In the process of 3D automatic modeling, the quality of the model is not only affected by various factors such as resolution and overlapping map, but also greatly affected by moving objects on the road, which often cause serious distortion and dislocation of the texture of the road model (Murtiyoso et al. 2020). Among the various factors affecting the model, the deformation caused by moving vehicles on the road is particularly significant. Therefore, it is crucial to focus on the fine modeling and visual management of road UAV images to address these dynamic changes effectively and enhance the accuracy of the representation.

In response to the analysis of the structural characteristics and limitations of the traditional U-Net network, an enhanced U-Net architecture is proposed. This revised network incorporates an Atrous Spatial Pyramid Pooling (ASPP) module at the juncture between the encoder and decoder to enhance segmentation accuracy. Additionally, a coordinate attention mechanism is integrated within the channel part to augment the capability of information extraction. Moreover, spatial attention is introduced in the decoder to improve the clarity of model segmentation. Ultimately, sophisticated 3D modeling software and a visualization platform are employed to accurately reconstruct the road scene.

As shown in Figure 1, Section 2 describes the image acquisition and preprocessing method. The road extraction method based on the improved U-Net model is given in Section 3. Then, Section 4 elaborated the modeling and visualization process of road scenes. Section 5 summarizes the experimental results and findings. Finally, Section 6 concludes this study.

Figure 1. Flowchart of the automatic extraction and 3D modeling of real road scenes.

2. Data preparation and preprocessing

The DJI Matrice M600 Pro hexacopter UAV was used as the image collection platform in this study, with a single photo resolution set to 7952 × 5304. Other hardware specifications are shown in Table 1.

Table 1. Hardware specifications for UAV road image collection.

Collection width (m)	Speed (m/s)	Shooting interval (s)	Number of Stitches	Working temperature (°C)	Flight control system
16.88	1.0	346	9	24	A3 Pro

As presented in Figure 2, two image sets were used for the model. Figure 2(a) shows a self-made UAV image set of a road collected in November 2022, and the shooting area is the North Ring road section of G312 National Highway in Wuxi, China. The image resolution is 0.1 m, which makes its size larger after stitching. Due to the use of large-size training images, the requirements for computer performance are higher, and the training phase will be more extended. This results in 1800 images, in which the training image is divided into 1024 × 1024 pixels blocks.

Figure 2. Road in remote sensing images and its annotation: (a) UAV-acquired, (b) DeepGlobe dataset, and (c) labels.

Figure 2(b) shows the DeepGlobe road extraction dataset derived from DeepGlobe 2018 (Demir et al. 2018). The training data for the Road Challenge consists of 6,226 RGB satellite images (1024 × 1024 pixels). The image has a 0.5 m resolution and was collected by Digital Globe’s satellites. One thousand two hundred of these samples were used in this study. In addition, the original images of all regions and the corresponding label images are arranged according to disorder and stored in different folders to avoid over-adapting to image features of a specific region during training stages.

Finally, 3000 data sets were established and randomly divided according to the ratio of 8:1:1, and 300 data sets were used for testing and result evaluation: path road and highway road, as shown in Figure 2(c). Labeling is divided into two labels according to different road characteristics: path road (road-P) and highway road (road-H). As a result, the dataset and labels information of roads in UAV images is given in Table 2.

Table 2. Road dataset information in UAV images.

Dataset	Labeled	road-H	road-P	No labeled	Total
Training set	2312	2864	1043	88	2400
Validation set	283	348	144	17	300
Testing set	276	346	128	24	300

3. Road extraction using the improved U-Net model

3.1. U-Net network

U-Net network structure comprises an encoder and a decoder with an encoder symmetry structure to realize down-sampling and up-sampling, respectively. Figure 3 describes its network structures. The encoder, also known as the compression path, is mainly used to extract image features. The compression path consists of four blocks, each of which uses three effective convolution and 1 maximum pooled down-sampling. Traditional sampling will reduce the resolution of the image. The decoder, also known as the extension path, is also composed of four blocks, each of which restores the size of the feature map by deconvolution and then merges it with the corresponding feature map of the encoder to form a new feature map. U-Net network, as a variant of FCN, has the advantage of full-CNN feature expression, and it solves the problem of rough feature maps extracted by FCN by reusing low-level semantic information through jumping connections.

Figure 3. Network structure of the original U-Net model.

3.2. Improved U-Net network

However, road extraction using U-Net would likely fail because that in RS images is complex and may be blocked. To enhance its anti-blocking ability, this study introduced the ASPP and fusion attention modules to improve the segmentation effect. Figure 4 illustrates the detail structure of improved model, which includes coding and decoding networks.

Figure 4. Network structure of the improved U-Net model.

The first part is the encoder part, which introduces a channel attention module in each channel to make the model channel relationship and adjust the weight of the channel dimension. The improved ASPP module is used to connect the bottom of encoder and decoder. Its purpose is to obtain multi-scale image context information using cavity convolution with different expansion rates, improve decoder classification accuracy, and increase model detail expression ability.

The second part is the decoder part, which introduces the spatial attention mechanism into the sampling module of each layer to help decoder recover the original feature map’s resolution. Thus, it could extract more and more complete road information.

3.2.1. Fusion attention mechanism module

(1)	Channel attention mechanism

Coordinate Attention (CA) (Hou, Zhou, and Feng 2021) is a new type of channel attention with the following advantages: an average pooling can introduce position information vertically and horizontally, which can adjust the weights along channel dimension and generate selective attention graphs to enhance the recognition and feature extraction of the road region, as shown in Figure 5. With fewer parameters and minimal increase in computational overhead, the approach remains highly flexible. Consequently, the CA module is integrated into each channel of the U-Net model to enhance road information extraction, minimize interference from irrelevant data, and improve segmentation outcomes.

Figure 5. Network structure of coordinate attention mechanism.

First, a feature graph U (H × W × C) is input into CA mode. After average pooling in the X and Y directions, a new feature graph (1× (H + W) ×C) is generated by splicing in the channel dimension. C/r is number of convolutional compression channels, r is the attenuation rate. Then, the nonlinear transformation operation is carried out by using the convolutional transformation function BN and the swich activation function. Feature graphs are then separated into above two feature graphs of size 1×H × C and 1×W × C and activated by the Sigmoid function to generate a pair of weights along and horizontally for each feature channel of the original input. Last, A new output feature graph U’ is created by multiplying learned weights with inputted feature graphs, then U’ has richer semantic information. The calculation process of this network is shown in Equation (1(1) $U^{\mathrm{\prime }}=U\times {\sigma }_1\times {\sigma }_2$(1) ). (1) $U^{\mathrm{\prime }}=U\times {\sigma }_1\times {\sigma }_2$(1) where, U and U’ are the input and out feature maps, respectively. σ₁ and σ₂ are channel weight along vertical and horizontal directions, respectively.

(2)	Spatial attention (SA) mechanism

Channels combine the high-level and low-level feature maps recovered by the decoder. At the same time, the original encoder part adopts traditional convolution method to extract features. There is a potential for interference from extraneous information, such as noise in the extracted feature maps, which can rapidly result in issues like edge blurring and road disconnection in road segmentation. To enhance the segmentation quality, an SA fusion module is incorporated into the decoder section of the model. This addition aims to concentrate on road features, improve their extraction, restore the continuity of roads, and enhance the clarity of their edges. Network structure of SA module is illustrated in Figure 6.

Figure 6. Network structure of SA module.

Firstly, pooling operations at the global maximum and average levels are carried out on feature graphs F, following spatial fusion, and then feature extraction is enhanced by a 7 × 7 convolution kernel, and the spatial weight value is obtained by Sigmoid activation function and multiplied with the original image pixel by pixel to get a new feature graph F’, as calculated in Equation (2(2) $F^{\mathrm{\prime }}=M_s(F)\otimes F,M_s(F)=\sigma (f^{7\times 7}([F_{max},F_{\mathrm{avg}}]))$(2) ). (2) $F^{\mathrm{\prime }}=M_s(F)\otimes F,M_s(F)=\sigma (f^{7\times 7}([F_{max},F_{\mathrm{avg}}]))$(2) where M_s(F) denotes the spatial weight value, σ represents the activation function, and f^7 × 7 denotes a 7 × 7 convolution operation.

3.2.2. ASPP module

Encoder outputs the most decadent semantic features in the last feature map. To enhance the feature utilization rate, an ASPP module is inserted following the feature map. This module utilizes dilated convolutions at various dilation rates to extract features, thereby capturing multi-scale contextual information. Additionally, it expands the receptive field without the addition of extra parameters, enabling the acquisition of a broader spectrum of feature information while preserving the original resolution. Different expansion rates can increase the filter's receptive field, as given in Equation (3(3) $S=(r-1)\times (k-1)+k$(3) ). (3) $S=(r-1)\times (k-1)+k$(3) where r is the expansion rate and k is the length of the filter. However, ASPP also has some shortcomings. It lacks mutual dependence in the results of a certain layer obtained by hollow convolution, which will cause local information loss and affect the classification results. Thus, ASPP module structure was improved by changing it from the original independent branch method to serial structure to realize more intensive pixel sampling, expand the receptive field, improve feature extraction ability, and avoid local information inconsistency. Figure 7 describes this improved ASPP module.

Figure 7. Network structure of the improved ASPP module.

The convolution of two voids can result in a larger receptive field. Equation (4(4) $S=S_1+S_2-1$(4) ) further gives the receptive field size after superposition: (4) $S=S_1+S_2-1$(4) where S₁ and S₂ are the respective receptive field sizes convolved by two voids. According to Equations (3(3) $S=(r-1)\times (k-1)+k$(3) ) and (4(4) $S=S_1+S_2-1$(4) ), the maximum receptive field of the original ASPP module is 37, and that of the improved ASPP module is 73. The enhanced module facilitates information sharing via skip-layer connections, while the use of dilated convolutions with varying dilation rates complements one another, thereby broadening the receptive field. This integration effectively increases the scope of feature detection and contributes to a more comprehensive understanding of the spatial context.

3.2.3. Loss function

It is a pixel-by-pixel classification issue for the DL-based model to determine whether a pixel is a road or a background. The proportion of roads in UAV images to the overall area of the image is small, and there are better choices than cross-entropy loss for such tasks. Thus, Dice coefficient (L_dice) and cross-entropy (L_cross) loss functions are combined to calculate loss function for this model. Equation (5(5) $L_{\mathrm{cross}}={\displaystyle \frac{1}{N}\sum _i-[(1\text{-}y)\times \log (1\text{-}f(x))+y\times \mathrm{logf}(x)]}$(5) ) shows how to calculate L_cross for binary classification (Mosinska et al. 2018; Liu et al. 2024). (5) $L_{\mathrm{cross}}={\displaystyle \frac{1}{N}\sum _i-[(1\text{-}y)\times \log (1\text{-}f(x))+y\times \mathrm{logf}(x)]}$(5) where y represents the corresponding real road label, ‘0’ and ‘1’ represent the pixel value of background and road, respectively. f is used to represent the parameterized form, then the output image output is y’ = f(x); N represents the number of pixel points.

Equation (6(6) ${\mathit{L}}_{\mathrm{dice}}=1-{\displaystyle \frac{2|X\cap Y|}{|X|+|Y|}}$(6) ) further shows how L_dice calculated (Liu et al. 2022). (6) ${\mathit{L}}_{\mathrm{dice}}=1-{\displaystyle \frac{2|X\cap Y|}{|X|+|Y|}}$(6) where X and Y, respectively represent the predicted graph and the actual label. |X∩Y| denotes the intersection between labels and prediction, |X| and |Y| indicate the number of labeled and predicted elements, respectively.

Thus, the final loss function L_loss is calculated in Equation (7(7) $L_{\mathrm{loss}}=L_{\mathrm{cross}}+L_{\mathrm{dice}}$(7) ). (7) $L_{\mathrm{loss}}=L_{\mathrm{cross}}+L_{\mathrm{dice}}$(7)

3.2.4. Experimental design and evaluation index

In this work, the training hardware configuration was A40 GPU, and the software configuration was PyTorch environment under Windows. For training models, the hyperparameters are set as below: The gradient descent method with the minimum batch of 4 was used for training. The initial learning rate is set to 0.00002, momentum is 0.9, and attenuation coefficient is 0.0001. The maximum batch is set to 200 (Liu et al. 2024c). Subsequently, the experiment employed the adaptive moment estimation (Adam) optimizer (Şen and Özkurt 2020) to adjust the model parameters and update the network weights, iterating 100 times. The learning rate was progressively decreased as the number of training iterations increased. To prevent the imbalance between positive and negative samples, a hybrid loss function as specified in Equation (7(7) $L_{\mathrm{loss}}=L_{\mathrm{cross}}+L_{\mathrm{dice}}$(7) ) was utilized.

To quantitatively compare the results, the evaluation indexes of Equations (8)–(11) (Liu et al. 2022; Liu et al. 2022; Liu, et al. 2024e) were adopted, which were precision (P), recall rate (R), F₁ score, average precision (AP), IoU, and mean IoU (mIoU), respectively. (8) $P={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}}$(8) (9) $R={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}}$(9) (10) $F_1={\displaystyle \frac{2\ast P\ast R}{P+R},\mathrm{AP}={\int }_0^{1}P(R)\mathrm{dR}}$(10) (11) $\mathrm{IoU}={\displaystyle \frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}+\mathrm{FP}},\mathrm{mIoU}={\displaystyle \frac{\mathrm{Io}{U}_{\mathrm{road}\text{-}H}+\mathrm{Io}{U}_{\mathrm{road}\text{-}P}}{2}}}$(11) where: TP represents correctly predicted pixels; FP represents pixels’ number whose predicted results are inconsistent with the real ones. FN represents pixels’ number that are not predicted. The values of the above indicators are all within the range of [0,1]. Road segmentation performs better when the value is close to 1.

4. 3D modeling of real road scenes

4.1. 3D modeling

To build the 3D model of the highway real scene, ContextCapture software was used to semi-automatically process the initial data of oblique photography and establish the 3D model of a multi-format real scene, and the preliminary results of the 3D model of the highway were obtained. ContextCapture software is one of the world's most commonly used 3D modeling software, which can semi-automatically generate high-quality, high-resolution 3D models based on high-definition images (the ground sampling resolution was 2.0 cm) (Cui and Wang 2023). It can generate geographic 3D models in any coordinate system and even in the custom plane system suitable for GIS applications. Authentic projective images and digital surface models that are compatible with all standard GIS tools can also be generated.

To optimize third-party visualization and analysis software for accurate 3D models, an open scene graph binary (OSGB) data format was selected. As shown in Figure 8, the oblique photography model of OSGB format also includes Data files and XML files composed of multiple sub-files. Each sub-file is an OSGB format file organized in a tree structure.

Figure 8. OSGB data format: (a) file structures and (b) organizational form.

The Data file adopts the PagedLOD mechanism based on OSG, which manages the 3D model data by establishing a top-down hierarchical pyramid structure. The PagedLOD organization form consists of Geode nodes and Group nodes. Leaf nodes are mainly used to store coordinates, textures, indexes, and other information for rendering and no longer contain child nodes. Group nodes can include any number of child nodes, and child nodes can also continue to distribute nodes, forming a large and orderly organization data structure, significantly improving rendering performances.

As shown in Figure 9(a), Context Capture software was used to process it and obtain the OSGB real 3D model data format based on the oblique photography data (WGS84 coordinate system) collected on the road section (the gray part), which was used for subsequent refinement, visualization and analysis software optimization. The tile set of production results contains 72 tiles (each tile includes approximately nine images), and Figure 9(b) shows its basic division rule. Starting from the top view of the model, it was divided equidistantly in the horizontal and vertical directions, respectively, to form several regular square tiles and rectangular tiles with edges and corners. In the tile file naming convention, the initial three digits represent the number of columns, while the subsequent set indicates the number of rows, ordered from bottom to top. This systematic naming facilitates the rapid identification and location of the 3D model corresponding to each tile.

Figure 9. 3D modeling process.

The third-party software SuperMap GIS can display the preliminary results of oblique photography models in OSGB format obtained by Context Capture software. Detailed steps are as follows:

1. The 3D slice cache layer of the oblique photography model was obtained through the generate profile function. The source path selects the Data file ‘Data,’ and the object path was consistent. The model reference point and projection settings were edited according to the < SRSOrigin > and < SRS > lines in the XML file. Click OK to get the 3D layer stored in SCP format (named ‘Config’);

2. Once the profile is complete, it can browse the tilt photography 3D model using the SuperMap iDesktopX software. A new spherical scene was created, and the newly generated Config 3D slice cache layer was inserted to show the preliminary results of the actual highway model in the scene.

4.2. Model refinement

In practical scenarios, the accuracy of 3D model results is often compromised due to interference from flight conditions, climatic environments, time of acquisition, and the image processing algorithms used. These factors can introduce flaws and defects into the 3D models. Consequently, it is necessary to modify and replace the preliminary models to achieve a refined and more accurate representation. This refinement process is crucial for enhancing the reliability and utility of the 3D models in practical applications.

As shown in Figure 10, DP-Modeler was selected as the refinement software. Before using DP-Modeler to repair the model, it is necessary to use DPSInManager software to create a new project, import OBJ format and OSGB format oblique photography data into the Mesh local model modification, and set the offset value according to the XML file. Import the JAS format file into DP-Modeler, and then it can start the local modification work of the model, such as road safety devices and bridge opening modification.

Figure 10. Workflow of the model refinement.

4.3. Model visualization

For GIS, the Web GIS development platform is commonly used as a 3D graphics engine to publish GIS spatial data and further realize the functions of retrieval, query, cartographic output and editing. Cesium uses the most advanced WebGL technology to achieve hardware acceleration (Liu et al. 2021), which can achieve smooth operation on HTML5 browsers without plug-ins and is suitable for displaying and simulating dynamic data on GIS layers. Thus, it was used for model visualization analysis.

ContextCapture Web View 2.0 is a Cesium-based custom Web application that enables user interface customization and real-world data display. By calling the HTML code of the 3D model, the URL can be generated, and the 3D model can be published on the web page. Any device running Cesium through a Node.js server can open the published highway 3D model visualizations directly from the web address without tilt photography data files.

5. Results and discussions

5.1. Automatic extraction results of roads

5.1.1. Results of road extraction

To verify the model performance, U-Net, FCNs, and Mask R-CNN networks were used for segmentation processing using the same testing dataset. Experimental results are shown in Figure 11.

Figure 11. Diagram of the detection results of road extraction for each model.

The distribution of roads is mainly straight lines or cross lines. Among them, the types of ground objects around the ‘road-H’ label are relatively consistent, and the prominence is somewhat apparent. The ‘road-P’ label is rather indistinct from the surrounding land because of its small area. In the figure, the yellow box indicates missed detection, and the red box indicates false detection. Compared with FCNs model, Mask R-CNN, U-Net, and improved U-Net models all showed better road segmentation effects. However, both the Mask R-CNN and U-Net models exhibited significant instances of missed and false detections, particularly at road intersections and with the ‘road-P’ label. In contrast, the improved U-Net model demonstrated substantially fewer missed and false detections, as well as a smaller erroneous pixel area, compared to other models. This indicates a more effective separation of the road-covered parts from the rest of the road, underscoring the superiority of the enhanced U-Net model in maintaining the integrity and accuracy of road segmentation.

5.1.2. Contrast analysis

To analyze the segmentation results more intuitively, the evaluation results of each segmentation model are calculated. The U-Net¹ model refers to the introduction of CA and SA modules into the original U-Net architecture, while U-Net² further incorporates an improved ASPP module to analyze the impact of the proposed improvements on the detection results.

Table 3 gives the segmentation results of different models. From the table, F₁ score, AP, and mIoU of improved U-Net network reached 85.6%, 82.9%, and 81.2%, respectively. Among these models, the proposed model’s IoU is 8.4% higher than the FCNs, 5.5% higher than the Mask R-CNN, and 3.4% higher than the U-Net models. F₁ score of improved U-Net model improved by 3.9% compared to original U-Net model, 5.5% compared to the Mask R-CNN model, and 10.1% compared to the FCNs model. AP of the improved U-Net model also improved by 5.6%.

Table 3. Comparisons of the testing results of four road extraction models.

	P		R		F1		IoU				mIoU
Models	road-H	road-P	road-H	road-P	road-H	road-P	road-H	road-P	AP	F1	road	Testing time (ms)
FCNs	0.827	0.734	0.836	0.695	0.831	0.714	0.831	0.625	0.746	0.755	0.728	25.77
Mask R-CNN	0.848	0.786	0.839	0.765	0.843	0.775	0.846	0.667	0.763	0.809	0.757	46.41
U-Net	0.865	0.803	0.821	0.762	0.842	0.782	0.843	0.712	0.773	0.817	0.778	35.43
U-Net1	0.886	0.825	0.856	0.775	0.871	0.799	0.864	0.732	0.812	0.839	0.803	36.04
U-Net2	0.894	0.831	0.862	0.781	0.878	0.805	0.871	0.739	0.821	0.848	0.807	36.58
Improved U-Net	0.901	0.832	0.875	0.786	0.888	0.808	0.876	0.748	0.829	0.856	0.812	36.58

Specifically, the indexes of the ‘road-H’ label were higher than those of the ‘road-P’ label because the road features of highways are more regular and the scope is broader, so the segmentation is more accurate. In addition, although the index of the ‘road-P’ label was not as good as that of ‘road-H,’ improved U-Net performance on the ‘road-P’ label was still satisfactory, with the P of the ‘road-H’ label reaching 90.1% and the P of ‘road-P’ label exceeding 80%, achieving the highest value among these models. Quantitative comparison results of these indicators suggested that improved U-Net network had improved road extraction accuracy from UAV images. The main reason is that by introducing CA, SA and improved ASPP modules in the improved U-Net model, the segmented road structure is more complete, more road information can be obtained, and the accuracy rate of road segmentation can be improved. A more detailed analysis revealed that the U-Net¹ model offers the most significant improvement over the original U-Net model, with an increase in AP of about 4%. The subsequent U-Net² model and the final improved U-Net model showed less than a 1% improvement, indicating that the uneven scale of road labels has the greatest impact on network detection performance.

In terms of the detection speed, although the FCNs model boasted the fastest detection speed among the experimental models, its segmentation accuracy failed to meet the required standards. The Mask R-CNN model exhibited the slowest testing speed. The improved U-Net model increased the detection time by only 1.14 ms per image (3.22%) compared to the original model, but the detection accuracy of the improved model has significantly increased. Taking everything into consideration, the model designed in this study offers the most balanced segmentation performance for road scene images captured from UAVs.

In addition, the improved model demonstrates excellent automatic road extraction capabilities on a mixed dataset (comprising both proprietary and public datasets), indicating that the proposed model adapts well to datasets from various sources. However, due to significant scale variations among different road types, some errors still occur in the extraction of images of roads on a smaller scale. Therefore, future research should focus on optimizing the algorithm and utilizing data sources from different platforms to achieve better application.

5.2. Results of 3D modeling of real road scenes

5.2.1. 3D models of road

Figure 12 presents the preliminary modeling results of the highway. It can be seen that the outline of the primary model coincides with the actual road and is relatively smooth. Still, the modeling effect of the area with vehicles was poor, indicating the need for local fine processing.

Figure 12. 3D model renderings: (a) global top view and (c) to (d) local details.

5.2.2. Model refinement

Figure 13 describes the model refinement process and the final result. It can be seen that after modifying the local details, the abnormal undulation of the road surface had been smoothed out. The overall model was smoother and closer to the actual situation than the preliminary model.

Figure 13. 3D model renderings: (a) global top view and (c) to (d) local details.

5.2.3. Model visualization

Figure 14 illustrates the real-world 3D model of a highway on the Website published by Cesium. The toolbar on the left was divided into a browsing view editing plate, an operation editing plate, and a model measurement plate. For the 3D model of the actual highway scene, the utility of distance and coordinate measurements within the model was substantiated. Distance measurement facilitates the determination of the distance and elevation difference between any two points in the model, while coordinate measurement provides the longitude, latitude, and elevation of any point in the model. This enhances the model's applicability in real-world spatial analysis and precision mapping. Furthermore, this platform provides embedded semantic segmentation networks from the research, enabling online model training, testing, and evaluation. It accomplishes road recognition and extraction from drone aerial imagery and serves as a basis for further understanding of road scenes.

Figure 14. Web-based 3D real-scene highway model using Cesium.

6. Conclusions

This research provides a method for automatic extraction and 3D modeling of real road scenes using UAV imagery and the improved U-Net model. The main conclusions are drawn as follows:

1. Compared to the conventional network utilized for road information extraction, the enhanced training model demonstrated a superior capability to capture more detailed road data. The feasibility of incorporating an improved ASPP module to extract roads in complex environments was confirmed. Additionally, the CA and SA modules effectively mitigated issues related to missed and false detections, thereby enhancing the overall accuracy and reliability of the model.

2. he F₁ score and IoU of the improved U-Net model for road extraction from UAV imagery reached 85.6% and 81.2%, respectively. These metrics signify a notable enhancement over other network models, demonstrating the improved model's superior performance. This model effectively reconstructs the actual road layout, showcasing its exceptional capability for automatic road extraction. This improvement in performance indices highlights the model's advanced ability to capture and delineate road features accurately in diverse aerial scenarios.

3. The ContextCapture, DP-Modeler, and Cesium software platforms were employed to facilitate the preliminary modeling, fine modeling, and visual analysis of road scenes. The model demonstrated high overall accuracy and effectively restored local details to closely replicate the actual scene. Additionally, the web-based model incorporated functionalities for coordinate and distance measurement, enhancing its utility for detailed spatial analysis and accurate geographical assessments.

However, this work only realizes the segmentation of road images. A further research topic will use a more optimized model to segment diverse objects in remote-sensing images, especially for the analysis of factors such as land, urban buildings, and rivers.

Acknowledgements

This research was funded by the Key Science and Technology Research Project of Jinhua, China, grant number 2022-3-056.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

Data in Figure 2(b) are publicly available and the DeepGlobe road extraction dataset derived from DeepGlobe 2018 (https://www.kaggle.com/datasets/balraj98/deepglobe-road-extraction-dataset). Data in Figure 2(a) were privately acquired and are available on request from the corresponding author.

Additional information

Funding

This work was supported by Jinhua Science and Technology Bureau: [Grant Number 2022-3-056].