Automatic extraction and 3D modeling of real road scenes using UAV imagery and deep learning semantic segmentation

Figures & data

Figure 1. Flowchart of the automatic extraction and 3D modeling of real road scenes.

Table 1. Hardware specifications for UAV road image collection.

Collection width (m)	Speed (m/s)	Shooting interval (s)	Number of Stitches	Working temperature (°C)	Flight control system
16.88	1.0	346	9	24	A3 Pro

Figure 2. Road in remote sensing images and its annotation: (a) UAV-acquired, (b) DeepGlobe dataset, and (c) labels.

Table 2. Road dataset information in UAV images.

Dataset	Labeled	road-H	road-P	No labeled	Total
Training set	2312	2864	1043	88	2400
Validation set	283	348	144	17	300
Testing set	276	346	128	24	300

Figure 3. Network structure of the original U-Net model.

Figure 4. Network structure of the improved U-Net model.

Figure 5. Network structure of coordinate attention mechanism.

Figure 6. Network structure of SA module.

Figure 7. Network structure of the improved ASPP module.

Figure 8. OSGB data format: (a) file structures and (b) organizational form.

Figure 9. 3D modeling process.

Figure 10. Workflow of the model refinement.

Figure 11. Diagram of the detection results of road extraction for each model.

Table 3. Comparisons of the testing results of four road extraction models.

	P		R		F1		IoU				mIoU
Models	road-H	road-P	road-H	road-P	road-H	road-P	road-H	road-P	AP	F1	road	Testing time (ms)
FCNs	0.827	0.734	0.836	0.695	0.831	0.714	0.831	0.625	0.746	0.755	0.728	25.77
Mask R-CNN	0.848	0.786	0.839	0.765	0.843	0.775	0.846	0.667	0.763	0.809	0.757	46.41
U-Net	0.865	0.803	0.821	0.762	0.842	0.782	0.843	0.712	0.773	0.817	0.778	35.43
U-Net1	0.886	0.825	0.856	0.775	0.871	0.799	0.864	0.732	0.812	0.839	0.803	36.04
U-Net2	0.894	0.831	0.862	0.781	0.878	0.805	0.871	0.739	0.821	0.848	0.807	36.58
Improved U-Net	0.901	0.832	0.875	0.786	0.888	0.808	0.876	0.748	0.829	0.856	0.812	36.58

Figure 12. 3D model renderings: (a) global top view and (c) to (d) local details.

Figure 13. 3D model renderings: (a) global top view and (c) to (d) local details.

Figure 14. Web-based 3D real-scene highway model using Cesium.

Data availability statement

Data in Figure 2(b) are publicly available and the DeepGlobe road extraction dataset derived from DeepGlobe 2018 (https://www.kaggle.com/datasets/balraj98/deepglobe-road-extraction-dataset). Data in Figure 2(a) were privately acquired and are available on request from the corresponding author.