Automatic identification of building structure types using unmanned aerial vehicle oblique images and deep learning considering facade prior knowledge

Figures & data

Figure 1. The workflow for automatically identifying building structural types from UAV oblique photography images based on deep learning.

Figure 2. Data flow chart and web crawling object diagram for establishing the sample set.

Figure 3. Examples of common classification of building structure types in China: (a) steel frame; (b) reinforced concrete shear wall; (c) reinforced concrete frame; (d) reinforced masonry; (e) unreinforced masonry.

Figure 4. Structure of deep convolution network with building facade prior knowledge attention branch (PKAB) for identifying seismic building structural types.

Figure 5. Flow chart of the method of automatically extracting building facades from the oblique image.

Figure 6. Extracting building facades from oblique images using feature points distribution. (a)Classification of feature points and distribution of outliers; (b)Classification of feature points after removing outliers. (The figure uses random different colors to show the feature points of different classifications, and uses a yellow circle to indicate that there are misclassification points.)

Figure 7. Direction gradient map of building facade. (a) The building facade extracted by the proposed method; (b) The binarized distribution map of x-direction gradient; (c) The binarized distribution map of y-direction gradient.

Figure 8. Devices used for obtaining the data and the orthoimage of datasets from the 3D high-resolution building model. (a) Four-rotor unmanned aerial vehicle (UAV); (b) Three-camera photography system; (c) The orthoimage of Area 1; (d) The orthoimage of Area 2.

Table 1. Statistical table of the number of buildings of different building structure types in each area.

Building structure type	Area1	Area2
Steel frame	6	0
Reinforced concrete shear wall	3	0
Reinforced concrete frame	52	21
Reinforced masonry	103	60
Unreinforced masonry	14	31

Table 2. The number of buildings of various structural types extracted by the building facade extraction metohd in different flight experiment cases.

Building structure type	Case 1 (Area 1 + Summer)	Case 2 (Area 2 + Summer)	Case 3 (Area 1 + Winter)
Steel frame	6	0	6
Reinforced concrete shear wall	3	0	3
Reinforced concrete frame	52	21	52
Reinforced masonry	103	59	103
Unreinforced masonry	14	20	14

Table 3. Evaluation results of building structural types based on different processing methods and model structures on Case 1. ($P=Precision,R=Recall,F1=F1-score$).

Classification	Xception			Xception + PKAB			Xception + opt			Xception + PKAB + opt
Metrics	P	R	$F1$	P	R	$F1$	P	R	$F1$	P	R	$F1$
Steel frame	0.55	0.83	0.66	0.55	0.83	0.67	0.56	0.83	0.67	0.71	0.83	0.77
RC shearwall	1.00	0.33	0.50	0.75	1.00	0.86	1.00	0.33	0.50	0.75	1.00	0.86
RC frame	0.83	0.96	0.89	0.90	0.88	0.89	0.82	0.88	0.85	0.85	0.90	0.88
Reinforced masonry	1.00	0.50	0.66	0.97	0.54	0.70	0.94	0.63	0.76	0.93	0.63	0.75
Unreinforced masonry	0.25	1.00	0.39	0.25	1.00	0.40	0.30	0.93	0.46	0.33	1.00	0.50
Average	0.73	0.72	0.62	0.68	0.85	0.70	0.72	0.72	0.65	0.72	0.87	0.75
Overall accuracy	0.68			0.70			0.73			0.74

Table 4. Evaluation results of building structural types based on different processing methods and model structures on Case 2. ($P=Precision,R=Recall,F1=F1-score$).

Classification	Xception			Xception + PKAB			Xception + opt			Xception + PKAB + opt
Metrics	P	R	$F1$	P	R	$F1$	P	R	$F1$	P	R	$F1$
RC frame	0.42	0.95	0.58	0.68	0.71	0.70	0.44	0.81	0.57	0.72	0.81	0.72
Reinforced masonry	0.81	0.51	0.63	0.84	0.83	0.84	0.87	0.58	0.69	0.85	0.80	0.85
Unreinforced masnory	0.86	0.65	0.74	0.85	0.85	0.85	0.86	0.95	0.90	0.90	0.95	0.90
Average	0.70	0.70	0.65	0.79	0.80	0.80	0.72	0.78	0.72	0.83	0.85	0.83
Overall accuracy	0.63			0.81			0.70			0.83

Table 5. Evaluation results of building structural types based on different processing methods and model structures on Case 3. ($P=Precision,R=Recall,F1=F1-score$).

Classification	Xception			Xception + PKAB			Xception + opt			Xception + PKAB + opt
Metrics	P	R	$F1$	P	R	$F1$	P	R	$F1$	P	R	$F1$
Steel frame	0.71	0.83	0.77	0.63	0.83	0.71	0.75	1.00	0.86	0.67	1.00	0.80
RC frame shearwall	1.00	0.33	0.50	1.00	0.67	0.80	0.33	0.33	0.33	1.00	0.67	0.80
RC frame	0.88	0.88	0.88	0.91	0.83	0.87	0.80	0.67	0.73	0.85	0.87	0.86
Reinforced masnory	0.94	0.49	0.64	0.88	0.62	0.73	0.84	0.69	0.76	0.92	0.65	0.76
Unreinforced masnory	0.22	1.00	0.35	0.25	1.00	0.44	0.37	1.00	0.54	0.34	1.00	0.51
Average	0.75	0.70	0.63	0.74	0.79	0.71	0.62	0.74	0.64	0.75	0.84	0.75
Overall accuracy	0.65			0.72			0.71			0.75

Figure 9. Confusion matrix of building structural types identification using different processing methods and model structures on Case 1. (a) Xception + unprocess; (b) Xception + PKAB-net + unprocess; (c) Xception + process; (d) Xception + PKAB-net + process

Figure 10. Confusion matrix of building structural types identification using different processing methods and model structures on Case 2. (a) Xception + unprocess; (b) Xception + PKAB-net + unprocess; (c) Xception + process; (d) Xception + PKAB-net + process.

Figure 11. Confusion matrix of building structural types identification using different processing methods and model structures on Case 3. (a) Xception + unprocess; (b) Xception + PKAB-net + unprocess; (c) Xception + process; (d) Xception + PKAB-net + process

Table 6. Performance comparison of different benchmark models and attention mechanisms in three cases.

Description	Backbone	Param.(m)	Case1			Case2			Case3
			$P_{avg}$	$R_{avg}$	$F{1}_{avg}$	$P_{avg}$	$R_{avg}$	$F{1}_{avg}$	$P_{avg}$	$R_{avg}$	$F{1}_{avg}$
ResNet50	ResNet50	23.60	0.63	0.61	0.55	0.67	0.66	0.67	0.65	0.59	0.53
ResNet50 + PKAB	ResNet50	23.61	0.64	0.62	0.57	0.68	0.66	0.66	0.65	0.62	0.54
ResNet101	ResNet101	42.67	0.44	0.43	0.35	0.67	0.63	0.64	0.42	0.44	0.39
ResNet101 + PKAB	ResNet101	42.68	0.40	0.50	0.42	0.62	0.57	0.58	0.59	0.51	0.44
InceptionV3	InceptionV3	21.81	0.64	0.76	0.67	0.68	0.70	0.63	0.64	0.74	0.66
InceptionV3 + PKAB	InceptionV3	21.83	0.73	0.78	0.71	0.68	0.72	0.65	0.67	0.78	0.69
MobileNet	MobileNet	3.23	0.71	0.78	0.68	0.78	0.78	0.78	0.60	0.70	0.61
MobileNet + PKAB	MobileNet	3.24	0.72	0.80	0.69	0.78	0.79	0.78	0.56	0.66	0.57
Xception	Xception	20.87	0.72	0.72	0.65	0.72	0.78	0.72	0.62	0.74	0.64
Xception + PKAB	Xception	20.89	0.72	0.86	0.74	0.83	0.85	0.83	0.75	0.84	0.75
Xception + SE	Xception	20.97	0.73	0.77	0.70	0.65	0.69	0.63	0.72	0.74	0.65
Xception + CBAM	Xception	20.97	0.69	0.76	0.64	0.69	0.70	0.63	0.59	0.75	0.62
Xception + PAM	Xception	20.99	0.71	0.85	0.72	0.74	0.78	0.74	0.63	0.67	0.62

Figure 12. The CAM visualization results of integrated network based on the Xception model combined with each attention mechanism. The ground-truth label is displayed below each input image, and P represents the predicted score of each network for the ground-truth class.