Scene-level buildings damage recognition based on Cross Conv-Transformer

Figures & data

Figure 1. Map of vision in transformer.

Figure 2. Location of the study areas.

Table 1. Classification of building damaged by aerial images after earthquake.

Class label	Feature description	Damaged rate (%)
Severely damaged	All buildings are destroyed or collapsed	>70%
Moderate damaged	At least half of the buildings are damaged or collapsed	50–70%
Slightly damaged	Basic intact or only a small part of the building is damaged	<50%
Negative samples	Background of waters, roads, etc.	/

Table 2. Example images of classification instances.

Class labels	Damaged rate (%)	Sample instance
Severely damaged (5561)	>70%
Moderate damaged (5272)	50-70%
Slightly damaged (5165)	<50%
Negative samples (5046)	Background of water bodies, open spaces, etc.

Table 3. Distribution of the sample set.

Sample set	Numbers
Training Set	13640
Validation Set	5819
Test Set	1584
Total	21,043

Figure 3. The flowchart of the Conv-Transformer structure.

Figure 4. The flowchart of the Conv-Projection structure.

Figure 5. Self Attention (a), Local Attention (b) and Global Attention (c). (Local Attention can only obtain the information of the image in the window, while Global attention can pay attention to the information of the entire image).

Figure 6. Cross Attention implementation in details (Only the Class tokens are fused because the Class tokens represent all the information of the branch patches).

Figure 7. Cross Attention implementation in detail (The class token in Big-branch interacts with the patch tokens in Small-branch as query, and the class token in Small-branch also performs the same operation to complete the interaction of the feature information between the two branches).

Figure 8. Cross Conv-Transformer network structure. (The network is divided into two branches, each branch is composed of a different number of Conv-Transformer. The output of the last two branches is sent to the Cross-Attention module for information fusion and filtering, and finally to the MLP for classification).

Figure 9. Precision and loss curve of the four models during training.

Table 4. Confusion matrix of the ResNet.

Category	Severely damage	Moderate damaged	Slightly damaged	Negative samples	Total	Damaged
Severely damage	488	33	8	0	529	92.23%
Moderate damage	27	291	25	0	343	84.84%
Sight damage	14	21	558	2	595	93.78%
Negative samples	0	0	6	111	117	94.87%
Total	529	345	597	113	1584
Producer accuracy	92.23%	84.35%	93.47%	98.23%		91.34% (OA)

Table 5. Confusion matrix of the ViT.

Category	Severely damage	Moderate damage	Slightly damaged	Negative samples	Total	Damaged
Severely damage	503	22	0	0	525	95.81%
Moderate damage	30	294	16	0	340	86.47%
Sight damage	1	31	585	0	617	94.81%
Negative samples	1	0	2	99	102	97.05%
Total	535	347	603	99	1584
Producer accuracy	94.03%	84.72%	97%	100%		93.64% (OA)

Table 6. Confusion matrix of the EfficientNet.

Category	Severely damage	Moderate damaged	Slightly damaged	Negative samples	Total	Damaged
Severely damage	507	18	5	0	530	95.66%
Moderate damage	8	300	10	0	318	94.34%
Sight damage	11	29	586	0	626	93.31%
Negative samples	0	0	3	105	107	97.04%
Total	535	349	604	105	1584
Producer accuracy	96.38%	85.96%	97.02%	100%		94.89% (OA)

Table 7. Confusion matrix of the Crosss Conv-Transformer.

Category	Severely damage	Moderate damaged	Slightly damaged	Negative samples	Total	Damaged
Severely damage	529	6	0	0	535	98.96%
Moderate damage	7	309	4	0	320	96.67%
Sight damage	1	22	600	0	623	96.31%
Negative samples	0	0	1	105	106	99.03%
Total	537	337	604	105	1584
Producer accuracy	98.51%	85.96%	99.17%	100%		97.61%(OA)

Figure 10. Block level distribution map of earthquake damage degree in Yushu.

Table 8. Confusion matrix of the Crosss Conv-Transformer.

	Single branch	Two branch	Concatenate	Cross attention	Accuracy/%
a	√				94.38
b		√	√		96.85
c		√		√	97.61

Figure 11. Heat map of the model’s feature.