LWIR hyperspectral image classification based on a temperature-emissivity residual network and conditional random field model

Figures & data

Figure 1. The Fourier transform based LWIR hyperspectral imaging system: the Telops Hyper-Cam.

Figure 2. The three datasets (a) Mineral dataset. (b) Tree leaves dataset. (c) Dongguo Lake dataset.

Figure 3. Spectra and temperature of the three datasets. (a) Radiance spectra of the mineral dataset classes. (b) Radiance spectra of the tree leaves dataset classes. (c) Radiance spectra of the Dongguo Lake dataset classes. (d) Emissivity spectra of the mineral dataset classes. (e) Emissivity spectra of the tree leaves dataset classes. (f) Emissivity spectra of the Dongguo Lake dataset classes. (g) Temperature of the mineral dataset. (h) Temperature of the tree leaves dataset. (i) Temperature of the Dongguo Lake dataset.

Figure 4. Flowchart of the proposed TERN-CRF method. The model includes two steps: the temperature-emissivity residual network and post-processing based on the CRF model. The network includes two spectral and two spatial residual blocks, an average pooling layer, and a fully connected layer. The CRF model includes a segmentation prior and inference of CRF.

Figure 5. (a), (b) and (c) respectively represent three-dimensional CONVBN, spectral residual blocks and spatial residual blocks.

Figure 6. Flowchart of the conditional random field model.

Table 1. Training and test samples of the mineral dataset.

No.	Class	Train	Test
1	Calcite	100	5260
2	Fibrous gypsum	100	5376
3	Anhydrite	100	4079
4	Dolomite	100	4418
5	Talc	100	2447
6	Diopside	100	3784
7	Concrete floor	100	537
	Total	700	25,901

Table 2. Training and test samples of the tree leaves dataset.

No.	Class	Train	Test
1	Photinia	100	3809
2	Evergreen magnolia	100	4981
3	Grass	100	1020
4	Camphor	100	3745
5	Osmanthus	100	3055
6	Mulberry	100	4483
7	Privet	100	1226
8	Concrete floor	100	3191
	Total	800	25,510

Table 3. Training and test samples of the Dongguo Lake dataset.

No.	Class	Train	Test
1	Building	100	122
2	Road	100	10,234
3	Sand	100	3783
4	Water	100	1929
5	Vegetation	100	2151
6	Bare soil	100	800
	Total	600	19,019

Figure 7. The classification results for the mineral dataset.

Table 4. Classification results of the different methods for the mineral dataset.

	SVM	SVM-CRF	FNEA-OO	Hybrid-SN	AF_3D CNN	CNN	CNN- CRF	TERN	TERN-CRF
OA(%)	99.05	99.49	96.41	99.46	99.34	99.66	99.68	99.72	99.74
AA(%)	99.12	99.56	92.54	99.21	98.46	99.34	99.39	99.38	99.45
Kappa	0.99	0.99	0.96	0.99	0.99	0.99	0.99	0.99	0.99
Calcite	99.41	99.68	97.45	99.87	99.98	99.81	99.83	99.85	99.90
Fibrous gypsum	98.85	99.22	96.28	99.27	99.46	99.94	99.94	99.57	99.55
Anhydrite	98.45	98.79	95.98	99.41	98.84	99.85	99.88	99.93	99.88
Dolomite	99.12	99.55	96.72	98.87	99.27	99.19	99.39	99.80	99.59
Talc	98.36	99.89	98.04	99.96	99.14	99.18	100	99.55	99.97
Diopside	99.68	100.00	98.68	99.89	99.94	100	97.58	99.95	97.39
Concrete floor	100	100	64.62	97.21	92.55	97.39	97.58	97.02	97.39

Figure 8. The classification results for the tree leaves dataset.

Table 5. Classification results of the different methods for the tree leaves dataset.

	SVM	SVM-CRF	FNEA-OO	Hybrid-SN	AF_3D CNN	CNN	CNN- CRF	TERN	TERN-CRF
OA(%)	83.32	94.85	85.24	94.99	92.26	92.32	94.67	97.76	98.27
AA(%)	80.60	93.53	79.77	94.32	90.39	90.03	92.99	97.21	98.01
Kappa	0.81	0.94	0.83	094	0.91	0.91	0.94	0.97	0.98
Photinia	84.30	95.21	89.44	96.10	94.39	97.13	97.55	98.55	98.79
Evergreen magnolia	93.94	95.84	94.88	95.48	96.58	97.11	97.17	97.37	97.39
Grass	85.78	94.61	63.92	91.27	89.50	91.76	93.82	95.59	95.59
Camphor	80.88	89.05	86.65	94.02	93.21	89.48	91.11	98.56	99.57
Osmanthus	56.04	97.84	84.98	95.19	80.52	83.18	94.40	98.76	99.38
Paper mulberry	87.66	98.53	91.97	96.72	96.07	95.27	96.25	97.75	98.53
Glossy privet	59.30	79.61	52.61	92.01	79.28	70.80	76.92	92.99	97.63
Concrete floor	96.90	97.56	73.68	93.73	93.60	95.49	96.68	98.09	97.18

Figure 9. The classification results for the Dongguo Lake dataset.

Table 6. Classification results of the different methods for the Dongguo Lake dataset.

	SVM	SVM-CRF	FNEA-OO	Hybrid- SN	AF_3D CNN	CNN	CNN- CRF	TERN	TERN-CRF
OA(%)	82.17	84.66	86.56	90.80	90.67	89.06	91.72	93.95	94.42
AA(%)	80.36	83.25	82.22	89.23	90.11	86.19	89.65	89.58	89.59
Kappa	0.74	0.78	0.80	0.86	0.86	0.84	0.88	0.91	0.92
Building	87.70	97.54	89.34	94.26	98.36	98.36	98.36	98.36	96.72
Road	80.67	87.18	88.09	90.00	89.05	89.41	91.27	95.55	95.99
Sand	94.45	97.65	96.83	98.33	99.28	98.68	99.37	99.50	99.42
Water	94.56	97.25	91.50	98.60	98.49	97.77	98.60	99.90	99.95
Vegetation	64.30	39.38	65.83	80.57	82.24	73.04	82.94	83.31	86.19
Bare soil	60.50	80.50	61.75	73.63	73.25	59.88	67.38	60.88	59.25

Table 7. Average running times for the different classification methods. Time: s.

Dataset		SVM	SVM-CRF	FNEA-OO	Hybrid -SN	AF_3D CNN	CNN	CNN-CRF	TERN	TERN-CRF
Mine-ral	Training	3.54	3.54	6.31	58.08	64.20	32.46	32.46	562.22	562.22
	Test	2.15	4.18	2.78	19.67	16.26	4.26	6.12	50.82	52.63
Tree leaves	Training	3.21	3.21	6.53	55.09	71.49	33.46	33.46	559.87	559.87
	Test	2.33	5.01	2.54	17.53	15.53	4.79	7.55	52.32	53.87
Dongguo Lake	Training	5.19	5.19	14.03	65.90	58.23	56.11	56.11	775.65	775.65
	Test	3.58	7.23	3.93	13.02	12.27	8.51	10.09	84.18	86.18

Figure 10. Comparison of the classification accuracies of the CNN, CNN-CRF, TERN, and TERN-CRF methods for the three datasets.