Near Real-time Map Building with Multi-class Image Set Labeling and Classification of Road Conditions Using Convolutional Neural Networks

Figures & data

Figure 1. Sample raw images from street and road cameras representing (a) Dry (b) Offline, (c) Poor, (d) Snow and (e) Wet categories

Figure 2. A summary of near real-time road condition classification process

Figure 3. An overview of the data preparation and experimentation phases

Figure 4. Augmented data samples

Table 1. Labeling of images by road condition sensors located near the cameras

Road Condition Id	Title	# Images	Content (Inferred Visually)
1	Dry	28,709	Mostly dry, some wet, snow
2	Moist	3,546	Mostly damp or wet, some dry
3	Moist (treated)	954	Mostly damp or wet, some dry
4	Wet	5,216	Mostly wet, some slush,snow, dry
5	Wet (treated)	584	Mostly wet, some slush, snow, dry
6	Ice	1,545	Wet, some snow, slush, dry
7	Frost	29	Seemingly dry
8	Snow	1,718	Mostly snow, some dry, wet
15	Dew	737	Mostly wet, some rainy, some dry
16	Black ice warning	66	Seemingly dry
18	Slush	2	N/A
21	Unknown	236	Wet snowflakes on road
22	Unknown	38	Light snowflakes on road
23	Unknown	149	Mostly heavier snow or slush
24	Unknown	48	Seemingly assorted

Table 2. Phase 1 classification metrics and the confusion matrix for ResNet-50 (top) and VGG-16 (bottom) frameworks

		ResNet-50
		Classification Metrics
	Precision	Recall	F1-Score	Support	Accuracy
Dry	0.61	0.97	0.75	200
Non-dry	0.91	0.38	0.53	200
Macro Avg.	0.76	0.67	0.64	400
Training					88.7%
Validation					67.0%
		Confusion Matrix
	True\Predicted	Dry	Non-dry
	Dry	193	7
	Non-dry	125	75
		VGG-16
		Classification Metrics
	Precision	Recall	F1-Score	Support	Accuracy
Dry	0.79	0.88	0.83	200
Non-dry	0.86	0.77	0.81	200
Macro Avg.	0.83	0.82	0.82	400
Training					94.0%
Validation					82.3%
		Confusion Matrix
	True\Predicted	Dry	Non-dry
	Dry	176	24
	Non-dry	47	153

Table 3. Phase 2 classification metrics and the confusion matrix for VGG-16 framework

		VGG-16
		Classification Metrics
	Precision	Recall	F1-Score	Support	Accuracy
Dry	0.81	0.78	0.79	89
Offline	0.97	1.00	0.99	33
Snow	0.83	0.78	0.80	45
Wet	0.51	0.58	0.54	43
Macro Avg.	0.78	0.78	0.78	210
Training					83.5%
Validation					77.1%
		Confusion Matrix
True\Predicted	Dry	Offline	Snow	Wet
Dry	69	0	2	18
Offline	0	33	0	0
Snow	3	1	35	6
Wet	13	0	5	25

Table 4. Phase 3 classification metrics and the confusion matrix for VGG-16 framework

		VGG-16
		Classification Metrics
	Precision	Recall	F1-Score	Support	Accuracy
Dry	0.88	0.93	0.90	1924
Offline	1.00	0.98	0.99	136
Poor	0.91	0.91	0.91	222
Snow	0.92	0.91	0.91	806
Wet	0.87	0.79	0.83	1003
Macro Avg.	0.92	0.90	0.91	4091
Training					94.1%
Validation					89.0%
	Confusion Matrix
True\Predicted	Dry	Offline	Poor	Snow	Wet
Dry	1780	0	6	41	97
Offline	0	133	3	0	0
Poor	9	0	202	5	6
Snow	55	0	1	733	17
Wet	181	0	9	19	794

Table 5. Classification judgment for 1000 random images from combined (1.5 M) data sets

# Images	Verdict	Dry	Offline	Poor	Snow	Wet	Total
VGG-16	Acceptable:	616	65	136	9	62	888
20 K	Refused:	31	0	7	21	53	112

Table 6. VGG-16 pseudo-labeling results the 352 K and 1.1 M data sets

VGG-89%	352 K Set		1.1 M Set
Pseudo-Labels	# Images	%	# Images	%
Dry	216,614	61.50	748,165	67.16
Poor	49,707	14.11	143,016	12.84
Wet	43,479	12.34	114,700	10.30
Offline	26,827	7.62	91,163	8.18
Snow	15,613	4.43	16,876	1.52
Total	352,240	100.00	1,113,920	100.00

Table 7. Final 47 K labeled data set with 5 classes

# Images	Total	Train	Validation
Dry	16,065	14,458	1607
Offline	5225	4702	523
Poor	9259	8333	926
Snow	7565	6808	757
Wet	9228	8305	923
Overall	47,342	42,606	4736

Table 8. Phase 4 classification metrics and the confusion matrix for VGG-16 framework

		VGG-16
		Classification Metrics
	Precision	Recall	F1-Score	Support	Accuracy
Dry	0.87	0.88	0.87	1607
Offline	0.99	0.98	0.99	523
Poor	0.88	0.84	0.86	926
Snow	0.90	0.85	0.88	757
Wet	0.80	0.85	0.82	923
Macro Avg.	0.89	0.88	0.88	4736
Training					88.9%
Validation					87.3%
	Confusion Matrix
True\Predicted	Dry	Offline	Poor	Snow	Wet
Dry	1419	0	53	22	113
Offline	1	511	11	0	0
Poor	67	3	779	31	46
Snow	36	0	35	646	40
Wet	117	0	8	17	781

Table 9. Phase 4 classification metrics and the confusion matrix for inceptionResNetV2 framework

		IRN-V2
		Classification Metrics
	Precision	Recall	F1-Score	Support	Accuracy
Dry	0.90	0.92	0.91	1607
Offline	0.99	0.99	0.99	523
Poor	0.88	0.87	0.88	926
Snow	0.92	0.90	0.91	757
Wet	0.88	0.88	0.88	923
Macro Avg.	0.92	0.91	0.91	4736
Training					90.8%
Validation					90.7%
		Confusion Matrix
True\Predicted	Dry	Offline	Poor	Snow	Wet
Dry	1474	0	67	11	55
Offline	1	519	3	0	0
Poor	58	4	810	18	36
Snow	25	0	30	684	18
Wet	75	0	10	30	808

Table 10. Phase 4 classification metrics and the confusion matrix for efficientNet-B4 framework

		EN-B4
		Classification Metrics
	Precision	Recall	F1-Score	Support	Accuracy
Dry	0.90	0.92	0.91	1607
Offline	0.98	0.99	0.99	523
Poor	0.91	0.84	0.87	926
Snow	0.94	0.92	0.93	757
Wet	0.86	0.92	0.89	923
Macro Avg.	0.92	0.92	0.92	4736
Training					90.3%
Validation					90.9%
		Confusion Matrix
True\Predicted	Dry	Offline	Poor	Snow	Wet
Dry	1471	0	56	7	73
Offline	0	520	3	0	0
Poor	75	8	776	23	44
Snow	29	1	17	693	17
Wet	54	0	5	18	846

Table 11. Summary of classification judgments for 1000 random images from the combined (1.5 M) data set

# Images	Verdict	Dry	Offline	Poor	Snow	Wet	Total (1000)
VGG-16	Acceptable:	616	65	136	9	62	888
20 K	Refused:	31	0	7	21	53	112
VGG-16	Acceptable:	599	79	205	8	70	961
47 K	Refused:	17	0	0	1	21	39
IRN-V2	Acceptable:	587	76	247	10	54	974
47 K	Refused:	20	0	0	3	3	26
EN-B4	Acceptable:	608	78	217	9	66	978
47 K	Refused:	18	1	0	1	2	22

Table 12. Common configurations

Hyper-parameter	: Value
Optimizer	: Rectified Adam
Learning Rate	: 0.0001
Loss Func.	: Categorical Crossentropy
Activation Func.	: Softmax
Global Pooling	: Average
Batch Size	: 16
Max Epochs	: 12
0–1 Rescaling	: Yes
Training Set	: 42,606
Validation Set	: 4736
Target Classes	: 5
Augmented Training	: Yes

Table 13. Algorithm-specific configurations. H: Hidden. D: Dropout

	VGG-16	RN-50	XCep.	IRN-V2	EN-B0	EN-B4
Input Dims	(224, 224)	(224, 224)	(299,299)	(299, 299)	(256, 256)	(256, 256)
Global Pooling	None	Average	Average	Average	Average	Average
Top Layers	H-D-H-D	D	D	D	D	D
FC Neurons	4096	0	0	0	0	0
Dropout Rate	0.4	0.4	0.2	0.4	0.2	0.4
Total Layers	25	178	135	783	233	470
Total Prms.	134 M	23 M	20 M	54 M	4 M	18 M
Trainable Prms	134 M	23 M	20 M	54 M	4 M	18 M
Nontrain. Prms	0	53 K	54 K	60 K	42 K	125 K

Figure 5. Training accuracy (left) & validation accuracy (right) over 12 epochs for 6 algorithms

Figure 6. Execution times (training + validation + model saving) over 12 epochs for 6 algorithms. Left: Per epoch. Right: Cumulative

Table 14. Classification judgment for the 782 images captured real-time in Canada and the United States at 2100 UTC 11 January 2020

# Images	Verdict	Dry	Offline	Poor	Snow	Wet	Total (782)
EN-B4	Acceptable:	432	109	24	55	136	756
Live Images	Refused:	20	0	0	1	5	26
Accuracy							96.7%

Figure 7. An example map of 782 classified camera images over Canada and the United States at 2100 UTC 11 January 2020. Each marker represents one classified image with the legend indicating the color corresponding to each respective class