FPGA-oriented lightweight multi-modal free-space detection network

Figures & data

Figure 1. Our proposed lightweight multi-modal free-space detection network. An encoder-decoder architecture is employed. The input of the network is a pair of RGB and depth images. They are processed by the ResNets (K. He et al., 2016) encoders. The multi-modal fusion strategy is the concatenation of the feature maps. A U-shape segmentation decoder (Ronneberger et al., 2015) is propagated for the final prediction.

Figure 2. Replacing the $7\times 7$ convolution with $3\times 3$ convolutions.

Figure 3. (a) Detail of the optimised accelerating $7\times 7$ convolution layer. (b) Detail of the dimensionality reduction for large features.

Figure 4. Block diagram of hardware architecture.

Figure 5. (a) $3\times 3$ convolution unit. (b) $1\times 1$ convolution unit.

Table 1. Performance on Kitti-road compared to other methods(GPU).

Methods	MaxF(%)	AP(%)	Runtime(ms)
PLARD	96.83	93.98	160
DFM-RTFNet	96.78	94.05	80
SNE-RoadSeg	96.75	94.07	100
RoadNet-RT	92.55	93.21	9
ChipNet on GPU (camera)	82.50	86.13	945
ChipNet on GPU (LiDAR)	93.73	–	162
ChipNet on FPGA (LiDAR)	94.05	88.29	18
Ours on GPU	95.54	90.14	81

Table 2. Pruned multi-modal unet-resnet50 on Kitti-road.

Rate	Methods	MaxF(%)	FLOPs(G)	Params(M)	Runtime(ms)
–	$\begin{array}{c}\mathrm{Multi}-\mathrm{modal}\\ \mathrm{Unet}-\mathrm{ResNet}50\end{array}$	95.54	192.10	61.64	81.76
0.25×	L1-norm pruning	93.09	134.47	43.15	69.49
	Ours	95.52	147.92	47.46	71.94
0.5×	L1-norm pruning	92.87	109.49	35.13	63.77
	Ours	94.17	99.89	32.05	62.95
0.75×	L1-norm pruning	86.33	55.71	17.87	48.23
	Ours	89.77	59.55	19.10	48.40

Figure 6. Examples of detection results by FPGA.

Table 3. 8-Bit quantised results on FPGA.

Methods (pruning rate)	MaxF(%)	Ops(G)	Params(M)	Runtime(ms)
ChipNet	93.73	–	–	12
Ours(0%)	94.28	48	16	18
Ours(25%)	94.39	37	12	16
Ours(50%)	93.93	25	8	14

Table 4. Resource usage on the FPGA implementation.

FPGA Resource	LUT	FF	BRAM	URAM	DSP
RoadNet-RT	260335	115684	1340	–	1560
ChipNet	38082	–	1543	–	3072
Our Used	298560	597120	600	80	2928
Our Utilization(%)	75.32%	53.69%	58.92%	91.25%	73.98%

Table 5. Performance evaluation of FPGA inference.

Benchmark	MaxF	AP	PRE	REC	FPR	FNR
UM_ROAD	93.45	89.09	93.91	93.07	1.96	5.61
UMM_ROAD	95.41	92.46	95.51	95.31	3.13	4.22
UU_ROAD	93.02	87.22	92.53	92.92	1.93	5.88
URBAN_ROAD	94.17	89.83	94.31	94.04	2.08	5.53

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). Las Vegas, USA.

 Google Scholar

Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In International conference on medical image computing and computer-assisted intervention (pp. 234–241). Munich, Germany.

 Google Scholar