FPGA-oriented lightweight multi-modal free-space detection network

Abstract

For autonomous vehicles, free-space detection is an essential part of visual perception. With the development of multi-modal convolutional neural networks (CNNs) in recent years, the performance of driving scene semantic segmentation algorithms has been dramatically improved. Therefore most free-space detection algorithms are developed based on multiple sensors. However, multi-modal CNNs have high data throughput and contain a large number of computationally intensive convolution calculations, limiting their feasibility for real-time applications. Field Programmable Gate Arrays (FPGAs) provide a unique combination of flexibility, performance, and low power for these problems to accommodate multi-modal data and the computational acceleration of different compression algorithms. Network lightweight methods offer great assurance for facilitating the deployment of CNNs on such resource-constrained devices. In this paper, we propose a network lightweight method for a multi-modal free-space detection algorithm. We first propose an FPGA-friendly multi-modal free-space detection lightweight network. It comprises operators that FPGA prefers and achieves a $95.54\mathrm{\%}$ MaxF score on the test set of KITTI-Road free-space detection tasks and 81 ms runtime when running on 700 W GPU devices. Then we present a pruning approach for this network to reduce the number of parameters in case the complete model exceeds the FPGA chip memory. The pruning is in two parts. For the feature extractors, we propose a data-dependent filter pruner according to the principle that the low-rank feature map contains less information. To not compromise the integrity of the multi-modal information, the pruner is independent for each modality. For the segmentation decoder, we apply a channel pruning approach to remove redundant parameters. Finally, we implement our designs on an FPGA board using 8-bit quantisation, and the accelerator achieves outstanding performance. A real-time application of scene segmentation on KITTI-Road is used to evaluate our algorithm, and the model achieves a $94.39\mathrm{\%}$ MaxF score and minimum 14 ms runtime on 20W FPGA devices.

Keywords:

• Pruning
• free-space detection
• lightweight network
• multi-modal learning
• FPGA

1. Introduction

Free-space detection is a part of driving scene understanding, which classifies each pixel in an image as drivable or non-drivable areas, usually realised by image segmentation algorithms. Other modules in the autonomous system benefit from these pixel-level segmentation results, such as trajectory prediction and path planning, to ensure autonomous vehicles safely navigate in complex environments. In recent years, multi-modal data fusion convolutional neural networks (CNN) architectures have greatly improved the performance of free-space detection algorithms (X. Zhang et al., 2021). For robust and accurate scene understanding, autonomous vehicles are usually equipped with different sensors, and multiple sensing modalities can be fused through their complementarity (X. Bai et al., 2021). In multi-modal learning, modalities can be combined from bottom to top according to fusion level. Pixel level fusion is to fuse the original data with the smallest granularity (C. Wang et al., 2020). Feature level fusion sends images of different modalities into different segmentation sub-networks, fuses the feature maps obtained by different sub-networks, and then inputs them into the decision layer (C. Wang et al., 2022). Decision level fusion sends images of different modalities in various networks and fuses the results after segmentation (Zhou et al., 2020). Feature-level fusion is the most commonly used method among them. This fusion strategy's models usually consist of multiple encoders and one decoder. However, the number of parameters in the network doubles with each additional input modality, which makes the model computationally expensive (Yan et al., 2021).

The calculation of such a large model relies heavily on computational acceleration devices. Graphics Processing Units (GPUs) provide the infrastructure of multi-core parallel computing and has many cores, which can support parallel computing of large amounts of data. Its high floating-point computing capabilities are often used to accelerate CNN algorithms. For autonomous vehicles, both real-time processing speed and low power consumption are desirable. As GPU consumes much electricity, few GPU devices can be installed in an autonomous vehicle due to a limited power supply. But dozens of perception and planning tasks need to be processed simultaneously on the vehicle. Field Programmable Gate Arrays (FPGAs) are low-power devices with highly-parallel bit-oriented architecture, support both pipeline parallelism and data parallelism. The training of CNN models uses Single Instruction Multiple Data (SIMD) calculation, in which large batches of data can be processed in parallel by executing only one instruction. However, the calculation of model inference after the training is an application of Multiple Instruction Single Data (MISD), which is precisely the advantage of FPGA over GPU. We evaluate FPGAs as a platform to meet computational power requirements and constraints.

Since multi-modal data fusion CNNs usually have larger weight matrix and activation values than single-modal models, the FPGA chip memory cannot afford the entire model (L. Bai et al., 2020). Model compression reduces the computation and data storage by simplifying the network structure and has become an essential step in deploying CNN algorithms on the FPGA platforms (Xiong et al., 2021; J. Zhang et al., 2021). There are several model compression methods, such as pruning, quantisation, low-rank factorisation, and knowledge distillation. Among them, pruning and quantisation are techniques that compress both activation and weight representations, allowing the FPGA platform to store all intermediate results.

Pruning is a technique for eliminating unnecessary weights from neural networks based on the over-parameterisation of CNNs (Ye et al., 2018). As a machine learning method, CNNs can be divided into two stages: training and inference. In the training phase, the parameters are learned according to the data; in the inference phase, new data are fed into the model, and the result is obtained after calculation (Guo et al., 2016). Over-parameterisation means that there are a large number of parameters in the training phase to capture the information in the dataset. Once the training is completed to the inference phase, most of these parameters are redundant, which means the network can be pruned before deployment. There are many benefits of network pruning. The most immediate one is reducing the high amount of calculation, resulting in less computation time and less power consumption. The smaller memory footprint also allows algorithms to run on lower-end devices, such as replacing DRAMs with faster and more power-efficient SRAMs. Finally, smaller package size is conducive to model updates, making product upgrades more convenient.

According to the granularity of pruned feature, pruning techniques can be grouped into weight pruning and filter pruning. Early methods are based on weight pruning, whose pruning granularity is the weight kernels. The kernels after pruning are sparse matrices filled with element zero. Under the support of today's hardware, sparse matrices cannot be optimised by using the existing Basic Linear Algebra Subprograms (BLAS) libraries, so it is difficult for the pruned model to achieve substantial performance improvement. Therefore, researches in recent years have focussed on filter pruning. Different granularities of filter pruning exist, such as filter-based, channel-based, block-based, and layer-based. Since filter pruning does not change the sparsity of the weight matrices, it can be well supported by existing computing platforms and frameworks. In this paper, we focus on filter pruning to achieve model compression and acceleration, aiming to provide a versatile solution for FPGA devices.

A typical neural network pruning framework is training, pruning, and fine-tuning. The first step is to train the complete network. The second step is to delete parameters according to the filter evaluation function. The third step is to train the network with additional epochs using a low learning rate so that the network has an opportunity to recover from the performance penalty. Generally, the last two steps are iterative, and each iteration increases the pruning rate. The core of a pruner lies in selecting pruning filter evaluation functions, whose goal is to achieve a minor accuracy loss under the highest compression ratio. Based on whether the training data is utilised to determine the pruned filters, the filter pruning can be categorised into data-dependent and data-independent. The methods of data-independent pruning are based on the inherent weights of the network and do not rely on the input data. After the pruning, fine-tuning is required to restore the accuracy. Typical methods include L1 or L2 norm, first-order gradient metric, the rank of feature maps, and geometric median in network layers. These filters bring advantages in low time complexity but have limitations in accuracy and compression ratio. The methods of data-dependent pruning are adding additional regular terms to the input data, making it sparse, and embedding the pruning into the training process so that the data flow raises a better pruning strategy during the network training. This category belongs to methods such as scaling factors in BN layers and masking structural sparsity parameters. These methods of directly adding sparse constraints to the network generally achieve better results than the first one. In multi-modal learning, the model usually supports multiple data inputs, which makes the data have a significant influence on the network. Therefore we prefer a data-dependent filter pruning as the evaluation function.

Quantize is to convert the trained model into a low-precision representation for calculation. Today's maturing quantisation method is to convert 32-bit floating-point tensors into 8-bit fixed-point tensors for calculation. The memory space of each parameter reduces from 32-bit to 8-bit. Thus, the model size decreases to a quarter. Another advantage of quantisation in model acceleration is improving throughput and reducing latency. After quantification, the theoretical computing peak of the chips can be quadrupled, further accelerating the inference of the model. The hardware digital signal processor (DSP) core integrated into the FPGA chip is a high-performance hardware multiplier and accumulator, which is highly appropriate for convolution operation. However, DSP units can only operate multiplications below the specified bit width, i.e. fixed-point calculations. Therefore, quantisation is also essential for running CNNs on FPGA platforms. We choose the 8-bit quantisation method in the TensorFlow platform proposed by Google (Krishnamoorthi, 2018). As one of the most basic quantisation methods, it is supported by mature software tools on TensorFlow and Pytorch to facilitate subsequent model deployment on the FPGA platform.

In this research, we propose a lightweight multi-modal free-space detection network appropriate for FPGA platforms. The foundation is a lightweight multi-input U-shaped architecture that supports multi-modal data input and returns reliable free-space detection results adequate for self-driving. Then we prune this lightweight model with two strategies. First, according to the rank of each modal feature map, layer pruning is performed separately for each modality to ensure their independence. Second, for the segmentation decoder, channel pruning further removes unnecessary parameters. With the pruned model, we quantise the model to 8-bit and deploy the inference on FPGA.

Our contributions in this paper are four-fold:

1. We propose a multi-modal learning CNN performing free-space detection tasks with RGB and depth images. The network achieves excellent performance on drivable area segmentation and has a suitable structure for transplanting to FPGA devices.

2. We propose a two-stage filter pruning strategy, including a data-dependent pruning based on the rank of feature maps and a data-independent pruning for the segmentation decoder.

3. We apply an 8-bit quantisation to maximise the computation capability of FPGA, which improves the operations per second(OPS) by $43\mathrm{\%}$ with little accuracy penalty.

4. We introduce a hardware architecture design for executing the lightweight model, which helps compute neural networks with high performance.

2. Related work

We review some related work on free-space detection and pruning below.

2.1. Free-space detection

Free-space detection presents an informative perception of the environment toautonomous vehicles. For several decades, research based on image processing has been expanding to address road segmentation problems. RBNet (Z. Chen & Chen, 2017) investigates the contextual relationship between the road structure and its boundary arrangement. RBA (Sun et al., 2019) uses a residual refinement module composed of the reverse attention and boundary attention units. Along with algorithms based on low-level features, many studies attempting to solve segmentation problems with deep CNNs have been carried out (Cordts et al., 2017), and studies that considered free space problems to be semantic segmentation tasks have been conducted (X. Han et al., 2018). Earlier approaches, including FCN (Long et al., 2015) and SegNet (Badrinarayanan et al., 2017), proposed the encoder-decoder architecture, where the encoder generates feature maps in multiple scales and the decoder provides high prediction accuracy pixel-wise classification. Strong backbone networks, e.g. GoogLeNet (Szegedy et al., 2015), ResNet (K. He et al., 2016), DenseNet (G. Huang et al., 2017), performance better segmentation features. U-Net (Ronneberger et al., 2015) based models concatenate the lower-level features to the feature maps as skip connections in order to predict more detailed output. Recent researches focus on non-local operations (X. Wang et al., 2018) in semantic segmentation models for the purpose of eliminating the noises in feature maps (Fu et al., 2019; Vaswani et al., 2017). These methods use transformers with extremely large parameters to ensure that the network learns the correlation between semantics, featuring high-accuracy results.

Methods combining geometric information from multi-modal data were gradually introduced to tackle this segmentation problem. Early works (Couprie et al., 2013; Eigen & Fergus, 2015; Hazirbas et al., 2016) convert depth into a single-channel image and used the early fusion to concatenate depth and RGB as a four-channel image input simply. HHA (Gupta et al., 2014) is another depth encoding of three channels, including horizontal disparity, height above ground, and norm angle. In other studies (Z. Li et al., 2016; D. Lin et al., 2013; Park et al., 2017), RGB and HHA images were fed into two DCNNs to extract features separately and finally use the middle fusion to concatenate or sum up the features. Other researches using LiDAR output with RGB data for free space detection have been published (Caltagirone et al., 2019; Teichmann et al., 2018). SNE-RoadSeg (Fan et al., 2020) generates surface normal information from dense depth images and aggregates it with RGB images by a densely-connected fusion. PLARD (Z. Chen et al., 2019) introduced an altitude difference image as a second input of the network. The cross-fusion approach (Caltagirone et al., 2019) and the cascaded fusion approach (Gu et al., 2021) were proposed to get better RGB and projected point cloud fused features.

2.2. Pruning

Many works (Carreira-Perpinán & Idelbayev, 2018; S. Han, Mao et al., 2015; H. Li et al., 2022; Tung & Mori, 2018; T. Zhang et al., 2018) focus on pruning the fine-grained weight of filters. S. Han, Pool et al. (2015) proposes an iterative method to discard the small weights whose values are below the predefined threshold. Carreira-Perpinán and Idelbayev (2018) formulates pruning as an optimisation problem of finding the weights that minimise the loss while satisfying a pruning cost condition. However, the sparse matrix obtained by weight pruning does not support the corresponding accelerated operation.

Opposed to weight pruning, filter pruning removes the entire filter according to certain metrics. It is one of the most popular methods to accelerate over-parameterised CNNs since the pruned deep networks can be directly applied on any off-the-shelf platforms and hardware to obtain the online speed-up. Some data-independent filter pruning strategies (Y. He et al., 2018, 2019; H. Li et al., 2016) have been explored. H. Li et al. (2016) utilises an l1-norm criterion to prune unimportant filters. Y. He et al. (2018) proposes to select filters with an l2-norm criterion and prune those selected filters in a soft manner. Ye et al. (2018) proposes to prune models by enforcing sparsity on the scaling parameter of batch normalisation layers. Zhuo et al. (2018) uses spectral clustering on filters to select unimportant ones. Y. He et al. (2019) proposed a filter pruning via geometric median to compress the model. Some filter pruning approaches (Dubey et al., 2018; Q. Huang et al., 2018; M. Lin et al., 2020; S. Lin et al., 2019; Suau et al., 2018; D. Wang et al., 2018; Yu et al., 2018; Zhuang et al., 2018) need to utilise training data to determine the pruned filters. Luo et al. (2017) adopts the statistics information from the next layer to guide the filter selections. Dubey et al. (2018) aims to obtain a decomposition by minimising the reconstruction error of training set sample activation. Suau et al. (2018) proposes an inherently data-driven method which use Principal Component Analysis (PCA) to specify the proportion of the energy that should be preserved. D. Wang et al. (2018) applies subspace clustering to feature maps to eliminate the redundancy in convolutional filters. M. Lin et al. (2020) develops a method that is mathematically formulated to prune filters with low-rank feature maps.

Efficient free-space detection algorithms are of great importance for deployment on mobile devices. However, there are few works discussing the pruning of semantic segmentation neural networks for free-space detection tasks. Pruning methods designed over the classification task have been straightforwardly applied to segmentation neural networks (W. He et al., 2021; Q. Huang et al., 2018; Yamamoto & Maeno, 2018). Luo et al. (2017) pruned filters in the backbone network on ImageNet and transferred it to the segmentation network. X. Chen et al. (2020) proposed a multi-task channel pruning to obtain a lightweight semantic segmentation network.

3. Method

The purpose of this article is to implement a free-space detection algorithm on an autonomous driving platform using low-power devices. FPGA is a feasible solution due to its high performance and energy efficiency. We first introduce a lightweight multi-modal free-space detection network, with less convolutional operators and smaller feature maps. Then reduce the parameters of the model through filter pruning and 8-bit quantisation. Finally, the model is transplanted to a FPGA so that it can predict independently in the low-power devices.

3.1. Multi-modal free-space detection network

The free-space detection network is built with a classic encoder-decoder architecture as shown in Figure 1. Unlike software platforms, convolutional computing resources built by hardware cannot be released during the program. It means that the fewer operators that are implemented, the FPGA resources are better used. Common image encoders include VGG (Simonyan & Zisserman, 2014), Deeplab (L. C. Chen et al., 2017), mobilenet (Howard et al., 2017), ResNet (K. He et al., 2016), etc. VGG uses stacked $3\times 3$ convolution, but due to the problem of gradient disappearance, the network is shallow and it is difficult to achieve high accuracy. Deeplab achieves good results, but the spatial pyramid structure contains a variety of atrous convolution operators, and it generates large feature tensors during multi-scale fusion, which is difficult to implement in FPGA. Mobilenet is a lightweight network using depthwise separable convolution to reduce the parameters of $3\times 3$ convolution, but the cost is the increased calculation steps and the detail loss in low-dimensional features. ResNet greatly increases the depth of the network through shortcut connection, has strong feature extraction ability, and basically only consists of $3\times 3$ convolution and $1\times 1$ convolution, which is the first choice for our encoder (Figure 2).

Figure 1. Our proposed lightweight multi-modal free-space detection network. An encoder-decoder architecture is employed. The input of the network is a pair of RGB and depth images. They are processed by the ResNets (K. He et al., 2016) encoders. The multi-modal fusion strategy is the concatenation of the feature maps. A U-shape segmentation decoder (Ronneberger et al., 2015) is propagated for the final prediction.

Figure 2. Replacing the $7\times 7$ convolution with $3\times 3$ convolutions.

Convolution decomposition: The calculations in a ResNet are $1\times 1$ convolution, $3\times 3$ convolution, $7\times 7$ convolution, and shortcut concatenation. It should be noted that in FPGAs, concatenation can be achieved only by scheduling the memories without the involvement of computing units. Therefore, the computation of $7\times 7$ convolution is the largest one. Meanwhile, this convolution appears only once in the first layer, and is not reused subsequently, which led us to the idea of optimising it. Convolutions with $7\times 7$ spatial filters means the receptive field of this operation is $7\times 7$. Without changing the size of the receptive field, we try to find a multi-layer network with a smaller convolution kernel to replace this operation. The theoretical receptive field of a convolutional layer can be calculated by the recursive formula: (1) $r_i=r_{i-1}-(k_i-1)\ast \prod _{j=0}^{i-1}{s}_j,$(1) where r is the receptive field, k is the kernel size, and s is the stride. Assume that the length and width of a feature map are both $x(x>=7)$. We use a $7\times 7$ convolution kernel to slide with a stride of 1, which requires $(x-7+1)$ slides. The same in the vertical direction, sliding $(x-7+1)$ times, so there are $(x-6)\ast (x-6)$ convolutions, where the size of the receptive field is $(x-6)\ast (x-6)$. In the same way, the output size after the $3\times 3$ convolution is $(x-3+1)\ast (x-3+1)$. Use two more $3\times 3$ convolutions on the output map to get the receptive field $(x-3+1-3+1-3+1)\ast (x-3+1-3+1-3+1)=(x-6)\ast (x-6)$, which is equal to the $7\times 7$ convolution result. Therefore, it can be concluded that three $3\times 3$ convolutions and one $7\times 7$ convolution have the same feature extraction ability.

The operations for a $3\times 3$ convolution is 9, for a $7\times 7$ convolution is 49. For images whose length and width are both x, the computations of three $3\times 3$ convolution and one $7\times 7$ convolution are $O_3=9\ast (x-2{)}^2+9\ast (x-4{)}^2+9\ast (x-6{)}^2$ and $O_7=49\ast (x-6{)}^2$ respectively. Calculating $O_3<=O_7$, we can get x>12, which means for image larger than $12\times 12$, the three-layers $3\times 3$ convolution have advatage in both number of parameters and calculation. For the first layer of Resnet, the input is the original image, which is much larger than $12\times 12$, so using three layers of $3\times 3$ convolution instead of $7\times 7$ convolution always performances better. The optimised accelerating $7\times 7$ convolution layer designed in this paper is shown in Figure 3(a).

Figure 3. (a) Detail of the optimised accelerating $7\times 7$ convolution layer. (b) Detail of the dimensionality reduction for large features.

Dimensionality reduction: In the semantic segmentation model, the deeper the network, the larger the receptive field, but the more information is missing in the decoder. Thus, the retrieval of edge features is extremely important. U-Net uses the method of skip connection to fuse the same-level feature map in the encoder with the recovery result of deconvolutional layer to achieve a retrieval of edge features. Referring to this U-shaped structure, we concatenate the multi-modal same-level features to the skip connection, and used them for the calculation of the deconvolution layer at the same time. The disadvantage is that the concatenated feature tensor will be very thick. Taking the KITTI dataset as an example. The size of the original image is $1392\times 512$, the size of the feature map after four times downsampling is $87\times 32$, the dimension of the feature map of the 4th layer of ResNet-50 is 1024. So the size of the feature tensor of the two-modality network after concatenation is $87\times 32\times 3072$, occupying about 8.13 Mb of FPGA block RAM. Although this size is within the acceptable range, it just takes up more than half of the memory space, which makes the device unable to run double batch parallelism. This is wasteful, so we reduced the dimensionality of the fused features, using two $1\times 1$ convolutional layers as shown in Figure 3(b).

The entire lightweight free space detection network uses ResNet-50 as the backbone, U-net as the segmentation head, and uses concatenation as the feature fusion method, which effectively maintains the complexity and feature extraction capability of the network. The lightweight network only contains two convolution operators, $3\times 3$ convolution and $1\times 1$ convolution, which greatly saves computing resources on the FPGA. At the same time, the size of the intermediate result of the algorithm is precisely controlled, so that the algorithm can prosess inside the FPGA in multi-batch parallelism, avoiding the delay caused by external data exchange.

3.2. Filter pruning

Filter pruning is a method of acceleration by removing the convolution kernels with weak feature extraction ability and making the network sparse. This method can greatly reduce the parameters in the network. The accuracy of can be recovered quickly after retraining the pruned model. The procedure can be divided into five steps. Step 1 is to train the entire network with the dataset to generate the initial network model file. Step 2 is to record the weight distribution of the convolution kernel of the first layer, and set the pruning rate according to the importance of convolution kernel. Step 3 is to delete the corresponding relationship between the convolution kernel and feature map with little impact on the network performance according to the pruning rate. Step 4 is to retrain the network to recover the performance penalty. Step 5 is to repeat the above process layer-by-layer, pruning and fine-tuning until output the pruned network model file.

Assume $C_i$ is the ith convolutional layer of a pre-trained CNN model. The weights in $C_i$ can be represented as $W_{C_i}=\{w_1,w_2,\dots ,w_{n_i}\}\in {\mathbb{R}}^{n_i\times n_{i-1}\times k_i\times k_i}$, where $n_i$ represents the number of filters in $C_i$ and $k_i$ denotes the kernel size. The input feature maps are denoted as $X_i=\{x_1,x_2,\dots ,x_{n_i}\}\in {\mathbb{R}}^{b\times n_{i-1}\times h_i\times w_i}$, where b is the batch size, $h_i$ and $w_i$ are the height and width of the feature map. Filter pruning aims to identify and remove the less important weights set from $W_{C_i}$, which can be fomulated as an optimisation problem: (2) $p=\underset{\delta }{\arg min}\sum _{i=1}^{\mathrm{layers}}\sum _{j=1}^{n_i}\delta \cdot F(w_{i,j}),$(2) where $F()$ measures the importance of a weight in the CNN. δ is a filter which is 1 if $w_{i,j}$ is important or 0 if $w_{i,j}$ is unimportant. Minimizing p is to removing the least important weights in $C_i$.

Encoder pruning: Our key problem falls in designing a function $F()$ which can well reflect the information richness of multi-modal features. Since the feature maps of different modalities are relatively independent, most pruning methods that directly design $F()$ based on network weights will concentrate on a certain modality, which causes the algorithm to ignore important cross-modal information. Therefore, for the encoder, we propose to define $F()$ according to the feature map for each independent modality, because feature maps are intermediate steps that can reflect both the filter properties and the input images. The optimisation function p is thus reformulated as: (3) $p=\underset{\delta }{\arg min}\sum _{i=1}^L\sum _{j=1}^{n_i}\delta \cdot F^{\ast }(w_{i,j}{x}_{i,j}),$(3) where $F^{\ast }()$ estimates the information of feature maps generated by $w_{i,j}$ and $x_{i,j}$. The more information the feature map contains, the more important the corresponding filter is.

The rank of a matrix is the number of irrelevant row or column vectors. For the matrix of an image, the rank can represent the degree of information redundancy and the amount of information in the image. We perform the singular value decomposition (SVD) onimage $x_{i,j}$: (4) $\begin{array}{rl}{x}_{i,j}& =\sum _{i=1}^R{\sigma }_i{u}_i{v}_i^{\ast }\\ & =\sum _{i=1}^{R^{\prime }}{\sigma }_i{u}_i{v}_i^{\ast }+\sum _{i=R^{\prime }}^R{\sigma }_i{u}_i{v}_i^{\ast },\end{array}$(4) where R is the rank of an input feature map, ${\sigma }_i$, $u_i$ and $v_i$ are the singular values. A feature map with rank R can be decomposed into a lower-rank feature map with rank $R^{\prime }$ and additional information, which demonstrates that higher-rank feature maps contain more information than lower-rank feature maps. It can be not only an effective measure of information, but also a stable representation. We thus define our information measurement as: (5) $F^{\ast }(w_{i,j}{x}_{i,j})=R(x_{i,j}).$(5)

Decoder pruning: Since the scaling factors in encoder and decoder network are optimised alternately, setting the same global threshold for both the backbone and decoder is inappropriate. No need to consider the influence of intermediate results, applying traditional pruning methods on the decoder is enough to remove redundant parameters. We sort the convolution kernels according to their L1-norm: (6) $F(w_{i,j})=\Vert w_{i,j}{\Vert }_1.$(6) We introduce a scaling factor for each channel, which is multiplied by the output of that layer. We then jointly train the network weights and these scaling factors, and sparse regularise them. Finally, we prune those channels with small factors and fine-tune the pruned network. Since the pruning corresponds to removing all inputs and outputs of the layer, we can directly obtain a narrow network. The scaling factor acts as a weight selection and are jointly optimised with the weights. The network can automatically identify insignificant channels and safely remove them without greatly affecting the generalisation performance.

3.3. Quantization and layer fusion

8-bit quantisation: Our quantisation strategy is to use floating-point computations in network training and integer computations in inference. Against the characteristics of FPGA, the goal of quantisation is to use only integer calculations to complete all arithmetic operations. It is implemented by an affine transformation between the real value r and the integer value q: (7) $r=S(q-Z),$(7) where Z is the quantisation zero-point, which is constant. r is the real value to be quantised, usually a 32-bit floating-point number. For B-bit quantisation, q is an integer of B-bit. In this paper, we set B = 8. S is the quantisation scale, calculated by counting the maximum and minimum values of the entire array: (8) $S=\frac{{\mathrm{array}}_{max}-{\mathrm{array}}_{min}}{2^B-1}.$(8) Z is the quantisation zero-point, representing the bias of q relative to 0, which is the same data type as q.

Quantized layer fusion: Generally, a layer in CNN consists of convolution, BN and ReLU. Since all parameters during inference are constant, the fusion of these calculations can greatly reduce the number of parameters. Because of the batch normalisation, the convolution computation can be set as unbiased. The convolution can thus be expressed as: (9) $y^{\prime }=\sum \omega x,$(9) where ω is the weight, x is the input feature, and $y^{\prime }$ is the convolutional result. Quantize Equation  (9(9) $y^{\prime }=\sum \omega x,$(9) ) with $x=S_x(q_x-Z_x)$ and $\omega =S_{\omega }(q_{\omega }-Z_{\omega })$, the convolution is equivalent to: (10) $y^{\prime }=S_{\omega }{S}_x\sum (q_{\omega }-Z_{\omega })(q_x-Z_x).$(10) Batch Normalization(BN) is widely used to solve the internal covariate shift problem. The computation of BN during inference can be summarised as a combination of normalisation and scaling: (11) $y^{″}=\alpha \frac{y^{\prime }-ϵ}{\sigma }+\beta ,$(11) where ϵ is the mean, σ is the variance, α is the scale, and β is the shift. Then we have the quantised BN: (12) $\begin{array}{rl}{y}^{″}& =\alpha \frac{S_{\omega }{S}_x\sum (q_{\omega }-Z_{\omega })(q_x-Z_x)-ϵ}{\sigma }+\beta \\ & =\frac{\alpha S_{\omega }{S}_x}{\sigma }\sum (q_{\omega }-Z_{\omega })(q_x-Z_x)+(\beta -\frac{\alpha ϵ}{\sigma }).\end{array}$(12) The convolution operations are followed by nonlinear activation units, to process the linear output of the previous layer through a nonlinear activation function to simulate any function, thereby enhancing the representation ability of the network. ReLU is a piecewise linear function, which is the most commonly used activation layer. A ReLU operator be expressed as: (13) $y=\{\begin{array}{ll}{y}^{″}& y^{″}\ge 0\\ 0& y^{″}<0.\end{array}$(13) The output y should also be quantised as $y=S_y(q_y-Z_y)$. Thus when $y^{″}\ge 0$: (14) $\begin{array}{rl}y& =S_y(q_y-Z_y)\\ & =\frac{\alpha S_{\omega }{S}_x}{\sigma }\sum (q_{\omega }-Z_{\omega })(q_x-Z_x)+(\beta -\frac{\alpha ϵ}{\sigma }).\end{array}$(14) Then we have the quantisation result as: (15) $q_y=\frac{\alpha S_{\omega }{S}_x}{\sigma S_y}\sum (q_{\omega }-Z_{\omega })(q_x-Z_x)+(\frac{\sigma \beta -\alpha ϵ}{\sigma S_y}+Z_y).$(15) When $y^{″}<0$: (16) $\begin{array}{rl}y& =S_y(q_y-Z_y)=0,\\ q_y& =Z_y.\end{array}$(16) Note that $M=\frac{\alpha S_{\omega }{S}_x}{\sigma S_y}$ and $b=\frac{\sigma \beta -\alpha ϵ}{\sigma S_y}+Z_y$, we have the quantised convolutional layer: (17) $q_y=\{\begin{array}{ll}M\sum (q_{\omega }-Z_{\omega })(q_x-Z_x)+b& q_y\ge Z_y\\ Z_y& q_y<Z_y,\end{array}$(17) where M, b, $Z_x$, $Z_{\omega }$ and $Z_y$ are five constants which can be offline calculated before inference. The fused convolution layer greatly reduces the computational steps in the inference, further accelerating the network.

3.4. Hardware architecture

The calculation and data exchange of the entire CNN are independently completed by the FPGA core. The CPU side is only responsible for sending start signals and receiving interrupts. The overview of hardware architecture is demonstrated in Figure 4. It consists of computation kernels, data buffers for memory optimisation, and several kinds of signal control units. A Process Logic Initialization block generates the internal control signals for each layer. The Data Buffer reads the weights and features of the current layer, then sent them into the computation kernels. We adopted DSP48E2 cores to perform two 8-bit multiplications simultaneously. The convolutional operations are performed in a pipelined Processing Element Arrays, and additional assist calculations such as maxpooling, up-sampling, etc. are performed according to the characteristics of each layer. The quantised features are re-assembled into feature maps after dequantization, and are sent to DDR for the next loop.

Figure 4. Block diagram of hardware architecture.

$3\times 3$ convolution unit: Each unit in the weight buffer stores a $3\times 3$ weight of a fliter, with a length of 9 bits for the carry-save addition. In the feature map buffer, a $3\times 3$ window is sent to the DSP for convolution operation input every cycle, and each element is 8bit. A group of 9 DSPs corresponds to one $3\times 3$ window accumulation.The channel accumulation is performed in each DSP, and summed up after accumulating three times. A subscriber is introduced to calculate Equation (16(16) $\begin{array}{rl}y& =S_y(q_y-Z_y)=0,\\ q_y& =Z_y.\end{array}$(16) ). The architecture contains 16 computing units (Figure 5(a)), which output 16 channels of 1 pixel per frame.

Figure 5. (a) $3\times 3$ convolution unit. (b) $1\times 1$ convolution unit.

$1\times 1$ convolution unit: The weight buffer shares the same unit as it of $3\times 3$ Conv, and each unit stores the weight of 4 channels corresponding to 1 Kernel. The feature map buffer sends a $1\times 1$ window of 4 channels to the DSP for convolution operation input. A group of 4 DSPs is used for channel accumulation. Each DSP is responsible for the 1/4 channel-accumulation of 1 kernel. The DSPs differ by one cycle. After the accumulation, the result is output to the accumulator to complete the convolution operation of the entire kernel. The bias-add is displayed at the end of convolution computing.

4. Experiments

4.1. Dataset

We construct our experiments on KITTI-ROAD (Fritsch et al., 2013) dataset to demonstrate our efficiency in reducing model complexity. : This dataset provides RGB-D data in real-world driving scenarios. Specifically, it contains 289 pairs of training data with ground truth for freespace detection and 290 pairs of testing data without ground truth.

4.2. Implementation details

Our network is trained on an NVIDIA Tesla V100 platform using CUDA10.0, which requires 600–700 W power suply. The quantised model is infrenced on a Xilinx Zynq UltraScale+ MPSoC ZCU106 kit with XCZU7EV-2FFVC1156 FPGA core, powered by a 12V5A direct current(DC) electricity supply, draws 20–60 W of power.

We use PyTorch to implement the network training. The training batch size is set to 6, the initial learning rate is set to 1e−4, and the Adam solver is used to optimise the network. We train the network over 100 epochs, and decay the learning rate linearly with a decay rate of 0.99. For pruning the pre-trained model, we reduce the learning rate to one-tenth of the original learning rate, and use fewer epochs to finetune the pruned model. In the filter pruning step, we prune all the weighted layers in a encoder/decoder with the same pruning rate at the same time. The pruning operation is conducted at the end of every training epoch.

4.3. Evaluation metrics

We take MaxF as segmentation evaluation metric: (18) $F1=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}},$(18) where TP, FP, FN are the numbers of true positive, false positive, false negative pixels. We adopt the widely-used protocols, i.e. number of parameters and required Float Points Operations (denoted as FLOPs), to evaluate model size and computational requirement. We also compare the inference speed on GPU and FPGA devices.

4.4. Performance evaluation

Performance of lightweight model: We list the performance of some SOTA algorithms from the KITTI leaderboards in Table 1. Most of them use [email protected] for the evaluation. ChipNet (Lyu et al., 2018) only used a single sensor, which preprocessed point cloud data by organising them in a spherical view. RoadNet-RT (L. Bai et al., 2020) achieves a very high-speed single-model segmentation network but too much loss in accuracy. PLARD (Z. Chen et al., 2019), DFM-RTFNet (H. Wang et al., 2021), and SNE-RoadSeg (Fan et al., 2020) took advantage of multi-modal data. They transformed the 3D point cloud into the image plane by projecting its LiDAR coordinates and sending it into the network with the RGB image. When comparing our proposed method with other SOTA algorithms using the evaluation measures, it was $1.51\mathrm{\%}$ less accurate than heavy data-fusion networks but two times faster. Furthermore, compared to the FPGA-faced ChipNet, our network showed a $1.81\mathrm{\%}$ improvement. As a result, the proposed lightweight multi-modal free-space detection network was competitive in inference speed while ensuring high accuracy.

Table 1. Performance on Kitti-road compared to other methods(GPU).

Methods	MaxF(%)	AP(%)	Runtime(ms)
PLARD	96.83	93.98	160
DFM-RTFNet	96.78	94.05	80
SNE-RoadSeg	96.75	94.07	100
RoadNet-RT	92.55	93.21	9
ChipNet on GPU (camera)	82.50	86.13	945
ChipNet on GPU (LiDAR)	93.73	–	162
ChipNet on FPGA (LiDAR)	94.05	88.29	18
Ours on GPU	95.54	90.14	81

Performance of pruning: As shown in Table 2, the classic l1-norm pruning suffered from a performance drop of $2.45\mathrm{\%}$ when pruning $25\mathrm{\%}$ of all channels. In contrast, our proposed method boosted the performance of the pruned model and had an insignificant accuracy drop. As the pruning rate increased, the accuracy dropped significantly because of the reduced model complexity. Similarly, the inference speed of the model also increases as the number of parameters decreases. Our method obtained a MaxF of $94.17\mathrm{\%}$ when keeping $50\mathrm{\%}$ channels, which led to a $1.37\mathrm{\%}$ decline. Pruning more channels led to a slightly worse score but consistently outperformed the prior method. To prioritise the accuracy of the model, we choose the result of $25\mathrm{\%}$ pruning rate for quantisation.

Table 2. Pruned multi-modal unet-resnet50 on Kitti-road.

Rate	Methods	MaxF(%)	FLOPs(G)	Params(M)	Runtime(ms)
–	$\begin{array}{c}\mathrm{Multi}-\mathrm{modal}\\ \mathrm{Unet}-\mathrm{ResNet}50\end{array}$	95.54	192.10	61.64	81.76
0.25×	L1-norm pruning	93.09	134.47	43.15	69.49
	Ours	95.52	147.92	47.46	71.94
0.5×	L1-norm pruning	92.87	109.49	35.13	63.77
	Ours	94.17	99.89	32.05	62.95
0.75×	L1-norm pruning	86.33	55.71	17.87	48.23
	Ours	89.77	59.55	19.10	48.40

Performance of quantisation: As shown in Table 3, after the 8-bit quantised, the storage of the network was reduced by $75\mathrm{\%}$, and the MaxF dropped by $1.13\mathrm{\%}$. The score of no pruning and $25\mathrm{\%}$ pruning are the same, indicating that the drop entirely depends on the quantisation. We also performed a comparison between our model and ChipNet on FPGA. The main advantage of ChipNet in speed is that it only uses one modality, so it has a smaller network structure. However, LiDAR-based free space detection cannot be applied in urban road scenes. In Figure 6, we demonstrate some detection results of the FPGA model. The performance of this model is hardly affected by lightweight processing and can provide quite good detection results.

Figure 6. Examples of detection results by FPGA.

Table 3. 8-Bit quantised results on FPGA.

Methods (pruning rate)	MaxF(%)	Ops(G)	Params(M)	Runtime(ms)
ChipNet	93.73	–	–	12
Ours(0%)	94.28	48	16	18
Ours(25%)	94.39	37	12	16
Ours(50%)	93.93	25	8	14

FPGA implementation: We used a independent FPGA platform to run our model. All data were stored in Double Data Rate SDRAM (DDR) in advance through the ARM core on the platform, and the FPGA program run through the start signal sent by the arm core. The ARM core will not participate in the whole inference process. It only receives the final output result. System clock frequency is set to 300 MHz. The resource usage of our proposed neural network is listed in Table 4. We illustrate the performance of the FPGA implemented model in Table 5.

Table 4. Resource usage on the FPGA implementation.

FPGA Resource	LUT	FF	BRAM	URAM	DSP
RoadNet-RT	260335	115684	1340	–	1560
ChipNet	38082	–	1543	–	3072
Our Used	298560	597120	600	80	2928
Our Utilization(%)	75.32%	53.69%	58.92%	91.25%	73.98%

Table 5. Performance evaluation of FPGA inference.

Benchmark	MaxF	AP	PRE	REC	FPR	FNR
UM_ROAD	93.45	89.09	93.91	93.07	1.96	5.61
UMM_ROAD	95.41	92.46	95.51	95.31	3.13	4.22
UU_ROAD	93.02	87.22	92.53	92.92	1.93	5.88
URBAN_ROAD	94.17	89.83	94.31	94.04	2.08	5.53

5. Conclusion

In this paper, we propose a lightweight multi-modal free-space detection network suitable for the FPGA platform by simplifying the network structure according to the MISD design of FPGA. The network uses ResNet-50 as the backbone and U-net as the segmentation head and only contains two convolution operators, $3\times 3$ convolution and $1\times 1$ convolution, which fully uses computing resources on the FPGA chip. Meanwhile, the size of the intermediate results is precisely controlled so that the algorithm can perform multiple batches of parallel processing inside the FPGA, avoiding the delay caused by external data exchange. We adopt concatenation as the feature fusion method, which effectively maintains the complexity of the network and the feature extraction ability. Then we prune this lightweight network with two strategies. The first is the encoder pruning according to the rank of each modal feature map. Layers are pruned on each modality separately to remove the convolution kernels with weak feature extraction capabilities and make the network sparse. The second is the decoder pruning, using l1-norm as the pruning function. Since the scale factors in the encoder and decoder networks are alternately optimised, the independence of the multi-modal features is guaranteed. Finally, in order to facilitate the convenient and rapid model deployment on the FPGA platform, we use a mature quantisation method on the pruned lightweight model to convert 32-bit floating-point tensors into 8-bit fixed-point tensors for calculation. The size of the model is reduced to a quarter. The theoretical calculation peak of the chip can be increased by four times because of the improvement of the throughput and the delay. In a common autonomous driving platform, the period of sensors is 100 ms, while the running time of this road perception algorithm is 16 ms. Sufficient time resources are reserved for the following tasks (planning, control, etc.), which is essential in building a real-time autonomous driving system.

Disclosure statement

No potential conflict of interest was reported by the author(s).