Lane detection based on IBN deep neural network and attention

Abstract

Lane detection is an indispensable technology for environmental perception and is a function of autonomous vehicles. Although many researchers have applied deep learning to lane detection and achieved good results, their application scenarios were relatively simple. When a lane is blocked, lost, or met with other challenges, the accuracy of lane detection decreases tremendously. This paper proposes a novel semantic segmentation network for lane detection that includes IBN-Net, an attention module, and an encoder-decoder structure called IAED. IBN-Net improves the modelling and generalisation capability with little computational complexity. The attention module can better capture context information and improve the performance of the network. We evaluated the performance of the proposed method, which improved by 6.3% over ResNet34 on the TuSimple datasets. Tests on the CULane datasets showed that our method is at least 3% better than existing methods. Experimental results showed that the proposed method has strong robustness under complex conditions such as insufficient illumination and shadow occlusion.

KEYWORDS:

• Lane detection
• deep learning
• semantic segmentation
• IBN-Net
• attention module

1. Introduction

Thousands of people worldwide lose their lives in traffic accidents every year, and more than 90% of these losses are caused by driver distraction. An advanced driving assistance system (ADAS) can reduce the incidence of traffic accidents by calculating some form of feedback by taking inputs from sensors around the vehicle and providing assistance to the driver. A lane keeping system and lane departure warning system are important components of the ADAS and ensure that vehicles do not deviate from the lane.

In research on the autonomous navigation of driverless vehicles, vision-based lane line detection is one of the important aspects. It is the key to achieving autonomous navigation of driverless vehicles by obtaining location information on a road through the fast and accurate detection of lane lines. The correct detection of lane lines guarantees the safe driving of driverless vehicles. The method proposed in this paper saves on costs and is effective.

Traditional lane detection methods require considerable manual marking and postprocessing, which lead to a large amount of calculation. Lane line detection based on manual features generally utilises colour features, structure tensors, strip filters, ridge features (Chiu & Lin, 2005; Loose et al., 2009; Teng et al., 2010), etc. However, these traditional methods have poor robustness and are easily affected by illumination, weather, shielding, and other factors, which are not conducive to their application expansion in changeable road scenarios.

In 2012, Geoffrey and his student Alex won the competition in ImageNet by establishing the significance of deep learning in computer vision (Krizhevsky et al., 2012). Recently, many researchers on lane line detection have adopted the method of deep learning (Wang, Sun, et al., 2022;  Duan et al., 2021; Ghafoorian et al., 2018; Gurghian et al., 2016; Yu et al., 2022;  Lu et al., 2021; Zhang & Mahale, 2018). This process can be divided into three categories: image classification (Cai, Liu, et al., 2021;  Chougule et al., 2018; Wang, Wang, et al., 2022; Qin, Wang, et al., 2020; Zhang, Li , et al., 2021;  Yoo et al., 2020 ), target detection (Qi et al., 2021; Huang et al., 2018; Huval et al., 2015; Lee et al., 2017; Ning, Duan, et al., 2020), and semantic segmentation (Ning, Gong, et al., 2020;  Wang et al., 2021; Chen et al., 2018; Neven et al., 2018). The method based on image classification cannot accurately represent lanes, nor can it detect a relatively high number of lanes. The method based on target detection is complex in data collection and requires a large amount of computation. At present, the method based on semantic segmentation is the most widely used. However, this technology cannot capture the global context information, which reduces the performance of lane line detection, and most of the methods are completed under sunny conditions. Therefore, the lane detection performance decreased sharply when the test was carried out on cloudy days, poorly lit days, rainy days, and in shaded conditions.

Recently, many scholars have found that the attention mechanism can emphasize feature maps and improve the network representation ability. By introducing a spatial transformer module, Max (Ning et al., 2021) made corresponding spatial changes to the image spatial domain information to accurately determine the regions that need to be considered in the image information and extract the key information. Hu (Zhang, Sun, Yu, Dong, et al., 2021) proposed a squeeze and excite module (SeNet) that enables the network to focus on the relationship between channels and helps the model to automatically learn the importance of different channel characteristics. Based on the literature (Zhang, Sun, Yu, Dong, et al., 2021), Woo et al. (Niu et al., 2021) expanded and proposed the CBAM module, which successively integrates the channel attention module and spatial attention module. The performance of CBAM was superior to that of SENET, and all of these methods achieved excellent performance in image classification by introducing an attention mechanism. This mechanism can improve the robustness of lane detection under complex weather and model the specific context information of the road to determine whether the pixels belong to the lane. This helps to improve the accuracy of lane detection.

IBN-Net is composed of instance normalization (IN) and batch normalization (BN) to enhance its learning ability and generalisation ability. IN enables the model to learn features that are invariant to appearance changes, while BN speeds up convergence and preserves content-related information.

Lane detection relies heavily on global information because of its slender appearance, strong perspective relationship, and obvious mutual constraints between different targets. Therefore, to make the network not completely dependent on the appearance feature under the condition of lane occlusion and insufficient light, this paper focuses on modelling the specific context information of the road and uses the appropriate context information for inference. In this paper, we consider introducing a novel encoder-decoder network (called IAED) with IBN-Net and attention modules to meet the above requirements. IAED is an effective semantic segmentation neural network that uses U-Net as the backbone and combines the advantages of jump connections, residual blocks and codec structures. IBN-Net can not only learn the colour and style of lane lines but also retain the features related to texture (Pan, Luo, et al., 2018). The spatial attention and channel attention modules are added to the encoder-decoder architecture to obtain the characteristic dependencies on spatial and channel dimensions separately, and the output of the two attention modules is fused to further improve the learning and generalisation ability of the network. The two modules of IBN-Net and attention modules can improve the accuracy of semantic segmentation. We combine the advantages of these two modes to improve the performance of the network. A more accurate lane segmentation result can be obtained by obtaining rich context dependencies. The purpose of this paper is to enhance the performance of lane line detection, which is verified in the TuSimple and CULane datasets.

Our main contributions are as follows:

1. We propose a new framework, IAED, which fully utilises the superficial appearance feature information and deep semantic information of the neural network and aggregates more global information.

2. IBN-Net is added to ResNet, which improves the generalisation ability and learning ability of the network structure with only a small amount of calculation.

3. The attention module is appended to the encoder-decoder architecture and can overcome lane occlusion by using context information, which improves the feature representation ability of the lane detection network structure.

4. The new architecture is adopted to improve the computational efficiency while maintaining accuracy. The experiment verifies the effectiveness and high performance of our study.

The paper is organised as follows. In Section 2, some recent excellent lane line detection algorithms and attention mechanisms are introduced in detail. The method of this paper is proposed in Section 3. In Section 4, the method proposed in this paper is verified by experiments and discussed. Finally, conclusions are given in Section 5.

2. Related work

2.1. Lane detection

The continuous development of semantic segmentation networks has improved the performance of lane detection (Zhang, Lu, Zhang, Xue, et al., 2021). Because lane lines have the characteristics of slender structure space and regular correlation of position, many scholars have proposed different network structures according to these characteristics. Pan and others proposed SCNN (Pan, Shi, et al., 2018), which extends the deep convolution neural network to a rich spatial level. The obtained spatial information can be propagated on the neurons of the same layer to enhance the spatial information, which is particularly effective for recognising structured objects. This method improves the accuracy of lane line detection, but it requires orderly stacking convolution, which greatly reduces the efficiency of the algorithm. Considering that vehicles or pedestrians often block lane boundaries and local characteristics of lane boundaries are not obvious, Fan et al. (2019 ) proposed a new rotational convolutional layer and a new parameterised lane branch to detect lane boundaries from a global perspective. The feature maps in the convolutional module rotate at different angles to collect more information about the entire lane boundary in multiple directions. However, these two methods require a large amount of computation, and there are too many lane lines to detect, which is difficult to apply in practice. To solve the problem of lane line changes, Neven (Neven et al., 2018) proposed LaneNet, which is composed of a lane line splitting branch and a lane line embedding branch. This network can learn the structure of lane line marking from multiple angles and carry out end-to-end training. This method is a multibranch network structure that can detect any number of lane lines and has good robustness. However, this method is very time-consuming in clustering, and it is difficult to meet the requirements of real-time in practical engineering applications.

Zou (Zou et al., 2019) used multiple frames of continuous driving scenes for lane detection and proposed a hybrid depth structure combining CNN and RNN. Although this method can guarantee accuracy in the case of occlusion, the calculation is very complex. To solve the problem of colour distortion in rainy and low-light conditions, an improved lane line detection method based on vanishing point location was proposed by Seokju (Lee et al., 2017 ). However, the postprocessing cost of this method is very high.

To retain internal data and avoid the loss of useful information in the subsampling, Shao (Chen et al., 2018) embedded three dilated convolution layers between the encoder and decoder based on VGG to increase the receptive field without losing road information. However, the effect of dilated convolution is not obvious for lane structures with slender appearances. Some attention mechanism methods are applied to the lane detection algorithm to establish a large global receptive field. Xiao (Xiao et al., 2020) added an attention module to the network structure. This module can effectively capture the global background. To obtain better lane line detection results, Xiao placed the two modules in parallel. Although this method achieves the same level of performance as SCNN and has higher efficiency, it still does not solve the fundamental problem of complex SCNN computation. Lee et al. (2022 ) added expanded self-attention (ESA), which uses the confidence of lane line prediction results to indicate the clarity of lane lines to the network. ESA enhances the network’s attention to the blocked lane line and improves the lane line detection effect. Hou, Ma, & Liu (2019) proposed the self-attention distillation (SAD) method, which allows models to study themselves and achieve materiality improvements without increasing computation time and without any additional supervision or labelling. This method uses a lighter backbone to achieve high performance while maintaining real-time efficiency.

The lane detection method based on object detection has achieved excellent results, and anchor-based methods in lane detection are widely used. LaneATT (Torres et al., 2020 ) uses a new anchor-based attention mechanism that can aggregate global information. SGNet (Su et al., 2021) uses a top-down vanishing point guidance anchoring mechanism to generate strong anchors to effectively capture various lane lines and improve lane perception by using multilevel structural constraints. CondLaneNet (Liu et al., 2021) has a conditional lane detection strategy based on conditional convolution and row formulas for lane instance level discrimination. UFLD (Qin, Wang, et al., 2020) regards the process of lane detection as a row-based selection problem using global features, thus reducing computational cost but sacrificing accuracy. CLRNet (Zheng et al., 2022) utilises a linear RoI extraction operator based on bilinear sampling and constructed the overall dimension of the lane IoU loss to constrain the regression quality of the overall lane line.

2.2. Attention mechanisms

The study of attention mechanisms comes from the study of human vision, which selectively pays attention to part of all information and ignores other visible information. The attention mechanism will not process all information but only key parts of information to improve the efficiency of information processing. In recent years, some scholars (Cai, Zhai, et al., 2021;  Wang et al., 2017; Wang et al., 2018; Yuan et al., 2018; Bai et al., 2021) introduced the attention mechanism of the spatial domain and the domain channel to improve the performance of the model. Max et al. added spatial transformer networks to the CNN networks, making the model have powerful performance. This keeps translation, rotation and scaling unchanged, aiming to improve the accuracy of classification. Hu proposed the squeeze and exclusion module (SeNet), which models the relationship between feature channels and can automatically learn the importance of different channel features. According to the importance degree, the important features are enhanced, while unusable features are suppressed. This method adopts global average pooling (GAP), which is simple and efficient, but it is easy to ignore the useful input information. Fair regards nonlocal operations as a generic family of building blocks to capture long-term dependencies. Woo et al. proposed CBAM, which added a channel attention module based on SeNet, and its performance was better than that of SeNet. On the basis of references, Fu et al. (2019) and others proposed Danet to adaptively integrate local features and global dependencies. Two types of attention modules were extended FCN to simulate the semantic interdependence in the spatial and channel dimensions. Yuan (Yuan et al., 2018) proposed OCNet, which is basically the same as DaNet’s method and achieves accurate results in image segmentation. The accuracy of the segmentation results of these two algorithms has been effectively improved, but the time consumption also increases. Based on references, Huang et al. (2019) proposed a cross-attention module that enables each pixel to capture the long dependence of all pixels on it. Compared with the references, the calculation time is reduced by 85%, but the accuracy of segmentation results is sacrificed (Wang et al., 2018). Qin proposed FcaNet (Qin, Zhang, et al., 2020), which is a multispectral channel attention method. In the channel attention mechanism, a discrete cosine transform (DCT) is used to compress the channel, and multiple but limited frequency components of DCT are used as the channel attention mechanism. This does not increase the amount of calculation, and the effect is better than the baseline SeNet. In our article, the attention channel is also improved based on SeNet. The simplicity of the global average pool (GAP) makes it difficult to capture the complex information of various inputs well. Following the example of CBAM, we further use the global maximum pool and the global standard difference pool to enhance the performance of GAP.

Based on the segmentation method, lane line detection is modelled as a per-pixel classification problem, and each pixel is classified as a lane area or background. Our approach is to model long-term correlations between pixels and create correlations between feature maps. Therefore, the generalisation ability of the method is improved by capturing context information.

3. Method

Due to the simple structure of lane lines, position information is more important than shape information in complex environments such as occlusion and low light. Therefore, the lane detection task relies heavily on global context information. This study has two main objectives: to use a space and channel attention module to obtain global context information and improve the accuracy of lane detection in complex environments such as occlusion and low light, and to increase the IBN network to improve network generalisation ability and reduce the amount of computation.

3.1. Architecture

Inspired by the literature (Huang et al., 2019), different types of encoder and decoder combinations can increase the computational efficiency on the premise of ensuring the accuracy of the reference frame. The main structure of the U-Net (Ronneberger et al., 2015) network consists of an encoder, decoder, and bottleneck layer, in which the encoding and decoding are symmetric and skip connection is used. In our study, ResNet34 is used as the encoder module, which gives up average pooling and a fully connected layer retaining the first four feature extraction blocks. The main reason for this is that a shortcut mechanism is introduced by using ResNet34. This avoids gradient disappearance and accelerates network convergence. In the encoder phase, IBN networks are added to ResNet34 to improve network performance without additional computation. The attention mechanism is introduced at the connection between the encoding stage and decoding stage to refine the result of deconvolution by jump connection. The feature map of the attention layer in the encoder is fused with the result of the corresponding deconvolution layer, the structure of which is shown in Figure 1.

Figure 1. Based on ResNet34-UNet network structure, IBN-NET is added to encoding stage, and attention mechanism is added to encoding and decoding stages. Each blue rectangular block represents a multichannel feature map. Left part of network is encoder. Right part of network is decoder. Middle part of network is attention mechanism, which obtains context information.

3.2. IBN-Net

Instance normalization (IN) is more commonly used in style transformations and similar tasks, which allow instance-specific contrast information to be filtered out of the content as they attempt to change the appearance of the image while preserving the content. During the IN statistics calculation, all elements of a single sample and a single channel are considered in the normalisation statistics. Therefore, IN can provide visual and appearance invariance (Ulyanov et al., 2016).

BN, which retains texture-related attributes, is more often applied to the tasks of image classification and semantic segmentation. The traditional deep neural network only standardises the input data before the input layer. The main reason why the gradient disappears is that when the number of neural network layers increases, the weight between layers closer to the input layer cannot be effectively corrected. The disappearance of the gradient causes the deeper neural network to update slowly and even stagnate in the hidden layer close to the input layer. BN normalises the input data of each layer and reduces the scale difference of the input characteristic values, which enlarges the gradient of the network and avoids the problem of gradient disappearance. Therefore, BN can accelerate the convergence speed of model training and make the model training process more stable. According to the literature (Pan et al., 2018), the shallow layer of a neural network expresses more appearance feature information, while the deep layer expresses more semantic information. Therefore, IN is used in the shallow layer, BN is used in the deep layer of the network, and BN is also retained in the shallow layer due to its great role. In this paper, the BN layer is reserved as the feature in the deep layer and is replaced by the IN layer as half the feature in the shallow layer of ResNet34. The other half of the feature is the BN layer. IBN-Net integrates IN and BN as building blocks, which retains their advantages and enhances the learning and generalisation ability of the network, as shown in Fig. 2.

Figure 2. Instance-batch normalization (IBN) block. In first convolution layer, we apply half the IN and BN in channel. In second convolution layer, we apply only BN in channel.

3.3. Attention module

Although most lane detection models adopt a convolution operation, they cannot obtain long-distance context information, which results in incorrect lane detection classification. However, the addition of an attention mechanism has achieved good results (Zhang et al., 2019 ). The structure of a lane line is simple, and its distribution has regularity and continuity, so channel and space information is important. When lane lines are mostly occluded, the local context cannot adequately infer pixel attributes. In this case, the channel attention module and spatial attention module can capture rich context and solve the problem of lane occlusion. In this paper, we combine the channel attention module of the CBAM with the spatial attention module of Danet. To enhance the presentation capability of the network, this paper extends the presentation of each pixel by aggregating the representation of context pixels and considers the relationship between pixels. As shown in Figure 3, the attention module in this paper has two submodules: the spatial attention module and the channel attention module.

Figure 3. Overview of attention module, which consists of two parallel submodules: channel attention module and spatial attention module.

3.3.1. Spatial attention module

The IAED-Net proposed in this paper retains the spatial attention module of DANet, which combines local features with their global dependencies to infer the information of the lane line occlusion area. Rich context relationship models are built on local features, which enhances the presentation capability of the network and improves the accuracy of lane detection. As shown in Figure 3, input features $X\in R^{C\times H\times W}$, A, and B are two new feature maps of X, where $\{B,C\}\in R^{C\times H\times W}$. Then, we perform a matrix multiplication between the transpose of A and B and apply a softmax layer to compute the spatial attention map S: (1) $\begin{array}{r}{S}_{i,j}=\frac{\text{exp}(A_i\cdot B_j)}{{\sum }_{i=1}^N\text{exp}(A_i\cdot B_j)}\end{array}$(1) where $S_{ji}$ measures the $i^{th}$ position’s impact on the $j^{th}$ position. The more similar feature representations of the two positions contribute to greater correlation between them. The final output is as follows: (2) $\begin{array}{r}{E}_S=\alpha \sum _{i=1}^N(s_{ji}{D}_i)+X_j\end{array}$(2) where $\alpha$ is initialised as 0 and gradually learns to assign more weight. The resulting feature $E_S$ for each location is the weighted sum of all the location features and the original features of the lane line. Therefore, the resulting feature has a global context view of lane lines and selectively aggregates contexts based on a spatial attention map.

3.3.2. Channel attention module

Channel attention can more accurately identify lane lines because each channel in the feature map is regarded as a feature detector. Lane detection should take advantage of the interdependencies between channel maps rather than just checking the context on a single channel. Each channel performs a certain function; for example, some channels that should be given special attention play a very important role in the pixel recovery of occluded areas. CBAM computes the spatial information of the feature map, aggregates the spatial information of the feature map by means of the average pool and maximum pool operations, and generates a global context vector, which is input into the multilayer perceptron (MLP). The channel attention is computed as follows: (3) $\begin{array}{rl}C& =\sigma (MIP(AvgPool(X))+MLP(MaxPool(X)))\\ & =\sigma (W_1(W_0(X_{avg}^C))+W_1(W_0(X_{max}^C)))\end{array}$(3) where $\sigma$ denotes the sigmoid function, $W_0ϵR^{C\times C/r}$, and $W_1ϵR^{C\times C/r}$. $X_{avg}^C$ and $X_{max}^C$ represent evaluation pooling characteristics and maximum pooling characteristics, respectively. The MLP weights $W_0$ and $W_1$ are shared for both inputs, and the ReLU activation function is followed by $W_0$. The final output is (4) $\begin{array}{r}{E}_c=C\otimes X\end{array}$(4)

3.3.3. Combination of spatial attention module and combination of modules

To take advantage of the remote context information, we combine the characteristics of the two attention modules. Generally, there are two ways to combine the two modules: sequential connection and parallel connection. In this paper, we use a parallel connection that sums the elements to complete feature fusion to obtain a better representation of lane line features. Finally, the final prediction diagram $E_{fusion}$ is generated by the convolution layer. To reduce the GPU calculation burden, no cascading process is used in this research.

3.4. Loss function

The choice of loss function directly affects the performance of the deep learning network model, which is used to solve and evaluate models by minimising loss functions. We have to choose or design a proper loss function for a particular problem rather than a fixed loss function. The cross entropy based on softmax shows remarkable performance in various lane line detection tasks. However, it cannot clearly guide network learning with highly distinguishing characteristics, resulting in the neglect of the difference in false labels (Tang et al., 2021). To overcome this shortcoming, L-Softmax enables the learned feature distribution to be highly clustered within classes and the distance between classes to be large enough, which enables the model to have a certain ability to prevent overfitting. The literature (Zhang et al., 2018) proposes a boundary perception loss function for lane detection. The formula is shown as follows: (5) $\begin{array}{r}{L}_{IoU-soft}=1-\frac{{\sum }_{p\in \mathcal{L}}(y_p\times g_p)}{{\sum }_{p\in \mathcal{L}}(y_p+g_p-y_p\times g_p)}\end{array}$(5)

4. Experiment

To evaluate the proposed method, we carry out extensive lane detection experiments on the TuSimple and CULane datasets. We use PyTorch as the deep learning framework, and all models are trained on 2 NVIDIA 3080 GPUs with 16 GB memory.

4.1. Dataset

The Tusimple database consists of 3626 training images and 2782 test images. Each image provides the first 19 frames of the image, which are unmarked. The dataset was taken on American highways at different times of the day with good weather conditions. This dataset is relatively simple because the roads were not crowded.

Culane was collected by different drivers with cameras on traffic roads in Beijing. The dataset is divided into 88,880 training sets, 9675 verification sets, and 34,680 test sets. The data include many challenging driving scenarios such as overcast days, insufficient light, rainy days, and shaded roads. It has more diversified traffic conditions and more complex scenes than the Tusimple database.

4.2. Evaluation metrics

In the field of image processing, there are many indices used to evaluate the performance of a model, including true positive (TP), true negative (TN), false-positive (FP) and false negative (FN), precision, recall, and accuracy. Accuracy is the proportion of data (TP + TN) correctly judged by the model in the total data and is the most intuitive evaluation index. The recall rate is the proportion of positive cases (TP) correctly judged by the model to all positive cases (TP + FN) in the dataset. Then, the calculation formulas are as follows: (6) $\begin{array}{rl}Accuracy& =\frac{TP+TN}{N}\end{array}$(6) (7) $\begin{array}{rl}Recall& =\frac{TP}{TP+FN}\end{array}$(7) where N represents the total number of ground truth lane points.

It is not accurate to only use recall or precision to measure the detection of lane markings. The F₁ score considers both precision and recall, and is an indicator to measure the accuracy of two types of models in statistics: (8) $\begin{array}{r}{F}_1=2\times \frac{Precision\times Recall}{Precision+Recall}\end{array}$(8) To determine whether a lane mark is successfully detected, we treat the lane mark as a line with a width of 30 pixels and calculate the cross union (IoU) between the ground truth and the predicted value.

4.3. Implementation details

In this paper, we choose ResNet-34 as our backbone. The images of TuSimple and CULane are resized to 360 × 640 and 288 × 800, respectively. We use SGD to train our model, and the learning rate is set to 0.01. We trained 70 and 16 periods for TuSimple and CULane, respectively, with a batch size of 8. All models were trained and tested using PyTorch, the CPU is an Intel Core i9-10900 K, and the GPUs are 2 NVIDIA 3080s with 16 GB memory. To verify the advantages and disadvantages of the proposed method, different test conditions are used such as night, cloudy, and sunny scenes. In our experiments, except for the improved methods in this paper, other methods are from the literature (Pan et al., 2018), and the evaluation metrics are the same.

4.4. Results

4.4.1. Experiment on TuSimple dataset

To demonstrate that ResNet-UNet and IBN-Net can effectively improve lane detection capability, we compared them with the baseline in the TuSimple database. Table 1 shows that the accuracy of ResNet34-UNet is better than that of UNet and ResNet34, indicating that the improvement in performance comes from the effective design of the network structure. The UNet method of incremental upsampling within the network provides the best accuracy. The experimental results show that the accuracy is improved by 2.3% when we add IBN-Net to ResNet34-UNet. Therefore, when IN is added moderately, content-related information can be well-preserved.

Table 1. Results of lane detection with baseline and IBN-Net in TuSimple datasets.

Model	Accuracy	FP	FN
UNet	84.3%	0.09	0.09
ResNet34	85.3%	0.10	0.12
ResNet34-UNet	87.5%	0.08	0.09
ResNet34-UNet + IBN	89.8%	0.08	0.07

To prove that IBN-Net not only has better lane detection ability but also stronger modelling ability than ResNet34, we compare it with other ResNet families on the TuSimple dataset. The results are listed in Table 2. IBN-Net achieves better results than the other ResNet series, indicating that it has better modelling capabilities. IBN-NET does not introduce any more parameters.

Table 2. Results of IBN-Net over other ResNet families on TuSimple datasets.

Model	Original/Acc	IBN-Net/Acc
ResNet34	85.3%	87.5%
ResNet50	86.6%	87.8%
ResNet101	86%	87.7%

To verify the attention module’s ability to improve lane detection, we compared our model with the baseline in the TuSimple database. Table 3 shows that the accuracy of ResNet34-UNet + IBN with the attention module is 91.6%, which is 1.8% higher than that of ResNet34-UNet + IBN. The results show that attention modules bring great benefits to lane detection.

Table 3. Results of attention over baseline on TuSimple datasets.

Model	Accuracy	FP	FN
ResNet34-Unet + IBN	89.8%	0.08	0.07
ResNet34-Unet + IBN + Attention	91.6%	0.06	0.05

We use ResNet34 as our baseline. The experiment shows in Table 4 that our model, which is the same as ResNet34-UNet, is faster than the baseline. This finding indicates that the ResNet-UNet structure improves speed while maintaining accuracy, and our model has little increase in computational costs even when additional parameters are introduced.

Table 4. Runtimes of Baseline, ResNet34-UNet, and our model. Runtimes of ResNe34t, ResNet34-UNet, and our model CPU are from Intel Core i9-10900K, and GPUs are 2 NVIDIA 3080s with 16 GB memory.

Models	Baseline	ResNet34-UNet	Our model
Input Size $(\mathit{C}\times \mathit{H}\times \mathit{W})$	3 × 360 × 640	3 × 360 × 640	3 × 360 × 640
Times(ms)	70.3	68.7	69.1

4.4.2. Experiment on CULane dataset

To prove the effectiveness of IAED, we compared it with a series of baselines under different scenarios in the CUlane dataset. As shown in Figure 4, which presents the visualisations, the performance of lane detection with IAED is better than ResNet34 and Resnet34-UNet + IBN in the case of shadows, dazzling light, crowding, and so on. Our attention network, which captures long-range contextual information effectively, achieves outstanding performance consistently on five challenging driving scenarios.

Figure 4. Comparison between baseline and our model. First line: strong light. Second line: moderate crowding. Third line: moderate shadows. Fourth line: heavily crowded. Fifth line: heavy shadows.

We compared the approach proposed in this paper with a series of ResNet34 methods in CUlane datasets that contain several challenging driving scenarios. It can be seen from Table 5 that the approach proposed in this paper is better than other methods in lane detection under normal conditions, but the performance difference is not obvious. This is because the image marking of the middle lane line under normal conditions is relatively simple and easy. In some complex cases, the performance of our method in lane line detection is slightly reduced. However, other methods cannot transfer the long-range dependence between pixel and channel dependence, making the performance decrease sharply, which reduces the lane detection ability. Therefore, the proposed method has better generalisation ability compared with other methods. The results show that IBN-Net and the attention module in the encoder-decoder network have excellent lane detection capability in the CUlane datasets. IBN-Net unifies instance normalisation and batch normalisation layers to increase modelling and generalisation capabilities. The attention module combines local features with global information to effectively capture remote context information, which gives accurate lane line detection results to make the network more robust.

Table 5. Results of Attention over baseline on CULane datasets in different scenarios. Experimental evaluation standard is F₁.

Model	Normal	Crowed	Night	Shadow	Noline	Dazzle light
Baseline	88.0	56.7	55.4	53.4	40.1	48.2
ResNet34-Unet	88.5	60.8	55.5	55.4	441	49.9
ResNet34-Unet + IBN	89.8	62.8	60.2	60.8	45.2	55.4
ResNet34-Unet + IBN + Attention	90.8	70.6	70.3	64.5	50.2	63.9

The method proposed in this paper is compared with the existing methods, which proves the excellent effect of the network structure. All other experimental results in Table 6 are from reference (Pan et al., 2018), and all experimental settings remain consistent. It can be seen from Table 6 that our method improves by 3.4% and 4.5% compared with ResNet-50 and MRFN, respectively, in normal scenarios, which is similar to the SCNN. However, in complex cases, the method in this paper is better than the other three methods. The results show that our method can provide more accurate lane line detection results than the existing methods and has better generalisation ability and robustness.

Table 6. Comparison with state-of-the-art methods on CULane datasets in different scenarios. Experimental evaluation standard is F₁.

Model	Normal	Crowed	Night	Shadow	Noline	Dazzle light
ResNet-50	87.4	64.1	60.6	60.7	48.1	58.5
MRFN	86.3	65.2	61.3	54.7	37.2	53.7
SCNN	90.8	69.7	66.1	60.7	43.4	58.5
Our model	90.8	70.6	70.3	64.5	50.2	63.9

To illustrate the advantages of our proposed method more intuitively, we use Acc on the CULane Datasets to compare with other existing methods. As shown in Table 7, our method achieves the best results with the highest Acc.

Table 7. Comparison with state-of-the-art methods on CULane datasets in different scenarios.

	Acc (%)
Model	Normal	Crowed	Night	Shadow	Noline	Dazzle light
ResNet-50	88.4	73.1	72.9	71.8	64.1	68.5
MRFN	87.3	72.2	70.8	65.2	63.2	69.7
SCNN	91.8	80.7	79.1	75.8	69.4	76.5
Our model	92.6	81.6	80.3	79.8	70.9	79.9

At the end of the experiment, we compared the performance and efficiency of our method with those of the SCNN. As seen from Table 6, taking F1 as the evaluation criterion, our proposed method is slightly superior to the SCNN. For a more detailed comparison with the SCNN, we use the mean IoU to evaluate the performance of the two models. The results listed in Table 8 indicate that our method achieves significantly better performance.

Table 8. Comparison between our model and SCNN using mIoU evaluation criteria.

Models	SCNN	Our model
mIou	55.8	56.6

To compare our model with the efficiency of the SCNN, we calculate their runtime experimentally. In the fair comparison case, the input size is modified to be the same as that in the SCNN, and both methods are tested on a GPU. The results show that our model is faster than the SCNN in Table 9. This is because the SCNN requires a high number of convolution operations.

Table 9. Runtimes of our model and SCNN. CPU is Intel Core i9-10900K, and GPUs are 2 NVIDIA 3080s with 16 GB memory.

Models	SCNN	Our model
Input Size $(\mathit{C}\times \mathit{H}\times \mathit{W})$	3 × 288 × 800	3 × 288 × 800
Times(ms)	95.3	74.9

5. Conclusion

This paper proposed a semantic segmentation network IAED for lane detection. This network can retain the details and effectively capture remote context information to deal with the performance degradation of lane detection in complex environments such as occlusion and low light. IBN-net increased modelling and generalisation capabilities. The spatial attention module and channel attention module can effectively capture the global background and judge whether the pixel is in the lane line sign, providing accurate lane line detection results. Our proposed network consistently delivers superior performance in the TuSimple and CULane datasets, especially in complex environments where this approach outperforms previous approaches.

Since lane line detection based on the segmentation method models the lane line as a pixel classification problem rather than a specified line shape, the lane is not regarded as a whole unit. In the future, we want to use rowwise detection methods for lane line detection. We hope to use data enhancement to improve the generalisation ability of the model and consider transformers to improve the detection accuracy of the model.

Author contributions

Conceptualization, Y. S., and L.-Y.W.; formal analysis, investigation, methodology, and validation, Y. S.; software, Y. S., and H.-D.W.; writing – original draft preparation, Y. S.; writing – review and editing, Y. S. and L.-Y.W.; supervision, L.-Y.W. All authors have read and agreed to the published version of the manuscript.

Acknowledgments

The authors would like to acknowledge the technical and administrative support from Key Laboratory of Modern Measurement and Control Technology members throughout the data collections and data calibration.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was funded by the National Defense Science and Technology Project [grant number JCCPCX201705], and Qin Xin Talents Cultivation Program, Beijing Information Science & Technology University [grant numbers QXTCPA201903; QXTCPB201901].