MobileDepth: Monocular Depth Estimation Based on Lightweight Vision Transformer

ABSTRACT

As deep learning takes off, monocular depth estimation based on convolutional neural networks (CNNs) has made impressive progress. CNNs are superior at extracting local characteristics from a single image; however, they are unable to manage long-range dependence and thus have a substantial impact on the performance of monocular depth estimation. In addition to this, as architectures based on CNNs frequently utilize down sampling operations, numbers of pixel-level features, which are extremely crucial for dense prediction tasks, are lost in the encoder phase. Unlike CNNs, ViT is capable of capturing global feature information, but it requires numbers of parameters and data augmentation owing to its lack of inductive bias. To address the aforementioned difficulties, in this study, we propose a Dilated Self Attention Block (DSAB) as well as a Local and Global Feature Extraction (LGFE) module. The former resolves the inference speed issue of standard ViT models, and we accomplish this by limiting the number of self-attention computations among tokens. The latter combines the advantages of CNNs and ViT, first extracting local representation information in low-dimensional space through standard convolution and then mapping the input tensor to high-dimensional space to capture global information, achieving the simultaneous extraction of global and local characteristics.

Introduction

Monocular depth estimation is one of the classical problems in computer vision, which can be applied to several computer vision tasks, including 3D reconstruction, autonomous driving, augmented reality and virtual reality, etc. Traditional sensor-based methods of acquiring depth information can yield high-quality depth images, but the equipment is extremely expensive, so there is still a long way to go from large-scale marketization. Compared to binocular cameras, monocular cameras have advantages such as low cost, high popularity and convenient image acquisition, etc. Therefore, monocular depth estimation is currently a more popular and challenging technology.

With the development of deep learning, the ability of CNNs to capture features from a single RGB image has been validated in multiple computer vision tasks, including depth estimation, object detection, semantic segmentation, etc. However, CNNs have obvious drawbacks. Firstly, due to the large number of down sampling operations, the network loses numerous pixel-level features at the encoder edge, and these features cannot be recovered at the decoder stage, which can considerably degrade the performance of monocular depth estimation. Secondly, although the spatial induction bias of CNNs enables them to capture characteristics with lower parameters in diverse visual tasks, it is spatially local and unable to capture global representations, since its ability to withdraw features can only be limited to a fixed convolutional kernel size, such as 3$\times$3 or 5$\times$5.

Recently, the emergence of vision transformer has had a profound impact in the area of computer vision. It compensates for the aforementioned shortcomings of CNNs. Unlike CNNs, vision transformer can capture global features effectively by employing the MSA mechanism, which is shown in Figure 1.

Figure 1. Multi-head self-attention mechanism.

The output vectors b1, b2, b3 are generated from the input token a processed by the self-attention layer (here we take 3 heads for instance), then the vectors are concatenated and linearly transformed to get b. The same process is done for the other inputs in the sequence and all of them share the weight parameters.

The MSA mechanism enables the network to learn cross-relations between patches for a better understanding of the global characteristics of a single image. Compared with the self-attention mechanism, which is defined as equation (1)(1) $\mathit{Attention}\left(\mathit{Q},\mathit{K},\mathit{V}\right)=\mathit{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)\mathit{V}$(1) , where Q, K, V are query, key, and value. The MSA mechanism supports process parallelization, thereby boosting computational performance and guarding against overfitting. However, the MSA mechanism lacks the inductive bias of CNNs, resulting in vision transformer that

(1) $\mathit{Attention}\left(\mathit{Q},\mathit{K},\mathit{V}\right)=\mathit{softmax}\left(\frac{Q{K}^T}{\sqrt{d_k}}\right)\mathit{V}$(1)

requires tremendous parameters and high computation as well as extensive data augmentation. Although the performance of the network is boosted, the cost comes in the form of model size and latency, therefore, deploying the model to edge devices is challenging.

In this paper, we propose a lightweight and efficient monocular depth estimation model that can capture more pixel-level features with fewer parameters. The network as a whole employs an encoder-decoder architecture that is suitable for dense prediction tasks. The encoder consists of the Dilated Self-Attention Block (DSAB) module as well as the MobilenetV2 (Sandler et al. 2018) module that extracts global and local features, respectively. The decoder side employs a multi-scale fusion module, which can extract multi-scale features from images in a coarse-to-fine process. The network architecture is shown in Figure 2.

Figure 2. Overview of the proposed MobileDepth.

This paper is organized as follows. Section 2 introduces some related research works. In Section 3, we will illustrate our network architecture in detail. Section 4 will describe our experimental procedures and results. Section 5 is the conclusion of this paper.

Related Work

CNNs Based Monocular Depth Estimation

Monocular depth estimation is an ill-posed problem. A single RGB image may corresponds to a variety of real scenes, but there are no stable cues in the image to constrain these possibilities. Therefore, it is difficult to capture depth information directly from a single image. In recent years, researchers have proposed several methods to extract depth information from a single RGB image (Fu et al. 2018; Kim 2014; Long, Shelhamer, and Darrell 2015). Eigen et al. (2014) used neural networks for the first time to obtain depth maps. They integrated local and global characteristics to predict the depth map, first design a coarse network to emphasize the global features, and then using a fine network to enhance the quality of the depth map. Soon after, Li et al. (2015) adopted regression of deep convolutional neural networks and conditional random fields for post-processing refinement to address the issue of unpredictable surface normal to monocular images. Fu et al. (2018) introduced a ranking mechanism in their model to estimate the depth images more accurately. Specifically, they first pre-divided the truth depth into many sub-intervals according to interval increments method and designed a pixel-to-pixel ordered regression loss function to model the ordered relationship between these depth sub-intervals. Unlike traditional encoder-decoder architectures, they employed dilated convolution network to better extract multi-scale features. Liu et al. (2015) proposed a model that does not require geometric prior knowledge and can be applied in general scenarios, which aims to jointly explore the capabilities of deep CNNs and continuous conditional random fields, proposing an equivalent model based on fully CNNs and a novel super-pixel merging method. Different from Liu et al., Gan et al. (2018) exploited the prior knowledge, and they introduced vertical pooling in the proposed model to aggregate the features in the vertical direction and improve the accuracy of depth prediction. Considering that the ground truth collected by radar is generally sparse, they also utilized a stereo matching network that takes left and right paired images as input to obtain a high-quality dense depth map. In 2019, Kaur et al. introduced attention to tools for enhancer prediction working on common features of enhancer (Kaur, Chauhan, and Aggarwal 2019). Soon, discovered the importance of the loss function and they proposed Monodepth2 Godard et al. (2018), which used the minimum reprojection loss function to handle the occlusion problem and the auto-masking loss function to ignore training pixels that violate the camera motion assumption and filter pixels that are stationary and relatively stationary.

Transformer Based Monocular Depth Estimation

Although the above-mentioned CNNs-based methods have achieved promising performance in the domain of monocular depth estimation, the limitation of these networks is that they are unable to extract global features well. Recently, transformer has made a significant breakthrough in natural language processing. The model discards the traditional recurrent neural network and convolutional neural network and purely utilizes the attention mechanism, addressing the issue of long-distance dependency of input sequences. Vision transformer was the first model to introduce transformer into computer vision. Experiments have shown that the ability of vision transformer to extract image features is quite promising. In the domain of monocular depth estimation, researchers have also employed vision transformer as a backbone network for encoders, and have achieved strong performance. (Bhat et al. 2020). proposed AdaBins, a transformer-based model that divides the depth range into multiple units. The central value of each unit is estimated adaptively for each image, and the final depth value of the image is estimated as a linear combination of the units. Vision Transformers for Dense Prediction (DPT) employs a traditional encoder-decoder architecture suitable for dense prediction tasks Ranftl et al. (2021), the encoder side utilizes transformer as the backbone to extract the global features of the image. Compared to fully convolutional neural networks, DPT eliminates the down sampling operation and vision transformer provides finer-grained and more globally consistent predictions. After DPT was proposed, Bae et al. (2022) proposed the MonoFormer. They observed that transformer has a strong shape bias, while the CNN exhibits a sharp texture bias. Compared to models that heavily rely on texture bias, models based on shape bias shows better generalization performance in terms of monocular depth estimation. Based on these observations, they designed a hybrid CNN-Transformer network with a multi-level adaptive feature fusion module, which enhances the shape bias by transformer while compensating for the weak local bias of Transformer by adaptively fusing multi-level representations. MonoViT Zhao et al. (2022), proposed by Zhao et al., is a brand-new framework combining the global reasoning enabled by ViT models with the flexibility of self-supervised monocular depth estimation. It achieves SOTA performance on the KITTI dataset by virtue of the combination of the local detail-awareness capability of CNN and the global long-range feature extraction capability of Transformer. It can be seen that Transformer plays an increasingly important role in the field of monocular depth estimation. However, due to the excessive use of the transformer layers that include many fully connected layers, the inference speed of transformer-based model is still an issue. To address the matter, Ibrahem et al. proposed RT-ViT (Ibrahem, Salem, and Kang 2022), which has different combinations composed of four ViT-based encoder architectures and four different CNN-based decoder architectures. This main architecture of the proposed method consists of a vision transformer encoder and a convolutional neural network decoder. They started by training the base vision transformer with 12 transformer layers then they reduced the transformer layers to six layers, and four layers, to obtain real-time processing. It is not the only case. In 2023, investigated the effective combination of CNN and Transformer and proposed a hybrid architecture named Lite-Mono Zhang et al. (2023) They proposed a Consecutive Dilated Convolution (CDC) module and a Local-Global Features Interaction (LGFI) module. The former is used to extract abundant multi-scale local features, and the latter encodes long-range global information into the feature map by utilizing self-attention mechanism. Experiments show that Lite-Mono greatly outperforms Monodepth2 in terms of accuracy, and the trainable parameters are reduced by about 80%.

Compared to the methods mentioned above, we take full advantage of the ability of transformer to capture long-distance dependencies and compensate for the disadvantage of transformer which lacks the capacity to extract local features with MobileNetV2 (Sandler et al. 2018). Furthermore, our method utilizes extremely fewer parameters to achieve stronger performance so that it can be deployed on edge devices, which greatly contributes to the development and popularity of MDE.

Method

Design Motivation

Several papers have demonstrated that vision transformer achieves better performance in dense prediction tasks. Employing vision transformer as an encoder can extract more effective features from a single image than CNNs, and therefore get better performance on the decoder side. We investigated several vision transformer-based monocular depth estimation models and found that they provide a significant performance improvement over traditional CNNs. (See details in Table 1). However, there are two sides to everything. The significant performance improvement comes at the cost of a large model and a huge computational effort that results in their inability to apply on edge devices. DPT-large contains amazingly 343 M parameters, and even DPT-base with 112 M parameters, which makes it difficult to deploy on mobile devices. Compared to CNNs-based models, parameters of transformer are tremendous. Taking Monodepth2-Res18 for example, it only takes 14.3 M parameters.

Table 1. Comparison of CNNs-based models and transformer-based models.

	Method	δ < 1.25	δ < ${1.25}^2$	δ < ${1.25}^3$	Abs Rel	RMSE	RMSE log
Eigen (Eigen et al. 2014)	CNNs	0.702	0.890	0.890	0.203	6.307	0.282
Zhou (Zhou et al. 2017)	CNNs	0.734	0.902	0.959	0.183	6.709	0.902
Liu (Liu et al. 2015)	CNNs	0.680	0.898	0.967	0.201	6.471	0.273
3Net (VGG) (Poggi et al. 2018)	CNNs	0.844	0.941	0.978	0.119	5.888	0.208
Monodepth2 (Godard et al. 2018)	CNNs	0.864	0.948	0.875	0.109	4.960	0.209
DPT-Hybrid (Ranftl et al. 2021)	Transformer	0.959	0.995	0.999	0.062	2.573	0.092
Adabins (Bhat et al. 2020)	Transformer	0.964	0.995	0.999	0.103	2.360	0.088

Therefore, how to make the model smaller while maintaining its existing performance is a topic worth investigating. To solve the above problems, in this paper, by taking full advantage of the ability of vision transformer to capture long-distance relations and compensate for the shortcomings of vision transformer in local feature extraction through MobileNetV2 (Sandler et al. 2018), we designed a lightweight encoder–decoder network architecture. Experiments demonstrate that our performance outperforms traditional CNNs-based monocular depth estimation models, at the same time, the parameters of our method are drastically reduced compared to the vision transformer-based model.

Architecture

The network employs an encoder-decoder architecture in general. The encoder consists of the MobileNetV2 block (MV2) and the Local and Global Feature Extraction (LGFE), which are responsible for capturing short-distance relationship and extracting global features, respectively. The decoder is composed of a reshape block, which is responsible for transforming the feature map generated from the encoder into multiple scales, as well as a fusion block, which is designed to fuse the rich hierarchical features and generate a high-resolution depth map.

Dilated Self-Attention Block

In order to fine-tune the global properties of a single image and decrease the parameters of the vision transformer, we propose the Dilated Self-Attention Block (DSAB). The traditional vision transformer, which approach consists generally of the following steps that are shown in Figure 3. (1) Split the input image into many patches. Taking ViT-B/16 as an example, it divides the input image, whose resolution is 224 $\times$ 224, into many patches of 16$\times$16 size, generating 196 patches. (2) Each patch is processed through the embedding layer, which maps the three-dimensional matrix of each patch into one-dimensional vector that is called a token. The length of each token is 768. (3) Position encoding operation for all tokens so that the input token has position information. (4) Input each layer-normalized token to the MSA layer. It can be found that the MSA layer of traditional vision transformer requires each token to perform self-attention operation with all other tokens, including itself, as illustrated in Figure 4. Although features of images can be extracted better, it will consume tremendous computation resources.

Figure 3. Processing of vision transformer.

Figure 4. Dilated self-attention block (Dsab)(top) and standard self-attention block(bottom).

Inspired by MobileViT (Lyu et al. 2020), we reduce parameters by limiting the amount of self-attention calculations between tokens, as there are enormous redundant pixels in RGB images, especially for images with high resolution. Furthermore, since there is not much feature variation between adjacent pixels, calculating self-attention among all tokens is both superfluous and highly expensive. Specifically, we first split the tokens into groups, and then the tokens in the same position of each group perform self-attention computation, as shown in Figure 4. In this method, it not only drastically decreases parameters by three quarters but also enhances the performance of global learning of the network.

Local and Global Feature Extraction (LGFE)

LGFE is designed to extract local and global features of single image with smaller parameters, as illustrated in Figure 5. The input image is first processed by the convolution layer; here we use the standard n × n convolution to extract the local features of the image. Then the input tensor is projected into a high-dimensional space by 1 × 1 convolution. Long-distance dependencies on feature points can be effectively handled by DSAB. The feature maps obtained by convolution layer are fed to the DSAB module so that the purpose of global modeling can be achieved. Finally, projecting the feature map from high-dimension into raw image through point-wise convolution. The whole module makes good use of inductive bias property of CNNs (short distance) and global feature extraction capability of vision transformer (long-distance dependency) at a small cost in parameters.

Figure 5. Local and global feature extraction block (LGFE).

MobileNetv2 Block

MobileNetV2 (Sandler et al. 2018), which is proposed by Howard et al., is a lightweight network architecture that can be applied to edge devices, as shown in Figure 6. Compared to traditional convolutional neural networks, MV2 considerably reduces parameters and the number of computations of model in the price of a tiny drop in accuracy. Firstly, the input feature map is raised to a higher dimension by pointwise convolution, which aims to prevent missing characteristic information, and here the Relu6 activation function is used. Then, instead of standard convolution, MV2 uses depth-wise convolution to extract the features of each channel. Finally, the feature map is reverted to the original dimension through pointwise convolution.

Figure 6. MobileNetV2 block.

Fusion Block

In the decoder end, we use fusion block based on RefineNet (Lin et al. 2016) to progressively fuse features from different scale layers so that we can generate high-resolution images. The Fusion block consists mainly of the residual convolution unit (RCU), as shown in Figure 7. This module is similar to Resnet in design philosophy, which enables the network to capture more hierarchical features. After RCU, feature maps of hierarchical layers are progressively up-sampled by twice the size, and here we use bilinear interpolation. Finally, the bottom layer is processed by the output head to generate the final prediction.

Figure 7. Residual convolution unit (RCU)(left) and fusion block(right).

Scale-And-Shift Invariant Trimmed Loss

Mixing multiple datasets in training can improve the performance of monocular depth estimation. Because it is challenging to obtain dense ground truth maps in different scenes, this results in many datasets with different characteristics and biases. The depth range is inconsistent across datasets, for instance, the ground truth depth values for ETH3D, KITTI, NYU, and TUM are within 72, 80, 10, and 10 m respectively. However, although the depth range varies in the datasets, the distribution of depth values is similar. Therefore, it is important to keep the scale and shift consistent. In this paper, we design the following loss function to address this problem.

(2) $L_{\mathit{mse}}\left(\hat{d},{\hat{d}}^{\ast }\right)=\frac{1}{2\mathit{M}}{\sum }_{i=1}^M\mathit{\rho }\left(\widehat{d_i},{\widehat{d_i}}^{\ast }\right)$(2)

where $\hat{d},{\hat{d}}^{\ast }$are the aligned predicted depth values and ground truth, respectively; M denotes the number of valid ground truth depth values; ρ denotes specific loss function, here we use MSE_Loss, see equation 3.

(3) $\mathit{MSELoss}\left(x_i,y_i\right)=(x_i-y_i{)}^2$(3)

$\hat{d}$, ${\hat{d}}^{\ast }$ is calculated by least squares method, see equation 4.

$\left(\mathit{s},\mathit{t}\right)=\mathit{argmin}\sum _{\mathit{i}=1}^M(\mathit{s}{d}_i+\mathit{t}-d_i^{\ast }{)}^2$

(4) $\hat{\mathit{d}}=\mathit{s}\mathit{d}+\mathit{t},{\hat{\mathit{d}}}^{\mathbf{\ast }}={\mathit{d}}^{\mathbf{\ast }}$(4)

where d, $d^{\ast }$ denotes predicted depth value and ground truth, ensuring the consistency of the scale and distribution. Due to the direction of MSE gradient update is easily affected by outliers, it is not robust. Therefore, we add a robust loss function, and here we take the gradient descent loss, see equation 5.

$L_{\mathit{gradient}}\left(\hat{d},{\hat{d}}^{\ast }\right)=\frac{1}{M}{\sum }_{k=1}^k{\sum }_{i=1}^M\left(\left|{\mathrm{\nabla }}_x{R}_i^k\left|+\right|{\mathrm{\nabla }}_y{R}_i^k\right|\right)$

(5) $R_i={\hat{\mathit{d}}}_{\mathit{i}}-{{\hat{\mathit{d}}}^{\mathbf{\ast }}}_{\mathit{i}}$(5)

$\mathrm{where}{R}^k$denotes the difference of the depth map at k scales, where k = 4,and halve image resolution at each scale. Ultimately, the loss function takes the following form:

(6) $\mathit{L}=L_{\mathit{mse}}\left(\hat{d},{\hat{d}}^{\ast }\right)+L_{\mathit{gradient}}\left(\hat{d},{\hat{d}}^{\ast }\right)$(6)

The loss function supports training on different datasets and can quantify the values of existing depth estimation datasets, exploring the best strategy for mixed datasets during training. Compared to the traditional L1 and L2 loss function, it converges faster and possesses stronger generalization and robustness.

Experiment

Datasets

KITTI is the largest monocular depth estimation evaluation dataset currently, which is captured by mobile vehicles equipped with four cameras,3D lidar and GPU/IMU. It contains multiple scenes, such as road, city, residential, campus, person. We use the raw data of the city, residential, and road scenes to train the model. Following Eigen split, we use 29K images for training, 4424 images for validation, and 697 images for testing. The resolution of the original RGB images is around 1242 $\times$ 375, while the corresponding ground truth depth image captured by lidar is sparse. When evaluating, we limit the predicted depth image to [0,80] meters and use NYU Depth Toolbox to complete the predicted depth image to ground truth resolution.

The NYU dataset, created by researchers from the Tandon School of Engineering at New York University (NYU), is an open dataset widely used in the field of computer vision. The main purpose of this dataset is to promote and drive research on computer vision and deep learning algorithms. It is mainly composed of the NYU Depth V2 dataset, which contains numbers of RGB images and corresponding depth maps of indoor scenes, covering multiple scenes such as the living room, bedroom, and kitchen. RGB images provide visible surface information about the scene, while depth maps provide distance information from each pixel to the nearest surface in the scene, providing a foundation for studying depth perception and 3D understanding.

Metric and Relative Depth

Monocular depth estimation is categorized into metric depth estimation and relative depth estimation, as shown in Figure 8, and the former can predict the realistic ground depth. Relative depth estimation, which is unable to represent realistic depth information, merely predicts the relative depth discrepancy among pixels. Metric depth estimation obtains the distance between camera and target object via Lidar, binocular camera, TOF, and it is usually measured in meters. Compared to relative depth estimation, absolute depth estimation is more useful in downstream tasks such as navigation, 3D modeling, and object recognition. In our experiments, we use the KITTI raw dataset, which consists of sparse absolute depth maps obtained by Lidar, to estimate metric depth information.

Figure 8. (a) Input RGB image (b) Relative depth map predicted by Monodepth2 (c) Metric depth map predicted by MobileDepth.

Evaluation Methodology

In our experiments, our evaluation method follows the standard approach of previous work on monocular depth estimation (Eigen et al. 2014). It mainly includes the following six evaluation criteria. (1) Absolute Relative error (Abs Rel), which normalize per-pixel errors, reducing the effect of large errors with distance. (2) Square Relative error (Sq Rel), the squared penalizes for larger depth error. (3) Root Mean Squared error (RMSE), a traditional method for measuring regression errors. (4) Root Mean Squared logarithm error ($\mathit{RMS}{E}_{\mathit{log}}$), the logarithm makes this error relative, reducing the effect of large errors with the distance. We also use accuracy under thresholds (${\delta }_i<{1.25}^i,\mathit{i}=1,2,3$).

(7) $\mathit{AbsRel}=\frac{1}{n}\sum \left|\frac{{\mathit{y}}_{\mathit{pred}}-y_{\mathit{gt}}}{y_{\mathit{gt}}}\right|$(7)

(8) $\mathit{SqRel}=\frac{1}{n}\sum {\left(\frac{{\mathit{y}}_{\mathit{pred}}-y_{\mathit{gt}}}{y_{\mathit{gt}}}\right)}^2\mathit{\hspace{0.17em}}$(8)

(9) $\mathit{RMSE}=\sqrt{\frac{1}{n}\sum {\left({\mathit{y}}_{\mathit{pred}}-y_{\mathit{gt}}\right)}^2}$(9)

(10) $\mathit{RMS}{E}_{\mathit{log}}=\sqrt{\frac{1}{n}\sum {\left(log\left({\mathit{y}}_{\mathit{pred}}\right)-\mathit{log}\left(y_{\mathit{gt}}\right)\right)}^2}$(10)

Implementation Details

We build the entire network using PyTorch, and experiments are deployed on Nvidia RTX 3090 GPUs. In our experiments, we take batch size as 4, and the entire model achieves convergence after 50 rounds of training, which takes about 35 hours. To take advantage of the features pre-trained on ImageNet (Deng et al. 2009), we use two optimizers to train the model. Specifically, we optimize the network parameters from scratch with $O_{\mathit{scratch}}$ optimizer and optimize our backbone network with $O_{\mathit{backbone}}$ optimizer, where $O_{\mathit{scratch}}$ and $O_{\mathit{backbone}}$ are both Adam optimizers, but the learning rates are $3.0\times {10}^{-5}$ and $1.0\times {10}^{-6}$ respectively, and for a better weight initialization, the momentum is set to be 0.9.

Date Augmentation

In order to enhance robustness of the model, following Ranftl et al. (2021), we apply a random horizontal flipping (70%) and a random rotation (20%, with a maximum angle value of ${10}^o$) as well as a random cropping (10%).

Experimental Results and Discussions

We evaluated the proposed model and existing lightweight networks on the KITTI test set and conducted both qualitative and quantitative comparisons. Figure 9 shows the qualitative comparison results, where we compared the depth map predicted by our model with the ground truth. As shown in the figure, the model can perfectly duplicate ground truth map and effectively predict the metric depth information in detail as well. In Table 2, the proposed model is compared quantitatively with existing works in recent years. It can be clearly seen that our model not only outperforms traditional CNNs-based models but also outperforms transformer-based networks with much fewer parameters. Table 3 shows the quantitative results obtained from the NYU dataset. In Tables 4 and 5, we compared our model with the existing models mainly in terms of parameters and the Floating-Point Operations (FLOPs). The results show that our model achieves outstanding improvements in terms of number of parameters, and simultaneously, the performance has been enhanced as well. For instance, compared to MonoFormer-VIT (Bae et al. 2022), which is transformer-based and needs 138 M parameters, our model only consumes 17.65 M parameters and achieves better performance. As can be noticed, the parameters have decreased by approximately 87.8%.

Figure 9. Qualitative comparison with ground truth.

Table 2. Quantitative results. Compare our method to existing methods on KITTI.

Methods	Year	Abs Rel$\downarrow$	RMSE log$\downarrow$	δ < 1.25$\uparrow$	δ < ${1.25}^2\uparrow$	δ < ${1.25}^3\uparrow$
Eigen (Eigen et al. 2014)	2014	0.203	0.282	0.702	0.890	0.890
Liu (Liu et al. 2015)	2015	0.201	0.273	0.680	0.898	0.967
Zhou (Zhou et al. 2017)	2017	0.183	0.902	0.734	0.902	0.959
Zhan (Zhan et al. 2018)	2018	0.144	0.501	0.815	0.962	0.992
AdaDepth (Kundu et al. 2018)	2018	0.167	0.237	0.771	0.922	0.971
PackNet-Sfm (Guizilini et al. 2019)	2020	0.111	0.189	0.878	0.960	0.982
SDC-Depth (Wang et al. 2020)	2020	0.128	0.497	0.845	0.966	0.990
HR-Depth (Lyu et al. 2020)	2021	0.109	0.185	0.884	0.962	0.983
MonoFormer-Hybrid (Bae et al. 2022)	2022	0.104	0.183	0.891	0.962	0.982
MonoFormer-VIT (Bae et al. 2022)	2022	0.118	0.193	0.873	0.956	0.981
MobileDepth (Ours)	2023	0.094	0.182	0.969	0.993	0.996

Table 3. Quantitative results. Compare our method to existing methods on NYU.

Methods	Year	Abs Rel$\downarrow$	RMSE log$\downarrow$	δ < 1.25$\uparrow$	δ < ${1.25}^2\uparrow$	δ < ${1.25}^3\uparrow$
Eigen (Eigen et al. 2014)	2014	0.158	0.641	0.769	0.950	0.988
Liu (Liu et al. 2015)	2015	0.127	0.506	0.836	0.966	0.991
Xu (Xu et al. 2017)	2017	0.121	0.586	0.811	0.954	0.987
Fu (Fu et al. 2018)	2018	0.115	0.509	0.828	0.965	0.992
GeoNet (Klingner et al.)	2018	0.149	0.226	0.796	0.935	0.975
BTS (Lee et al. 2019)	2019	0.110	0.392	0.885	0.978	0.994
MobileDepth (Ours)	2023	0.087	0.164	0.935	0.991	0.998

Table 4. Comparison of MobileDepth with existing models in parameters.

Model	Year	Method	RMSE$\downarrow$	Parameters$\downarrow$
GeoNet (Klingner et al.)	2018	CNN	5.567	31.6M
DDVO (Wang et al. 2017)	2018	CNN	5.583	28.1M
Monodepth2-Res50 (Godard et al. 2018)	2019	CNN	5.067	32.5M
PackNet-Sfm (Guizilini et al. 2019)	2020	CNN	4.601	128M
SGDepth	2020	CNN	4.693	18.3M
CADepth-Res18 (Yan et al. 2021)	2021	Transformer	4.686	18.8M
MonoFormer-Hybrid (Bae et al. 2022)	2022	Transformer	4.580	159M
MonoFormer-VIT (Bae et al. 2022)	2022	Transformer	4.840	138M
MobileDepth (Ours)	2023	Transformer	4.469	17.6M

Table 5. Complexity of models.

Model	Year	RMSE$\downarrow$	δ < ${1.25}^3\uparrow$	FLOPs (G)	Inference time (ms)
GeoNet (Klingner et al.)	2018	5.567	0.975	42.7	51.3
Monodepth2-Res50 (Godard et al. 2018)	2019	5.067	0.981	49.6	78.5
PackNet-Sfm (Guizilini et al. 2019)	2019	4.601	0.982	205	24
R-MSFM6 (Zhou et al.)	2021	4.981	0.981	31.2	12
MonoViT (Zhao et al. 2022)	2022	4.459	0.984	23.7	47.4
MonoFormer-Hybrid (Bae et al. 2022)	2022	4.580	0.982	111.4	25
MonoFormer-VIT (Bae et al. 2022)	2022	4.594	0.983	96.86	18
MobileDepth (Ours)	2023	4.469	0.996	15.22	9

Moreover, we introduce the FLOPs to measure model complexity, which is often used as an indirect measure of model speed. It should be pointed out that the MACs (Multiply-Accumulate Operations), which are often confused with the concept of FLOPs, are actually twice as many as FLOPs. The result can be found in Table 5, in comparison to MonoFormer-VIT, FLOPs of MobileDepth are just 15.22 G, an 84.24% reduction in operations. Furthermore, Inference time is an important indicator for evaluating the response speed of a model in practical applications. MobileDepth leads all other models at a speed of 9 ms, which is 25% faster than the second fastest model R-MSFM6 at 12 ms. This is crucial for applications that require quick response, such as autonomous driving and intelligent navigation. In terms of accuracy, at a threshold size of δ < ${1.25}^3$, the accuracy of MobileDepth is 0.996, which is a significant improvement compared to other models. For example, compared to PackNet Sfm’s 0.982, MobileDepth has increased by approximately 1.4%, indicating that MobileDepth can more accurately predict depth. In summary, the MobileDepth model has achieved significant efficiency improvements with minimal accuracy loss. While maintaining high accuracy, it significantly reduces the demand for computing resources and inference time, making it very suitable for use on mobile and embedded devices, especially those applications that have strict requirements for real-time performance and energy consumption.

Ablation Study

To check the effectiveness of the proposed model, we performed ablation experiments on some modules of the architecture. We adjusted and replaced some modules to meet our purpose. Furthermore, we performed quantitative and qualitative ablation comparisons on the KITTI dataset, as shown in Table 6 and Figure 10, respectively.

Table 6. Ablation study for components and loss function.

	Abs Rel$\downarrow$	RMSE$\downarrow$	δ < 1.25$\uparrow$	δ < ${1.25}^2\uparrow$	δ < ${1.25}^3\uparrow$	Parameters
MobileDepth full model	0.094	4.469	0.969	0.993	0.996	17.6 M
without MV2 block	0.139	4.743	0.947	0.958	0.961	23.8 M
without DSAB block	0.103	4.531	0.962	0.990	0.994	320 M
with MSE loss function	0.124	4.617	0.958	0.967	0.978	17.6 M

Figure 10. Quantitative ablation comparison experiments. (a) The input image. (b) The ground truth. (c) The predicted depth map using MSE loss function. (d)The predicted depth map via full model. (e) The predicted depth map without DSAB block. (f) The predicted depth map without MV2 block.

Impact of MobileNetv2

When we remove the MV2 block and only employ the DSAB module to extract features from a single image, we see that the number of parameters increases and the model cannot predict the depth information in the image details. According to the findings, we believe that the MV2 module is essential for the entire architecture because it compensates well for the DSAB module, which lacks the ability to extract local characteristics, so that depth information on the image details can be recovered.

Impact of DSAB Block

In the DSAB module, we attempt to set the size of each group to 1. In other words, the DSAB module was replaced with the standard self-attention block. It turns out that without the DSAB module not only the number of parameters in the model increases significantly (17 M vs. 320 M) but also the accuracy drops. Consequently, when compared to the standard self-attention block, the DSAB module considerably enhances global learning ability of the model by decreasing self-attention computation between redundant pixels in the image, and more importantly, the number of parameters is substantially reduced as a result.

Impact of Loss Function

In this paper, we use the scale-and-shift-invariant trimmed loss function, which aims to improve the performance of the model when training on mixed datasets. When we replace our loss function with others, such as MSE and SGD, the performance suffers as well. This is because the depth range is inconsistent across datasets. When training on a mixed dataset, the scale and shift must be kept consistent, which is extremely critical for the training results. Furthermore, even within the same dataset, the scenes differ, for example, KITTI contains road, city, residential, campus, and human scenes; therefore, there is bias in the characteristics among different scenes. The proposed loss function addresses this problem well and updates the weights of the model wonderfully via back propagation.

Conclusion

We suggest MobileDepth, a novel and lightweight monocular depth estimation model. From a single image, our method can effectively extract both local and global features. On the encoder side, we combine the advantages of CNN and vision transform, fully exploiting the inductive bias property of CNN and the ability to handle long-distance dependencies of vision transformer. Furthermore, we propose the Dilated Self-Attention Block (DSAB) to shrink the model by reducing the redundant self-attention computation operations among patches of vision transformer. MobileDepth can predict high-resolution metric depth maps that are helpful for downstream tasks and can be deployed on edge devices. Experiments demonstrate that MobileDepth improves monocular depth estimation significantly with a small number of parameters. In the future, we will concentrate on lightweight monocular depth estimation based on self-supervised method, which is one of the urgent improvements in this paper. Compared with supervised learning, self-supervision has gradually become one of the research hotspots in recent years for its easy training and low cost.

Disclosure Statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

The work was supported by the National Natural Science Foundation of China [62071006].