MobileDepth: Monocular Depth Estimation Based on Lightweight Vision Transformer

Figures & data

Figure 1. Multi-head self-attention mechanism.

Figure 2. Overview of the proposed MobileDepth.

Table 1. Comparison of CNNs-based models and transformer-based models.

	Method	δ < 1.25	δ < ${1.25}^2$	δ < ${1.25}^3$	Abs Rel	RMSE	RMSE log
Eigen (Eigen et al. 2014)	CNNs	0.702	0.890	0.890	0.203	6.307	0.282
Zhou (Zhou et al. 2017)	CNNs	0.734	0.902	0.959	0.183	6.709	0.902
Liu (Liu et al. 2015)	CNNs	0.680	0.898	0.967	0.201	6.471	0.273
3Net (VGG) (Poggi et al. 2018)	CNNs	0.844	0.941	0.978	0.119	5.888	0.208
Monodepth2 (Godard et al. 2018)	CNNs	0.864	0.948	0.875	0.109	4.960	0.209
DPT-Hybrid (Ranftl et al. 2021)	Transformer	0.959	0.995	0.999	0.062	2.573	0.092
Adabins (Bhat et al. 2020)	Transformer	0.964	0.995	0.999	0.103	2.360	0.088

Figure 3. Processing of vision transformer.

Figure 4. Dilated self-attention block (Dsab)(top) and standard self-attention block(bottom).

Figure 5. Local and global feature extraction block (LGFE).

Figure 6. MobileNetV2 block.

Figure 7. Residual convolution unit (RCU)(left) and fusion block(right).

Figure 8. (a) Input RGB image (b) Relative depth map predicted by Monodepth2 (c) Metric depth map predicted by MobileDepth.

Figure 9. Qualitative comparison with ground truth.

Table 2. Quantitative results. Compare our method to existing methods on KITTI.

Methods	Year	Abs Rel$\downarrow$	RMSE log$\downarrow$	δ < 1.25$\uparrow$	δ < ${1.25}^2\uparrow$	δ < ${1.25}^3\uparrow$
Eigen (Eigen et al. 2014)	2014	0.203	0.282	0.702	0.890	0.890
Liu (Liu et al. 2015)	2015	0.201	0.273	0.680	0.898	0.967
Zhou (Zhou et al. 2017)	2017	0.183	0.902	0.734	0.902	0.959
Zhan (Zhan et al. 2018)	2018	0.144	0.501	0.815	0.962	0.992
AdaDepth (Kundu et al. 2018)	2018	0.167	0.237	0.771	0.922	0.971
PackNet-Sfm (Guizilini et al. 2019)	2020	0.111	0.189	0.878	0.960	0.982
SDC-Depth (Wang et al. 2020)	2020	0.128	0.497	0.845	0.966	0.990
HR-Depth (Lyu et al. 2020)	2021	0.109	0.185	0.884	0.962	0.983
MonoFormer-Hybrid (Bae et al. 2022)	2022	0.104	0.183	0.891	0.962	0.982
MonoFormer-VIT (Bae et al. 2022)	2022	0.118	0.193	0.873	0.956	0.981
MobileDepth (Ours)	2023	0.094	0.182	0.969	0.993	0.996

Table 3. Quantitative results. Compare our method to existing methods on NYU.

Methods	Year	Abs Rel$\downarrow$	RMSE log$\downarrow$	δ < 1.25$\uparrow$	δ < ${1.25}^2\uparrow$	δ < ${1.25}^3\uparrow$
Eigen (Eigen et al. 2014)	2014	0.158	0.641	0.769	0.950	0.988
Liu (Liu et al. 2015)	2015	0.127	0.506	0.836	0.966	0.991
Xu (Xu et al. 2017)	2017	0.121	0.586	0.811	0.954	0.987
Fu (Fu et al. 2018)	2018	0.115	0.509	0.828	0.965	0.992
GeoNet (Klingner et al.)	2018	0.149	0.226	0.796	0.935	0.975
BTS (Lee et al. 2019)	2019	0.110	0.392	0.885	0.978	0.994
MobileDepth (Ours)	2023	0.087	0.164	0.935	0.991	0.998

Table 4. Comparison of MobileDepth with existing models in parameters.

Model	Year	Method	RMSE$\downarrow$	Parameters$\downarrow$
GeoNet (Klingner et al.)	2018	CNN	5.567	31.6M
DDVO (Wang et al. 2017)	2018	CNN	5.583	28.1M
Monodepth2-Res50 (Godard et al. 2018)	2019	CNN	5.067	32.5M
PackNet-Sfm (Guizilini et al. 2019)	2020	CNN	4.601	128M
SGDepth	2020	CNN	4.693	18.3M
CADepth-Res18 (Yan et al. 2021)	2021	Transformer	4.686	18.8M
MonoFormer-Hybrid (Bae et al. 2022)	2022	Transformer	4.580	159M
MonoFormer-VIT (Bae et al. 2022)	2022	Transformer	4.840	138M
MobileDepth (Ours)	2023	Transformer	4.469	17.6M

Table 5. Complexity of models.

Model	Year	RMSE$\downarrow$	δ < ${1.25}^3\uparrow$	FLOPs (G)	Inference time (ms)
GeoNet (Klingner et al.)	2018	5.567	0.975	42.7	51.3
Monodepth2-Res50 (Godard et al. 2018)	2019	5.067	0.981	49.6	78.5
PackNet-Sfm (Guizilini et al. 2019)	2019	4.601	0.982	205	24
R-MSFM6 (Zhou et al.)	2021	4.981	0.981	31.2	12
MonoViT (Zhao et al. 2022)	2022	4.459	0.984	23.7	47.4
MonoFormer-Hybrid (Bae et al. 2022)	2022	4.580	0.982	111.4	25
MonoFormer-VIT (Bae et al. 2022)	2022	4.594	0.983	96.86	18
MobileDepth (Ours)	2023	4.469	0.996	15.22	9

Table 6. Ablation study for components and loss function.

	Abs Rel$\downarrow$	RMSE$\downarrow$	δ < 1.25$\uparrow$	δ < ${1.25}^2\uparrow$	δ < ${1.25}^3\uparrow$	Parameters
MobileDepth full model	0.094	4.469	0.969	0.993	0.996	17.6 M
without MV2 block	0.139	4.743	0.947	0.958	0.961	23.8 M
without DSAB block	0.103	4.531	0.962	0.990	0.994	320 M
with MSE loss function	0.124	4.617	0.958	0.967	0.978	17.6 M

Figure 10. Quantitative ablation comparison experiments. (a) The input image. (b) The ground truth. (c) The predicted depth map using MSE loss function. (d)The predicted depth map via full model. (e) The predicted depth map without DSAB block. (f) The predicted depth map without MV2 block.

Eigen, D., C. Puhrsch, and R. Fergus. 2014. Depth map prediction from a single image using a multi-scale deep network. NIPS 27. http://arxiv.org/abs/1406.2283.

 Google Scholar

Zhou, T., M. Brown, N. Snavely and D.G Lowe. 2017. Unsupervised learning of depth and ego-motion from video. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 6612–19. Honolulu, HI, USA. July 21–26, 2017.

 Google Scholar

Liu, F., C. Shen, G. Lin, and I. Reid. 2015. Learning depth from single monocular images using deep convolutional neural fields. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38:2024–39. NW Washington, DC, United States.

 Google Scholar

Poggi, M., F. Tosi, and S. Mattoccia. 2018. Learning monocular depth estimation with unsupervised trinocular assumptions. In 2018 International Conference on 3D Vision (3DV), 324–33. Verona, ltaly. September 5–8, 2018.

 Google Scholar

Godard, C., O. Mac Aodha, M. Firman and G.J. Brostow. 2018. Digging into self-supervised monocular depth estimation. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), 3827–37. Seoul, Korea (South), Oct 27–Nov 2, 2019.

 Google Scholar

Ranftl, R., A. Bochkovskiy and V. Koltun. 2021. Vision transformers for dense prediction. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) ,12159–68. Montreal, Canada. Oct 10–17, 2021.

 Google Scholar

Bhat, S., I. Alhashim and P. Wonka. 2020. AdaBins: depth estimation using adaptive bins. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 4008–20. Nashville, TN, USA. June 10–25, 2021.

 Google Scholar

Zhan, H., R. Garg, C.S. Weerasekera, K. Li, H. Agarwal and I. Reid. 2018. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstructionC. In Proceedings of the IEEE conference on computer vision and pattern recognition, 340–49. Salt Lake City, UT, USA. June 18–22, 2018.

 Google Scholar

Kundu, J. N. 2018. AdaDepth: Unsupervised content congruent adaptation for depth estimation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2656–65. Salt Lake City, UT, USA. June 18–22, 2018.

 Google Scholar

Guizilini, V. C., R. Ambrus, S. Pillai, A. Raventos and A. Gaidon. 2019. 3D Packing for self-supervised monocular depth estimation. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2482–91. Seattle, WA, USA. June 13–19, 2020.

 Google Scholar

Wang, L., J. Zhang, O. Wang, Z. Lin and H. Lu. 2020. Sdc-depth: Semantic divide-and-conquer network for monocular depth estimation C. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 541–50. Seattle, WA, USA. June 13–19, 2020.

 Google Scholar

Lyu, X., L. Liu, M. Wang, X. Kong, L. Liu, Y. Liu, X. Chen, and Y. Yuan. 2020. HR-Depth: High resolution self-supervised monocular depth estimation. arXiv Preprint arXiv: 1503025312 35: 07356.

 Google Scholar

Bae, J.-H., S. Moon, and S. Im. 2022. MonoFormer: Towards generalization of self-supervised monocular depth estimation with transformers. arXiv Preprint arXiv: 150302531 2 (37): 11083.

 Google Scholar

Xu, D., E. Ricci, W. Ouyang, X. Wang and N.Sebe. 2017. Multi-scale continuous crfs as sequential deep networks for monocular depth estimationC. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5354–62. Honolulu, HI, USAJuly 21–26, 2017.

 Google Scholar

Fu, H., M. Gong, C. Wang, K. Batmanghelich and D. Tao. 2018. Deep ordinal regression network for monocular depth estimation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2002–11. Salt Lake City, UT, USA. June 18–22, 2018.

 Google Scholar

Lee, J.H., M.K. Han, D.W Ko, and I.H Suh. 2019. From big to small: Multi-scale local planar guidance for monocular depth estimation. arXiv Preprint arXiv: 1907 abs/1907.10326: 10326. http://arxiv.org/abs/1907.10326.

 Google Scholar

Wang, C., J.M. Buenaposada, R. Zhu and S. Lucey. 2017. Learning depth from monocular videos using direct methods. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022–30. Salt Lake City, UT, USA. June 18–22, 2018.

 Google Scholar

Yan, J., H. Zhao, P. Bu, and Y. Jin. 2021. Channel-wise attention-based network for self-supervised monocular depth estimation. In 2021 International Conference on 3D Vision (3DV), 464–73. December 1–3, 2021.

 Google Scholar

Zhao, C., Y. Zhang, M. Poggi, Tosi, F., Guo, X., Zhu, Z., Huang, G., Tang, Y. and S. Mattoccia. 2022. Monovit: Self-supervised monocular depth estimation with a vision transformer C. In 2022 International Conference on 3D Vision(3DV), 668–78. IEEE: Prague, CZ. September 12–16, 2022.

 Google Scholar