Advanced search
Publication Cover
Connection Science Volume 35, 2023 - Issue 1
Submit an article Journal homepage
939
Views
0
CrossRef citations to date
0
Altmetric
Special Issue: Efficient Deep Neural Networks for Image Processing in End Side Devices

FPGA-oriented lightweight multi-modal free-space detection network

Feiyi Fanga School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, People's Republic of China
,
Junzhu Maoa School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, People's Republic of China
,
Wei Yub Beijing RICH AI information Technology Co., Ltd, Beijing, People's Republic of China
&
Jianfeng Lua School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, People's Republic of ChinaCorrespondence[email protected]
Article: 2159333 | Received 31 May 2022, Accepted 06 Dec 2022, Published online: 28 Dec 2022

References

  • Badrinarayanan, V., Kendall, A., & Cipolla, R. (2017). Segnet: A deep convolutional encoder–decoder architecture for image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(12), 2481–2495. https://doi.org/10.1109/TPAMI.34
  • Bai, L., Lyu, Y., & Huang, X. (2020). Roadnet-rt: High throughput CNN architecture and SOC design for real-time road segmentation. IEEE Transactions on Circuits and Systems I: Regular Papers, 68(2), 704–714. https://doi.org/10.1109/TCSI.8919
  • Bai, X., Wang, X., Liu, X., Liu, Q., Song, J., Sebe, N., & Kim, B. (2021). Explainable deep learning for efficient and robust pattern recognition: A survey of recent developments. Pattern Recognition, 120, Article ID 108102. https://doi.org/10.1016/j.patcog.2021.108102
  • Caltagirone, L., Bellone, M., Svensson, L., & Wahde, M. (2019). LIDAR–camera fusion for road detection using fully convolutional neural networks. Robotics and Autonomous Systems, 111, 125–131. https://doi.org/10.1016/j.robot.2018.11.002
  • Carreira-Perpinán, M. A., & Idelbayev, Y. (2018). ‘Learning-compression’ algorithms for neural net pruning. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 8532–8541). Salt Lake City, USA.
  • Chen, L. C., Papandreou, G., Kokkinos, I., Murphy, K., & Yuille, A. L. (2017). Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40(4), 834–848. https://doi.org/10.1109/TPAMI.2017.2699184
  • Chen, X., Wang, Y., Zhang, Y., Du, P., Xu, C., & Xu, C. (2020). Multi-task pruning for semantic segmentation networks. arXiv preprint arXiv:2007.08386.
  • Chen, Z., & Chen, Z. (2017). RBNet: A deep neural network for unified road and road boundary detection. In International conference on neural information processing (pp. 677–687). Guangzhou, China.
  • Chen, Z., Zhang, J., & Tao, D. (2019). Progressive lidar adaptation for road detection. IEEE/CAA Journal of Automatica Sinica, 6(3), 693–702. https://doi.org/10.1109/JAS.6570654
  • Cordts, M., Rehfeld, T., Schneider, L., Pfeiffer, D., Enzweiler, M., Roth, S., Pollefeys, M., & Franke, U. (2017). The stixel world: A medium-level representation of traffic scenes. Image and Vision Computing, 68, 40–52. https://doi.org/10.1016/j.imavis.2017.01.009
  • Couprie, C., Farabet, C., Najman, L., & LeCun, Y. (2013). Indoor semantic segmentation using depth information. arXiv preprint arXiv:1301.3572.
  • Dubey, A., Chatterjee, M., & Ahuja, N. (2018). Coreset-based neural network compression. In Proceedings of the European conference on computer vision (ECCV) (pp. 454–470). Munich, Germany.
  • Eigen, D., & Fergus, R. (2015). Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture. In Proceedings of the IEEE international conference on computer vision (pp. 2650–2658). Santiago, Chile.
  • Fan, R., Wang, H., Cai, P., & Liu, M. (2020). SNE-RoadSeg: Incorporating surface normal information into semantic segmentation for accurate freespace detection. In European conference on computer vision (pp. 340–356).
  • Fritsch, J., Kuehnl, T., & Geiger, A. (2013). A new performance measure and evaluation benchmark for road detection algorithms. In 16th international IEEE conference on intelligent transportation systems (ITSC 2013) (pp. 1693–1700). Hague, Netherland.
  • Fu, J., Liu, J., Tian, H., Li, Y., Bao, Y., Fang, Z., & Lu, H. (2019). Dual attention network for scene segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3146–3154). Long Beach, USA.
  • Gu, S., Yang, J., & Kong, H. (2021). A cascaded LiDAR-camera fusion network for road detection. In 2021 IEEE international conference on robotics and automation (ICRA) (pp. 13308–13314). Xian, China.
  • Guo, Y., Yao, A., & Chen, Y. (2016). Dynamic network surgery for efficient DNNs. Advances in Neural Information Processing Systems, 29, 1379–1387.
  • Gupta, S., Girshick, R., Arbeláez, P., & Malik, J. (2014). Learning rich features from RGB-D images for object detection and segmentation. In European conference on computer vision (pp. 345–360). Zurich, Switzerlan.
  • Han, S., Mao, H., & Dally, W. J. (2015). Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149.
  • Han, S., Pool, J., Tran, J., & Dally, W. (2015). Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 1135–1143.
  • Han, X., Lu, J., Zhao, C., You, S., & Li, H. (2018). Semisupervised and weakly supervised road detection based on generative adversarial networks. IEEE Signal Processing Letters, 25(4), 551–555. https://doi.org/10.1109/LSP.2018.2809685
  • Hazirbas, C., Ma, L., Domokos, C., & Cremers, D. (2016). FuseNet: Incorporating depth into semantic segmentation via fusion-based cnn architecture. In Asian conference on computer vision (pp. 213–228). Taipei, China.
  • He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770–778). Las Vegas, USA.
  • He, W., Wu, M., Liang, M., & Lam, S. K. (2021). CAP: Context-aware pruning for semantic segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision (pp. 960–969).
  • He, Y., Kang, G., Dong, X., Fu, Y., & Yang, Y. (2018). Soft filter pruning for accelerating deep convolutional neural networks. arXiv preprint arXiv:1808.06866.
  • He, Y., Liu, P., Wang, Z., Hu, Z., & Yang, Y. (2019). Filter pruning via geometric median for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 4340–4349). Long Beach, USA.
  • He, Y., Zhang, X., & Sun, J. (2017). Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference on computer vision (pp. 1389–1397). Venice, Italy.
  • Howard, A. G., Zhu, M., Chen, B., Kalenichenko, D., Wang, W., Weyand, T., Andreetto, M., & Adam, H. (2017). Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861.
  • Huang, G., Liu, Z., Van Der Maaten, L., & Weinberger, K. Q. (2017). Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700–4708). Hawaii, USA.
  • Huang, Q., Zhou, K., You, S., & Neumann, U. (2018). Learning to prune filters in convolutional neural networks. In 2018 IEEE winter conference on applications of computer vision (WACV) (pp. 709–718). Nevada, USA.
  • Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
  • Krishnamoorthi, R. (2018). Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv:1806.08342.
  • Li, H., Kadav, A., Durdanovic, I., Samet, H., & Graf, H. P. (2016). Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710.
  • Li, H., Yue, X., Wang, Z., Chai, Z., Wang, W., Tomiyama, H., & Meng, L. (2022). Optimizing the deep neural networks by layer-wise refined pruning and the acceleration on FPGA. Computational Intelligence and Neuroscience, 2022. https://doi.org/10.1155/2022/8039281
  • Li, X., Zhong, Z., Wu, J., Yang, Y., Lin, Z., & Liu, H. (2019). Expectation–maximization attention networks for semantic segmentation. In Proceedings of the IEEE international conference on computer vision (pp. 9167–9176). Seoul, Korea.
  • Li, Z., Gan, Y., Liang, X., Yu, Y., Cheng, H., & Lin, L. (2016). LSTM-CF: Unifying context modeling and fusion with LSTMs for RGB-D scene labeling. In European conference on computer vision (pp. 541–557). Amsterdam, Netherlands.
  • Lin, D., Fidler, S., & Urtasun, R. (2013). Holistic scene understanding for 3D object detection with RGBD cameras. In Proceedings of the IEEE international conference on computer vision (pp. 1417–1424). Sydney, Australia.
  • Lin, M., Ji, R., Wang, Y., Zhang, Y., Zhang, B., Tian, Y., & Shao, L. (2020). HRank: Filter pruning using high-rank feature map. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 1529–1538). Seattle, USA.
  • Lin, S., Ji, R., Yan, C., Zhang, B., Cao, L., Ye, Q., Huang, F., & Doermann, D. (2019). Towards optimal structured cnn pruning via generative adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 2790–2799). Long Beach, USA.
  • Liu, Z., Li, J., Shen, Z., Huang, G., Yan, S., & Zhang, C. (2017). Learning efficient convolutional networks through network slimming. In Proceedings of the IEEE international conference on computer vision (pp. 2736–2744). Venice, Italy.
  • Long, J., Shelhamer, E., & Darrell, T. (2015). Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431–3440). Boston, USA.
  • Luo, J. H., Wu, J., & Lin, W. (2017). ThiNet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference on computer vision (pp. 5058–5066). Venice, Italy.
  • Lyu, Y., Bai, L., & Huang, X. (2018). Chipnet: Real-time lidar processing for drivable region segmentation on an FPGA. IEEE Transactions on Circuits and Systems I: Regular Papers, 66(5), 1769–1779. https://doi.org/10.1109/TCSI.8919
  • Molchanov, P., Tyree, S., Karras, T., Aila, T., & Kautz, J. (2016). Pruning convolutional neural networks for resource efficient inference. arXiv preprint arXiv:1611.06440.
  • Mukherjee, S., & Guddeti, R. M. R. (2014). A hybrid algorithm for disparity calculation from sparse disparity estimates based on stereo vision. In 2014 international conference on signal processing and communications (SPCOM) (pp. 1–6). Bangalore, India.
  • Park, S. J., Hong, K. S., & Lee, S. (2017). RDFNeT: RGB-D multi-level residual feature fusion for indoor semantic segmentation. In Proceedings of the IEEE international conference on computer vision (pp. 4980–4989). Venice, Italy.
  • Ronneberger, O., Fischer, P., & Brox, T. (2015). U-Net: Convolutional networks for biomedical image segmentation. In International conference on medical image computing and computer-assisted intervention (pp. 234–241). Munich, Germany.
  • Simonyan, K., & Zisserman, A. (2014). Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
  • Suau, X., Zappella, L., Palakkode, V., & Apostoloff, N. (2018). Principal filter analysis for guided network compression. arXiv preprint arXiv:1807.10585 2.
  • Sun, J. Y., Kim, S. W., Lee, S. W., Kim, Y. W., & Ko, S. J. (2019). Reverse and boundary attention network for road segmentation. In Proceedings of the IEEE international conference on computer vision workshops (pp. 0–0). Seoul, Korea.
  • Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., & Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 1–9). Boston, USA.
  • Teichmann, M., Weber, M., Zoellner, M., Cipolla, R., & Urtasun, R. (2018). MultiNet: Real-time joint semantic reasoning for autonomous driving. In 2018 IEEE intelligent vehicles symposium (IV) (pp. 1013–1020). Suzhou, China.
  • Tung, F., & Mori, G. (2018). CLIP-Q: Deep network compression learning by in-parallel pruning-quantization. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7873–7882). Salt Lake, USA.
  • Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). Attention is all you need. In Advances in neural information processing systems (pp. 5998–6008). Long Beach, USA.
  • Wang, C., Bai, X., Wang, X., Liu, X., Zhou, J., Wu, X., Li, H., & Tao, D. (2020). Self-supervised multiscale adversarial regression network for stereo disparity estimation. IEEE Transactions on Cybernetics, 51(10), 4770–4783. https://doi.org/10.1109/TCYB.2020.2999492
  • Wang, C., Wang, X., Zhang, J., Zhang, L., Bai, X., Ning, X., Zhou, J., & Hancock, E. (2022). Uncertainty estimation for stereo matching based on evidential deep learning. Pattern Recognition, 124, Article ID 108498. https://doi.org/10.1016/j.patcog.2021.108498
  • Wang, D., Zhou, L., Zhang, X., Bai, X., & Zhou, J. (2018). Exploring linear relationship in feature map subspace for convnets compression. arXiv preprint arXiv:1803.05729.
  • Wang, H., Fan, R., Sun, Y., & Liu, M. (2021). Dynamic fusion module evolves drivable area and road anomaly detection: A benchmark and algorithms. IEEE Transactions on Cybernetics, 10750–10760. https://doi.org/10.1109/TCYB.2021.3064089
  • Wang, X., Girshick, R., Gupta, A., & He, K. (2018). Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7794–7803). Boston, USA.
  • Xiong, S., Wu, G., Fan, X., Feng, X., Huang, Z., Cao, W., Zhou, X., Ding, S., Yu, J., Wang, L., & Shi, Z. (2021). MRI-based brain tumor segmentation using FPGA-accelerated neural network. BMC Bioinformatics, 22(1), 1–15. https://doi.org/10.1186/s12859-021-04347-6
  • Yamamoto, K., & Maeno, K. (2018). PCAS: Pruning channels with attention statistics for deep network compression. arXiv preprint arXiv:1806.05382.
  • Yan, C., Pang, G., Bai, X., Liu, C., Ning, X., Gu, L., & Zhou, J. (2021). Beyond triplet loss: Person re-identification with fine-grained difference-aware pairwise loss. IEEE Transactions on Multimedia, 24, 1665–1677. https://doi.org/10.1109/TMM.2021.3069562
  • Ye, J., Lu, X., Lin, Z., & Wang, J. Z. (2018). Rethinking the smaller-norm-less-informative assumption in channel pruning of convolution layers. arXiv preprint arXiv:1802.00124.
  • Yu, R., Li, A., Chen, C. F., Lai, J. H., Morariu, V. I., Han, X., Gao, M., Lin, C. Y., & Davis, L. S. (2018). NISP: Pruning networks using neuron importance score propagation. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 9194–9203). Salt Lake, USA.
  • Zhang, J., Yang, T., Li, Q., Zhou, B., Yang, Y., Luo, G., & Shi, J. (2021). An FPGA-based neural network overlay for ADAS supporting multi-model and multi-mode. In 2021 IEEE international symposium on circuits and systems (ISCAS) (pp. 1–5). Daegu, Korea.
  • Zhang, T., Ye, S., Zhang, K., Tang, J., Wen, W., Fardad, M., & Wang, Y. (2018). A systematic DNN weight pruning framework using alternating direction method of multipliers. In Proceedings of the European conference on computer vision (ECCV) (pp. 184–199). Munich, Germany.
  • Zhang, X., Yang, Y., Li, T., Zhang, Y., Wang, H., & Fujita, H. (2021). CMC: A consensus multi-view clustering model for predicting Alzheimer's disease progression. Computer Methods and Programs in Biomedicine, 199, Article ID 105895. https://doi.org/10.1016/j.cmpb.2020.105895
  • Zhou, L., Bai, X., Liu, X., Zhou, J., & Hancock, E. R. (2020). Learning binary code for fast nearest subspace search. Pattern Recognition, 98, Article ID 107040. https://doi.org/10.1016/j.patcog.2019.107040
  • Zhuang, Z., Tan, M., Zhuang, B., Liu, J., Guo, Y., Wu, Q., Huang, J., & Zhu, J. (2018). Discrimination-aware channel pruning for deep neural networks. Advances in Neural Information Processing Systems, 31, 875–886.
  • Zhuo, H., Qian, X., Fu, Y., Yang, H., & Xue, X. (2018). SCSP: Spectral clustering filter pruning with soft self-adaption manners. arXiv preprint arXiv:1806.05320.

Related research

People also read lists articles that other readers of this article have read.

Recommended articles lists articles that we recommend and is powered by our AI driven recommendation engine.

Cited by lists all citing articles based on Crossref citations.
Articles with the Crossref icon will open in a new tab.