Adjustable patch and feature prior token-based transformer for weakly supervised semantic segmentation

Abstract

Weakly supervised semantic segmentation is a challenging task, utilizing only low-cost weak supervision to produce pixel-level predictions. Existing transformer-based methods for weakly supervised semantic segmentation have some limitations: (1) Fixing patch size might destroy the structured semantics, which is unfriendly to objects of different scales, and (2) Ignoring the prior features when using multi-head attention mechanisms might lead to inaccurate segmentation localization. To tackle these issues, we proposed an effective transformer framework coupled with the adjustable patch and prior feature tokens, termed as APFPformer, in which an adjustable patch module is developed to split the image according to the area of the salient object for preserving the structured semantics in patches. A prior feature token module is devised to exploit the edge and texture information as prior tokens, ensuring the gain of discriminative representation. Additionally, a single-stage scheme is applied to reduce the computation and rapid segmentation process. Our experiments demonstrate the superiority of our approach over early methods, gaining competitive mean Intersection-over-Union scores of 67.9% on the PASCAL VOC2012 dataset and 40.2% on the MS COCO dataset.

Keywords:

• Semantic segmentation
• weakly supervised learning
• vision transformer
• adjustable patch
• prior feature token

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The data that support the findings of this study are available from the corresponding author upon reasonable request.

Additional information

Funding

This work was supported by the Youth Project of Fundamental Research Program of Shanxi Province [grant number 202303021212222], the Industry-University-Research Innovation Fund for Chinese Universities [grant number 2021ZYA11005], the Key Research and Development Plan of Shanxi Province [grant numbers 202102020101002 and 202202010101005], and the General Program of Basic Research Plan of Shanxi Province [grant number 201901D111247].

Notes on contributors

Linjuan Li

Linjuan Li is currently working toward the PH.D degree in born in Shanxi Key Laboratory of Advanced Control and Intelligent Information System, School of Electronic and Information Engineering, Taiyuan University of Science and Technology, in China. She received B.S. and M.S. degree from Taiyuan University of Technology. Her research interests including computer vision and deep learning technology.

Haoxue Zhang

Haoxue Zhang is currently working toward the PH.D degree in born in Shanxi Key Laboratory of Advanced Control and Intelligent Information System, School of Electronic and Information Engineering, Taiyuan University of Science and Technology, in China. She received B.S. degree from Beijing University of Chemical Technology, and M.S. degree from Taiyuan University of Science and Technology. Her research interests including computer vision and deep learning technology.

Gang Xie

Gang Xie received the B.S. degree in control theory and the Ph.D. degree in circuits and systems from the Taiyuan University of Technology, China, in 1994 and 2006, respectively. He is currently the Vice President of the Taiyuan University of Science and Technology, China, and has also been a Professor of the Taiyuan University of Technology since 2008. He has authored over 100 papers and held five invention patents. His main research interests cover intelligent information processing, complex networks, and big data.

Yanhong Bai

Yanhong Bai is a Professor of Taiyuan University of Science and Technology. She received the PH.D degree from Nanjing university of science and technology, B.S. degree and M.S. degree from Taiyuan University of Technology. Her research interests including control intelligence algorithm and deep learning technology.