Comprehensive survey on the effectiveness of sharpness aware minimization and its progressive variants

ABSTRACT

As advancements push for larger and more complex Artificial Intelligence (AI) models to improve performance, preventing the occurrence of overfitting when training overparameterized Deep Neural Networks (DNNs) remains a challenge. Despite the presence of various regularization techniques aimed at mitigating this issue, poor generalization remains a concern, especially when handling diverse and limited data. This paper explores one of the latest and most promising strategies to address this challenge, Sharpness Aware Minimization (SAM), which concurrently minimizes loss value and sharpness-related loss. While this method exhibits substantial effectiveness, it comes with a notable trade-off in increased training time and is founded on several approximations. Consequently, several variants of SAM have emerged to alleviate these limitations and bolster model performance. This survey paper examines the significant advancements achieved by SAM, delves into its constraints, and categorizes recent progressive variants that further enhance current State-of-the-Art results.

CO EDITOR-IN-CHIEF :

• Su, Shun-Feng

ASSOCIATE EDITOR :

• Su, Shun-Feng

KEYWORDS:

• Overfitting
• generalization
• robustness
• loss landscape
• weight perturbation
• sharpness

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was financially supported by the “Chinese Language and Technology Center” of National Taiwan Normal University (NTNU) from The Featured Areas Research Center Program within the framework of the Higher Education Sprout Project by the Ministry of Education (MOE) in Taiwan, and National Science and Technology Council, Taiwan, under Grants no. NSTC 112-2221-E-003-007, NSTC 112-2221-E-003-008, and NSTC 112-2221-E-003 -010. Furthermore, the Ministry of Education of Taiwan also contributed to fund this project via the Taiwan Scholarship Program.