Reinforcement learning for process control with application in semiconductor manufacturing

Abstract

Process control is widely discussed in the manufacturing process, especially in semiconductor manufacturing. Due to unavoidable disturbances in manufacturing, different process controllers are proposed to realize variation reduction. Since Reinforcement Learning (RL) has shown great advantages in learning actions from interactions with a dynamic system, we introduce RL methods for process control and propose a new controller called RL-based controller. Considering the fact that most existing run-to-run (R2R) controllers mainly rely on a linear model assumption for the process input–output relationship, we first discuss theoretical properties of RL-based controllers based on the linear model assumption. Then the performance of RL-based controllers and traditional R2R controllers (e.g., Exponentially Weighted Moving Average (EWMA), double EWMA, adaptive EWMA, and general harmonic rule controllers) are compared for linear processes. Furthermore, we find that the RL-based controllers have potential advantages to deal with other complicated nonlinear processes. The intensive numerical studies validate the advantages of the proposed RL-based controllers.

Keywords:

• Process control
• reinforcement learning
• semiconductor manufacturing

Additional information

Funding

This work was supported by Guangzhou Municipal Science and Technology Program under grant No. 202201011235, Guangdong Basic and Applied Basic Research Foundation under grant No. 2023A1515011656, Foshan HKUST Projects under grant No. FSUST20-FYTRI03B, and the National Natural Science Foundation of China under Grants 71831006, 72001139 and 72122013.

Notes on contributors

Yanrong Li

Yanrong Li is a PhD candidate in management science and engineering at Antai College of Economics and Management, Shanghai Jiao Tong University. She received BE and ME degrees from Tianjin University in 2015 and 2018, respectively. Her research interests include data analytics for process control and operational optimization in manufacturing systems.

Juan Du

Juan Du is currently an Assistant Professor with the Smart Manufacturing Thrust, Systems Hub, The Hong Kong University of Science and Technology (Guangzhou), China. She is also affiliated with the Department of Mechanical and Aerospace Engineering, The Hong Kong University of Science and Technology, Hong Kong SAR, China, and Guangzhou HKUST Fok Ying Tung Research Institute, Guangzhou, China. Her current research interests include data analytics and machine learning for modeling, monitoring, control, diagnosis and optimization in smart manufacturing systems. Her research has received 7 best paper awards and two outstanding doctoral thesis awards.

Wei Jiang

Wei Jiang is a distinguished professor of management science at Antai College of Economics and Management, Shanghai Jiao Tong University. Prior to joining Shanghai Jiao Tong University, he worked in AT&T Labs, Stevens Institute of Technology, and Hong Kong University of Science and Technology. His research interests include big data analytics and innovation, Industry 4.0, and operations management, etc. He received the NSF CAREER award in 2006 and NSFC National Funds for Distinguished Young Scientists award in 2013.