Adaptive scheduling for assembly job shop with uncertain assembly times based on dual Q-learning

Abstract

To address the uncertainty of production environment in assembly job shop, in combination of the real-time feature of reinforcement learning, a dual Q-learning (D-Q) method is proposed to enhance the adaptability to environmental changes by self-learning for assembly job shop scheduling problem. On the basis of the objective function of minimising the total weighted earliness penalty and completion time cost, the top level Q-learning is focused on localised targets in order to find the dispatching policy which can minimise machine idleness and balance machine loads, and the bottom level Q-learning is focused on global targets in order to learn the optimal scheduling policy which can minimise the overall earliness of all jobs. Some theoretical results and simulation experiments indicate that the proposed algorithm achieves generally better results than the single Q-learning (S-Q) and other scheduling rules, under the arrival frequency of product with different conditions, and show good adaptive performance.

Abbreviations: AFSSP, assembly flow shop scheduling problem; AJSSP, assembly job shop scheduling problem; RL, reinforcement learning; TASP, two-stage assembly scheduling problem

KEYWORDS:

• Adaptive scheduling
• assembly job shop
• Q-learning
• Reinforcement learning
• uncertain assembly

Acknowledgments

The authors would like to thank the Editor, Associate Editor and all anonymous reviewers for their thoughtful comments and suggestionsthat greatly helped improve the presentation and technical quality of this paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research is supported by the National Natural Science Foundation of China [grant number #71401076], The Ministry of education of Humanities and Social Science project of China [grant number 19YJCZH149], and the Fundamental Research Funds for the Central Universities of Nanjing Agricultural University [grant number #SKYZ2019010].

Notes on contributors

Haoxiang Wang

Haoxiang Wang was born in 1979 in Jiangxi Province, China. He got his B.S. degree in Information and Computational Science from Southeast University in 2004. He received his M.S. degree from Jiangsu University on Systems Engineering in 2009. He got his Ph.D. degree from Southeast University by a thesis on Adaptive Scheduling Based On Extended Q-learning for Knowledgeable Manufacturing System in 2014. He has been in the Department of Management Engineering, College of Engineering, Nanjing Agricultural University as a teacher for six years. He is interested in Production Planning & Control, Scheduling, Logistics and Distribution Systems, etc.

Bhaba R. Sarker

Bhaba R. Sarker is the Elton G. Yates Distinguished Professor of Engineering at the Louisiana State University. Before joining LSU, he taught at UT-Austin and Texas A&M University. Professor Sarker published more than 170 papers in refereed journals and more than 90 papers in conference proceedings. He won the 2006 David F. Baker Distinguished Research Award from IISE for outstanding research contributions in Industrial Engineering. He has served on the editorial boards of more than 10 journals and is currently on the editorial boards of six journals including International Journal of Production Economics and Production Planning & Control. He is a Fellow in Institute of Industrial & Systems Engineers, USA and South Asia Institute of Science and Engineering. He is also a member of DSI, INFORMS, POMS, and New York Academy of Sciences. He is currently working in the area of optimization, supply chain logistics, lean production systems and renewable energy.

Jing Li

Jing Li, Ph.D. Professor of Management Science and Engineering, Dean of the Department of Management Engineering, College of Engineering, Nanjing Agricultural University. He is interested in System modeling and simulation, Job shop planning, Agricultural system engineering, etc.

Jian Li

Jian Li, associate professor in college of engineering, Nanjing agricultural university. He received his PhD degree in management science and engineering from Southeast University in 2009. His research interests include logistics, vehicle routing problem, and optimization. He has teaching experience of more than 10 years. Moreover, he has published more than 20 papers in various journals and completed ten scientific research projects.