Learning to schedule (L2S): adaptive job shop scheduling using double deep Q network

References

• Anand Deva Durai C, Azath M, Jeniffer J. Integrated search method for flexible job shop scheduling problem using hhs–alns algorithm. SN Comput Sci. 2020;1(2):1–6.
   Google Scholar
• Wu X, Peng J, Xiao X, et al. An effective approach for the dual-resource flexible job shop scheduling problem considering loading and unloading. J Intell Manuf. 2021;32(3):707–728. DOI:10.1007/s10845-020-01697-5
   Web of Science ®Google Scholar
• Kubiak W, Feng Y, Li G, et al. Efficient algorithms for flexible job shop scheduling with parallel machines. Naval Res Logist (NRL). 2020;67(4):272–288. DOI:10.1002/nav.21901
   Web of Science ®Google Scholar
• Zhang M, Tao F, Nee A. Digital twin enhanced dynamic job-shop scheduling. J Manuf Syst. 2021;58:146–156.
   Web of Science ®Google Scholar
• Zhang H, Zhang G, Yan Q. Digital twin-driven cyber-physical production system towards smart shop-floor. J Ambient Intell Humaniz Comput. 2019;10(11):4439–4453.
   Web of Science ®Google Scholar
• Hagras T, Janeček J. Static vs. dynamic list-scheduling performance comparison. Acta Polytechnica. 2003;43(6). DOI:10.14311/490
   Google Scholar
• Kopetz H. Real-time scheduling. Real-time systems: The International Series in Engineering and Computer Science. Boston, MA: Springer. 2002;395:227–243. DOI:10.1007/0-306-47055-1_11
   Google Scholar
• Zhang C, Zhou Y, Peng K, et al. Dynamic flexible job shop scheduling method based on improved gene expression programming. Meas Control. 2021;54(7–8):1136–1146. DOI:10.1177/0020294020946352
   Web of Science ®Google Scholar
• Liu Z, Chen W, Zhang C, et al. Intelligent scheduling of a feature-process-machine tool supernetwork based on digital twin workshop. J Manuf Syst. 2021;58:157–167.
   Web of Science ®Google Scholar
• Boufellouh R, Belkaid F. Bi-objective optimization algorithms for joint production and maintenance scheduling under a global resource constraint: application to the permutation flow shop problem. Comput Oper Res. 2020;122:104943.
   Web of Science ®Google Scholar
• Branda A, Castellano D, Guizzi G, et al. Metaheuristics for the flow shop scheduling problem with maintenance activities integrated. Comput Ind Eng. 2021;151:106989.
   Web of Science ®Google Scholar
• Li L, Zhen HL, Yuan M, et al. Bilevel learning for large-scale flexible flow shop scheduling. Comput. Ind. Eng. 2022;168:108140 .
   Web of Science ®Google Scholar
• Du Y, Wang T, Xin B, et al. A data-driven parallel scheduling approach for multiple agile earth observation satellites. IEEE Trans Evol Comput. 2019;24(4):679–693. DOI:10.1109/TEVC.2019.2934148
   Web of Science ®Google Scholar
• Ikonen TJ, Heljanko K, Harjunkoski I. Reinforcement learning of adaptive online rescheduling timing and computing time allocation. Comput Chem Eng. 2020;141:106994.
   Web of Science ®Google Scholar
• Lu H, Zhang X, Yang S A learning-based iterative method for solving vehicle routing problems. In: International conference on learning representations; Addis Ababa, Ethiopia; 2020.
   Google Scholar
• Liu CL, Chang CC, Tseng CJ. Actor-critic deep reinforcement learning for solving job shop scheduling problems. IEEE Access. 2020;8:71752–71762.
   Web of Science ®Google Scholar
• Kuhnle A, Schäfer L, Stricker N, et al. Design, implementation and evaluation of reinforcement learning for an adaptive order dispatching in job shop manufacturing systems. Procedia CIRP. 2019;81:234–239.
   Google Scholar
• Liu Y, Zhang L, Wang L, et al. A framework for scheduling in cloud manufacturing with deep reinforcement learning. In: 2019 IEEE 17th International Conference on Industrial Informatics (INDIN); Helsinki, Finland; Vol. 1; IEEE; 2019. p. 1775–1780.
   Google Scholar
• Han BA, Yang JJ. Research on adaptive job shop scheduling problems based on dueling double dqn. IEEE Access. 2020;8:186474–186495.
   Web of Science ®Google Scholar
• Xiao Z, Ma S, Zhang S Learning task allocation for multiple flows in multi-agent systems. In: 2009 International Conference on Communication Software and Networks; Chengdu Sichuan, China; IEEE; 2009. p. 153–157.
   Google Scholar
• Sutton RS, Barto AG. Introduction to reinforcement learning. IEEE Trans Neural Networks. 1998;9(5):1054. DOI:10.1109/TNN.1998.712192
   Google Scholar
• Williams RJ. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach Learn. 1992;8(3):229–256.
   Web of Science ®Google Scholar
• Wang Z, Bapst V, Heess N, et al. Sample efficient actor-critic with experience replay. arXiv Preprint. 2016;arXiv:1611.01224.
   Google Scholar
• Nachum O, Norouzi M, Xu K, et al. Bridging the gap between value and policy based reinforcement learning. Adv Neural Inf Process Syst. 2017;30:2772–2782.
   Google Scholar
• Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms. arXiv Preprint. 2017;arXiv:1707.06347.
   Google Scholar
• Mnih V, Kavukcuoglu K, Silver D, et al. Playing atari with deep reinforcement learning. arXiv Preprint. 2013;arXiv:1312.5602.
   Google Scholar
• Watkins CJ, Dayan PQL. Machine learning. Mach Learn. 1992;8(3):279–292.
   Web of Science ®Google Scholar
• Kober J, Bagnell JA, Peters J. Reinforcement learning in robotics: a survey. Int J Rob Res. 2013;32(11):1238–1274.
   Web of Science ®Google Scholar
• Heess N, Wayne G, Silver D, et al. Learning continuous control policies by stochastic value gradients. Adv Neural Inf Process Syst. 2015;28:2944–2952.
   Google Scholar
• Lillicrap TP, Hunt JJ, Pritzel A, et al. Continuous control with deep reinforcement learning. arXiv Preprint. 2015;arXiv:1509.02971.
   Google Scholar
• Mnih V, Badia AP, Mirza M, et al. Asynchronous methods for deep reinforcement learning. In: International conference on machine learning; New York, NY, USA. PMLR; 2016. p. 1928–1937.
   Google Scholar
• Schulman J, Levine S, Abbeel P, et al. Trust region policy optimization. In: International conference on machine learning; Lille, France. PMLR; 2015. p. 1889–1897.
   Google Scholar
• Silver D, Lever G, Heess N, et al. Deterministic policy gradient algorithms. In: International conference on machine learning; Beijing, China. PMLR; 2014. p. 387–395.
   Google Scholar
• Silver D, Huang A, Maddison CJ, et al. Mastering the game of go with deep neural networks and tree search. Nature. 2016;529(7587):484–489. DOI:10.1038/nature16961
   PubMed Web of Science ®Google Scholar
• Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor. In: International conference on machine learning; Stockholm, Sweden. PMLR; 2018. p. 1861–1870.
   Google Scholar
• Sutton RS, Barto AG. Reinforcement learning: an introduction. Cambridge, MA: MIT press; 2018.
   Google Scholar
• Abed-Alguni BH, Ottom MA. Double delayed q-learning. Int J Artif Intell. 2018;16(2):41–59.
   Google Scholar
• Van Hasselt H, Guez A, Silver D Deep reinforcement learning with double q-learning. In: Proceedings of the AAAI conference on artificial intelligence; Arizona, USA; Vol. 30; 2016.
   Google Scholar
• Sewak M. Deep q network (dqn), double dqn, and dueling dqn. In: Deep reinforcement learning. Singapore: Springer; 2019. pp. 95–108. DOI:10.1007/978-981-13-8285-7_8.
   Google Scholar
• Chen R, Yang B, Li S, et al. A self-learning genetic algorithm based on reinforcement learning for flexible job-shop scheduling problem. Comput Ind Eng. 2020;149:106778.
   Web of Science ®Google Scholar
• Wang L, Hu X, Wang Y, et al. Dynamic job-shop scheduling in smart manufacturing using deep reinforcement learning. Comput Netw. 2021;190:107969.
   Web of Science ®Google Scholar
• Wang Z, Zhang J, Yang S. An improved particle swarm optimization algorithm for dynamic job shop scheduling problems with random job arrivals. Swarm Evol Comput. 2019;51:100594.
   Web of Science ®Google Scholar
• Cao Z, Zhou L, Hu B, et al. An adaptive scheduling algorithm for dynamic jobs for dealing with the flexible job shop scheduling problem. Bus Inf Syst Eng. 2019;61(3):299–309. DOI:10.1007/s12599-019-00590-7
   Web of Science ®Google Scholar
• Abed-Alguni BH, Paul D. Island-based cuckoo search with elite opposition-based learning and multiple mutation methods for solving optimization problems. Soft Comput. 2022;26(7):3293–3312.
   Web of Science ®Google Scholar
• Alawad NA, Abed-Alguni BH. Retraction note to: accurate computation: COVID-19 rRT-PCR positive test dataset using stages classification through textual big data mining with machine learning. J Supercomput. 2022;79(6):1–22.
   Web of Science ®Google Scholar
• Wang J, Liu Y, Ren S, et al. Evolutionary game based real-time scheduling for energy-efficient distributed and flexible job shop. J Clean Prod. 2021;293:126093.
   Web of Science ®Google Scholar
• Nie L, Wang X, Pan F. A game-theory approach based on genetic algorithm for flexible job shop scheduling problem. In: Journal of Physics: Conference Series; Xi'an, China. Vol. 1187; IOP Publishing; 2019. p. 032095.
   Google Scholar
• Renna P. Decision-making method of reconfigurable manufacturing systems’ reconfiguration by a gale-shapley model. J Manuf Syst. 2017;45:149–158.
   Web of Science ®Google Scholar
• Hu J, Wellman MP. Multiagent reinforcement learning: theoretical framework and an algorithm. ICML. 1998;98:242–250.
   Google Scholar
• Littman ML. Markov games as a framework for multi-agent reinforcement learning. In: Cohen WW, Hirsh H, editors. Proceedings of the Eleventh International Conference on International Conference on Machine Learning; ICML'94. San Francisco, CA, USA: Morgan Kaufmann Publishers Inc; 1994. pp. 157–163.
   Google Scholar
• Hu J, Wellman MP. Nash q-learning for general-sum stochastic games. J Mach Learn Res. 2003;4(Nov):1039–1069.
   Google Scholar
• Heinrich J, Lanctot M, Silver D Fictitious self-play in extensive-form games. In: Proceedings of the 32nd International conference on machine learning; Lille, France; 2015. p. 805–813.
   Google Scholar
• Monaci M, Agasucci V, Grani G. An actor-critic algorithm with deep double recurrent agents to solve the job shop scheduling problem. arXiv Preprint. 2021;arXiv:2110.09076.
   Google Scholar
• Tian W, Zhang H. A dynamic job-shop scheduling model based on deep learning. Adv Prod Eng Manage. 2021;16(1):23–36.
   Web of Science ®Google Scholar
• Ali KB, Telmoudi AJ, Gattoufi S. Improved genetic algorithm approach based on new virtual crossover operators for dynamic job shop scheduling. IEEE Access. 2020;8:213318–213329.
   Google Scholar
• Kardos C, Laflamme C, Gallina V, et al. Dynamic scheduling in a job-shop production system with reinforcement learning. Procedia CIRP. 2021;97:104–109.
   Google Scholar
• Park J, Chun J, Kim SH, et al. Learning to schedule job-shop problems: representation and policy learning using graph neural network and reinforcement learning. Int J P Res. 2021;59(11):3360–3377. DOI:10.1080/00207543.2020.1870013
   Web of Science ®Google Scholar
• Luo S, Zhang L, Fan Y. Dynamic multi-objective scheduling for flexible job shop by deep reinforcement learning. Comput Ind Eng. 2021;159:107489.
   Web of Science ®Google Scholar
• Yang S, Xu Z. Intelligent scheduling and reconfiguration via deep reinforcement learning in smart manufacturing. Int J P Res. 2021;60:1–18. DOI:10.1080/00207543.2021.1943037.
   Web of Science ®Google Scholar
• Altenmüller T, Stüker T, Waschneck B, et al. Reinforcement learning for an intelligent and autonomous production control of complex job-shops under time constraints. Prod Eng. 2020;14(3):319–328. DOI:10.1007/s11740-020-00967-8
   Google Scholar
• Wang H, Sarker BR, Li J, et al. Adaptive scheduling for assembly job shop with uncertain assembly times based on dual q-learning. Int J P Res. 2021;59(19):5867–5883. DOI:10.1080/00207543.2020.1794075
   Web of Science ®Google Scholar
• Luo S. Dynamic scheduling for flexible job shop with new job insertions by deep reinforcement learning. Appl Soft Comput. 2020;91:106208.
   Web of Science ®Google Scholar
• Zhao Y, Zhang H. Application of machine learning and rule scheduling in a job-shop production control system. Int J Simul Model. 2021;20(2):410–421.
   Web of Science ®Google Scholar