Optimisation of recovery policies in the era of supply chain disruptions: a system dynamics and reinforcement learning approach

References

• Alves, J. C., and G. R. Mateus. 2022. “Multi-Echelon Supply Chains with Uncertain Seasonal Demands and Lead Times Using Deep Reinforcement Learning.” arXiv preprint arXiv:2201.04651.
   Google Scholar
• Aslam, T., and A. H. C. Ng. 2016. “Combining System Dynamics and Multi-Objective Optimization with Design Space Reduction.” Industrial Management & Data Systems 116 (2): 291–321. https://doi.org/10.1108/IMDS-05-2015-0215.
   Web of Science ®Google Scholar
• Bottani, E., and R. Montanari. 2010. “Supply Chain Design and Cost Analysis Through Simulation.” International Journal of Production Research 48 (10): 2859–2886. https://doi.org/10.1080/00207540902960299.
   Web of Science ®Google Scholar
• Campuzano, F., and J. Mula. 2011. Supply Chain Simulation: A System Dynamics Approach for Improving Performance. London, Dordrecht, Heidelberg, New York: Springer Science & Business Media.
   Google Scholar
• Degris, T., P. M. Pilarski, and R. S. Sutton. 2012. “Model-Free Reinforcement Learning with Continuous Action in Practice.” In 2012 American Control Conference (ACC), 2177–2182. IEEE.
   Google Scholar
• Dolgui, A., D. Ivanov, and B. Sokolov. 2018. “Ripple Effect in the Supply Chain: An Analysis and Recent Literature.” International Journal of Production Research 56 (1–2): 414–430. https://doi.org/10.1080/00207543.2017.1387680.
   Web of Science ®Google Scholar
• El-Aal, A., M. A. El-Sharief, A. E. El-Deen, and A.-B. Nassr. 2010. “A Framework for Evaluating and Comparing Inventory Control Policies in Supply Chains.” JES. Journal of Engineering Sciences 38 (2): 449–465. https://doi.org/10.21608/jesaun.2010.124377.
   Google Scholar
• Esteso, A., D. Peidro, J. Mula, and M. Díaz-Madroñero. 2023. “Reinforcement Learning Applied to Production Planning and Control.” International Journal of Production Research 61 (16): 5772–5789. https://doi.org/10.1080/00207543.2022.2104180.
   Web of Science ®Google Scholar
• Gadewadikar, J., and J. Marshall. 2023. “A Methodology for Parameter Estimation in System Dynamics Models Using Artificial Intelligence.” Systems Engineering 27 (2): 253–266.
   Web of Science ®Google Scholar
• Ghadge, A., M. Er, D. Ivanov, and A. Chaudhuri. 2022. “Visualisation of Ripple Effect in Supply Chains Under Long-Term, Simultaneous Disruptions: A System Dynamics Approach.” International Journal of Production Research 60 (20): 6173–6186. https://doi.org/10.1080/00207543.2021.1987547.
   Web of Science ®Google Scholar
• Golan, M. S., L. H. Jernegan, and I. Linkov. 2020. “Trends and Applications of Resilience Analytics in Supply Chain Modeling: Systematic Literature Review in the Context of the Covid-19 Pandemic.” Environment Systems and Decisions 40 (2): 222–243. https://doi.org/10.1007/s10669-020-09777-w.
   PubMedGoogle Scholar
• Gu, Q., and T. Gao. 2017. “Production Disruption Management for R/m Integrated Supply Chain Using System Dynamics Methodology.” International Journal of Sustainable Engineering 10 (1): 44–57. https://doi.org/10.1080/19397038.2016.1250838.
   Web of Science ®Google Scholar
• Hayes, C. F., R. Rădulescu, E. Bargiacchi, J. Källström, M. Macfarlane, M. Reymond, T. Verstraeten, et al. 2022. “A Practical Guide to Multi-Objective Reinforcement Learning and Planning.” Autonomous Agents and Multi-Agent Systems 36 (1): 26. https://doi.org/10.1007/s10458-022-09552-y.
   Web of Science ®Google Scholar
• Heidary, M. H., and A. Aghaie. 2019. “Risk Averse Sourcing in a Stochastic Supply Chain: A Simulation-Optimization Approach.” Computers & Industrial Engineering 130:62–74. https://doi.org/10.1016/j.cie.2019.02.023.
   Web of Science ®Google Scholar
• Hubbs, C. D., H. D. Perez, O. Sarwar, N. V. Sahinidis, I. E. Grossmann, and J. M. Wassick. 2020. “Or-Gym: A Reinforcement Learning Library for Operations Research Problems.” arXiv preprint arXiv:2008.06319.
   Google Scholar
• Ivanov, D. 2017. “Simulation-Based Ripple Effect Modelling in the Supply Chain.” International Journal of Production Research 55 (7): 2083–2101. https://doi.org/10.1080/00207543.2016.1275873.
   Web of Science ®Google Scholar
• Ivanov, D. 2019. “Disruption Tails and Revival Policies: A Simulation Analysis of Supply Chain Design and Production-Ordering Systems in the Recovery and Post-Disruption Periods.” Computers & Industrial Engineering 127:558–570. https://doi.org/10.1016/j.cie.2018.10.043.
   Web of Science ®Google Scholar
• Ivanov, D., A. Dolgui, A. Das, and B. Sokolov. 2019. “Digital Supply Chain Twins: Managing the Ripple Effect, Resilience, and Disruption Risks by Data-Driven Optimization, Simulation, and Visibility.” In Handbook of Ripple Effects in the Supply Chain, 309–332. Cham, Switzerland: Springer.
   Google Scholar
• Ivanov, D., A. Dolgui, B. Sokolov, and M. Ivanova. 2017. “Literature Review on Disruption Recovery in the Supply Chain.” International Journal of Production Research 55 (20): 6158–6174. https://doi.org/10.1080/00207543.2017.1330572.
   Web of Science ®Google Scholar
• Ivanov, D., B. Sokolov, I. Solovyeva, A. Dolgui, and F. Jie. 2016. “Dynamic Recovery Policies for Time-Critical Supply Chains Under Conditions of Ripple Effect.” International Journal of Production Research 54 (23): 7245–7258. https://doi.org/10.1080/00207543.2016.1161253.
   Web of Science ®Google Scholar
• Jaenichen, F.-M., C. J. Liepold, A. Ismail, C. J. Martens, V. Dörrsam, and H. Ehm. 2021. “Simulating and Evaluating Supply Chain Disruptions Along an End-To-End Semiconductor Automotive Supply Chain.” In 2021 Winter Simulation Conference (WSC), 1–12. IEEE.
   Google Scholar
• Jafarnejad, A., M. Momeni, S. H. R. Hajiagha, and M. F. Khorshidi. 2019. “A Dynamic Supply Chain Resilience Model for Medical Equipment's Industry.” Journal of Modelling in Management 14 (3): 816–840.
   Web of Science ®Google Scholar
• Katsaliaki, K., P. Galetsi, and S. Kumar. 2022. “Supply Chain Disruptions and Resilience: A Major Review and Future Research Agenda.” Annals of Operations Research 319: 965–1002. https://doi.org/10.1007/s10479-020-03912-1.
   PubMedGoogle Scholar
• Kegenbekov, Z., and I. Jackson. 2021. “Adaptive Supply Chain: Demand–supply Synchronization Using Deep Reinforcement Learning.” Algorithms 14 (8): 240. https://doi.org/10.3390/a14080240.
   Web of Science ®Google Scholar
• Kurian, D. S., V. M. Pillai, A. Raut, and J. Gautham. 2022. “Deep Reinforcement Learning-Based Ordering Mechanism for Performance Optimization in Multi-Echelon Supply Chains.” Applied Stochastic Models in Business and Industry.
   Web of Science ®Google Scholar
• Li, Y., and C. W. Zobel. 2020. “Exploring Supply Chain Network Resilience in the Presence of the Ripple Effect.” International Journal of Production Economics 228:107693. https://doi.org/10.1016/j.ijpe.2020.107693.
   Web of Science ®Google Scholar
• Liu, M., Z. Liu, F. Chu, F. Zheng, and C. Chu. 2021. “A New Robust Dynamic Bayesian Network Approach for Disruption Risk Assessment Under the Supply Chain Ripple Effect.” International Journal of Production Research 59 (1): 265–285. https://doi.org/10.1080/00207543.2020.1841318.
   Web of Science ®Google Scholar
• Liu, M., H. Tang, F. Chu, F. Zheng, and C. Chu. 2022. “A Reinforcement Learning Variable Neighborhood Search for the Robust Dynamic Bayesian Network Optimization Problem Under the Supply Chain Ripple Effect.” IFAC-PapersOnLine 55 (10): 1459–1464. https://doi.org/10.1016/j.ifacol.2022.09.596.
   Google Scholar
• Llaguno, A., J. Mula, and F. Campuzano-Bolarin. 2022. “State of the Art, Conceptual Framework and Simulation Analysis of the Ripple Effect on Supply Chains.” International Journal of Production Research 60 (6): 2044–2066. https://doi.org/10.1080/00207543.2021.1877842.
   Web of Science ®Google Scholar
• Martin-Martinez, E., R. Samsó, J. Houghton, and J. Solé Ollé. 2022. “Pysd: System Dynamics Modeling in Python.” The Journal of Open Source Software 7 (78): 4329. https://doi.org/10.21105/joss.
   Google Scholar
• Mnih, V., A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. 2016. “Asynchronous Methods for Deep Reinforcement Learning.” In International Conference on Machine Learning, 1928–1937. PMLR.
   Google Scholar
• Moerland, T. M., J. Broekens, A. Plaat, and C. M. Jonker. 2023. “Model-Based Reinforcement Learning: A Survey.” Foundations and Trends$\circledR$ in Machine Learning 16 (1): 1–118. https://doi.org/10.1561/2200000086.
   Web of Science ®Google Scholar
• Mortazavi, A., A. A. Khamseh, and P. Azimi. 2015. “Designing of An Intelligent Self-Adaptive Model for Supply Chain Ordering Management System.” Engineering Applications of Artificial Intelligence 37:207–220. https://doi.org/10.1016/j.engappai.2014.09.004.
   Web of Science ®Google Scholar
• Olivares-Aguila, J., and W. ElMaraghy. 2021. “System Dynamics Modelling for Supply Chain Disruptions.” International Journal of Production Research 59 (6): 1757–1775. https://doi.org/10.1080/00207543.2020.1725171.
   Web of Science ®Google Scholar
• Oroojlooyjadid, A., M. Nazari, L. V. Snyder, and M. Takáč. 2022. “A Deep Q-network for the Beer Game: Deep Reinforcement Learning for Inventory Optimization.” Manufacturing & Service Operations Management 24 (1): 285–304. https://doi.org/10.1287/msom.2020.0939.
   Web of Science ®Google Scholar
• Perez, H. D., C. D. Hubbs, C. Li, and I. E. Grossmann. 2021. “Algorithmic Approaches to Inventory Management Optimization.” Processes 9 (1): 102. https://doi.org/10.3390/pr9010102.
   Web of Science ®Google Scholar
• Raffin, A., A. Hill, A. Gleave, A. Kanervisto, M. Ernestus, and N. Dormann. 2021. “Stable-Baselines3: Reliable Reinforcement Learning Implementations.” The Journal of Machine Learning Research 22 (1): 12348–12355.
   Google Scholar
• Rahmandad, H., and S. Fallah-Fini. 2008. “Learning Control Policies in System Dynamics Models.” Systemdynamics.Org.
   Google Scholar
• Rolf, B., I. Jackson, M. Müller, S. Lang, T. Reggelin, and D. Ivanov. 2023. “A Review on Reinforcement Learning Algorithms and Applications in Supply Chain Management.” International Journal of Production Research 61 (20): 7151–7179. https://doi.org/10.1080/00207543.2022.2140221.
   Web of Science ®Google Scholar
• Schmitt, T. G., S. Kumar, K. E. Stecke, F. W. Glover, and M. A. Ehlen. 2017. “Mitigating Disruptions in a Multi-Echelon Supply Chain Using Adaptive Ordering.” Omega 68:185–198. https://doi.org/10.1016/j.omega.2016.07.004.
   Web of Science ®Google Scholar
• Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. “Proximal Policy Optimization Algorithms.” arXiv preprint arXiv:1707.06347.
   Google Scholar
• Sinha, D., V. Bagodi, and D. Dey. 2020. “The Supply Chain Disruption Framework Post Covid-19: A System Dynamics Model.” Foreign Trade Review 55 (4): 511–534. https://doi.org/10.1177/0015732520947904.
   Web of Science ®Google Scholar
• Sutton, R. S., and A. G. Barto. 2018. Reinforcement Learning: An Introduction. Cambridge, MA: MIT press.
   Google Scholar
• Sutton, R. S., D. McAllester, S. Singh, and Y. Mansour. 1999. “Policy Gradient Methods for Reinforcement Learning with Function Approximation.” Advances in Neural Information Processing Systems 12: 1057–1063.
   Google Scholar
• Timme, S. G., and C. Williams-Timme. 2003. “The Real Cost of Holding Inventory.” Supply Chain Management Review 7 (4): 30–37. (July/August 2003), ILL.
   Google Scholar
• Tordecilla, R. D., A. A. Juan, J. R. Montoya-Torres, C. L. Quintero-Araujo, and J. Panadero. 2021. “Simulation-Optimization Methods for Designing and Assessing Resilient Supply Chain Networks Under Uncertainty Scenarios: A Review.” Simulation Modelling Practice and Theory 106:102166. https://doi.org/10.1016/j.simpat.2020.102166.
   PubMed Web of Science ®Google Scholar
• Wang, T., X. Bao, I. Clavera, J. Hoang, Y. Wen, E. Langlois, S. Zhang, et al. 2019. “Benchmarking Model-Based Reinforcement Learning.” arXiv preprint arXiv:1907.02057.
   Google Scholar
• Wang, H., J. Tao, T. Peng, A. Brintrup, E. E. Kosasih, Y. Lu, R. Tang, and L. Hu. 2022. “Dynamic Inventory Replenishment Strategy for Aerospace Manufacturing Supply Chain: Combining Reinforcement Learning and Multi-Agent Simulation.” International Journal of Production Research 60 (13): 4117–4136. https://doi.org/10.1080/00207543.2021.2020927.
   Web of Science ®Google Scholar
• Wang, X., S. Wang, X. Liang, D. Zhao, J. Huang, X. Xu, B. Dai, and Q. Miao. 2022. “Deep Reinforcement Learning: A Survey.” IEEE Transactions on Neural Networks and Learning Systems 35 (4): 5064–5078. https://doi.org/10.1109/TNNLS.2022.3207346.
   Web of Science ®Google Scholar
• Yan, Y., A. H. Chow, C. P. Ho, Y.-H. Kuo, Q. Wu, and C. Ying. 2022. “Reinforcement Learning for Logistics and Supply Chain Management: Methodologies, State of the Art, and Future Opportunities.” Transportation Research Part E: Logistics and Transportation Review 162:102712. https://doi.org/10.1016/j.tre.2022.102712.
   Web of Science ®Google Scholar
• Zhang, H., and T. Yu. 2020. “Taxonomy of Reinforcement Learning Algorithms.” In Deep Reinforcement Learning: Fundamentals, Research and Applications, 125–133. Singapore.
   Google Scholar
• Zhou, J., and X. Zhou. 2019. “Multi-Echelon Inventory Optimizations for Divergent Networks by Combining Deep Reinforcement Learning and Heuristics Improvement.” In 2019 12th International Symposium on Computational Intelligence and Design (ISCID), Vol. 1, 69–73. IEEE.
   Google Scholar
• Zhu, Q., H. Krikke, and M. C. Caniëls. 2021. “The Effects of Different Supply Chain Integration Strategies on Disruption Recovery: A System Dynamics Study on the Cheese Industry.” Logistics 5 (2): 19. https://doi.org/10.3390/logistics5020019.
   Google Scholar