Performance of deep reinforcement learning algorithms in two-echelon inventory control systems

Abstract

This study conducts a comprehensive analysis of deep reinforcement learning (DRL) algorithms applied to supply chain inventory management (SCIM), which consists of a sequential decision-making problem focussed on determining the optimal quantity of products to produce and ship across multiple capacitated local warehouses over a specific time horizon. In detail, we formulate the problem as a Markov decision process for a divergent two-echelon inventory control system characterised by stochastic and seasonal demand, also presenting a balanced allocation rule designed to prevent backorders in the first echelon. Through numerical experiments, we evaluate the performance of state-of-the-art DRL algorithms and static inventory policies in terms of both cost minimisation and training time while varying the number of local warehouses and product types and the length of replenishment lead times. Our results reveal that the Proximal Policy Optimization algorithm consistently outperforms other algorithms across all experiments, proving to be a robust method for tackling the SCIM problem. Furthermore, the (s, Q)-policy stands as a solid alternative, offering a compromise in performance and computational efficiency. Lastly, this study presents an open-source software library that provides a customisable simulation environment for addressing the SCIM problem, utilising a wide range of DRL algorithms and static inventory policies.

Keywords:

• inventory management
• inventory control systems
• inventory control policies
• artificial intelligence
• deep learning
• reinforcement learning

Disclosure statement

No potential conflict of interest was reported by the author(s).

Data availability statement

The datasets generated and analysed during the current study can be generated using the open-source software library developed for this research, which can be accessed at https://github.com/frenkowski/SCIMAI-Gym.

Notes

1 Our open-source library is available at https://github.com/frenkowski/SCIMAI-Gym.

Additional information

Notes on contributors

Francesco Stranieri

Francesco Stranieri is a Ph.D. student enrolled in the Italian National Ph.D. programme in Artificial Intelligence (Industry 4.0 area) at the Polytechnic of Turin and the University of Milano-Bicocca. He obtained his Bachelor's and Master's degrees in Computer Science from the University of Milano-Bicocca in 2019 and 2021, respectively. He won the 2nd prize in the ‘Green Technologies and Applications’ line in the Best Student Research ‘Video & Poster Competition’ at the IEEE EUROCON 2023 Conference. In 2023, he completed a six-month internship at Bristol Myers Squibb in Switzerland, focussing on deep reinforcement learning and digital twin for supply chain operations and risk assessment. He is an active member of several associations, including the Italian Association of Operations Research (AIRO), the Italian Association for Artificial Intelligence (AIxIA), and IEEE Eta Kappa Nu (IEEE-HKN).

Fabio Stella

Fabio Stella, Ph.D. in Computational Mathematics and Operations Research, is an Associate Professor of Operations Research at the Department of Informatics, Systems, and Communication of the University of Milano-Bicocca. His main research interests are artificial intelligence (AI) and machine learning (ML). Fabio has published more than 100 papers and served as Programme Chair/Reviewer for top international conferences on AI and ML, including AISTATS, ECAI, ICLR, ICML, IJCAI, NeurIPS, PAKDD, PGM, RecSys, SIGIR, and UAI. He has been awarded as a top 10% reviewer at NeurIPS in 2020, 2022, and 2023, ICML and AISTATS in 2022, and IJCAI in 2023. He has been Associate Editor of IEEE Intelligent Systems since 2021 and is currently leading two international research projects funded through a competitive process by the European Commission.

Chaaben Kouki

Chaaben Kouki is a Professor of Operations Management at the ESSCA School of Management in Angers, France. He received his Ph.D. and Master's degrees in Industrial Engineering from Ecole Centrale Paris in 2010 and 2007, respectively, and his Bachelor's degree in Industrial Engineering from Ecole Nationale d'Ingénieurs de Tunis – ENIT in 2005. Before joining ESSCA School of Management, he was a faculty member at Rennes School of Business from 2012 to 2015 and an inventory manager at Michelin Group from 2010 to 2012. His primary research focus is on inventory management systems.