A policy-based Monte Carlo tree search method for container pre-marshalling

References

• Arslan, B., and B. Y. Ekren. 2023. “Transaction Selection Policy in Tier-to-Tier SBSRS by Using Deep Q-Learning.” International Journal of Production Research 61 (21): 7353–7366.https://doi.org/10.1080/00207543.2022.2148767.
   Web of Science ®Google Scholar
• Bai, F., X. Ju, S. Wang, W. Zhou, and F. Liu. 2022. “Wind Farm Layout Optimization Using Adaptive Evolutionary Algorithm with Monte Carlo Tree Search Reinforcement Learning.” Energy Conversion and Management 252: 115047. https://doi.org/10.1016/j.enconman.2021.115047
   Web of Science ®Google Scholar
• Bortfeldt, A., and F. Florian. 2012. “A Tree Search Procedure for the Container Pre-Marshalling Problem.” European Journal of Operational Research 217 (3): 531–540. https://doi.org/10.1016/j.ejor.2011.10.005
   Web of Science ®Google Scholar
• Boschma, R., M. R. Mes, and L. R. de Vries. 2023. “Approximate Dynamic Programming for Container Stacking.” European Journal of Operational Research 310 (1): 328–342. https://doi.org/10.1016/j.ejor.2023.02.034
   Web of Science ®Google Scholar
• Caserta, M., S. Schwarze, and S. Voß. 2012. “A Mathematical Formulation and Complexity Considerations for the Blocks Relocation Problem.” European Journal of Operational Research 219 (1): 96–104. https://doi.org/10.1016/j.ejor.2011.12.039
   Web of Science ®Google Scholar
• Chen, S., S. Fang, and R. Tang. 2019. “A Reinforcement Learning Based Approach for Multi-Projects Scheduling in Cloud Manufacturing.” International Journal of Production Research 57: 3080–3098. https://doi.org/10.1080/00207543.2018.1535205
   Web of Science ®Google Scholar
• Coulom, R. 2006. “Efficient Selectivity and Backup Operators in Monte-Carlo Tree Search.” In International Conference on Computers and Games, edited by H. J. Herik, P. Ciancarini, and H. H. L. M. Donkers, 72–83. Berlin: Springer.
   Google Scholar
• Crippa, M., P. L. Lanzi, and F. Marocchi. 2022. “An Analysis of Single-Player Monte Carlo Tree Search Performance in Sokoban.” Expert Systems with Applications 192: 116224. https://doi.org/10.1016/j.eswa.2021.116224
   Web of Science ®Google Scholar
• Expósito-Izquierdo, C., B. Melián-Batista, and M. Moreno-Vega. 2012. “Pre-Marshalling Problem: Heuristic Solution Method and Instances Generator.” Expert Systems with Applications 39: 8337–8349. https://doi.org/10.1016/j.eswa.2012.01.187
   Web of Science ®Google Scholar
• Feng, Y., D. P. Song, D. Li, and Q. Zeng. 2020. “The Stochastic Container Relocation Problem with Flexible Service Policies.” Transportation Research Part B: Methodological 141: 116–163. https://doi.org/10.1016/j.trb.2020.09.006
   Web of Science ®Google Scholar
• Forster, F., and A. Bortfeldt. 2012. “A Tree Search Procedure for the Container Relocation Problem.” Computers & Operations Research 39: 299–309. https://doi.org/10.1016/j.cor.2011.04.004
   Web of Science ®Google Scholar
• Ge, P., Y. Meng, J. Liu, L. Tang, and R. Zhao. 2020. “Logistics Optimisation of Slab Pre-Marshalling Problem in Steel Industry.” International Journal of Production Research 58: 4050–4070. https://doi.org/10.1080/00207543.2019.1641238
   Web of Science ®Google Scholar
• Gharehgozli, A. H., Y. Yu, R. de Koster, and J. T. Udding. 2014. “A Decision-Tree Stacking Heuristic Minimising the Expected Number of Reshuffles at a Container Terminal.” International Journal of Production Research 52: 2592–2611. https://doi.org/10.1080/00207543.2013.861618
   Web of Science ®Google Scholar
• Hakan Akyüz, M., and C. Y. Lee. 2014. “A Mathematical Formulation and Efficient Heuristics for the Dynamic Container Relocation Problem.” Naval Research Logistics (NRL) 61: 101–118. https://doi.org/10.1002/nav.21569
   Web of Science ®Google Scholar
• Hottung, A., S. Tanaka, and K. Tierney. 2020. “Deep Learning Assisted Heuristic Tree Search for the Container Pre-marshalling Problem.” Computers & Operations Research 113: 104781. http://doi.org/10.1016/j.cor.2019.104781.
   Web of Science ®Google Scholar
• Hottung, A., and K. Tierney. 2016. “A Biased Random-key Genetic Algorithm for the Container pre-Marshalling Problem.” Computers & Operations Research 75: 83–102. https://doi.org/10.1016/j.cor.2016.05.011
   Web of Science ®Google Scholar
• Jang, B., M. Kim, G. Harerimana, and J. W. Kim. 2019. “Q-learning Algorithms: A Comprehensive Classification and Applications.” IEEE Access 7: 133653–133667. https://doi.org/10.1109/ACCESS.2019.2941229
   Google Scholar
• Jin, B. 2020. “On the Integer Programming Formulation for the Relaxed Restricted Container Relocation Problem.” European Journal of Operational Research 281 (2): 475–482. http://doi.org/10.1016/j.ejor.2019.08.041.
   Web of Science ®Google Scholar
• Jin, B., and M. Yu. 2021. “Note on the Dominance Rules in the Exact Algorithm for the Container Pre-marshalling Problem by Tanaka & Tierney (2018).” European Journal of Operational Research 293 (2): 802–807. http://doi.org/10.1016/j.ejor.2020.12.041.
   Web of Science ®Google Scholar
• Jin, B., W. Zhu, and A. Lim. 2015. “Solving the Container Relocation Problem by an Improved Greedy Look-Ahead Heuristic.” European Journal of Operational Research 240: 837–847. https://doi.org/10.1016/j.ejor.2014.07.038
   Web of Science ®Google Scholar
• Jooken, J., P. Leyman, T. Wauters, and P. De Causmaecker. 2023. “Exploring Search Space Trees Using an Adapted Version of Monte Carlo Tree Search for Combinatorial Optimization Problems.” Computers & Operations Research 150: 106070. https://doi.org/10.1016/j.cor.2022.106070.
   Google Scholar
• Kim, K. H., and G. P. Hong. 2006. “A Heuristic Rule for Relocating Blocks.” Computers & Operations Research 33 (4): 940–954. https://doi.org/10.1016/j.cor.2004.08.005.
   Web of Science ®Google Scholar
• Kocsis, L., and C. Szepesvári. 2006. “Bandit Based Monte-Carlo Planning.” In European Conference on Machine Learning, edited by J. Fürnkranz, T. Scheffer, and M. Spiliopoulou, 282–293. Berlin: Springer. https://doi.org/10.1007/11871842_29
   Google Scholar
• Lee, Y., and S. L. Chao. 2009. “A Neighborhood Search Heuristic for Pre-Marshalling Export Containers.” European Journal of Operational Research 196: 468–475. https://doi.org/10.1016/j.ejor.2008.03.011
   Web of Science ®Google Scholar
• Lee, Y., and N. Y. Hsu. 2007. “An Optimization Model for the Container Pre-Marshalling Problem.” Computers & Operations Research 34 (11): 3295–3313. https://doi.org/10.1016/j.cor.2005.12.006.
   Web of Science ®Google Scholar
• Li, K., Q. Deng, L. Zhang, Q. Fan, G. Gong, and S. Ding. 2021. “An Effective MCTS-Based Algorithm for Minimizing Makespan in Dynamic Flexible job Shop Scheduling Problem.” Computers & Industrial Engineering 155: 107211. https://doi.org/10.1016/j.cie.2021.107211
   Web of Science ®Google Scholar
• Li, Y., M. C. Fu, and J. Xu. 2021. “An Optimal Computing Budget Allocation Tree Policy for Monte Carlo Tree Search.” IEEE Transactions on Automatic Control 67 (6): 2685–2699. https://doi.org/10.1109/TAC.2021.3088792
   Web of Science ®Google Scholar
• Parreño-Torres, C., R. Alvarez-Valdes, and R. Ruiz. 2019. “Integer Programming Models for the pre-Marshalling Problem.” European Journal of Operational Research 274: 142–154. https://doi.org/10.1016/j.ejor.2018.09.048
   Web of Science ®Google Scholar
• Saqlain, M., S. Ali, and J. Y. Lee. 2022. “A Monte-Carlo Tree Search Algorithm for the Flexible Job-Shop Scheduling in Manufacturing Systems.” Flexible Services and Manufacturing Journal 35 (2): 548–571.https://doi.org/10.1007/s10696-021-09437-4.
   Web of Science ®Google Scholar
• Schadd, M. P., M. H. Winands, M. J. Tak, and J. W. Uiterwijk. 2012. “Single-Player Monte-Carlo Tree Search for SameGame.” Knowledge-Based Systems 34: 3–11. https://doi.org/10.1016/j.knosys.2011.08.008
   Web of Science ®Google Scholar
• Scholl, J., D. Boywitz, and N. Boysen. 2018. “On the Quality of Simple Measures Predicting Block Relocations in Container Yards.” International Journal of Production Research 56: 60–71. https://doi.org/10.1080/00207543.2017.1394595
   Web of Science ®Google Scholar
• Silver, D., J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, … D. Hassabis. 2017. “Mastering the Game of Go Without Human Knowledge.” Nature 550: 354–359. https://doi.org/10.1038/nature24270.
   PubMed Web of Science ®Google Scholar
• Świechowski, M., K. Godlewski, B. Sawicki, and J. Mańdziuk. 2023. “Monte Carlo Tree Search: A Review of Recent Modifications and Applications.” Artificial Intelligence Review 56: 2497–2562. https://doi.org/10.1007/s10462-022-10228-y
   Web of Science ®Google Scholar
• Ting, C. J., and K. C. Wu. 2017. “Optimizing Container Relocation Operations at Container Yards with Beam Search.” Transportation Research Part E: Logistics and Transportation Review 103: 17–31. https://doi.org/10.1016/j.tre.2017.04.010
   Web of Science ®Google Scholar
• Wang, N., B. Jin, and A. Lim. 2015. “Target-Guided Algorithms for the Container Pre-Marshalling Problem.” Omega 53: 67–77. https://doi.org/10.1016/j.omega.2014.12.002
   Web of Science ®Google Scholar
• Wang, N., B. Jin, Z. Zhang, and A. Lim. 2017. “A Feasibility-Based Heuristic for the Container pre-Marshalling Problem.” European Journal of Operational Research 256: 90–101. https://doi.org/10.1016/j.ejor.2016.05.061
   Web of Science ®Google Scholar
• Wang, H., Q. Yan, and J. Wang. 2023. “Blockchain-Secured Multi-Factory Production with Collaborative Maintenance using Q Learning-Based Optimisation Approach.” International Journal of Production Research 61 (11): 3685–3702. https://doi.org/10.1080/00207543.2021.2002968.
   Web of Science ®Google Scholar
• Zehendner, E., and D. Feillet. 2014. “A Branch and Price Approach for the Container Relocation Problem.” International Journal of Production Research 52: 7159–7176. https://doi.org/10.1080/00207543.2014.965358
   Web of Science ®Google Scholar
• Zhang, C., H. Guan, Y. Yuan, W. Chen, and T. Wu. 2020. “Machine Learning-Driven Algorithms for the Container Relocation Problem.” Transportation Research Part B: Methodological 139: 102–131. https://doi.org/10.1016/j.trb.2020.05.017
   Web of Science ®Google Scholar
• Zhou, C., S. Zhu, M. G. Bell, L. H. Lee, and E. P. Chew. 2022. “Emerging Technology and Management Research in the Container Terminals: Trends and the COVID-19 Pandemic Impacts.” Ocean & Coastal Management 230: 106318. https://doi.org/10.1016/j.ocecoaman.2022.106318
   PubMed Web of Science ®Google Scholar
• Zhu, W., H. Qin, A. Lim, and H. Zhang. 2012. “Iterative Deepening A* Algorithms for the Container Relocation Problem.” IEEE Transactions on Automation Science and Engineering 9 (4): 710–722. http://doi.org/10.1109/TASE.2012.2198642.
   Web of Science ®Google Scholar