First steps towards real-world traffic signal control optimisation by reinforcement learning

References

• Andrychowicz, O. M., Baker, B., Chociej, M., Jozefowicz, R., McGrew, B., & Pachocki, J. (2020). Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1), 3–20. https://doi.org/10.1177/0278364919887447
   Web of Science ®Google Scholar
• Aslani, M., Mesgari, M. S., & Wiering, M. (2017). Adaptive traffic signal control with actor- critic methods in a real-world traffic network with different traffic disruption events. Transportation Research Part C: Emerging Technologies, 85, 732–752. https://www.sciencedirect.com/science/article/pii/S0968090X17302681
   Web of Science ®Google Scholar
• Bachechi, C., Rollo, F., & Po, L. (2022). Detection and classification of sensor anomalies for simulating urban traffic scenarios. Cluster Computing, 25(4), 2793–2817. https://doi.org/10.1007/s10586-021-03445-7
   Web of Science ®Google Scholar
• Bakker, B. (2001). Reinforcement learning with long short-term memory. Advances in Neural Information Processing Systems, 14. https://proceedings.neurips.cc/paper/2001/hash/a38b16173474ba8b1a95bcbc30d3b8a5-Abstract.html
   Google Scholar
• Behrisch, M., Bieker, L., Erdmann, J., & Krajzewicz, D. (2011). Sumo–simulation of urban mobility: An overview. In Proceedings of Simul 2011, the third International Conference on Advances in System Simulation, Barcelona, Spain.
   Google Scholar
• Bhouri, N., Mayorano, F. J., Lotito, P. A., Salem, H. H., & Lebacque, J. P. (2015). Public transport priority for multimodal urban traffic control. Cybernetics and Information Technologies, 15(5), 766. https://doi.org/10.1515/cait-2015-0014
   Google Scholar
• Bouktif, S., Cheniki, A., & Ouni, A. (2021). Traffic signal control using hybrid action space deep reinforcement learning. Sensors, 21(7), 2302. https://doi.org/10.3390/s21072302
   PubMed Web of Science ®Google Scholar
• Brilon, W. (2015). Handbuch für die bemessung von straßenverkehrsanlagen. FGSV.
   Google Scholar
• Busch, J. V., Latzko, V., Reisslein, M., & Fitzek, F. H. (2020). Optimised traffic light management through reinforcement learning: Traffic state agnostic agent vs. holistic agent with current v2i traffic state knowledge. IEEE Open Journal of Intelligent Transportation Systems, 1, 201–216. https://doi.org/10.1109/OJITS.2020.3027518
   Google Scholar
• Chen, R., Fang, F., & Sadeh, N. (2022). The Real Deal: A Review of Challenges and Opportunities in Moving Reinforcement Learning-Based Traffic Signal Control Systems Towards Reality. ATT ’22: Workshop on Agents in Traffic and Transportation, Vienna, Austria.
   Google Scholar
• Chen, C., Wei, H., Xu, N., Zheng, G., Yang, M., Xiong, Y., Xu, K., and Li, Z. (2020). Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control. In Proceedings of the Aaai Conference on Artificial Intelligence, New York, USA (Vol. 34., pp. 3414–3421).
   Google Scholar
• Chu, T., Wang, J., Codecà, L., & Li, Z. (2020). Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 21(3), 1086–1095. https://doi.org/10.1109/TITS.2019.2901791
   Web of Science ®Google Scholar
• Cools, S.-B., Gershenson, C., & D’Hooghe, B. (2013). Self-Organizing Traffic Lights: A Realistic Simulation. Advances in Applied Self-Organizing Systems (pp. 45–55). London: Springer. https://doi.org/10.1007/978-1-4471-5113-5_3
   Google Scholar
• DOT. (2012). Manual on uniform traffic signal control devices. US Department of Transportation.
   Google Scholar
• Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., & Hester, T. (2021). Challenges of real-world reinforcement learning: Definitions, benchmarks and analysis. Machine Learning, 110(9), 2419–2468. https://doi.org/10.1007/s10994-021-05961-4
   Web of Science ®Google Scholar
• FGSV. (2015/2021). Guidelines for traffic signals (rilsa). German Road and Transport Research Association.
   Google Scholar
• GEVAS software GmbH. (2015). Vtcontrol. Author.
   Google Scholar
• GEVAS software GmbH. (2020). Open trelan user manual. Author.
   Google Scholar
• Gregurić, M., Vujić, M., Alexopoulos, C., & Miletić, M. (2020). Application of deep reinforcement learning in traffic signal control: An overview and impact of open traffic data. Applied Sciences, 10(11), 4011. https://doi.org/10.3390/app10114011
   Google Scholar
• Haarnoja, T., Zhou, A., Abbeel, P., & Levine, S. (2018). Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.
   Google Scholar
• Harth, M., Langer, M., & Bogenberger, K. (2021). Automated calibration of traffic demand and traffic lights in sumo using real-world observations. In Sumo Conference Proceedings (Vol. 2. pp. 133–148).
   Google Scholar
• Haydari, A., & Yilmaz, Y. (2020). Deep reinforcement learning for intelligent transportation systems: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(1), 11–32. https://doi.org/10.1109/TITS.2020.3008612
   Web of Science ®Google Scholar
• Henderson, P., Islam, R., Bachman, P., Pineau, J., Precup, D., & Meger, D. (2018). Deep reinforcement learning that matters. In Proceedings of the AAAI Conference on Artificial Intelligence, New Orleans, USA (Vol. 32).
   Google Scholar
• Kaspar, M., Osorio, J. D. M., & Bock, J. (2020). Sim2real transfer for reinforcement learning without dynamics randomization. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Las Vegas, USA (pp. 4383–4388).
   Google Scholar
• Kemper, C. (2006). Dynamische simulation des verkehrsablaufs unter verwendung statischer verflechtungsmatrizen [ Unpublished doctoral dissertation]. Universität Hannover.
   Google Scholar
• Kemper, C. (2015). Drivers. Mobil.tum 2015 – international scientific conference on mobility and transport technologies, solutions and perspectives for intelligent transport systems. 14.
   Google Scholar
• Krajzewicz, D. (2010). Traffic Simulation with SUMO – Simulation of Urban Mobility. Fundamentals of Traffic Simulation. International Series in Operations Research & Management Science (Vol. 145., pp. 269–293). New York, USA: Springer. https://doi.org/10.1007/978-1-4419-6142-6_7
   Google Scholar
• Krauß, S. (1998). Microscopic modeling of traffic flow: Investigation of collision free vehicle dynamics (Report No. DLR-FB-98-08). U.S. Department of Energy Office of Scientific and Technical Information.
   Google Scholar
• Li, M., Alhajyaseen, W., & Nakamura, H. (2010). A traffic signal optimization strategy considering both vehicular and pedestrian flows. In Compendium of Papers Cd-rom, the 89th Annual Meeting of the Transportation Research Board, Washington , USA (pp. 10–14).
   Google Scholar
• Liang, X., Du, X., Wang, G., & Han, Z. (2019). A deep reinforcement learning network for traffic light cycle control. IEEE Transactions on Vehicular Technology, 68(2), 1243–1253. https://doi.org/10.1109/TVT.2018.2890726
   Web of Science ®Google Scholar
• Lillicrap, T. P., Hunt, J. J., Pritzel, A., Heess, N., Erez, T., Tassa, Y. Silver, D. Wierstra, D. (2019). Continuous control with deep reinforcement learning.
   Google Scholar
• Lin, Y., Dai, X., Li, L., & Wang, F.-Y. (2018). An efficient deep reinforcement learning model for urban traffic control.
   Google Scholar
• Li, Z., Yu, H., Zhang, G., Dong, S., & Xu, C.-Z. (2021). Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning. ArXiv , abs/Transportation Research Part C: Emerging Technologies, 2104(9936), 103059. https://doi.org/10.1016/j.trc.2021.103059
   Google Scholar
• Mankowitz, D. J., Dulac-Arnold, G., & Hester, T. (2019). Challenges of real-world reinforcement learning.
   Google Scholar
• Mannion, P., Duggan, J., & Howley, E. (2016). An Experimental Review of Reinforcement Learning Algorithms for Adaptive Traffic Signal Control. Autonomic Road Transport Support Systems. Autonomic Systems. (pp. 47–66). Springer. https://doi.org/10.1007/978-3-319-25808-9_4
   Google Scholar
• Meess, H., Gerner, J., Hein, D., Schmidtner, S., & Elger, G. (2022a). Real world traffic optimization by reinforcement learning: A concept. In International Workshop on Agent- based Modelling of Urban Systems (abmus) Proceedings: 2022, Auckland, New Zealand (pp. 49–54).
   Google Scholar
• Meess, H., Gerner, J., Hein, D., Schmidtner, S., & Elger, G. (2022b). Reinforcement learning for traffic signal control optimization: A concept for real-world implementation. Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, Auckland, New Zealand (pp. 1699–1701).
   Google Scholar
• Müller, A., Rangras, V., Ferfers, T., Hufen, F., Schreckenberg, L., Jasperneite, J., Schnittker, G., Waldmann, M., Friesen, M., and Wiering, M. (2021). Towards real-world deployment of reinforcement learning for traffic signal control. 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, USA (pp. 507–514).
   Google Scholar
• Müller, A., & Sabatelli, M. (2022). Safe and psychologically pleasant traffic signal control with reinforcement learning using action masking. 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), Macau, China (pp. 951–958).
   Google Scholar
• Noaeen, M., Naik, A., Goodman, L., Crebo, J., Abrar, T., Abad, Z. S. H. Bazzan, A.L. Far, B. (2022). Reinforcement learning in urban network traffic signal control: A systematic literature review. Expert Systems with Applications, 199, 116830. https://doi.org/10.1016/j.eswa.2022.116830
   Web of Science ®Google Scholar
• Oertel, R., Wagner, P., Krimmling, J., & Körner, M. (2011). Verlustzeitenbasierte lsasteuerung eines einzelknotens. Straßenverkehrstechnik (pp. 561–568).
   Google Scholar
• Oroojlooy, A., & Hajinezhad, D. (2022). A review of cooperative multi-agent deep reinforcement learning. Applied Intelligence, 53(11), 13677–13722. https://doi.org/10.1007/s10489-022-04105-y
   Web of Science ®Google Scholar
• Pandit, K., Ghosal, D., Zhang, H. M., & Chuah, C.-N. (2013). Adaptive traffic signal control with vehicular ad hoc networks. IEEE Transactions on Vehicular Technology, 62(4), 1459–1471. https://doi.org/10.1109/TVT.2013.2241460
   Web of Science ®Google Scholar
• Pohlmann, T. (2010). New approaches for online control of urban traffic signal systems [ Unpublished doctoral dissertation]. Technische Universität Braunschweig.
   Google Scholar
• Qadri, S. S. S. M., Gökçe, M. A., & Öner, E. (2020). State-of-art review of traffic signal control methods: Challenges and opportunities. European Transport Research Review, 12(1), 1–23. https://doi.org/10.1186/s12544-020-00439-1
   Web of Science ®Google Scholar
• Rodrigues, F., & Azevedo, C. L. (2019). Towards robust deep reinforcement learning for traffic signal control: Demand surges, incidents and sensor failures. In2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand (pp. 3559–3566).
   Google Scholar
• Schulman, J., Levine, S., Abbeel, P., Jordan, M., & Moritz, P. (2015). Trust region policy optimization. In International Conference on Machine Learning, Lille, France (pp. 1889–1897).
   Google Scholar
• Shaikh, P. W., El-Abd, M., Khanafer, M., & Gao, K. (2022). A review on swarm intelligence and evolutionary algorithms for solving the traffic signal control problem. IEEE Transactions on Intelligent Transportation Systems, 23(1), 48–63. https://doi.org/10.1109/TITS.2020.3014296
   Web of Science ®Google Scholar
• Sutton, R. S., & Barto, A. G. (2018). In Reinforcement Learning: An Introduction. MIT press.
   Google Scholar
• Tan, K. L., Sharma, A., & Sarkar, S. (2020). Robust deep reinforcement learning for traffic signal control. Journal of Big Data Analytics in Transportation, 2(3), 263–274. https://doi.org/10.1007/s42421-020-00029-6
   Google Scholar
• Treiber, M., Hennecke, A., & Helbing, D. (2000, August). Congested traffic states in empirical observations and microscopic simulations. Physical Review E, 62(2), 1805–1824. https://doi.org/10.1103/PhysRevE.62.1805
   PubMed Web of Science ®Google Scholar
• United Nations, D. O. E., & Social Affairs, P. D. (2018). World urbanization prospects: the 2018 revision. Custom data acquired via website. https://population.un.org/wup/DataQuery/
   Google Scholar
• Varaiya, P. (2013). Max pressure control of a network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36, 177–195. https://doi.org/10.1016/j.trc.2013.08.014
   Web of Science ®Google Scholar
• Wang, Z., Bapst, V., Heess, N., Mnih, V., Munos, R., Kavukcuoglu, K., & de Freitas, N. (2016). Sample efficient actor-critic with experience replay. International Conference on Learning Representations (ICLR), Toulon, France.
   Google Scholar
• Wang, L., Ma, Z., Dong, C., & Wang, H. (2022). Human-centric multimodal deep (HMD) traffic signal control. IET Intelligent Transport Systems, 17(4), 744–753. https://doi.org/10.1049/itr2.12300
   Web of Science ®Google Scholar
• Wang, L., Pan, Z., & Wang, J. (2021). A review of reinforcement learning based intelligent optimization for manufacturing scheduling. Complex System Modeling and Simulation, 1(4), 257–270. https://doi.org/10.23919/CSMS.2021.0027
   Google Scholar
• Wei, H., Zheng, G., Yao, H., & Li, Z. (2018). Intellilight: A reinforcement learning approach for intelligent traffic light control. In Proceedings of the 24th ACM Sigkdd International Conference on Knowledge Discovery & Data mining, London, UK (pp. 2496–2505).
   Google Scholar
• Wu, T., Zhou, P., Liu, K., Yuan, Y., Wang, X., Huang, H., & Wu, D. O. (2020). Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks. IEEE Transactions on Vehicular Technology, 69(8), 8243–8256. https://doi.org/10.1109/TVT.2020.2997896
   Web of Science ®Google Scholar
• Zeng, J., Hu, J., & Zhang, Y. (2018). Adaptive traffic signal control with deep recurrent q-learning. In 2018 IEEE Intelligent Vehicles Symposium (iv), Changshu, China (pp. 1215–1220).
   Google Scholar
• Zhao, D., Dai, Y., & Zhang, Z. (2012). Computational intelligence in urban traffic signal control: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(4), 485–494. https://doi.org/10.1109/TSMCC.2011.2161577
   Web of Science ®Google Scholar
• Zheng, G., Zang, X., Xu, N., Wei, H., Yu, Z., Gayah, V. Xu, K. Li, Z. (2019). Diagnosing reinforcement learning for traffic signal control. arXiv preprint arXiv:1905.04716.
   Google Scholar