Deep Q learning-based traffic signal control algorithms: Model development and evaluation with field data

References

• Abdoos, M., Mozayani, N., & Bazzan, A. L. (2011). Traffic light control in non-stationary environments based on multi agent q-learning [Paper presentation]. 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), IEEE. pp. 1580–1585.
   Google Scholar
• Arel, I., Liu, C., Urbanik, T., & Kohls, A. G. (2010). Reinforcement learning-based multi-agent system for network traffic signal control. IET Intelligent Transport Systems, 4(2), 128–135. https://doi.org/10.1049/iet-its.2009.0070
   Web of Science ®Google Scholar
• Bazzan, A. L., do Amarante, M. d B., Sommer, T., & Benavides, A. J. (2010). Itsumo: An agent-based simulator for its applications [Paper presentation]. Proc. of the 4th Workshop on Artificial Transportation Systems and Simulation, IEEE. p. 8.
   Google Scholar
• Cai, C., Wong, C. K., & Heydecker, B. G. (2009). Adaptive traffic signal control using approximate dynamic programming. Transportation Research Part C: Emerging Technologies, 17(5), 456–474. https://doi.org/10.1016/j.trc.2009.04.005
   Web of Science ®Google Scholar
• Casas, N. (2017). Deep deterministic policy gradient for urban traffic light control. arXiv preprint arXiv:1703.09035.
   Google Scholar
• Chia, I., Wu, X., Dhaliwal, S. S., Thai, J., & Jia, X. (2017). Evaluation of actuated, coordinated, and adaptive signal control systems: A case study. Journal of Transportation Engineering, Part A: Systems, 143(9), 05017007. https://doi.org/10.1061/JTEPBS.0000068
   Web of Science ®Google Scholar
• Chu, T., Wang, J., Codecà, L., & Li, Z. (2020). Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 21(3), 1086–1095. https://doi.org/10.1109/TITS.2019.2901791
   Web of Science ®Google Scholar
• Chun-Gui, L., Meng, W., Zi-Gaung, S., Fei-Ying, L., & Zeng-Fang, Z. (2009). Urban traffic signal learning control using fuzzy actor-critic methods. ICNC’09. Fifth International Conference on Natural Computation, 2009, IEEE. Vol. 1, pp. 368–372.
   Google Scholar
• Coifman, B., & Dhoorjaty, S. (2004). Event data-based traffic detector validation tests. Journal of Transportation Engineering, 130(3), 313–321. https://doi.org/10.1061/(ASCE)0733-947X(2004)130:3(313)
   Web of Science ®Google Scholar
• Dion, F., & Rakha, H. (2006). Estimating dynamic roadway travel times using automatic vehicle identification data for low sampling rates. Transportation Research Part B: Methodological, 40(9), 745–766. https://doi.org/10.1016/j.trb.2005.10.002
   Web of Science ®Google Scholar
• Fei, X., Zhang, Y., Liu, K., & Guo, M. (2013). Bayesian dynamic linear model with switching for real-time short-term freeway travel time prediction with license plate recognition data. Journal of Transportation Engineering, 139(11), 1058–1067. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000538
   Google Scholar
• Gao, J., Shen, Y., Liu, J., Ito, M., & Shiratori, N. (2017). Adaptive traffic signal control: Deep reinforcement learning algorithm with experience replay and target network. arXiv preprint arXiv:1705.02755.
   Google Scholar
• Genders, W., & Razavi, S. (2016). Using a deep reinforcement learning agent for traffic signal control. arXiv preprint arXiv:1611.01142.
   Google Scholar
• Genders, W., & Razavi, S. (2019). Asynchronous n-step q-learning adaptive traffic signal control. Journal of Intelligent Transportation Systems, 23(4), 319–331. https://doi.org/10.1080/15472450.2018.1491003
   Web of Science ®Google Scholar
• Gettman, D., Madrigal, G., Allen, S., Boyer, T., Walker, S., Tong, J., Phillips, S., Liu, H., Wu, X., & Hu, H. (2012). Operation of traffic signal systems in oversaturated conditions, vol. 2. Technical report.
   Google Scholar
• Gong, Y., Abdel-Aty, M., & Park, J. (2021). Evaluation and augmentation of traffic data including bluetooth detection system on arterials. Journal of Intelligent Transportation Systems, 25(6), 561–573. https://doi.org/10.1080/15472450.2019.1632707
   Web of Science ®Google Scholar
• Gregurić, M., Vujić, M., Alexopoulos, C., & Miletić, M. (2020). Application of deep reinforcement learning in traffic signal control: An overview and impact of open traffic data. Applied Sciences, 10(11), 4011. https://doi.org/10.3390/app10114011
   Google Scholar
• He, K., Zhang, X., Ren, S., & Sun, J. (2015). Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. Proceedings of the IEEE International Conference on Computer Vision, pp. 1026–1034.
   Google Scholar
• Hessel, M., Modayil, J., Van Hasselt, H., Schaul, T., Ostrovski, G., Dabney, W., Horgan, D., Piot, B., Azar, M., & Silver, D. (2018). Rainbow: Combining improvements in deep reinforcement learning [Paper presentation]. Thirty-Second AAAI Conference on Artificial Intelligence.
   Google Scholar
• Ho, J., & Ermon, S. (2016). Generative adversarial imitation learning. Advances in Neural Information Processing Systems, 1, 4565–4573.
   Google Scholar
• Hunt, P., Robertson, D., Bretherton, R., & Winton, R. (1981). Scoot-a traffic responsive method of coordinating signals. Technical report.
   Google Scholar
• Jin, J., & Ma, X. (2015). Adaptive group-based signal control using reinforcement learning with eligibility traces [Paper presentation]. 2015 IEEE 18th International Conference on Intelligent Transportation Systems (ITSC), IEEE. pp. 2412–2417.
   Google Scholar
• Kazagli, E., & Koutsopoulos, H. (2013). Estimation of arterial travel time from automatic number plate recognition data. Transportation Research Record: Journal of the Transportation Research Board, 2391(1), 22–31. https://doi.org/10.3141/2391-03
   Google Scholar
• Khamis, M. A., Gomaa, W., & El-Shishiny, H. (2012). Multi-objective traffic light control system based on bayesian probability interpretation [Paper presentation]. 2012 15th International IEEE Conference on Intelligent Transportation Systems (ITSC), IEEE. pp. 995–1000.
   Google Scholar
• Kingma, D. P., & Ba, J. (2015). Adam: A method for stochastic optimization. Presented at the 3rd International Conference for Learning Representations, San Diego.
   Google Scholar
• Li, L., Lv, Y., & Wang, F.-Y. (2016). Traffic signal timing via deep reinforcement learning. IEEE/CAA Journal of Automatica Sinica, 3(3), 247–254.
   Google Scholar
• Lian, F., Chen, B., Zhang, K., Miao, L., Wu, J., & Luan, S. (2021). Adaptive traffic signal control algorithms based on probe vehicle data. Journal of Intelligent Transportation Systems, 25(1), 41–57. https://doi.org/10.1080/15472450.2020.1750384
   Web of Science ®Google Scholar
• Liang, X., Du, X., Wang, G., & Han, Z. (2018). Deep reinforcement learning for traffic light control in vehicular networks. arXiv preprint arXiv:1803.11115.
   Google Scholar
• Liu, H. X., Ma, W., Wu, X., & Hu, H. (2008). Development of a real-time arterial performance monitoring system using traffic data available from existing signal systems. Technical report. Minnesota Department of Transportation.
   Google Scholar
• Liu, M., Deng, J., Xu, M., Zhang, X., & Wang, W. (2017). Cooperative deep reinforcement learning for traffic signal control [Paper presentation]. UrbComp’17, Halifax, Nova Scotia, Canada.
   Google Scholar
• Lotufo, R., Morgan, A., & Johnson, A. (1990). Automatic number-plate recognition [Paper presentation]. IEE Colloquium on Image Analysis for Transport Applications, IET. pp. 6/1–6/6.
   Google Scholar
• Ma, D., Xiao, J., & Ma, X. (2021). A decentralized model predictive traffic signal control method with fixed phase sequence for urban networks. Journal of Intelligent Transportation Systems, 25(5), 455–468. https://doi.org/10.1080/15472450.2020.1734801
   Web of Science ®Google Scholar
• Mannion, P., Duggan, J., & Howley, E. (2016). An experimental review of reinforcement learning algorithms for adaptive traffic signal control. In Autonomic road transport support systems (pp. 47–66). Springer.
   Google Scholar
• Medina, J. C., & Benekohal, R. F. (2012). Traffic signal control using reinforcement learning and the max-plus algorithm as a coordinating strategy [Paper presentation]. 2012 15th International IEEE Conference on Intelligent Transportation Systems (ITSC), IEEE. pp. 596–601.
   Google Scholar
• Merel, J., Tassa, Y., Srinivasan, S., Lemmon, J., Wang, Z., Wayne, G., & Heess, N. (2017). Learning human behaviors from motion capture by adversarial imitation. arXiv preprint arXiv:1707.02201.
   Google Scholar
• Mnih, V., Kavukcuoglu, K., Silver, D., Rusu, A. A., Veness, J., Bellemare, M. G., Graves, A., Riedmiller, M., Fidjeland, A. K., Ostrovski, G., Petersen, S., Beattie, C., Sadik, A., Antonoglou, I., King, H., Kumaran, D., Wierstra, D., Legg, S., & Hassabis, D. (2015). Human-level control through deep reinforcement learning. Nature, 518(7540), 529–533. https://doi.org/10.1038/nature14236
   PubMed Web of Science ®Google Scholar
• Mousavi, S. S., Schukat, M., & Howley, E. (2017). Traffic light control using deep policy-gradient and value-function-based reinforcement learning. IET Intelligent Transport Systems, 11(7), 417–423. https://doi.org/10.1049/iet-its.2017.0153
   Web of Science ®Google Scholar
• Norouzi, M., Abdoos, M., & Bazzan, A. L. (2021). Experience classification for transfer learning in traffic signal control. The Journal of Supercomputing, 77(1), 780–795. https://doi.org/10.1007/s11227-020-03287-x
   Web of Science ®Google Scholar
• Oliveira-Neto, F., Han, L., & Jeong, M. (2009). Tracking large trucks in real time with license plate recognition and text-mining techniques. Transportation Research Record: Journal of the Transportation Research Board, 2121(1), 121–127. https://doi.org/10.3141/2121-13
   Google Scholar
• Prabuchandran, K., An, H. K., & Bhatnagar, S. (2015). Decentralized learning for traffic signal control. 2015 7th International Conference on Communication Systems and Networks (COMSNETS), IEEE. pp. 1–6.
   Google Scholar
• Prashanth, L., & Bhatnagar, S. (2011). Reinforcement learning with function approximation for traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 12(2), 412–421. https://doi.org/10.1109/TITS.2010.2091408
   Web of Science ®Google Scholar
• Rahman, T. A., & Rahim, S. K. A. (2013). RFID vehicle plate number (e-plate) for tracking and management system. 2013 International Conference on Parallel and Distributed Systems, IEEE. pp. 611–616.
   Google Scholar
• Roderick, M., MacGlashan, J., & Tellex, S. (2017). Implementing the deep q-network. arXiv preprint arXiv:1711.07478.
   Google Scholar
• Salkham, A. A., Cunningham, R., Garg, A., & Cahill, V. (2008). A collaborative reinforcement learning approach to urban traffic control optimization [Paper presentation]. Proceedings of the 2008 IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology, IEEE Computer Society. Vol. 02, pp. 560–566. https://doi.org/10.1109/WIIAT.2008.88
   Google Scholar
• Shuldiner, P. W., D'Agostino, S. A., & Woodson, J. B. (1996). Determining detailed origin-destination and travel time patterns using video and machine vision license plate matching. Transportation Research Record: Journal of the Transportation Research Board, 1551(1), 8–17. https://doi.org/10.1177/0361198196155100102
   Google Scholar
• Su, S., & Tham, C.-K. (2007). Sensorgrid for real-time traffic management. ISSNIP 2007. 3rd International Conference on Intelligent Sensors, Sensor Networks and Information, 2007, IEEE. pp. 443–448.
   Google Scholar
• Tan, T., Bao, F., Deng, Y., Jin, A., Dai, Q., & Wang, J. (2020). Cooperative deep reinforcement learning for large-scale traffic grid signal control. IEEE Transactions on Cybernetics, 50(6), 2687–2700. https://doi.org/10.1109/TCYB.2019.2904742
   PubMed Web of Science ®Google Scholar
• Teo, K. T. K., Yeo, K. B., Chin, Y. K., Chuo, H. S. E., & Tan, M. K. (2014). Agent-based traffic flow optimization at multiple signalized intersections. Modelling Symposium (AMS), 2014 8th Asia, IEEE. pp. 21–26.
   Google Scholar
• Urbanik, T., Tanaka, A., Lozner, B., Lindstrom, E., Lee, K., Quayle, S., Beaird, S., Tsoi, S., Ryus, P., & Gettman, D. (2015). Signal timing manual. Transportation Research Board.
   Google Scholar
• Walsh, T. J., Goschin, S., & Littman, M. L. (2010). Integrating sample-based planning and model-based reinforcement learning [Paper presentation]. Twenty-Fourth AAAI Conference on Artificial Intelligence.
   Google Scholar
• Wei, H., Zheng, G., Yao, H., & Li, Z. (2018). Intellilight: A reinforcement learning approach for intelligent traffic light control [Paper presentation]. Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, ACM. pp. 2496–2505.
   Google Scholar
• Wilson, C., Willis, C., Hendrikz, J. K., Le Brocque, R., & Bellamy, N. (2010). Speed cameras for the prevention of road traffic injuries and deaths. Cochrane Database of Systematic Reviews, 11, CD004607.
   Google Scholar
• Xiong, C., Yang, X. T., Zhang, L., Lee, M., Zhou, W., & Raqib, M. (2021). An integrated modeling framework for active traffic management and its applications in the washington, DC area. Journal of Intelligent Transportation Systems, 25(6), 609–619. https://doi.org/10.1080/15472450.2021.1878891
   Web of Science ®Google Scholar
• Xu, N., Zheng, G., Xu, K., Zhu, Y., & Li, Z. (2019). Targeted knowledge transfer for learning traffic signal plans [Paper presentation]. PAKDD (2). pp. 175–187.
   Google Scholar
• Yau, K.-L. A., Qadir, J., Khoo, H. L., Ling, M. H., & Komisarczuk, P. (2017). A survey on reinforcement learning models and algorithms for traffic signal control. ACM Computing Surveys, 50(3), 1–38. https://doi.org/10.1145/3068287
   Web of Science ®Google Scholar
• Yu, H., Yang, S., Wu, Z., & Ma, X. (2018). Vehicle trajectory reconstruction from automatic license plate reader data. International Journal of Distributed Sensor Networks, 14(2), 155014771875563. https://doi.org/10.1177/1550147718755637
   Web of Science ®Google Scholar
• Zhang, K., Jia, N., Zheng, L., & Liu, Z. (2019). A novel generative adversarial network for estimation of trip travel time distribution with trajectory data. Transportation Research Part C: Emerging Technologies, 108, 223–244. https://doi.org/10.1016/j.trc.2019.09.019
   Web of Science ®Google Scholar
• Zhang, X., Nihan, N., & Wang, Y. (2005). Improved dual-loop detection system for collecting real-time truck data. Transportation Research Record: Journal of the Transportation Research Board, 1917(1), 108–115. https://doi.org/10.1177/0361198105191700113
   Google Scholar
• Zhang, X., Wang, Y., Nihan, N., & Hallenbeck, M. (2003). Development of a system for collecting loop-detector event data for individual vehicles. Transportation Research Record: Journal of the Transportation Research Board, 1855(1), 168–175. https://doi.org/10.3141/1855-21
   Google Scholar
• Zhao, D., Dai, Y., & Zhang, Z. (2012). Computational intelligence in urban traffic signal control: A survey. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(4), 485–494.
   Web of Science ®Google Scholar