First steps towards real-world traffic signal control optimisation by reinforcement learning

Figures & data

Table 1. Excerpt of previous works for DRL in traffic signal optimisation and their system design decisions with regard to the issues mentioned above.

Work	Action Definition	Real-world traffic	Multimodality	Baseline	Scalability
Aslani et al. (2017)	next phase duration in cycle	-real-world traffic network -real-world traffic demand	yes	actuated traffic control	yes
Wei et al. (2018)	keep or change in cycle	-symmetric intersection -real-world traffic demand	no	SOTL (Cools et al., 2013)	no
Liang et al. (2019)	phase duration in cycle	-symmetric intersection -no real-world traffic demand	no	ATSC (Pandit et al., 2013)	no
Wu et al. (2020)	keep or change in cycle	-artificial traffic network -no real-world traffic demand	yes	SOTL (Cools et al., 2013)	yes
C. Chen et al. (2020)	choose next phase	-real-world traffic network -real-world traffic demand	no	max pressure (Varaiya, 2013)	yes
Chu et al. (2020)	choose next phase	-real-world traffic network -real-world traffic demand	no	greedy	yes
Z. Li et al. (2021)	keep or change in cycle	-real-world traffic network -real-world traffic demand	no	max pressure (Varaiya, 2013)	yes
Müller et al. (2021)	choose next phase	-real-world intersection -real-world traffic demand	yes	greedy	no
Bouktif et al. (2021)	choose next phase and duration	-symmetric intersection -no real-world traffic demand	no	fixed-time	no

The abbreviated action definitions hereby indicate:

Phase duration in cycle: The agent chooses N continuous values after each traffic cycle which defines the length of the N different phases.

Next phase duration in cycle: The agent decides after each phase within the cycle how long the next phase should be active.

Keep or change in cycle: The agent decides if the current phase within the cycle should be changed or extended by Δt.

Choose next phase: The agent chooses one of the N phases which will be held for a fixed duration Δt.

Choose next phase and duration: The agent chooses one of the N phases which will be held for duration Δt, also chosen by the agent.

Figure 1. Simplified scheme of the time gap control.

Figure 2. Full concept for the RL training based on real-world data and transfer to the real-world application based on DRIVERS.

Figure 3. Traffic network of Ingolstadt together with the corresponding microscopic traffic simulation and estimation representation by DRIVERS.

Table 2. Mapping of concept components to the issues discussed in Section 3.

Issue	Concept Component	Description
Real-world traffic	DRIVERS-OD, Simulation	The system is trained in a simulated real-world traffic network and supplied with real-world traffic demand by DRIVERS OD.
Multimodality	Feature Extractor, RL-System	The Feature Extractor includes multimodal traffic within its state and reward definition. Through the optimization of the traffic signals on a parameter level, existing control mechanisms for pedestrians, cyclists, as well as public transport acceleration, will be inherited.
Uncertainty in traffic state	DRIVERS	The traffic estimation model DRIVERS generates a microscopic traffic simulation from sparse real-world traffic detectors with proven correlation to the real traffic used to generate the traffic state for the RL agents.
Action space	RL-System	The chosen action definition can directly be incorporated into the existing traffic control units via existing signal messages from a central traffic server without changes in the traffic control or communication framework.
Robust policy	RL-System	The constrained action space can ensure that agents achieve a minimum level of performance in all situations. Even completely unknown traffic scenarios do not lead to a full failure of the traffic system due to the lower and upper bounds of the T-times defined by traffic engineers. Thereby it can be guaranteed, that the traffic network has no critical failure.
Safety constraints	RL-System	Safety is secured by the traffic engineer that planned the traffic light control since the agent optimizes the traffic signals only on a parameter level.
Baselines	Emulated Real Traffic Controller	The framework is compared to the already existing T-time-based traffic-actuated control scheme of the road network which is optimized by traffic engineers using state-of-the-art methods. Thus, the already carefully chosen T-times are used within the same traffic-actuated control scheme which is used as well by the agent. The agent thereby is just adapting the T-times during runtime. This results in a competitive baseline already manually optimized for the respective intersections.
Real-world training	Core RL-System	Training of the system in simulation and deployment to reality is enabled through the Core RL-System since it is the central unit used with identical functionality in both systems. As it has the same interfaces to both environments, a seamless integration and transfer from training to real deployment can be achieved. In addition, offline training could be supported, as the real-world can be used togehter with DRIVERS OD to generate further training data.
Scalability	RL-System	The setup of the RL-System allows the training and deployment to several intersections and complete traffic networks. Through MARL techniques joint and even asynchronous optimization can also be trained.

Figure 4. First training scheme for training based on real-world data and real-world compliant control concept.

Figure 5. Optimized agent performance at a single crossing. Blue and red lines indicate agent and benchmark rewards, respectively, with corresponding standard deviations shown in shaded areas.

Figure 6. Comparison of mean, median, upper and lower quantiles, standard deviation, and outliers between the benchmark and the agent across all data points. The dotted lines represent the mean, +1 standard deviation, and −1 standard deviation.

Figure 7. Comparison of agent-based traffic actuated control improvements in percentage relative to real-world traffic actuated control with default T-times. Median improvements are indicated with a solid line, while mean improvements are indicated with a horizontal dotted line. The dotted triangular lines represent +1 standard deviation and −1 standard deviation.

Figure 8. Visual comparison between the benchmark behavior and the agent’s behavior on a simulative example traffic volume.

Table 3. Definition of mathematical symbols used in this work.

Symbol	Description
$A^k$	Set of actions for crossing $\mathit{k}$
${\mathit{T}}_{\min }\mathit{ik}$	Minimal T-time for crossing $\mathit{k}$ and phase transition $\mathit{i}$
${\mathit{T}}_{\max }\mathit{ik}$	Maximal T-time for crossing $\mathit{k}$ and phase transition $\mathit{i}$
$\mathit{a}\mathrm{\_}\mathrm{out}\mathit{i},\mathit{max}\mathit{k}$	Neural Network output for crossing $\mathit{k}$ and phase transition $\mathit{i}$ corresponding to the upper T-time $\in [-1,1]$
$\mathit{a}\mathrm{\_}\mathrm{out}\mathit{i},\mathit{min}\mathit{k}$	Neural Network output for crossing $\mathit{k}$ and phase transition $\mathit{i}$ corresponding to the lower T-time $\in [-1,1]$
${\mathit{T}}_{\min \_\mathrm{adm}}\mathit{ik}$	Minimum admissible T-time for crossing $\mathit{k}$ and phase transition $\mathit{i}$
${\mathit{T}}_{\max \_\mathrm{adm}}\mathit{ik}$	Maximum admissible T-time for crossing $\mathit{k}$ and phase transition $\mathit{i}$
$\mathit{aik}$	Action for crossing crossing $\mathit{k}$ and phase transition $\mathit{i}$ $\in [{\mathit{T}}_{\min \_\mathrm{adm}}\mathit{ik},{\mathit{T}}_{\max \_\mathrm{adm}}\mathit{ik}]$
PI	performance index metric defined in Brilon (2015)
$G_w$	Weighting factor for the waiting time within the performance index
$G_H$	Weighting factor for the number of stops within the performance index
$\mathit{q}$	Traffic volume
$\mathit{H}$	Average number of stops of approaching vehicles
$t_w$	Average waiting time of approaching vehicles

Aslani, M., Mesgari, M. S., & Wiering, M. (2017). Adaptive traffic signal control with actor- critic methods in a real-world traffic network with different traffic disruption events. Transportation Research Part C: Emerging Technologies, 85, 732–752. https://www.sciencedirect.com/science/article/pii/S0968090X17302681

 Web of Science ®Google Scholar

Wei, H., Zheng, G., Yao, H., & Li, Z. (2018). Intellilight: A reinforcement learning approach for intelligent traffic light control. In Proceedings of the 24th ACM Sigkdd International Conference on Knowledge Discovery & Data mining, London, UK (pp. 2496–2505).

 Google Scholar

Cools, S.-B., Gershenson, C., & D’Hooghe, B. (2013). Self-Organizing Traffic Lights: A Realistic Simulation. Advances in Applied Self-Organizing Systems (pp. 45–55). London: Springer. https://doi.org/10.1007/978-1-4471-5113-5_3

 Google Scholar

Liang, X., Du, X., Wang, G., & Han, Z. (2019). A deep reinforcement learning network for traffic light cycle control. IEEE Transactions on Vehicular Technology, 68(2), 1243–1253. https://doi.org/10.1109/TVT.2018.2890726

 Web of Science ®Google Scholar

Pandit, K., Ghosal, D., Zhang, H. M., & Chuah, C.-N. (2013). Adaptive traffic signal control with vehicular ad hoc networks. IEEE Transactions on Vehicular Technology, 62(4), 1459–1471. https://doi.org/10.1109/TVT.2013.2241460

 Web of Science ®Google Scholar

Wu, T., Zhou, P., Liu, K., Yuan, Y., Wang, X., Huang, H., & Wu, D. O. (2020). Multi-agent deep reinforcement learning for urban traffic light control in vehicular networks. IEEE Transactions on Vehicular Technology, 69(8), 8243–8256. https://doi.org/10.1109/TVT.2020.2997896

 Web of Science ®Google Scholar

Chen, C., Wei, H., Xu, N., Zheng, G., Yang, M., Xiong, Y., Xu, K., and Li, Z. (2020). Toward a thousand lights: Decentralized deep reinforcement learning for large-scale traffic signal control. In Proceedings of the Aaai Conference on Artificial Intelligence, New York, USA (Vol. 34., pp. 3414–3421).

 Google Scholar

Varaiya, P. (2013). Max pressure control of a network of signalized intersections. Transportation Research Part C: Emerging Technologies, 36, 177–195. https://doi.org/10.1016/j.trc.2013.08.014

 Web of Science ®Google Scholar

Chu, T., Wang, J., Codecà, L., & Li, Z. (2020). Multi-agent deep reinforcement learning for large-scale traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 21(3), 1086–1095. https://doi.org/10.1109/TITS.2019.2901791

 Web of Science ®Google Scholar

Li, Z., Yu, H., Zhang, G., Dong, S., & Xu, C.-Z. (2021). Network-wide traffic signal control optimization using a multi-agent deep reinforcement learning. ArXiv , abs/Transportation Research Part C: Emerging Technologies, 2104(9936), 103059. https://doi.org/10.1016/j.trc.2021.103059

 Google Scholar

Müller, A., Rangras, V., Ferfers, T., Hufen, F., Schreckenberg, L., Jasperneite, J., Schnittker, G., Waldmann, M., Friesen, M., and Wiering, M. (2021). Towards real-world deployment of reinforcement learning for traffic signal control. 2021 20th IEEE International Conference on Machine Learning and Applications (ICMLA), Pasadena, USA (pp. 507–514).

 Google Scholar

Bouktif, S., Cheniki, A., & Ouni, A. (2021). Traffic signal control using hybrid action space deep reinforcement learning. Sensors, 21(7), 2302. https://doi.org/10.3390/s21072302

 PubMed Web of Science ®Google Scholar

Brilon, W. (2015). Handbuch für die bemessung von straßenverkehrsanlagen. FGSV.

 Google Scholar