A Reinforcement Learning Based Decision Support Tool for Epidemic Control: Validation Study for COVID-19

Figures & data

Figure 1. Population class model for the developed EM.

Table 1. Summary of the variables and parameters of the developed EM

Variables	Description
Cur Inf	Current infections in the studied region, initialized by the user at the first step, then changed automatically according to the dynamics of the simulation.
N	The population size of the studied region, initialized by the user at the first step, then changed automatically according to the dynamics of the simulation.
First Inf	The number of infected individuals for the first time, computed during the simulation.
Re Inf	The number of infected individuals after recovery (for the second time), computed during the simulation.
Nxt Inf	The number of next infection cases, (which is the sum of First Inf and Re Inf), computed during the simulation.
K	The number of known carriers, (those that were tested positive and hospitalized), computed during the simulation.
U	The number of unknown carriers, (those that neither showed symptoms nor have been tested), computed during the simulation.
R	The number of recovered cases, computed during the simulation.
D	The number of death cases, computed during the simulation.
Parameters	Description
R0	The reproduction rate of the studied epidemic. Varies between 1.4 and 2.4.
Incubation	The incubation period of the studied epidemic. Varies between 5.6 and 12.5 days.
Fatality	The fatality rate of the studied epidemic when no healthcare treatment is taken. Varies between 0.3% and 4.3%.
Reinfection	The reinfection probability of the studied epidemic, fixed at 0.16.
Density	The density of the studied region (in people/km^2), defined by the user.
S1	The transmission rate, defined in equation (1)(1) $s1=\left\{\begin{array}{c}\left[\left(1-\frac{A1+A2+A3+A4}{4}\right)\ast Density\ast \left(R0+\frac{CurInf}{N}\right)\right]-\left(A7\ast 0.61\right),ifV=1\\ \left[\left(1-\frac{A1+A2+A3+A4}{4}\right)\ast Density\ast R0\ast \frac{CurInf}{N}\right],ifV=0\end{array}\right.$(1) .
S2	The identification rate, defined in equation (2)(2) $\mathrm{s}2=A5\ast \left(1-incubationperiod\right)$(2) .
S3	The death rate, defined in equation (3)(3) $\mathrm{s}3=\left(1-A6\right)\ast fatalityrate$(3) .
S4	The reinfection rate, defined in equation (4)(4) $\mathrm{s}4=\left(1-\left(A7\ast 0.61\right)\right)\ast Probabilityofreinfection$(4) .

Table 2. Summary of the result of the validation according to the population size and density for the 10 Moroccan cities

City	Population size	Density ppl/km2	Corr Inf/day	P-value Inf/day	Corr Death/day	P-value Death/day
Casablanca	3,535,127	11,380	0.758	1.68e-40	0.462	1.51e-12
Marrakech	1,393,206	6521	0.708	4.54e-32	0.422	1.63e-10
Ouarzazat	297,502	24	0.540	1.17e-10	0.203	0.072
Rabat	544,422	4853	0.750	2.43e-39	0.205	0.002
Tetouane	550,374	216.6	0.609	8.66e-23	0.261	0.001
O.Eddahab	126,765	1.82	0.372	6.34e-05	0.244	0.040
Fes	1,232,798	657.3	0.578	3.80e-20	0.175	0.190
Meknes	891,155	457.3	0.555	2.13e-18	0.220	0.201
Laayoune	256,482	5.3	0.559	1.08e-18	0.207	0.195
Tanger	1,188,815	7026	0.651	3.66e-14	0.324	0.033

Figure 2. The actual number of infections versus the predicted number of infections per day on top of each subfigure. The actual number of deaths versus the predicted number of deaths per day on the bottom of each subfigure, in the ten Moroccan cities.

Figure 3. Episodic reward and variance monitoring for the three models for different health score thresholds.

Figure 4. PPO performance against different economic thresholds (Te).

Figure 5. The effect of the economic score weights on the performance and policy.

Figure 6. The effect of stochasticity on the performance of the PPO model.

Figure 7. The performance of the DDPG, TD3, and PPO on the easier reward function.

Table 3. Timeline of the interventions and relaxation implemented in Morocco

Intervention	History of the related measures implemented
A1: travel restriction	In September 2020, international and national travel restriction were imposed. in October and November, international borders were opened with condition (i.e., after having permission papers such as negative COVID-19 test) and some local transportations were allowed in low-risk zones only. In December, because of last year’s holidays, internal travel restrictions and conditioned permission of international travel (i.e., after having permission papers such as negative COVID-19 test). This policy has been repeatedly extended in January, February, and March 2021.
A2: Lockdown	In September 2020, Curfew from 6pm to 5am was imposed, only some mosques were allowed to open (i.e., the biggest mosques in districts). In December, national curfew was implemented, and restaurants and cafes were closed. This policy has been repeatedly extended in January, February, and March 2021.
A3: Distance work & education	In September 2020, schools opened for registration in low-risk zones and in high-risk zones in the second half of October. Most businesses worked at distance. Restaurant were working only via delivery and a lot of other public businesses were closed such as Hammams, clubs, etc. In October and November, schools and universities worked presently in low-risk zones and distantly in high-risk zones. This policy has been repeatedly extended in January, February, and March 2021.
A4: Mask imposing	In September 2020, a lawsuit of 300MAD was imposed for non-wearing masks. A survey by (PERC 2021b) has reported that 94% of participants wore their masks in public. However, in October and November, restrictions related to this measure were weakened and the adherence of citizens started decreasing. As the wave got to its peak in December, strict campaigns by the government were taking place and the lawsuit was strictly applied. This policy has been repeatedly extended in January, February, and March 2021.
A5: Test rate	In September 2020, the number of tests per day was relatively sufficient compared to the number of infections which was relatively low. However, as the wave started to peak in November, the number of tests per day was not sufficient yielding more and more infections as more unknown carriers were not isolated. This motivated the government to buy more test kits. Morocco had sufficient test kits that allowed it to conduct more than 200,000 tests per day from late December to March.
A6: Healthcare capacity	In September 2020, the death rate was relatively low and hospital beds were sufficient. As the wave started to peak in November, the death rate increased proportionally. The government invested in the healthcare capacity in several ways: buying/providing more intensive care unit beds, buying more respiratory equipment, transforming other public places to public hospitals dedicated for COVID patients only. This policy has been extended to January, February, and March 2021.
A7: Vaccination	No vaccination was implemented in this period.

Figure 8. Optimal hyperparameter sets for the three models used (DDPG, TD3, and PPO from left to right respectively).

Table 4. Hyperparameters tested for each deep RL model

Algorithm	Hyperparameter	Description
DDPG	Gamma	Specifies the discount factor for computing the return in the Bellman equation (Richard, Sutton, and Barto Andrew 2017)
	Tau	Specifies the soft update coefficient, between 0 and 1 (Lillicrap et al. 2016)
	Sigma	Specifies the standard deviation of the action noise used, that is, the Ornstein Uhlenbeck distribution noise. High values for sigma mean more exploration (Lillicrap et al. 2016)
TD3	Gamma	Specifies the discount factor for computing the return in the Bellman equation (Richard, Sutton, and Barto Andrew 2017)
	Tau	Specifies the soft update coefficient for the Polyak update, between 0 and 1 (Lillicrap et al. 2016)
	Sigma	Specifies the standard deviation of the action noise used, that is, the Gaussian distribution noise. High values for sigma mean more exploration (Fujimoto, Van Hoof, and Meger 2018)
	Policy delay	Specifies the frequency of updating the policy network compared to the update of the value network (Fujimoto, Van Hoof, and Meger 2018)
PPO	Gamma	Specifies the discount factor for computing the return in the Bellman equation (Richard, Sutton, and Barto Andrew 2017)
	GAE lambda	Specifies the trade-off of bias versus variance for the Generalized Advantage Estimator (Schulman et al. 2017)
	Clip range	Specifies how far can the new policy go after the update from the old policy. It is used to prevent large updates, thus it is usually a small number between 0.1 and 0.3 (Schulman et al. 2017)

PERC. 2021b. Finding the Balance: Public Health and Social Measures in Morocco. Prevent Epidemics. https://preventepidemics.org/wp-content/uploads/2021/03/morocco_en_20210316_2047.pdf

 Google Scholar

Richard, S., Sutton, and G. Barto Andrew. 2017. Reinforcement learning: An introduction. 2nd ed. Cambridge, Massachusetts London, England: The MIT Press. 978-0262039246.

 Google Scholar

Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2016. “Continuous control with deep reinforcement learning.” lillicrap2019continuous. https://arxiv.org/abs/1509.02971

 Google Scholar

Fujimoto, S., H. Van hoof, and D. Meger. 2018. “addressing function approximation error in actor-critic methods.” 35th International Conference on Machine Learning, ICML 2018, July 10th to July 15th, 2018, Stockholm, Sweden, 4, PMLR, 2587–601. https://proceedings.mlr.press/v80/fujimoto18a.html

 Google Scholar

Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. Proximal Policy Optimization Algorithms. schulman2017proximal. http://arxiv.org/abs/1707.06347

 Google Scholar