A Reinforcement Learning Based Decision Support Tool for Epidemic Control: Validation Study for COVID-19

References

• Achaiah, N. C., S. B. Subbarajasetty, and R. M. Shetty. 2020. R0 and Re of COVID-19: Can We Predict When the Pandemic Outbreak will be Contained? Indian Journal of Critical Care Medicine 24 (11): 1125–1908. doi:10.5005/jp-journals-10071-23649.
   PubMed Web of Science ®Google Scholar
• Arango, M., and L. Pelov. 2020. COVID-19 Pandemic Cyclic Lockdown Optimization Using Reinforcement Learning. arango2020covid19. http://arxiv.org/abs/2009.04647
   Google Scholar
• Askitas, N., K. Tatsiramos, and B. Verheyden. 2021. Estimating worldwide effects of non-pharmaceutical interventions on COVID-19 incidence and population mobility patterns using a multiple-event study. Scientific Reports 11 (1):1–13. doi:10.1038/s41598-021-81442-x.
   PubMed Web of Science ®Google Scholar
• Bergwerk, M., T. Gonen, Y. Lustig, S. Amit, M. Lipsitch, C. Cohen, M. Mandelboim, Levin, E. G., Rubin, C., Indenbaum, V., Tal, I., Zavitan, M., Zuckerman, N., Bar-Chaim, A., Kreiss, Y., Regev-Yochay, G. et al. 2021. Covid-19 breakthrough infections in vaccinated health care workers. New England Journal of Medicine 385 (16):1474–84. doi:10.1056/nejmoa2109072.
   PubMed Web of Science ®Google Scholar
• Bhadra, A., A. Mukherjee, and K. Sarkar. 2021. Impact of population density on covid-19 infected and mortality rate in India. Modeling Earth Systems and Environment 7 (1):623–29. doi:10.1007/s40808-020-00984-7.
   PubMed Web of Science ®Google Scholar
• Busoniu, L., R. Babuska, and B. De schutter. 2010. Chapter 7 multi-agent reinforcement learning : An overview. Technology 38 (2):183–221.
   Google Scholar
• Chinazzi, M., J. T. Davis, M. Ajelli, C. Gioannini, M. Litvinova, S. Merler, A. P. Y Piontti, Mu, K., Rossi, L., Sun, K., Viboud, C., Xiong, X., Yu, H., Halloran, M. E., Longini Jr, I. M., Vespignani, A., et al. 2020. The effect of travel restrictions on the spread of the 2019 novel coronavirus (covid-19) outbreak. Science 368 (6489):395–400. doi:10.1126/science.aba9757.
   PubMed Web of Science ®Google Scholar
• CHO, S. A. N. G. W. O. O. K. 2020. Quantifying the impact of nonpharmaceutical interventions during the covid-19 outbreak: The case of Sweden. Econometrics Journal 23 (3):323–44. doi:10.1093/ECTJ/UTAA025.
   Web of Science ®Google Scholar
• Cohen, K., and A. Leshem. 2021. Suppressing the impact of the covid-19 pandemic using controlled testing and isolation. Scientific Reports 11 (1):1–15. doi:10.1038/s41598-021-85458-1.
   PubMed Web of Science ®Google Scholar
• Corona, G., A. Pizzocaro, W. Vena, G. Rastrelli, F. Semeraro, A. M. Isidori, R. Pivonello, A. Salonia, A. Sforza, and M. Maggi. 2021. Diabetes is most important cause for mortality in covid-19 hospitalized patients: Systematic review and meta-analysis. Reviews in Endocrine & Metabolic Disorders 22 (2):275–96. doi:10.1007/s11154-021-09630-8.
   PubMed Web of Science ®Google Scholar
• Cotton, C., B. Crowley, B. Kashi, A. Huw Lloyd-Ellis, and F. Tremblay. 2020. “Quantifying the economic impacts of covid-19 policy responses on Canada's provinces in (almost) real time.” https://www.econ.queensu.ca/sites/econ.queensu.ca/files/wpaper/qed_wp_1441.pdf.
   Google Scholar
• Daoust, J.-F., R. Nadeau, R. Dassonneville, E. Lachapelle, É. Bélanger, J. Savoie, and C. van der Linden. 2020. How to survey citizens’ compliance with covid-19 public health measures: Evidence from three survey experiments. Journal of Experimental Political Science 1–8. doi:10.1017/xps.2020.25.
   Google Scholar
• Daron, A., V. Chernozhukov, I. Werning, and M. Whinston. 2020. A multi-risk SIR model with optimally targeted lockdown. NBER Working Paper Series 27102:1–39. https://mr-sir.herokuapp.com/.
   Google Scholar
• Deschepper, M., K. Eeckloo, S. Malfait, D. Benoit, S. Callens, and S. Vansteelandt. 2021. Prediction of hospital bed capacity during the COVID− 19 pandemic. BMC Health Services Research 21 (1):1–10. doi:10.1186/s12913-021-06492-3.
   PubMed Web of Science ®Google Scholar
• Eker, S. 2020. Validity and usefulness of COVID-19 models. Humanities and Social Sciences Communications 7 (1):1–5. doi:10.1057/s41599-020-00553-4.
   Google Scholar
• Ferguson, N., D. Laydon, Nedjati-Gilani, G., N. Imai, K. Ainslie, M. Baguelin, S. Bhatia, Boonyasiri, A., Cucunubá, Z., Cuomo-Dannenburg, G., Dighe, A., Dorigatti, I., Fu, H., Gaythorpe, K., Green , W., Hamlet, A., Hinsley, W., Okell, L. C, van Elsland, S., Thompson, H., Verity, R. et al. 2020. Impact of non-pharmaceutical interventions (NPIs) to reduce covid-19 mortality and healthcare demand. https://Www.Imperial.Ac.Uk/Media/Imperial-College/Medicine/Sph/Ide/Gida-Fellowships/Imperial-College-COVID19-NPI-Modelling-16-03-2020.Pdf. Imperial College COVID-19 Response Team. March. 1–20.
   Google Scholar
• Fujimoto, S., H. Van hoof, and D. Meger. 2018. “addressing function approximation error in actor-critic methods.” 35th International Conference on Machine Learning, ICML 2018, July 10th to July 15th, 2018, Stockholm, Sweden, 4, PMLR, 2587–601. https://proceedings.mlr.press/v80/fujimoto18a.html
   Google Scholar
• Giordano, G., F. Blanchini, R. Bruno, P. Colaneri, A. Di Filippo, A. Di Matteo, and M. Colaneri. 2020. Modelling the COVID-19 epidemic and implementation of population-wide interventions in Italy.Nature medicine 26 (6): 855–860. https://www.nature.com/articles/s41591-020-0883-7. doi:10.1038/s41591-020-0883-7.
   PubMed Web of Science ®Google Scholar
• Haddad, E. A., E. Karim, A. Abdelaaziz, A. Ali, M. Arbouch, and I. F. Araújo. 2020. The Impact Of Covid-19 In Morocco: Macroeconomic, Sectoral And Regional Effects. Rabat, Morocco: Policy Center for the New South. https://www.policycenter.ma/publications/impact-covid-19-morocco-macroeconomic-sectoral-and-regional-effects
   Google Scholar
• Hall, V. J., S. Foulkes, A. Charlett, A. Atti, E. J. M. Monk, R. Simmons, E. Wellington, J Cole, M., Saei, A., Oguti, B., Munro, K., Wallace, S., D Kirwan, P., Shrotri, M., Vusirikala, A., Rokadiya, S., Kall, M., Zambon, M., Ramsay, M., Brooks, T., S Brown, C. et al. 2021. SARS-CoV-2 infection rates of antibody-positive compared with antibody-negative health-care workers in england: A large, multicentre, prospective cohort study (SIREN). The Lancet 397 (10283):1459–69. doi:10.1016/S0140-6736(21)00675-9.
   PubMed Web of Science ®Google Scholar
• Harshad, K., T. Ganu, and D. P. Seetharam. 2020. Optimising lockdown policies for epidemic control using reinforcement learning. Transactions of the Indian National Academy of Engineering 5 (2):129–32. doi:10.1007/s41403-020-00129-3.
   Google Scholar
• HCP. 2021. “46_website HCP.” https://www.hcp.ma/Demographie-population_r142.html.
   Google Scholar
• Herik, H. Jaap, V. D., W. H. M. U. Jos, and J. Van Rijswijck. 2002. Games solved: Now and in the future. Artificial Intelligence 134 (1–2):277–311. doi:10.1016/S0004-3702(01)00152-7.
   Web of Science ®Google Scholar
• Ian, C., A. Mondal, and C. G. Antonopoulos. 2020. A SIR model assumption for the spread of COVID-19 in different communities. Chaos, Solitons, and Fractals 139:1–14. doi:10.1016/j.chaos.2020.110057.
   Web of Science ®Google Scholar
• IMF. 2020 October. The great lockdown: Dissecting the economic effects. World Economic Outlook. A Long and Difficult Ascent, pp. 65–84. International Monetary Fund. 978-1-51355-605-5.
   Google Scholar
• Jarvis, C. I., K. Van Zandvoort, A. Gimma, K. Prem, M. Auzenbergs, K. O’Reilly, G. Medley, Emery, J. C., Houben, R. M. G. J., Davies, N., Nightingale, E. S., Flasche, S., Jombart, T., Hellewell, J., Abbott, S., Munday, J. D., Bosse, N. I., Funk, S., Sun, F., Endo, A., Rosello, A. et al. 2020. Quantifying the impact of physical distance measures on the transmission of COVID-19 in the UK. BMC Medicine 18 (1):1. doi:10.1186/s12916-020-01597-8.
   PubMed Web of Science ®Google Scholar
• Jasper, V., E. E. Koks, and J. W. Hall. 2021. April. 4. Global economic impacts of covid-19 lockdown measures stand out in high frequency shipping data. PLoS ONE. 16:1–16. doi: 10.1371/journal.pone.0248818.
   Web of Science ®Google Scholar
• Kadanali, A., and G. Karagoz. 2016. An overview of Ebola virus disease. Northern Clinics of Istanbul 2 (1):81–86. doi:10.14744/nci.2015.97269.
   Google Scholar
• Kadi, N., and M. Khelfaoui. 2020. population density, a factor in the spread of covid-19 in Algeria: Statistic study. Bulletin of the National Research Centre 44 (1). doi: 10.1186/s42269-020-00393-x.
   PubMedGoogle Scholar
• Karin, O., Y. M. Bar-On, T. Milo, I. Katzir, A. Mayo, Y. Korem, B. Dudovich, Yashiv, E., Zehavi, A. J., Davidovitch, N., Milo, R., Alon, U., et al. 2020. cyclic exit strategies to suppress covid-19 and allow economic activity. MedRxiv 1–18. doi:10.1101/2020.04.04.20053579.
   Google Scholar
• Khalilpourazari, S., and H. Hashemi doulabi. 2021a. Designing a hybrid reinforcement learning based algorithm with application in prediction of the COVID-19 pandemic in quebec. Annals of Operations Research. Annals of Operations Research. doi:10.1007/s10479-020-03871-7.
   PubMed Web of Science ®Google Scholar
• Khalilpourazari, S., and H. Hashemi doulabi. 2021b. Robust modelling and prediction of the COVID-19 pandemic in Canada. International Journal of Production Research1–17. doi:10.1080/00207543.2021.1936261.
   Web of Science ®Google Scholar
• Khalilpourazari, S., H. Hashemi doulabi, A. Özyüksel çiftçioğlu, and G. Wilhelm weber. 2021. Gradient-based grey wolf optimizer with gaussian walk: Application in modelling and prediction of the COVID-19 pandemic. Expert Systems with Applications 177 (March):114920. doi:10.1016/j.eswa.2021.114920.
   PubMed Web of Science ®Google Scholar
• Khalilpourazari, Soheyl, and D. Hossein hashemi. 2021. “using reinforcement learning to forecast the spread of COVID-19 in France.” ICAS 2021 - 2021 IEEE International Conference on Autonomous Systems, Proceedings, France, IEEE. doi: 10.1109/ICAS49788.2021.9551174.
   Google Scholar
• Konda, V. R., and J. N. Tsitsiklis. 2000. Actor-critic algorithms. Neural Information Processing Systems, 2000 12, NIPS, USA. 1008–14. https://proceedings.neurips.cc/paper/1999/file/6449f44a102fde848669bdd9eb6b76fa-Paper.pdf.
   Google Scholar
• Leech, G., C. Rogers-Smith, J. Benjamin Sandbrink, B. Snodin, R. Zinkov, B. Rader, J. S. Brownstein, Gal, Y., Bhatt, S., Sharma, M., Mindermann, S., Brauner, J. M., Aitchison, L., et al. 2021. Mass mask-wearing notably reduces COVID-19 transmission. MedRxiv 1 (1):2021.06.16.21258817. http://medrxiv.org/content/early/2021/06/18/2021.06.16.21258817.abstract.
   Google Scholar
• Libin, P. J. K., A. Moonens, T. Verstraeten, F. Perez-Sanjines, N. Hens, P. Lemey, and A. Nowé. 2021. Deep reinforcement learning for large-scale epidemic control. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 12461 LNAI, 155–70. doi:10.1007/978-3-030-67670-4_10.
   Google Scholar
• Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2016. “Continuous control with deep reinforcement learning.” lillicrap2019continuous. https://arxiv.org/abs/1509.02971
   Google Scholar
• Martin, E., S. F. Dowell, and N. Firese. 2011. Incubation period of Ebola hemorrhagic virus subtype zaire. Osong Public Health and Research Perspectives 2 (1):3–7. doi:10.1016/j.phrp.2011.04.001.
   PubMedGoogle Scholar
• Masbah, M., and R. Aourraz. 2020. How Moroccans view the government ’ s measures? March: The Moroccan Institute for Policy Analysis. https://mipa.institute/7486
   Google Scholar
• Matheron, G., N. Perrin, and O. Sigaud. 2019. The problem with DDPG: understanding failures in deterministic environments with sparse rewards. matheron2019problem. http://arxiv.org/abs/1911.11679
   Google Scholar
• Mercer, T. R., and M. Salit. 2021. Testing at scale during the COVID-19 pandemic. Nature Reviews. Genetics 22 (7):415–26. doi:10.1038/s41576-021-00360-w.
   PubMed Web of Science ®Google Scholar
• Mnih, V., K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013. Playing Atari with Deep Reinforcement Learning. mnih2013playing. http://arxiv.org/abs/1312.5602
   Google Scholar
• Moein, S., N. Nickaeen, A. Roointan, N. Borhani, Z. Heidary, S. Haghjooy Javanmard, J. Ghaisari, and Y. Gheisari. 2021. Inefficiency of SIR models in forecasting COVID-19 epidemic: a case study of isfahan. Scientific Reports 11 (1):1–9. doi:10.1038/s41598-021-84055-6.
   PubMed Web of Science ®Google Scholar
• Moroccan Minstry of Health. 2020. “47_website covidmaroc evolution reports portail.” http://www.covidmaroc.ma/Pages/LESINFOAR.aspx.
   Google Scholar
• Murano, Y., R. Ueno, S. Shi, T. Kawashima, Y. Tanoue, S. Tanaka, S. Nomura, Shoji, H., Shimizu, T., Nguyen, H., Miyata, H., Gilmour, S., Yoneoka, D., et al. 2021. Impact of domestic travel restrictions on transmission of COVID-19 infection using public transportation network approach. Scientific Reports 11 (1):1–9. doi:10.1038/s41598-021-81806-3.
   PubMed Web of Science ®Google Scholar
• OECD. 2020. THE COVID-19 CRISIS IN Morocco as of may 6, 2020 COVID-19 update economic impact policy reactions. Organisation for Economic Co-operation and Development (OECD). https://www.oecd.org/mena/competitiveness/The-Covid-19-Crisis-in-Morocco.pdf
   Google Scholar
• Ohi, A. Q., M. F. Mridha, M. Mostafa Monowar, and M. Abdul Hamid. 2020. Exploring optimal control of epidemic spread using reinforcement learning. Scientific Reports 10 (1):1–19. doi:10.1038/s41598-020-79147-8.
   PubMed Web of Science ®Google Scholar
• Okhuese, A. V. 2020. Estimation of the probability of reinfection with COVID-19 by the susceptible-exposed-infectious-removed-undetectable-susceptible model. JMIR Public Health and Surveillance 6 (2):1–11. doi:10.2196/19097.
   Google Scholar
• Ontario Agency for Health Protection and Promotion (Public Health Ontario). 2021. economic impacts related to public health measures in response and recovery during the COVID-19 pandemic. Queen's Printer for Ontario. https://www.publichealthontario.ca/-/media/documents/ncov/phm/2021/03/eb-covid-19-economic-impacts.pdf?la=en
   Google Scholar
• Oraby, T., M. G. Tyshenko, J. Campo Maldonado, K. Vatcheva, S. Elsaadany, W. Q. Alali, J. C. Longenecker, and M. Al-Zoughool. 2021. Modeling the effect of lockdown timing as a COVID-19 control measure in countries with differing social contacts. Scientific Reports 11 (1):1–13. doi:10.1038/s41598-021-82873-2.
   PubMed Web of Science ®Google Scholar
• Padmanabhan, R., N. Meskin, T. Khattab, M. Shraim, and M. Al-Hitmi. 2021. reinforcement learning-based decision support system for COVID-19. Biomedical Signal Processing and Control 68:102676. doi:10.1016/j.bspc.2021.102676.
   PubMed Web of Science ®Google Scholar
• PERC. 2021a. Finding the balance: public health and social measures in ghana. Partnership for Evidence-Based Response to COVID-19. August 2020. 1–9. https://preventepidemics.org/wp-content/uploads/2021/03/ghana_en_20210323_1721.pdf.
   Google Scholar
• PERC. 2021b. Finding the Balance: Public Health and Social Measures in Morocco. Prevent Epidemics. https://preventepidemics.org/wp-content/uploads/2021/03/morocco_en_20210316_2047.pdf
   Google Scholar
• Prague, M., L. Wittkop, A. Collin, D. Dutartre, Q. Clairon, P. Moireau, R. Thiébaut, and B. Hejblum. 2020. Multi-level modeling of early COVID-19 epidemic dynamics in French regions and estimation of the lockdown impact on infection rate. medRxiv. doi:10.1101/2020.04.21.20073536.
   Google Scholar
• Probert, W. J. M., S. Lakkur, C. J. Fonnesbeck, K. Shea, M. C. Runge, M. J. Tildesley, and M. J. Ferrari. 2019. Context matters: Using reinforcement learning to develop human-readable, state-dependent outbreak response policies. Philosophical Transactions of the Royal Society B: Biological Sciences 374 (1776):20180277. doi:10.1098/rstb.2018.0277.
   PubMed Web of Science ®Google Scholar
• Quesada, J. A., A. López-Pineda, V. F. Gil-Guillén, J. M. Arriero-Marín, F. Gutiérrez, and C. Carratala-Munuera. 2021. Incubation period of covid-19: A systematic review and meta-analysis. Revista Clínica Española (English Edition) 221 (2):109–17. doi:10.1016/j.rceng.2020.08.002.
   Web of Science ®Google Scholar
• Rajgor, D. D., M. Har Lee, S. Archuleta, N. Bagdasarian, and S. Chye Quek. 2020. The many estimates of the COVID-19 case fatality rate. The Lancet Infectious Diseases 20 (7):776–77. doi:10.1016/S1473-3099(20)30244-9.
   PubMed Web of Science ®Google Scholar
• Richard, S., Sutton, and G. Barto Andrew. 2017. Reinforcement learning: An introduction. 2nd ed. Cambridge, Massachusetts London, England: The MIT Press. 978-0262039246.
   Google Scholar
• Santos, I. F. F. D., G. M. A. Almeida, and F. A. B. F. de Moura. 2021. Adaptive SIR model for propagation of SARS-CoV-2 in Brazil. Physica a: Statistical Mechanics and Its Applications 569:125773. doi:10.1016/j.physa.2021.125773.
   Web of Science ®Google Scholar
• Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. Proximal Policy Optimization Algorithms. schulman2017proximal. http://arxiv.org/abs/1707.06347
   Google Scholar
• Sen-Crowe, B., M. Sutherland, M. McKenney, and A. Elkbuli. 2021. A closer look into global hospital beds capacity and resource shortages during the COVID-19 pandemic. Journal of Surgical Research 260:56–63. doi:10.1016/j.jss.2020.11.062.
   PubMed Web of Science ®Google Scholar
• Silver, D., J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., van den Driessche, G., Graepel, T., Hassabis, D., et al. 2017. Mastering the game of go without human knowledge. Nature 550 (7676):354–59. doi:10.1038/nature24270.
   PubMed Web of Science ®Google Scholar
• Sun, J., X. Chen, Z. Zhang, S. Lai, B. Zhao, H. Liu, S. Wang, Huan, W., Zhao, R., Ng, M. T. A., Zheng, Y., et al. 2020. Forecasting the long-term trend of COVID-19 epidemic using a dynamic model. Scientific Reports 10 (1):1–10. doi:10.1038/s41598-020-78084-w.
   PubMed Web of Science ®Google Scholar
• Sun, P., X. Sun, L. Han, J. Xiong, Q. Wang, B. Li, Y. Zheng, Liu, J., Liu, Y., Liu, H., Zhang, T., et al. 2018. TStarBots: Defeating the Cheating Level Builtin AI in StarCraft II in the Full Game. sun2018tstarbots. http://arxiv.org/abs/1809.07193
   Google Scholar
• Sutton, R. S., D. McAllester, S. Singh, and Y. Mansour. 1996. Policy gradient methods for reinforcement learning with function approximation Advances in Neural Information Processing Systems 12 (NIPS 1999). 12. MIT Press. https://proceedings.neurips.cc/paper/1999/file/464d828b85b0bed98e80ade0a5c43b0f-Paper.pdf
   Google Scholar
• WHO. 2020. “who coronavirus (covid-19) dashboard | who coronavirus (COVID-19) dashboard with vaccination data 2020.”https://covid19.who.int/.
   Google Scholar
• Woolcott, O. O., and J. P. Castilla-Bancayán. 2021. The effect of age on the association between diabetes and mortality in adult patients with COVID-19 in Mexico. Scientific Reports 11 (1):1–10. doi:10.1038/s41598-021-88014-z.
   PubMedGoogle Scholar
• World Bank Group. 2020. Morocco economic monitor, fall 2020. Morocco Economic Monitor, Fall. doi:10.1596/34976.
   Google Scholar
• World Health Organization. 2015. Anticipating emerging infectious disease epidemics. World Health Organization. https://apps.who.int/iris/bitstream/handle/10665/252646/WHO-OHE-PED-2016.2-eng.pdf
   Google Scholar