On-policy learning-based deep reinforcement learning assessment for building control efficiency and stability

References

• Andersson, C., J. Åkesson, and C. Führer. 2016. Pyfmi: A python package for simulation of coupled dynamic models with the functional mock-up interface. Lund, Sweden: Centre for Mathematical Sciences, Lund University.
   Google Scholar
• ASHRAE Guideline 36P. 2018. ASHRAE guideline 36P: High performance sequences of operation for HVAC systems. Standard. Atlanta GA: American Society of Heating, Refrigerating and Air-Conditioning Engineers.
   Google Scholar
• Chen, B., Z. Cai, and M. Bergés. 2019. Gnu-rl: A precocial reinforcement learning solution for building hvac control using a differentiable mpc policy. In Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, 316–25.
   Google Scholar
• Crawley, D. B., L. K. Lawrie, F. C. Winkelmann, W. F. Buhl, Y. Huang, C. O. Pedersen, R. K. Strand, R. J. Liesen, D. E. Fisher, M. J. Witte, et al. 2001. EnergyPlus: Creating a new-generation building energy simulation program. Energy and Buildings 33 (4):319–31. doi:10.1016/S0378-7788(00)00114-6
   Web of Science ®Google Scholar
• Deru, M., K. Field, D. Studer, K. Benne, B. Griffith, P. Torcellini, B. Liu, et al. 2011. US Department of Energy commercial reference building models of the national building stock.
   Google Scholar
• Ding, X., Du, W., and A. Cerpa. 2019. Octopus: Deep reinforcement learning for holistic smart building control. In Proceedings of the 6th ACM International Conference on Systems for Energy-Efficient Buildings, Cities, and Transportation, 326–35.
   Google Scholar
• Drgoňa, J., D. Picard, and L. Helsen. 2020. Cloud-based implementation of white-box model predictive control for a GEOTABS office building: A field test demonstration. Journal of Process Control 88:63–77. doi:10.1016/j.jprocont.2020.02.007
   Web of Science ®Google Scholar
• Drgoňa, J., J. Arroyo, I. C. Figueroa, D. Blum, K. Arendt, D. Kim, E. Perarnau Ollé, et al. 2020. All you need to know about model predictive control for buildings. Annual Reviews in Control 50:190–232.
   Web of Science ®Google Scholar
• Goel, S., M. Rosenberg, R. Athalye, Y. Xie, W. Wang, R. Hart, J. Zhang, et al. 2014. Enhancements to ASHRAE standard 90.1 prototype building models. Technical Report. Pacific Northwest National Laboratory, Richland, WA.
   Google Scholar
• Han, M., R. May, X. Zhang, X. Wang, S. Pan, D. Yan, Y. Jin, and L. Xu. 2019. A review of reinforcement learning methodologies for controlling occupant comfort in buildings. Sustainable Cities and Society 51:101748. http://www.sciencedirect.com/science/article/pii/S2210670719307589. doi:10.1016/j.scs.2019.101748
   Web of Science ®Google Scholar
• Hill, A., A. Raffin, M. Ernestus, A. Gleave, A. Kanervisto, R. Traore, P. Dhariwal, et al. 2018. Stable baselines. https://github.com/hill-a/stable-baselines.
   Google Scholar
• Huang, S., Y. Lin, V. Chinde, X. Ma, and J. Lian. 2021. Simulation-based performance evaluation of model predictive control for building energy systems. Applied Energy 281:116027. doi:10.1016/j.apenergy.2020.116027
   Web of Science ®Google Scholar
• Ibarz, J., J. Tan, C. Finn, M. Kalakrishnan, P. Pastor, and S. Levine. 2021. How to train your robot with deep reinforcement learning: Lessons we have learned. The International Journal of Robotics Research 40 (4–5):698–721. doi:10.1177/0278364920987859
   Web of Science ®Google Scholar
• “International Energy Outlook 2019.” 2019. https://www.eia.gov/outlooks/archive/ieo19/pdf/ieo2019.pdf.
   Google Scholar
• Jiru, T. E. 2014. Combining HVAC energy conservation measures to achieve energy savings over standard requirements. Energy and Buildings 73:171–5. doi:10.1016/j.enbuild.2014.01.009
   Web of Science ®Google Scholar
• Kingma, D. P., and J. Ba. 2014. Adam: A method for stochastic optimization. arXiv Preprint arXiv:1412.6980.
   Google Scholar
• Kiran, B. R., I. Sobh, V. Talpaert, P. Mannion, A. A. A. Sallab, S. Yogamani, and P. Perez. 2022. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems 23 (6):4909–26. doi:10.1109/TITS.2021.3054625
   Web of Science ®Google Scholar
• Lample, G., and D. S. Chaplot. 2017. Playing FPS games with deep reinforcement learning. In 31st AAAI Conference on Artificial Intelligence. doi:10.1609/aaai.v31i1.10827
   Google Scholar
• Lee, J.-Y., S. Huang, A. Rahman, A. D. Smith, and S. Katipamula. 2020. Flexible reinforcement learning framework for building control using EnergyPlus-Modelica energy models. In Proceedings of the 1st International Workshop on Reinforcement Learning for Energy Management in Buildings & Cities, 34–38.
   Google Scholar
• Li, Y. 2017. Deep reinforcement learning: An overview. arXiv Preprint arXiv:1701.07274.
   Google Scholar
• Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2015. Continuous control with deep reinforcement learning. arXiv Preprint arXiv:1509.02971.
   Google Scholar
• Mnih, V., A. Puigdomenech Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. 2016. Asynchronous methods for deep reinforcement learning. In International Conference on Machine Learning, 1928–1937. PMLR.
   Google Scholar
• Mnih, V., K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al. 2015. Human-level control through deep reinforcement learning. Nature 518 (7540):529–33. doi:10.1038/nature14236
   PubMed Web of Science ®Google Scholar
• Mocanu, E., D. C. Mocanu, P. H. Nguyen, A. Liotta, M. E. Webber, M. Gibescu, and J. G. Slootweg. 2019. On-line building energy optimization using deep reinforcement learning. IEEE Transactions on Smart Grid 10 (4):3698–708. doi:10.1109/TSG.2018.2834219
   Web of Science ®Google Scholar
• Nouidui, T., M. Wetter, and W. Zuo. 2014. Functional mock-up unit for co-simulation import in EnergyPlus. Journal of Building Performance Simulation 7 (3):192–202. doi:10.1080/19401493.2013.808265
   Web of Science ®Google Scholar
• Schreiber, T., S. Eschweiler, M. Baranski, and D. Müller. 2020. Application of two promising Reinforcement Learning algorithms for load shifting in a cooling supply system. Energy and Buildings 229:110490. doi:10.1016/j.enbuild.2020.110490
   Web of Science ®Google Scholar
• Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. Proximal policy optimization algorithms. arXiv Preprint arXiv:1707.06347.
   Google Scholar
• Schulman, J., S. Levine, P. Abbeel, M. Jordan, and P. Moritz. 2015. Trust region policy optimization. In International Conference on Machine Learning, 1889–1897. PMLR.
   Google Scholar
• Silver, D., A. Huang, C. J. Maddison, A. Guez, L. Sifre, G. van den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, M. Lanctot, et al. 2016. Mastering the game of Go with deep neural networks and tree search. Nature 529 (7587):484–9. doi:10.1038/nature16961
   PubMed Web of Science ®Google Scholar
• Silver, D., G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. 2014. Deterministic policy gradient algorithms. In International Conference on Machine Learning, 387–395. PMLR.
   Google Scholar
• Silver, D., J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, et al. 2017. Mastering the game of go without human knowledge. Nature 550 (7676):354–9. doi:10.1038/nature24270
   PubMed Web of Science ®Google Scholar
• Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning: An introduction. MIT press.
   Google Scholar
• “Tracking Buildings 2020.” 2020. https://www.iea.org/reports/tracking-buildings-2020.
   Google Scholar
• Vázquez-Canteli, J. R., and Z. Nagy. 2019. Reinforcement learning for demand response: A review of algorithms and modeling techniques. Applied Energy 235:1072–89. doi:10.1016/j.apenergy.2018.11.002
   Web of Science ®Google Scholar
• Verhelst, J., G. Van Ham, D. Saelens, and L. Helsen. 2017. Economic impact of persistent sensor and actuator faults in concrete core activated office buildings. Energy and Buildings 142:111–27. doi:10.1016/j.enbuild.2017.02.052
   Web of Science ®Google Scholar
• Vinyals, O., I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, et al. 2019. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575 (7782):350–4. doi:10.1038/s41586-019-1724-z
   PubMed Web of Science ®Google Scholar
• Wang, N., S. Goel, A. Makhmalbaf, and N. Long. 2018. Development of building energy asset rating using stock modelling in the USA. Journal of Building Performance Simulation 11 (1):4–18. doi:10.1080/19401493.2015.1134668
   Web of Science ®Google Scholar
• Wang, Z., and T. Hong. 2020. Reinforcement learning for building controls: The opportunities and challenges. Applied Energy 269:115036. doi:10.1016/j.apenergy.2020.115036
   Web of Science ®Google Scholar
• Wei, T., Y. Wang, and Q. Zhu. 2017. Deep reinforcement learning for building HVAC control. In Proceedings of the 54th Annual Design Automation Conference 2017, 1–6.
   Google Scholar
• Zhang, Z., A. Chong, Y. Pan, C. Zhang, and K. P. Lam. 2019. Whole building energy model for HVAC optimal control: A practical framework based on deep reinforcement learning. Energy and Buildings 199:472–90. doi:10.1016/j.enbuild.2019.07.029
   Web of Science ®Google Scholar
• Zhao, J., B. Lasternas, K. P. Lam, R. Yun, and V. Loftness. 2014. Occupant behavior and schedule modeling for building energy simulation through office appliance power consumption data mining. Energy and Buildings 82:341–55. doi:10.1016/j.enbuild.2014.07.033
   Web of Science ®Google Scholar