برنامه‌ریزی بهره‌برداری ریزشبکه‌ها مبتنی بر الگوریتم یادگیری تقویتی عمیق

نویسندگان

1 دانشگاه علم و صنعت ایران، تهران، ایران

2 دانشگاه علوم و فنون هوایی شهید ستاری

3 استادیار گروه مهندسی برق، دانشگاه فنی و حرفه‌ای

چکیده

در این مقاله، برنامه‌ریزی بهره‌برداری ریزشبکه‌ها مشتمل بر منابع تولید انرژی و سیستم‌های ذخیره انرژی مبتنی بر یادگیری تقویتی عمیق ارائه شده است. با توجه به خاصیت پویایی مسئله، ابتدا در قالب یک فرایند تصمیم‌گیری مارکوف متشکل از چهارتایی (حالت، اقدام، تابع احتمال انتقال و پاداش) فرمول‌بندی شده است. سپس، الگوریتم گرادیان استراتژی قطعی عمیق به‌منظور یادگیری استراتژی بهینۀ برنامه‌ریزی بهره‌برداری ریزشبکه با هدف کمینه کردن هزینه‌های بهره‌برداری ارائه شده است. این الگوریتم یک روش بی‌نیاز از مدل، مستقل از استراتژی و بر مبنای معماری عامل-نقاد است که می‌تواند به‌خوبی فضای حالت و اقدام مسئله را به‌صورت پیوسته مدل‌سازی و بر چالش بزرگ بودن ابعاد مسئله غلبه کند. به‌منظور ارزیابی الگوریتم ارائه‌شده، نتایج با الگوریتم یادگیری Q عمیق و روش تحلیلی مقایسه شد. نتایج حاصل از شبیه‌سازی، کارایی الگوریتم گرادیان استراتژی قطعی عمیق ارائه‌شده را از جهت همگرایی، زمان اجرا و هزینۀ کل نشان دادند. 

کلیدواژه‌ها


[1] Amini Badri A, Taghizadegan Kalantari N. Reliability Evaluation of Active Radial Distribution Systems Based on an Improved Classification Algorithm. JEM., Vol.10, No. 3, pp. 2-11, 2020 [2] Shahgholian G, Fani B, Moazzami M, Keyvani B, Karimi H. An Improvement inthe Reactive Power Sharing by the Use of Modified Droop Characteristics in Autonomous Microgrids. JEM.; Vol.9, No. 3, pp. 64-71, 2019 [3] Esmaeili, S., Anvari-Moghaddam, A. and Jadid, S., "Retail market equilibrium and interactions among reconfigurable networked microgrids", Sustainable Cities and Society, Vol.49, pp.101628, 2019 [4] Ahmad, S., Alhaisoni, M.M., Naeem, M., Ahmad, A. and Altaf, M., "Joint energy management and energy trading in residential microgrid system", IEEE Access, Vol.8, pp.123334-123346, 2020. [5] Zhu, J., Yuan, Y. and Wang, W., "An exact microgrid formation model for load restoration in resilient distribution system", International Journal of Electrical Power & Energy Systems, Vol.116, p.105568, 2020. [6] Huang, Z. and Dinavahi, V., "An efficient hierarchical zonal method for large-scale circuit simulation and its real-time application on more electric aircraft microgrid", IEEE Transactions on Industrial Electronics, Vol.66, pp.5778-5786, 2020. [7] Zhaoxia, X., Tianli, Z., Huaimin, L., Guerrero, J.M., Su, C.L. and Vásquez, J.C., "Coordinated control of a hybrid-electric-ferry shipboard microgrid", IEEE Transactions on Transportation Electrification, Vol.5, pp.828-839, 2020. [8] Liu, D., Xu, Y., Wei, Q. and Liu, X., "Residential energy scheduling for variable weather solar energy based on adaptive dynamic programming", IEEE/CAA Journal of Automatica Sinica, Vol.5, pp.36-46, 2017. [9] Karimi, H. and Jadid, S., "Optimal microgrid operation scheduling by a novel hybrid multi-objective and multi-attribute decision-making framework", Energy, Vol.186, p.115912, 2019. [10] Farzin, H., Fotuhi-Firuzabad, M. and Moeini-Aghtaie, M., "A stochastic multi-objective framework for optimal scheduling of energy storage systems in microgrids", IEEE Transactions on Smart Grid, Vol.8, pp.117-127, 2017. [11] Kim, H.J., Kim, M.K. and Lee, J.W., "A two-stage stochastic p-robust optimal energy trading management in microgrid operation considering uncertainty with hybrid demand response", International Journal of Electrical Power & Energy Systems, Vol.124, p.106422, 2021. [12] Shuai, H., Fang, J., Ai, X., Tang, Y., Wen, J. and He, H., "Stochastic optimization of economic dispatch for microgrid based on approximate dynamic programming", IEEE Transactions on Smart Grid, Vol.10, pp.2440-2452, 2018. [13] Zakaria, A., Ismail, F.B., Lipu, M.H. and Hannan, M.A., "Uncertainty models for stochastic optimization in renewable energy applications", Renewable Energy, Vol.145, pp.1543-1571, 2020. [14] Craparo, E., Karatas, M. and Singham, D.I., "A robust optimization approach to hybrid microgrid operation using ensemble weather forecasts", Applied energy, Vol.201, pp.135-147, 2017. [15] Qiu, H., Long, H., Gu, W. and Pan, G., "Recourse-Cost Constrained Robust Optimization for Microgrid Dispatch With Correlated Uncertainties", IEEE Transactions on Industrial Electronics, 2020. [16] Zhang, Y., Fu, L., Zhu, W., Bao, X. and Liu, C., "Robust model predictive control for optimal energy management of island microgrids with uncertainties", Energy, Vol.164, pp.1229-1241, 2018. [17] Zhu, J., Mo, X., Zhu, T., Guo, Y., Luo, T. and Liu, M., "Real-time stochastic operation strategy of a microgrid using approximate dynamic programming-based spatiotemporal decomposition approach", IET Renewable Power Generation, Vol.13, pp.3061-3070, 2019. [18] Li, B., Chen, T., Wang, X. and Giannakis, G.B., "Real-time energy management in microgrids with reduced battery capacity requirements", IEEE Transactions on Smart Grid, Vol.10, pp.1928-1938, 2017. [19] Liu, W., Zhuang, P., Liang, H., Peng, J. and Huang, Z., "Distributed economic dispatch in microgrids based on cooperative reinforcement learning", IEEE transactions on neural networks and learning systems, Vol.29, pp.2192-2203, 2018. [20] Khooban, M.H. and Gheisarnejad, M., "A Novel Deep Reinforcement Learning Controller Based Type-II Fuzzy System: Frequency Regulation in Microgrids", IEEE Transactions on Emerging Topics in Computational Intelligence, 2020. [21] Kofinas, P., Vouros, G. and Dounis, A.I., "Energy management in solar microgrid via reinforcement learning using fuzzy reward", Advances in Building Energy Research, Vol.12, pp.97-115, 2018. [22] Zeng, P., Li, H., He, H. and Li, S., "Dynamic energy management of a microgrid using approximate dynamic programming and deep recurrent neural network learning", IEEE Transactions on Smart Grid, Vol.10, pp.4435-4445, 2018. [23] Xu, Xu, Youwei Jia, Yan Xu, Zhao Xu, Songjian Chai, and Chun Sing Lai., "A multi-agent reinforcement learning-based data-driven method for home energy management", IEEE Transactions on Smart Grid, Vol. 11, pp. 3201-3211, 2020. [24] Huang, Bin, and Jianhui Wang, "Deep-Reinforcement-Learning-Based Capacity Scheduling for PV-Battery Storage System", IEEE Transactions on Smart Grid, Vol. 12, pp. 2272-2283, 2020. [25] Han, M., Zhao, J., Zhang, X., Shen, J. and Li, Y., "The reinforcement learning method for occupant behavior in building control: A review", Energy and Built Environment, 2020. [26] Dos Santos Mignon, A. da Rocha, R.L.D.A., "An Adaptive Implementation of ε-Greedy in Reinforcement Learning", Procedia Computer Science, Vol.109, pp.1146-1151, 2017. [27] Afrasiabi, M., Mohammadi, M., Rastegar, M. and Kargarian, A., "Multi-agent microgrid energy management based on deep learning forecaster", Energy, Vol.186, p.115873, 2019. [28] Devraj, Adithya M., Ioannis Kontoyiannis, Sean P. Meyn, "Differential temporal difference learning", IEEE Transactions on Automatic Control, 2020. [29] Beere, N., McPhail, D. and Sharma, R., "A general methodology for utility microgrid planning: A Cairns case study", In 2015 IEEE PES Asia-Pacific Power and Energy Engineering Conference (APPEEC), pp. 1-5, 2015. [30] Carpinelli, G., Celli, G., Mocci, S., Mottola, F., Pilo, F. and Proto, D., "Optimal integration of distributed energy storage devices in smart grids", IEEE Transactions on smart grid, Vol.4, pp.985-995, 2013. [31] Rabiee, A., Sadeghi, M., Aghaeic, J. and Heidari, A., "Optimal operation of microgrids through simultaneous scheduling of electrical vehicles and responsive loads considering wind and PV units uncertainties", Renewable and Sustainable Energy Reviews, Vol.57, pp.721-739, 2016. [32] Ustun, T.S., Ozansoy, C. and Zayegh, A., "Recent developments in microgrids and example cases around the world—A review", Renewable and Sustainable Energy Reviews, Vol.15, pp.4030-4041, 2011. [33] Lei, L., Tan, Y., Dahlenburg, G., Xiang, W. and Zheng, K., "Dynamic Energy Dispatch Based on Deep Reinforcement Learning in IoT-Driven Smart Isolated Microgrids", IEEE Internet of Things Journal, 2020.