Distributed Reinforcement Learning Based Efficient Radio Resource Allocation For Heterogeneous IoT Wireless Network
DOI:
https://doi.org/10.22399/ijcesen.2852Keywords:
Internet of Things (IoT), Power Allocation, Heterogeneous Network , Reinforcement Learning, Quality of Service (QoS)Abstract
The heterogeneous cellular network has emerged as a critical infrastructure in supporting diverse Internet of Things (IoT) based services. As next-generation technologies continue to evolve, they will provide a unified framework capable of seamlessly connecting huge number of IoT devices. This integration will support the complex requirements of modern IoT-driven business processes. To achieve maximum capacity for IoT applications there is a need to integrate multiple wireless network technologies which makes the environment dense. However, this network densification also raises interference levels from neighbouring devices which can negatively impact the Quality of Service (QoS). In a dense IoT environment with limited radio resources there is need of efficient radio resource allocation strategy to maintain QoS across various connected devices. This work presents a distributed reinforcement learning based power allocation algorithm for heterogeneous IoT networks. We also propose a reward function that accounts for the QoS needs of multiple IoT users, promotes fairness, and ensures reliable connectivity. We carry out complexity analysis and convergence analysis of our proposed algorithm and also we explore different learning frameworks to evaluate the performance of the algorithm. Results demonstrated that the proposed method is effective in improving network capacity and other performance measures in dense heterogeneous environment.
References
[1] Zhou, I., Li, X., Chen, Y., & Zhang, H. (2021). Internet of Things 2.0: Concepts, applications, and future directions. IEEE Access, 9, 70961–71012. https://doi.org/10.1109/ACCESS.2021.3078549
[2] Palattella, M. R., Dohler, M., Grieco, L. A., Boggia, G., & Mendes, P. (2016). Internet of Things in the 5G era: Enablers, architecture, and business models. IEEE Journal on Selected Areas in Communications, 34(3), 510–527. https://doi.org/10.1109/JSAC.2016.2525418
[3] Peng, M., Zhang, K., Jiang, J., Wang, J., & Wang, W. (2015). Energy-efficient resource assignment and power allocation in heterogeneous cloud radio access networks. IEEE Transactions on Vehicular Technology, 64(11), 5275–5287. https://doi.org/10.1109/TVT.2014.2379922
[4] Zhang, H., Huang, S., Jiang, C., Long, K., Leung, V. C. M., & Poor, H. V. (2017). Energy-efficient user association and power allocation in millimeter-wave-based ultra-dense networks with energy harvesting base stations. IEEE Journal on Selected Areas in Communications, 35(9), 1936–1947. https://doi.org/10.1109/JSAC.2017.2720898
[5] Bin Farooq, U., Sajid Hashmi, U., Qadir, J., Imran, A., & Mian, A. N. (2018). User transmit power minimization through uplink resource allocation and user association in HetNets. Proceedings of the IEEE Global Communications Conference (GLOBECOM), 1–6. https://doi.org/10.1109/GLOCOM.2018.8647409
[6] Song, L., Chai, K. K., Chen, Y., Schormans, J., Loo, J., & Vinel, A. (2017). QoS-aware energy-efficient cooperative scheme for cluster-based IoT systems. IEEE Systems Journal, 11(3), 1447–1455. https://doi.org/10.1109/JSYST.2015.2465292
[7] Wang, J., Jiang, C., Wei, Z., Pan, C., Zhang, H., & Ren, Y. (2019). Joint UAV hovering altitude and power control for space-air-ground IoT networks. IEEE Internet of Things Journal, 6(2), 1741–1753. https://doi.org/10.1109/JIOT.2018.2875493
[8] Bakht, K., Ahmed, R., Ijaz, S., Ahmed, A., Jabeen, F., & Khwaja, A. S. (2019). Power allocation and user assignment scheme for beyond 5G heterogeneous networks. Wireless Communications and Mobile Computing, 1–9. https://doi.org/10.1155/2019/2472783
[9] Ha, V. N., & Le, L. B. (2014). Fair resource allocation for OFDMA femtocell networks with macrocell protection. IEEE Transactions on Vehicular Technology, 63(3), 1388–1401. https://doi.org/10.1109/TVT.2013.2284572
[10] Kai, C., Li, H., Xu, L., Li, Y., & Jiang, T. (2019). Joint subcarrier assignment with power allocation for sum rate maximization of D2D communications in wireless cellular networks. IEEE Transactions on Vehicular Technology, 68(5), 4748–4759. https://doi.org/10.1109/TVT.2019.2903815
[11] Saad, H., Mohamed, A., & Elbatt, T. (2012). Distributed cooperative Q-learning for power allocation in cognitive femtocell networks. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–5. https://doi.org/10.1109/VTCFall.2012.6399230
[12] Tefft, J. R., & Kirsch, N. J. (2013). A proximity-based Q-learning reward function for femtocell networks. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–5. https://doi.org/10.1109/VTCFall.2013.6692057
[13] Amiri, R., Mehrpouyan, H., Fridman, L., Mallik, R. K., Nallanathan, A., & Matolak, D. (2018). A machine learning approach for power allocation in HetNets considering QoS. Proceedings of the IEEE International Conference on Communications (ICC), 1–6. https://doi.org/10.1109/ICC.2018.8422864
[14] Zhang, G. A., Gu, J. Y., Bao, Z. H., Xu, C., & Zhang, S. B. (2014). Joint routing and channel assignment algorithms in cognitive wireless mesh networks. Transactions on Emerging Telecommunications Technologies, 25(3), 294–307. https://doi.org/10.1002/ett
[15] Zhang, Y., Kang, C., Ma, T., Teng, Y., & Guo, D. (2018). Power allocation in multi-cell networks using deep reinforcement learning. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–6. https://doi.org/10.1109/VTCFall.2018.8690757
[16] Meng, F., Chen, P., Wu, L., & Cheng, J. (2020). Power allocation in multi-user cellular networks: Deep reinforcement learning approaches. IEEE Transactions on Wireless Communications, 19(10), 6255–6267. https://doi.org/10.1109/TWC.2020.3001736
[17] Ding, H., Zhao, F., Tian, J., Li, D., & Zhang, H. (2020). A deep reinforcement learning for user association and power control in heterogeneous networks. Ad Hoc Networks, 102, 102069. https://doi.org/10.1016/j.adhoc.2019.102069
[18] Giannopoulos, A., Oikonomou, K., Koutsopoulos, I., & Antoniou, E. (2021). Demand-driven power allocation in wireless networks with deep Q-learning. Proceedings of the IEEE International Symposium on a World of Wireless, Mobile and Multimedia Networks (WoWMoM), 248–251. https://doi.org/10.1109/WoWMoM51794.2021.00045
[19] Wang, H., Fan, Y., & Yang, L. (2021). A power allocation algorithm for underwater acoustic communication networks based on reinforcement learning. Proceedings of the IEEE International Conference on Civil Aviation Safety and Information Technology (ICCASIT), 986–990. https://doi.org/10.1109/ICCASIT53235.2021.9633599
[20] Jiang, S., & Zheng, J. (2021). A Q-learning based dynamic power control algorithm for D2D communication underlaying cellular networks. Proceedings of the IEEE Wireless Communications and Signal Processing (WCSP), 1–5. https://doi.org/10.1109/WCSP52459.2021.9613167
[21] Anzaldo, A., & Andrade, A. G. (2022). Deep reinforcement learning for power control in multi-tasks wireless cellular networks. Proceedings of the IEEE Mediterranean Communication and Networking Conference (MeditCom), 61–65. https://doi.org/10.1109/MeditCom55741.2022.9928617
[22] Huangi, R., Si, J., Shi, J., & Li, Z. (2021). Deep-reinforcement-learning-based resource allocation in ultra-dense networks. Proceedings of the IEEE Wireless Communications and Signal Processing (WCSP), 1–5. https://doi.org/10.1109/WCSP52459.2021.9613186
[23] Zhao, Y., Peng, T., Guo, Y., & Wang, W. (2021). Energy-efficient uplink power allocation in ultra-dense networks through multi-agent reinforcement learning. Proceedings of the IEEE Vehicular Technology Conference (VTC-Fall), 1–7. https://doi.org/10.1109/VTC2021-Fall52928.2021.9625554
[24] Sun, M., Mei, E., Wang, S., & Jin, Y. (2023). Joint DDPG and unsupervised learning for channel allocation and power control in centralized wireless cellular networks. IEEE Access, 11, 42191–42203. https://doi.org/10.1109/ACCESS.2023.3270316
[25] Valcarce, A., & Zhang, J. (2010). Empirical indoor-to-outdoor propagation model. IEEE Access, 9, 682–685. https://doi.org/10.1109/ACCESS.2010.5643660
[26] Bin Sediq, A., Gohary, R. H., Schoenen, R., & Yanikomeroglu, H. (2013). Optimal and low-complexity power allocation for OFDMA-based relay systems. IEEE Transactions on Wireless Communications, 12(1), 171–184. https://doi.org/10.1109/TWC.2012.112112.110796
[27] Khenak, F. (2010). V-learning. Proceedings of the 2010 International Conference on Computer Information Systems and Industrial Management Applications (CISIM), 228–232. https://doi.org/10.1109/CISIM.2010.5643660
[28] Smith, J. Q., & White, D. J. (1994). Markov decision processes. Mathematics of Operations Research, 157(1), 1–25.
[29] Hierons, R. (1999). Machine learning (T. M. Mitchell). McGraw-Hill.
[30] Sutton, R. S., & Barto, A. G. (2018). Reinforcement learning: An introduction. MIT Press.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 International Journal of Computational and Experimental Science and Engineering
This work is licensed under a Creative Commons Attribution 4.0 International License.
