Comparative Analysis of Modified Q-Learning and DQN for Autonomous Robot Navigation
DOI:
https://doi.org/10.62411/faith.3048-3719-49Keywords:
Deep-Q network (DQN), Gym environment, Mobile robotic, Path planning; Q-Learning , Reinforcement Learning.Abstract
Autonomous mobile robot navigation integrates localization, mapping, and path planning to enable effective operation in complex environments. This study compares a modified Q-learning algorithm with a Deep Q-Network (DQN) in a simulated gym environment, focusing on convergence speed, success rate, and computational efficiency. The modified Q-learning algorithm converged after 44 episodes, outperforming the DQN, which required 400 episodes. It achieved a success rate of 69.6% with a cumulative reward that surpassed the DQN in fewer episodes, while completing simulations in just 9 minutes compared to 400 minutes for the DQN. These results demonstrate the modified Q-learning’s efficiency in addressing the exploration-exploitation trade-off and navigating complex environments. This study highlights the potential of the modified Q-learning algorithm for real-world applications in robotics and autonomous navigation, providing a foundation for future research in intelligent path planning
Downloads
References
N. Khlif, N. Khraief, and S. Belghith, “Reinforcement Learning for Mobile Robot Navigation: An overview,” in 2022 IEEE Information Technologies & Smart Industrial Systems (ITSIS), Jul. 2022, pp. 1–7. doi: 10.1109/ITSIS56166.2022.10118362.
G. Chen and L. Hong, “Research on Environment Perception System of Quadruped Robots Based on LiDAR and Vision,” Drones, vol. 7, no. 5, p. 329, May 2023, doi: 10.3390/drones7050329.
M. Reda, A. Onsy, A. Y. Haikal, and A. Ghanbari, “Path planning algorithms in the autonomous driving system: A comprehensive review,” Rob. Auton. Syst., vol. 174, p. 104630, Apr. 2024, doi: 10.1016/j.robot.2024.104630.
M. Gök, “Dynamic path planning via Dueling Double Deep Q-Network (D3QN) with prioritized experience replay,” Appl. Soft Comput., vol. 158, p. 111503, Jun. 2024, doi: 10.1016/j.asoc.2024.111503.
R. Siegwart, I. R. Nourbakhsh, and D. Scaramuzza, Introduction to Autonomous Mobile Robots, 2nd ed. Autonomous Systems Lab; Carnegie Mellon University; University of Zurich: MIT Press, 2011.
H. R. Everett, Sensors for Mobile Robots. Natick, MA: A K Peters, 1995.
H. Belaidi and F. Demim, “NURBs Based Multi-robots Path Planning with Obstacle Avoidance,” J. Comput. Theor. Appl., vol. 1, no. 4, pp. 478–491, May 2024, doi: 10.62411/jcta.10387.
N. Khlif, K. Nahla, and B. Safya, “Reinforcement learning with modified exploration strategy for mobile robot path planning,” Robotica, vol. 41, no. 9, pp. 2688–2702, Sep. 2023, doi: 10.1017/S0263574723000607.
J. Muhammad and I. O. Bucak, “An improved Q-learning algorithm for an autonomous mobile robot navigation problem,” in 2013 The International Conference on Technological Advances in Electrical, Electronics and Computer Engineering (TAEECE), May 2013, pp. 239–243. doi: 10.1109/TAEECE.2013.6557278.
C. Xia and A. El Kamel, “A Reinforcement Learning Method of Obstacle Avoidance for Industrial Mobile Vehicles in Unknown Environments Using Neural Network,” in Proceedings of the 21st International Conference on Industrial Engineering and Engineering Management 2014, 2015, pp. 671–675. doi: 10.2991/978-94-6239-102-4_136.
L. Khriji, F. Touati, K. Benhmed, and A. Al-Yahmedi, “Mobile Robot Navigation Based on Q-Learning Technique,” Int. J. Adv. Robot. Syst., vol. 8, no. 1, Mar. 2011, doi: 10.5772/10528.
Hee Rak Beom and Hyung Suck Cho, “A sensor-based navigation for a mobile robot using fuzzy logic and reinforcement learning,” IEEE Trans. Syst. Man. Cybern., vol. 25, no. 3, pp. 464–477, Mar. 1995, doi: 10.1109/21.364859.
K. Macek, I. Petrovic, and N. Peric, “A reinforcement learning approach to obstacle avoidance of mobile robots,” in 7th International Workshop on Advanced Motion Control. Proceedings (Cat. No.02TH8623), 2002, pp. 462–466. doi: 10.1109/AMC.2002.1026964.
X. Ruan, D. Ren, X. Zhu, and J. Huang, “Mobile Robot Navigation based on Deep Reinforcement Learning,” in 2019 Chinese Control And Decision Conference (CCDC), Jun. 2019, pp. 6174–6178. doi: 10.1109/CCDC.2019.8832393.
Guo-Sheng Yang, Er-Kui Chen, and Cheng-Wan An, “Mobile robot navigation using neural Q-learning,” in Proceedings of 2004 International Conference on Machine Learning and Cybernetics (IEEE Cat. No.04EX826), 2005, vol. 1, pp. 48–52. doi: 10.1109/ICMLC.2004.1380601.
L. Tai and M. Liu, “Towards Cognitive Exploration through Deep Reinforcement Learning for Mobile Robots,” arXiv. Oct. 06, 2016. [Online]. Available: http://arxiv.org/abs/1610.01733
W. Wang, Z. Wu, H. Luo, and B. Zhang, “Path Planning Method of Mobile Robot Using Improved Deep Reinforcement Learning,” J. Electr. Comput. Eng., vol. 2022, pp. 1–7, Apr. 2022, doi: 10.1155/2022/5433988.
X. Lei, Z. Zhang, and P. Dong, “Dynamic Path Planning of Unknown Environment Based on Deep Reinforcement Learning,” J. Robot., vol. 2018, pp. 1–10, Sep. 2018, doi: 10.1155/2018/5781591.
Q. Zhang, J. Lin, Q. Sha, B. He, and G. Li, “Deep Interactive Reinforcement Learning for Path Following of Autonomous Underwater Vehicle,” IEEE Access, vol. 8, pp. 24258–24268, 2020, doi: 10.1109/ACCESS.2020.2970433.
Y. Wang, H. He, and C. Sun, “Learning to Navigate Through Complex Dynamic Environment With Modular Deep Reinforcement Learning,” IEEE Trans. Games, vol. 10, no. 4, pp. 400–412, Dec. 2018, doi: 10.1109/TG.2018.2849942.
J. C. de Jesus, V. A. Kich, A. H. Kolling, R. B. Grando, M. A. de S. L. Cuadros, and D. F. T. Gamarra, “Soft Actor-Critic for Navigation of Mobile Robots,” J. Intell. Robot. Syst., vol. 102, no. 2, p. 31, Jun. 2021, doi: 10.1007/s10846-021-01367-5.
M. Towers et al., “Gymnasium: A Standard Interface for Reinforcement Learning Environments,” arXiv. Jul. 24, 2024. [Online]. Available: http://arxiv.org/abs/2407.17032
Z. Miljković, M. Mitić, M. Lazarević, and B. Babić, “Neural network Reinforcement Learning for visual control of robot manipulators,” Expert Syst. Appl., vol. 40, no. 5, pp. 1721–1736, Apr. 2013, doi: 10.1016/j.eswa.2012.09.010.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 1970 Nessrine Khlif, Nahla Khraief, Safya Belghith
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
