AUV traversal planning method based on AMDQN
-
摘要: 为了提高AUV的遍历路径规划能力, 文中提出了一种基于自适应多目标深度Q网络(AMDQN)的自主水下航行器(AUV)遍历路径规划算法。首先, 文中建立起契合实际场景的遍历规划环境, 一方面利用射线覆盖方法代替传统的矩形栅格式建模方法提升建模精度, 另一方面约束了AUV的可选动作集限制其机动能力; 其次, 在上述环境下\文中对算法的各个组件进行精心设计。具体地, 在状态空间设计上, 文中将模糊的全局环境信息、精确的以AUV为中心的局部环境信息以及AUV自身位置信息进行融合, 系统地对环境进行表达; 在奖励函数设计上, 文中引入了“规则惩罚”与“边缘指导”, 使得AUV可以沿环境边缘稳定提高覆盖率; 在参数更新方法上, 文中在M-DQN算法的基础上设计了自适应的温度参数更新策略, 同时引入了多步奖励与“竞争”网络架构缓解训练方差; 在训练过程中, 文中“软重置”全连接层参数缓解局部最优现象。最后, 通过仿真发现所提算法相较于传统方法和未改进的强化学习算法有更强的环境适应性和更高的遍历覆盖率。Abstract: To improve the traversal path planning capability of autonomous underwater vehicle(AUV), this paper proposes an AUV traversal path planning algorithm based on advanced M-DQN (AMDQN). First, this paper establishes a traversal planning environment suitable for practical scenarios. On the one hand, the ray coverage method is used instead of the traditional rectangular grid modeling method to improve modeling accuracy; on the other hand, the optional action set of the AUV is constrained to limit its maneuverability. Second, each component of the algorithm is carefully designed under the above environment. Specifically, in the state space design, this paper fuses vague global environmental information, accurate local environmental information centered on the AUV, and the AUV’s own position information to systematically represent the environment. In the reward function design, “rule penalty” and “edge guidance” are introduced, enabling the AUV to stably improve coverage along environmental edges. For the parameter update method, an adaptive temperature parameter update strategy is designed based on M-DQN, while the multi-step reward and a “dueling” network architecture are introduced to alleviate training variance. During the training process, “soft reset” of fully connected layer parameters is adopted to mitigate the local optimum problem. Finally, simulation results show that the proposed algorithm has stronger environmental adaptability and higher traversal coverage compared with traditional methods and unimproved reinforcement learning algorithms.
-
图 5 有障碍和无障碍环境中温度参数的动态调整情况
Figure 5. Dynamic adjustment of temperature parameters under obstacle-free and obstacle environments
图 7 无障碍环境5种算法奖励函数曲线
Figure 7. Five algorithm reward function curves for barrier-free environment
图 8 有障碍环境5种算法奖励函数曲线
Figure 8. Five algorithm reward function curves for barrier environment
图 9 无障碍环境不同算法组件覆盖率
Figure 9. Coverage of different algorithm components in an accessible environment
图 13 无障碍环境下两种算法的可视化路径
Figure 13. Visualization path of two algorithms in an accessible environment
图 14 圆形障碍环境下两种算法的可视化路径
Figure 14. Visualization path of two algorithms in a circular obstacle environment
图 15 多障碍环境下两种算法的可视化路径
Figure 15. Visualized paths of two algorithms in multi-obstacle environments
图 16 “U”型障碍环境下两种算法的可视化路径
Figure 16. Visualized paths of two algorithms in a "U"-shaped obstacle environment
图 17 AUV探测范围变化和海洋环境存在扰动时的遍历轨迹
Figure 17. Traversal trajectories with varying auv detection ranges and disturbances in marine environments
表 1 算法伪代码
Table 1. Algorithm pseudocode
基于AMDQN的遍历规划算法, 最终输出柔性策略$ \boldsymbol{\pi } $: 初始化: 经验池$ D $, 状态价值函数$ \theta $, 目标状态价值函数$ {\theta }^{-}=\theta $, 超参数$ \alpha ,\tau ,\varsigma $等; 初始化: AUV与障碍物位姿信息; $ \text{ for }i\text{ to }{E}_{\max }\colon $ $ \text{ for }i\text{ to }{T}_{\text{end}}\colon $ 依式(4)计算状态空间; 依策略采样$ {a}_{t}\sim \pi $; 依式(2)(10)(23)计算N步状态$ {s}_{t+N} $, 折扣奖励和$ r_{t}^{M,N} $, 并将$ ({s}_{t},{a}_{t}, $$ r_{t}^{M,N}, {s}_{t+N}) $储存在经验池$ D $中; $ \text{if len }(D)> {D}_{s}\colon $ 依式(24)(25)为损失函数更新参$ \theta $; 依式(27)为损失函数更新参数$ \tau $; 依式(28)计算目标参数$ {\theta }^{-} $; $ \text{if }i> {E}_{s}\text{ and }i{\text{%}} 20=0\colon $ 依式(39)”软”重置部分网络参数; $ \text{if Done}\colon $ $ \text{break} $; $ \mathrm{end}~\text{for } $ $ \mathrm{end}~\text{for} $ 
下载: 导出CSV
表 2 不同超参数下算法性能表
Table 2. Performance of the algorithm with different hyperparameters
参数类型 参数值 首次遍历
成功回合首次遍历
成功时间/s池化后图像维度 10×10 — — 20×20 595 1266.42 50×50 1312 4938.65 奖励步数 1 942 2034.72 3 595 1266.42 5 — — 卷积池化层数 1 — — 3 595 1266.42 5 602 1632.66
下载: 导出CSV
表 3 不同奖励超参数下算法性能表
Table 3. Performance of the algorithm with different reward hyperparameters
超参数值 首达幕数 路径长度/m 转弯次数 [−1,−1,−0.75] 821 447 81 [−0.5,−1,−0.75] 766 502.5 114 [−2,−1,−0.25] — — — [−2,−1,−1.5] 1215 414 54 [−2,−0.5,−0.75] 802 493.5 105 [−2,−1.5,−0.75] 1010 421.5 76 [−2,−1,−0.75] 824 426 66
下载: 导出CSV
表 4 算法超参数表
Table 4. Algorithm Hyperparameter Table
超参数 值 超参数 值 经验池容量 60 000 网络学习率 0.0003 $ {T}_{\max } $ 350 折扣因子$ \gamma $ 0.99 目标熵$ \overline{\mathcal{H}} $ 0.25 奖励参数$ {a}_{1} $ 0.15 奖励参数$ {a}_{2} $ -1 奖励参数$ \alpha $ 0.1 奖励参数$ {R}_{1} $ 50 奖励参数$ {R}_{2} $ −5 软重置参数$ {\alpha }_{f} $ 0.05 软更新参数$ \varsigma $ 0.001
下载: 导出CSV
表 5 不同算法组件下算法性能表
Table 5. Performance table of algorithms under different algorithm components
环境信息 无障碍 有障碍 首达指定覆盖率回合 $ {\mathrm{D}}_{90} $ $ {\mathrm{D}}_{99} $ $ {\mathrm{D}}_{90} $ $ {\mathrm{D}}_{99} $ SAC 960 / 960 / MDQN 740 / 820 / AMDQN(不含奖励修正) 720 / 940 / AMDQN(不含奖励修正与局部地图) 1110 / 1180 / AMDQN 960 1320 640 900
下载: 导出CSV
表 6 不同算法的训练时长和单步推理时长
Table 6. Training and inference time for different algorithms
SAC MDQN PPO-penalty AMDQN 单步推理时长/ms 1.13 1.02 1.09 1.12 训练时长/s 3627.45 3125.12 2746.32 1848.51
下载: 导出CSV
表 7 无障碍环境不同算法的覆盖率与路径长度
Table 7. Coverage and planning length of different algorithms in an obstacle-free environment
覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {A}^{*} $ 99.16 507 38 SAC 97.71 525 79 MDQN 99.02 460 63 PPO-penalty 89.86 525 80 AMDQN 99.02 430 42
下载: 导出CSV
表 8 圆形障碍环境不同算法的覆盖率与路径长度
Table 8. Coverage and planning length of different algorithms in circular obstacle environments
覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {A}^{*} $ 92.32 379.5 49 SAC 90.38 525.0 101 MDQN 97.54 525.0 87 PPO-penalty 87.40 525.0 100 AMDQN 99.07 387.0 55
下载: 导出CSV
表 9 多障碍环境不同算法的覆盖率与路径长度
Table 9. Coverage rate and planning length of different algorithms in multi-obstacle environments
覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {\mathrm{A}}^{*} $ 86.57 525 60 SAC 99.02 523 94 MDQN 94.81 525 117 PPO-penalty 78.11 525 102 AMDQN 99.11 403 64
下载: 导出CSV
表 10 “U”型障碍环境不同算法的覆盖率与路径长度
Table 10. Coverage Rate and Planning length of Different Algorithms in a "U"-Shaped Obstacle Environment
覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {A}^{*} $ 89.45 375 68 SAC 92.48 525 75 MDQN 95.7 525 138 PPO-penalty 78.78 525 71 AMDQN 99.01 430 68
下载: 导出CSV
-
[1] 苗润龙, 庞硕, 姜大鹏, 等. 海洋自主航行器多海湾区域完全遍历路径规划[J]. 测绘学报, 2019, 48(2): 256-264.Miao R L, Pang S, Jiang D P, et al. Complete coverage path planning for autonomous marine vehicle used in multi-bay areas[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(2): 256-264. [2] Tang G, Tang C, Zhou H, et al. R-DFS: A coverage path planning approach based on region optimal decomposition[J]. Remote Sensing, 2021, 13(8): 1525. [3] Xu A, Viriyasuthee C, Rekleitis I. Optimal complete terrain coverage using an unmanned aerial vehicle[C]//2011 IEEE International Conference on Robotics and Automation. IEEE, 2011: 2513-2519. [4] 熊亿民. 基于改进蚁群算法的全向移动机器人全遍历路径规划[J]. 计算机系统应用, 2021, 30(6): 209-214.Xiong Y M. Full Traversal Path Planning of Omnidirectional Mobile Robot Based on Improved Ant Colony Algorithm[J]. Computer Systems and Applications, 2021, 30(6): 209-214. [5] 池志猛, 李智刚, 赵洋, 等. 针对深海采矿车的遍历路径规划方法研究[J]. 矿业研究与开发, 2022, 42(7): 160-166.Chi Z M, Li Z G, Zhao Y, et al. Research on traversal path planning method for deep-sea mining vehicles[J]. Mining Research and Development, 2022, 42(7): 160-166. [6] Bouman A, et al. Adaptive coverage path planning for efficient exploration of unknown environments[C]//2022 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). IEEE, 2022: 11916-11923. [7] 温志文, 杨春武, 蔡卫军, 等. 复杂环境下UUV完全遍历路径规划方法[J]. 鱼雷技术, 2017, 25(1): 22-26.Wen Z W, Yang C W, Cai W J, et al. A complete coverage path planning method of UUV under complex environment[J]. Journal of Unmanned Undersea Systems, 2017, 25(1): 22-26. [8] 史方青, 黄华, 张昊, 等. 玻璃幕墙清洗机器人内螺旋完全遍历路径规划研究[J]. 哈尔滨工程大学学报, 2024, 45(6): 1170-1178.Shi F Q, Huang H, Zhang H, et al. Study on the path planning of the internal spiral complete traversal for glass-curtain wall-cleaning robots[J]. Journal of Harbin Engineering University, 2024, 45(6): 1170-1178. [9] 吕霞付, 程启忠, 李森浩, 等. 基于改进A*算法的无人船完全遍历路径规划[J]. 水下无人系统学报, 2019, 27(6): 695-703.LÜ X F, Cheng Q Z, Li S H, et al. Unmanned surface vehicle full traversal path planning based on improved A* algorithm[J]. Journal of Unmanned Undersea Systems, 2019, 27(6): 695-703. [10] 时天龙. 基于A*算法的无人船完全遍历路径规划研究[D]. 大连: 大连海事大学, 2024: 32-38. [11] 王岩, 李春书, 张福龙. 一种改进的爬壁机器人全遍历路径规划[J]. 计算机仿真, 2023, 40(7): 447-452.Wang Y, Li C S, Zhang F L. An improved full traversal path planning for wall climbing robot[J]. Computer Simulation, 2023, 40(7): 447-452. [12] 孟浩德, 吴征天, 吴闻笛, 等. 基于记忆模拟退火算法的扫地机器人遍历路径规划[J]. 计算机与数字工程, 2024, 52(3): 821-826, 857.Meng H D, Wu Z T, Wu W D et al. Traversal path planning of sweeping robot based on memory simulated annealing algorithm[J]. Computer and Digital Engineering. 2024, 52(3): 821-826, 857. [13] Lin B, Han G H, Song C C, et al. Traversal path planning and simulation of robot based on radiation scanning[J]. Journal of System Simulation, 2021, 33(1): 83-90. [14] Carvalho J P, Aguiar A P. Deep reinforcement learning for zero-shot coverage path planning with mobile robots[J]. IEEE/CAA Journal of Automatica Sinica, 2025, 12(8): 1594-1609. [15] Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533. [16] Chen Y, Lu Z M, Cui J L, et al. A complete coverage path planning algorithm for lawn mowing robots based on deep reinforcement learning[J]. Sensors, 2025, 25(2): 416. [17] Xing B, Wang X, Yang L, et al. An algorithm of complete coverage path planning for unmanned surface vehicle based on reinforcement learning[J]. Journal of Marine Science and Engineering, 2023, 11(3): 645. [18] Ai B, Jia M, Xu H, et al. Coverage path planning for maritime search and rescue using reinforcement learning[J]. Ocean Engineering, 2021, 241: 110098. [19] Heydari J, Saha O, Ganapathy V. Reinforcement learning-based coverage path planning with implicit cellular decomposition[EB/OL]. (2021-10-18)[2026-07-21]. https://doi.org/10.48550/arXiv.2110.09018. [20] Van H H, Guez A, Silver D. Deep reinforcement learning with double Q-learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2016, 30(1): 2094-2100. [21] Theile M, Bayerlein H, Nai R, et al. UAV coverage path planning under varying power constraints using deep reinforcement learning[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS). IEEE, 2020: 1444-1449. [22] Zheng T, Jin Y, Zhao H, et al. Deep reinforcement learning based coverage path planning in unknown environments[C]//2024 6th International Conference on Frontier Technologies of Information and Computer (ICFTIC). IEEE, 2024: 1608-1611. [23] Carvalho J P, Aguiar A P. A reinforcement learning based online coverage path planning algorithm[C]//2023 IEEE International Conference on Autonomous Robot Systems and Competitions(ICARSC). IEEE, 2023: 81-86. [24] Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//International Conference on Machine Learning. PMLR, 2018: 1861-1870. [25] Jonnarth A, Zhao J, Felsberg M. Learning coverage paths in unknown environments with deep reinforcement learning[EB/OL]. (2024-06-07)[2026-07-21]. https://doi.org/10.48550/arXiv.2306.16978. [26] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[EB/OL]. (2017-07-20)[2026-07-21]. https://doi.org/10.48550/arXiv.1707.06347. [27] Wijegunawardana I D, Samarakoon S M B P, Muthugala M A V J, et al. Risk-aware complete coverage path planning using reinforcement learning[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2025, 55(4): 2476-2488. [28] Vieillard N, Pietquin O, Geist M. Munchausen reinforcement learning[C]//Advances in Neural Information Processing Systems. 2020, 33: 4235-4246. [29] Von B M. An open-source benchmark simulator: control of a BlueROV2 underwater robot[J]. Journal of Marine Science and Engineering, 2022, 12(10): 1898. -
下载:
点击查看大图
计量
- 文章访问数: 71
- HTML全文浏览量: 43
- PDF下载量: 15
- 被引次数: 0
