AUV Traversal Planning Method Based on AMDQN
-
摘要: 为了提高自主水下航行器(AUV)的遍历路径规划能力, 文中提出了一种基于改进的蒙豪森深度Q网络(AMDQN)的AUV遍历路径规划算法。首先建立贴合实际场景的遍历规划环境, 利用射线覆盖方法代替传统矩形栅格式建模, 以此提升环境建模精度, 同时约束AUV的可选动作集, 限制其机动能力。在上述环境下对算法各组件进行设计: 状态空间方面, 将模糊的全局环境信息、以AUV为中心的高精度局部环境信息, 以及AUV自身位置信息进行融合, 对环境进行系统化描述; 奖励函数方面, 引入了规则惩罚项与边缘指导机制, 使AUV能够沿环境边缘稳定航行, 提高覆盖率; 参数更新方面, 在DQN算法的基础上设计自适应温度参数更新策略, 同时结合多步奖励与竞争网络架构缓解训练方差; 在训练过程中, 对全连接层参数执行软重置操作, 以缓解局部最优现象。仿真实验结果证明, 相较于传统方法和未改进的强化学习算法, 文中所提算法环境适应性更强, 同时能实现更高的遍历覆盖率。
-
关键词:
- 自主水下航行器 /
- 路径规划 /
- 强化学习 /
- 改进的蒙豪森深度Q网络
Abstract: To improve the traversal path planning capability of autonomous undersea vehicle(AUV), this paper proposed an AUV traversal path planning algorithm based on an advanced Munchausen deep Q-network(AMDQN). First, the paper established a traversal planning environment suitable for practical scenarios. On the one hand, the ray coverage method was used instead of the traditional rectangular grid modeling method to improve modeling accuracy; on the other hand, the optional action set of the AUV was constrained to limit its maneuverability. Second, each component of the algorithm was carefully designed under the above environment. Specifically, in the state space design, the paper fused vague global environmental information, accurate local environmental information centered on the AUV, and the AUV’s own position information to systematically represent the environment. In the reward function design, rule penalty and edge guidance were introduced, enabling the AUV to stably improve coverage along environmental edges. For the parameter update method, an adaptive temperature parameter update strategy was designed based on DQN, while the multi-step reward and a dueling network architecture were introduced to alleviate training variance. During the training process, a soft reset of fully connected layer parameters was adopted to mitigate the local optimum problem. The simulation results show that the proposed algorithm has stronger environmental adaptability and higher traversal coverage than traditional methods and unimproved reinforcement learning algorithms. -
图 5 有障碍和无障碍环境中温度参数的动态调整情况
Figure 5. Dynamic adjustment of temperature parameters under obstacle-free and obstacle environments
图 7 无障碍环境5种算法奖励函数曲线
Figure 7. Five algorithm reward function curves for barrier-free environment
图 8 有障碍环境5种算法奖励函数曲线
Figure 8. Reward function curves of five algorithms in obstacle environment
图 9 无障碍环境不同算法组件覆盖率
Figure 9. Component coverage of different algorithms in barrier free environment
图 10 有障碍环境不同算法组件覆盖率
Figure 10. Coverage of different algorithm components in obstacle environment
图 13 无障碍环境下2种算法可视化路径
Figure 13. Visualization path of two algorithms in barrier free environment
图 14 圆形障碍环境下两种算法的可视化路径
Figure 14. Visual path of two algorithms in circular obstacle environment
图 15 多障碍环境下2种算法的可视化路径
Figure 15. Visualized paths of two algorithms in multi-obstacle environments
图 16 U型障碍环境下2种算法可视化路径
Figure 16. Visualized paths of two algorithms in a U-shaped obstacle environment
图 17 AUV探测范围变化和海洋环境存在扰动时的遍历轨迹
Figure 17. Traversal trajectories with varying AUV detection ranges and disturbances in marine environments
表 1 算法伪代码
Table 1. Algorithm pseudocode
基于AMDQN的遍历规划算法, 最终输出柔性策略$ {{\pi} } $ 初始化: 经验池$ D $, 状态价值函数$ \theta $, 目标状态价值函数$ {\theta }^{-}=\theta $, 超参数$ \alpha ,\tau ,\varsigma $等; 初始化: AUV与障碍物位姿信息; for i to Emax: $ \text{ for }i\text{ to }{T}_{\text{end}}\colon $ 根据式(4)计算状态空间; 根据策略采样$ {a}_{t}\sim {\pi} $; 根据式(2)、式(10)、式(23)计算N步状态$ {s}_{t+N} $, 折扣奖励和$ r_{t}^{M,N} $, 并将$ ({{\boldsymbol{s}}}_{t},{a}_{t}, r_{t}^{M,N}, {{\boldsymbol{s}}}_{t+N}) $储存在经验池$ D $中; $ \text{if len }(D)> {D}_{s}\colon $ 根据式(24)、式(25)为损失函数更新参数$ \theta $; 根据式(27)为损失函数更新参数$ \tau $; 根据式(28)计算目标参数$ {\theta }^{-} $; $ \text{if }i> {E}_{s}\text{ and }i{\text{%}} 20=0\colon $ 根据式(39)软重置部分网络参数; $ \text{if Done}\colon $ $ \text{break} $; $ \mathrm{end}~\text{for } $ $ \mathrm{end}~\text{for} $ 
下载: 导出CSV
表 2 不同超参数下算法性能
Table 2. Performance of the algorithm with different hyperparameters
参数类型 参数值 首次遍历
成功回合首次遍历
成功时间/s池化后图像维度 10×10 — — 20×20 595 1266.42 50×50 1312 4938.65 奖励步数 1 942 2034.72 3 595 1266.42 5 — — 卷积池化层数 1 — — 3 595 1266.42 5 602 1632.66
下载: 导出CSV
表 3 不同奖励超参数下算法性能表
Table 3. Performance of the algorithm with different reward hyperparameters
超参数值 首达幕数 路径长度/m 转弯次数 [−1, −1, −0.75] 821 447 81 [−0.5, −1, −0.75] 766 502.5 114 [−2, −1, −0.25] — — — [−2, −1, −1.5] 1215 414 54 [−2, −0.5, −0.75] 802 493.5 105 [−2, −1.5, −0.75] 1010 421.5 76 [−2, −1, −0.75] 824 426 66
下载: 导出CSV
表 4 算法超参数表
Table 4. Algorithm Hyperparameter Table
超参数 值 超参数 值 经验池容量 60 000 网络学习率 0.0003 $ {T}_{\max } $ 350 $ \gamma $ 0.99 $ \overline{\mathcal{H}} $ 0.25 $ {a}_{1} $ 0.15 $ {a}_{2} $ -1 $ \alpha $ 0.1 $ {R}_{1} $ 50 $ {R}_{2} $ −5 $ {\alpha }_{f} $ 0.05 $ \varsigma $ 0.001
下载: 导出CSV
表 5 不同算法组件下算法性能表
Table 5. Performance table of algorithms under different algorithm components
算法 无障碍 有障碍 $ {\mathrm{D}}_{90} $ $ {\mathrm{D}}_{99} $ $ {\mathrm{D}}_{90} $ $ {\mathrm{D}}_{99} $ SAC 960 / 960 / MDQN 740 / 820 / AMDQN(不含奖励修正) 720 / 940 / AMDQN(不含奖励修正与局部地图) 1110 / 1180 / AMDQN 960 1320 640 900
下载: 导出CSV
表 6 不同算法的训练时长和单步推理时长
Table 6. Training and single-step inference time for different algorithms
算法 单步推理时长/ms 训练时长/s SAC 1.13 3627.45 MDQN 1.02 3125.12 PPO-penalty 1.09 2746.32 AMDQN 1.12 1848.51
下载: 导出CSV
表 7 无障碍环境不同算法覆盖率与路径长度
Table 7. Coverage and path length of different algorithms in barrier free environment
算法 覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {A}^{*} $ 99.16 507 38 SAC 97.71 525 79 MDQN 99.02 460 63 PPO-penalty 89.86 525 80 AMDQN 99.02 430 42
下载: 导出CSV
表 8 圆形障碍环境不同算法的覆盖率与路径长度
Table 8. Coverage and path length of different algorithms in circular obstacle environment
算法 覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {A}^{*} $ 92.32 379.5 49 SAC 90.38 525.0 101 MDQN 97.54 525.0 87 PPO-penalty 87.40 525.0 100 AMDQN 99.07 387.0 55
下载: 导出CSV
表 9 多障碍环境不同算法的覆盖率与路径长度
Table 9. Coverage rate and planning length of different algorithms in multi-obstacle environments
算法 覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {{A}}^{*} $ 86.57 525 60 SAC 99.02 523 94 MDQN 94.81 525 117 PPO-penalty 78.11 525 102 AMDQN 99.11 403 64
下载: 导出CSV
表 10 U型障碍环境不同算法的覆盖率与路径长度
Table 10. Coverage Rate and Planning length of Different Algorithms in a U-shaped obstacle environment
算法 覆盖率/% 路径长度/m 转弯次数 往复遍历+$ {A}^{*} $ 89.45 375 68 SAC 92.48 525 75 MDQN 95.7 525 138 PPO-penalty 78.78 525 71 AMDQN 99.01 430 68
下载: 导出CSV
-
[1] 苗润龙, 庞硕, 姜大鹏, 等. 海洋自主航行器多海湾区域完全遍历路径规划[J]. 测绘学报, 2019, 48(2): 256-264.Miao R L, Pang S, Jiang D P, et al. Complete coverage path planning for autonomous marine vehicle used in multi-bay areas[J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(2): 256-264. [2] Tang G, Tang C, Zhou H, et al. R-DFS: a coverage path planning approach based on region optimal decomposition[J]. Remote Sensing, 2021, 13(8): 1525. [3] Xu A, Viriyasuthee C, Rekleitis I. Optimal complete terrain coverage using an unmanned aerial vehicle[C]//2011 IEEE International Conference on Robotics and Automation, 2011: 2513-2519. [4] 熊亿民. 基于改进蚁群算法的全向移动机器人全遍历路径规划[J]. 计算机系统应用, 2021, 30(6): 209-214.Xiong Y M. Full traversal path planning of omnidirectional mobile robot based on improved ant colony algorithm[J]. Computer Systems and Applications, 2021, 30(6): 209-214. [5] 池志猛, 李智刚, 赵洋, 等. 针对深海采矿车的遍历路径规划方法研究[J]. 矿业研究与开发, 2022, 42(7): 160-166.Chi Z M, Li Z G, Zhao Y, et al. Research on traversal path planning method for deep-sea mining vehicles[J]. Mining Research and Development, 2022, 42(7): 160-166. [6] Bouman A, Ott J, Kim S K, et al. Adaptive coverage path planning for efficient exploration of unknown environments[C]//2022 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), 2022: 11916-11923. [7] 温志文, 杨春武, 蔡卫军, 等. 复杂环境下UUV完全遍历路径规划方法[J]. 水下无人系统学报, 2017, 25(1): 22-26.Wen Z W, Yang C W, Cai W J, et al. A complete coverage path planning method of UUV under complex environment[J]. Journal of Unmanned Undersea Systems, 2017, 25(1): 22-26. [8] 史方青, 黄华, 张昊, 等. 玻璃幕墙清洗机器人内螺旋完全遍历路径规划研究[J]. 哈尔滨工程大学学报, 2024, 45(6): 1170-1178.Shi F Q, Huang H, Zhang H, et al. Study on the path planning of the internal spiral complete traversal for glass-curtain wall-cleaning robots[J]. Journal of Harbin Engineering University, 2024, 45(6): 1170-1178. [9] 吕霞付, 程启忠, 李森浩, 等. 基于改进A*算法的无人船完全遍历路径规划[J]. 水下无人系统学报, 2019, 27(6): 695-703.Lü X F, Cheng Q Z, Li S H, et al. Unmanned surface vehicle full traversal path planning based on improved A* algorithm[J]. Journal of Unmanned Undersea Systems, 2019, 27(6): 695-703. [10] 时天龙. 基于A*算法的无人船完全遍历路径规划研究[D]. 大连: 大连海事大学, 2024: 32-38. [11] 王岩, 李春书, 张福龙. 一种改进的爬壁机器人全遍历路径规划[J]. 计算机仿真, 2023, 40(7): 447-452.Wang Y, Li C S, Zhang F L. An improved full traversal path planning for wall climbing robot[J]. Computer Simulation, 2023, 40(7): 447-452. [12] 孟浩德, 吴征天, 吴闻笛, 等. 基于记忆模拟退火算法的扫地机器人遍历路径规划[J]. 计算机与数字工程, 2024, 52(3): 821-826, 857.Meng H D, Wu Z T, Wu W D, et al. Traversal path planning of sweeping robot based on memory simulated annealing algorithm[J]. Computer and Digital Engineering, 2024, 52(3): 821-826, 857. [13] Lin B, Han G H, Song C C, et al. Traversal path planning and simulation of robot based on radiation scanning[J]. Journal of System Simulation, 2021, 33(1): 83-90. [14] Carvalho J P, Aguiar A P. Deep reinforcement learning for zero-shot coverage path planning with mobile robots[J]. IEEE/CAA Journal of Automatica Sinica, 2025, 12(8): 1594-1609. [15] Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518(7540): 529-533. [16] Chen Y, Lu Z M, Cui J L, et al. A complete coverage path planning algorithm for lawn mowing robots based on deep reinforcement learning[J]. Sensors, 2025, 25(2): 416. [17] Xing B, Wang X, Yang L, et al. An algorithm of complete coverage path planning for unmanned surface vehicle based on reinforcement learning[J]. Journal of Marine Science and Engineering, 2023, 11(3): 645. [18] Ai B, Jia M, Xu H, et al. Coverage path planning for maritime search and rescue using reinforcement learning[J]. Ocean Engineering, 2021, 241: 110098. [19] Heydari J, Saha O, Ganapathy V. Reinforcement learning-based coverage path planning with implicit cellular decomposition[PP/OL]. (2021-10-18)[2026-07-21]. https://doi.org/10.48550/arXiv.2110.09018. [20] Van H H, Guez A, Silver D. Deep reinforcement learning with double Q-learning[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2016, 30(1): 2094-2100. [21] Theile M, Bayerlein H, Nai R, et al. UAV coverage path planning under varying power constraints using deep reinforcement learning[C]//2020 IEEE/RSJ International Conference on Intelligent Robots and Systems(IROS), 2020: 1444-1449. [22] Zheng T, Jin Y, Zhao H, et al. Deep reinforcement learning based coverage path planning in unknown environments[C]//2024 6th International Conference on Frontier Technologies of Information and Computer(ICFTIC), 2024: 1608-1611. [23] Carvalho J P, Aguiar A P. A reinforcement learning based online coverage path planning algorithm[C]//2023 IEEE International Conference on Autonomous Robot Systems and Competitions(ICARSC), 2023: 81-86. [24] Jonnarth A, Zhao J, Felsberg M. Learning coverage paths in unknown environments with deep reinforcement learning[PP/OL]. (2024-06-07)[2026-07-21]. https://doi.org/10.48550/arXiv.2306.16978. [25] Haarnoja T, Zhou A, Abbeel P, et al. Soft actor-critic: off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//International Conference on Machine Learning, 2018: 1861-1870. [26] Wijegunawardana I D, Samarakoon S M B P, Muthugala M A V J, et al. Risk-aware complete coverage path planning using reinforcement learning[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2025, 55(4): 2476-2488. [27] Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[PP/OL]. (2017-07-20)[2026-08-20]. https://doi.org/10.48550/arXiv.1707.06347. [28] Von B M. An open-source benchmark simulator: control of a BlueROV2 underwater robot[J]. Journal of Marine Science and Engineering, 2022, 12(10): 1898. [29] 王路, 朱衍明, 宋家平, 等. 基于激光雷达的无人船水面路径规划方法研究[J]. 指挥控制与仿真, 2026, 48(1): 36-44.WANG L, ZHU Y M, SONG J P, et al. Research on method of path planning of unmanned surface vehicle based on LiDAR[J]. Command Control & Simulation, 2026, 48(1): 36-44. [30] Vieillard N, Pietquin O, Geist M. Munchausen reinforcement learning[C]//Advances in Neural Information Processing Systems, 2020, 33: 4235-4246. -
下载:
点击查看大图
计量
- 文章访问数: 435
- HTML全文浏览量: 279
- PDF下载量: 39
- 被引次数: 0
