Energy-optimal Path Planning Algorithm for Unmanned Surface Vessel Based on Reinforcement Learning

LI Peijuan; YAN Tingwu; YANG Shutao; LI Rui; DU Junfeng; QIAN Fufu; LIU Yiting

doi:10.11993/j.issn.2096-3920.202203002

Volume 31 Issue 2

Apr 2023

Turn off MathJax

Article Contents

Article Navigation > Journal of Unmanned Undersea Systems > 2023 > 31(2): 237-243

LI Peijuan, YAN Tingwu, YANG Shutao, LI Rui, DU Junfeng, QIAN Fufu, LIU Yiting. Energy-optimal Path Planning Algorithm for Unmanned Surface Vessel Based on Reinforcement Learning[J]. Journal of Unmanned Undersea Systems, 2023, 31(2): 237-243. doi: 10.11993/j.issn.2096-3920.202203002

Citation:

LI Peijuan, YAN Tingwu, YANG Shutao, LI Rui, DU Junfeng, QIAN Fufu, LIU Yiting. Energy-optimal Path Planning Algorithm for Unmanned Surface Vessel Based on Reinforcement Learning[J]. Journal of Unmanned Undersea Systems, 2023, 31(2): 237-243. doi: 10.11993/j.issn.2096-3920.202203002

Citation:

LI Peijuan, YAN Tingwu, YANG Shutao, LI Rui, DU Junfeng, QIAN Fufu, LIU Yiting. Energy-optimal Path Planning Algorithm for Unmanned Surface Vessel Based on Reinforcement Learning[J]. Journal of Unmanned Undersea Systems, 2023, 31(2): 237-243. doi: 10.11993/j.issn.2096-3920.202203002

PDF( 1349 KB)

Energy-optimal Path Planning Algorithm for Unmanned Surface Vessel Based on Reinforcement Learning

doi: 10.11993/j.issn.2096-3920.202203002

Industrial center/School of Innovation and Entrepreneurship, Nanjing Institute of Technology, Nanjing 211167, China

Received Date: 2022-03-03
Accepted Date: 2022-05-11
Rev Recd Date: 2022-04-18

Available Online: 2023-02-21

Abstract

Abstract

To address path planning for unmanned surface vessels(USVs) in ocean environments affected by physical disturbances, such as ocean currents and obstacles, an energy-optimal algorithm based on improved reinforcement learning is proposed. First, a two-dimensional ocean-current model comprising multiple random vortices and a plane kinematic model of the USV is established. Then, the study determined whether a waypoint is reachable using the relative velocity relationship between the USV and the ocean current. An improved reward function, an action set, and a state set are used to obtain a global optimal path, and the B-spline method is applied to smooth the plan. Finally, numerical simulations are performed in two typical environments. The simulation results show that a path with optimal energy consumption and smoothness can be planned based on the proposed algorithm.
- unmanned surface vessel,
- path planning,
- energy optimization,
- reinforcement learning,
- ocean current model

FullText(HTML)

References(15)

References

[1]	Fan X J, Guo J Y, Liu H, et al. Improved artificial potential field method applied for AUV path planning[J]. Mathematical Problems in Engineering, 2020(1): 1-21.
[2]	Liu C, Mao Q, Chu X, et al. An improved A-star algorithm considering water current, traffic separation and berthing for vessel path planning[J]. Applied Scineces Basel, 2019, 9(6): 1057. doi: 10.3390/app9061057
[3]	Wang L, Pang S. Chemical plume tracing using an AUV based on POMDP source mapping and A-star path planning[C]//2019 MTS/IEEE Oceans Conference. Parc Chanotl, Marseille, France: IEEE, 2019.
[4]	高博, 徐德民, 张福斌, 等. 海流建模及其在路径规划中的应用[J]. 系统仿真学报, 2010, 22(4): 957-961. doi: 10.16182/j.cnki.joss.2010.04.016 Gao Bo, Xu Deming, Zhang Fubin, et al. Method of designing optimal smooth way for vehicle[J]. Journal of System Simulation, 2010, 22(4): 957-961. doi: 10.16182/j.cnki.joss.2010.04.016
[5]	向金林, 王鸿东, 欧阳子路, 等. 基于改进双向RRT的无人艇局部路径规划算法研究[J]. 中国造船, 2020, 61(1): 157-166. Xiang Jinlin, Wang Hongdong, Ouyang Zilu, et al. Algorithm of local path planning for unmanned surface vehicles based on improved Bi-RRT[J]. Shipbuilding of China, 2020, 61(1): 157-166.
[6]	Alvarez A, Caiti A. A genetic algorithm for autonomous underwater vehicle route planning in ocean environments with complex space-time variability[J]. IFAC Proceedings Volumes, 2001, 34(7): 237-242. doi: 10.1016/S1474-6670(17)35089-9
[7]	Alvarez A, Caiti A, Onken R. Evolutionary path planning for autonomous underwater vehicles in a variable ocean[J]. IEEE Journal of Oceanic Engineering, 2004, 29(2): 418-429. doi: 10.1109/JOE.2004.827837
[8]	Zeng Z, Sammut K, Lian L, et al. A comparison of optimization techniques for AUV path planning in environments with ocean currents[J]. Robotics and Autonomous Systems, 2016(82): 61-72.
[9]	Zeng Z, Lammas A, Sammut K, et al. Shell space decomposition based path planning for AUVs operating in a variable environment[J]. Ocean Engineering, 2014, 91: 181-195. doi: 10.1016/j.oceaneng.2014.09.001
[10]	马焱, 肖玉杰, 陈轶, 等. 基于改进烟花-蚁群算法的海流环境下水下无人水面艇的避障路径规划[J]. 导航与控制, 2019, 18(1): 51-59. doi: 10.3969/j.issn.1674-5558.2019.01.007 Ma Yan, Xiao Yujie, Chen Yi, et al. Obstacle avoidance path planning of unmanned underwater vehicle in ocean current environment based on improved fireworks-ant colony algorithm[J]. Navigation and Control, 2019, 18(1): 51-59. doi: 10.3969/j.issn.1674-5558.2019.01.007
[11]	Chen C, Chen X Q, Ma F, et al. A knowledge-free path planning approach for smart ships based on reinforcement learning[J]. Ocean Engineering, 2019, 189: 106299. doi: 10.1016/j.oceaneng.2019.106299
[12]	Guo S, Zhang X, Zheng Y, et al. An autonomous path planning model for unmanned ships based on deep reinforcement learning[J]. Sensors, 2020, 20(2): 426. doi: 10.3390/s20020426
[13]	Woo J, Kim N. Collision avoidance for an unmanned surface vehicle using deep reinforcement learning[J]. Ocean Engineering, 2020, 199(1): 1-16.
[14]	Garau B, Alvarez A, Oliver G. AUV navigation through turbulent ocean environments supported by onboard H-ADCP[C]//2006 IEEE International conference on robotics and automation. Orlando, FL, USA: IEEE, 2006.
[15]	Li X, Sun Z, Cao D, et al. Real-time trajectory planning for autonomous urban driving: framework, algorithms, and verifications[J]. IEEE-ASME Transactions on Mechatronics, 2016, 21(2): 740-753. doi: 10.1109/TMECH.2015.2493980