基于延迟鲁棒多智能体强化学习的协同围捕方法

付松琛; 白乐天; 赵少靖; 蒙傲萌; 梁宏; 黎塔

doi:10.11993/j.issn.2096-3920.2025-0173

基于延迟鲁棒多智能体强化学习的协同围捕方法

doi: 10.11993/j.issn.2096-3920.2025-0173

付松琛^{1, 2,},
白乐天^{1, 2,},
赵少靖^{1, 2,},
蒙傲萌^{1, 2,},
梁宏^{1, 2,},
黎塔^{1, 2,}

1.
中国科学院声学研究所语音与智能信息处理实验室, 北京, 100190
2.
中国科学院大学, 北京, 100049

详细信息

作者简介:
付松琛(1999-), 男, 在读博士, 主要研究方向为多智能体强化学习

中图分类号: TJ630; U664.82
计量
- 文章访问数: 7
- HTML全文浏览量: 3
- PDF下载量: 3
- 被引次数: 0
出版历程
- 收稿日期: 2025-12-26
- 修回日期: 2026-01-06
- 录用日期: 2026-01-27
- 网络出版日期: 2026-07-20

A Delay-Robust Multi-Agent Reinforcement Learning Approach for Cooperative Target Encirclement

FU Songchen^{1, 2
,},
BAI Letian^{1, 2
,},
ZHAO Shaojing^{1, 2
,},
MENG Aomeng^{1, 2
,},
LIANG Hong^{1, 2
,},
LI Ta^{1, 2
,}

1.
Laboratory of Speech and Intelligent Information Processing, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China

摘要

摘要: 针对多个无人水下航行器(UUV)协同作业时因水声通信机制导致的延迟观测问题, 提出了一种基于延迟鲁棒多智能体强化学习的协同围捕方法。首先,在常用通信场景下分析延迟观测的产生原理; 其次, 利用二维纳维-斯托克斯方程对湍流场进行建模, 搭建仿真环境, 模拟真实任务场景; 然后, 提出延迟鲁棒多智能体强化学习方法, 并详细介绍各组成模块; 在此基础上, 设计围捕任务的奖励函数、不同网络的模型结构以及训练方法, 完成不同任务、不同延迟条件下的消融实验。实验结果表明, 文中方法能有效应对延迟观测问题, 并且在不同程度的延迟下保持较好的表现, 部分任务上接近无时隙情况的理论性能上限。此外, 消融实验证明了所提出方法中各组成模块在对抗延迟观测时的有效性, 为延迟观测下的多UUV协作策略研究提供了新的理论支撑与技术路径。
- 无人水下航行器 /
- 目标围捕 /
- 多智能体强化学习 /
- 延迟观测 /
- 课程学习
Abstract: To address delayed observations caused by acoustic communication in cooperative operations of multiple unmanned underwater vehicles (UUVs), a delay-robust multi-agent reinforcement learning based cooperative encirclement method is proposed. First, the mechanism of delayed observations is analyzed under common communication scenarios. Second, a turbulent flow field is modeled using the two-dimensional Navier–Stokes equations to construct a simulation environment that reflects realistic task settings. Then, a delay-robust multi-agent reinforcement learning method is proposed, and its constituent modules are described in detail. On this basis, the reward functions for the encirclement task, the network architectures, and the training procedures are designed, and ablation studies are conducted under different tasks and delay conditions. Experimental results demonstrate that the proposed method effectively handles delayed observations and maintains strong performance under varying delay levels, approaching the theoretical upper bound of the no-delay case in some tasks. Furthermore, the ablation results verify the effectiveness of each module in mitigating delayed observations, providing new theoretical support and a practical methodology for multi-UUV cooperative strategies under delayed observations.
- unmanned undersea vehicle /
- target encirclement /
- multi-agent reinforcement learning /
- delayed observation /
- curriculum learning.

HTML全文

图 1 UUV探测与通信范围

Figure 1. UUV detection and communication range

下载: 全尺寸图片幻灯片

图 2 正三角形围捕示意图

Figure 2. Equilateral triangle encirclement

下载: 全尺寸图片幻灯片

图 3 3个UUV围捕2个目标的初始化示意图

Figure 3. Initialization of three UUVs surrounding two targets

下载: 全尺寸图片幻灯片

图 4 基于Transformer结构的SSMR示意图

Figure 4. SSMR based on Transformer structure

下载: 全尺寸图片幻灯片

图 5 延迟鲁棒多智能体强化学习方法框架图

Figure 5. Delay robust method framework diagram

下载: 全尺寸图片幻灯片

图 6 不同方法训练过程中的奖励值变化

Figure 6. The variation of reward values during the training process of different methods.

下载: 全尺寸图片幻灯片

图 7 策略网络的输入观测与真实值的差距

Figure 7. The difference between the input observation of the policy network and the real value

下载: 全尺寸图片幻灯片

表 1 环境参数

Table 1. Parameters of environment

参数	符号	取值
地图左下角坐标	$ {p}_{\text{lb}} $	-3000, -3000
地图右上角坐标	$ {p}_{\text{rt}} $	3000, 3000
涡旋数量	$ {N}_{f} $	5
涡旋强度	$ {\varOmega }_{c} $	8000 m²/s
最小涡旋半径	$ r_{c}^{\min } $	500 m
最大涡旋半径	$ r_{c}^{\max } $	1000 m
UUV通信范围半径	$ {r}_{\mathrm{comm}} $	2000 m
UUV探测范围半径	$ {r}_{\mathrm{de}\text{tc}} $	1500 m
UUV探测范围开角	$ {\theta }_{\mathrm{de}\text{tc}} $	1.57 rad
UUV最大对水速率	$ v_{\text{water}}^{\max } $	6 m/s
UUV最大加速度	$ {a}_{\text{max}} $	0.15 m/s²
UUV最大转向角速度	$ {\omega }_{\text{max}} $	0.1 rad/s
UUV初始化圆半径	$ r_{u}^{\text{init}} $	600 m
目标最小速率	$ v_{\text{target}}^{\min } $	0 m/s
目标最大速率	$ v_{\text{target}}^{\max } $	3 m/s
目标朝向噪声强度	$ {\sigma }_{n} $	0.03
目标初始化圆环内径	$ r_{t}^{\text{in}} $	1200 m
目标初始化圆环外径	$ r_{t}^{\text{out}} $	2400 m
目标初始化最小间距	$ {d}_{\text{sep}} $	1600 m
围捕质量阈值	$ {s}_{\text{thre}} $	0.3
单步仿真时间	$ \Delta t $	30 s
速度衰减系数	$ {\gamma }_{\text{water}} $	0.99

下载: 导出CSV

表 2 奖励函数参数

Table 2. Parameters of reward function

参数	符号	取值
步数惩罚	$ {R}_{\text{step}} $	0.01
越界惩罚	$ {R}_{\text{bound}} $	1
碰撞惩罚	$ R_{\text{colli}}^{\text{fail}} $	5
碰撞判定距离	$ D_{\text{colli}}^{\text{fail}} $	50 m
碰撞警告基础惩罚	$ R_{\text{colli}}^{\text{warn}} $	1
碰撞警告最大距离	$ D_{\text{colli}}^{\text{warn}} $	500 m
阵型基础奖励	$ R_{\text{form}}^{\text{base}} $	1
角度基础奖励	$ R_{\text{angle}}^{\text{base}} $	1
捕获基础奖励	$ R_{\text{cap}}^{\text{base}} $	100
围捕数量增益系数	$ {g}_{\text{cap}} $	1.2

下载: 导出CSV

表 3 3种任务上训练时的延迟条件

Table 3. Delay conditions during training on 3 tasks

任务	时隙长度	丢包率	描述
3围1	30 s	20%	任务简单, 延迟低
3围2	60 s	0%	任务难, 延迟高
6围3	60 s	0%	任务最难, 延迟最高

下载: 导出CSV

表 4 3个UUV围捕1个目标场景下的评估结果

Table 4. Evaluation results under three UUVs encircle one target scenario

方法	围捕数量				围捕质量
方法	1	2	3	4	1	2	3	4
Oracle	0.95(0.04)	0.91(0.05)	0.85(0.07)	0.76(0.08)	0.81(0.03)	0.78(0.04)	0.72(0.06)	0.63(0.07)
Oracle-DA	0.97(0.03)	0.9(0.05)	0.82(0.07)	0.7(0.09)	0.86(0.03)	0.79(0.05)	0.72(0.07)	0.6(0.07)
Oracle-DA-CL	0.94(0.04)	0.88(0.06)	0.79(0.06)	0.69(0.07)	0.81(0.03)	0.76(0.05)	0.69(0.06)	0.6(0.06)
Base	0.92(0.05)	0.91(0.05)	0.87(0.06)	0.79(0.07)	0.78(0.04)	0.77(0.04)	0.74(0.05)	0.66(0.06)
Base-DA	0.94(0.04)	0.93(0.04)	0.9(0.05)	0.87(0.06)	0.8(0.04)	0.8(0.03)	0.78(0.05)	0.74(0.05)
Base-DA-CL	0.95(0.04)	0.94(0.04)	0.92(0.05)	0.87(0.06)	0.81(0.03)	0.81(0.04)	0.8(0.04)	0.75(0.05)
MSSR-DA-CL	0.94(0.04)	0.95(0.04)	0.9(0.05)	0.86(0.06)	0.8(0.03)	0.81(0.03)	0.77(0.04)	0.74(0.05)
SSMR-DA-CL	0.95(0.04)	0.94(0.04)	0.92(0.05)	0.89(0.05)	0.82(0.03)	0.82(0.04)	0.8(0.04)	0.77(0.05)

	围捕耗时				碰撞次数
	1	2	3	4	1	2	3	4
Oracle	18.05(1.52)	20.55(1.51)	22.4(1.76)	23.9(2.08)	0.02(0.02)	0.02(0.03)	0.03(0.03)	0.04(0.04)
Oracle-DA	18.56(1.58)	21.57(1.74)	24.26(1.83)	26.29(2.24)	0.01(0.02)	0.02(0.02)	0.03(0.03)	0.03(0.03)
Oracle-DA-CL	17.93(1.46)	20.98(1.65)	22.98(2.28)	23.63(2.16)	0.02(0.02)	0.02(0.03)	0.02(0.02)	0.03(0.03)
Base	18.6(1.48)	19.76(1.46)	20.6(1.64)	21.84(1.85)	0.02(0.02)	0.02(0.02)	0.02(0.02)	0.03(0.03)
Base-DA	19.01(1.63)	20.32(1.73)	21.65(2.03)	22.83(1.96)	0.02(0.02)	0.02(0.03)	0.02(0.03)	0.03(0.03)
Base-DA-CL	19.29(1.66)	20.74(1.65)	21.47(1.65)	23.81(1.99)	0.01(0.02)	0.02(0.02)	0.01(0.02)	0.02(0.02)
MSSR-DA-CL	18.93(1.6)	19.79(1.57)	20.62(1.72)	21.37(1.9)	0.02(0.02)	0.01(0.02)	0.02(0.02)	0.02(0.02)
SSMR-DA-CL	18.27(1.62)	19.12(1.66)	20.14(1.72)	20.94(1.69)	0.01(0.02)	0.01(0.02)	0.02(0.02)	0.02(0.03)

下载: 导出CSV

表 5 3个UUV围捕2个目标场景下的评估结果

Table 5. Evaluation results under three UUVs encircle two targets scenario

方法	围捕数量				围捕质量
方法	1	2	3	4	1	2	3	4
Oracle	1.51(0.1)	1.32(0.11)	1.2(0.13)	0.99(0.13)	0.79(0.03)	0.73(0.05)	0.68(0.06)	0.59(0.06)
Oracle-DA	1.61(0.09)	1.39(0.11)	1.25(0.13)	1.03(0.12)	0.82(0.03)	0.77(0.04)	0.72(0.05)	0.62(0.06)
Oracle-DA-CL	1.54(0.1)	1.34(0.11)	1.18(0.13)	0.9(0.13)	0.79(0.03)	0.75(0.04)	0.69(0.06)	0.57(0.07)
Base	1.27(0.1)	1.25(0.1)	1.23(0.11)	1.12(0.11)	0.74(0.04)	0.74(0.05)	0.74(0.04)	0.69(0.05)
Base-DA	1.38(0.11)	1.34(0.11)	1.31(0.12)	1.2(0.12)	0.76(0.04)	0.76(0.04)	0.75(0.04)	0.7(0.05)
Base-DA-CL	1.42(0.12)	1.36(0.11)	1.33(0.11)	1.19(0.12)	0.78(0.04)	0.76(0.04)	0.75(0.04)	0.7(0.05)
MSSR-DA-CL	1.38(0.11)	1.39(0.12)	1.36(0.1)	1.27(0.11)	0.77(0.04)	0.78(0.04)	0.77(0.04)	0.75(0.05)
SSMR-DA-CL	1.46(0.1)	1.43(0.11)	1.44(0.12)	1.36(0.11)	0.78(0.04)	0.78(0.04)	0.78(0.04)	0.76(0.04)

	围捕耗时				碰撞次数
	1	2	3	4	1	2	3	4
Oracle	20.02(1.32)	22.37(1.5)	23.41(1.73)	24.29(2.12)	0.05(0.04)	0.06(0.04)	0.07(0.04)	0.08(0.05)
Oracle-DA	20.23(1.08)	22.59(1.2)	23.72(1.59)	25.19(2.13)	0.04(0.04)	0.05(0.04)	0.06(0.04)	0.07(0.05)
Oracle-DA-CL	20.07(1.14)	22.15(1.38)	23.58(1.69)	25.15(2.07)	0.05(0.04)	0.06(0.04)	0.07(0.04)	0.08(0.04)
Base	22.23(1.6)	22.95(1.55)	22.95(1.58)	23.94(1.56)	0.06(0.04)	0.07(0.04)	0.07(0.04)	0.08(0.05)
Base-DA	21.4(1.44)	22.02(1.33)	22.09(1.37)	23.03(1.61)	0.06(0.04)	0.06(0.04)	0.06(0.04)	0.06(0.04)
Base-DA-CL	21.41(1.35)	22.44(1.39)	22.68(1.32)	23.89(1.71)	0.05(0.04)	0.05(0.04)	0.06(0.04)	0.07(0.04)
MSSR-DA-CL	21.33(1.19)	21.19(1.3)	21.38(1.32)	22.28(1.35)	0.06(0.04)	0.06(0.04)	0.05(0.04)	0.06(0.04)
SSMR-DA-CL	21.24(1.23)	21.77(1.49)	21.48(1.4)	22.4(1.35)	0.05(0.04)	0.05(0.04)	0.05(0.04)	0.05(0.04)

下载: 导出CSV

表 6 6个UUV围捕3个目标场景下的评估结果

Table 6. Evaluation results under six UUVs encircle three targets scenario

方法	围捕数量				围捕质量
方法	1	2	3	4	1	2	3	4
Oracle	1.13(0.14)	0.98(0.14)	0.86(0.13)	0.79(0.14)	0.61(0.06)	0.53(0.07)	0.49(0.06)	0.45(0.07)
Oracle-DA	1.44(0.16)	1.03(0.16)	0.77(0.14)	0.7(0.15)	0.66(0.05)	0.52(0.06)	0.41(0.07)	0.38(0.07)
Oracle-DA-CL	1.56(0.18)	1.24(0.18)	0.91(0.15)	0.81(0.15)	0.67(0.05)	0.57(0.07)	0.47(0.07)	0.43(0.07)
Base	0.7(0.13)	0.81(0.13)	0.83(0.12)	0.83(0.13)	0.45(0.08)	0.49(0.06)	0.5(0.07)	0.5(0.07)
Base-DA	0.78(0.14)	0.91(0.12)	1.05(0.14)	0.99(0.13)	0.48(0.07)	0.54(0.06)	0.6(0.06)	0.58(0.06)
Base-DA-CL	1.07(0.15)	1.21(0.14)	1.22(0.15)	1.16(0.14)	0.58(0.06)	0.62(0.06)	0.63(0.06)	0.6(0.06)
MSSR-DA-CL	1.32(0.16)	1.41(0.16)	1.41(0.16)	1.35(0.16)	0.63(0.05)	0.65(0.05)	0.65(0.06)	0.64(0.06)
SSMR-DA-CL	1.48(0.17)	1.54(0.18)	1.51(0.15)	1.45(0.15)	0.66(0.05)	0.68(0.05)	0.67(0.06)	0.66(0.05)

	围捕耗时				碰撞次数
	1	2	3	4	1	2	3	4
Oracle	18.96(1.86)	20.38(2.96)	21.03(3.31)	20.55(2.71)	0.16(0.06)	0.2(0.07)	0.2(0.07)	0.22(0.08)
Oracle-DA	16.29(1.56)	18.48(2.36)	20.09(2.63)	19.89(2.65)	0.22(0.08)	0.34(0.08)	0.38(0.09)	0.41(0.09)
Oracle-DA-CL	16.65(1.42)	19.11(2.96)	20.31(2.54)	20.41(3.04)	0.17(0.06)	0.19(0.07)	0.21(0.06)	0.21(0.07)
Base	20.8(3.14)	19.07(3.14)	19.08(2.39)	19.58(2.38)	0.23(0.07)	0.25(0.08)	0.26(0.08)	0.25(0.07)
Base-DA	18.01(2.27)	16.57(2.27)	17.04(2.19)	17.35(2.05)	0.22(0.07)	0.21(0.07)	0.2(0.08)	0.2(0.07)
Base-DA-CL	16.87(2.01)	16.57(1.75)	16.94(1.89)	17.44(1.98)	0.21(0.08)	0.21(0.08)	0.21(0.08)	0.21(0.07)
MSSR-DA-CL	17.72(1.68)	17.01(1.74)	17.63(1.88)	17.68(1.73)	0.18(0.07)	0.19(0.07)	0.2(0.07)	0.19(0.07)
SSMR-DA-CL	16.51(1.3)	16.28(1.66)	16.62(1.57)	16.6(1.6)	0.18(0.08)	0.18(0.07)	0.2(0.07)	0.2(0.08)

下载: 导出CSV

参考文献(31)

[1]	胡桥, 赵振轶, 冯豪博, 等. AUV智能集群协同任务研究进展[J]. 水下无人系统学报, 2023, 31(2): 189-200. Hu Q, Zhao Z Y, Feng H B, et al. Progress of AUV intelligent swarm collaborative task[J]. Journal of Unmanned Undersea Systems, 2023, 31(2): 189-200.
[2]	陈昭, 丁一杰, 张治强. 无人潜航器发展历程及运用优势研究[J]. 舰船科学技术, 2024, 46(23): 98-102. Chen Z, Ding Y J, Zhang Z Q. Research on the development history and application advantages of unmanned underwater vehicle[J]. Ship Science and Technology, 2024, 46(23): 98-102.
[3]	韦良才, 李东升. 无人潜航器集群作业的应用[J]. 电子技术, 2024, 53(9): 310-311. Wei L C, Li D S. Application of unmanned underwater vehicle cluster operation[J]. Electronic Technology, 2024, 53(9): 310-311.
[4]	王科翔, 卢野, 李银城, 等. 面向海战场下无人集群作战发展研究综述[J]. 现代防御技术, 2025, 53(3): 11-22. Wang K X, Lu Y, Li Y C, et al. Review of unmanned swarm operations development for maritime battlefields[J]. Modern Defence Technology, 2025, 53(3): 11-22.
[5]	王光诚. 基于多智能体强化学习的无人水下航行器决策方法研究[D]. 吉林大学, 2024: 1-2.
[6]	林学航. 基于强化学习的无人潜航器三维路径规划方法研究[D]. 哈尔滨工程大学, 2024: 5.
[7]	赵剑楠, 覃琪琪, 李云, 等. 基于分布式强化学习的AUV水下三维洋流目标跟踪控制算法[J]. 电信科学, 2025, 41(10): 88-101. Zhao J N, Qin Q Q, Li Y, et al. Distributed reinforcement learning-based AUV 3D underwater current target tracking control algorithm[J]. Telecommunication Science, 2025, 41(10): 88-101.
[8]	杨晓康. 基于深度强化学习的多智能体协同围捕控制[D]. 杭州电子科技大学, 2025: 2.
[9]	王景璟, 魏维, 王永越, 等. 水下自主潜航器集群协同围捕技术[J]. 指挥与控制学报, 2023, 9(6): 637-650. Wang J J, Wei W, Wang Y Y, et al. Autonomous underwater vehicle swarm collaborative hunting technologies[J]. Journal of Command and Control, 2023, 9(6): 637-650.
[10]	赵少靖, 付松琛, 白乐天, 等. 基于自适应多目标优化的UUV全覆盖路径规划方法[J]. 水下无人系统学报, 2025, 33(3): 459-472. Zhao S J, Fu S C, Bai L T, et al. Adaptive multi-objective optimization-based coverage path planning method for UUVs[J]. Journal of Unmanned Undersea Systems, 2025, 33(3): 459-472.
[11]	刘千里, 张松, 李乐琦. 水下无人通信载荷技术综述与应用前景[J]. 舰船电子工程, 2025, 45(1): 7-12. Liu QL, Zhang S, Li L Q. Overview of underwater unmanned communication payload technology and application[J]. Ship Electronic Engineering, 2025, 45(1): 7-12.
[12]	董佳明. 基于多智能体强化学习的多UUV博弈围捕方法研究[D]. 哈尔滨工程大学, 2024: 6-7.
[13]	Wu G, Xu T, Sun Y, et al. Review of multiple unmanned surface vessels collaborative search and hunting based on swarm intelligence[J]. International journal of advanced robotic systems, 2022, 19(2): 17298806221091885.
[14]	Zhao Z, Hu Q, Feng H, et al. A cooperative hunting method for multi-AUV swarm in underwater weak information environment with obstacles[J]. Journal of marine science and engineering, 2022, 10(9): 1266. doi: 10.3390/jmse10091266
[15]	Li J, Lu H, Zhang H, et al. Dynamic target hunting under autonomous underwater vehicle (AUV) motion planning based on improved dynamic window approach (DWA)[J]. Journal of Marine Science and Engineering, 2025, 13(2): 221. doi: 10.3390/jmse13020221
[16]	Cao X, Liu W, Ren L. Underwater target capture based on heterogeneous unmanned system collaboration[J]. IEEE Transactions on Intelligent Vehicles, 2024.
[17]	Jiang B, Wang Y, Kong F, et al. Multi-autonomous underwater vehicle trajectory planning in ocean current based on hierarchical hunting and evolutionary learning[J]. ACM Transactions on Intelligent Systems and Technology, 2025, 16(5): 1-26. doi: 10.1145/3757928
[18]	Zhang M, Chen H, Cai W. Collaborative hunting method of multi-AUV in 3-D IoUT: Searching, tracking, and encirclement keeping[J]. IEEE Internet of Things Journal, 2025, 12(8): 10958-10973. doi: 10.1109/JIOT.2024.3514632
[19]	Wang Z, Du J, Jiang C, et al. Task scheduling for distributed AUV network target hunting and searching: An energy-efficient AoI-aware DMAPPO approach[J]. IEEE Internet of Things Journal, 2022, 10(9): 8271-8285.
[20]	Hou X, Xing T, Wang J, et al. Age of information-aware multi-objective optimization for heterogeneous UAV-USV-UUV Networks in Underwater Target Hunting[J]. IEEE Transactions on Mobile Computing, 2025.
[21]	Chen J, Wang Y, Zhang Y, et al. Extrinsic-and-intrinsic reward-based multi-agent reinforcement learning for multi-UAV cooperative target encirclement[J]. IEEE Transactions on Intelligent Transportation Systems, 2025.
[22]	Wei W, Wang J, Du J, et al. Differential game-based deep reinforcement learning in underwater target hunting task[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[23]	伍雨辰. 基于深度强化学习的水声传感器网络时隙可变MAC协议[D]. 天津大学, 2022: 2.
[24]	Rashid T, Samvelyan M, De Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020, 21(178): 1-51.
[25]	Oliehoek F A, Amato C. A concise introduction to decentralized POMDPs[M]. Cham, Switzerland: Springer International Publishing, 2016: 14-15.
[26]	Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518: 529-533. doi: 10.1038/nature14236
[27]	Tampuu A, Matiisen T, Kodelja D, et al. Multiagent cooperation and competition with deep reinforcement learning[J]. PLOS One, 2017, 12(4): e0172395. doi: 10.1371/journal.pone.0172395
[28]	Sunehag P, Lever G, Gruslys A, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward[C]//Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. Stockholm, 2018: 2085-2087.
[29]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in neural information processing systems, 2017, 30: 6000-6010. doi: 10.1007/978-3-031-84300-6_13
[30]	Dey R, Salem F M. Gate-variants of gated recurrent unit (GRU) neural networks[C]//2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS). Massachusetts, 2017: 1597-1600.
[31]	Ha D, Dai A, Le Q V. Hypernetworks[EB/OL]. 2016[2025-12-22]. https://arxiv.org/abs/1609.09106.