基于延迟鲁棒多智能体强化学习的协同围捕方法

付松琛; 白乐天; 赵少靖; 蒙傲萌; 梁宏; 黎塔

doi:10.11993/j.issn.2096-3920.2025-0173

基于延迟鲁棒多智能体强化学习的协同围捕方法

doi: 10.11993/j.issn.2096-3920.2025-0173

付松琛^{1, 2,},
白乐天^{1, 2,},
赵少靖^{1, 2,},
蒙傲萌^{1, 2,},
梁宏^{1, 2,},
黎塔^{1, 2,}

1.
中国科学院声学研究所语音与智能信息处理实验室, 北京, 100190
2.
中国科学院大学, 北京, 100049

详细信息

作者简介:
付松琛(1999-), 男, 在读博士, 主要研究方向为多智能体强化学习

中图分类号: TJ630; U664.82
计量
- 文章访问数: 538
- HTML全文浏览量: 261
- PDF下载量: 40
- 被引次数: 0
出版历程
- 收稿日期: 2025-12-26
- 修回日期: 2026-01-06
- 录用日期: 2026-01-27
- 网络出版日期: 2026-07-20

A Cooperative Encirclement Method Based on Delay-Robust Multi-Agent Reinforcement Learning

FU Songchen^{1, 2
,},
BAI Letian^{1, 2
,},
ZHAO Shaojing^{1, 2
,},
MENG Aomeng^{1, 2
,},
LIANG Hong^{1, 2
,},
LI Ta^{1, 2
,}

1.
Laboratory of Speech and Intelligent Information Processing, Institute of Acoustics, Chinese Academy of Sciences, Beijing 100190, China
2.
University of Chinese Academy of Sciences, Beijing 100049, China

摘要

摘要: 针对多个无人水下航行器(UUV)协同作业时因水声通信机制导致的延迟观测问题, 提出了一种基于延迟鲁棒多智能体强化学习的协同围捕方法。首先, 在常用通信场景下分析延迟观测的产生原理, 利用二维纳维-斯托克斯方程对湍流场进行建模, 搭建仿真环境, 模拟真实任务场景; 然后, 提出延迟鲁棒多智能体强化学习方法, 并详细介绍各组成模块。在此基础上设计围捕任务的奖励函数、不同网络的模型结构以及训练方法, 完成不同任务和延迟条件下的消融实验。实验结果表明, 文中方法能有效应对延迟观测问题, 在不同程度的延迟下均保持较好表现, 部分任务中接近无时隙情况的理论性能上限。消融实验证明了所提方法中各组成模块在对抗延迟观测时的有效性, 为延迟观测下的多UUV协作策略研究提供了新的理论支撑与技术路径。
- 无人水下航行器 /
- 协同围捕 /
- 多智能体强化学习 /
- 延迟观测
Abstract: To address delayed observations caused by acoustic communication in cooperative operations of multiple unmanned undersea vehicles(UUVs), a cooperative encirclement method based on delay-robust multi-agent reinforcement learning was proposed. First, the mechanism of delayed observations was analyzed under common communication scenarios. Second, a turbulent flow field was modeled using the two-dimensional Navier-Stokes equations to construct a simulation environment that reflects realistic task settings. Then, a delay-robust multi-agent reinforcement learning method was proposed, and its constituent modules were described in detail. On this basis, the reward functions for the encirclement task, the network architectures, and the training procedures were designed, and ablation studies were conducted under different tasks and delay conditions. Experimental results demonstrate that the proposed method effectively handles delayed observations and maintains strong performance under varying delay levels, approaching the theoretical upper bound of the no-delay case in some tasks. Furthermore, the ablation results verify the effectiveness of each module in mitigating delayed observations, providing new theoretical support and a practical methodology for multi-UUV cooperative strategies under delayed observations.
- unmanned undersea vehicle /
- cooperative encirclement /
- multi-agent reinforcement learning /
- delayed observation

HTML全文

图 1 UUV探测与通信范围

Figure 1. UUV detection and communication range

下载: 全尺寸图片幻灯片

图 2 正三角形围捕示意图

Figure 2. Equilateral triangle encirclement

下载: 全尺寸图片幻灯片

图 3 3个UUV围捕2个目标的初始化示意图

Figure 3. Initialization of three UUVs encircling two targets

下载: 全尺寸图片幻灯片

图 4 基于Transformer结构的SSMR示意图

Figure 4. SSMR based on Transformer structure

下载: 全尺寸图片幻灯片

图 5 延迟鲁棒MARL方法框架图

Figure 5. Framework diagram of delay-robust multi-agent RL method

下载: 全尺寸图片幻灯片

图 6 不同方法训练过程奖励值变化

Figure 6. Variation of reward values during the training process of different methods

下载: 全尺寸图片幻灯片

图 7 策略网络输入观测与真实值差距

Figure 7. The difference between the input observation of the policy network and the real value

下载: 全尺寸图片幻灯片

表 1 仿真参数

Table 1. Parameters of simulation

参数	符号	取值
地图左下角坐标	$ {p}_{\text{lb}} $	−3000, −3000
地图右上角坐标	$ {p}_{\text{rt}} $	3000, 3000
涡旋数量	$ {N}_{f} $	5
涡旋强度/(m²/s)	$ {\varOmega }_{c} $	8000
最小涡旋半径/m	$ r_{c}^{\min } $	500
最大涡旋半径/m	$ r_{c}^{\max } $	1000
UUV通信范围半径/m	$ {r}_{\mathrm{comm}} $	2000
UUV探测范围半径/m	$ {r}_{\mathrm{de}\text{tc}} $	1500
UUV探测范围开角/rad	$ {\theta }_{\mathrm{de}\text{tc}} $	1.57
UUV最大对水速率/(m/s)	$ v_{\text{water}}^{\max } $	6
UUV最大加速度/(m/s²)	$ {a}_{\text{max}} $	0.15
UUV最大转向角速度/(rad/s)	$ {\omega }_{\text{max}} $	0.1
UUV初始化圆半径/m	$ r_{u}^{\text{init}} $	600
目标最小速率/(m/s)	$ v_{\text{target}}^{\min } $	0
目标最大速率/(m/s)	$ v_{\text{target}}^{\max } $	3
目标朝向噪声强度	$ {\sigma }_{n} $	0.03
目标初始化圆环内径/m	$ r_{t}^{\text{in}} $	1200
目标初始化圆环外径/m	$ r_{t}^{\text{out}} $	2400
目标初始化最小间距/m	$ {d}_{\text{sep}} $	1600
围捕质量阈值	$ {s}_{\text{thre}} $	0.3
单步仿真时间/s	$ \Delta t $	30
速度衰减系数	$ {\gamma }_{\text{water}} $	0.99

下载: 导出CSV

表 2 奖励函数参数

Table 2. Parameters of reward function

参数	符号	取值
耗时惩罚	$ {R}_{\text{step}} $	0.01
越界惩罚	$ {R}_{\text{bound}} $	1
碰撞失败惩罚	$ R_{\text{colli}}^{\text{fail}} $	5
碰撞判定距离/m	$ D_{\text{colli}}^{\text{fail}} $	50
碰撞警告基础惩罚	$ R_{\text{colli}}^{\text{warn}} $	1
碰撞警告最大距离/m	$ D_{\text{colli}}^{\text{warn}} $	500
阵型基础奖励	$ R_{\text{form}}^{\text{base}} $	1
角度基础奖励	$ R_{\text{angle}}^{\text{base}} $	1
捕获基础奖励	$ R_{\text{cap}}^{\text{base}} $	100
围捕数量增益系数	$ {g}_{\text{cap}} $	1.2

下载: 导出CSV

表 3 3种任务训练对应的延迟条件

Table 3. Delay conditions for training on three kinds of tasks

任务	时隙长度/s	丢包率/%	描述
“3围1”	30	20	任务简单, 延迟低
“3围2”	60	0	任务难, 延迟高
“6围3”	60	0	任务最难, 延迟最高

下载: 导出CSV

表 4 “3围1”任务场景下评估结果

Table 4. Evaluation results under three UUVs encircling one target scenario

方法	围捕数量				围捕质量
方法	①	②	③	④	①	②	③	④
Oracle	0.95(0.04)	0.91(0.05)	0.85(0.07)	0.76(0.08)	0.81(0.03)	0.78(0.04)	0.72(0.06)	0.63(0.07)
Oracle-DA	0.97(0.03)	0.9(0.05)	0.82(0.07)	0.7(0.09)	0.86(0.03)	0.79(0.05)	0.72(0.07)	0.6(0.07)
Oracle-DA-CL	0.94(0.04)	0.88(0.06)	0.79(0.06)	0.69(0.07)	0.81(0.03)	0.76(0.05)	0.69(0.06)	0.6(0.06)
Base	0.92(0.05)	0.91(0.05)	0.87(0.06)	0.79(0.07)	0.78(0.04)	0.77(0.04)	0.74(0.05)	0.66(0.06)
Base-DA	0.94(0.04)	0.93(0.04)	0.9(0.05)	0.87(0.06)	0.8(0.04)	0.8(0.03)	0.78(0.05)	0.74(0.05)
Base-DA-CL	0.95(0.04)	0.94(0.04)	0.92(0.05)	0.87(0.06)	0.81(0.03)	0.81(0.04)	0.8(0.04)	0.75(0.05)
MSSR-DA-CL	0.94(0.04)	0.95(0.04)	0.9(0.05)	0.86(0.06)	0.8(0.03)	0.81(0.03)	0.77(0.04)	0.74(0.05)
SSMR-DA-CL	0.95(0.04)	0.94(0.04)	0.92(0.05)	0.89(0.05)	0.82(0.03)	0.82(0.04)	0.8(0.04)	0.77(0.05)

	围捕耗时				碰撞次数
	①	②	③	④	①	②	③	④
Oracle	18.05(1.52)	20.55(1.51)	22.4(1.76)	23.9(2.08)	0.02(0.02)	0.02(0.03)	0.03(0.03)	0.04(0.04)
Oracle-DA	18.56(1.58)	21.57(1.74)	24.26(1.83)	26.29(2.24)	0.01(0.02)	0.02(0.02)	0.03(0.03)	0.03(0.03)
Oracle-DA-CL	17.93(1.46)	20.98(1.65)	22.98(2.28)	23.63(2.16)	0.02(0.02)	0.02(0.03)	0.02(0.02)	0.03(0.03)
Base	18.6(1.48)	19.76(1.46)	20.6(1.64)	21.84(1.85)	0.02(0.02)	0.02(0.02)	0.02(0.02)	0.03(0.03)
Base-DA	19.01(1.63)	20.32(1.73)	21.65(2.03)	22.83(1.96)	0.02(0.02)	0.02(0.03)	0.02(0.03)	0.03(0.03)
Base-DA-CL	19.29(1.66)	20.74(1.65)	21.47(1.65)	23.81(1.99)	0.01(0.02)	0.02(0.02)	0.01(0.02)	0.02(0.02)
MSSR-DA-CL	18.93(1.6)	19.79(1.57)	20.62(1.72)	21.37(1.9)	0.02(0.02)	0.01(0.02)	0.02(0.02)	0.02(0.02)
SSMR-DA-CL	18.27(1.62)	19.12(1.66)	20.14(1.72)	20.94(1.69)	0.01(0.02)	0.01(0.02)	0.02(0.02)	0.02(0.03)

下载: 导出CSV

表 5 “3围2”任务场景下评估结果

Table 5. Evaluation results under three UUVs encircling two targets scenario

方法	围捕数量				围捕质量
方法	①	②	③	④	①	②	③	④
Oracle	1.51(0.1)	1.32(0.11)	1.2(0.13)	0.99(0.13)	0.79(0.03)	0.73(0.05)	0.68(0.06)	0.59(0.06)
Oracle-DA	1.61(0.09)	1.39(0.11)	1.25(0.13)	1.03(0.12)	0.82(0.03)	0.77(0.04)	0.72(0.05)	0.62(0.06)
Oracle-DA-CL	1.54(0.1)	1.34(0.11)	1.18(0.13)	0.9(0.13)	0.79(0.03)	0.75(0.04)	0.69(0.06)	0.57(0.07)
Base	1.27(0.1)	1.25(0.1)	1.23(0.11)	1.12(0.11)	0.74(0.04)	0.74(0.05)	0.74(0.04)	0.69(0.05)
Base-DA	1.38(0.11)	1.34(0.11)	1.31(0.12)	1.2(0.12)	0.76(0.04)	0.76(0.04)	0.75(0.04)	0.7(0.05)
Base-DA-CL	1.42(0.12)	1.36(0.11)	1.33(0.11)	1.19(0.12)	0.78(0.04)	0.76(0.04)	0.75(0.04)	0.7(0.05)
MSSR-DA-CL	1.38(0.11)	1.39(0.12)	1.36(0.1)	1.27(0.11)	0.77(0.04)	0.78(0.04)	0.77(0.04)	0.75(0.05)
SSMR-DA-CL	1.46(0.1)	1.43(0.11)	1.44(0.12)	1.36(0.11)	0.78(0.04)	0.78(0.04)	0.78(0.04)	0.76(0.04)

	围捕耗时				碰撞次数
	①	②	③	④	①	②	③	④
Oracle	20.02(1.32)	22.37(1.5)	23.41(1.73)	24.29(2.12)	0.05(0.04)	0.06(0.04)	0.07(0.04)	0.08(0.05)
Oracle-DA	20.23(1.08)	22.59(1.2)	23.72(1.59)	25.19(2.13)	0.04(0.04)	0.05(0.04)	0.06(0.04)	0.07(0.05)
Oracle-DA-CL	20.07(1.14)	22.15(1.38)	23.58(1.69)	25.15(2.07)	0.05(0.04)	0.06(0.04)	0.07(0.04)	0.08(0.04)
Base	22.23(1.6)	22.95(1.55)	22.95(1.58)	23.94(1.56)	0.06(0.04)	0.07(0.04)	0.07(0.04)	0.08(0.05)
Base-DA	21.4(1.44)	22.02(1.33)	22.09(1.37)	23.03(1.61)	0.06(0.04)	0.06(0.04)	0.06(0.04)	0.06(0.04)
Base-DA-CL	21.41(1.35)	22.44(1.39)	22.68(1.32)	23.89(1.71)	0.05(0.04)	0.05(0.04)	0.06(0.04)	0.07(0.04)
MSSR-DA-CL	21.33(1.19)	21.19(1.3)	21.38(1.32)	22.28(1.35)	0.06(0.04)	0.06(0.04)	0.05(0.04)	0.06(0.04)
SSMR-DA-CL	21.24(1.23)	21.77(1.49)	21.48(1.4)	22.4(1.35)	0.05(0.04)	0.05(0.04)	0.05(0.04)	0.05(0.04)

下载: 导出CSV

表 6 “6围3”任务场景下评估结果

Table 6. Evaluation results under six UUVs encircling three targets scenario

方法	围捕数量				围捕质量
方法	①	②	③	④	①	②	③	④
Oracle	1.13(0.14)	0.98(0.14)	0.86(0.13)	0.79(0.14)	0.61(0.06)	0.53(0.07)	0.49(0.06)	0.45(0.07)
Oracle-DA	1.44(0.16)	1.03(0.16)	0.77(0.14)	0.7(0.15)	0.66(0.05)	0.52(0.06)	0.41(0.07)	0.38(0.07)
Oracle-DA-CL	1.56(0.18)	1.24(0.18)	0.91(0.15)	0.81(0.15)	0.67(0.05)	0.57(0.07)	0.47(0.07)	0.43(0.07)
Base	0.7(0.13)	0.81(0.13)	0.83(0.12)	0.83(0.13)	0.45(0.08)	0.49(0.06)	0.5(0.07)	0.5(0.07)
Base-DA	0.78(0.14)	0.91(0.12)	1.05(0.14)	0.99(0.13)	0.48(0.07)	0.54(0.06)	0.6(0.06)	0.58(0.06)
Base-DA-CL	1.07(0.15)	1.21(0.14)	1.22(0.15)	1.16(0.14)	0.58(0.06)	0.62(0.06)	0.63(0.06)	0.6(0.06)
MSSR-DA-CL	1.32(0.16)	1.41(0.16)	1.41(0.16)	1.35(0.16)	0.63(0.05)	0.65(0.05)	0.65(0.06)	0.64(0.06)
SSMR-DA-CL	1.48(0.17)	1.54(0.18)	1.51(0.15)	1.45(0.15)	0.66(0.05)	0.68(0.05)	0.67(0.06)	0.66(0.05)

	围捕耗时				碰撞次数
	①	②	③	④	①	②	③	④
Oracle	18.96(1.86)	20.38(2.96)	21.03(3.31)	20.55(2.71)	0.16(0.06)	0.2(0.07)	0.2(0.07)	0.22(0.08)
Oracle-DA	16.29(1.56)	18.48(2.36)	20.09(2.63)	19.89(2.65)	0.22(0.08)	0.34(0.08)	0.38(0.09)	0.41(0.09)
Oracle-DA-CL	16.65(1.42)	19.11(2.96)	20.31(2.54)	20.41(3.04)	0.17(0.06)	0.19(0.07)	0.21(0.06)	0.21(0.07)
Base	20.8(3.14)	19.07(3.14)	19.08(2.39)	19.58(2.38)	0.23(0.07)	0.25(0.08)	0.26(0.08)	0.25(0.07)
Base-DA	18.01(2.27)	16.57(2.27)	17.04(2.19)	17.35(2.05)	0.22(0.07)	0.21(0.07)	0.2(0.08)	0.2(0.07)
Base-DA-CL	16.87(2.01)	16.57(1.75)	16.94(1.89)	17.44(1.98)	0.21(0.08)	0.21(0.08)	0.21(0.08)	0.21(0.07)
MSSR-DA-CL	17.72(1.68)	17.01(1.74)	17.63(1.88)	17.68(1.73)	0.18(0.07)	0.19(0.07)	0.2(0.07)	0.19(0.07)
SSMR-DA-CL	16.51(1.3)	16.28(1.66)	16.62(1.57)	16.6(1.6)	0.18(0.08)	0.18(0.07)	0.2(0.07)	0.2(0.08)

下载: 导出CSV

参考文献(31)

[1]	胡桥, 赵振轶, 冯豪博, 等. AUV智能集群协同任务研究进展[J]. 水下无人系统学报, 2023, 31(2): 189-200. Hu Q, Zhao Z Y, Feng H B, et al. Progress of AUV intelligent swarm collaborative task[J]. Journal of Unmanned Undersea Systems, 2023, 31(2): 189-200.
[2]	陈昭, 丁一杰, 张治强. 无人潜航器发展历程及运用优势研究[J]. 舰船科学技术, 2024, 46(23): 98-102. Chen Z, Ding Y J, Zhang Z Q. Research on the development history and application advantages of unmanned underwater vehicle[J]. Ship Science and Technology, 2024, 46(23): 98-102.
[3]	韦良才, 李东升. 无人潜航器集群作业的应用[J]. 电子技术, 2024, 53(9): 310-311. Wei L C, Li D S. Application of unmanned underwater vehicle cluster operation[J]. Electronic Technology, 2024, 53(9): 310-311.
[4]	王科翔, 卢野, 李银城, 等. 面向海战场下无人集群作战发展研究综述[J]. 现代防御技术, 2025, 53(3): 11-22. Wang K X, Lu Y, Li Y C, et al. Review of unmanned swarm operations development for maritime battlefields[J]. Modern Defence Technology, 2025, 53(3): 11-22.
[5]	王光诚. 基于多智能体强化学习的无人水下航行器决策方法研究[D]. 长春: 吉林大学, 2024: 1-2.
[6]	林学航. 基于强化学习的无人潜航器三维路径规划方法研究[D]. 哈尔滨: 哈尔滨工程大学, 2024: 5.
[7]	赵剑楠, 覃琪琪, 李云, 等. 基于分布式强化学习的AUV水下三维洋流目标跟踪控制算法[J]. 电信科学, 2025, 41(10): 88-101. Zhao J N, Qin Q Q, Li Y, et al. Distributed reinforcement learning-based AUV 3D underwater current target tracking control algorithm[J]. Telecommunication Science, 2025, 41(10): 88-101.
[8]	杨晓康. 基于深度强化学习的多智能体协同围捕控制[D]. 杭州: 杭州电子科技大学, 2025: 2.
[9]	王景璟, 魏维, 王永越, 等. 水下自主潜航器集群协同围捕技术[J]. 指挥与控制学报, 2023, 9(6): 637-650. Wang J J, Wei W, Wang Y Y, et al. Autonomous underwater vehicle swarm collaborative hunting technologies[J]. Journal of Command and Control, 2023, 9(6): 637-650.
[10]	赵少靖, 付松琛, 白乐天, 等. 基于自适应多目标优化的UUV全覆盖路径规划方法[J]. 水下无人系统学报, 2025, 33(3): 459-472. Zhao S J, Fu S C, Bai L T, et al. Adaptive multi-objective optimization-based coverage path planning method for UUVs[J]. Journal of Unmanned Undersea Systems, 2025, 33(3): 459-472.
[11]	刘千里, 张松, 李乐琦. 水下无人通信载荷技术综述与应用前景[J]. 舰船电子工程, 2025, 45(1): 7-12. Liu Q L, Zhang S, Li L Q. Overview of underwater unmanned communication payload technology and application[J]. Ship Electronic Engineering, 2025, 45(1): 7-12.
[12]	董佳明. 基于多智能体强化学习的多UUV博弈围捕方法研究[D]. 哈尔滨: 哈尔滨工程大学, 2024: 6-7.
[13]	Wu G, Xu T, Sun Y, et al. Review of multiple unmanned surface vessels collaborative search and hunting based on swarm intelligence[J]. International Journal of Advanced Robotic Systems, 2022, 19(2): 248452788.
[14]	Zhao Z, Hu Q, Feng H, et al. A cooperative hunting method for multi-AUV swarm in underwater weak information environment with obstacles[J]. Journal of Marine Science and Engineering, 2022, 10(9): 1266. doi: 10.3390/jmse10091266
[15]	Li J, Lu H, Zhang H, et al. Dynamic target hunting under autonomous underwater vehicle(AUV) motion planning based on improved dynamic window approach(DWA)[J]. Journal of Marine Science and Engineering, 2025, 13(2): 221. doi: 10.3390/jmse13020221
[16]	Cao X, Liu W, Ren L. Underwater target capture based on heterogeneous unmanned system collaboration[J]. IEEE Transactions on Intelligent Vehicles, 2024, 9(10): 6049-6062.
[17]	Jiang B, Wang Y, Kong F, et al. Multi-autonomous underwater vehicle trajectory planning in ocean current based on hierarchical hunting and evolutionary learning[J]. ACM Transactions on Intelligent Systems and Technology, 2025, 16(5): 1-26. doi: 10.1145/3757928
[18]	Zhang M, Chen H, Cai W. Collaborative hunting method of multi-AUV in 3D IoUT: searching, tracking, and encirclement keeping[J]. IEEE Internet of Things Journal, 2025, 12(8): 10958-10973. doi: 10.1109/JIOT.2024.3514632
[19]	Wang Z, Du J, Jiang C, et al. Task scheduling for distributed AUV network target hunting and searching: an energy-efficient AoI-aware DMAPPO approach[J]. IEEE Internet of Things Journal, 2022, 10(9): 8271-8285.
[20]	Hou X, Xing T, Wang J, et al. Age of information-aware multi-objective optimization for heterogeneous UAV-USV-UUV Networks in Underwater Target Hunting[J]. IEEE Transactions on Mobile Computing, 2025, 24(11): 11292-11304.
[21]	Chen J, Wang Y, Zhang Y, et al. Extrinsic-and-intrinsic reward-based multi-agent reinforcement learning for multi-UAV cooperative target encirclement[J]. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(10): 17653-17665.
[22]	Wei W, Wang J, Du J, et al. Differential game-based deep reinforcement learning in underwater target hunting task[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023, 36(1): 462-474.
[23]	伍雨辰. 基于深度强化学习的水声传感器网络时隙可变MAC协议[D]. 天津: 天津大学, 2022: 2.
[24]	Rashid T, Samvelyan M, Schroeder De Witt C, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020, 21(178): 1-51.
[25]	Oliehoek F A, Amato C. A concise introduction to decentralized POMDPs[M]. Cham: Springer International Publishing, 2016: 14-15.
[26]	Mnih V, Kavukcuoglu K, Silver D, et al. Human-level control through deep reinforcement learning[J]. Nature, 2015, 518: 529-533. doi: 10.1038/nature14236
[27]	Tampuu A, Matiisen T, Kodelja D, et al. Multiagent cooperation and competition with deep reinforcement learning[J]. PLOS One, 2017, 12(4): e0172395. doi: 10.1371/journal.pone.0172395
[28]	Sunehag P, Lever G, Gruslys A, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward[C]//Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, 2018: 2085-2087.
[29]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[J]. Advances in Neural Information Processing Systems, 2017, 30: 6000-6010. doi: 10.1007/978-3-031-84300-6_13
[30]	Dey R, Salem F M. Gate-variants of gated recurrent unit (GRU) neural networks[C]//2017 IEEE 60th International Midwest Symposium on Circuits and Systems(MWSCAS), 2017: 1597-1600.
[31]	Ha D, Dai A, Le Q V. Hypernetworks[EB/OL]. [2025-12-22]. https://arxiv.org/abs/1609.09106.