MAD3PG: A Framework for Multi-Agent Deep Denoising Diffusion Policy Gradient Optimization

<p dir="ltr">Distributional reinforcement learning enhances policy robustness and learning efficiency by modeling the full value distribution rather than merely estimating its expectation. However, most existing methods are designed for single-agent discrete tasks and typically rely on prior assumptions about the form of the value distribution, such as fixed discrete atoms or quantile regression. These assumptions become restrictive in multi-agent scenarios, where information from different agents must be fused and the resulting joint value distribution often exhibits complex multimodal characteristics that cannot be effectively captured by predefined distributional forms. To address the above limitations, this paper proposes Multi-Agent Deep Denoising Diffusion Policy Gradient Optimization (MAD3PG), a novel framework that leverages diffusion models to model the value distribution of joint state-action pairs. Unlike existing distributional methods that rely on prior assumptions, diffusion models inherently possess the capacity to capture value distribution, thereby demonstrating clear advantages in multi-agent environments. Since true value samples are unavailable during online training, we introduce a K-repeated sampling strategy combined with temporal-difference targets to efficiently optimize the conditional variational evidence lower bound. Furthermore, we provide a theoretical convergence analysis of MAD3PG. We demonstrate MAD3PG’s ability to fit multi-modal distributions through toy examples. Furthermore, we validate its effectiveness by conducting comprehensive experiments in the Multi-Agent Particle Environment (MPE) as well as in the MuJoCo environment. The experimental results indicate that MAD3PG significantly outperforms traditional algorithms based on deterministic value estimation</p>