In this paper, we propose a novel algorithm that integrates diffusion models with reinforcement learning, called Diffusion-Q Synergy (DQS). The methodology formalizes an equivalence relationship between the iterative denoising process in diffusion models and the policy improvement mechanism in Markov Decision Processes. Central to this framework is a dual-learning mechanism: (1) a parametric Q-function is trained to evaluate noise prediction trajectories through temporal difference learning, effectively serving as a differentiable critic for action quality assessment; and (2) this learned Q-sc...
