Diffusion Policy: Visuomotor Policy Learning via Action Diffusion

8 min read

Generate a coherent action sequence by repeatedly denoising noise, execute only its near-term prefix, then observe and plan again.

Evaluation15 tasks, 4 benchmarksAverage improvement46.9%Outputaction sequenceSampling10 DDIM stepsReported latency0.1s on RTX 3080

training

Corrupt a demonstrated plan, then learn to recover it

Training is supervised behavior cloning in action space. There is no reward, value function, policy gradient, or negative sampling loop.

Slice a demonstration

Pair the latest Tₒ observations Oₜ with the next Tₚ expert actions Aₜ⁰.

Add known noise

Sample a diffusion step k and Gaussian noise εᵏ, then corrupt the entire action sequence.

Predict the noise

Give the model Oₜ, the noisy actions, and k. It estimates the exact noise that was added.

Apply one MSE loss

Update the vision encoder and noise predictor so the estimated noise matches εᵏ.

one training example

Oₜ = [oₜ₋Tₒ₊₁, ..., oₜ]

Aₜ⁰ = [aₜ, aₜ₊₁, ..., aₜ₊Tₚ₋₁]

The observation window says what is happening now. The action window is the clean trajectory sketch the model must learn to generate.

noise-prediction objective

L = MSE(εᵏ, εθ(Oₜ, Aₜ⁰ + εᵏ, k))

Across many noise levels, the model learns the vector field that can turn random action sequences back into demonstration-like plans.

inference

Start with noise, then refine an action plan

The policy does not regress directly to one answer. It follows a learned score field that pulls a random action sequence toward the conditional demonstration distribution.

Sample noisy actions

Initialize the full prediction horizon Aᵏ from Gaussian noise. Different initial samples can enter different behavior modes.

Condition on observations

Encode recent images and robot state once, then provide that context and the diffusion timestep to the noise predictor.

Denoise repeatedly

Subtract predicted noise across K iterations. DDIM reduces the paper's 100 training steps to 10 inference steps.

Execute a short prefix

Run the next Tₐ actions, observe again, and repeat. The remaining prediction is a plan, not a commitment.

learned update

Aᵏ⁻¹ = α(Aᵏ - γεθ(O, Aᵏ, k) + noise)

εθ acts like the gradient of an energy landscape over action sequences. Each step moves the sample toward actions that better fit the observation.

why diffusion

The representation fits the shape of robot behavior

Robot demonstrations are precise, sequential, and often ambiguous. Diffusion handles those properties in one output representation.

Multimodal actions

Separate samples can commit to distinct valid strategies instead of averaging incompatible actions.

High-dimensional output

The model generates a whole action sequence jointly, which encourages temporal consistency.

Stable training

Noise prediction avoids the negative-sampling loop used to train normalized energy-based policies.

architecture

Two noise predictors, one conditional diffusion objective

The visual encoder is shared conceptually, but the action-sequence model can favor smooth trajectories or rapid changes.

CNN-based

Temporal U-Net with FiLM

A 1D convolutional network processes the noisy action sequence. Every block is modulated by the observation feature and diffusion timestep through channel-wise scale and bias values.

action feature x → a(O, k) · x + b(O, k)

Transformer-based

Causal decoder with cross-attention

Action tokens use causal self-attention and read observation tokens through cross-attention. The paper finds this useful for tasks with high-frequency action changes and velocity control.

noisy actions → causal attention → ∇E(A)

figures from the paper

Policy representation, full architecture, and multimodal behavior

These are the authors' original diagrams from the paper. They connect the abstract score-field view to the deployed visuomotor policy and its observed behavior.

Paper diagram comparing explicit, implicit, and diffusion policy representations, including the learned action gradient field — Paper Figure 1. Policy representations. Explicit policies choose an output parameterization, implicit policies optimize an energy, and Diffusion Policy iteratively follows a learned gradient field from noise to an action.Full resolution

Diffusion Policy paper overview showing image observations, receding-horizon action sequences, CNN FiLM conditioning, and Transformer cross-attention — Paper Figure 2. Diffusion Policy overview. Recent observations condition K denoising updates over an action sequence. The paper instantiates the noise predictor as either a temporal CNN or a Transformer decoder.Full resolution

Paper trajectories comparing Diffusion Policy, LSTM-GMM, BET, and IBC around a T-shaped block — Paper Figure 3. Multimodal behavior. In Push-T, Diffusion Policy represents both paths around the block while committing to one coherent route per rollout. The baselines are biased toward one route or switch modes inconsistently.Full resolution

The mental model

Ordinary behavior cloning asks a network to output the correct action directly. That is awkward when several actions are valid. Averaging them can produce an action that belongs to no successful behavior. Diffusion Policy instead learns how to move a noisy action sequence toward regions that look like demonstrations.

At inference time, the policy samples a whole sequence from noise. A CNN or Transformer repeatedly predicts the noise to remove while conditioning on recent images and robot state. The result is a temporally coherent plan that can preserve distinct modes, such as going around the left or right side of an obstacle.

The robot does not blindly execute the whole prediction. It commits to a short prefix, collects new observations, and replans. This receding-horizon loop combines the consistency of sequence prediction with the responsiveness of closed-loop control.

Sources

Paper Project Code