Action-to-Action Flow Matching
5 min read
Use the robot's recent motion, rather than random noise, as the starting point for generating its next action chunk.
the key shift
Start close to the answer
A robot already has a structured prior: the motion it just executed. A2A promotes that history from a small conditioning signal to the source distribution itself.
Regression
One deterministic mapping
Diffusion / flow
A long, iterative transport path
A2A
A short, informed transport path
architecture
History latent → conditioned flow → future actions
Action history defines the source point z₀. Images tell the vector field where that point should move.
Encode
Executed actions a≤t
8-step history → 1D CNN ×3 → source latent z₀
Visual history I≤t
8 frames → ResNet-18 → linear projection → condition c
Transport
Flow net
AdaLN-MLP ×4
Learn vτ(zτ, τ, c), then integrate the ODE from the history latent to the future latent in as little as one Euler step.
Decode
Residual MLP ×4
Decode z₁ into the next 8-step action chunk a>t.
training
Three losses make the short path executable
The vector field is only one part of the job: the latent must preserve actions, and ODE sampling must land on a physically valid chunk.
Flow matching
LFMPredict the target vector field between z₀ and z₁.
Reconstruction
LAEKeep action chunks recoverable from the shared latent space.
Inference consistency
LICAlign integrated latents and decoded actions with ground truth.
evidence
Fast enough for one-step control
A2A was evaluated on five simulated manipulation tasks and two real-world Franka tasks.
0.56 ms
one-step latency
Measured on an RTX 5090
>90%
one-step success
Close Box after 32 epochs
30–40%
unseen visual scenes
Six-step A2A across levels 1–3
Success rate after 30 epochs / 100 demonstrations
Five simulation tasks · percentages
| Method | Steps | Close box | Pick cube | Stack cube | Open drawer | Pick-place bowl |
|---|---|---|---|---|---|---|
| A2A | 6 | 92 | 92 | 86 | 92 | 90 |
| VITA | 6 | 88 | 88 | 80 | 90 | 92 |
| FM-UNet | 10 | 82 | 70 | 28 | 34 | 68 |
| ACT | 1 | 82 | 86 | 32 | 80 | 60 |
takeaways
The useful intuition
Why it works
Adjacent action chunks are physically similar. Encoding both into one latent space aligns their distributions and straightens the transport path.
Why it generalizes
Action history and images travel through separate paths, so low-dimensional proprioception is not drowned out by visual features.
Where it bends
The continuity prior is weaker for binary gripper switches and is sensitive to uncertain histories. Small injected noise helps, but the balance is task-specific.
In one sentence: A2A turns the robot's recent executed motion into the initial state of a vision-guided flow, making the route to its next action chunk short enough for single-step inference.
The mental model
A2A starts from an observation about robot motion: the action chunk that just happened is usually close to the action chunk that should happen next. Instead of transporting Gaussian noise all the way to an executable action, it encodes the robot's executed action history into a latent starting point and learns a shorter, vision-conditioned flow to the future-action latent.
That shorter path is simple enough for a four-block AdaLN MLP and can be integrated in one step. The trade-off is equally direct: the method is strongest when actions change smoothly, while discrete switches and corrupted action histories weaken its continuity prior.
Sources