π₀: A Vision-Language-Action Flow Model for General Robot Control
7 min read
A pre-trained vision-language model learns to control many robots by repeatedly turning noise into a coherent chunk of continuous actions.
how it works
Observation prefix → two experts → action chunk
π₀ keeps the VLM's image-language pathway and adds continuous state and action tokens for robot control.
Build the sequence
2-3 RGB views
SigLIP vision encoder → image tokens
Language instruction
Gemma tokens, e.g. "fold shirt"
Robot state qₜ
Fixed context; expert-routed because proprioception is new to the VLM
Noisy action suffix A^τ
50 action slots + flow timestep τ
One sequence, two weight sets
PaliGemma VLM
3B pre-trained image-language backbone for image and text tokens.
Action expert
~300M randomly initialized parameters for state and action tokens.
The bridge
Different weights, one self-attention sequence. Action tokens can read the full observation prefix.
Generate actions
Predict a velocity field
Each pass says how the 50 noisy action slots should move toward a coherent, observation-conditioned action chunk.
Final output
A = [aₜ ... aₜ₊₄₉]Continuous motor commands, produced after 10 Euler updates.
reference figure
The architecture, as drawn in the paper
The pre-training mixture enters a PaliGemma-initialized VLM and a smaller action expert; one policy can then serve robot embodiments with different action spaces.
attention
How the action expert reads the VLM
The connection is ordinary self-attention with a blockwise causal mask, not a separate cross-attention module.
At every transformer layer, both the VLM and action expert produce their own queries, keys, and values using separate weights. The blockwise attention mask lets action tokens query the VLM's representations while preventing VLM tokens from reading robot-state and action tokens. Only the 50 action-token outputs are decoded and directly supervised by the flow-matching loss. Those gradients pass through attention into the VLM, so both pathways adapt for robot control.
Preserve the VLM pathway
Image and language tokens cannot look ahead to the new robotics tokens, minimizing distribution shift from PaliGemma pre-training.
Keep the context cacheable
State cannot attend to the changing action suffix, so its keys and values stay fixed during the flow integration steps.
Coordinate the whole chunk
Action tokens see the full prefix and one another bidirectionally, so all 50 steps can form one coherent motion.
flow matching
Training learns the path; inference follows it
τ = 0 is noise and τ = 1 is the demonstrated action chunk. The model learns the vector field between them.
During training
1
Take a demonstrated chunk
A contains the next 50 continuous robot actions.
2
Mix actions with noise
A^τ = τA + (1 - τ)ε, with ε sampled from a Gaussian.
3
Predict the direction
The expert estimates the velocity target A - ε at every slot.
4
Minimize flow MSE
The learned field is conditioned on images, instruction, and state.
The paper samples τ to emphasize lower, noisier timesteps, where the observation must do more work to constrain the action.
During inference
A^(τ + δ) = A^τ + δ · vθ(A^τ, o)
Forward Euler integration, repeated 10 times with δ = 0.1.
Encode context once
Compute image, language, and state prefix keys and values.
Start from Gaussian noise
Initialize all 50 action slots as A⁰ ~ N(0, I).
Refine ten times
Use forward Euler with δ = 0.1, re-running only the action suffix.
Execute and replan
Run part of the resulting action chunk, then observe the world again.
The paper reports 73 ms on-board inference for three cameras on an RTX 4090, including all ten action-expert passes.
training recipe
Breadth first, fluency second
The architecture supplies capacity; the two-stage data recipe supplies generality, recovery behavior, and task-specific polish.
Inherited visual semantics
PaliGemma starts with Internet-scale image-language pre-training before it ever sees robot data.
Broad robot pre-training
10,000+ hours across 7 robot configurations and 68 tasks, plus OXE, Bridge, and DROID data.
High-quality post-training
Curated demonstrations specialize the base policy for fluent, dexterous downstream tasks.
Why diverse pre-training data?
It covers varied scenes, mistakes, corrections, and recovery behaviors that polished demonstrations rarely contain.
Why curated post-training data?
It teaches the policy to complete a target task efficiently, consistently, and with a fluent strategy.
The mental model
Think of π₀ as one token sequence with two specialists. The PaliGemma vision-language backbone reads the images and instruction, while a smaller action expert handles robot state and a 50-step noisy action suffix. The specialists use different transformer weights, but the action tokens can read the observation tokens through shared self-attention.
The model does not emit the final motor commands in one pass. It starts the action suffix as Gaussian noise and makes ten flow-matching updates. Because the observation prefix stays fixed, its attention keys and values can be cached while only the action suffix is recomputed.
Sources