π₀.₅: A Vision-Language-Action Model with Open-World Generalization
8 min read
One VLA learns from robot actions, semantic subtasks, and web data, then plans in language before generating continuous control.
π₀ → π₀.₅
Same continuous-control core, a different route to it
π₀ learns continuous control directly. π₀.₅ first builds a broad token-based robot-and-vision model, then adds continuous control and language-level planning.
| Difference | π₀ | π₀.₅ |
|---|---|---|
| Pretraining | Continuous noisy action chunks · flow matching | FAST, text, and location tokens · cross-entropy |
| Action expert | Present throughout robot training | Added for continuous-control post-training |
| Planning | Usually maps the prompt directly to actions | Predicts a textual subtask, then acts on it |
| Data mixture | Primarily multi-robot demonstrations | Robot data + subtasks + web vision-language data |
inference
See the scene, name the next subtask, then act
The same VLA handles both levels: it first generates a semantic command as text, then conditions its action expert on that command.
Observe
Multi-camera images
The VLM reads the room, objects, and robot workspace.
Overall task + robot state
For example, “clean the bedroom” plus tokenized proprioception.
oₜ + ℓ → pre-trained VLA
Plan in language
Predict one high-level subtask
Autoregressive text decoding turns a long-horizon goal into the next locally useful command.
Semantic output ℓ̂
“pick up the pillow”
Generate control
Denoise a continuous action chunk
The 300M action expert reads the observation and predicted subtask, then transforms noise into low-level motor commands.
Continuous output
noise → 10 flow steps → aₜ:ₜ₊H
reference figure
The paper's full two-stage system
Discrete multimodal pre-training builds the VLA; post-training adds the continuous action expert and the high-level-to-low-level inference loop.
training recipe
Learn broadly with tokens, specialize later for control
π₀.₅ separates scalable semantic pre-training from the final real-time control specialization.
Stage 1 · pre-training
One autoregressive token problem
Images, text, boxes, semantic subtasks, and FAST-compressed robot actions are all trained with next-token prediction. The action expert is not used in this stage.
Mobile manipulators
~400 hours across ~100 homes
Diverse environments
Other fixed and mobile robots in homes
Cross-embodiment
Many robots and tasks in laboratory settings
High-level labels
Observation → semantic subtask
Multimodal web data
Captions, VQA, and object detection
outputs: text · boxes · FAST action tokens
Stage 2 · post-training
Mobile manipulation + flow matching
The data mix narrows toward successful, task-relevant mobile manipulation. A 300M action expert is added and trained jointly with the discrete output path.
Mobile manipulators
~400 hours across ~100 homes
Diverse environments
Other fixed and mobile robots in homes
Cross-embodiment
Many robots and tasks in laboratory settings
High-level labels
Observation → semantic subtask
Multimodal web data
Captions, VQA, and object detection
Verbal instructions
Human supervisors supply language corrections and guidance.
Added
Verbal instructions + continuous action expert
Omitted
Lab cross-embodiment data (CE)
hybrid action learning
FAST for training scale, flow matching for real-time control
The two action representations solve different bottlenecks and are kept separate by the attention mask.
Discrete path
FAST action tokens
[-17] [12] [34] [142] ...
- Compresses an action chunk into a short token sequence.
- Fits standard, scalable next-token VLM training.
- Autoregressive decoding is less suitable for fast control.
Continuous path
Flow-matching action expert
noise → velocity field → action chunk
- Represents fine-grained, continuous motor commands.
- Refines every action slot in parallel over ten denoising steps.
- Added only during the mobile-manipulation post-training stage.
During pre-training
The flow loss weight is zero: the VLM learns entirely through discrete next-token prediction.
During post-training
Cross-entropy and flow-matching losses train the discrete and continuous outputs together.
knowledge insulation
The VLM teaches the action expert without reading it back
A blockwise attention mask controls exactly which token families can exchange information.
Protect the pre-trained pathway
Context embeddings never attend to flow-action embeddings, so the randomly initialized expert cannot inject noise back into the VLM.
Separate the action views
Flow tokens cannot read FAST action tokens, preventing the continuous branch from leaking the discrete answer during joint training.
Transfer in one direction
The action expert can read the complete visual, language, and state prefix, so semantic knowledge still conditions every denoising step.
what the experiments support
Generalization came from the recipe, not one magic dataset
The ablations attribute different parts of the result to cross-embodiment transfer, semantic web knowledge, and high-level supervision.
New homes, long tasks
The system cleaned kitchens and bedrooms in homes absent from training, with multi-stage rollouts lasting 10-15 minutes.
Different data, different benefits
Other-robot data was important for overall task success; web data mattered especially for language following on unseen object categories.
Explicit subtasks help
The full high-level + low-level system was strongest in the paper's ablation, while removing web or verbal-instruction data degraded performance.
What “open-world” means here
π₀.₅ generalizes familiar household behaviors to new environments and objects; the authors do not claim arbitrary new skills. They also report failures from unfamiliar hardware, partial observability, and distracted high-level subtask selection.
The mental model
π₀.₅ treats heterogeneous supervision as one multimodal language problem first. Images, captions, object locations, semantic subtasks, and FAST-compressed robot actions all become tokens that a PaliGemma-based VLA can learn with next-token prediction. This stage transfers visual semantics, task structure, and behavior from other robots into one shared backbone.
Post-training adds a smaller flow-matching action expert for fine-grained real-time control. At deployment, the VLA first names the next subtask, such as “pick up the pillow,” and then the action expert turns noise into a continuous action chunk conditioned on that subtask. A one-way attention mask lets the expert consume the VLM's knowledge without perturbing its pre-trained representation.
Sources