π₀.₇: A Steerable Generalist Robotic Foundation Model with Emergent Capabilities
11 min read
Describe not only what the robot should do, but how it should do it. Rich multimodal prompts turn demonstrations, failures, autonomous rollouts, and non-robot experience into one steerable generalist policy.
architecture
One VLA, steered by a richer description of behavior
The familiar VLM-plus-action-expert architecture now reads a multimodal context that says what success looks like and which behavior mode to produce.
Observation + memory
Up to four cameras, each with up to six history frames. The MEM-style encoder compresses any history length to one frame's token count.
Multimodal context Cₜ
Gemma 3 VLM backbone
A 400M vision encoder handles observation history and subgoal images. Text and visual tokens meet under block-causal attention.
Flow action expert
The action expert attends to backbone activations and predicts 50 continuous action steps with flow matching.
Steered action chunk
aₜ:ₜ₊₅₀
Joint-space or end-effector commands, conditioned on the requested speed, quality, strategy, and near-future scene.
Execute 15 or 25, then replan with RTC
diversifying the prompt
Resolve ambiguity instead of averaging it away
Every prompt component names a hidden variable that would otherwise mix distinct behaviors under the same task label.
Task + subtask
"peel vegetables" + "pick up the peeler"
A stable task intent plus the next semantic step from a person or high-level policy.
Visual subgoals
Up to 3 desired near-future views
Base and wrist views specify geometry that is awkward or ambiguous to describe in words.
Episode metadata
Speed: 8000 · Quality: 5 · Mistake: false
Labels separate fast expert behavior from slow, failed, or otherwise suboptimal experience.
Control mode
joint or end-effector
A text identifier tells the same policy which action interface it should use for the task.
Component dropout makes the prompt composable
Visual subgoals
Present in 25% of training examples
Prevents the easy inverse-dynamics shortcut from dominating.
Subtask text
Dropped in 30% of examples that have subgoals
Lets an image substitute for the equivalent verbal description.
All metadata
Dropped entirely 15% of the time
Keeps the model usable without episode annotations.
Each metadata field
Individually dropped with 5% probability
Allows partial steering at inference time.
Control mode
Never dropped
The requested action interface must stay unambiguous.
training data
The prompt is what makes a much broader mixture usable
Quality and mistake labels preserve the information in weak experience without asking the model to imitate its flaws at test time.
Robot demonstrations
- many platforms
- static + mobile
- single-arm + bimanual
Autonomous experience
- prior-policy rollouts
- human interventions
- failures + mistakes
Human + open data
- egocentric video
- open robot datasets
- video-language tasks
Auxiliary web data
- VQA + captions
- object localization
- attribute prediction
Same task label, different behavior modes
Expert demo
quality 5
fast
no mistake
Slow success
quality 4
slow
no mistake
Failed rollout
quality 1
slow
mistake
Distill without flattening
At inference, ask for high quality and no mistakes. Autonomous data can transfer specialist behavior while its labels keep weak modes separated from the desired one.
inference
Semantic plans, visual plans, and motor plans run on different clocks
The subtask policy and world model run in separate threads. Motor inference never waits for them; it uses the newest context already available.
Choose the next subtask
A learned high-level policy or a human coach supplies the current semantic instruction.
changes on semantic progress
Generate visual subgoals
A BAGEL-initialized world model edits current views toward the requested near-future outcome.
refresh every 4s or on subtask change
Request the desired mode
Quality 5, mistake false, a task-specific fast speed bin, and the selected control interface.
metadata is steerable
Act and replan asynchronously
The VLA always uses the latest available context, executes part of each chunk, and blends replans with RTC.
50 Hz control · 15 or 25 steps executed
RTC trains through inference delay
Training simulates 0 to 12 delayed timesteps, up to 240 ms at 50 Hz, so refreshed chunks can join the action stream smoothly.
emergent capabilities
Generalization appears where detailed context meets diverse data
The ablations show an interaction: broader data becomes substantially more valuable when the model can tell its behavior modes apart.
Specialist-level dexterity
One generalist matches or exceeds RL-trained specialists on laundry, espresso, and box-building evaluations.
no task-specific post-training
Instruction generalization
The model follows open-ended and referential language across entirely unseen kitchens and bedrooms.
new scenes + new references
Cross-embodiment transfer
A much larger bimanual UR5e folds shirts without any laundry-folding data from that robot.
zero-shot task transfer
Compositional tasks
Language coaching guides new appliance use, then trains a high-level policy without additional teleoperation.
new skill compositions
the interaction
What changed from prior π models
π₀.₇ builds on the π₀.₆ VLA and MEM memory system. Its central contribution is a training and prompting methodology: expose the strategy, quality, desired future, and control interface so one model can absorb heterogeneous experience and recombine it at inference.
The paper argues for strong signs of compositional generalization, not a solved definition of general intelligence.
From the paper
Architecture and prompt examples
The original figures show how the three runtime models connect and how different context combinations steer real task sequences.
The mental model
A short task label leaves too much unspecified. Two trajectories can both satisfy "fold the shirt" while differing in speed, quality, strategy, control interface, or even whether they succeed. If those differences are hidden, a generalist policy risks averaging incompatible behaviors. π₀.₇ makes them visible in the prompt.
The prompt combines the overall task, the current semantic subtask, multi-view images of a desired near-future state, episode metadata, and a control-mode label. During training, components are randomly removed so the same model remains useful with anything from a plain language command to the complete multimodal prompt.
At runtime, a high-level policy proposes the next textual subtask while a separate world model turns that subtask into visual subgoals. The VLA consumes the freshest available context and emits continuous action chunks asynchronously. The architecture is still recognizable as a flow-based VLA; the step change comes from making diverse experience legible and steerable.
Sources