RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
8 min read
A pretrained vision-language model learns to answer robot instructions with action tokens, carrying web knowledge into closed-loop control.
Paper §3.1; Appendix D
The model: turn a VLM into a robot policy
RT-2 reuses a vision-language model’s image understanding, language processing and token decoder. Given a camera image and a task instruction, it predicts low-level end-effector commands as a token sequence. The same weights still answer visual questions. There is no newly added action-only head: robot control becomes another output task for the pretrained model.
RT-2-PaLI-X
The 55B variant combines a ViT-22B image encoder with a roughly 32B, 50-layer encoder-decoder language backbone similar to UL2. Projected image patches and instruction embeddings enter the multimodal encoder; the decoder generates action tokens autoregressively. The paper also studies a 5B variant.
RT-2-PaLM-E
The 12B variant uses a ViT-4B visual encoder and a decoder-only PaLM-based language model. Visual features are projected into the language embedding space and combined with text. The decoder predicts the same action representation. These are alternative VLA backbones, not two stages run in sequence.
The important transfer is semantic: a model that understands animals, numbers or object relationships can use that knowledge to select a target for a learned manipulation skill. Robot demonstrations still supply the connection between what is seen and how the arm should move.
Paper §3.2
Eight values become the action language
Each command contains a termination flag, three end-effector translation changes, three rotation changes and gripper extension. Each continuous dimension is uniformly quantized into 256 bins. The bin indices become tokens, so a robot training target has the same sequence format as a text answer.
terminate · Δx · Δy · Δz · Δrotₓ · Δrotᵧ · Δrot𝓏 · gripper
PaLI-X already has individual integer tokens up to 1000, so action bins use existing number tokens. PaLM-E instead repurposes its 256 least-used tokens. Both reuse the existing vocabulary and output layers. During robot-action decoding, the vocabulary is constrained to valid action tokens; ordinary vision-language tasks retain natural-language outputs.
This predicts one discretized command at a time. It does not use the continuous flow-matching action expert or action chunks found in π₀.₅. After decoding the bins back into physical commands, the robot acts, takes a new observation and queries the model again.
Paper §3.2, §4; Appendices B, E
Co-fine-tuning preserves web knowledge while teaching control
01 · Start pretrained
Initialize from a VLM trained on image-text, captioning and visual-question-answering data. This supplies broad visual and semantic knowledge before robot learning.
02 · Mix the tasks
Keep original vision-language examples while adding image + instruction → action-token examples. Robot data is upweighted to about 50% of PaLI-X’s mixture and 66% of PaLM-E’s.
03 · Predict the next token
Use the same next-token prediction objective across text answers and action strings. On robot examples, this is behavior cloning of demonstrated actions.
What is annotated? Robot trajectories pair camera observations with recorded actions and a natural-language instruction describing the demonstrated task. Training examples select the current image, the instruction and that timestep’s action target. These are action labels, not object masks or manually written reasoning traces.
How much robot data? Roughly 130,000 real-robot demonstrations from the RT-1 dataset, whose collection reports over 130K episodes and 700+ distinct task instructions. These were collected with 13 Everyday Robots over 17 months in office-kitchen environments, covering picking, placing and drawer manipulation. A trajectory is a complete demonstration with many timesteps, not a single image-action pair. RT-1 dataset source.
What comes from the web? Image-text pairs, captions, question-answer pairs and interleaved image/text examples. Appendix B describes WebLI as a major source: about 10B pairs across 109 languages, filtered to roughly 1B high-similarity examples. These are vision-language data, not billions of robot demonstrations. Each backbone retains its own mixture; RT-2-PaLI-X excludes Episodic WebLI during co-fine-tuning.
Web-scale VLM examples + ~130K robot trajectories
The web examples retain semantic knowledge; the robot trajectories ground that knowledge in actions. Co-fine-tuning keeps both in the training mix, so 130K describes the robotic portion rather than the entire training corpus.
Scale comparison: OpenVLA later trained on roughly 970K robot episodes from Open X-Embodiment, compared with RT-2’s roughly 130K. That provides a useful sense of dataset scale; different robot embodiments, task coverage and data mixtures mean episode counts alone do not measure model quality. OpenVLA source.
| Variant | Learning rate | Batch size | Gradient steps |
|---|---|---|---|
| PaLI-X 55B | 10⁻³ | 2,048 | 80K |
| PaLI-X 5B | 10⁻³ | 2,048 | 270K |
| PaLM-E 12B | 4 × 10⁻⁴ | 512 | 1M |
Ablations compare training from scratch, robot-only fine-tuning and mixed co-fine-tuning. Pretraining is essential, keeping web examples improves generalization, and the larger PaLI-X model performs better. The result supports preserving useful VLM knowledge while grounding it in robot behavior.
Paper §3.3, §4.4
Closed-loop control, with an optional language plan
The large models run on a multi-TPU cloud service queried by the robot. The paper reports 1–3 Hz for PaLI-X 55B and about 5 Hz for the 5B version. These are policy-update rates, not motor-servo rates. Large-model inference is a practical bottleneck for fast, precise behavior.
A separate PaLM-E experiment adds a natural-language Plan before the action tokens and fine-tunes for a few hundred additional steps. This supports examples such as choosing a rock when asked for something useful for hammering. It is an augmented variant with qualitative demonstrations, not a requirement for every RT-2 prediction or proof of reliable general-purpose planning.
Paper §4–5; Appendix Tables 4–6
Better semantic generalization, within the learned motor skills
Across roughly 6,000 evaluation trials, the gains are strongest on new objects, backgrounds, environments and semantic instructions. Appendix Table 4 reports 62% unseen-average success for both main RT-2 variants versus 32% for RT-1. Seen-task success remains similar: 91–93% for RT-2 and 92% for RT-1. These are the paper’s evaluation conditions, not a universal robot success rate.
Additional tests probe symbols, relationships and human recognition, with the best RT-2 variant exceeding three times RT-1’s average success on those emergent-skill tests. A separate smaller PaLI-based Language-Table experiment reaches 90% success in simulation versus 77% for the prior baseline; it uses a different, two-dimensional action setting.
The boundary is clear: web data improves how existing skills are applied, but does not teach unseen physical motions by itself. The robot’s motor repertoire still comes from robot demonstrations. Quantized actions, expensive inference and incomplete task success remain limitations. RT-2’s lasting contribution is the direct connection from pretrained visual-language knowledge to low-level action prediction in one model.