RL Token: Bootstrapping Online RL with Vision-Language-Action Models
10 min read
Compress a frozen VLA's internal state into one trainable readout, then let a small actor-critic refine its action chunks with minutes to hours of real robot practice.
figures from the paper
The complete method, then the extraction mechanism
These are the authors' original architecture diagrams. The first shows RLT as a complete system; the second opens the RL-token module and makes the representation shapes explicit.
the rl token
Many VLA vectors go in; one equally wide vector comes out
The paper's figure calls the input count N, while its equations use M. Both refer to the sequence of final-layer VLA embeddings that the readout compresses.
read frozen vla features
π₀.₆ final-layer embeddings
Two wrist cameras plus one base camera in the experiments. Each embedding has width 2,048.
z₁:N ∈ ℝᴺˣ²⁰⁴⁸
With a fixed task instruction, the experiments omit language embeddings from this readout step.
mix with a learned readout
[z₁:N, eᵣₗ] ∈ ℝ⁽ᴺ⁺¹⁾ˣ²⁰⁴⁸
Encoder transformer gϕ
Self-attention lets the learned special position collect information from the complete VLA sequence.
one continuous rl state
RL token zᵣₗ
1 × 2,048
Exactly one vector for the current observation. Its width is not reduced; the sequence length is.
shape bookkeeping
What the paper specifies, and what it leaves open
The 2,048-wide input and output shapes are explicit. Several implementation details are not reported, so they should not be reverse-engineered from the diagram.
Original sequence
N vectors
Each vector is 2,048D. The equations call this count M.
Learned input
+1 vector
eᵣₗ starts as a trainable special embedding, also 2,048D.
RL readout
1 vector
The encoder output at the appended position is retained.
Token compression
N : 1
An inferred ratio in token count. Vector width stays at 2,048.
Not disclosed: the numeric value of N, the encoder and decoder layer counts, their hidden widths, and the RL-token module's parameter count. N depends on the VLA input-token sequence produced for the observation.
how the token learns
Reconstruct the sequence through a one-token bottleneck
The decoder is a training constraint, not part of the deployed actor-critic. If one vector helps reconstruct all original VLA embeddings, it must preserve information the downstream controller can reuse.
Condition on zᵣₗ
1 × 2,048
the only summary vector
The decoder also receives earlier ground-truth embeddings while training, so it reconstructs the sequence autoregressively.
Decoder transformer dϕ
minimize Σᵢ ‖ẑᵢ − stopgrad(zᵢ)‖²
A linear projection maps decoder outputs back to the 2,048D VLA embedding space. Gradients from this loss do not alter the source embeddings.
Task demonstrations
Train on 1 to 10 hours collected for the target task.
Adaptation length
Train the VLA and RL token for 2,000 to 10,000 gradient steps.
Then freeze it
Freeze the VLA and readout before any online actor-critic updates.
online rl
Use the token as state, and the VLA chunk as a proposal
The frozen foundation model supplies perception and a strong behavioral prior. Small MLP actor and critic networks learn the precise correction.
Frozen π₀.₆
The full VLA predicts 50 actions for one second. Online RL refines a shorter 10-action prefix.
Actor πθ
πθ(a₁:₁₀ | zᵣₗ, sᵖ, ā₁:₁₀)
- Gaussian action distribution
- Reference masked in 50% of training samples
- L2 penalty toward the VLA proposal
Twin critics Qψ
Qψ(zᵣₗ, sᵖ, a₁:₁₀) → value
- TD3-style off-policy updates
- Replay mixes VLA, RL, and human actions
- Sparse terminal success reward
Execute 0.2 seconds
10 × 14
140D action chunk
Chunking shortens credit assignment while preserving a 50 Hz motor interface. Intermediate chunks are stored every two control steps.
complete recipe
Spend the learning budget where millimeters matter
The base VLA handles broad task progress. A human hands control to RLT for insertion, fastening, or rotation, concentrating sparse-reward learning on the failure-prone phase.
Adapt on demonstrations
Fine-tune the task VLA and train the reconstruction bottleneck.
Warm up the replay buffer
Collect competent transitions from the frozen VLA reference policy.
Practice the critical phase
Alternate robot rollouts with high-ratio actor-critic updates.
what improved
Faster completion without giving up reliability
Project-page throughput is measured as successful completions per 10 minutes. The scales differ by task, so compare each pair rather than bar height across tasks.
Screwdriver
1.7
base
14
rlt
Zip tie
2.8
base
13
rlt
Ethernet
147
base
400
rlt
Charger
136
base
600
rlt
On Ethernet insertion, replacing the RL token with a frozen ImageNet-pretrained ResNet-10 cuts final throughput by about 50%. The manipulation-aware representation matters, not just the small network.
The learned policy stops the base VLA's repeated probing near contact. It approaches and inserts more fluidly, and can discover a useful pressure-and-wiggle strategy absent from demonstrations.
The mental model
The RL token is not a discrete action token and it is not a shorter action vocabulary. It is one continuous 2,048-dimensional vector produced from the VLA's final-layer embeddings for the current observation. A learned special embedding is appended to that token sequence, a small encoder mixes the sequence, and the output at the special position becomes the RL state.
An autoregressive decoder makes this readout useful. During the task-specific adaptation stage, it must reconstruct every original VLA embedding from the single RL token and the embeddings that came before. This turns the readout into an information bottleneck. Once trained, the decoder is no longer part of online RL, and both the VLA and RL-token module remain frozen.
The lightweight actor and critic can now learn on top of a compact, manipulation-aware state instead of updating the multi-billion parameter VLA. The actor edits a sampled VLA action chunk rather than searching the full action space from scratch. Ten 14D actions form each 140D RL decision, while a behavior-cloning penalty keeps the edit close to the VLA's proposal unless the critic finds a better local behavior.
Sources