FAST: Frequency-space Action Sequence Tokenization
8 min read
Turn a smooth, high-frequency robot trajectory into a compact motion sentence that an autoregressive VLA can predict.
why FAST
Stop making the model predict the same motion 700 times
Per-timestep binning preserves temporal redundancy. FAST compresses the whole trajectory before the Transformer sees it.
Scalar binning
0.31, 0.32, 0.32, 0.33 → 124, 124, 124, 125
One token per dimension per timestep. A 50 Hz, 14-dimensional, one-second chunk becomes about 700 tokens, and copying the previous token can be an easy shortcut.
FAST
smooth trajectory → sparse frequency coefficients
Describe the whole chunk in frequency space, then compress recurring patterns. On the paper's 50 Hz shirt-folding data, roughly 700 scalars became 53 tokens, a 13.2× reduction.
encode
Action chunk → compact token sequence
Only rounding loses information; BPE can exactly recover the quantized coefficient stream.
Normalize
Map each dataset/action dimension's 1st and 99th percentiles to approximately [−1, 1].
A → ÃDCT over time
Rewrite each dimension as average motion, slow shape, then progressively faster corrections.
C = DCT(Ã)Scale + round
Quantize continuous coefficients. Tiny high-frequency terms become zero, creating a sparse matrix.
C̄ = round(γC)Frequency first
Place frequency 0 for every action dimension first, then frequency 1, and so on.
[C̄₁,₀ … C̄ᴰ,₀, C̄₁,₁ …]BPE compression
Merge frequent neighboring integer patterns into tokens from a compact fitted vocabulary.
[12, 0, 0, 0] → T₄₂A FAST token has no fixed meaning like “one joint” or “one timestep.” It may encode several neighboring coefficients, dimensions, or frequencies, depending on which patterns BPE found in the training corpus.
ordering
Broad motion first, fine corrections later
Frequency-major flattening makes the autoregressive sequence progressively refine the complete multi-dimensional trajectory.
tokenizer fitting
FAST is mostly fixed math, not a neural encoder
There is no learned encoder, decoder, reconstruction loss, or codebook loss inside the tokenizer.
Fixed operations
The learned part
One BPE merge dictionary
Count adjacent pairs in quantized action sequences, merge the most frequent pair, add a vocabulary entry, and repeat. FAST+ fits one shared dictionary on about one million one-second chunks from diverse robots and action spaces, padding action vectors to 32 dimensions so different embodiments can share the corpus.
This fitting is frequency counting and pair merging, not backpropagation. Images, instructions, and cross-entropy only enter later, when a VLA learns what the fitted tokens mean.
decode
Reverse the pipeline to recover continuous actions
An autoregressive VLA such as π₀-FAST generates tokens one at a time, then decodes the entire predicted chunk.
FAST tokens
T₁ … Tₙ
BPE decode
integer stream
Unflatten
C̄ ∈ ℤᴰˣᴴ
Divide by γ
Ĉ = C̄ / γ
Inverse DCT
 ∈ ℝᴴˣᴰ
Denormalize
motor commands
Quantization is lossy; BPE encoding and decoding are lossless with respect to the rounded coefficient sequence.
FAST in π₀.₅
FAST teaches the VLM; the action expert controls the robot
The same demonstrated action chunk has a discrete representation for next-token learning and a continuous representation for flow matching.
Ground-truth continuous action chunk
Pretraining · 280k steps
FAST(A) → VLM
The PaliGemma-based VLM autoregressively predicts FAST action tokens with cross-entropy alongside web, localization, and semantic-subtask data. The flow-matching weight is α = 0, so this stage makes the VLM robot-action-aware before an action expert is trained.
L = −Σᵢ log p(Tᵢ | o, ℓ, T<ᵢ)Post-training · 80k steps
Noisy Aᵗ → action expert
A randomly initialized ~300M-parameter expert learns a continuous velocity field with flow matching while token prediction continues. The combined loss uses a flow weight of α = 10.
L = L_token + αL_flowThe action expert does not receive predicted FAST tokens. It reads the VLM prefix built from images, language, and robot state, plus a noisy continuous action chunk and flow timestep. Its mask blocks access to FAST action tokens, preventing ground-truth action leakage through the discrete branch.
where the tokens go
π₀-FAST and π₀.₅ use FAST differently
FAST is an inference representation in π₀-FAST, but primarily a pretraining target in π₀.₅.
π₀-FAST
Generate FAST tokens at inference
The autoregressive policy predicts and decodes the compact token sequence into a continuous action chunk.
π₀.₅
Generate continuous actions at inference
The VLM first predicts a textual subtask; the flow-matching expert then produces the low-level continuous chunk in ten integration steps. FAST action tokens are not generated for real-time low-level control, avoiding the roughly 30–60 autoregressive action-token steps used by π₀-FAST.
The mental model
FAST does not assign one token to every scalar action. It first rewrites a complete action chunk in frequency space, where smooth robot motion is concentrated in a few low-frequency coefficients. It rounds those coefficients, orders the broad trajectory before its fine corrections, and uses BPE to merge recurring patterns into a short sequence of discrete tokens.
Think of the result as a motion sentence: early values describe the overall movement across every action dimension, while later values refine it. The only fitted part of the tokenizer is its BPE vocabulary; normalization, DCT, rounding, flattening, and inverse DCT are fixed operations.
Sources