Sequence Modeling

Earlier robot policies were often written like:

single observation -> policy -> next action

That works when the current image contains everything the robot needs. But many real tasks are not like that. A single frame may show a pedestrian, a cube, or a hand near an object, but it does not always reveal where things came from, how fast they are moving, or what happened just before.

So the first weakness is memory. The robot needs history to infer the real state behind the visible observation.

In an ideal MDP, the current state is enough to choose the next action. Robotics is messier: the robot often gets partial observations.

For example, one image of a road does not tell you whether the road is icy. One image of a person does not tell you their velocity. One image of a robot hand does not tell you whether it has already touched the object.

So the policy should not only ask, "What do I see now?" It should ask, "What has been happening over time?"

POMDP - Partially Observable Markov Decision Process

The second weakness is smoothness. If the policy predicts each action independently, every tiny error can create a new correction at the next time step.

That gives behavior like:

move left -> move right -> move left -> correct again

Real manipulation needs short coherent motions: approach the object, align the gripper, close, lift. A grasp is not one action; it is a small trajectory.

Instead of treating each action as isolated, sequence modeling treats the data as an ordered stream.

A trajectory can contain observations, states, language, and actions:

o₁, a₁, o₂, a₂, o₃, a₃, ...

The model learns the probability of the next item given the previous items. This is the same basic idea behind language modeling: predict the next token from the context so far.

A full trajectory probability can be written as a chain of conditional predictions:

p(x₁, x₂, x₃, ...) = p(x₁) p(x₂ | x₁) p(x₃ | x₁, x₂) ...

So the model does not need to predict the whole future in one magical step. It repeatedly asks:

Given the previous tokens, what should the next token be?

For text, that next token is a word or subword. For robotics, it could eventually be an action token.

Recurrent neural networks were an early answer to sequence modeling. They process items one at a time and keep a hidden state that summarizes the past.

The good part: the same network can handle variable-length sequences.

The problem: everything important from the past must be compressed into one hidden state. Long-range dependencies become hard, gradients can vanish or explode, and training is slow because the sequence has to be processed step by step.

Transformers ask every token to look directly at the other relevant tokens.

Instead of passing information through a long recurrent chain, attention creates short paths between far-apart parts of the sequence. A token near the end can directly attend to a token near the beginning.

This helps with long context, parallel training, and avoiding the fixed-size memory bottleneck of RNNs.

The lecture frames attention like a differentiable lookup table.

Each token produces:

The model compares queries with keys, uses softmax to decide which tokens matter, and mixes their values into a new representation.

So attention is not hard-coded reasoning. It is a learned way to retrieve the useful parts of context.

Self-attention sees a set of tokens. By itself, it does not know whether one token came first, later, nearby, or far away.

That is why transformers add positional information. Absolute positions say, "this is token 17." Relative positions say, "this token is three steps before that token."

Relative encodings are often more flexible because many patterns depend on distance, not on the exact index.

The original transformer architecture had two sides.

The encoder reads the input sequence with bidirectional attention. For translation, that means the encoder can look at the whole source sentence.

The decoder generates the output sequence autoregressively. It uses a causal mask so it can only see previous output tokens, then cross-attends to the encoder output to use the source information.

During training, the decoder is given the true previous tokens instead of its own earlier predictions.

That means the model can learn all next-token predictions in one pass:

At test time, the model no longer has the true future tokens. It generates one token, feeds that back in, and continues.

Before a transformer can process text, the text has to become tokens.

Characters are flexible but make sequences very long. Words make sequences shorter but struggle with unknown words. Subwords are the middle ground.

This matters because attention cost grows with sequence length. Better tokenization is not just cosmetic; it changes how much context the model can afford to use.

Byte Pair Encoding starts from tiny pieces and repeatedly merges frequent adjacent pieces.

Common patterns become single tokens. Rare or unseen words can still be represented as smaller subword pieces.

So instead of choosing between characters and full words, BPE creates a vocabulary that compresses common language while preserving fallback flexibility.

Modern GPT-style models mostly use the decoder side: causal self-attention plus next-token prediction.

Scaling this simple recipe produced major jumps in capability. GPT-1 showed pretraining plus finetuning. GPT-2 showed stronger zero-shot behavior. GPT-3 made in-context learning much more visible. (this is why LLMs became few shot learners)

The lecture connects this to the Bitter Lesson: methods that can absorb more data and compute tend to win over carefully hand-engineered special cases.

A bigger model is not automatically better if it does not see enough data.

Chinchilla-style scaling showed that, for a fixed training budget, model parameters and training tokens should grow together. GPT-3-style scaling was powerful, but undertrained relative to later compute-optimal recipes.

This is useful because small experiments can predict how larger training runs will behave.

Vision Transformers turn an image into a sequence by splitting it into patches.

For example, each 16x16 patch is flattened, projected into a vector, given positional information, and treated like a token.

Now the same attention machinery can operate over image patches, text tokens, or mixtures of both.

CLIP trains an image encoder and a text encoder together.

The goal is simple: the embedding for an image should be close to the embedding for its matching caption and far from unrelated captions.

After training, an image of an apple and the text "apple" live near each other in representation space. This gives later models a useful bridge between vision and language.

LLaVA is an early-fusion style vision-language model.

It uses a vision encoder to turn the image into visual features, projects those features into the language model's token space, and prepends them to the text prompt.

Then the LLM can self-attend across image tokens and text tokens together. The benefit is rich interaction. The risk is that finetuning can disturb the pretrained language model.

Flamingo is a late-fusion style approach.

The language model stays mostly preserved. Visual information enters through cross-attention layers, where text queries can attend to visual keys and values.

A learned gate starts near zero, so at the beginning the model behaves almost exactly like the original LLM. During training, the gate gradually opens and lets visual information flow in.

Early fusion and late fusion usually start from a language model and attach vision to it.

Native multimodal models take the more expensive route: train one transformer over all modalities from scratch.

Text, image, audio, and video tokens can be interleaved in arbitrary order. The model does not treat language as the main system with other senses attached; it just learns over mixed token streams.

The lecture's bridge into robotics is this question:

If text can be tokens and images can be patch tokens, can robot actions become tokens in the same sequence?

Then a robot policy becomes a multimodal sequence model:

language tokens + image tokens + action tokens -> next action tokens

The transformer can use attention over the whole history to decide what matters for the next command.

Robot actions are usually continuous numbers: joint targets, end-effector deltas, gripper commands, or velocities.

A simple VLA approach, used by models like RT-2 and OpenVLA, discretizes each action dimension at each time step into bins.

The lecture describes a more robust version using quantile normalization: clip each action dimension to its 1st and 99th percentile, normalize to [-1, 1], then map into bins. This avoids wasting most bins on extreme outliers.

Once actions are binned, the bins can be represented as tokens in the model vocabulary.

A practical trick is to reuse rare vocabulary slots for action tokens. Then the model can keep the same next-token prediction setup and cross-entropy loss used for language.

Conceptually, the model is now saying:

Given the instruction, images, state, and previous action tokens, predict the next action token.

Instead of predicting one action, the model predicts the next K actions.

A chunk might represent a short motion like:

approach -> align -> lower -> close gripper -> lift

Because the actions are planned together, the motion is smoother than independent one-step decisions. This is especially important for high-frequency, dexterous manipulation.

If a robot predicts 100 actions and executes all of them before looking again, it may be smooth but not reactive.

The fix is temporal ensembling. The robot queries the model repeatedly, keeps overlapping future predictions, and combines the predictions for each time step with decaying weights.

That gives the robot both benefits: smooth action chunks and fresh observations at every step.

Action Chunking with Transformers was used for precise bimanual manipulation with the ALOHA robot.

In the lecture's example, the robot runs at 50 Hz with 14 degrees of freedom and predicts chunks of 100 actions. That is 1,400 numeric values per forward pass.

This kind of chunked planning helps tasks like battery insertion, where tiny jitter can ruin the motion.

At high frequency, consecutive robot actions are often almost identical.

That creates a bad learning signal for autoregressive VLAs. The model can get low prediction error by copying the previous action token instead of learning the real dexterous behavior.

So per-time-step tokenization works poorly when the robot needs dense, high-frequency control.

FAST stands for Frequency-space Action Sequence Tokenization.

The key idea is borrowed from signal processing. Robot action chunks are smooth trajectories, and smooth signals mostly live in low frequencies.

FAST applies a Discrete Cosine Transform to action chunks, keeps a compact frequency-space representation, quantizes it, and then uses BPE-like merging to create dense action tokens.

This is like finding the robotics equivalent of subwords: larger than one time step, smaller than a whole trajectory, and much more informative than repeated near-identical actions.

With compressed action tokens, the model is no longer wasting sequence length on tiny redundant differences between adjacent high-frequency commands.

The lecture describes this as preserving the useful motion information while discarding redundant high-frequency correlation.

This lets autoregressive VLAs move closer to the dexterity of specialist policies while keeping the generality of language-conditioned robot foundation models.

The lecture starts with a robotics problem: reactive policies lack memory and produce jitter.

Then it builds the transformer toolkit: attention for history, tokenization for turning data into sequences, scaling laws for making general models stronger, and multimodal modeling for mixing text and vision.

The final robotics move is to add actions to the same sequence. But actions need the right representation. Once actions are tokenized or compressed well, robot learning can use the same scalable sequence modeling recipe:

find the right tokens -> train a general transformer -> let data and compute do the work