MEM: Multi-Scale Embodied Memory for Vision Language Action Models
8 min read
A VLA remembers recent experience as video and long-running task state as a compact language summary - enough context to act coherently for up to fifteen minutes.
why memory is hard
Longer history creates two different bottlenecks
Giving a policy more context is not automatically helpful. The model must learn to use that history reliably, and still act within a robot's real-time control budget.
Effectiveness
More context can make policies worse
A policy can learn shortcuts from its history instead of the real cause of an action, then fail when deployment histories differ from clean demonstrations.
Distribution shift between training and repeated real-world failures
Too little diverse long-sequence data to learn robust memory use
MEM’s response
Compress the language memory and pre-train visual memory on diverse robot, correction, and internet video data.
Efficiency
More context costs more compute
Every extra observation adds visual tokens and attention work. Long histories make both training and inference slower and more resource intensive.
More activations and attention make training slower
Inference latency grows while the robot still needs real-time control
MEM’s response
Compress short-term video before the VLA backbone and store distant semantic state in compact text.
paper architecture
High-level memory meets low-level control
The high-level VLM generates the updated memory and next subtask as natural-language strings. The low-level VLA combines the subtask with dense video memory to produce continuous actions.
decompose the context
Two memory types, each with a different job
Short-context keeps dense frames from the last few seconds for dexterous manipulation. Long-context compresses minutes of history into language for general reasoning.
Short-context
Dense frames for dexterous manipulation
A dense window of recent images and proprioceptive state preserves the motion, geometry, and contact details needed for fine control.
Fine-grained scene and robot dynamics
Objects hidden by the robot’s own arm
Evidence needed to retry with a different strategy
Long-context
Compressed language for general reasoning
A compact language summary carries minutes of task progress without forcing the policy to replay every frame.
Which recipe or cleanup steps are complete
Where useful objects were placed
What remains open, dirty, missing, or unfinished
efficient visual memory
Compress time before the backbone
Naively passing every frame into the VLA makes latency climb with history length. A key property of MEM's video encoder is that it introduces no new learnable parameters compared to standard, single-image ViTs.
learned memory writing
The model decides what is worth remembering
Training targets are generated from annotated subtasks and their success or failure. The high-level policy learns to rewrite the summary, not append an endless transcript.
Before · mₜ
relevant past“I placed a plate in the cabinet and moved to the counter.”
New event
Picked up the next bowl successfully.
After · mₜ₊₁
rewritten“I placed a plate in the cabinet, moved to the counter, and picked up a bowl.”
Minimal semantic state, ready for the next decision.
Select
Keep only facts that can change a future decision.
Compress
Replace incidental details with a shorter semantic fact.
Update on progress
A repeated failed subtask need not expand the summary.
what memory unlocks
Remember state, recover from mistakes, keep moving
The useful result is not a bigger context window. It is behavior that stays coherent when the relevant evidence is no longer visible.
Long tasks
Tracks recipe and cleanup state for tasks lasting up to fifteen minutes.
recipe setup · clean kitchen
Partial observability
Recalls hidden objects, occluded targets, counts, timing, and cleaned regions.
find object · unpack groceries
Adapt after failure
Uses recent failed attempts to change grasp height or door-opening direction.
+11% chopstick · +62% fridge
Retains dexterity
Matches the memoryless π₀.₆ policy on hard manipulation tasks that do not require memory.
no observed memory tax
Core takeaway
Memory for embodied agents is a representation problem, not just a context-length problem. Keep recent experience in the modality that preserves detail; carry distant history in the modality that compresses meaning.
The mental model
MEM does not force one representation to remember everything. Raw observations preserve the fine details needed to recover from an occlusion or change a failed grasp, but they are expensive. A short language summary is cheap enough to carry across an entire recipe, but it cannot preserve motion and geometry. MEM gives each kind of memory the job it is best suited for.
The high-level policy reads the task, current scene, and previous language memory. It chooses the next subtask and rewrites the memory with only the semantic facts that remain useful. The low-level policy receives that subtask together with a dense window of recent observations, then its action expert produces continuous robot actions.
The key engineering trick is to compress time before the visual tokens reach the VLA backbone. Spatial attention processes patches within each frame; periodic causal temporal attention lets matching patches gather evidence from earlier frames. Past-frame tokens are then dropped, leaving one current-frame-sized representation that already contains recent history.
Sources