MolmoMotion: Forecasting Point Trajectories in 3D with Language Instruction
5 min read
A vision-language model that forecasts where an object's points travel in 3D from one language instruction and a few frames.
how it works
Inputs & encoding → model → outputs
Every modality is turned into tokens, fused inside one Molmo2 backbone, then decoded into future 3D motion by two interchangeable heads.
Inputs & Encoding
RGB history
SigLIP2 ViT · 14px patches → 1152-D tokens
Action text
Qwen3-4B tokenizer → text tokens
2D query points
Bilinear-sampled t0 features → 1 token / point
Initial 3D points
Metric XYZ deltas from the first query point
Fused into one token sequence; RGB patch tokens dominate the count (bar is schematic, not to scale).
Model
Molmo2-4B
VLM backbone: SigLIP2 vision + Qwen3-4B language.
Reads the token sequence and conditions both prediction heads.
Outputs
Autoregressive head
The unmodified Molmo2 LM head emits millimeter-quantized <track> coordinate text.
loss · next-token cross-entropyFlow-matching head
A 36-block DiT expert denoises continuous future XYZ from Gaussian noise.
loss · flow-matching MSEFinal output
future_3d · P×F×3Absolute XYZ meters in the t0 world frame.
reference figure
The same flow, as drawn in the paper
Image and point tokens enter Molmo2; the autoregressive variant decodes quantized coordinate text, while the flow-matching variant works directly in continuous 3D.
training data
How MolmoMotion-1M is built
No manual labels: an automatic pipeline turns raw public videos into 3D point-track supervision.
Caption → object
Qwen3-0.6B extracts the manipulated-object phrase; Molmo2-8B re-captions vague clips.
Query points
MolmoPoint-Vid-4B localizes the object, SAM 3 masks it, then K-means spreads points over the surface.
2D tracks
AllTracker follows the dense 2D points through time and outputs visibility masks.
Metric 3D
ViPE estimates depth, intrinsics, and camera poses; visible points back-project into a shared t0 frame.
Clean tracks
MAD outlier filtering drops bad tracks; Stereo4D-style smoothing reduces depth jitter.
Motion clip
Keep intervals where median 3D displacement is meaningfully above the static baseline.
Figure 3: overview of the data annotation pipeline (paper).
in short
The representation, distilled
Target representation
Object-attached 3D points in a t0-anchored metric frame, not pixels, camera poses, or robot actions.
AR coordinates
Anchor-relative dx/dy/dz deltas quantized to millimeter bins and serialized as <tracks> text.
FM coordinates
The flow head keeps coordinates continuous and learns a velocity field from noise to the clean future.
Why it matters
Sparse world-frame points stay class-agnostic, view-stable, and compact for robot planning or video generation.
The mental model
MolmoMotion uses Molmo2-4B as the grounding backbone: SigLIP2 handles RGB patch tokens, Qwen3-4B handles language, and sampled point features bind each query point to the object surface. The model then predicts future XYZ trajectories either as structured coordinate text or as a continuous flow-matching trajectory.
The crux is the representation: sparse object-attached points in a world frame. This keeps motion class-agnostic, view-stable, compact, and directly useful for downstream robot planning or video generation.
Sources