Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons
8 min read
A video-language reward model turns robot rollouts into three complementary signals: how far the task has progressed, whether it succeeded, and which of two trajectories did better.
model interface
One critic, two ways to ask
Robometer reads a task instruction and sampled visual trajectory, not robot actions, state, or proprioception. A second trajectory activates its comparative judgment.
Input
Task instruction
“place the spoon on the plate”
Trajectory A
up to 8 frames
Trajectory B
optional comparison
Robometer
Qwen3-VL-4B-Instruct
shared causal backboneOutputs
Progress
p₁ … pₜ
Expected completion at each queried video prefix
Success
s₁ … sₜ
Per-frame probability that the task is complete
Preference
P(A ≻ B)
Probability that trajectory A is better than B
Single-video mode: progress + success. Paired-video mode: those signals for A plus the A-versus-B preference probability.
architecture
Special tokens turn one video-language model into three heads
A progress token follows each frame in trajectory A. A split token separates trajectories, and one final preference token summarizes the comparison.
Progress + success heads
Each <pₜ> can attend to the task and frames up to time t, so it produces a local judgment without looking ahead.
Preference head
The final <pref> token sees both trajectories, then chooses which better follows the instruction.
reference figure
Comparisons make imperfect trajectories useful
The paper constructs pairs from differing expertise, instruction mismatches, and rewound video segments. Preference supervision supplies ordering even when absolute progress labels are unavailable.
training pairs
Three routes to a useful preference label
The pair only needs a reliable ordering; it does not need a hand-authored numeric reward at every frame.
Different expertise
expert ≻ failure / suboptimal
Teaches what higher-quality completion looks like for a task.
Different tasks
correct instruction ≻ mismatch
Prevents generic motion from scoring well for the wrong goal.
Video rewind
original ≻ rewound segment
Shows that progress may reverse after a mistake or regression.
learning objective
Absolute progress anchors magnitude; preference supplies order
The three losses are trained jointly: L = Lpref + Lprog + Lsucc. Each signal answers a different downstream question.
Progress is a distribution
Successful demonstrations receive a roughly linear progress target. The model predicts one of 10 bins from 0 to 1; their probability- weighted expectation becomes a continuous reward.
A reward trace can fall after a regression. The red bar illustrates the kind of decrease encouraged by rewind augmentation.
Success says when to stop
A binary head estimates whether the task is complete at each queried point. It is related to, but separate from, progress.
Preference says which is better
A relative label can use abundant failures and suboptimal runs without pretending their exact progress is known.
Dense inference from sparse views
Robometer queries successive prefixes while retaining the first and current frame, producing a time-varying trace from sampled visual context.
evidence and boundary
It generalizes across videos, but remains a video-only judge
The paper evaluates 976 out-of-distribution trajectories from six embodiments and reports stronger ranking and downstream policy learning than its baselines.
0.94
OOD value-order correlation
VOC r on the paper's held-out trajectory benchmark
0.64
OOD rank correlation
Kendall τ across out-of-distribution trajectories
85%
Online RL success
single-stage task; RoboReward 55%, base π₀ 20%
What the score cannot directly observe
No force, contact, depth, robot state, or action stream enters the model. Sparse frames can also omit brief safety-critical failures. Visually similar outcomes, such as placing versus dropping, may therefore collapse to similar rewards even when their physical quality differs.
Read the outputs separately
Progress: how far along?
Success: is it complete?
Preference: which rollout is better?
The mental model
Robometer is a critic, not a controller. Give it a language task and a sampled video trajectory, and it estimates progress and success from visual evidence alone. Give it two trajectories, and it also predicts which one better satisfies the instruction. The same Qwen3-VL backbone supports all three judgments.
The important training idea is to combine absolute and relative supervision. Successful demonstrations provide a rough progress clock; expert-versus-failure, mismatched-task, and rewound-video pairs teach the model how trajectories should be ordered. This lets failed and suboptimal data improve the reward model even when precise frame-level labels are unavailable.
Its boundary is equally important: Robometer sees pixels and the task description, not force, depth, actions, or proprioception. Two outcomes that look alike in sparse frames, such as gently placing an object versus dropping it from a small height, can therefore receive similar scores. Progress is an estimate, and the separate success head should not be confused with a guaranteed terminal score of 100%.
Sources