VLS: Steering Pretrained Robot Policies via Vision-Language Models
4 min read
A vision-language model writes differentiable rewards that steer a frozen robot policy toward new task constraints, without fine-tuning.
From the paper
Overview and method
How it works
A policy may already know how to place an object but fail when the target moves or an obstacle appears. VLS keeps the diffusion or flow-matching policy frozen and adjusts its action generation at deployment.
- Ground the scene. SAM masks, DINOv2 features, and depth produce task-relevant 3D keypoints. A VLM decomposes the instruction into stages and writes a differentiable PyTorch reward for each stage.
- Guide denoising. Reward gradients refine action proposals, RBF repulsion preserves diversity, and Feynman-Kac resampling favors higher-reward candidates. Gradients pass through the generated reward program, not through the VLM.
- Close the loop. Reward feedback adjusts guidance strength. Separate stage-switching thresholds provide hysteresis, reducing repeated transitions near a stage boundary.
The whole task does not need to fit into one action chunk. The VLM chooses a task-dependent sequence of stages, and each stage can span multiple chunks. VLS observes the scene and steers successive chunks with the current stage's reward until feedback supports moving on. For example, moving a cup to a shelf may take several chunks within the same stage.
Three mechanisms behind steering
RBF diversity: keep alternatives alive
RBF stands for radial basis function: a function of distance. VLS uses distance-based repulsion between candidate action trajectories during early denoising so they do not all collapse onto the same option. Think of keeping both a leftward and a rightward approach available while exploring how to reach an object. Distance here is between action sequences, not a physical force applied to the robot.
Feynman-Kac resampling: give promising candidates more chances
Each candidate is a particle with a reward-based weight. VLS normalizes these weights and randomly samples a replacement batch with replacement. Better-scoring trajectories are more likely to be copied; weaker ones may disappear. For example, paths that clear an obstacle can receive more of the next round's refinement effort. This selects among candidates without gradients, while gradient refinement adjusts the candidates themselves.
Schmitt trigger: avoid switching stages on noisy feedback
A Schmitt trigger uses two thresholds, creating a buffer called hysteresis. In VLS, reward above the upper threshold signals advancement, reward between the thresholds maintains the stage, and reward below the lower threshold reinforces it. A VLM interprets the outcome when switching is triggered. For a grasp-then-lift task, this helps prevent uncertain grasp feedback from repeatedly triggering stage changes.
See Sections IV-B and IV-C for the sampling and stage-control rules.
Results and tradeoffs
On LIBERO-PRO, the reported overall success rate for π₀.₅ (LeRobot) rises from 23.69% to 36.81% with VLS, a 13.12 percentage-point gain across task and position perturbations. In the real-world unseen-mug experiment, success rises from 0% to 40% over 20 trials.
Training-free adaptation still costs inference time: multiple candidates, refinement steps, and resampling add latency. The method also depends on the VLM generating an appropriate task decomposition and reward. These experiments demonstrate useful adaptation, rather than guaranteed success under arbitrary scene changes.
Inference time
In the CALVIN door_left experiment, average inference takes roughly 0.7 to 1.2 seconds as the number of candidate trajectories increases. More candidates give steering more options, at the cost of latency.
| Candidate trajectories | Average inference time |
|---|---|
| 1 | 665 ms |
| 2 | 799 ms |
| 5 | 953 ms |
| 10 | 1239 ms |
These are overall inference measurements from Figure 4, rather than timings for the gradient update alone. VLM reward generation latency is not reported separately. Each prediction produces an action chunk containing multiple future actions, so the timing is not a delay for every individual motor command.
Sources