CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation
8 min read
Measure how much robot control comes from the coding model versus its tools, then recover reliability with execution feedback, visual differencing, reusable skills, and parallel reasoning.
NVIDIA / UC Berkeley / Stanford / CMU · March 2026, revised July 2026
The idea to remember
A coding agent can look capable when a hand-written API already solves most of the manipulation problem. CaP-X varies that scaffolding explicitly, then shows how additional interaction and reasoning can recover performance over lower-level primitives.
What improves: CaP-Agent0 keeps model weights fixed and improves the inference-time harness and reusable skill library. The separate CaP-RL experiment post-trains the coding model itself with reinforcement learning.
The architecture
How the improvement loop works
Step 1
Observe through CaP-Gym
Receive the task and available APIs. A stateful Python executor connects generated code to perception services, motion planning, and robot control in simulation or on hardware.
Step 2
Reason with grounded feedback
A Visual Differencing Module describes the scene and what changed. Parallel coding-model queries propose solutions; a central coding agent combines them into one program.
Step 3
Execute, inspect, continue
Run the program, inspect execution logs and visual feedback, and generate another program if necessary. Reusable helpers reduce repeated low-level implementation work across trials.
What the agents do
- Visual Differencing Module
- Converts the initial scene and subsequent visual changes into text for the coding agent. This is a specific alternative to feeding raw RGB images directly into the code-generation context.
- Parallel coding candidates
- At each turn, generate nine candidates: nine queries to one model, or three each to Gemini-3-Pro, GPT-5.2, and Claude Opus 4.5. Sampling diversity broadens the search.
- Central coding agent
- Synthesizes candidate programs into a final code snippet and participates in multi-turn continuation decisions. The environment executes that combined program.
- Skill synthesis pipeline
- Extracts helpers from successful low-level rollouts, then uses Gemini-3-Pro to identify recurring task-agnostic logic. The reported implementation produces nine verified utilities in one synthesis pass.
Four names, four different parts of the framework
CaP-Gym is the execution environment: generated Python programs call perception and control primitives through a REPL-like interface. One code turn can contain many primitive calls and many internal simulator or controller updates.
CaP-Bench is the controlled evaluation. It varies primitive abstraction, single-turn versus multi-turn interaction, and perceptual grounding. CaP-Agent0 is the training-free harness designed from these results. CaP-RL is a separate experiment that trains the code-generating model with rewards from robot tasks.
The benchmark separates model capability from scaffolding
The full CaP-Gym release integrates 187 tasks: seven Robosuite tasks, 130 LIBERO-PRO tasks, and 50 BEHAVIOR tasks. The main 12-model analysis uses seven core tasks, each evaluated with 100 trials per tier. These two scopes should not be confused.
S1 through S4 are single-turn tiers; M1 through M4 allow multiple turns. High-level APIs encode more human-designed task structure. Low-level APIs leave more perception, geometry, and control composition to the model. S1 also provides privileged simulator state, so the tier comparison changes more than code abstraction alone.
Within a trial, the environment is not reset between code turns. Success therefore includes inspection, debugging, and recovery in the scene produced by previous actions. A multi-turn trial is not equivalent to repeated independent attempts from a fresh initial state.
Why text about visual changes helps
In the reported experiments, execution logs improve recovery, while directly interleaving raw images can hurt code-generation performance. The Visual Differencing Module instead turns observations into structured natural-language descriptions of the scene and its changes.
This result concerns the tested models and interfaces. It does not establish that raw images are always inferior. CaP-Agent0 combines this grounding method with reusable helpers and an ensemble, so its full performance should not be attributed to any one component.
Persistent skills, without claiming an endless learning run
The skill library pools successful S3 rollouts across all 12 models and seven Robosuite tasks. Function definitions are extracted, and a model identifies common utilities that can be reused with low-level interfaces. The resulting library is shared across trials rather than specific to one coding model.
The paper describes a single synthesis pass yielding nine verified utilities. It motivates iterative updates as more successful experience becomes available, but that possibility should not be presented as a demonstrated long-running self-improvement curriculum.
CaP-RL changes the model, not just the harness
CaP-RL applies GRPO to Qwen2.5-Coder-7B-Instruct on Cube Lift, Cube Stack, and Spill Wipe. Training uses privileged S1 state-based APIs to reduce perception noise in the reward signal. After 50 iterations per task, evaluation uses noisy S2 perception.
The coding model learns to compose shared tools; it is not merely writing a reward function for a separate motor policy. On a real Franka, lifting rises from 24% to 84% and stacking from 12% to 76%, with 25 trials per task. These are CaP-RL results, distinct from Agent0’s training-free demonstrations.
What the experiments show
- 187 tasks
- Full CaP-Gym release
- The core CaP-Bench analysis covers seven tasks and 12 models, with 100 trials per task per tier.
- 4 / 7
- Agent0 matches expert code
- Comparable or better success on four core tasks. The reference is human-written programs in a single-turn setting, not human teleoperation.
- 84% / 76%
- CaP-RL on real Franka
- Cube lift / cube stack over 25 trials each, compared with 24% / 12% before RL. This result updates the coding model weights.
Where humans still contribute
- People design and implement the perception and control interfaces, task definitions, evaluation protocols, and human-written reference programs. High-level tiers intentionally contain more human-crafted structure than low-level tiers.
- Agent0 synthesizes helpers from successful execution traces rather than requiring a person to hand-code each library utility. It still depends on the supplied robot stack and pretrained perception and coding models.
- Some physical demonstrations allow optional human feedback between turns, such as pointing out that a grasp was too high. They should not all be described as intervention-free autonomous evaluations.
How to read the result
- The seven-task, 12-model benchmark is smaller than the 187-task environment release. Agent0’s broader task studies do not imply that all 12 models were evaluated across the entire suite.
- Task success and code execution success measure different things. A program can execute without errors while its grasp or placement fails physically.
- The four-of-seven expert comparison uses single-turn human-written code as the reference. It does not establish superiority to expert humans with matching recovery and computation budgets.
- CaP-RL’s quantitative real-world transfer covers two tasks with 25 trials each. Training-free Agent0 examples on other embodiments are a separate source of evidence.
Figures and numerical results are attributed to the linked paper. The step cards are explanatory summaries.