Towards Human-Like Manipulation through RL-Augmented Teleoperation and Mixture-of-Dexterous-Experts VLA
9 min read
Keep a pretrained VLA in charge of broad visual-language behavior, then route force and fingertip contact through sparse experts that add action-specific corrections exactly where contact begins to matter.
the design choice
Do not make contact sensing pretend to be vision
MoDE keeps the pretrained visual-language pathway intact and gives fast physical signals a dedicated route into action generation.
Naive concatenation
One state vector
Arm torque, fingertip contact, and proprioception are flattened into one interface despite having different meanings and time scales.
Dedicated contact pathway
Attend, route, correct
Read VLA context
Choose an expert
Add a residual
mode-vla architecture
The backbone proposes; MoDE corrects
Force and tactile tokens read the same task context as the action expert, but only their residual corrections reach the final velocity field.
Pretrained proposal
3 camera views
Task prompt
Robot state
Noisy actions
π₀ action expert
PaliGemma context plus a Gemma-300M suffix produces 50 flow-matching action slots.
Contact refinement
Force f ∈ R¹⁴
Arm joint torques: 7 joints × 2 arms
Tactile g ∈ R⁶⁰
6D wrench: 5 fingertips × 2 hands
Self-attention + sparse router
Contact tokens attend to the visual-language prefix and denoising suffix, then each token selects one of eight MLP experts.
Residual outputs
Force route
Arm action adjustment
Contact onset and insertion resistance
Tactile route
Hand action adjustment
Fingertip slip and grasp stability
why token-level routing matters
One sensor snapshot becomes 50 routing decisions
The same current reading is copied across the action horizon and given a different step encoding at every position. Experts can therefore change inside one predicted chunk.
h1
reach
E2h2
reach
E2h3
contact
E6h4
insert
E3…
track
E3h50
settle
E7These regime labels illustrate the specialization the architecture enables. The paper motivates contact-onset and steady-state experts, but does not publish expert-usage traces that prove each expert learned a named role.
the small hierarchical piece
MoDE refines control until the VLA delegates the hand
IMCopilot matters at inference, but it is a separate RL skill rather than one of the eight dexterous experts.
Option 1: VLA drives the hand
Flow matching generates arm and hand actions. MoDE adds force-guided arm residuals and tactile-guided hand residuals.
Option 2: IMCopilot drives the hand
The RL rotation primitive overrides the VLA's hand actions. The VLA still produces the arm motion, refined by MoDE's force route.
what the experiments support
Contact sensing helps, with force carrying the larger average gain
Each result uses 20 real-robot trials. Apple peeling combines MoDE-VLA with the separate IMCopilot skill.
Apple peeling
73% peel completion
Tube rearranging
Bimanual
Gear assembling
Force-sensitive
Charger plugging
Force-sensitive
Remove force
Average success falls from 34% to 23%. Contact onset and compliant insertion lose their strongest cue.
Remove tactile
Average success falls to 26%, mainly through more slip in grasp-intensive phases.
Important limit
No reported ablation isolates sparse MoE routing from the rest of the sensor-fusion pathway.
takeaways
What is worth carrying forward
Preserve the prior
Use residual corrections when a new sensor is most useful locally and the pretrained policy is already strong elsewhere.
Route by future step
Per-horizon tokens let one action chunk move through several contact regimes without paying for every expert at every position.
Separate semantics
Arm torque corrects arm motion; fingertip wrench corrects hand motion. The output structure mirrors what each sensor measures.
The mental model
MoDE-VLA is best understood as a contact-aware refinement layer on top of π₀, not as a collection of complete VLA policies. The pretrained backbone still interprets the camera views, instruction, robot state, and noisy action chunk. MoDE gives force and tactile sensing a separate route so those signals do not have to compete with vision and language inside an interface learned mostly from parallel-gripper data.
A single current force reading and a single tactile reading are each projected into the backbone width, copied across all 50 future action positions, and tagged with step encodings. After attention lets them read the VLA context and current denoising state, a top-1 router chooses one of eight expert MLPs independently for each token. This gives different points in the same action chunk room to specialize for free motion, contact onset, steady force, or slip.
The outputs remain deliberately narrow. Force-conditioned tokens correct arm actions, while tactile-conditioned tokens correct hand actions. Both are added residually to the action expert's suffix, so the side path can become quiet in free space instead of overwriting the useful behavior already stored in π₀. For the hardest in-hand rotation, the VLA can also emit a trigger that hands control to IMCopilot, while MoDE continues refining arm motion.
The evidence is promising but early. Across 20 trials on each of four tasks, the full system raises average success from 15% to 34%. Removing force costs 11 points and removing tactile costs 8, but the paper does not report a clean ablation that removes only the sparse expert block while retaining both sensor streams. Apple peeling also mixes MoDE with IMCopilot, so its improvement should not be credited to the VLA fusion module alone.
Sources