Frontier & Open Problems

The robotics hype cycle deserves closer examination.

Every few weeks, someone claims robotics is close to its endgame or having its ChatGPT moment.

The warning is simple: a polished demo in one lab is not the same as a robot that works in the world.

The not-so-secret recipe for a strong robot demo is straightforward.

Collect a lot of data on the exact target platform. Train a behavior cloning policy. Keep collecting data until the task becomes interpolation for the model.

Then test under the same conditions as training: same room, same lighting, same camera setup, same table height, and the same objects if possible.

Generalization can be understood as a hierarchy.

Most impressive demos live near the small end: one object, one room, one setup, and one training distribution.

The real goal is world-scale generalization, where a robot can enter a new kitchen, face new objects, and still compose its knowledge into useful behavior.

A human does not need to have cooked in every possible kitchen to cook in a new one.

We combine object knowledge, physical intuition, task structure, and reasoning about the current scene.

Robot policies still struggle with this kind of compositional generalization.

Large internet-trained models can describe images and reason semantically.

But they often lack physical grounding: what happens when a glass is pushed off an edge, how contact forces behave, or why a fragile object breaks.

Generative video models can make physical scenes look convincing while still violating basic physics in subtle ways.

On the other side, we have embodied specialist policies that can perform dexterous or contact-rich tasks.

They are grounded in robot interaction, but they often fail when moved to new rooms, objects, camera viewpoints, or task compositions.

The open question is how to combine broad semantic generalization with real physical skill.

Passive internet data is not enough.

Robotics needs data that is multimodal, spatial, temporal, and grounded in physical interaction.

The goal is to build agents that do more than describe the world: they should reason about it and interact with it intelligently.

One camp wants to use vision-language models because they already contain semantic knowledge and scene reasoning.

Another camp wants generative video models because video better captures temporal dynamics and consequences.

A third camp argues that the backbone may matter less if we can collect enough real robot data and train from scratch.

Each backbone choice implies a different data recipe.

A VLM route leans on web-scale image-language knowledge. A video route treats internet video and egocentric video as central. A from-scratch route bets heavily on real robot experience.

The field does not yet know which recipe scales best.

The layered recipe is: use web data for broad world knowledge, simulation for cheaper scalable interaction, and real robot data to bridge the sim-to-real gap.

This is appealing because no single data source gives everything.

But simulation has its own cost: accurately simulating the whole world is extremely hard and engineering-heavy.

A video-first strategy starts from internet videos, then adds human egocentric videos, and finally post-trains on robot actions.

The intuition is that video is closer to embodied prediction than static visual question answering.

Robots need to understand how scenes evolve, not only what objects are present.

The more aggressive approach is to train on real robot data directly, possibly from scratch.

This avoids relying too much on proxies like text-image data or imperfect simulation.

The obvious challenge is cost: real robot data is slow, expensive, fragile, and hard to scale.

Different interfaces trade off data volume, embodiment alignment, and task complexity.

Egocentric human videos can scale to huge amounts of data, but they have a large embodiment gap.

Teleoperation gives robot-aligned data, but it is slower and more expensive. Wearables, UMI-style grippers, VR teleoperation, and puppeteering sit at different points in between.

Handheld gripper systems like UMI make it easier to collect in-the-wild manipulation demonstrations.

They are cheaper and more portable than full robot teleoperation.

But the data still has to transfer from the human-held interface to the deployed robot, so cross-embodiment learning becomes part of the problem.

Leader-follower arm setups can produce high-quality robot demonstrations.

The tradeoff is that they require more hardware, more setup, and more things that can break.

The broader point is that data collection is not just a dataset problem. It is a hardware, interface, cost, and alignment problem.

The dream policy should be both dexterous and general.

Dexterity needs fast control, contact handling, and rich sensing. Generalization needs large models, broad data, and semantic abstraction.

Those requirements currently pull in different directions.

A concrete failure illustrates the problem: a robot can visually estimate an object's pose but still crush it because it has no touch or force feedback.

When the gripper makes contact, vision alone may not tell the robot how much pressure it is applying.

Many current VLA-style policies are still effectively blind at the moment of contact.

Adding touch, force, proprioception, audio, and other modalities sounds natural.

But the data is scarce, sensors are expensive or fragile, and paired multimodal datasets are rare.

Different robots also use different sensors, which makes transfer across embodiments difficult.

One idea is to use language as semantic glue between modalities that were not collected together at scale.

A model may have lots of vision-language data, some audio data, and a tiny amount of touch data, without one giant paired dataset containing everything.

Language can help bind those experiences into shared concepts like rough, soft, hollow, metallic, or an object that feels like a pineapple but is brown.

Reasoning should probably be adaptive.

If the robot sees a known object in a known task, it should run the policy fast and avoid wasting compute.

If the situation is uncertain, it should spend more test-time compute. If it is truly out of distribution, it should stop and ask for help.

The hard part is knowing when the model does not know. Introspection is an open problem.

Large models can be confident pattern-completers even in genuinely novel situations.

In robotics this is harder because uncertainty can come from objects, sensors, lighting, embodiment, control dynamics, task novelty, or environment changes.

Most current robot learning relies heavily on imitation.

If the dataset only shows state 1 to 2 to 3, the policy may never learn that 1 to 3 is possible or better.

Behavior cloning reproduces what humans demonstrated; it does not automatically discover better ways to solve the task.

RL can, in principle, learn from suboptimal data and stitch trajectories into better behavior.

It has powered progress in reasoning agents and other domains.

But scaling RL for real robots remains difficult because real interaction is expensive, failures can damage hardware, and reward design and evaluation are hard.

The ideal loop is: better models make more capable robots, more capable robots get deployed more widely, deployment generates more data, and that data trains better models.

That loop would compound progress.

But the first turns of the flywheel are hard because autonomous rollout data is messy, mixed-quality, and distributionally different from expert teleoperation data.

Expert data is usually smoother, shorter, and higher quality.

Autonomous rollouts may be slower, jerkier, longer, and full of recovery behavior or partial failures.

Naively mixing those distributions into imitation learning can confuse the policy, so new algorithms are needed to learn from every kind of robot experience.

A useful robot should improve over time from its own physical experience.

The current default is usually train, freeze, deploy, and hope the model generalizes.

Lifelong learning asks how the robot can keep learning from deployment data while preserving the skills and general knowledge it already had.

Sometimes retraining is too slow.

If a home robot fails to open a fridge, a human might give language feedback like grasp more to the left.

If that is not enough, the human might demonstrate the action once. The vision is in-context learning for robotics: adapt from whatever signal is available now.

Many mobile manipulation stacks still use classical SLAM and navigation first, then switch to a learned VLA-style manipulation policy near the object.

That hard switch is awkward.

The open problem is a more unified model that can handle navigation, whole-body control, manipulation, state estimation, and task progress together.

For long-range navigation, what should the robot remember about the world?

Maybe it needs an explicit 3D map. Maybe a bird's-eye-view projection is enough. Maybe a topological map, scene graph, or walkthrough video can carry the right information.

The answer likely depends on the task, time horizon, and how dynamic the environment is.

A deployed robot produces a constant stream of video, depth, language, touch, audio, bounding boxes, actions, and outcomes.

Not all of that can be stored or retrieved naively.

The robot needs memory at multiple time scales: dense short-term sensory memory, compressed event memory, long-term semantic memory, and mechanisms for deciding what to forget.

VLMaps fuses vision-language features into 3D spatial maps.

This lets a robot use open-vocabulary language queries for spatial navigation, such as finding landmarks or objects without task-specific retraining.

It is an example of using foundation model representations inside a structured spatial memory.

LeLaN learns language-conditioned navigation from action-free egocentric video.

The important idea is that unlabeled video can still teach useful navigation priors.

This supports the broader theme: not all robot learning data has to come from expensive teleoperated robot rollouts.

PALO adapts generalist policies to new long-horizon tasks using a small number of demonstrations and language decompositions proposed by a VLM.

Instead of full fine-tuning, it uses language structure to find useful subtask sequences.

This fits the rapid-adaptation vision: use a little new information to steer a pretrained robot policy quickly.

In the personal story, the lecturer describes building infrastructure for end-to-end language-conditioned manipulation.

The CALVIN benchmark emerged almost as a byproduct of needing a simulation environment for long-horizon, language-conditioned robot policies.

The lesson is that useful research artifacts often come from building the tool you needed for your own problem.

The open problems cluster into a few pillars: intelligent embodied reasoning, dexterous mobile manipulation, lifelong learning, rapid adaptation, memory, and mobile interaction.

Solving only one is not enough.

A real-world robot will need to reason, sense contact, navigate, remember, adapt, and learn from its own deployment data.

Most research does not arrive as one giant leap. It usually builds on years of accumulated work.

Self-driving cars, for example, sit on decades of progress rather than one magic paper.

A healthy research funnel loses a lot of ideas.

You may start with many ideas, run a smaller number of experiments, find a couple of interesting results, and publish one paper.

That is not failure. That is the normal shape of research. Also, most papers do not stand the test of time.

In a field where scale matters, simple ideas often travel farther.

If an idea needs 17 moving parts to work, it may be fragile and hard for others to reproduce or build on.

A simpler version that gets 80 percent of the benefit may be more impactful.

An important problem without a plan is too vague.

A clever method on an unimportant benchmark may not move the field.

The sweet spot is an important problem where you have a concrete angle of attack and can explain what becomes possible if the idea works.

Research takes a long time, and many experiments fail.

Deadlines, rejections, and hardware problems make the process painful.

If the problem genuinely excites you, the hard parts feel worth enduring.

Before committing to a long project, ask the harsh question: what is the most likely reason this idea fails?

If you can answer honestly and still believe the idea is worth pursuing, continue.

This protects the most valuable research resource: time.

Method-driven research starts with a technique and looks for a problem to apply it to.

Problem-driven research starts with a real bottleneck and finds or invents whatever method solves it.

Both can work, but problem-driven research gives you an anchor if the first method does not pan out.

Do not begin at full complexity and then wonder why the experiment failed.

Start with overfitting, simple baselines, controlled settings, and incremental difficulty increases.

Always visualize model inputs and outputs so you can see what is actually happening.

Discuss ideas with colleagues and with authors whose work you build on.

A short conversation can save weeks of duplicated effort or point out a failure mode you missed.

Research is not supposed to happen in a sealed room.

A great paper that nobody reads has little impact.

Share results, release code and data when possible, polish figures, and explain the work for the audience you want to reach.

The caution is to communicate clearly without sliding into robotics-is-solved hype.

The personal story starts with frustration: too many ROS nodes, separate perception, planning, and control modules, brittle calibration, and errors accumulating across the pipeline.

That pain led to the question of whether end-to-end language-conditioned manipulation could replace the brittle stack.

The benchmark and later demos came from building the infrastructure needed to ask that question seriously.

Robot learning remains unfinished in the best possible way.

There are real open problems everywhere: data, sensing, reasoning, RL, lifelong learning, memory, navigation, adaptation, evaluation, and communication.

The goal is not only to learn the existing toolkit, but also to see where the field still needs new ideas.