Egocentric world models
Do the task once with your own hands. The robot learns it from whatever cameras were watching — no teleoperation, no scripted path.
Visualisation · procedural scene
The landscape
Manipulation is not the language story. Language is already the most distilled thing humans produce. The physical world is far higher-dimensional and has no corpus.
01Pixel-space world models
Trained to generate what a scene will look like next. Remarkable video, and nothing a body can act on. A plausible next frame is not a metric scene, so nothing in the output tells a robot where anything actually is.
02Vision-language-action
Trained to map an instruction to joint commands. They work, and the price is tens of thousands of teleoperated demonstrations. Change the robot and the policy does not come with you.
Both camps, and the industry push toward ever more perfect simulators and digital twins, are reaching for a world model that is not currently buildable. The useful thing sits between them, and it is smaller than either: a scene a body can act in, and a record of what the person was trying to do in it.
The approach
Three layers. A foundation model over the top two, and a bottom layer that makes it portable across bodies.
What the person was trying to do, in what order, and why that grip and not another one. The part a demonstration actually carries and a trajectory does not.
One coherent, measurable scene fused from however many cameras happen to be watching — a headset, the robot’s own wrist, a phone on a stand — with poses changing online as people and robots move. Geometry a body can act in, rather than pixels from a viewpoint.
Embodiment, actuators, camera intrinsics and placement. Different bodies resolve into the same scene, which is what lets a skill move between them.
Why now
Self-supervised visual features, feed-forward 3D from unposed images, and video-native prediction all became usable across 2024 and 2025. None of this was buildable two years ago.
Built on
Public backbones, with our own forecaster on top — the model that predicts how the scene and the hands move next, which is what turns a reconstruction into something a robot can act ahead of.
Status
The loop
Closer to how you work with a capable colleague than to how you program a machine.
A person does the task once with their own hands, wearing camera glasses. Any other view in the room joins in — the robot’s wrist camera, a phone on a stand. No teleoperation rig, no scripted trajectory, no motion capture suit.
The robot asks when it is unsure, instead of failing confidently. Uncertainty becomes a question rather than a dropped part.
Human and robot in the same space on the same task, with the person able to correct it mid-task rather than after it.
Per-environment fine-tuning runs online until the correction loop goes quiet. Full automation is the end state, not the starting requirement.
Where this goes
Everything in this section is where the work is aimed, not what runs today.
Human intention and task semantics, transferred to robots. Collaboration first, because that is what is reachable now, with full autonomy as the end state rather than the entry price.
Environments reconstructed from point clouds of real, unknown spaces, more dynamic than what current simulators offer. Articulation data is the gap, and it is the one we have to close.
Easier cross-embodiment, better handling of long-horizon tasks, and policies that keep improving online in the environment they were actually deployed in.
Get in touch
If you run a robot-learning lab, build with manipulation, or invest in this, we should talk.