The Infrastructure Layer for Physical AI

Robotics has better models than ever.

What it still does not have is enough worlds.

@svlevine and the Physical Intelligence team have described the problem plainly: language models can learn from an enormous body of internet data; robotics has no equivalent treasure trove. For a new robot, task or environment, somebody still has to collect physical interaction data. And for complex, long-horizon tasks, simply covering every plausible situation with more real-world collection becomes impractical.

@JensenHuang and NVIDIA are attacking the same bottleneck from another direction: simulation, synthetic data and physically grounded virtual environments. NVIDIA's own framing is that physical AI needs high-fidelity 3D environments where robots can learn safely before deployment.

And @drfeifei has spent the last few years making an even broader argument: spatial intelligence is still a missing component of modern AI. Language alone is not enough for systems that have to understand and act inside a 3D world.

So the industry broadly agrees on the problem.

The harder question is how to manufacture enough useful worlds.

The approaches we have today only solve pieces of it

1. Collect more real robot data

This is still the gold standard.

Teleoperation, fleet data and real deployments give you the dynamics, sensor noise, failures and weird edge cases that simulation struggles to reproduce.

But the economics are ugly.

Every extra kitchen, warehouse, object distribution, lighting condition, robot embodiment and failure mode costs physical time. Scaling the dataset also means scaling hardware, operators, maintenance and safety procedures.

Real data is indispensable.

It is just a very expensive way to discover the long tail.

---

2. Build everything in simulation

Isaac Sim, Omniverse and conventional game-engine pipelines give us something video models cannot: persistent geometry, physics, cameras, collision, controllable robots and repeatable experiments.

The weakness moves upstream.

Someone still has to build the world.

Factories need machines. Kitchens need cabinets, appliances and clutter. Warehouses need shelves, pallets, aisles and realistic layouts. Assets have to be modeled, placed, textured and configured.

Procedural generation helps, but purely procedural scenes often look procedural because they are built from predefined rules and asset libraries.

This is why synthetic-data scaling is not only a simulation problem.

It is also a world-authoring problem.

---

3. Generate the world as pixels

Video world models have pushed the opposite direction.

Instead of manually constructing a simulator, generate what the agent should see.

That gives remarkable visual diversity and can be extremely useful for augmentation, prediction and learning representations.

But pixels are not automatically a simulator.

A beautiful generated warehouse video does not necessarily give you:

• persistent object geometry
• editable objects
• collision surfaces
• navigation structure
• reusable assets
• arbitrary camera access
• a stable environment that an embodied agent can revisit

Gaussian-splat world generators sit somewhere between these two worlds. They can create impressive explorable environments, but the representation is still different from a conventional object-level mesh scene.

The WorldGen paper makes this distinction concrete in its comparison with World Labs' Marble: Marble preserves strong visual quality near its conditioned viewpoint, while fidelity drops as the camera moves farther away. WorldGen instead targets a broader, explicitly traversable area represented as geometry.

Likewise, today's strongest single-image-to-3D systems are excellent at objects and compact scenes, but the WorldGen authors found their geometry and texture resolution insufficient for the kind of compositional, large-scale environment they wanted to put directly into an interactive engine.

None of these approaches is useless.

They optimize different parts of the stack.

That is what makes WorldGen interesting.

WorldGen doesn't choose between procedural simulation and generative AI

It combines them.

@AIatMeta's WorldGen, developed by Dilin Wang, Rakesh Ranjan, Andrea Vedaldi, @t_monnier and a large Reality Labs team, starts from a text prompt and produces an explicit, textured, traversable 3D scene.

Not simply an image of one.

An actual world made from meshes.

The pipeline is surprisingly practical.

1. Plan before generating

An LLM interprets the prompt and converts it into parameters for a procedural scene generator.

The procedural stage establishes coarse terrain, spatial layout and object placement.

Most importantly, it produces a navigation mesh.

So traversability is not something WorldGen hopes will appear after generation.

It is part of the specification given to the generator.

2. Let generative models decide what the world looks like

The blockout is rendered, and an image generator produces a reference view that establishes appearance, style and scene detail.

Procedural generation therefore provides structural constraints.

Generative AI provides visual diversity.

Neither has to do the other's job.

3. Reconstruct the whole world together

WorldGen then extends Meta's AssetGen2 so that reconstruction is conditioned not only on the reference image, but also on the navmesh.

This matters.

Generating every building, rock and tree independently would make local assets look good while destroying global spatial consistency.

Instead, WorldGen first generates a holistic scene.

The paper's 50-scene evaluation reports a navmesh Chamfer distance of 0.022, versus roughly 0.038–0.049 for the compared models—a roughly 40–50% reduction in this alignment error.

4. Break the world back into objects

A single giant mesh would still be awkward to use.

So WorldGen adapts AutoPartGen to decompose the reconstructed environment into individual components and then enhances those objects at higher resolution.

That decomposition reaches an [email protected] of 0.853 in the paper's benchmark and takes about one minute, versus roughly ten minutes for the reported AutoPartGen baseline.

The result is the important part:

a coherent world globally, but editable objects locally.

That is much closer to what an actual production environment needs.

Why this matters outside a paper demo

WorldGen's generated environments span roughly 50 × 50 meters, can be freely navigated, contain explicit textured meshes and are designed for standard game-engine workflows. The reported end-to-end pipeline runs in roughly five minutes.

That changes the economics of synthetic environment creation.

Imagine a robotics team starting with:

“A crowded fulfillment warehouse with narrow aisles, loading zones, pallets and irregular obstacle placement.”

Instead of asking a 3D team to build fifty warehouse variants, a system in this direction could generate the structural starting point automatically.

Then vary:

layout.

density.

obstacles.

appearance.

terrain.

lighting.

object placement.

And because the output is explicit geometry rather than a generated video, those worlds can become inputs to an actual simulation stack.

For mobile robots, navigation, perception and planning stress tests are obvious applications.

For embodied AI, generated environments could become another axis of curriculum diversity.

For synthetic-data pipelines, the environment itself can become generative rather than being the fixed stage on which randomization happens.

There is an important distinction here:

game-engine-ready does not yet mean robot-ready.

WorldGen demonstrates environment generation.

It does not demonstrate a complete robot simulator.

What WorldGen still does not solve

The paper is unusually clear about several limits.

Its single reference-view setup currently restricts generation to bounded, single-story environments. Truly unbounded worlds and multi-layer layouts remain outside the demonstrated system.

There is also no asset instancing/reuse yet, which can hurt runtime efficiency in dense scenes.

More importantly for robotics, WorldGen's paper evaluates geometry, decomposition, traversability and scene generation.

It does not establish that the generated worlds reproduce all of the physical properties a robot requires.

A manipulation simulator also cares about:

mass.

friction.

compliance.

contact.

articulation.

object affordances.

sensor characteristics.

dynamic state.

And eventually, whether policies trained inside that world transfer back into reality.

Those are different problems.

WorldGen builds the stage remarkably quickly.

It does not yet generate everything that can happen on that stage.

The next step is not a bigger WorldGen

It is WorldGen + World Dynamics.

This is where several currently separate research directions should meet.

WorldGen gives us:

prompt → explicit 3D world

Physics engines give us:

3D world + actions → physical state transitions

World action models give us:

observations + actions → learned future

Robot foundation models give us:

observations + goals → actions

Put those together and the synthetic-data loop starts looking much more interesting:

Generate world → populate semantics and physics → insert robot → generate or execute behavior → observe consequences → select useful failures → train policy → validate in reality.

@ylecun has repeatedly emphasized the underlying principle: an intelligent system that acts needs some ability to predict the consequences of its actions.

That is the gap after WorldGen.

Not simply generating more beautiful scenes.

Generating operational worlds.

Worlds with explicit geometry, semantics, physics and controllable dynamics.

Worlds that can be randomized automatically but still obey constraints.

Worlds where a robot can fail 100,000 times without breaking 100,000 real objects.

Worlds that can eventually be reconstructed from reality, altered counterfactually, simulated forward and converted back into training data.

That is when world generation stops being primarily a graphics problem.

It becomes robotics infrastructure.

WorldGen does not solve that entire stack.

But it solves one layer that has been surprisingly expensive for a very long time:

creating the world itself.

#WorldGen #Robotics #PhysicalAI #WorldModels #EmbodiedAI #RobotLearning #Simulation #SyntheticData #3DGenAI #SpatialIntelligence