# Physical AI Datasets and World Models for Robotics Training

> What physical AI datasets and world models are, which open ones exist (Open X-Embodiment, DROID, Cosmos, V-JEPA 2, Genie 3), and how to combine them.

Source: https://hyperlake.cloud/blog/physical-ai-datasets-and-world-models-for-robotics-training-2026
By Hyperlake Team · Hyperlake

Robot policies learn from episodes: time-aligned camera frames, robot state and actions, usually collected by human teleoperation, in simulation, or both. Physical AI datasets like these teach the behaviour. A world model is a different asset. It learns to predict what happens next in a scene, so it can generate synthetic experience, score candidate actions or stand in as a simulator. Most teams end up needing both, using a world model where real data is too scarce, slow or risky to collect.

We make HyperLake, a sovereign AI operating platform. It doesn't supply datasets or world models, and the section on where it fits says what it does cover.

![Real, simulated and synthetic episodes feed a world model and robot policy, tested on real held-out data.](https://hyperlake.cloud/blog/img/production/081ae4466440b76edcca1cb999addfa9b3855c1a-1200x840.png?w=1600&fit=max&auto=format)

## What do physical AI datasets contain?

A robot learning dataset is a pile of episodes. Each episode is one attempt at a task, recorded as a sequence of timesteps. A timestep usually holds:

- one or more camera images (sometimes depth as well)
- the robot's state, such as joint positions
- the action that was commanded
- a language instruction or task label
- an episode identifier and a success flag

Format matters more than you'd expect. Hugging Face's [LeRobot](https://github.com/huggingface/lerobot) project, released under Apache-2.0, stores vision data as synchronized MP4 videos (or images) and state and action data as Parquet files, with datasets hosted on the Hugging Face Hub. It also ships tools to delete episodes, split data, add or remove features and merge datasets.

The next block is a hypothetical, generic illustration of one episode's metadata. It isn't the schema of any particular library.

```json
{
  "episode_id": "ep_000123",
  "robot": "arm_a",
  "task": "place the cup on the shelf",
  "fps": 30,
  "num_steps": 412,
  "success": true,
  "cameras": ["wrist", "front"],
  "operator": "teleop_07"
}
```

Fields like robot, operator and success are what let you filter, audit and rebalance the dataset later. Strip that metadata out and the data gets hard to reuse.

## Which open robotics datasets are worth knowing?

Three public datasets show the range. The figures come from each project's own page, read in October 2026.

| Dataset | What it is | Scale as published |
| --- | --- | --- |
| [Open X-Embodiment](https://robotics-transformer-x.github.io/) | Pooled robot datasets from many labs | 60 datasets, 22 robot embodiments, 527 skills, over 1M real robot trajectories |
| [DROID](https://droid-dataset.github.io/) | Manipulation demonstrations collected by teleoperation on one hardware setup | 76k trajectories, about 350 hours, 564 scenes, 86 tasks, 13 institutions |
| [NVIDIA GR00T X-Embodiment Sim](https://huggingface.co/datasets/nvidia/PhysicalAI-Robotics-GR00T-X-Embodiment-Sim) | Simulated trajectories for post-training NVIDIA's GR00T N1 model | Five groups of datasets; 1.91 TB total; cc-by-4.0 |

Open X-Embodiment is about breadth, with many robots and labs pooled into one resource. The page gives two different counts for contributors (34 labs in one place, 21 institutions in another), so quote it carefully. DROID is about consistency. Every institution used the same hardware, which makes it easier to reason about than a pooled collection, though it only covers one robot arm design.

The GR00T simulation dataset page lists groups of 9, 24, 24, 24 and 1 datasets. Adding up the per-dataset trajectory counts it states (9 x 1,000, 24 x 10,000, 24 x 1,000, 24 x 3,000, and 102) gives 345,102 trajectories. That's our own sum from the page's figures.

Check every licence before you use the data. Terms differ between datasets, and a research licence doesn't automatically permit commercial training.

## Real, simulated or synthetic: where does the data come from?

| Source | Strength | Weakness |
| --- | --- | --- |
| Real teleoperation | Matches the true robot and scene | Slow, costly, hard to scale |
| Simulation | Fast, cheap, safe, labelled by construction | Sim-to-real gap |
| Generated video | Cheap variety of scenes and lighting | Physics may be wrong; needs checking |

Simulation frameworks such as [Isaac Lab](https://github.com/isaac-sim/IsaacLab), an open-source GPU-accelerated framework that supports reinforcement learning and imitation learning, let you generate a lot of labelled experience. The price is the gap between simulated and real physics. Our article on [why one robot's walking took seven months to transfer](https://hyperlake.cloud/blog/7-months-to-make-a-robot-walk-simulation-vs-reality-skyentific) is a concrete look at that gap.

Here's the mix we'd start from. Use real data for the behaviours that have to be right, simulation for coverage and safety, and generated data to widen visual variety. Tag every source, so you can measure what each one contributes.

## What is a world model for robotics?

A world model learns how an environment behaves. Give it the current state, and possibly an action, and it predicts a future state. Robotics uses that prediction in three ways:

1. As a data generator, producing new video or trajectories, say in different lighting or scenes.
1. As a planner, imagining the result of candidate actions and picking the best.
1. As an evaluator, testing a policy inside the model instead of on a real robot.

"World model" is a loose term. Some systems predict pixels, some predict in a compact latent space, and some also output text or actions.

## Which world models exist today?

Everything below comes from each owner's page, as of October 2026.

- NVIDIA Cosmos. The [Cosmos repository](https://github.com/NVIDIA/Cosmos) describes omnimodal world models for world simulation, future prediction, synthetic data and policy learning. Cosmos 3 has a Reasoner (text and vision in, text out) and a Generator (text, vision, audio and action in; vision, audio and action out), in Super (64B), Nano (16B) and Edge (4B) base sizes. The page states source code and models are released under OpenMDW-1.1, and lists separate tools for post-training, data curation and evaluation.
- Meta V-JEPA 2. According to [Meta's announcement](https://ai.meta.com/blog/v-jepa-2-world-model-benchmarks/), it's a 1.2-billion-parameter model pre-trained on more than a million hours of video and about a million images. Meta says it trained the model on the open-source DROID dataset and then deployed it directly on robots in its labs, and separately that 62 hours of robot data was enough for a model usable for planning and control. Meta reports 65% to 80% success on pick-and-place of new objects in unseen environments when using visual subgoals.
- Google DeepMind Genie 3. [DeepMind's post](https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/) describes a model that generates navigable worlds from text at 24 frames per second and 720p, consistent for a few minutes. It lists limits including a restricted set of agent actions and interaction lasting minutes, not hours, and calls availability a limited research preview.

They're built for different jobs. Cosmos is described as a family of models for simulation, prediction and synthetic data. V-JEPA 2 plans from goal images. Genie 3 generates interactive environments. Choose by the job you need done, not by the headline.

## How do datasets and world models work together?

World models need data as well. Meta says V-JEPA 2 was trained on the open-source DROID dataset for robot planning. So the usual pipeline looks like this:

1. Collect and curate real episodes with metadata.
1. Generate simulation and synthetic episodes for coverage.
1. Post-train or fine-tune a policy or world model on the mix.
1. Evaluate on held-out real episodes, not just generated ones.
1. Feed the failures back in as new collection targets.

Step 4 is where teams fool themselves. A world model can look completely convincing and still get contact, friction or object behaviour wrong. Keep a real held-out set that no generated data was derived from. We'd guard that set more carefully than anything else in the pipeline.

## How should you manage robot training data?

Treat every episode as a governed data asset:

- Record provenance: robot, operator, site, software version and licence.
- Keep raw and processed versions, with lineage between them. See [data lineage](https://hyperlake.cloud/blog/what-is-data-lineage) and [AI data governance](https://hyperlake.cloud/blog/ai-data-governance).
- Split train, validation and test at the episode or site level, not the frame level, since neighbouring frames are nearly identical.
- Plan for mixed data, with video, tables and text side by side, as in [structured vs unstructured data for AI](https://hyperlake.cloud/blog/structured-vs-unstructured-data-for-ai).
- Decide where the data may live. Factory footage can show people, products and processes.

Storage layout and pipelines are in our [physical AI reference architecture](https://hyperlake.cloud/physical-ai/sovereign-physical-ai-stack-on-kubernetes), and the [physical AI overview](https://hyperlake.cloud/physical-ai) explains the field. This article sticks to data and models.

## Where does HyperLake fit?

HyperLake isn't a dataset or a world model, and it won't train foundation models for you. It's a platform for assembling, deploying and governing the data, models and applications behind AI in a customer's own environment or a client's. For physical AI, its product description lists simulation, orchestration, training, data and model services in the customer's cloud, integrating tools such as NVIDIA OSMO, Omniverse, Isaac Sim and Isaac Lab, subject to licensing and validated integration. If all you want is to download a public dataset and run one experiment, you don't need a platform.

## Frequently asked questions

### What is the largest open robotics dataset?

We can't name one with confidence. Open X-Embodiment reports over a million real robot trajectories pooled from many datasets, and DROID reports 76k trajectories from one setup. "Largest" depends on whether you count episodes, hours or bytes.

### What is a world foundation model?

It's a large model trained to predict how the physical world evolves, usually from video, which developers then adapt for jobs like synthetic data generation or planning. People use the term in different ways, and "world model" often gets used more loosely still.

### Can synthetic data replace real robot data?

Not on current evidence from the sources above. Meta's V-JEPA 2 robot planning still used real robot data (DROID), and generated worlds such as Genie 3 list limits on actions and duration. Synthetic data widens coverage, and real held-out data checks it.

### What format should robot datasets use?

Whatever your training stack reads without conversion. LeRobot uses Parquet for state and action with MP4 for video, which keeps tabular and video data apart and easy to inspect.

### Are open robotics datasets free for commercial use?

Not automatically. The GR00T simulation dataset page lists cc-by-4.0 and LeRobot is Apache-2.0, but every dataset has its own terms, so read the licence for each source before training.

### How much robot data does a world model need?

That varies by model. Meta says 62 hours of robot data was enough for a V-JEPA 2 model usable for planning and control, on top of large-scale video pre-training. Don't expect that figure to carry over to other systems.

## Related reading

- [Physical AI Infrastructure You Control](https://hyperlake.cloud/physical-ai)
- [Sovereign Physical AI Stack on Kubernetes: Reference Architecture](https://hyperlake.cloud/physical-ai/sovereign-physical-ai-stack-on-kubernetes)
- [Open-Source Physical AI Stack Map: 49 Projects by Layer](https://hyperlake.cloud/physical-ai/open-source-stack-map)
