Stop looking for the one right file format. To manage robotics training data you need a three-tier pipeline: append-only ROS 2 MCAP (rosbag2) containers for edge recording, with message loss and interrupted-recording recovery checked on the actual device, a governed object-and-table lakehouse (Apache Iceberg and Apache Parquet) for episode curation and dataset versioning, and training formats matched to your policy dataloader. In 2026, use Hugging Face LeRobotDataset v3.0 for standardized Vision-Language-Action (VLA) and imitation learning with multi-episode Parquet and MP4 shards, Zarr for cloud-native chunked array streaming in Diffusion Policy and UMI pipelines, or HDF5 (h5py) for low-latency random slicing on local NVMe.
*Disclosure: We build Hyperlake, a sovereign AI operating platform. This guide is our engineering assessment, and it links directly to each open-source specification and each vendor's official documentation.*

Why Do Robotics Training Pipelines Need Multiple Data Formats?
Because a robot trajectory isn't a text file or a photo. It's a bundle of sensor streams running on different clocks that still have to line up within milliseconds Trossen Robotics. A typical bimanual manipulation rig or mobile robot logs:
- High-frequency proprioception and control: joint positions, velocities, motor currents and 6-axis force/torque wrench readings at 100 Hz to 500 Hz.
- Multi-view visual streams: two to four RGB or RGB-D camera feeds at 30 to 90 FPS.
- Spatial and navigation telemetry: LiDAR point clouds, IMU packets and odometry transforms (
/tf). - Task and episode context: natural-language task instructions, embodiment calibration matrices, operator IDs and success/failure labels Encord.
No single storage format is good at both ends of that lifecycle Trossen Robotics. Each stage is bottlenecked by something different:
- Edge recording is write-bound and fault-sensitive. Onboard compute has to append asynchronous pub/sub messages to disk in sequence, without dropping frames, and without corrupting the archive if the battery disconnects halfway through an episode.
- Dataset curation is query-bound. Data engineers need columnar predicate pushdown to filter millions of trajectories by robot embodiment, lighting, task success or collision recovery, without decoding terabytes of video to do it.
- GPU training is random-read-bound. Imitation learning and VLA dataloaders sample random $(t)$ indices across thousands of episodes and fetch temporal windows (observations at $[t - 0.2\text{s}, t - 0.1\text{s}, t]$ alongside an action horizon of $[t, t + 1, \dots, t + 16]$, for example), which demands fast random seek and chunk decompression.
Proposed Reference Architecture: From Robot Edge Recording to GPU Training
Production robotics teams keep edge ingestion, lakehouse curation and training consumption in separate layers. That stops context getting lost at each format handoff and avoids file-system bottlenecks Trossen Robotics, Encord.
*Note: The diagram is a Proposed Reference Architecture that combines open-source robotics formats with governed lakehouse and Kubernetes infrastructure.*

For how these components deploy on Kubernetes more broadly, see our Sovereign Physical AI Stack on Kubernetes: Reference Architecture and the Open-Source Physical AI Stack Map.
Robotics Data Formats Compared
The following five formats and frameworks address different parts of a robotics data pipeline. Select them against the recording, curation and training tools you will use.
| Format / Framework | Primary Pipeline Stage | Storage Layout | Video & Image Handling | Cloud Object Streaming | Best For | Key Limitation |
|---|---|---|---|---|---|---|
Hugging Face LeRobot (LeRobotDataset v3.0) | Training & Hub Distribution | Directory of multi-episode Parquet shards (data/), MP4 shards (videos/), and relational metadata (meta/) LeRobot v3 Docs | Hardware/codec compressed MP4 shards indexed by frame/timestamp offsets; optional image storage | Native via StreamingLeRobotDataset over Hugging Face Hub or hf://buckets/ LeRobot v3 Docs | Standardized PyTorch imitation learning and VLA models (ACT, SmolVLA, π₀, GR00T) combining video, tabular state, and episode metadata LeRobot Docs | Requires calling dataset.finalize() to close Parquet writers; random video frame decoding adds CPU overhead vs. raw arrays LeRobot v3 Docs |
HDF5 (h5py) | Local GPU Training | Single hierarchical binary container file organizing POSIX-like groups and typed multidimensional datasets HDF5 Docs | Stored as raw or chunk-compressed (GZIP/LZF) 4D tensor datasets ([T, H, W, C]) inside the container | Poor over HTTP/S3 without intermediate caching or specialized range-request wrappers | High-speed local NVMe training and classic manipulation benchmarks (ALOHA, robomimic) requiring low-latency slice reads | Monolithic files bottleneck concurrent cloud object reads; one-file-per-episode layouts exhaust filesystem inodes at scale |
| Zarr | Cloud & Distributed Training | Chunked, compressed N-dimensional arrays where each chunk maps to a distinct object key or file Zarr Docs | Stored as compressed array chunks (Blosc, Zstd, LZ4) addressable by coordinate index | Native; workers fetch individual chunk keys concurrently over S3, GCS, or local disk Zarr Docs | Distributed cloud training and trajectory policies (Diffusion Policy, UMI) streaming multidimensional tensors directly from object storage | No built-in robotics metadata standard; naive small chunk sizes in Zarr v2 can create millions of tiny S3 objects unless sharded in Zarr v3 |
| Apache Parquet | Curation, Indexing & Tabular State | Columnar binary files (PAR1) split into row groups and column chunks with trailing footer metadata Apache Parquet Docs | Not suited for high-bitrate video (stores raw byte blobs poorly compared to MP4 or Zarr) | High; readers read the file footer first and fetch only required byte ranges and columns Apache Parquet Docs | High-frequency scalar time-series (joint states, actions, timestamps) and SQL-queryable episode catalogs in Apache Iceberg | Inefficient for large multidimensional visual tensors unless paired with external video shards (as in LeRobot v3.0) |
ROS 2 (rosbag2 / MCAP) | Real-Time Edge Recording & Playback | Append-only message container (.mcap default) with embedded schemas, channels, and indexed LZ4/Zstd chunks MCAP Docs, rosbag2 Docs | Logs serialized sensor_msgs/Image or CompressedImage (H.264/JPEG) messages alongside time-synced topics | Supported via HTTP range requests on indexed .mcap files for inspection in Foxglove | Deterministic pub/sub recording on physical robot hardware, fault-tolerant edge logging, and timeline debugging | Sequential time-ordered layout is not structured for random batch sampling across episodes in PyTorch DataLoader |
How should formats be selected?
Go with what your recording and training tools support, and keep a record of every conversion. LeRobot defines a training-oriented dataset structure, while MCAP is a recording format. HDF5 and Zarr serve array-oriented workflows (HDF5 is still a common robot dataset format for local benchmarks), and Parquet holds tabular information. All of these can live happily in one pipeline.
Test the library versions you've picked, their supported operations and their recovery behaviour on representative data. Whatever you choose, you still have to preserve timing, calibration and dataset lineage yourself.
How to Version and Curate Multi-Terabyte Robotics Datasets
For large robotics datasets, evaluate file sizes, transfer patterns, repository limits and storage costs before choosing a versioning system. Keep binary-object immutability and catalog metadata snapshots as separate design concerns. The fix is to treat binary blob immutability and catalog metadata snapshots as two separate problems. This is also, in practice, how you store and query ROS 2 and rosbag data at scale.
1. Immutable Object Storage + Open Table Snapshots
Write raw .mcap files and materialized training shards (LeRobotDataset v3.0 or Zarr) to object storage under immutable, content-addressed or run-partitioned paths. Then, instead of duplicating video shards every time you make a new training split, keep an Apache Iceberg episode catalog table (robotics.episodes_catalog) that you query through Trino.
Each row tracks episode-level metadata and pointers:
episode_uuid,robot_id,embodiment_version,operator_idtask_instruction,lighting_condition,outcome_status(SUCCESS,RECOVERY,COLLISION_ABORT)raw_mcap_uri(s3://robot-raw/2026-10/run_882.mcap)lerobot_shard_uri,shard_from_index,shard_to_index
When you curate a training dataset, query that table at a specific snapshot ID (see Apache Iceberg Snapshots Explained). If you also want Git-like branch, commit and merge over raw S3 prefixes, say to branch a LeRobotDataset directory before a data-cleaning job, LakeFS or DVC give you zero-copy object pointer versioning. MLflow or Weights & Biases then ties the exact Iceberg snapshot ID or LakeFS commit hash to the trained policy checkpoint.
2. Preserve Failed and Recovery Trajectories
Here's a mistake we see a lot in imitation learning: throwing away failed teleoperation runs at ingestion Encord. A policy trained only on clean expert demonstrations suffers covariate shift the moment the robot drifts slightly off course. Keep the aborted, near-miss and human-recovery trajectories in your object store. Tag them in the Iceberg catalog (outcome_status = 'RECOVERY' or 'FAILURE') and filter them at materialization time, so they're still there for offline reinforcement learning, reward modeling and safety evaluation Encord.
How should storage and bandwidth be sized?
From your own recordings. A universal per-robot estimate won't survive contact with your camera settings, compression, sensor rates, episode length and retention policy.
Size raw recordings, curated training data and temporary working files separately. Work out growth and transfer demand for each, add replication and recovery requirements, and check the estimate against a real capture before you buy capacity.
Where Hyperlake Fits (and When to Use Standalone Open-Source Tools)
Not every team needs an enterprise platform for robotics data. Honestly, many don't.
- When standalone open-source tools are the better choice: a single academic lab arm, or a small research team publishing public benchmarks, will move fastest with the standalone
lerobotPython library on local NVMe, pushing public datasets straight to the Hugging Face Hub LeRobot Docs. - Where Hyperlake fits: once a physical AI team moves from public demos to proprietary robot fleets, defense or industrial facilities, or customer sites, raw camera feeds and facility maps can't go to public hubs. Hyperlake is a sovereign AI operating platform that runs on Kubernetes in your own cloud account, with no compute markup. Today that means AWS and Google Cloud; private cloud, on-premises and other clouds are coming soon.
Inside one governed foundation, Hyperlake lets teams assemble and run:
- Lakehouse curation and streaming: object storage with Apache Iceberg and Trino for SQL episode curation, ClickHouse for fast telemetry analytics, Kafka for streaming ingestion, and Qdrant or Milvus for embedding-based visual trajectory retrieval.
- Identity-to-data governance: people, training jobs and agents reach trajectory datasets through an OAuth/OIDC proxy, validated JWTs carrying scoped workload identity, and OPA (Open Policy Agent) policy checks logged at every access point (see AI Data Governance: Lineage, Access, and Versioning).
- Physical AI simulation and training orchestration: reusable deployment patterns for simulation, training, data and model serving (
KServe), integrating tools such as NVIDIA OSMO, Omniverse, Isaac Sim and Isaac Lab (subject to licensing and validated integration) alongside real-world robot feeds, as shown in our live Physical AI Reliability & Safety Operations demonstration.
Frequently Asked Questions
How should a LeRobot dataset version be chosen?
Pick the version your training and recording tools support. Check the official LeRobot documentation, and store the version number with the dataset so nobody has to guess later.
Should I use HDF5, Zarr, or LeRobotDataset for robot imitation learning?
It depends on where training runs. Use LeRobotDataset v3.0 for PyTorch VLA or imitation learning policies (such as ACT, SmolVLA, π₀ or GR00T) when you want compact MP4 video sharding, built-in temporal windowing (delta_timestamps) and cloud streaming LeRobot Docs. Choose Zarr if your pipeline (Diffusion Policy or UMI, for example) streams pre-decoded multidimensional array chunks concurrently from S3 or GCS Zarr Docs. And reach for HDF5 (h5py) when the whole dataset fits on local NVMe and you want low-latency array slicing without video decoding overhead Trossen Robotics.
What should be recorded with a training dataset?
Format and library versions, source recording identifiers, transformations, splits and the relevant calibration. With those, another engineer can reproduce the exact training input.
Why should robots record in MCAP (rosbag2) instead of writing training formats directly?
Robots lose power without warning, and their sensors run at different rates (200 Hz joint states next to 30 FPS camera frames, say). MCAP is an append-only container with indexed, chunk-level LZ4/Zstandard compression built for structured recording MCAP Docs. Writing HDF5 or multi-episode MP4/Parquet shards directly on the edge risks corrupting the whole file on power loss, and it forces real-time timestamp interpolation onto constrained onboard compute Trossen Robotics.
How do I version multi-terabyte robotics datasets without duplicating video files?
Keep raw .mcap logs and sharded training files (LeRobotDataset v3.0 or Zarr) as immutable objects in S3-compatible storage, and manage versions through an Apache Iceberg episode catalog table or an object-level layer like LakeFS or DVC. A new training split then becomes a new Iceberg snapshot or LakeFS commit pointing at approved byte/frame ranges in existing shards. No terabytes of MP4 or Zarr files get copied.
*All trademarks belong to their respective owners. Spotted an outdated format specification? Contact the Hyperlake team and we'll verify it and update the documentation.*



