# Apache Iceberg: ACID Tables, Snapshots, and Time Travel > Apache Iceberg is an open table format that adds ACID transactions, snapshots, time travel, schema evolution, and engine interoperability to data lakes. Source: https://hyperlake.cloud/blog/what-is-apache-iceberg Published 2026-10-14 · by Hyperlake Team · Hyperlake Video: [Watch: What is Apache Iceberg (1:58)](https://www.youtube.com/watch?v=OjcWL6uI_9I) Apache Iceberg is an open table format for large analytical datasets in object storage. It sits between storage and compatible compute engines, adding atomic commits, immutable snapshots, schema evolution, time travel, and hidden partitioning. Iceberg improves data lake reliability without replacing the underlying files or storage system. These capabilities matter when a data lake must support concurrent workloads, changing schemas, record-level changes, and reproducible analysis without fragile file management. The video above walks through the core ideas. ## What is Apache Iceberg and where does it fit? Apache Iceberg is a metadata layer that makes collections of data files behave like reliable analytical tables. It is not a database, query engine, file format, or object store. The table data can remain in formats such as Parquet within object storage. Iceberg maintains metadata describing schemas, partition specifications, data files, statistics, manifests, and snapshots. Compatible engines consult that metadata instead of inferring the table from directory names and file paths. This design separates storage from compute. Teams can choose compatible processing engines while retaining a common table definition, an important characteristic of [open table formats](https://hyperlake.cloud/blog/open-table-formats-explained). ## How do snapshots provide ACID transactions and time travel? Iceberg commits table changes atomically, so readers see a complete snapshot rather than a partially written collection of files. Each immutable snapshot identifies the exact files that represent the table at that point. A write operation first creates or removes data and delete files, then prepares new metadata. The commit atomically updates the table’s current metadata pointer. Readers already using the previous snapshot can finish consistently, while later readers use the newly committed state. This model provides several practical capabilities: - Concurrent reads remain isolated from incomplete writes. - Conflicting writes can be detected instead of silently corrupting table state. - Updates can replace affected files rather than requiring an entire partition rewrite, depending on the engine and operation. - Queries can target a retained snapshot or timestamp for reproducible analysis. Time travel is not a substitute for backup. Snapshots form a queryable history, but expiration policies can remove old metadata and files, and failures affecting the same storage environment may affect both current and historical states. Separate backups still protect against broader loss. ![Diagram: An Iceberg write creates files and metadata, commits a snapshot, then exposes one consistent table view.](https://hyperlake.cloud/blog/img/production/5c2933baec7c9d3f459f628a311a3987c15dd005-1200x750.png?w=1600&fit=max&auto=format) *Atomic snapshot commits keep incomplete table changes hidden from readers.* ## How does Iceberg evolve schemas without rewriting all data? Iceberg tracks columns with stable field identifiers, allowing compatible engines to distinguish a renamed column from an unrelated new column. This makes many schema changes metadata operations rather than full data rewrites. Teams can add, rename, reorder, or remove columns while older files retain their original physical schema. When those files are read, Iceberg maps their fields to the table’s current logical schema. Removing a column from the schema does not necessarily erase its bytes from existing files; physical cleanup requires later file rewriting or retention procedures. Compatibility still matters. Before making a change, teams should confirm that every engine reading or writing the table supports the relevant Iceberg specification and schema operation. ## What are hidden partitioning and partition evolution? Hidden partitioning lets query writers filter logical columns without knowing the table’s physical directory layout. Iceberg applies declared partition transforms and uses metadata to identify which files can be skipped. For example, a table may partition timestamps by day without requiring users to include a separate partition column in every query. This avoids coupling analytical logic to path conventions and reduces errors caused by missing or incorrectly expressed partition filters. Partition evolution allows the strategy to change as data volume and access patterns change. New files can use a new partition specification while existing files keep the old one, and Iceberg evaluates both through table metadata. That separation between logical queries and physical organization is a major improvement over directory-driven tables. ![Diagram: Directory-driven tables expose physical layouts, while Iceberg keeps queries separate from partition organization.](https://hyperlake.cloud/blog/img/production/df30637fceb1575e593c2241716db59d10310969-1200x750.png?w=1600&fit=max&auto=format) *Hidden partitioning separates logical filters from physical file organization.* ## Which engines can use the same Iceberg table? Multiple compatible engines can access the same Iceberg table because the format defines shared table metadata and commit behavior. Spark, Trino, Flink, and other supporting engines can read the same underlying files without creating separate copies for each engine. Safe multi-engine operation still requires compatible Iceberg versions, a correctly configured catalog, and supported write behavior. The format provides the common contract, while engines execute queries, transformations, streaming jobs, and maintenance procedures. This interoperability makes Iceberg a common foundation for a [data lakehouse architecture](https://hyperlake.cloud/blog/data-lakehouse-architecture-explained-the-four-layers-that-make-it-work) that separates durable storage from independently scalable compute. ## Key takeaways - Apache Iceberg adds reliable table semantics to analytical files in object storage. - Atomic snapshots give concurrent readers a consistent view of committed data. - Retained snapshots support time travel and auditability but do not replace backups. - Stable field identifiers enable many schema changes without rewriting existing files. - Hidden partitioning and open engine support reduce coupling between queries, storage layouts, and compute tools. ## How Hyperlake helps Hyperlake can assemble governed data foundations using Iceberg and object storage with a fitting query engine, including Trino where appropriate for the workload. Teams can deploy these services in their own or a client’s environment alongside identity, access policy, observability, audit, and lifecycle controls, with operational procedures varying by engine and solution pack. To discuss an Iceberg-based data and AI environment, [talk to our team](https://hyperlake.cloud/contact). ## Frequently asked questions ### Does Apache Iceberg replace Parquet or object storage? No. Apache Iceberg organizes data files that commonly remain in formats such as Parquet and continue to reside in object storage or another supported file system. Iceberg adds table-level metadata, snapshots, schemas, partition specifications, and commit semantics above those files, while compatible compute engines perform the actual reads and writes. ### How long does Apache Iceberg time travel remain available? Time travel remains available only while the required snapshots, metadata, and referenced files are retained. Administrators often expire old snapshots and remove unreferenced files to control storage growth. Retention policies should therefore reflect audit, reproducibility, recovery, and cost requirements rather than assuming every historical state will remain indefinitely. ### Can multiple engines write to one Iceberg table safely? Compatible engines can share an Iceberg table when they use a properly configured catalog and supported commit behavior. Iceberg’s optimistic concurrency model detects conflicting commits rather than exposing partially written state. Operational safety still depends on engine compatibility, transaction isolation requirements, retry handling, and disciplined maintenance procedures.