What does an open-source data lakehouse stack look like?
The five layers, the open-source options for each, how they connect, and a self-hosted example.
An open-source data lakehouse stack has five layers: S3-compatible object storage holding Parquet files, an open table format such as Apache Iceberg, a catalog such as Apache Polaris, query engines such as Trino and Spark, and governance for access policy and lineage. Open interfaces between layers let you swap one without moving the data.
A lakehouse keeps data as open files in object storage and adds the parts that used to need a warehouse: tables, transactions, a catalog and fast SQL. An open-source stack fills each of those parts with a project you can run yourself.
What are the layers, and what are the options for each?
Five layers, each with more than one open-source choice.
| Layer | Job | Open-source options | What to check |
|---|---|---|---|
| 1. Object storage | Hold the data and metadata files | Ceph Object Gateway, Garage, SeaweedFS, or a cloud object store you control | S3 API coverage, replication, who maintains it |
| 2. File and table format | Store columns efficiently, then turn files into tables with transactions and history | Parquet files with Apache Iceberg; Delta Lake, Apache Hudi or Apache Paimon as alternatives | Which engines read and write the format |
| 3. Catalog | Track each table's current metadata so engines agree on what the table is | Apache Polaris, Lakekeeper, Project Nessie, Hive Metastore | Iceberg REST support, access control, credential handling |
| 4. Query and processing engines | Run SQL, batch jobs and streams on the tables | Trino for interactive SQL, Apache Spark for batch, Apache Flink for streams, DuckDB on a laptop | Connector support for your catalog |
| 5. Governance | Decide who can see which rows and columns, and record lineage | Open Policy Agent or Apache Ranger for policy; OpenMetadata or DataHub for discovery and lineage | Where policies are enforced, and whether every engine is covered |
The blog post on lakehouse architecture explains the storage, format, catalog and compute layers in more depth. Governance is listed separately here because in an open stack it is a separate project you have to choose and wire in.
How do the layers fit together?
Engines ask the catalog where a table's current metadata is, read the Parquet files it points to from object storage, and check policy before returning rows.
- Data lands as Parquet files in object storage, written by Spark, Flink or an ingestion tool.
- The table format commits the new files: Iceberg writes a new metadata file and snapshot, so readers see all of the change or none of it. Iceberg's own description lists full schema evolution, hidden partitioning and time travel among its features.
- The catalog records which metadata file is current. Iceberg's REST catalog specification gives "a common API" for this. In the spec's words, "New languages and engines can support any catalog with just one client implementation."
- An engine queries the table. Trino's Iceberg connector, for example, supports REST, JDBC, Nessie, Hive Metastore, AWS Glue and Snowflake catalogs, and reads Parquet, ORC or Avro data files from S3, Azure, Google Cloud Storage or HDFS.
- Policy is checked. Trino's Open Policy Agent plugin sends each access check to OPA and can filter rows and mask columns for specific users.
Because the interfaces between layers are open (S3, Parquet, the Iceberg spec, the REST catalog API), you can replace one layer without rewriting the data. That is the main difference from a warehouse that stores data in its own format.
What does a self-hosted example look like?
Here is one combination that runs entirely on infrastructure you control, for example on Kubernetes:
| Layer | Example choice | Why it fits |
|---|---|---|
| Storage | Ceph Object Gateway | Compatible with "a large subset" of the Amazon S3 API, on your own disks |
| Table format | Apache Iceberg with Parquet | Read and written by many engines at once |
| Catalog | Apache Polaris | Implements Iceberg's REST API; a top-level Apache project since February 2026 |
| Engines | Trino for SQL, Spark for batch | Both work with Polaris and Iceberg |
| Governance | Open Policy Agent through Trino's plugin | Row filters and column masks in one place |
Smaller teams can swap in Garage, an S3-compatible store built for self-hosting across sites, or Lakekeeper, an Apache-2.0 Iceberg REST catalog written in Rust. Project Nessie adds Git-like branches and commits across tables.
Which choices matter most?
The table format and the catalog, because they are the hardest to change later.
- Table format. Iceberg is built for many engines sharing the same tables; Delta Lake fits stacks centred on Databricks. The post on Iceberg vs Delta Lake compares them.
- Catalog. It holds the pointer to every table and is the natural place to enforce access. Prefer one that speaks the Iceberg REST API. The post on open metadata catalogs covers Polaris.
- Object storage maintenance. Check who maintains the store. The community minio/minio repository was archived on 25 April 2026 and says it is no longer maintained.
When is a lakehouse the wrong choice?
When the data is small, the team is small, or nobody will operate five projects. A single PostgreSQL database, or DuckDB over Parquet files, handles modest data volumes with far less to run. A lakehouse earns its keep when several engines, teams or AI applications need the same large tables under one set of rules.
Hyperlake uses this pattern in its data and intelligence solution: Iceberg and object storage with a fitting query engine such as Trino, plus a reviewed semantic layer, deployed in the customer's environment so agents see only approved data. Its use cases page describes it.
Frequently asked questions
Is Parquet a table format?
No. Parquet is a file format: it stores columns of one file efficiently. A table format such as Iceberg sits on top and records which Parquet files make up the table at each point in time, which is what makes transactions, schema changes and time travel possible.
Can Trino and Spark use the same Iceberg table?
Yes. The Iceberg project says engines including Spark, Trino, Flink, Presto, Hive and Impala can safely work with the same tables at the same time, as long as they go through the same catalog.
Is MinIO still an option for lakehouse storage?
Be careful. The minio/minio GitHub repository was archived on 25 April 2026 and says it is no longer maintained, and the community edition is distributed as source code only. Existing installations still run, but plan for security fixes and upgrades before choosing it for new work.
Where did the term lakehouse come from?
A 2021 CIDR paper by Armbrust, Ghodsi, Xin and Zaharia described the lakehouse as a new architecture based on open direct-access formats such as Apache Parquet, with first-class support for machine learning and warehouse-level performance.
Sources
Sources for the facts on this page, last checked October 9, 2026.
- Apache Iceberg: the open table format for analytic datasets checked October 9, 2026
- Iceberg REST catalog specification (Apache Iceberg) checked October 9, 2026
- Iceberg connector (Trino documentation) checked October 9, 2026
- Open Policy Agent access control (Trino documentation) checked October 9, 2026
- Apache Polaris checked October 9, 2026
- Apache Polaris graduates to a Top Level Project (19 February 2026) checked October 9, 2026
- Lakekeeper: Apache Iceberg REST catalog in Rust (GitHub) checked October 9, 2026
- Project Nessie checked October 9, 2026
- Ceph Object Gateway (Ceph documentation) checked October 9, 2026
- Garage: S3-compatible object store checked October 9, 2026
- minio/minio repository status (GitHub) checked October 9, 2026
- Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics (CIDR 2021) checked October 9, 2026
Related guides
- Data lakehouse architecture: the four layers
- Apache Iceberg vs Delta Lake: how to choose
- Apache Polaris and open metadata catalogs explained
- What is sovereign AI?Sovereign AI is AI whose data, models, compute and operations stay under the control and laws of the country or organization that relies on it.
Start with a workload. Build the environment around it.
Tell us what you need to deploy, whose environment it must run in, and what it needs to connect to.