hyperlakeDiscuss a deployment ↗
Blog · 5 min read

Data Lakehouse Architecture: The Four Layers

Data lakehouse architecture uses four layers (object storage, table formats, metadata catalogs, and compute) to deliver open, governed analytics.

Video thumbnail: Data Lakehouse Architecture Explained   The Four Layers That Make It Work
Watch: Data Lakehouse Architecture Explained   The Four Layers That Make It Work (2:12) · Video page

In a lakehouse, keeping data and working on it are separate jobs. Four layers divide the responsibilities: object storage, an open table format, a metadata catalog and compute engines. The files stay open rather than tied to one engine. Above them, the other layers bring database-style table management, transactions, schema tracking, historical versions and governance to analytics and machine learning.

Saving Parquet files in a data lake doesn't give you all of that. The choices between layers matter: a poor fit can leave you maintaining more infrastructure, copying datasets or relying on an engine you can't easily replace.

What are the four layers of data lakehouse architecture?

Start at the bottom with the files, then work upward to the processing a workload needs. Each layer has a specific job. It doesn't have to know the details of every application or engine above it.

Object storage keeps the data files; Apache Parquet is a common choice for columnar analytical data. A table format manages those files as tables, adding transactional metadata, schemas, versions and historical snapshots. The catalog gives those tables names and records their schemas, locations and access information. Compute engines do the processing: SQL queries, large transformations, machine learning and other work.

A plain data lake could contain identical Parquet files. What's missing is the shared understanding of tables, their transactions and where to find them. That difference is easier to see in our comparison of a data lake, data warehouse, and data lakehouse.

Diagram: Four lakehouse layers from object storage through table formats and catalogs to compute engines

How do metadata catalogs and compute engines work together?

The engine needs a table; the catalog tells it where to look. Users can refer to the table by name instead of having to keep track of physical file paths. Finding the data and processing it remain separate jobs.

In that lookup, a catalog typically supplies the table's schema and storage location. It also holds information used for access governance. Rather than starting with a particular object-storage prefix, the engine starts with a table name and resolves it through the catalog to the relevant metadata and files.

This matters when a table moves or its underlying files change. Compaction, new partitions and advancing snapshots shouldn't leave engines with conflicting views of the same dataset. Without a shared catalog, each engine may maintain its own view, tied too closely to the storage layout.

There's no need for one engine to do every job. Interactive SQL can run on a SQL query engine, large-scale transformations on Spark, and machine learning frameworks can read prepared data. Compatible engines can work at the same time and scale separately. You can also replace an engine without copying the entire dataset, because storage doesn't belong to the compute layer.

Compatibility is the condition here, not an optional extra. Check that every engine supports the table format and the catalog's behavior. Identity, authorization and auditing also need consistent enforcement wherever the integrated systems allow access.

Diagram: A catalog connects stable lakehouse storage and tables to independent compute engines

What does an open table format add to object storage?

A file isn't a table on its own. The table format supplies the rules that let engines read a collection of files as a managed table. We'd call it one of the most consequential choices in data lakehouse architecture, because the format decides how writes, schema changes, snapshots and concurrent operations get coordinated.

Apache Iceberg, Delta Lake and Apache Hudi do this by keeping metadata about table membership and versions: which files belong, and which version is current. The available features depend on both the format and what the engine supports:

  • ACID transactions, for reliable reads and writes.
  • Schema evolution, without treating every change as a new dataset.
  • Version tracking across updates, and time travel queries against retained historical snapshots.
  • Safer coordination between concurrent readers and writers.

The files still live in object storage. A table format manages them; it doesn't take over their storage. Before choosing one, check the engines you need, your governance requirements, the operational tools and how portable the result will be. See our Apache Iceberg and Delta Lake comparison for differences between those two options.

Why is object storage the foundation of a lakehouse?

Object storage is where the data persists, regardless of which engine is currently reading or transforming it. That separation lets different compatible engines use the same files rather than own separate copies.

Parquet is common here for a practical reason: its columnar layout suits analytical queries that need only a few columns. Keeping data in open file formats also avoids dependence on a proprietary storage layer. Large collections of structured or semi-structured data can remain available for different workloads.

Storage alone can't tell readers which table version they should see. Nor does a directory of Parquet files provide transactions or reliable schema evolution. Table versions, a consistent current view and those other management tasks have to come from the layers above the files.

How Hyperlake helps

Hyperlake can assemble a governed data foundation from Iceberg and object storage with a query engine that fits, plus operational, vector, graph, search and streaming services when the workload calls for them. It runs in infrastructure you control and provides shared controls for identity, access, observability, lifecycle operations, lineage and audit, with procedures that vary by engine and solution pack. To work through a lakehouse architecture for your environment, talk to our team.

Frequently asked questions

Can multiple query engines use the same lakehouse data?

Yes, provided they support the same table format and shared catalog. Interactive SQL and batch transformations can use the same tables through different engines. Check more than the format name, though: each engine must correctly handle the catalog implementation, transaction semantics and current metadata version as well.

What happens if object storage has no table format?

Then you have files in a data lake, not a shared mechanism for managing database-style tables. Readers may need to work out schemas and locate paths themselves. Replacing files, changing schemas and coordinating simultaneous writes become harder without the metadata and transaction model that a table format provides.

Is a data lakehouse the same as a data warehouse?

No. A lakehouse generally puts open files in object storage and keeps table management, the catalog and compute in separate services. A traditional warehouse more often manages storage and compute together in one system. Either can provide governed analytics. The differences concern storage, freedom to choose engines, operational responsibilities and which workloads the design suits.

Why is the metadata catalog important for governance?

Everyone needs a common inventory of table names, schemas and locations. The catalog provides that inventory and can connect table identities to authorization controls and audit processes. But recording access information isn't the same as enforcing it. Query engines, storage or service access points must still apply the controls; catalog metadata alone won't prevent access.

Start with a workload. Build the environment around it.

Explore example deployments, or see how the platform assembles, deploys, governs and operates the stack.