Zero-copy data sharing gives external analytics engines and authorized partners direct, governed access to data in its existing storage location. Instead of creating and synchronizing separate copies, the architecture uses open table formats, centralized metadata, fine-grained permissions, and temporary credentials so multiple systems can query a shared source of truth.
This matters when large datasets span cloud providers, analytics platforms, and organizational boundaries, where copying can increase storage costs, egress charges, synchronization lag, and governance complexity. The video above walks through the core ideas.
What is zero-copy data sharing?
Zero-copy data sharing is an access pattern that lets multiple query engines use the same underlying data files without creating a persistent copy for each consumer. The data remains in controlled object storage while a metadata catalog tells authorized systems how to interpret and locate it.
Traditional sharing often involves exporting files, replicating tables, or building pipelines into another platform. Every additional copy creates another dataset to secure, update, monitor, retain, and eventually delete. It can also leave consumers working with stale data between synchronization jobs.
A zero-copy design separates storage from compute. The producer manages shared tables in object storage, while external engines bring their own compute and read the files through standardized table metadata. Technologies covered in this overview of open table formats support this separation by representing schemas, partitions, snapshots, and file references independently from a single query engine.
“Zero-copy” does not mean that no bytes cross a network. Query engines still read data from storage, which can create network transfer, scan, and egress costs when systems sit in different regions or clouds. It means the architecture avoids creating another persistent dataset solely for sharing.
How does zero-copy data sharing work?
A centralized catalog governs table metadata and makes approved tables discoverable to compatible external engines. The engine receives permission and secure credentials, resolves the current table metadata, and reads the referenced columnar files directly from object storage.
The typical sequence is:
- Publish the table. The data owner registers an open-format table and its metadata in a governed catalog.
- Authorize the consumer. The catalog or an integrated access service evaluates read permissions for a user, workload, engine, or partner.
- Issue scoped access. The system provides temporary credentials or another limited access mechanism for the approved storage paths.
- Query in place. The external engine resolves the metadata and reads only the files needed for the query.
Because consumers resolve shared metadata rather than ingesting independent copies, committed data and schema changes can become available without running a separate replication pipeline. Visibility is not always literally instantaneous: catalog refresh behavior, engine caching, transaction support, and the table protocol determine when a consumer observes a new snapshot or schema.
Compatibility also matters. Each participating engine must support the relevant table format, catalog protocol, storage authentication method, and schema features.
How does zero-copy sharing improve governance?
Zero-copy sharing reduces the number of independent data copies that governance teams must control. Owners can manage access at a shared catalog and storage layer instead of relying on separate permissions and security processes for every replicated dataset.
Fine-grained read policies can limit which identities, tables, or approved data paths an external engine may access. Temporary, scoped credentials reduce dependence on long-lived shared secrets, while centralized metadata provides a consistent location for permission and schema management.
A governed implementation should combine catalog events, credential issuance, object access, and query activity into an audit trail. Direct storage access must remain within integrated enforcement and logging paths; otherwise, catalog logs alone may not show every file read.
This model can also support data sovereignty requirements because the authoritative files remain in infrastructure selected by the data owner. However, sovereignty still depends on where queries run, whether data crosses jurisdictions, where temporary results are stored, and which operators can access the environment.
What should teams check before adopting zero-copy sharing?
Teams should validate engine compatibility, network economics, policy enforcement, and metadata behavior before treating zero-copy sharing as a universal replacement for replication. Some workloads still need local copies for latency, isolation, transformation, resilience, or regulatory reasons.
Key design checks include:
- Format compatibility: Confirm that every engine correctly supports the required table operations, schema evolution, and snapshot semantics.
- Credential scope: Restrict temporary credentials to approved identities, operations, storage locations, and time windows.
- Network boundaries: Model cross-region and cross-cloud reads because zero-copy access can still incur egress charges and latency.
- Audit coverage: Verify that catalog, query-engine, identity, and object-storage events can be correlated.
- Cache behavior: Define when consumers must refresh metadata and how they handle changed or expired snapshots.
- Write ownership: Establish whether consumers are read-only or whether the architecture safely supports multiple writers.
Read-only sharing is generally simpler because one owner remains responsible for commits and schema changes. Multi-engine writes require stronger coordination around transactions, concurrency, compatible protocol versions, and operational recovery.
Key takeaways
- Zero-copy data sharing avoids persistent duplicate datasets by letting compatible engines query governed files in place.
- Open table formats and centralized catalogs separate shared storage from the compute engines that analyze it.
- Scoped credentials, fine-grained permissions, and integrated audit logs help preserve centralized governance.
- Metadata updates can reach consumers without replication pipelines, but visibility depends on protocol support and caching.
- Network transfer, egress, latency, jurisdiction, and engine compatibility still require careful evaluation.
How Hyperlake helps
Hyperlake can assemble data services, identity, access policy, scoped secrets, audit, and lifecycle controls in infrastructure controlled by the organization or its client. Its modular data foundation can run Iceberg and object storage with a fitting query engine, including Trino where appropriate, while governed services mediate access for people and agents. To discuss a zero-copy architecture for a specific workload and environment, talk to our team.
Frequently asked questions
Does zero-copy data sharing eliminate cloud egress costs?
No. Zero-copy sharing eliminates the need to create a persistent duplicate merely to provide access, but query engines still read bytes from object storage. Cross-region or cross-cloud reads may therefore incur network egress charges and additional latency. Storage and compute placement should be designed around expected query volume and data locality.
Can external partners query shared data without receiving database passwords?
Yes, when the architecture uses federated identity and temporary, scoped credentials instead of shared database accounts. The partner’s user or workload identity can be evaluated against access policy, then granted limited read access to approved tables or storage paths. The exact mechanism depends on the catalog, query engine, identity provider, and object store.
Do schema changes appear immediately in every external query engine?
Not necessarily. A committed schema change becomes part of the table’s authoritative metadata, avoiding a separate copy pipeline, but external engines may cache catalogs or table metadata. How quickly they observe the change depends on refresh intervals, protocol compatibility, transaction semantics, and whether the engine supports that type of schema evolution.
When is data replication better than zero-copy sharing?
Replication may be preferable when consumers need low-latency local access, independent availability, workload isolation, substantial transformations, or a jurisdiction-specific physical copy. It can also help when an external engine cannot read the chosen table format or catalog protocol. The tradeoff is additional storage, synchronization, security, lineage, and lifecycle management.
