Problem
Public cloud blob storage has been a standard for data lakes for the last 10 years. Blob storage, at first, came to solve data warehouse storage limitations. It is very cheap and has unlimited capacity. You can store any data format (structured, semi-structured, or unstructured) in the data lake located on a blob storage, and keep any amount of raw data for an unlimited time. When considering Apache Iceberg vs Delta Lake, both can manage data efficiently. Depending on the access frequency, data can be stored on cold or warm types of cloud storage, saving even more costs.
Once data is placed on a blob storage, we need to make it work and produce actionable insights. Data files can be written to or read from a data lake, and the append or read operation is fast and causes no conflicts. However, when you want to make changes to the existing data, like deleting or updating rows, the problem begins. Data read and data update workflows interfere with and lock each other, which complicates data analysis.
Data pruning is another problem with data lakes. If you need to process data from a specific month, you do not want to scan data for 10 years. It’s beneficial to have metadata that can help speed up data processing.
Solution
Alongside the data lake’s popularity, multiple open table frameworks have emerged in the data processing landscape. Those data format agnostic tools add a logical layer on top of the physical files and represent physical files as a table. This layer allows users to interact with files concurrently to power on efficient data load, data querying, and analysis. Another advantage that open table format tools provide is a “schema on read” – since files can contain various structures and data “fields,” the user defines the data that they need during the query and only rows (or documents) that have that specific field are returned.
Apache Iceberg and Delta Lake are very popular open table formats that are widely used in data Lakehouse architectures: solutions that combine data lakes and data warehouse principles.
In this tip, we will review their similarities and differences over the most interesting open table framework features.
Delta Lake
Delta Lake is an open-source storage layer, developed and open-sourced by Databricks, that brings ACID transactions and scalable metadata into a world of big data processing. It is Spark Native, built on top of parquet files, and maintains a transaction log to support ACID transactions. This open table framework natively integrates with any cloud Blob storage, AWS S3, Azure Blob Storage, or Google Cloud Storage. If you want to use Apache Delta Lake with a query engine other than Spark (for instance, Snowflake), you will be able to read the table data without the ability to modify it.
Apache Iceberg
Apache Iceberg is another open table format designed for large data lakes. It was built and open-sourced by Netflix and is now used by many data Lakehouse solutions. Apache Iceberg is engine agnostic, can be used for environments, not tightened to Apache Spark, and can work with many query engines, as Apache Spark, Apache Flink, Presto, Hive, Dremio, DuckDB, and Snowflake.
Features Supported by a Apache Iceberg vs Delta Lake
| Feature | Delta Lake | Apache Iceberg |
|---|---|---|
| Ecosystem | Primarily designed for Apache Spark | Supports multiple engines: Spark, Hive, Presto, Flink, Trino, Snowflake, and DuckDB |
| Changes Management/Schema Evolution | Enables user to make changes to the data structure (add, rename, and remove table columns) without writing the migration DDL script; logs table changes into JSON-formatted delta logs. | More flexible schema evolution. Schema updates are metadata operations. Allows users to add, rename, and remove table columns without the need for table rewrite. |
| Change Data Feed: tracking row-level data changes (merges, updates, deletes, inserts) This feature is useful for downstream updates, similar to the CDC. | Supports Merges, Deletes, and Updates. Insert-only operations are not logged! Highly scalable to reduce concern about feeds growing too much and impact write operations speed. Stores using parquet files that contain a list of data changes. | Provides a change log procedure that gives a view containing all changes made to a table for the specified period of time: inserts, updates, and deletes. |
| ACID Transactions Capabilities | Uses transaction logs for atomic insert, update, and delete operations to run reliably without conflicts or corruption. | Uses metadata layers to provide atomicity, consistency, isolation, and durability for insert, update, and delete operations. |
| Multi-table Transactions | Not supported. | Primarily supports transactions at the table level. Active development and community interest in extending its capabilities to support multi-table transactions, enabling all-or-nothing operations across multiple tables. |
| Strategy for Updates | Copy-on-write: when you change the data, you don’t change an original file. Affected rows are copied and new file with only affected rows is created. Transaction Log is updated to point to a new file. | Copy-on-write or Merge-on-read (for Flink, mode for streaming updates without rewriting full file) |
| Partitioning and Partition Pruning | Stores data in directories based on partition columns. Queries use metadata for partition pruning using Parquet checkpoint. | Uses hidden partition scheme (which reduces query overload), abstracting it from the user. Dynamically tracks partitions in the metadata without any directory structures. Can handle multiple partition schemes for the same dataset. Queries are aware of those partition schemes. |
| Partition Evolution | Evolves partitions (change partition keys) by having users manually specify new partitioning policy when writing new data. Old data continues to follow old partitioning strategy unless rewritten. | Supported. Allows seamless partition evolution without the need to rewrite the data. Can change partition columns at any time, and handles the changes in the metadata. |
| Data Deduplication (insert data without duplicates) | Supported during MERGE operation. | Supported during MERGE operation. |
| Time Travel and Rollback | Supports time travel capabilities, allowing users to query the state of data at specific points in time. | Changelog procedure allows time travel capability and rollback to a point in time. |
| File Format Supported | Parquet. | Parquet, Avro, or ORC. |
| Data Quality | Supports by using two types of Delta constraints: NOT NULL and CHECK. When the constraint is violated, Delta throws an exception and refuses to save the data. | Table constraints are not supported. |
| Vacuum/Cleaning | Manual for data files and managed for transaction logs. | Manual operation. |
| File Unified Sizing: configuration for a unified file size | Requires manual maintenance. | Requires manual maintenance. |
| Bootstrap: converting data in place or migrating the data | Supports converting in place | Needs to migrate the data. |
| Indexing | Has indexes on a table level stored as key/value metadata in a JSON file. If a column has a high cardinality, users can specify Z-ordering (multi-dimensional clustering) for optimization. Z-ordering is a technique of organizing related information in the same set of files, which can drastically reduce the amount of data that the query needs to scan. | Traditional secondary indexes are not supported. Relies on metadata-based indexes at file and column level: file stats, like row count, min and max values for each column, partition statistics. |
| Isolation Levels | Provides serializable writes (ensures that the reader never sees inconsistent data) and snapshot isolation for reads (readers see consistent snapshot views of the table even when a table is in the middle of a set of changes). | Supports snapshot-based isolation: each write creates a new snapshot of a table. |
Performance
Apache Iceberg lags behind Delta Lake in data loading and querying benchmarks.
- Delta is 1.7 times faster in data loads and querying benchmarks. In TPC-DS benchmark (link below), it took 3 TB of data 1.68 hours to load the dataset into Delta and perform a set of queries, and 5.99 hours to load data into Apache Iceberg.
- Table load difference grows when the table gets wider.
- This benchmark was performed two years ago, and I could not find anything that has more fresh information. Let us know if you have tested the performance of both frameworks in your work.
Security
If you are using Delta Lake on Databricks, you would have out-of-the-box fine-grained access controls, row/column level security, and role-based access controls.
Iceberg can match the Delta Lake strong security controls but needs additional catalogs, like Hive or Glue, and additional security integrations.
Scalability
Both Iceberg and Delta Lake elastically scale utilizing decoupled storage and compute. Iceberg Metadata is more scalable for very large datasets as Delta’s JSON logs with a very large table can grow and degrade performance.
Summary
Both Apache Delta Lake and Apache Iceberg are modern and versatile open-table frameworks that bring ACID transactions, data pruning, and rich metadata to your data lakes.
The choice depends on your tech stack and feature requirements. If you are heavily invested in Spark and Databricks then your choice would definitely be Apache Delta Lake. For multi-engine support and work with massive tables made of millions of files, you should consider Apache Iceberg.
Next Steps
- Performance benchmark Delta vs. Iceberg: Delta vs Iceberg : Performance as a decisive criteria
- Delta Lake included with Synapse Analytics: What is Delta Lake?
- Z-ordering vs. partitioning: Partitioning vs Z-Ordering – Explained
- Data skipping in Delta Lake: Data skipping for Delta Lake
- Delta Lake best practices: Best practices: Delta Lake
- Getting Started with Delta Lake Using Azure Data Factory
