What Is Apache Iceberg? The Open Table Format Explained

Apache Iceberg is an open table format for large analytical datasets stored in cloud object storage. It adds the capabilities that raw Parquet files lack — ACID transactions, schema evolution, time travel, and efficient partition management — while keeping data in open, cloud-native storage that any compatible engine can read.

Iceberg was created at Netflix to solve specific problems that arose from managing petabyte-scale Hive tables: slow metadata operations, unsafe concurrent writes, and broken queries after schema changes. It became an Apache project in 2018 and has since become one of the dominant open table formats alongside Delta Lake and Apache Hudi.

The Problem Iceberg Solves

Raw Parquet files on S3 are cheap and fast to read, but they have no transactional semantics. Writing to a Parquet table is a sequence of file operations — write new files, update a metadata file (like the Hive Metastore) to point to the new files. This process is not atomic. If a write fails halfway through, the table is in a partially written, inconsistent state. Concurrent writers can overwrite each other. Readers scanning during a write may see partial results.

Hive tables (the predecessor Iceberg replaced at Netflix and elsewhere) had additional problems: listing all files in a partition required enumerating objects in an S3 prefix — slow for large tables, and prone to race conditions during writes. Schema changes (adding a column) could silently break queries that expected the old schema.

Iceberg addresses these problems at the table format layer — the specification for how files are organized and tracked on storage — rather than requiring a specific database engine.

Core Iceberg Capabilities

**ACID transactions** — Iceberg uses optimistic concurrency control. Each write produces a new snapshot of the table. Writers check that their assumed base state is still current before committing. If two writers conflict, one retries. Readers always see a consistent snapshot; partial writes are never visible.

**Schema evolution** — columns can be added, removed, renamed, or reordered without rewriting existing data files. Iceberg tracks column IDs separate from column names — if a column is renamed, old data files still map correctly to the new name. This is a significant improvement over Hive, where renaming a column could break existing Parquet files.

**Time travel and rollback** — every write to an Iceberg table creates an immutable snapshot. You can query any prior snapshot by timestamp or snapshot ID. You can roll back a table to a previous state by replacing the current snapshot pointer. This is invaluable for debugging, auditing, and recovering from accidental data corruption.

**Partition evolution** — Iceberg allows changing the partitioning strategy of a table without rewriting existing data. Old data keeps its original partitions; new data uses the new partition scheme. Query planning handles both correctly. Hive required rewriting the entire table to change partitioning.

**Hidden partitioning** — Iceberg automatically derives partition values from column values. A table partitioned by month does not require users to manually compute the month value; Iceberg handles partition column management transparently, eliminating the class of user errors where wrong partition values cause data to be written to incorrect partitions.

**Metadata optimization** — instead of listing S3 objects to find relevant files, Iceberg maintains a metadata layer (manifest files, manifest lists) that records which files contain which partition ranges. Query planners use this metadata to skip irrelevant files without object listing — significantly faster for large tables with many files.

Iceberg vs Delta Lake vs Hudi

These three open table formats are functionally similar. Understanding the differences:

**Apache Iceberg** — engine-agnostic by design. Supported by Spark, Flink, Trino, Presto, Dremio, StarRocks, Snowflake (Iceberg tables on Snowflake), BigQuery (BigLake), and many others. No vendor controls the spec; it is governed by the Apache Software Foundation. If you want maximum flexibility to swap query engines, Iceberg is the most broadly supported format.

**Delta Lake** — created by Databricks, open-sourced via the Linux Foundation. The dominant format within the Databricks ecosystem. Excellent support in Spark; growing support in other engines. Unity Catalog governance integrates deeply with Delta Lake. If you are on Databricks, Delta Lake is the natural default.

**Apache Hudi** — optimized for streaming upserts and near-real-time ingestion. Offers more granular write modes (copy-on-write vs merge-on-read) that optimize for either read performance or write performance. Strong in environments with high-frequency upserts from CDC or streaming pipelines. Originated at Uber.

The format wars have largely concluded in a pragmatic draw. Snowflake, BigQuery, and most query engines now support reading (and in some cases writing) all three formats. For greenfield lakehouses, Iceberg's engine-agnosticism makes it the safer default; for Databricks-centric environments, Delta Lake is simpler.

Why Open Table Formats Matter

The strategic significance of open table formats extends beyond technical capabilities.

Before Iceberg, Delta Lake, and Hudi, analytical data in a data warehouse was locked to that warehouse's proprietary storage. Moving from Redshift to Snowflake meant exporting all data and reimporting it. Adding a new query engine to analyze existing data was not possible without duplication.

Open table formats separate storage from compute. Data stored as Iceberg tables on S3 can be queried by Spark, Trino, BigQuery (via BigLake), Snowflake (via Iceberg tables), Athena, Flink, and DuckDB — simultaneously, from the same files, with no duplication. You can switch query engines without migrating data. You can add a specialized engine (DuckDB for interactive analysis, Flink for streaming) without copying data to a new system.

This is the strategic foundation of the lakehouse architecture: open storage formats plus compute engines that read them directly, eliminating the lock-in that made switching data platforms expensive.

Iceberg in Practice

Iceberg is most commonly encountered in:

- Databricks environments where Unity Catalog supports Iceberg tables (alongside Delta Lake)

- Trino/Athena environments on AWS where S3-based analytics needs ACID semantics

- Snowflake environments where Snowflake can read and write Iceberg tables on customer-managed storage

- Apache Flink streaming pipelines writing to Iceberg tables for real-time analytics with ACID guarantees

- Organizations standardizing on the lakehouse pattern who want to avoid vendor lock-in on storage format

For organizations building a new data lakehouse, the choice between Iceberg and Delta Lake is often less about technical capabilities and more about the query engine ecosystem: Databricks-heavy environments default to Delta; AWS/multi-cloud/engine-diverse environments often prefer Iceberg's broader compatibility.

Our data architecture practice designs lakehouse architectures using open table formats for Snowflake, Databricks, and AWS environments — contact us to discuss your lakehouse architecture requirements.

A former Microsoft data architect audits your data foundation, identifies your top priorities, and sends you a written plan. Free. No pitch.