What Is Apache Iceberg? A Practical Guide for Small Data Teams
Learn what Apache Iceberg is, how its metadata turns Parquet files into tables, what upkeep it needs and when a small team should use it.

Data teams today often load data into BigQuery or write Parquet files to S3, and it works. But then you start hearing the terms "lakehouse" and "Iceberg" thrown around in every conference talk, vendor page and job post. And you wonder if you’re falling behind.
Apache Iceberg is an open table format. It consists of small metadata files that sit alongside your Parquet files, turning a folder of files into a table. Spark, Trino, Snowflake, BigQuery, DuckDB and other engines can safely read and write to Iceberg tables. It’s free, it’s a standard, and it’s not a database you have to install. If you have a small team, whether you need Iceberg depends on a few factors. Most often, the answer is "not yet".
Why a folder of Parquet files is not a table
This is where Parquet shines. It stores data by column, which is ideal for analytical queries where you only need a few columns. The OLAP vs OLTP guide has a great explanation of why column storage is faster for analytical queries. Parquet is also inexpensive to store in S3. However, each Parquet file is still just a file. When you drop a bunch of them into a folder and name it orders, that "table" only exists in your head. Nothing in your bucket knows about it.
That creates some issues:
- Readers can read partial data. A query engine will read whatever files it finds when it lists the directory. If a writer is still writing, or if it failed mid-way, the query will read a mix of old and new files.
- Writers can conflict. There's no lock to prevent multiple writers from writing to the same directory at the same time, or from deleting files that another writer is reading.
- Schema changes can be risky. If you rename a column, the name won't change in your old files. It's up to each query engine to decide how to map the old name to the new name.
- Deletes require rewriting files. Parquet files can't be updated in-place. If you need to delete a customer's data, you have to find all the files that contain their rows, rewrite them, and then replace the old files without disrupting running queries.
- Partitions are fixed. If you have a directory structure like
order_date=2026-10-01/, switching to hourly partitioning would require creating a new table and re-writing all your data. And to take advantage of that partitioning, queries need to explicitly filter on the partition column, otherwise they'll read all the data.
Without a table layer, a bucket of files is just a bucket of files. Iceberg is that layer.
What is Apache Iceberg?
Apache Iceberg is an open table format, which is a specification for managing a large, slow-changing collection of files as a table. Your data files are still stored as Parquet files in your own bucket, but Iceberg adds a few layers of metadata files on top, which list the data files that belong to the table at the current moment.
There are four layers in Iceberg, read from top to bottom:
- Catalog. A small service that stores the location of the current metadata file for each table. For example, AWS Glue, an Iceberg REST catalog, or the catalog built into Snowflake or Databricks.
- Metadata file. A JSON file that contains the schema of the table, the partitioning scheme, configuration settings, and a list of snapshots. A snapshot is a state of the table at a single point in time.
- Manifest list and manifests. Each snapshot points to a manifest list. The manifest list points to manifests. A manifest file has one row per data file, which contains the partition values of the data file, as well as stats about the data file such as row counts, minimum and maximum values. These stats are used by engines to skip files that are not relevant to a query.
- Data files. Plain Parquet, Avro or ORC files, the same files as you have today.
The key concept is that Iceberg tracks individual files instead of folders. So a writer can add new Parquet files to the bucket, but they are invisible until the writer commits.
This is the path an engine follows to read a table:
When performing a commit, a writer creates new data files, manifests, and metadata files. It then requests that the catalog atomically switch the ‘current’ pointer from the old metadata file to the newly created metadata file. This can fail if another writer has already performed a commit, in which case the writer will retry on top of the newer version. Readers never lock. Readers access whichever metadata file was current at the time of their query, thereby seeing a complete snapshot of the data. This is what the commit would look like in terms of flow:
We created a small orders table with Spark and made nine changes to it: create, insert, two column changes, a partition change, another insert, an update, a delete and a merge. Here is what the table folder held afterwards (trimmed):
data/order_ts_day=2026-10-01/00000-1-898fedd7-...-00001.parquetdata/order_ts_hour=2026-10-02-14/00000-7-881a285b-...-00001.parquet...metadata/9ddfbacd-26b4-4ed5-9091-8572bf1d7f8e-m0.avrometadata/snap-3903044455079110341-1-9ddfbacd-....avro...metadata/v1.metadata.json...metadata/v9.metadata.jsonmetadata/version-hint.text
We can also see that there are five rows in our table. Finally, we can see that we have 30 files in our folder. 8 of these are Parquet files. There are 9 metadata files (one for each change), 5 manifest lists (snap-*.avro files, one for each change to the data) and 8 manifests.
Iceberg never updates files, only creates them. It maintains old versions of files so we can use them for time travel. Eventually we need to clean them up. This is why we have the section on upkeep below.
In our local run, we used the simplest catalog, which updates the pointer using a version-hint.text file. This works fine on a laptop, but according to the spec, the file-based commit scheme is unsafe on object stores. On S3 we need to use a real catalog, like AWS Glue or a REST catalog.
What Iceberg adds, with a short example of each
All the examples below were executed on Spark with Iceberg and on the same orders table. Any engine with Iceberg write support could do the same, albeit with its own SQL syntax.
Hidden partitioning. You tell Iceberg how to split the data once, when you create the table. Here it groups rows by the day of order_ts:
CREATE TABLE demo.shop.orders ( order_id bigint, customer string, amount decimal(10,2), order_ts timestamp)USING icebergPARTITIONED BY (days(order_ts));
Queries then filter on the real column, and Iceberg skips the days that do not match. Nobody has to know about a separate order_date partition column:
SELECT * FROM demo.shop.ordersWHERE order_ts >= '2026-10-01' AND order_ts < '2026-10-02';
Schema evolution. Adding, dropping, renaming, widening or reordering columns are metadata changes, so no data files are rewritten. Iceberg gives every column an ID, so a renamed column still matches its old data:
ALTER TABLE demo.shop.orders ADD COLUMN country string;ALTER TABLE demo.shop.orders RENAME COLUMN customer TO customer_name;
Old rows show NULL for the new country column, and the renamed column keeps its values:
+--------+-------------+------+-------------------+-------+|order_id|customer_name|amount|order_ts |country|+--------+-------------+------+-------------------+-------+|1 |Ada |25.00 |2026-10-01 09:00:00|NULL ||2 |Ben |40.00 |2026-10-01 10:00:00|NULL |+--------+-------------+------+-------------------+-------+
Partition evolution. When daily partitions get too coarse, you change them in place:
ALTER TABLE demo.shop.ordersREPLACE PARTITION FIELD days(order_ts) WITH hours(order_ts);
Old files keep the daily layout and new files use the hourly one. The files metadata table shows both layouts (spec 0 and spec 1) living in the same table after one more insert:
+-------+------------------+------------+|spec_id|partition |record_count|+-------+------------------+------------+|0 |{2026-10-01, NULL}|2 ||0 |{2026-10-02, NULL}|2 ||1 |{NULL, 497486} |1 |+-------+------------------+------------+
Row-level updates, deletes and merges. Iceberg supports UPDATE, DELETE and MERGE INTO. When a delete matches only some rows, Iceberg rewrites only the affected data files. Version 2 of the format also added delete files, which mark rows as deleted without rewriting the data file at all. A merge is the usual way to apply a batch of changes:
MERGE INTO demo.shop.orders AS ordersUSING ( SELECT * FROM VALUES (2, 'Ben', 45.00, TIMESTAMP '2026-10-01 10:00:00', 'US'), (6, 'Finn', 120.00, TIMESTAMP '2026-10-02 15:00:00', 'FR') AS source(order_id, customer_name, amount, order_ts, country)) AS incomingON orders.order_id = incoming.order_idWHEN MATCHED THEN UPDATE SET amount = incoming.amount, country = incoming.countryWHEN NOT MATCHED THEN INSERT *;
ACID transactions: ACID stands for atomic, consistent, isolated, and durable. In other words, a change either happens entirely or doesn’t happen at all, and readers won’t see things halfway. In the merge above, Ben’s row was updated and Finn’s row was inserted, all in one commit. Any query starting during the merge will see the table as it was before the merge, and any query starting after will see both changes. If two jobs commit at the same time and start from the same version, only one commit succeeds and the other will have to retry.
Time travel and rollback. Every data change creates a snapshot, and you can list them:
SELECT snapshot_id, committed_at, operationFROM demo.shop.orders.snapshots;
+-------------------+-----------------------+---------+|snapshot_id |committed_at |operation|+-------------------+-----------------------+---------+|8712082876802806949|2026-10-23 01:48:50.471|append ||4468552588377488096|2026-10-23 01:48:52.206|append ||40938677677449855 |2026-10-23 01:48:53.061|overwrite||3903044455079110341|2026-10-23 01:48:53.712|overwrite||5403034263911535346|2026-10-23 01:48:54.631|overwrite|+-------------------+-----------------------+---------+
You can query any snapshot by ID or by time with VERSION AS OF or TIMESTAMP AS OF:
SELECT * FROM demo.shop.orders VERSION AS OF 8712082876802806949;
Note that the query returned the first four orders with their original values. We also have the old column name, customer and no country column. This is because time travel provides the schema as it was at the time of the snapshot. This can cause problems for dashboard queries that are written for the current columns in the table, but can fail when looking at an older snapshot of the table.
If a bad job ran, you can move the table back to an earlier snapshot. This call undid the delete and the merge in one step:
CALL demo.system.rollback_to_snapshot('shop.orders', 40938677677449855);
The upkeep that comes with Iceberg
Of course, there is a cost to time travel. Every write, update, delete, merge, and compaction creates a new snapshot and retains old data and metadata files. Also, many small loads can result in many small files. Small files create more metadata and slow down queries, as the engine has to open each small file.
So an Iceberg table needs three cleanup jobs:
Job | What it does | Default to know |
|---|---|---|
Compaction ( | Merges small data files into bigger ones | Targets 512 MB files and only acts on groups of at least 5 files |
Snapshot expiry ( | Deletes old snapshots and the files only they used | Expires snapshots older than 5 days, keeps at least 1 |
Orphan file cleanup ( | Deletes files no metadata points to, such as leftovers from failed jobs | Only touches files older than 3 days, has a dry run mode |
These are the calls we ran on the demo table:
CALL demo.system.rewrite_data_files(table => 'shop.orders');CALL demo.system.expire_snapshots( table => 'shop.orders', older_than => TIMESTAMP '2026-10-23 01:48:55', retain_last => 1);CALL demo.system.remove_orphan_files(table => 'shop.orders', dry_run => true);
The expiry of snapshots resulted in deletion of 4 data files, 5 manifests and 4 manifest lists. The compaction of files did not result in rewriting of files because of the small number of rows (five) present. After the expiry of the snapshots, the ability to time travel back to these snapshots is lost.
In a small team, someone would have to be responsible for running these jobs. This can be done in one of two ways.
- You can run these jobs on your own. You can schedule the execution of procedures in Spark (or any other engine supporting the same) to run on a regular basis. This is a very small amount of code to be added but is another job to be monitored.
- You can use a managed table service. Amazon S3 Tables allows you to run compaction as part of the service. You pay $0.002 per 1,000 objects and $0.005 per GB of data processed. So, to compact 30,000 files of 5MB each, you would pay about $0.06 for the objects and $0.73 for the data. BigQuery’s Iceberg managed tables provide automatic file sizing, clustering and garbage collection as well.
Which engines support Apache Iceberg?
Spark, Trino, Flink, Snowflake, BigQuery, Databricks, Amazon Athena and DuckDB can read and write Iceberg tables. ClickHouse can read Iceberg tables, with writes available in beta. The majority of engines are restricted to writing to a single catalog and may restrict features such as partition evolution.
This is what each engine's official docs confirmed in October 2026:
Engine | Read | Write | Conditions to know |
|---|---|---|---|
Yes | Yes | Connector maintained by the Iceberg project. The reference engine for maintenance procedures. | |
Yes | Yes | Connector maintained by the Iceberg project. Supports streaming writes with | |
Yes | Yes | The Iceberg connector reads and writes data and metadata. | |
Yes | Yes | Tables must be in the AWS Glue catalog. Creates and writes Iceberg v2 tables only. | |
Yes | Yes | Works with Snowflake as the catalog or an external catalog. External writes need Iceberg v2 or v3. | |
Yes | Yes | Iceberg managed tables keep data in your Cloud Storage bucket. No partition evolution, one | |
Yes | Yes | Writes managed Iceberg tables in Unity Catalog. Iceberg tables from other catalogs are read-only. | |
Yes | Yes | Writes need an attached REST catalog. Deletes use delete files only (merge-on-read). | |
Yes | Beta | Reads through the Iceberg table engine or table function. Inserts are a beta feature you turn on with a setting. |
For example, BigQuery explicitly warns that adding files to a BigQuery-managed Iceberg table outside of BigQuery can lead to data loss. While Iceberg is an “open format”, that doesn’t mean any engine can write to any table. Iceberg tables are owned by the catalog that has a pointer to the table; any engine that can write through that catalog is fine to write to the table.
In BigQuery’s docs, there are some additional restrictions on BigQuery Iceberg tables, such as no table renames, no clones, and no materialized views:
Iceberg vs Delta Lake vs Apache Hudi
Iceberg has two open table format rivals. All three add transactions, schema changes and time travel on top of Parquet files. They differ in where they came from and what they do best:
Format | Started at | Best at | Good fit |
|---|---|---|---|
Apache Iceberg | Being read and written by many engines | Teams that want to pick or switch engines freely | |
Delta Lake | Spark and Databricks pipelines | Teams that run everything on Databricks | |
Apache Hudi | Uber, now an Apache project | High-frequency record-level updates, such as CDC replication |
Those differences are starting to blur. Delta Lake’s UniForm feature enables a Delta table to be read as an Iceberg table. Also, Databricks writes its own managed Iceberg tables.
Iceberg is the easiest choice for small teams that don’t use Databricks. It works with more engines than the other formats. Delta Lake is the better choice for organizations that use only Databricks and don’t plan to use anything else. Hudi is the best choice for organizations that need to do a lot of record-level upserts.
Iceberg vs a cloud data warehouse
A data lake keeps raw files in cheap object storage and lets you choose the query engine later. A data warehouse stores structured tables and bundles storage, compute and management into one product. (Our data lake vs data warehouse guide covers the full comparison.) Iceberg gives a lake the table features people used to buy a warehouse for, so the real choice is between Iceberg tables in your bucket and native tables in BigQuery or Snowflake:
| Iceberg tables in your bucket | Cloud data warehouse |
|---|---|---|
Storage cost | $0.0265 per GB per month on S3 Tables for the first 50 TB | $23.55 per TiB per month for BigQuery active storage, $23 per TB per month for Snowflake on demand |
Compute | Any engine from the support table, and you can use more than one | The warehouse's own engine, billed by the warehouse |
Vendor lock-in | Low. The files and metadata follow an open spec in storage you own | Higher. Moving out means exporting your data |
Operational effort | You pick a catalog, set up access and run or pay for compaction and cleanup | The vendor handles files, compaction and access |
The cost of storage per gigabyte is around the same, so storage cost isn’t usually a deciding factor. Instead, the biggest cost difference is in compute. With Iceberg, you can run a big batch job using Spark, then run smaller queries using DuckDB or Athena on the same tables. In a data warehouse, you’re always running your queries on the warehouse’s meter. For example, BigQuery on-demand pricing is $6.25 per TiB scanned after the first free TiB each month.
These days, the two worlds are converging. Snowflake, for instance, can read and write Iceberg tables. When these tables are on your own external volume, Snowflake only charges you for the compute used to query them. Similarly, BigQuery Iceberg managed tables don’t have BigQuery storage cost, since the files are stored in your Cloud Storage bucket instead. This means that you can still use a data warehouse as your main compute engine, while storing your data as Iceberg.
Should a small team use Iceberg?
So when should a small team use Iceberg? If you need to access the same data across multiple engines, or if you already have files in S3 that you need to be able to update, delete or change the schema of, Iceberg is a great fit. If a single warehouse serves all your queries, stick to the warehouse; Iceberg adds the catalog and cleanup jobs that only make sense if you need its benefits.
Here’s a flowchart representation of the decision:
Iceberg is a good fit when you have one or more of these needs:
- Two or more engines use the same tables. You have data scientists training models in Spark or Python and analysts querying the same data in Athena, Trino or Snowflake. Without Iceberg, you’re trying to keep two copies of data in sync.
- Your files in S3 need real table behavior. You need to delete rows for a customer privacy request, correct a bad load or add a column to a table without breaking old files.
- You want to own your storage. You anticipate switching query engines, or you don’t want your history tied to a single vendor.
- Someone can own the upkeep, or you can pay for it. This means having someone on the team to run the catalog and cleanup jobs, or using a managed service like S3 Tables or BigQuery’s Iceberg managed tables.
A managed warehouse like BigQuery is more appealing when:
- One engine does all the work. All users are looking at BI dashboards or running SQL models. The data is structured tables from your apps and SaaS tools.
- Nobody has time for the plumbing. Nobody on the team wants to set up a catalog, cloud permissions and maintenance jobs.
- The bill is fine. The cost of storage is comparable in both setups, so you won’t move to Iceberg to save money on storage.
Or, you can use BigQuery or Snowflake as your main query engine but store tables as Iceberg in your own bucket. This lets you use the full warehouse experience but adds the option of adding a new engine later. It does have the same limitations as those in the engine table above.
How data gets into Iceberg tables
Data in Iceberg tables is only useful when data is in the table. There are three common ways to get data into an Iceberg table:
- Batch loads. This is a job that runs on a schedule and reads data from a source and appends new rows or merges in changes. The
INSERTandMERGE INTOexamples above would be a batch load. This type of job can be run with Spark, Trino, Athena, Snowflake and DuckDB. - CDC (change data capture) from databases. This is a tool that reads the change log of a database like PostgreSQL or MySQL and applies each insert, update and delete to the Iceberg table, typically as a
MERGE. Iceberg row-level changes makes this possible. See our guide to cursor-based sync vs CDC for more on the different ways of reading change data. - Streaming. Flink can stream into an Iceberg table with
INSERT INTO. Flink makes frequent small commits, so many small files and snapshots are created. Therefore streaming tables require compaction and snapshot expiry most.
To create any of these yourself, you have to write and run the load jobs and set up the catalog.
S3 with Apache Iceberg as a destination is now supported by Erathos, so Erathos pipelines can load data into Iceberg tables in your own S3 bucket. It uses the AWS Glue catalog, so the tables can be queried with Athena, Trino, Spark and Databricks. To use it, go to Settings -> Data Warehouse -> S3 Iceberg, and add your AWS access key, bucket name, bucket region and Glue catalog database. This is a beta connector.
FAQ
Is Apache Iceberg a database? No. Iceberg is a table format specification. It describes the metadata files which can, in turn, describe a table composed of data files. Iceberg does not have a server, nor can it execute queries. An engine like Spark, Athena, Snowflake, or DuckDB can read the metadata and execute the query.
Is Apache Iceberg free? Yes. Iceberg is an open source Apache project. You need to pay for the storage to store the files, the engines to execute the queries, and any managed service you may be using. For example, S3 Tables is $0.0265 per GB per month for the first 50 TB.
What is the difference between Iceberg and Parquet? Parquet is a file format that stores data column-wise in a single file. Iceberg is a table format that tracks which Parquet files belong to a table, its schema, partitions, and history. Iceberg can also track Avro and ORC files.
Do I need Spark to use Iceberg? No. Athena, Trino, Snowflake, BigQuery, and DuckDB can read and write Iceberg tables. The cleanup procedures mentioned in the Iceberg docs can be executed on Spark, so it’s useful to have around unless you’re using a managed service that handles cleanup for you.
What is an Iceberg catalog? A catalog stores the pointer to the current metadata file for each table and swaps it when a write is committed. AWS Glue, Snowflake, Databricks Unity Catalog, and any service that implements the Iceberg REST catalog API can be catalogs. The engines you use can limit which catalogs you can use. For example, Athena needs Glue as its catalog. DuckDB needs a REST catalog to write.
What is Amazon S3 Tables? S3 Tables is an AWS storage service designed for Iceberg. It organizes tables in table buckets and can perform compaction. Compaction is priced at $0.002 per 1,000 objects and $0.005 per GB processed.
Does Iceberg replace a data warehouse? It replaces the storage portion, but you still need an engine to run queries, which can be a data warehouse. Both Snowflake and BigQuery support querying Iceberg tables, so you can certainly use Iceberg with a data warehouse.
If you decide Iceberg is right for you, Erathos can load data into Iceberg tables on S3. Try Erathos for free for 14 days.