Back to blog

How does Apache Iceberg on S3 work?

Apache Iceberg on S3 brings ACID transactions, time travel, and schema evolution to data lakes. How Erathos supports S3 + Iceberg as a destination.

Apache Iceberg architecture diagram on Amazon S3 with ACID transaction support

What is Amazon S3 and Apache Iceberg?

Amazon S3 (Simple Storage Service) is a highly durable and scalable storage service by AWS, used by companies to store massive volumes of data with high availability. While widely adopted as the foundation for data lakes, on its own it does not provide the structure needed for efficient analytical table management.

Apache Iceberg is an open-source table format designed to overcome the limitations of traditional formats like Hive and Parquet. It brings ACID transactions, data versioning, and optimizations that significantly improve analytical query performance.

Why S3 Apache Iceberg?

Apache Iceberg revolutionized the analytical table format by solving historical pain points of formats like Hive and Parquet. It offers:

  • ACID Transactions: Update, delete, and upsert (merge) data with no headaches.
  • Schema Evolution: Adapt your data as business requirements change, without breaking downstream pipelines.
  • Time Travel: Query historical versions of your data for audits or rollback comparisons.
  • Scalability: Combine the massive scale of Amazon S3 with the optimized structure of Iceberg.

What changes for Erathos users?

Now, you can route your pipelines directly to Iceberg tables on Amazon S3, getting the best of both worlds:

  • Native integration with engines like Databricks, Trino, Athena, and Spark.
  • Automated partition and metadata management.
  • High-performance analytical queries, even on petabyte-scale datasets.

Practical use cases:

  1. Governed Lakes: Build highly scalable data lakes without sacrificing consistency and versioning.
  2. Near-Real-Time Analytics: Ingest data frequently without degrading query performance.
  3. Auditing and Compliance: Leverage time travel to track changes and guarantee data compliance.

Get started now!

Update your pipelines and test the new S3 Apache Iceberg destination. With just a few clicks, you can run a robust and optimized lakehouse.

➡️ Want to see it in action? Book a meeting with the Erathos team to learn how to simplify your data stack, or try it yourself by following our docs.