Data Lakehouse Reality Check After Three Years of Iceberg

It has been roughly three years since Apache Iceberg started showing up in serious enterprise data architecture diagrams, and long enough now to take stock honestly. The lakehouse pitch was simple: separate storage from compute, treat object storage as the source of truth, and let multiple engines read the same tables. The pitch was right in the broad strokes. The implementation details have been considerably messier than the keynote slides suggested, and where teams actually landed after their first or second iteration looks different from the original architecture.

This is a reality check on what works, what frustrates teams in production, and where the architecture is heading next.

The pitch held up better than expected

The core lakehouse promise has held. Object storage as the durable layer is dramatically cheaper than what came before, and the cost difference compounds at scale. Iceberg’s table format – ACID transactions, schema evolution, time travel – works well enough that teams trust it for production. Pluggable engines mean Spark, Trino, DuckDB, and the warehouse-native engines can all read from the same table without weird gymnastics. For organizations that previously had three different data systems competing for ground truth, the consolidation has been meaningful.

The cost story in particular has been the surprise. Teams who migrated from Snowflake-only or BigQuery-only to a lakehouse with Iceberg tables on S3 often report bills that are 40 to 60 percent lower for the same workloads, with the caveat that the engineering cost to run it is higher. Whether that trade favors you depends on how much engineering you have and what your steady-state query patterns look like.

Where Iceberg still bites

The Iceberg metadata layer is the part that surprises new teams. Iceberg manages tables by writing JSON manifests and a snapshot history into the object store, and the cost of mismanaging that metadata shows up in unexpected ways.

Compaction is the first thing new teams trip on. Without scheduled small-file compaction, Iceberg tables accumulate thousands of tiny data files from streaming ingestion. Query latency starts to degrade quietly. By the time someone investigates, the table has half a million files and the planner is spending more time reading manifests than scanning data. Every production Iceberg deployment I have seen eventually built or adopted a compaction job. The tooling for this is maturing but is still not as automatic as the keynote suggests.

Snapshot expiration is the second. Iceberg keeps every historical snapshot until you tell it not to, and the storage cost grows accordingly. Teams that left snapshot retention at defaults for a year ended up paying more for old snapshots than for current data. Expiry policy is something every team has to choose deliberately.

The third surprise is partition evolution. Iceberg supports evolving partitions without rewriting data, which sounds great until you realize that queries against the older partition scheme do not benefit from the new partitioning. In practice, big partition changes still imply a rewrite if you want consistent query performance, so the value of in-place partition evolution is narrower than the documentation implies.

The catalog war

The most underrated part of the lakehouse story is the catalog. Iceberg defines a table format but does not solve where the metadata lives or who controls writes. Through 2024 and 2025 a real fight emerged between Snowflake’s Polaris, Databricks’ Unity, AWS’s Glue-backed Iceberg REST catalog, and the various open implementations. The fight is not over, but a few patterns are becoming clear.

For organizations all-in on a single platform vendor, the vendor’s catalog is the default and there is little reason to fight it. For organizations that genuinely want engine portability, an open catalog – Polaris in open mode, Nessie, or the REST catalog wrapped around their own backing store – is the realistic path. The compromise pattern, where teams use a vendor catalog but write a thin abstraction layer in front, has mostly produced extra engineering work without delivering the portability it promised.

Whichever catalog you pick, owning the write path is the decision that matters. If you have multiple engines writing to the same Iceberg tables through different catalogs, you will eventually corrupt metadata. The teams that operate stable lakehouses have a clear answer to who writes and a clear answer to who only reads.

Streaming ingestion is still where things break

Batch ingestion into Iceberg is solved. Streaming ingestion is the area where teams still hit production issues. The combinations that work reliably are narrower than the marketing suggests.

Flink writing into Iceberg has matured into a workable pipeline for many shops. Spark Structured Streaming into Iceberg works but tends to produce many small files that need aggressive compaction. Direct Kafka-to-Iceberg connectors are emerging but the operational story is still uneven.

The pattern that survives in production for most teams is something like Kafka into a streaming engine, the engine batches every minute or two, and a separate maintenance job compacts in the background. Trying to get sub-minute latency end to end into Iceberg is possible, but the operational footprint goes up sharply, and most use cases that demand sub-minute freshness end up routing through a real-time store like Pinot or Clickhouse in addition to the lakehouse, with the lakehouse holding the durable record.

The cost models that surprise CFOs

Lakehouse architectures shift cost in ways that finance teams have not always understood. Storage drops dramatically. Compute is now consumed in chunks – cluster-hour or query-credit – rather than reserved continuously, which can either save or cost depending on workload.

The line items that surprise people are the object-store API costs. List operations on a table with millions of files are not free, and an unoptimized query planner can rack up significant S3 list charges. Egress between regions, especially in cross-region replicated lakes, can also produce six-figure surprises.

The teams that managed these cleanly did three things. They picked one region as primary for hot data and replicated only what needed to be replicated. They monitored S3 request metrics, not just byte counts. And they understood that Iceberg metadata reads are S3 reads, which means table layout choices have a direct cost dimension.

Where governance went

The classic data warehouse came with built-in row and column level security, audit logging, and role-based access. Splitting storage and compute fractured that, and for a while the lakehouse was a weaker governance story than what it replaced. Catalogs are starting to close the gap, but the picture is uneven.

Column masking and row filtering in the catalog now work reasonably well across the major engines. What is still messy is workload identity. A Trino cluster that talks to Iceberg with a single service account cannot easily distinguish queries by user for audit purposes. The fix involves passing user identity through to the engine and into catalog calls, which works but requires careful platform configuration.

Lineage is the other gap. The major lineage tools have plugged into the catalog APIs, which is an improvement, but lineage at the column level across multiple engines is still a project rather than a product. Most teams I know have lineage at the table level and have given up on column lineage as a near-term goal.

Where this is going

A few directions look likely to firm up over the next year. The big platform vendors all support Iceberg as a first-class table format now, which removes much of the lock-in concern that motivated lakehouse adoption in the first place. That paradoxically may slow lakehouse adoption among organizations who would otherwise have moved, because the trade between platform ease and openness has tightened.

Engine convergence on a smaller set of mature options is happening. DuckDB has become a credible interactive query layer for medium-sized lakehouses. Trino remains the default for cross-engine federation. Spark is still the bulk processing king. The proliferation of niche engines that characterized the early lakehouse years is settling down.

Catalog interoperability through the Iceberg REST spec is real progress. Multiple vendors now support reading and writing through the same REST API, which is a genuine win for portability.

Honest summary for anyone starting now

If you are designing a data architecture from scratch in 2026 and you do not have a strong reason to stay on a single warehouse, a lakehouse on Iceberg is a defensible default. It will cost you in engineering time and require you to pick a catalog deliberately. It will save you in storage and engine licensing if you can absorb the operational work.

If you have a working warehouse that meets your needs, the lakehouse migration is not free. The teams that benefited most had clear cost pressure, a strong platform team, and a real engine portability requirement. The teams that migrated because the architecture looked modern often spent two years rebuilding what they already had. The boring assessment of your current pain is the better guide than the conference talk.