Eighty percent of the "data engineering" work I’ve done in the last two years has been debugging catalog state inconsistencies, not actually processing data. It’s a bitter pill, but if your catalog isn't atomic, your petabytes of Parquet are just expensive digital trash.
The industry has collectively decided that the Iceberg REST catalog spec is the way out. It abstracts away the hive-metastore hell of s3:// path listings and S3-consistency-guarantee prayer circles. But now the industry is fragmenting again, forcing us to choose between running our own catalog, leveraging a managed provider, or locking ourselves into a vendor’s ecosystem.
You aren't choosing a "data strategy" here. You are choosing who gets to hold the kill switch for your entire data platform.
The contenders
We’re looking at three distinct paths for implementing the Iceberg REST spec:
First, Apache Polaris. It’s the new kid, open-sourced by Snowflake. It’s a pure-play REST catalog server designed to be deployed on Kubernetes. It’s vendor-neutral, which is refreshing, but it requires you to own the availability of the control plane.
Second, Databricks Unity Catalog (UC). It’s the "Enterprise Standard" if you’re already in the Databricks orbit. It’s proprietary, opaque, and incredibly powerful, but it effectively mandates that your compute lives within their walled garden.
Third, AWS Glue. The incumbent. It’s not a native REST catalog in the traditional sense, but AWS added a REST-compatible API layer for Iceberg. It’s serverless, it’s cheap, and it’s arguably the most "set it and forget it" option, provided you don't mind the inevitable IAM role hell.
Photo by 🇻🇪 Jose G. Ortega Castro 🇲🇽 on Unsplash
The operational tax on your SRE team
If you choose to self-host Polaris on EKS, you are signing up for a pager. Polaris needs a backing database—usually Postgres—to store the catalog state. If your RDS instance goes down or your K8s node group hits an OOM loop, your entire data lake goes dark.
I’ve seen teams try to save $500 a month by self-hosting metadata stores, only to spend three engineering days troubleshooting a split-brain scenario where the Iceberg metadata pointer drifted from the actual S3 manifest files. That’s a $10,000 incident for a $500 saving.
Unity Catalog, on the other hand, is managed. You don't manage the database; you manage the metastore. The failure mode here isn't infrastructure; it’s policy. I’ve spent more time debugging PERMISSION_DENIED errors in Unity than I ever did debugging database uptime in self-hosted solutions. If your Identity Provider (Okta/Entra ID) sync fails, your pipeline stops.
Glue is the middle ground. It’s managed, but it’s notoriously slow. If you have a massive table with 100,000+ partitions, Glue’s API latency becomes your primary bottleneck. You will see 5xx errors during high-concurrency partition pruning if you aren't careful with your glue:GetPartition call volume.
Cost structures and hidden tax
The cost of a catalog is never just the price of the service. It’s the integration cost.
AWS Glue is cheap—basically pennies per million requests. But it forces you to keep your compute inside AWS. If you decide to spin up a Spark job on a different provider, you’re looking at cross-cloud egress fees and the nightmare of cross-account IAM cross-pollination.
Unity Catalog is "free" if you use Databricks compute, but it’s a tax on your architectural freedom. Once you move your governance logic into Unity’s proprietary format, extracting it is non-trivial. You are paying for Unity with the loss of your ability to easily switch compute engines.
Polaris is the only one that is truly agnostic. It’s the "Linux approach." It costs exactly what it costs to run the K8s cluster and the RDS instance. But watch out for the storage of the metadata itself. If your Polaris instance is behind a load balancer that isn't configured for long-lived keep-alives, you will see intermittent ConnectionResetException errors during large metadata updates, which will leave your Iceberg tables in a corrupted staged state.
Photo by Elena Mozhvilo on Unsplash
Failure modes: When the catalog lies to you
The most dangerous thing a catalog can do is return a successful commit for a transaction that failed to write.
In the early days of Glue, we dealt with "ghost partitions." Glue would report a partition as existing, but the underlying S3 objects had been deleted due to a failed Spark job that hadn't properly rolled back. We ended up writing a custom "Catalog-to-S3 Auditor" script—essentially a nightly job that runs s3 ls against every partition entry in Glue and flags mismatches. If you choose Glue, you will need this script.
Unity Catalog is significantly more robust here because it handles the transaction commit in a single, atomic operation within the Databricks control plane. It is effectively impossible to have a drift between the Unity catalog and the underlying storage unless you have rogue users bypassing the catalog to write directly to S3.
Polaris is still maturing. I’ve encountered bugs where the metadata update fails to propagate to the read-replicas quickly enough, leading to "time-travel" anomalies where a reader sees the state from five seconds ago despite the writer confirming the commit. It’s getting better, but I wouldn't trust it with mission-critical financial reconciliation just yet.
What I'd pick, and why
If you are a startup with a small team and you’re already all-in on AWS, use Glue. It’s boring, it’s cheap, and the IAM integration, while painful to set up, is bulletproof once you get the policies right. The latency issues are real, but you can mitigate them by being disciplined about your partition schema. Don't over-partition your data.
If you are a large enterprise with a massive footprint and you already use Databricks, stop fighting the gravity. Use Unity Catalog. The governance features—specifically the lineage tracking and the fine-grained access control—are worth the lock-in. You’ll save more in headcount by not having to build your own governance layer than you’ll lose in vendor flexibility.
I would stay away from self-hosting Polaris unless you have a dedicated platform team that actually enjoys managing Postgres and Kubernetes. It’s an elegant piece of technology, and it’s the future of open-source data lakehouse architecture, but it is currently a "developer-experience" product, not an "infrastructure-as-a-service" product.
My personal preference? I’m waiting for a high-quality, managed SaaS version of Polaris. Once someone turns that into a reliable, "set-it-and-forget-it" managed service, the other two are going to be in trouble. But until then, pick your poison: Glue for the budget-conscious, Unity for the enterprise-locked, and Polaris for the infrastructure masochists.
Whatever you do, don't write your own catalog. I’ve seen that movie, and it ends with a 3:00 AM emergency partition recovery task you’ll regret for the rest of your career.
Top comments (0)