AWS Big Data Blog

Razor Group’s journey to a modern data lakehouse on AWS

Yaswanth Kothainti — Fri, 28 Aug 2026 16:25:02 +0000

Razor Group is one of Europe’s leading ecommerce aggregators, operating 250+ brands across multiple global marketplaces. With a portfolio exceeding $400M in revenue, the company relies on data to power every critical business decision, from dynamic pricing and inventory optimization to advertising spend and supply chain orchestration.

At the heart of this operation sits the Razor Operating System (ROS), a proprietary platform that processes 370M+ API calls monthly through 9,300+ data pipelines, transforming marketplace signals into automated actions at scale.

In this post, we share how Razor Group optimized their data platform by implementing a lakehouse architecture on AWS. We cover the architectural decisions, the phased migration approach, and the measurable business outcomes. Whether you’re looking to optimize workload performance, reduce infrastructure costs, or unlock multi-engine flexibility for your analytics, this blueprint provides actionable insights you can adapt for your organization.

The business challenge: Scaling data infrastructure for hypergrowth

As Razor Group’s brand portfolio expanded rapidly, the demands on their data platform grew significantly. The company needed their analytics infrastructure to keep pace with the speed of ecommerce, where pricing decisions, stock replenishment, and advertising bids happen in near real time.

Their existing architecture, built on Amazon Redshift provisioned clusters, had served them well during earlier growth stages. As workloads diversified and data volumes surged, several optimization opportunities emerged:

Figure 1: The Razor Operating System data architecture before the migration

Workload contention: Over 1,000 SQL models for ETL, transformation, and analytics competed for the same compute resources, creating resource contention during peak processing windows.
Cost-to-utilization mismatch: Always-on clusters ran 24/7, but workload analysis revealed that 98% of compute demand came from batch ETL rather than interactive analytics, which resulted in significant idle capacity during off-peak hours.
Data freshness gaps: Batch-oriented pipelines delivered data with 4–6 hour latency, limiting the team’s ability to react to fast-moving marketplace dynamics.
Scaling constraints: As concurrent users and pipeline complexity grew, vertical scaling alone couldn’t address the need for workload isolation and elastic capacity.

These weren’t failures of any single service. They were signals that the architecture needed to evolve to match the scale and diversity of Razor Group’s workloads.

Why a lakehouse architecture?

Rather than replacing their existing investments, Razor Group recognized the opportunity to optimize workload placement by adopting a modern lakehouse architecture. The core principles driving this decision:

Open table formats: Apache Iceberg provides ACID transactions, time travel, and schema evolution. Data is stored once and accessed by any compatible engine without duplication.
Elastic, per-workload scaling: With data persisted on Amazon Simple Storage Service (Amazon S3), each engine independently scales compute to match its workload. Each engine spins up for peak processing and scales to zero when idle, without over-provisioning shared infrastructure.
Multi-engine flexibility: Different workloads have different requirements. Heavy ETL benefits from distributed Spark processing, ad hoc exploration from serverless queries, and business intelligence (BI) dashboards from high-performance warehouse engines, each optimized for its purpose.

This approach allowed Razor Group to right-size each workload to the best-fit engine while maintaining a single, governed copy of data accessible across the entire platform.

Solution overview

Razor Group partnered with AWS to implement a comprehensive lakehouse architecture that brings together multiple AWS services, each playing a complementary role:

Figure 2: End-to-end lakehouse architecture on AWS

Designing for scale: The lakehouse vision

The core insight driving Razor Group’s new architecture was simple: build a single, open format data lake that any engine can query. In the old model, each tool maintained its own copy of the data. In the new model, a single open-format data lake on Amazon S3 serves as the source of truth, and multiple purpose-built compute engines read from it based on the workload at hand.

This shift, commonly called a lakehouse architecture, combines the cost economics and scalability of a data lake with the query performance and governance of a data warehouse. Its open table format, Apache Iceberg, provides ACID transactions, schema evolution, time travel, and no vendor lock-in.

Storage and governance: The open data foundation

Amazon S3 Tables (a capability of Amazon S3) with Apache Iceberg — The primary storage layer, providing open-format tables with ACID transactions, partition evolution, and time travel. Data is stored once and accessible by any Iceberg-compatible engine.
AWS Glue Data Catalog — A unified metadata repository for consistent data discovery across all compute engines.
AWS Lake Formation — Fine-grained access control with column-level and row-level security so that governance scales with the platform.

Compute: Right engine for the right workload

Apache Spark on Amazon Elastic Compute Cloud (Amazon EC2) — Elastic, distributed compute for heavy ETL and transformation workloads. It uses AWS Graviton instances and Amazon EC2 Spot Instances for cost optimization.
Amazon Athena — Serverless SQL for ad hoc exploration and lightweight queries directly on Iceberg tables, with no infrastructure to manage.
Amazon Redshift Serverless — High-performance serving layer for BI dashboards, Tableau workloads, and interactive analytics. Amazon Redshift Serverless automatically scales to meet demand and pauses when idle, so it stays cost-efficient for the analytics workloads it serves best.

Orchestration and observability

Apache Airflow — Pipeline orchestration that manages 9,300+ data pipelines with dependency tracking and service level agreement (SLA) monitoring.
Comprehensive observability stack — Cost attribution, pipeline health monitoring, and data quality checks across all layers.

Note: When the architecture was originally designed, Amazon Redshift lacked Iceberg write support, making self-managed Spark the only viable ingestion path. This constraint has since been removed. Amazon Redshift now supports full Apache Iceberg DML (UPDATE, DELETE, MERGE), complementing its earlier CREATE/INSERT capabilities and AWS Glue Iceberg materialized views. This makes it a complete read/write Iceberg engine.

Migration approach

Rather than a risky big-bang cutover, Razor Group adopted a phased migration of five stages, each delivering standalone value while building the foundation for the next. Both Amazon Redshift and Spark pipelines ran in parallel during the transition, which maintained business continuity and let the team compare outputs with confidence. At no point was a production pipeline paused or a dashboard unavailable.

The migration journey: Five phases

The migration unfolded across five structured phases, each building on the previous one and delivering incremental value before the next began.

Phase 1: Establish the lakehouse foundation

Before migrating a single query, Razor Group needed to answer three questions: where does the data live, how is it managed, and how do we query it?

Why S3 Tables over self-managed Iceberg

Razor Group had already committed to Apache Iceberg as the table format: open, engine-agnostic, and equipped with ACID transactions and time travel. The question was whether to self-manage Iceberg on standard S3 buckets or use Amazon S3 Tables.

Self-managed Iceberg is powerful but operationally expensive. Someone has to run compaction jobs to prevent small-file proliferation. Someone has to expire old snapshots before metadata bloat degrades query planning. Someone has to clean up orphaned data files after interrupted writes. With 700+ models running across 40+ schemas, many of them materializing multiple times per day, that maintenance burden would scale with the platform rather than shrink.

S3 Tables eliminated this entire category of work. Compaction, snapshot management, and unreferenced file removal run continuously and automatically. The integrated Iceberg REST Catalog API means any compatible engine, such as Spark, Trino, Athena, Amazon Redshift, and Flink, can discover and query tables without maintaining a separate metastore. Discovery is unified through AWS Glue Data Catalog, which now exposes the Iceberg REST Catalog protocol as its access interface. Because tables are first-class AWS resources, access control, encryption, and lifecycle policies operate at the table level rather than through complex S3 bucket policies layered on top of file-path conventions.

For a company that didn’t want the operational burden of self-managing open table format maintenance, this was the deciding factor.

AWS Glue Data Catalog provides unified metadata discovery across all tiers. Lake Formation handles column- and table-level access control, with AWS Identity and Access Management (IAM) roles that follow least-privilege principles and AWS CloudTrail turned on for a full audit trail.

Choosing the query protocol

Prior to the rearchitecture, the Amazon Redshift cluster was 98% ETL, and only a fraction of compute hours were analyst SELECT queries. The replacement engine needed to handle both heavy batch transformations and interactive ad hoc queries.

Traditional Spark (spark-submit) handles batch ETL well, but couples clients to the cluster. Every job requires packaging driver JARs, managing classpaths, and submitting from within the cluster. For a platform running 200+ production directed acyclic graphs (DAGs) that process massive data volumes daily, this operational friction was a non-starter.

Spark Connect is the gRPC-based client-server protocol introduced in Spark 3.4, and it solved the coupling problem entirely. The cluster runs a persistent gRPC endpoint. Clients connect remotely and submit queries over the wire. Airflow operators become thin clients: they open a session, submit SQL, and get results, with success and failure mapping directly to task states. There are no driver JARs and no polling. Multiple consumers, including pipeline orchestrators, the web application, and developer notebooks, share one cluster without any of them needing Spark installed locally.

Deploying Spark Connect

Razor Group deployed a self-hosted Spark cluster on Amazon EC2: an on-demand AWS Graviton leader node, Spot workers at about 70% cost savings, and the Spark Connect endpoint exposed through an internal Network Load Balancer. Custom Amazon Machine Images (AMIs) bake in the full Spark, Iceberg, and S3 Tables stack, so private-subnet nodes have everything they need without internet access at runtime.

This phase produced no immediate business value, but it made everything that followed possible.

Phase 2: Migrate data ingestion

Razor Group’s ingestion layer pulls data from Amazon Selling Partner API, Seller Central portals, NetSuite ERP, and custom web scrapers. In the previous architecture, all of this landed in Amazon Redshift through COPY commands, which meant data freshness was dictated by batch job schedules and competed for resources on the same cluster that served analytical queries.

Razor Group migrated these pipelines to AWS Lambda functions orchestrated by Apache Airflow, writing data directly to S3 Tables in Iceberg format. The shift from schedule-driven to event-driven significantly improved freshness. Lambda functions spin up only when there’s data to process, and Airflow sensors trigger downstream transformations the moment new data lands. This replaced rigid hourly batch windows with data freshness measured in minutes.

The orchestration layer manages 200+ DAGs across 90+ flows and processes data from dozens of sources at scale. The migration required rewiring destinations from Amazon Redshift COPY to Iceberg writes, but the orchestration logic itself carried over with minimal changes.

This phase alone eliminated roughly 40% of compute costs by severing the always-on cluster dependency for ingestion.

Phase 3: Transform processing pipelines

This was the most technically demanding phase, and where Razor Group learned the most. The team migrated 1,000+ SQL models from Amazon Redshift to Apache Spark, working incrementally up the dependency chain across 40+ schemas. The models moved through a medallion structure: Bronze for raw ingested data, Silver for cleaned and conformed data, and Gold for business-ready aggregates.

Razor Group built automated conversion tooling and a validation framework that ran both Amazon Redshift and Spark outputs in parallel, comparing results row-by-row before decommissioning anything. Several categories of transformation pushed the limits of what automation could handle:

Window functions: The QUALIFY clause in Amazon Redshift has no Spark equivalent. Each instance required wrapping in a subquery with explicit row numbering, which affected dozens of models in the inventory schema alone.
JSON serialization: The most time-consuming category. Complex columns stored as JSON STRING in Amazon Redshift needed from_json() with hand-written STRUCT definitions in Spark. Every nested payload column across ads, orders, and transaction pipelines required schema introspection, with no shortcuts.
Function dialect: More than 20 function-level conversions, including NVL to COALESCE, DATEADD to interval arithmetic, and LISTAGG to ARRAY_JOIN(COLLECT_LIST()).
Snapshot elimination: The single biggest hidden cost. Full table copies that ran multiple times daily only to preserve point-in-time state consumed more than 35 hours of weekly Amazon Redshift compute. With Iceberg’s native time travel, these became zero-cost operations overnight.

When migrating 1,000+ SQL models, automated tooling handles the mechanical syntax conversions well. But roughly 30% of the models required human judgment: those with complex JSON payloads, deeply nested window functions, or cross-schema snapshot dependencies. These models consumed 70% of the migration effort.

Razor Group built a structured migration workflow that used Claude to accelerate this work: read source SQL, identify dependencies, convert syntax, resolve missing base tables, add JSON parsing, validate outputs, and write to the lakehouse. The system did more than translate SQL. It applied schema context, traced cross-model dependencies, and flagged edge cases that would have taken engineers hours to find manually. What could have been a multi-year effort became a systematic, repeatable process measured in weeks. This approach fundamentally changed the speed of migration.

Phase 4: Unify the serving layer

With data flowing through Iceberg tables, Razor Group collapsed the serving layer. End users query Gold-layer Iceberg tables through Amazon Redshift Serverless, and internal exploration and machine learning (ML) workloads read the same tables through Spark Connect. This removed the need to maintain separate data copies, materialized views, or extract jobs for different consumers.

This is the strategic payoff of an open table format. Iceberg tables on S3 are engine-agnostic: Spark for batch transforms today, Trino for interactive queries tomorrow, Flink for streaming next quarter. Any engine that speaks Iceberg can read the data without conversion or migration. Razor Group went from being locked into a single vendor’s SQL dialect to having the freedom to adopt new engines without touching the storage layer.

Phase 5: Operationalize and observe

The final phase made the lakehouse production-grade. Razor Group built a comprehensive observability stack that aggregates metrics, traces, and logs from every pipeline component into a unified view. This view supports centralized log search, anomaly detection, and automated alerting that correlates failures across the entire data platform.

This observability layer did more than provide visibility. It gave the team confidence. When you’re running thousands of pipeline executions daily, you need to know within minutes when something breaks, what caused it, and which downstream consumers are affected. That’s the difference between reactive firefighting and proactive operations.

Pipeline orchestration consolidated around three patterns: a daily pipeline (ingestion to materialization to export to AI agent analysis), an operations worker polling every 15 minutes, and weekly scraper jobs.

The cutover was zero-downtime by design: both schedulers ran in parallel for two weeks. Automated comparison checks validated that every pipeline produced identical outputs before the prior architecture system was disabled.

Results and business impact

The lakehouse architecture delivered measurable improvements across every dimension:

Metric	Before	After	Improvement
P95 query runtime	180 seconds	63 seconds	65% faster
Infrastructure cost	Always-on provisioned clusters	Elastic, workload-optimized	63% reduction
Data freshness	4–6 hour batch cycles	Event-driven pipelines	15-minute freshness
Concurrent capacity	Limited by cluster size	Elastic, independent scaling	Unlimited
Engine flexibility	Single engine	Multi-engine (Spark, Athena, Amazon Redshift)	Open format portability

The 63% reduction compares the lakehouse run-rate (January–March 2026) with the pre-rearchitecture run-rate (October–December 2025), the trailing three months before the rearchitecture. The figure is an apples-to-apples blended infrastructure number that includes compute and storage across both architectures. The before column covers Amazon Redshift cluster compute and managed storage. The after column covers Amazon EC2 (Spark workers, both on-demand and Spot), AWS Lambda, AWS Glue, Amazon Athena, Amazon Redshift Serverless, and S3 Tables storage. Data-transfer and ancillary services are excluded because they were not materially different between the two periods. Workload mix (the number of pipelines, models, and end-user query volume) was held broadly comparable across the two windows.

Lessons learned

Start with the decision loops, not the tools, and know your workload before you replace your warehouse.

The most valuable activity of the entire migration wasn’t writing a line of code. It was the Amazon Redshift workload analysis we ran before making any architectural decisions. Discovering that 98% of compute was ETL, with only a sliver going to analyst queries, validated the move to on-demand Spark. It also prevented us from over-provisioning the replacement infrastructure for interactive workloads that barely existed. Architecture decisions should always trace back to core business requirements: pricing accuracy, promotional responsiveness, intraday P&L visibility. Start there, not with the technology.

Design for multiple compute engines, and choose the right engine per workload.

One of the clearest lessons from running a single-engine architecture is what you give up. Avoid locking yourself into one compute layer for BI, ingestion, backfills, and ML alike, because they have fundamentally different cost and performance profiles. Iceberg, Spark, and S3 Tables work well together out of the box once you make the shift. The technology isn’t the hard part. The hard part is mapping 1,000+ models across 40+ schemas, tracing dependencies through 200+ DAGs, and discovering that a column is actually a JSON string silently serialized differently between two engines. Migration is as much an excavation project as an engineering one.

Automate conversion, but budget for the 30%.

Automated tooling handles mechanical syntax conversions well, and it should be the first tool you reach for. But models with complex JSON payloads, deeply nested window functions, or cross-schema snapshot dependencies require human judgment, and that work doesn’t compress. Roughly 30% of our models needed significant manual intervention, and those models consumed 70% of the total migration effort. Plan for it honestly from the start.

Observability must include cost attribution, and watch out for hidden cost bombs.

Snapshot operations were our biggest surprise. Full table copies that ran multiple times daily to preserve point-in-time state were costing more than 35 hours of weekly compute, and nobody questioned it because “that’s how snapshots work.” Iceberg’s time-travel capability eliminated their cost, and that single feature justified a meaningful portion of the migration on its own. More broadly, you cannot optimize what you cannot see, so track query-level usage and attribute it to teams and functions. Cost observability is not a nice-to-have. It’s foundational.

Governance isn’t optional. Build it into the foundation, and align stakeholders from day one.

Catalog and access control need to come first, before you scale adoption, not after. The same principle applies to people: migration is a cross-functional program, not an infrastructure project. Our two-week parallel run caught edge cases that row-level validation missed entirely: time zone differences between Amazon Redshift and Spark, partition pruning behavior under concurrent writes, and subtle ordering differences in non-deterministic window functions. That parallel run wasn’t a safety net. It was where the migration actually proved itself. None of it works without the right stakeholders involved and aligned from the very beginning.

Conclusion

Razor Group’s journey offers valuable lessons for organizations looking to optimize their data architectures:

Analyze your workload mix first. Understanding that 98% of compute was ETL rather than interactive queries guided the decision to offload heavy processing to elastic Spark, while preserving Amazon Redshift Serverless for the interactive analytics it handles best.
Design for multi-engine flexibility. Open table formats like Apache Iceberg eliminate the need to choose a single engine. Each workload runs on the engine best suited to its access pattern, cost profile, and performance requirements.
Automate migration, but budget for complexity. Automated transpilation handled 70% of SQL models, but the remaining 30% consumed 70% of engineering effort. Plan accordingly.
Observability must include cost attribution. Without per-workload cost visibility, optimization is guesswork. Razor Group discovered that Iceberg snapshot maintenance alone consumed more than 35 hours of compute weekly, a hidden cost that observability surfaced and automation resolved.
Build governance into the foundation. AWS Lake Formation and AWS Glue Data Catalog provided fine-grained access control from day one, not retrofitted after the migration.
Validate with parallel systems. A two-week parallel run between old and new architectures caught edge cases that automated testing missed, which supported a confident production cutover.

The road ahead

With the lakehouse foundation in place, Razor Group is positioned to accelerate innovation, from real-time pricing models to AI-driven inventory optimization, all powered by a unified, open, and governed data platform on AWS.

The company’s transformation demonstrates that modern data architectures aren’t about choosing between services. They’re about placing each workload where it performs best, using open formats to eliminate silos, and scaling each layer independently as the business grows.

To learn how other organizations are implementing similar lakehouse architectures on AWS, see How BigBasket uses the Iceberg-based lakehouse architecture on AWS to power lightning-fast grocery delivery across India.

About the authors

How Picnic configured multiple OAuth providers for Amazon MQ

Oscar Mapfumo Sibanda — Thu, 27 Aug 2026 16:07:20 +0000

This post is co-written with Oscar Mapfumo Sibanda from Picnic.

Picnic is an Amsterdam-based tech scale-up that reinvents how people buy food. It isn’t a supermarket with a digital layer but a tech company that happens to deliver groceries. Picnic is engineered in-house: the customer app, the fulfillment platform, the supply chain, and the routing technology that guides a fleet of thousands of electric vehicles through the Netherlands, Germany, and France. Software doesn’t merely support business. Software is a business.

At the center of this system, RabbitMQ is the core component. It’s the communication backbone connecting hundreds of microservices across the entire business lifecycle, from ordering and logistics to delivery and finance. At peak, Picnic’s platform processes close to one million messages per second. At this scale, messaging is no longer only background infrastructure. It becomes part of the company’s operational nervous system. To keep that system highly available, scalable, and resilient as Picnic grows, the company decided to use Amazon MQ as a managed service.

The next challenge was identity. Picnic’s authentication strategy clearly distinguishes between people and services. Operators sign in through Keycloak, the company’s single sign-on provider, while Picnic’s Amazon Elastic Kubernetes Service (Amazon EKS) workloads are adopting AWS Identity and Access Management (IAM) authentication to eliminate static credentials. A single broker therefore must trust two identity providers at once. The Amazon MQ documentation covers configuring OAuth 2.0 with a single provider. This post extends that guidance to a multi-provider setup on the same broker.

In this post, we show how Picnic solved that problem. You will learn how to configure an Amazon MQ for RabbitMQ broker to accept tokens from multiple OAuth 2.0 identity providers, using Keycloak and IAM as the working example. You will also see how to map each provider’s scopes to RabbitMQ permissions and how to roll the change out on a running broker without disrupting connected users.

Background and prerequisites

Amazon MQ for RabbitMQ supports OAuth 2.0 authentication and authorization, where broker users and their permissions are managed by an external identity provider. User authentication and resource permissions for vhosts, exchanges, queues, and topics are centralized through the OAuth 2.0 provider’s scope system.

RabbitMQ’s OAuth 2.0 plugin supports multiple resource servers and audiences, allowing different OAuth 2.0 providers to issue tokens that a single broker can validate. This capability is essential if you operate in multiple environments or have teams registered with separate identity providers.

Prerequisites

To follow along with this post, you need:

An active AWS account.
An Amazon MQ for RabbitMQ broker with OAuth 2.0 configured for at least one identity provider (see Using OAuth 2.0 authentication and authorization for Amazon MQ for RabbitMQ).
A second OAuth 2.0 identity provider configured and operational.
Outbound web identity federation enabled for your AWS account (if using IAM as a provider).
Basic familiarity with RabbitMQ configuration and OAuth 2.0 concepts.
AWS Command Line Interface (AWS CLI) version 2.27 or later (required for the get-web-identity-token command used in the testing section).

Note: The information in this post reflects Amazon MQ for RabbitMQ features and behavior at the time of publication. We recommend checking the Amazon MQ documentation, release notes and best practices before implementation.

Solution architecture

The design rests on a single idea: a RabbitMQ broker can trust more than one identity provider at the same time, and it decides which one to apply per token rather than per broker. RabbitMQ does this by reading the aud (audience) claim of each incoming token and matching it against a configured resource server. Each resource server is bound to one OAuth 2.0 provider, so the audience determines both which signing keys validate the token and which permission rules apply.

Architecture diagram

The following two diagrams show how services and operators authenticate the broker through their respective identity providers.

Figure 1: Services (IAM) flow

Services authenticate through IAM. An Amazon EKS workload assumes an IAM role and generates a web identity token (1), which AWS Security Token Service (AWS STS) issues with an audience of rabbitmq-iam (2). The workload presents that token as its password when it connects to the broker over Advanced Message Queuing Protocol (AMQPS) on port 5671 (3). The broker selects the matching resource server, verifies the token’s signature against the AWS STS signing keys (4), and maps the role’s Amazon Resource Name (ARN) to the permissions the workload needs (5).

Figure 2: Operators (Keycloak) flow

Operators authenticate through Keycloak. An operator opens the RabbitMQ management console and initiates login (1), and the console redirects the browser to Keycloak (2). Keycloak authenticates the operator and issues a token whose audience targets the console resource server, rabbitmq-keycloak (3). The browser presents that token to the broker (4), which verifies the signature against Keycloak’s signing keys (5). The broker then reads the operator’s group membership and grants access (6): the Operator group receives read-only permissions, while the Administrator group receives full control.

Two constraints follow this design. First, audience verification is a single broker-wide setting that applies to every provider at once, so each provider must issue tokens carrying the exact audience its resource server expects. Second, the broker is private, deployed inside an Amazon Virtual Private Cloud (Amazon VPC) with no public exposure. Both providers’ endpoints must be resolvable, either through publicly addressable JSON Web Key Set (JWKS) endpoints or through private networking, because the broker fetches signing keys from those endpoints.

Implementation walkthrough

This walkthrough configures one Amazon MQ for RabbitMQ broker to trust two identity providers: Keycloak for operators and AWS IAM for services. The steps assume you already have a running broker, a Keycloak realm, and outbound web identity federation enabled for your AWS account. All configurations are applied through a RabbitMQ configuration revision using the AWS Command Line Interface (AWS CLI).

Enable OAuth 2.0 on the broker

The first block activates the OAuth 2.0 backend and keeps the internal backend in place. Internal authentication remains active deliberately: Amazon MQ creates an administrator user when the broker is provisioned, and that user is needed for break-glass access.

auth_backends.1 = oauth2
auth_backends.2 = internal
auth_oauth2.verify_aud = true

Setting verify_aud = true tells RabbitMQ to reject any token whose aud claim does not match a configured resource server. This single broker-wide setting governs every provider you add.

Add the first identity provider (Keycloak)

A resource server binds an audience value to a provider and a set of permission rules. The Keycloak resource server uses the id rabbitmq-keycloak, which is the audience the realm must place in its tokens. RabbitMQ reads the operator’s group membership from the group_membership claim and resolves it through scope aliases.

auth_oauth2.resource_servers.1.id = rabbitmq-keycloak
auth_oauth2.resource_servers.1.oauth_provider_id = keycloak
auth_oauth2.resource_servers.1.scope_prefix = rabbitmq.
auth_oauth2.resource_servers.1.additional_scopes_key = group_membership
auth_oauth2.resource_servers.1.preferred_username_claims.1 = email
auth_oauth2.resource_servers.1.scope_aliases.1.alias = Operator
auth_oauth2.resource_servers.1.scope_aliases.1.scope = rabbitmq.read:*/* rabbitmq.write:^$ rabbitmq.configure:^$ rabbitmq.tag:monitoring
auth_oauth2.resource_servers.1.scope_aliases.2.alias = Administrator
auth_oauth2.resource_servers.1.scope_aliases.2.scope = rabbitmq.read:*/* rabbitmq.write:*/* rabbitmq.configure:*/* rabbitmq.tag:administrator

The Operator group is read-only: it can read any resource and view the management UI through the monitoring tag. The Administrator group receives full permissions plus the administrator tag. This role is least privilege by design.

The provider is configured with its issuer and JWKS endpoint:

auth_oauth2.oauth_providers.keycloak.https.hostname_verification = wildcard
auth_oauth2.oauth_providers.keycloak.issuer = https://keycloak.example.com/auth/realms/test
auth_oauth2.oauth_providers.keycloak.jwks_uri = https://keycloak.example.com/auth/realms/test/protocol/openid-connect/certs

To let operators sign in from the management console, expose Keycloak as a management resource:

management.oauth_enabled = true
management.oauth_disable_basic_auth = false
management.oauth_scopes = openid email profile
management.oauth_resource_servers.1.id = rabbitmq-keycloak
management.oauth_resource_servers.1.oauth_client_id = rabbitmq-keycloak
management.oauth_resource_servers.1.label = Keycloak SSO

Add the second identity provider (AWS IAM)

Adding a second provider means adding a second resource server and a second entry under oauth_providers. The IAM resource server uses the id rabbitmq-iam because the audience is set when minting the token.

auth_oauth2.resource_servers.2.id = rabbitmq-iam
auth_oauth2.resource_servers.2.oauth_provider_id = aws_iam
auth_oauth2.resource_servers.2.scope_prefix = rabbitmq/
auth_oauth2.resource_servers.2.additional_scopes_key = sub
auth_oauth2.resource_servers.2.scope_aliases.1.alias = arn:aws:iam::123456789012:role/EKSWorkloadRole
auth_oauth2.resource_servers.2.scope_aliases.1.scope = rabbitmq/read:*/* rabbitmq/write:*/* rabbitmq/configure:*/* rabbitmq/tag:policymaker
auth_oauth2.oauth_providers.aws_iam.https.hostname_verification = wildcard
auth_oauth2.oauth_providers.aws_iam.issuer =
auth_oauth2.oauth_providers.aws_iam.jwks_uri =

The IAM workload receives the policymaker tag. It can publish, consume, and manage policies but does not receive the administrator tag.

Apply the configuration and restart the broker:

CONFIG_ID=$(aws mq describe-broker --broker-id $BROKER_ID \
  --query 'Configurations.Current.Id' --output text)
REVISION=$(aws mq update-configuration --configuration-id $CONFIG_ID \
  --data "$(cat rabbitmq.conf | base64 | tr -d '\n')" \
  --query 'LatestRevision.Revision' --output text)
aws mq update-broker --broker-id $BROKER_ID \
  --configuration Id=$CONFIG_ID,Revision=$REVISION
aws mq reboot-broker --broker-id $BROKER_ID

Note: The base64 command syntax differs between Linux and macOS. The preceding command (cat file | base64 | tr -d '\n') is portable on both operating systems. If running exclusively on Linux, you can also use base64 --wrap=0 rabbitmq.conf. On macOS, the equivalent command is base64 -i rabbitmq.conf.

Testing and validation

Validate each provider independently. For IAM, assume the role and request a token from AWS STS, then present it as the AMQP password:

TOKEN=$(aws sts get-web-identity-token \
  --audience "rabbitmq-iam" \
  --signing-algorithm ES384 \
  --duration-seconds 300 \
  --query 'WebIdentityToken' --output text)
# Username is empty (ignored by the OAuth plugin); the token is passed as the password
curl -u ":$TOKEN" https://<broker-id>.mq.<region>.on.aws/api/overview

Note: The get-web-identity-token API requires outbound web identity federation to be enabled on your AWS account and AWS CLI version 2.27 or later.

A successful response confirms the IAM resource server accepted the token. For Keycloak, open the management console, choose Keycloak SSO, and sign in as an operator.

When a login fails, decode the JSON Web Token (JWT) and check two claims. The aud claim must exactly match a resource server id. With verify_aud = true, a missing or mismatched audience is the most common cause of rejection. If the audience is correct but permissions are missing, verify the scope_prefix is set correctly.

Operational considerations

A few points deserve attention before you run this pattern in production.

Key rotation: When rotating signing keys at a provider, publish the new key in the JWKS endpoint before revoking the old one. The broker caches keys, so overlapping both during the transition window prevents authentication failures while the cache refreshes.
Audience validation: Audience remains the linchpin. With verify_aud enabled, every provider must issue tokens carrying the audience its resource server expects, so confirm this whenever you onboard a new one. Don’t disable audience validation in production. The RabbitMQ OAuth 2.0 plugin does not perform token revocation checks, which makes audience binding a critical control that prevents tokens issued for other services from granting access.
Scope prefix: scope_prefix values are optional. They’re needed only if the tokens don’t follow the default format. RabbitMQ only reads scopes carrying the expected prefix, so a token can authenticate yet grant nothing if the prefix is missing. Map each provider to the least privilege its principals need. For example, prefer narrow scopes like read:orders over blanket read:all to limit the scope of impact if a single provider’s credentials are compromised.
Token lifetime: Because the plugin does not support token revocation, token lifetime is your primary control over leaked credentials. Issue short-lived access tokens and have your client applications refresh them proactively at approximately 75 percent of the token’s lifetime to avoid connection disruptions when a token expires mid-session.
Monitoring: Authentication failures and refused tokens are recorded in the broker’s connection log group in Amazon CloudWatch, which can be reached through the Amazon CloudWatch Logs link on the broker’s page in the Amazon MQ console. Beyond logs, set up CloudWatch alarms on RabbitMQMemUsed, RabbitMQDiskFree, and ConnectionCount. An unexpected spike in failed connections is often the first sign of a token or audience misconfiguration. For unaggregated, per-node visibility, consider enabling the Prometheus metrics endpoint: metrics such as rabbitmq_auth_attempts_failed_total surface OAuth rejections faster than the CloudWatch one-minute polling interval.
Network controls: Enforce defense in depth by restricting broker access using security groups so that only authorized VPCs and IP ranges can reach the AMQPS and management endpoints. This matters especially in an OAuth setup because, once a token has been issued, the broker cannot revoke it before it expires.

Cleanup

To avoid incurring future costs, delete the resources created during this walkthrough if you no longer need them:

Delete Amazon MQ broker and configurations.
Remove test OAuth application registrations from your identity providers.
Delete any IAM roles created for testing.

Conclusion

In this post, we demonstrated how Picnic configured an Amazon MQ for RabbitMQ broker to authenticate tokens from two OAuth 2.0 identity providers: Keycloak for human operators and AWS IAM for machine-to-machine services on a single broker instance. The key mechanism is RabbitMQ’s support for multiple resource servers, where the audience claim in each token determines which provider’s signing keys and permission rules apply.

With this approach, the Picnic team was able to cleanly separate human and machine authentication without the operational overhead of running separate brokers, while retaining fine-grained access control for both token issuers.

This pattern works with any combination of OAuth 2.0 providers and is particularly valuable for organizations looking to consolidate messaging infrastructure while maintaining distinct identity boundaries.

To learn more about Amazon MQ for RabbitMQ and OAuth 2.0 authentication, see Authentication and authorization for Amazon MQ. For a hands-on walkthrough of configuring OAuth 2.0 with Amazon MQ for RabbitMQ, see Using OAuth 2.0 authentication and authorization for Amazon MQ for RabbitMQ. The configuration examples in this post are broker-level settings applied through the Amazon MQ API. No standalone code repository is required.

About the authors

Build with geospatial and variant types in Iceberg v3 on AWS Glue 6.0

Shoukat Ghouse — Thu, 27 Aug 2026 16:00:12 +0000

xfAs organizations build data lakes that combine geospatial data, high-frequency event streams, and heterogeneous payloads, the limitations of older table formats become acute. Without a native geospatial type, coordinates require separate float columns (latitude/longitude) with no spatial predicates. Without nanosecond-precision timestamps, sub-microsecond event ordering is lost. Without a variant type, semi-structured data forces a choice between rigid flattening and untyped JSON strings. Each workaround adds complexity, slows queries, and increases maintenance burden.

AWS Glue 6.0, powered by Apache Spark 4.1, removes these workarounds by adding support for Apache Iceberg v3, bringing new column-level capabilities to your data lake tables. These include new data types: native geospatial types (GEOMETRY with spatial predicates, and GEOGRAPHY), nanosecond-precision timestamps, and the VARIANT type for semi-structured data with automatic shredding. Iceberg v3 also adds support for DEFAULT column values. These are table format features. After they’re written, they’re readable by any Iceberg v3-compatible engine that supports these features.

In this post, we build a connected vehicle fleet monitoring pipeline that uses these capabilities in a single Iceberg v3 table. Vehicles emit telemetry events with GPS coordinates (geospatial), sub-microsecond event times (nanosecond), and sensor payloads that vary by vehicle type (variant). We ingest these events, run spatial queries to detect geofence violations, sequence events at nanosecond precision, and extract typed metrics from heterogeneous payloads, all without workarounds, flattening, or external libraries.

Solution overview

A logistics company operates a mixed fleet of delivery vehicles: vans, electric bikes, and delivery robots. Each vehicle type produces telemetry events with a different sensor payload schema. The operations team needs to:

Detect geofence violations: flag vehicles that enter restricted zones (airports, pedestrian areas, private property).
Sequence events precisely: at fleet scale, many events land in the same microsecond window. Nanosecond timestamps give a deterministic order and prevent ties when sequencing or deduplicating events during processing.
Extract metrics from heterogeneous payloads: query battery level from delivery robots, fuel level from vans, and pedal cadence from bikes, all stored in the same column.

We address all three requirements with a single Iceberg v3 table on AWS Glue 6.0. The following data definition language (DDL) shows the table structure. The AWS Glue job we provision in subsequent steps executes this statement.

CREATE TABLE fleet_monitoring_db.vehicle_telemetry (
event_id STRING,
vehicle_id STRING,
vehicle_type STRING DEFAULT 'UNKNOWN',
event_time TIMESTAMP_NTZ(9),
location GEOMETRY(4326),
service_area GEOGRAPHY(4326),
sensor_payload VARIANT,
speed_kmh DOUBLE DEFAULT 0.0,
region STRING DEFAULT 'EMEA'
) USING ICEBERG
TBLPROPERTIES (
'format-version' = '3',
'write.delete.mode' = 'merge-on-read'
)
PARTITIONED BY (days(event_time), vehicle_type)

In the preceding statement, the database is shown as fleet_monitoring_db for readability. The deployed stack creates it as fleet_monitoring_<account-id>.

The following list describes the key columns:

event_time TIMESTAMP_NTZ(9): Stores the event timestamp at nanosecond precision.
location GEOMETRY(4326): Stores GPS coordinates as native spatial objects using (SRID 4326). You can use predicates like ST_Intersects directly in SQL, replacing hand-coded spatial math on raw latitude/longitude doubles (WGS 84).
service_area GEOGRAPHY(4326): Stores geographic coordinates using a spherical (geodesic) model, distinct from GEOMETRY’s planar model. AWS Glue 6.0 writes and reads GEOGRAPHY in Iceberg v3, and the type is portable to any Iceberg v3-compatible engine. Geodesic spatial predicates over GEOGRAPHY are engine-dependent today. In this post we run spatial queries on the GEOMETRY location column, which Glue 6.0 supports natively.
sensor_payload VARIANT: Each vehicle type produces a different JSON schema. Vans report fuel and engine metrics, robots report battery and camera status, bikes report cadence and heart rate. All land in this single column without schema unions or separate tables using variant data type.
vehicle_type STRING DEFAULT ‘UNKNOWN’ and speed_kmh DOUBLE DEFAULT 0.0: When an ingestion writer omits these fields, Iceberg applies the declared defaults automatically. Useful when multiple producers write to the same table and not all of them populate every column.

The table uses PARTITIONED BY (days(event_time), vehicle_type) so that analytical queries can prune by date range and vehicle type without scanning the full table. 'write.delete.mode' = 'merge-on-read' supports fast row-level corrections (for example, correcting a misreported GPS coordinate) through compact deletion vectors (Roaring Bitmaps) instead of accumulating positional delete files.

In this post, we insert sample data directly to focus on the new Iceberg data types and how to use them together. In production, these events would stream from Amazon Managed Streaming for Apache Kafka (Amazon MSK) into an AWS Glue 6.0 streaming job.

The following diagram illustrates the production architecture for reference:

Figure 1: Reference architecture for a fleet telemetry pipeline on AWS Glue 6.0

The architecture processes vehicle telemetry through two paths, with a downstream batch analytics layer:

Hot path (real-time, milliseconds): A Spark Real-Time Mode (RTM) job reads telemetry from Amazon MSK and evaluates geofence violations using spatial predicates like ST_Intersects, routing alerts to a downstream Kafka topic within milliseconds.

Cold path (near-real-time, seconds): A micro-batch job reads the same MSK topic and writes events into an Iceberg v3 table, converting payloads to GEOMETRY, TIMESTAMP_NTZ(9), and VARIANT columns with DEFAULT values applied.

Batch analytics: An AWS Glue job reads the Iceberg v3 table to run batch analytics on geofence detection, nanosecond event sequencing, and per-vehicle-type metric extraction.

Prerequisites

To follow along, you need:

An AWS account and an AWS Region where AWS Glue 6.0 is available.
An AWS Identity and Access Management (IAM) role with permissions to deploy AWS CloudFormation stacks and create resources including AWS Glue, Amazon Simple Storage Service (Amazon S3), and Amazon CloudWatch Logs.

Deploy the CloudFormation stack

We provide an AWS CloudFormation template that provisions all the resources needed for this walkthrough.

The stack provisions the following resources:

An Amazon S3 bucket for Iceberg table storage.
An IAM role with permissions for AWS Glue, Amazon S3, and Amazon CloudWatch Logs.
An AWS Glue database (fleet_monitoring_<account-id>).
An AWS Glue job fleet-telemetry-ingest-<account-id> (PySpark): creates the Iceberg v3 table vehicle_telemetry described earlier and inserts sample telemetry from three vehicle types.
An AWS Glue job fleet-telemetry-queries-<account-id> (PySpark): demonstrates geofence detection, nanosecond sequencing, variant extraction, and default values.

Deploy the CloudFormation stack:

Download the CloudFormation template from the GitHub repository.
Sign in to the AWS CloudFormation console.
Choose Create stack, With new resources, Upload a template file, and upload the downloaded template.
Acknowledge the IAM capabilities and choose Create stack.

Stack creation takes approximately 2–5 minutes. No parameters are required.

After the stack completes, navigate to the AWS Glue console and run the jobs in this order:

Run fleet-telemetry-ingest-<account-id>. This job creates the Iceberg v3 table and inserts sample data (approximately 2 minutes).
After it succeeds, run fleet-telemetry-queries-<account-id>. This job executes all demonstration queries (approximately 2 minutes).

The following sections describe each job in detail.

Job 1: Ingest sample telemetry data

The ingestion job creates the Iceberg v3 table described earlier and inserts four sample telemetry events: one for each of the three vehicle types (van, robot, bike), plus one with omitted fields to demonstrate DEFAULT values. You can view the complete script in the GitHub repository. Note that the geospatial types require one additional Spark configuration (spark.sql.geospatial.enabled=true), which is already set in the job’s --conf argument by the CloudFormation template. All other types work with no extra configuration.

The following are the key snippets from the script:

Van telemetry: GPS coordinates with engine metrics and route information:

spark.sql(f"""
INSERT INTO {TABLE} VALUES (
'EVT-001', 'VAN-042', 'VAN',
CAST('2026-07-28 09:15:30.123456789' AS TIMESTAMP_NTZ(9)),
ST_SetSrid(ST_GeomFromWKB(X'0101000000E17A14AE47E1C0BF1F85EB51B84E4940'), 4326),
ST_SetSrid(ST_GeogFromWKB(X'0101000000E17A14AE47E1C0BF1F85EB51B84E4940'), 4326),
PARSE_JSON('{{"fuel_pct": 0.72, "cargo_kg": 450, "door_open": false,
"engine": {{"rpm": 2100, "temp_c": 88.5}},
"route": {{"stops_remaining": 4, "eta_minutes": 35}}}}'),
35.2, 'EMEA'
)
""")

Delivery robot telemetry: Same table, completely different sensor schema (battery, cameras, navigation):

spark.sql(f"""
INSERT INTO {TABLE} VALUES (
'EVT-002', 'ROB-117', 'ROBOT',
CAST('2026-07-28 09:15:30.123456790' AS TIMESTAMP_NTZ(9)),
ST_SetSrid(ST_GeomFromWKB(X'01010000000000000000001040000000000000F03F'), 4326),
ST_SetSrid(ST_GeogFromWKB(X'01010000000000000000001040000000000000F03F'), 4326),
PARSE_JSON('{{"battery_pct": 0.62, "obstacle_distance_m": 2.8,
"navigation_mode": "autonomous",
"cameras": {{"front": "active", "rear": "recording"}}}}'),
48.0, 'EMEA'
)
""")

Note: EVT-001 and EVT-002 are exactly 1 nanosecond apart (.123456789 vs .123456790). Without TIMESTAMP_NTZ(9), both would round to the same microsecond and be indistinguishable.

Default values test: Event inserted with vehicle_type, speed_kmh, and region omitted:

spark.sql(f"""
INSERT INTO {TABLE}
(event_id, vehicle_id, event_time, location, service_area, sensor_payload)
VALUES (
'EVT-004', 'UNK-999',
CAST('2026-07-28 10:00:00.000000000' AS TIMESTAMP_NTZ(9)),
ST_SetSrid(ST_GeomFromWKB(X'0101000000000000000000F03F000000000000F03F'), 4326),
ST_SetSrid(ST_GeogFromWKB(X'0101000000000000000000F03F000000000000F03F'), 4326),
PARSE_JSON('{{"status": "initializing"}}')
)
""")

The omitted columns automatically receive their DEFAULT values: vehicle_type = 'UNKNOWN', speed_kmh = 0.0, region = 'EMEA'.

Job 2: Query the data

The query job demonstrates all four data types working together. After the job succeeds, select the run in the AWS Glue console and choose Output logs to see the results.

The following sections walk through the key queries from the job and the results of each.

Geofence detection with ST_Intersects

The job defines a polygon and finds all vehicles inside it:

POLY = "010300...."
SELECT event_id, vehicle_id, vehicle_type, speed_kmh
FROM fleet_monitoring_db.vehicle_telemetry
WHERE ST_Intersects(
location,ST_SetSrid(ST_GeomFromWKB(X'{POLY}'), 4326)
)
ORDER BY event_id

The polygon covers coordinates (0,0)-(5,0)-(5,2)-(0,2). Three vehicles are inside (ROBOT at (4,1), BIKE at (3,1), UNKNOWN at (1,1)). The VAN at (-0.1278, 51.5074) is outside.

Figure 2: Geofence query results showing the three vehicles inside the polygon

Nanosecond event sequencing

Order events by their sub-microsecond timestamps:

SELECT event_id, vehicle_id, CAST(event_time AS STRING) AS precise_time
FROM fleet_monitoring_db.vehicle_telemetry
WHERE event_id IN ('EVT-001', 'EVT-002', 'EVT-003')
ORDER BY event_time ASC

EVT-001 and EVT-002 are correctly distinguished and ordered despite being only 1 nanosecond apart. With standard TIMESTAMP_NTZ (microsecond precision), both would show .123456 and their relative order would be undefined.

Figure 3: Nanosecond-precision ordering distinguishing two events one nanosecond apart

Variant extraction with variant_get

Different sensor schemas per vehicle type, all extracted with variant_get:

SELECT vehicle_id, vehicle_type,
CASE vehicle_type
WHEN 'VAN' THEN variant_get(sensor_payload, '$.fuel_pct', 'DOUBLE')
WHEN 'ROBOT' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
WHEN 'BIKE' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
ELSE NULL
END AS energy_level,
variant_get(sensor_payload, '$.engine.temp_c', 'DOUBLE') AS engine_temp,
variant_get(sensor_payload, '$.cameras.front', 'STRING') AS front_cam,
variant_get(sensor_payload, '$.deliveries.completed', 'INT') AS deliveries_done
FROM fleet_monitoring_db.vehicle_telemetry
WHERE vehicle_type != 'UNKNOWN'
ORDER BY vehicle_id

Figure 4: Variant extraction returning typed values from heterogeneous sensor payloads

variant_get takes three arguments: the column, a dot-path expression, and the expected return type. It supports arbitrary nesting depth. $.engine.temp_c reaches two levels deep, $.deliveries.completed reaches into a different structure entirely. When a path doesn’t exist in a particular row’s payload, it returns NULL.

Default values

Confirm that omitted columns received their defaults:

SELECT event_id, vehicle_type, speed_kmh, region
FROM fleet_monitoring_db.vehicle_telemetry
WHERE event_id = 'EVT-004'

Figure 5: Default column values applied to the event inserted with omitted fields

EVT-004 was inserted without vehicle_type, speed_kmh, or region. The declared defaults were applied automatically.

Combined query: Combining spatial, temporal, and variant operations

The following query runs a geospatial predicate, nanosecond ordering, and variant extraction in a single SELECT statement:

SELECT vehicle_id, vehicle_type,
CAST(event_time AS STRING) AS precise_time,
CASE vehicle_type
WHEN 'VAN' THEN variant_get(sensor_payload, '$.fuel_pct', 'DOUBLE')
WHEN 'ROBOT' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
WHEN 'BIKE' THEN variant_get(sensor_payload, '$.battery_pct', 'DOUBLE')
ELSE NULL
END AS energy_level,
speed_kmh
FROM fleet_monitoring_db.vehicle_telemetry
WHERE ST_Intersects(location, ST_SetSrid(ST_GeomFromWKB(X'0103000000...'), 4326))
ORDER BY event_time ASC

Figure 6: Combined query results over a single Iceberg v3 table

This single query combines a spatial predicate, nanosecond ordering, and variant extraction over one table, with no external libraries, pre-processing, or joins to separate geometry or payload tables.

Clean up

To avoid ongoing charges from the AWS Glue jobs and Amazon S3 storage, delete the CloudFormation stack when you’re done:

Open the AWS CloudFormation console.
Select the stack you deployed earlier and choose Delete.

Conclusion

In this post, we stored and analyzed geospatial coordinates, nanosecond timestamps, and heterogeneous sensor payloads in a single Iceberg v3 table on AWS Glue 6.0, with sensible defaults applied automatically, no external libraries, and no schema flattening.

GEOMETRY columns replace latitude/longitude doubles and support native spatial predicates like ST_Intersects for geofence detection. GEOGRAPHY is stored natively.
TIMESTAMP_NTZ(9) preserves full nanosecond precision for event sequencing where microsecond resolution is insufficient.
VARIANT stores heterogeneous payloads (different schema per vehicle type) in one column with typed extraction through variant_get.
DEFAULT values keep field population consistent across multiple ingestion writers without duplicating logic.

All capabilities require Iceberg format-version 3. Geospatial requires one additional configuration (spark.sql.geospatial.enabled=true). Nanosecond timestamps, Variant, and DEFAULT values work with no extra configuration.

These capabilities apply wherever schemas vary by source (IoT fleets, multi-tenant software as a service (SaaS), event-driven architectures), timestamps need sub-microsecond precision (trading, sensor fusion, autonomous systems), or spatial operations replace coordinate workarounds (logistics, real estate, delivery networks).

For more information, see the AWS launch announcement, the AWS Glue documentation, and the Apache Iceberg v3 specification. AWS Glue 6.0 includes additional capabilities such as Spark Real-Time Mode and Spark Declarative Pipelines, which we cover in separate posts.

About the authors

Migrate an OAuth 2.0 authenticated Apache Kafka cluster to Amazon MSK with MSK Replicator

Subham Rakshit — Wed, 26 Aug 2026 19:11:40 +0000

In an earlier post, we walked through how Amazon Managed Streaming for Apache Kafka (Amazon MSK) Replicator migrates external and self-managed Apache Kafka clusters to Amazon MSK. It replicates your topics and their configurations, keeps topic and consumer-group names intact, and synchronizes consumer-group offsets, so your producers and consumers can cut over on their own schedule instead of all at once. MSK Replicator now supports OAuth 2.0 (SASL/OAUTHBEARER) authentication to the external cluster, and that is what this post covers.

If your external Kafka cluster authenticates clients with OAuth, MSK Replicator can connect to it, but “OAuth” isn’t a single thing you switch on. It’s a family of grant types, and each one comes with its own trust model, its own set of inputs you need to supply, and its own configuration on both the Replicator side and your identity provider (IdP) side.

In this post, we walk you through the grant types one by one, show you how to configure Replicator for each, call out the network and TLS prerequisites that are commonly missed, and finish with how to handle IdPs that sit behind an additional identity layer. This mechanism works with any OAuth 2.0 (OIDC) identity provider, including Keycloak, Okta, Microsoft Entra ID, PingFederate, and Auth0. OAuth here governs only how Replicator authenticates to your external cluster, so the target can be either Amazon MSK Standard or Express brokers, which always use IAM.

How OAuth authentication works

Before you configure Replicator, it helps to be precise about how the OAuth Kafka handshake works.

The components

The Identity Provider (IdP) – Issues access tokens and publishes the public keys. Brokers use these keys to verify the tokens. Examples: Keycloak, Okta, Microsoft Entra ID, PingFederate, Auth0, or a custom OIDC server.
The client – In our case, MSK Replicator, acting as a Kafka consumer/producer against your external cluster.
The resource server – Your self-managed Kafka broker, which must decide whether to admit a connection.
The access token – A JWT (JSON Web Token): a base64url-encoded, three-part string header.payload.signature that the IdP cryptographically signs.

The SASL/OAUTHBEARER handshake, step by step

The following sequence diagram shows the full exchange, from Replicator requesting a token to the broker accepting the connection:

Figure 1: The SASL/OAUTHBEARER handshake. Replicator gets a signed JWT from the IdP and presents it to the broker, which verifies it against cached JWKS keys before accepting the connection.

Walking through it:

Request a token – Replicator asks the IdP for an access token. The exact request depends on the grant type (covered in the next section).
Receive a signed JWT – The IdP returns a signed JWT access token.
Present the token – Replicator opens a SASL/OAUTHBEARER connection to the external Kafka brokers and presents the JWT.
Verify locally – The broker verifies the JWT signature against the IdP’s cached JWKS public keys, without calling the IdP per message.
Connection accepted – The broker admits the connection and derives the Kafka principal from the preferred_username claim.

Step 4 is worth dwelling on: the broker validates the token locally. It fetches the IdP’s JWKS (JSON Web Key Set, the public half of the IdP’s signing keys, RFC 7517) from an endpoint like https://idp.example.com/realms/kafka/protocol/openid-connect/certs and caches it, refreshing on a configurable interval (and re-fetching if it sees a key ID it doesn’t recognize). Incoming JWT signatures are then verified against those cached keys. The IdP is not in the hot path of message traffic. It is contacted only to (a) issue tokens to clients, and (b) serve its public keys for the periodic JWKS refresh.

What the Kafka broker checks

When Replicator presents a JWT, the broker validates:

Signature – Proves the IdP issued the token and no one tampered with it (verified against JWKS).
iss (issuer) – Must match the broker’s configured oauth.valid.issuer.uri, byte-for-byte, including scheme, host, port, and path. A mismatch is a common configuration error.
exp (expiry) – Expired tokens are rejected. Strimzi’s client callback handler proactively refreshes before expiry, so you shouldn’t see mid-stream failures.
The principal claim – Typically preferred_username. The broker uses this as the Kafka principal in ACLs (for example, User:service-account-msk-replicator). This matters: the identity Replicator authenticates as on the external cluster must have ACLs that you configure to grant it the read/describe permissions it needs.

Mapping your IdP to a Replicator grant type

A grant type is the protocol by which the client proves its identity to the IdP and obtains a token. This is the front half of the preceding handshake (steps 1 and 2). MSK Replicator supports three of them. You already know how your Kafka clients authenticate to your IdP today, so start from that.

Which grant to use?

Find the row that matches how your clients get tokens today:

How your Kafka clients get tokens from the IdP today	Grant type	Long-lived secret?	What you trust/register on the IdP
A `client_id` / `client_secret` (confidential client)	CLIENT_CREDENTIALS	Yes (stored on AWS Secrets Manager)	Nothing new: reuse the existing client, or create one for Replicator
You want secretless, and your IdP can trust an external token issuer	IAM_JWT_BEARER	No	AWS STS as an external token (OIDC) issuer. Trust its JWKS
You want secretless, and your IdP models workloads as signed-JWT clients	CLIENT_CREDENTIALS_ASSERTION	No	AWS STS as the client’s signing authority (`private_key_jwt`). Trust its JWKS

The simplest mapping is like-for-like: if your clients use a client_id/client_secret, point Replicator at the same client with CLIENT_CREDENTIALS. If you’d rather not give Replicator a long-lived secret, the two secretless grants let it authenticate with its AWS identity instead. Choose between them based on how your IdP prefers to trust an external party.

The rest of this section explains why the three grants differ, using an analogy. If your row is clear and you only want the configuration, skip ahead to Configuring and creating the replicator.

A scenario: checking in at a secure office building

A visitor needs to get into a secure office building. They can’t walk straight in. First they stop at the reception desk to prove who they are and collect a temporary access pass. Only then can they use that pass at the building’s turnstile to get inside. In OAuth terms: the building is your external Kafka cluster, the reception desk is the IdP, the temporary access pass is the access token (JWT), and the visitor is MSK Replicator. Presenting the pass at the turnstile is the SASL/OAUTHBEARER step, and it works the same way for every grant type. What differs is how the visitor proves who they are at the reception desk before it prints a pass.

Scenario 1: CLIENT_CREDENTIALS (the shared PIN)

Figure 2: CLIENT_CREDENTIALS. The visitor authenticates at reception with a PIN (the client_secret), gets a temporary badge (the access token), and uses it to enter the building (the Kafka cluster).

At the reception desk the visitor keys in a PIN the desk already have on file (the client_secret), collects a temporary access pass in return (the access token), and uses that pass to get into the building. Both sides hold the same secret. In practice (RFC 6749 §4.4), Replicator authenticates to the IdP with a client_id/client_secret stored on AWS Secrets Manager, receives the access token, and presents it to the external Kafka brokers over SASL/OAUTHBEARER. Use it when your IdP already issues client secrets for machine clients. This is usually a like-for-like move that reuses the client your existing producers and consumers use, or a new one created for Replicator.

Scenario 2: IAM_JWT_BEARER (the badge is the request)

Figure 3: IAM_JWT_BEARER. The visitor shows an employer-signed badge (an STS JWT) to reception as the request itself and gets an access token. Reception accepts it because it trusts the employer’s stamp (the STS JWKS).

First, the visitor collects an employer-signed badge: Replicator calls STS GetWebIdentityToken to mint an STS JWT. At the reception desk the badge itself is the request. The visitor shows it to ask for a pass. Reception trusts the employer’s tamper-proof stamp (STS JWKS), so it accepts the badge and prints a temporary access pass. In practice (RFC 7523 §2.1), the STS JWT is sent as the authorization grant (assertion), and the IdP trusts AWS STS as an external token issuer. Use it when you want secretless authentication, and your IdP can trust an external issuer’s JWTs.

Scenario 3: CLIENT_CREDENTIALS_ASSERTION (the same badge, used as ID on the form)

Figure 4: CLIENT_CREDENTIALS_ASSERTION. The visitor fills out reception’s standard request form and attaches the same STS JWT as ID, getting an access token. Reception trusts the employer’s stamp (the STS JWKS).

The visitor again collects the same employer-signed badge (STS JWT). This time they fill out the reception desk’s standard access request form (the client_credentials grant) and attach the badge to it as identification, all in one submission. Reception trusts the same employer stamp (STS JWKS) and prints a temporary access pass. In practice (RFC 7521/RFC 7523 §2.2), the same STS JWT is sent as the client_assertion on the client_credentials grant, with the IdP trusting STS as the client’s signing authority (private_key_jwt). Use it when you want secretless authentication and your IdP models external workloads as signed-JWT clients.

Scenarios 2 and 3 in one sentence. Both mint the same STS JWT and share the same benefit: nothing shared can leak, because there is no secret. They differ only in where the STS JWT sits in the token request. IAM_JWT_BEARER sends it as the assertion (the badge is the request), while CLIENT_CREDENTIALS_ASSERTION sends it as the client_assertion on a standard client_credentials request (the badge is ID on the form). That single difference is what you register on the IdP: AWS STS as an external token issuer, or as the client’s signing authority.

Solution overview

Now that you can map your setup to a grant type, the next question is where these pieces actually run. MSK Replicator runs on AWS managed infrastructure but attaches elastic network interfaces (ENIs) into the subnets of the target Amazon MSK cluster’s virtual private cloud (VPC) and initiates every connection from there under a Service Execution Role (SER). Those ENIs sit in private subnets that typically have no NAT or internet gateway, so each external dependency needs an explicit network path. The following diagram shows the full topology for an OAuth migration, including the two pieces that are commonly missed: STS Outbound Web Identity Federation (for the secretless grants) and the interface VPC endpoints for STS and Secrets Manager.

Figure 5: Deployment architecture. The source environment holds the IdP and Kafka brokers. The AWS account holds STS, Secrets Manager, and the Amazon MSK VPC, whose private subnets contain the Replicator ENIs and target cluster, reached through interface VPC endpoints.

The source environment (on the left, shown as on-premises here, but it can equally be another cloud or a self-managed cluster on AWS) holds two components: the IdP token endpoint and JWKS (Keycloak, Okta, Entra ID) and the external Kafka brokers on a SASL_SSL / OAUTHBEARER listener. Everything else runs in your AWS account.

The two dotted lines are trust relationships you configure ahead of time, not runtime calls:

External Kafka validates token by using IdP JWKS – The broker checks every presented access token against the IdP’s published public keys. This applies to all grants.
IdP trusts STS issuer through JWKS – For the secretless grants only, the IdP is configured to trust your account’s STS issuer and validate the STS-signed JWT against STS’s JWKS. When STS Outbound Web Identity Federation is enabled, AWS provisions a per-account issuer URL (https://<id>.tokens.sts.global.api.aws) whose JWKS the IdP trusts. This trust is not used by CLIENT_CREDENTIALS.

The numbered arrows are the runtime flow, all originating from the Replicator ENIs:

Step 1: Fetch client credentials and the CA certificate from AWS Secrets Manager, through its VPC endpoint. For CLIENT_CREDENTIALS this includes the client_id/client_secret. For the secretless grants it is only the CA certificate(s).
Step 1a (optional): Call GetWebIdentityToken on AWS STS, through the STS VPC endpoint, to mint a JWT of Replicator’s AWS identity. Required only for IAM_JWT_BEARER and CLIENT_CREDENTIALS_ASSERTION.
Step 2: Get a signed JWT access token from the IdP token endpoint, exchanging either the client secret or the STS JWT depending on the grant.
Step 3: Present the token to the external Kafka brokers over SASL/OAUTHBEARER.
Step 4: Replicate to the target Amazon MSK cluster using IAM authentication.

The two supporting pieces inside the VPC, the Secrets Manager and STS interface VPC endpoints, are commonly overlooked precisely because the private subnets have no NAT or internet gateway. We cover exactly why they’re needed, and when, in the following section, Cross-cutting requirements.

Configuring and creating the replicator

With the architecture in mind, you can now configure Replicator itself. MSK Replicator models OAuth through a saslOAuthBearer structure on the external cluster’s clientAuthentication. Exactly one of three mechanism members must be present: clientCredentials, iamJwtBearer, or clientCredentialsAssertion. The control plane enforces this mutual exclusivity. Fields shared across all three (tokenEndpointUrl, scope, tokenEndpointAuthenticationMethod, tokenEndpointTlsCertificateArn, and saslExtensions) live at the saslOAuthBearer level.

Before the per-grant details, here are the requirements that apply to every OAuth migration, whichever grant you choose. Most OAuth setup failures trace back to one of these, so review them first.

Cross-cutting requirements

Here are the five items that apply to every grant: TLS trust, secret format, network reachability, the Service Execution Role, and STS federation.

a) TLS everywhere, and two separate trust settings

Replicator connects to two TLS endpoints, and they are configured independently:

encryptionInTransit.rootCaCertificate: the CA that signed your Kafka brokers’ TLS certificates (the SASL_SSL listener – :9096).
tokenEndpointTlsCertificateArn: the CA that signed your IdP’s token endpoint TLS certificate (for example – Keycloak on :8443).

If your broker and IdP are signed by the same private CA, you still must supply the CA in both fields. Omitting tokenEndpointTlsCertificateArn when the IdP uses a private or self-signed cert produces a PKIX path building failed error during token acquisition. Because that fails before workers stabilize, you’ll see a generic failure with no worker logs. If your IdP uses a publicly-trusted certificate (for example, it sits behind a public endpoint), you can omit tokenEndpointTlsCertificateArn entirely.

b) Secret format: store key/value pairs, not raw values

Every secret Replicator reads (client credentials, CA certificate) is parsed by the config provider as a set of key/value pairs. Use the Secrets Manager console’s Key/value editor rather than pasting raw text, and it will serialize and escape the values for you.

The keys the provider expects:

Key	Value	Used for
`certificate`	the CA in PEM (newlines escaped as `\n`)	CA-certificate secrets (`rootCaCertificate`, `tokenEndpointTlsCertificateArn`)
`client_id`, `client_secret`	your OAuth client credentials	the `CLIENT_CREDENTIALS` token-request secret

Custom parameters, headers, and SASL extensions. Some IdPs require extra data on the token request, and some brokers require SASL/OAUTHBEARER extensions. The config provider supports both through reserved key prefixes in the same secret:

Prefix	Effect	Example key	Example value
`custom_param.`	adds a parameter to the token request sent to the IdP	`custom_param.tenant_token`	`myTenantToken`
`custom_header.`	adds an HTTP header to the IdP token request	`custom_header.X-Tenant-Id`	`acme`
`extension.`	adds a SASL/OAUTHBEARER extension presented to the broker (for example, Confluent Cloud’s `logicalCluster`)	`extension.logicalCluster`	`myLogicalClusterId`

For example, an IdP that expects a tenant token as a request parameter and a Confluent Cloud broker that requires a logical-cluster extension would add custom_param.tenant_token and extension.logicalCluster as extra key/value pairs alongside client_id/client_secret in the same secret.

c) Network reachability from Replicator’s ENIs

Replicator attaches ENIs into the subnets you specify (through the target amazonMskCluster cluster’s vpcConfig) and initiates all connections from there. Those ENIs must be able to reach:

Your external brokers, over VPC peering, AWS Transit Gateway, AWS Direct Connect, or VPN, with security groups permitting the SASL_SSL port.
Your IdP’s token endpoint, over the same networking. The endpoint hostname must resolve from those subnets.
AWS Secrets Manager, to fetch credentials/CA. If the subnets have no NAT/internet gateway, add an interface VPC endpoint for com.amazonaws.<region>.secretsmanager with private DNS.
AWS STS (only for IAM_JWT_BEARER and CLIENT_CREDENTIALS_ASSERTION), to call GetWebIdentityToken. In no-egress subnets this will time out (STS GetWebIdentityToken call failed: Connect timed out) unless you add an interface VPC endpoint for com.amazonaws.<region>.sts with private DNS. This is the most common oversight for the secretless grants.

Both endpoints use private DNS, so the standard secretsmanager.<region>.amazonaws.com and sts.<region>.amazonaws.com hostnames resolve to the endpoint inside the VPC, with no client change needed.

A note on vpcConfig placement. For an external Apache Kafka cluster, vpcConfig is specified on the target amazonMskCluster entry, not the external apacheKafkaCluster entry. The API rejects a vpcConfig on the external cluster. The ENIs it creates are what reach both clusters and all AWS endpoints.

d) The Service Execution Role (SER)

Replicator assumes an IAM role to do its work. Two parts matter:

Trust policy – Must allow the Replicator service to assume it. kafka.amazonaws.com needs to be trusted. A trust policy that is too narrow fails with AccessDenied.ServiceExecutionRoleUnassumable.
Permissions – The replication permissions are extensive and depend on which features you enable, so follow the service execution role permissions reference to build a least-privilege policy.

e) Enabling STS Outbound Web Identity Federation (secretless grants only)

For IAM_JWT_BEARER and CLIENT_CREDENTIALS_ASSERTION, sts:GetWebIdentityToken must be enabled for your account/role. When enabled, AWS provisions a dedicated issuer URL of the form https://<uuid>.tokens.sts.global.api.aws. Every JWT STS mints for your account carries this as its iss claim, and its public keys are published under this issuer’s JWKS. You configure your IdP to trust this issuer. Granting the sts:GetWebIdentityToken IAM action is necessary but not sufficient. The account-level federation feature must also be turned on.

Create the replicator

A repeatable way to create the replicator is with a request file and --cli-input-json, so you can keep the full configuration under version control. The following example is a complete CLIENT_CREDENTIALS request. The two secretless variants change only the saslOAuthBearer block (shown after).

aws kafka create-replicator \
  --region <region> \
  --cli-input-json file://create-replicator.json

create-replicator.json:

{
  "replicatorName": "oauth-migration-replicator",
  "serviceExecutionRoleArn": "arn:aws:iam::<acct>:role/msk-replicator-execution-role",
  "kafkaClusters": [
    {
      "apacheKafkaCluster": {
        "apacheKafkaClusterId": "<source-cluster-id>",
        "bootstrapBrokerString": "b-1.ext-kafka.example.com:9096,b-2.ext-kafka.example.com:9096"
      },
      "clientAuthentication": {
        "saslOAuthBearer": {
          "tokenEndpointUrl": "https://idp.example.com/realms/kafka/protocol/openid-connect/token",
          "clientCredentials": {
            "tokenRequestSecretArn": "arn:aws:secretsmanager:<region>:<acct>:secret:<oauth-creds>"
          },
          "tokenEndpointAuthenticationMethod": "POST",
          "tokenEndpointTlsCertificateArn": "arn:aws:secretsmanager:<region>:<acct>:secret:<idp-ca>"
        }
      },
      "encryptionInTransit": {
        "encryptionType": "TLS",
        "rootCaCertificate": "arn:aws:secretsmanager:<region>:<acct>:secret:<broker-ca>"
      }
    },
    {
      "amazonMskCluster": {
        "mskClusterArn": "arn:aws:kafka:<region>:<acct>:cluster/target-msk/<uuid>"
      },
      "vpcConfig": {
        "subnetIds": [
          "subnet-aaaa",
          "subnet-bbbb",
          "subnet-cccc"
        ],
        "securityGroupIds": [
          "sg-xxxxxxxx"
        ]
      }
    }
  ],
  "replicationInfoList": [
    {
      "sourceKafkaClusterId": "<source-cluster-id>",
      "targetKafkaClusterArn": "arn:aws:kafka:<region>:<acct>:cluster/target-msk/<uuid>",
      "targetCompressionType": "NONE",
      "topicReplication": {
        "topicsToReplicate": [
          ".*"
        ],
        "detectAndCopyNewTopics": true,
        "copyTopicConfigurations": true
      },
      "consumerGroupReplication": {
        "consumerGroupsToReplicate": [
          ".*"
        ],
        "detectAndCopyNewConsumerGroups": true,
        "synchroniseConsumerGroupOffsets": true
      }
    }
  ]
}

Field names and exact nesting follow the create-replicator API reference. Check it for the full schema and any Region-specific values.

The example above uses CLIENT_CREDENTIALS. For the full schema, any Region-specific values, and detailed examples for the other grant types, check the MSK documentation.

Recommended order of operations

With the requirements and configuration in hand, here is the order to put them in:

Pick your grant type using the preceding decision table. CLIENT_CREDENTIALS is the fastest path if you already manage a client secret. Otherwise choose a secretless grant based on how your IdP models external workloads. For a multi-hop internal chain, use IAM_JWT_BEARER against the proxy pattern described in the next section.
Prepare the IdP: create the client (or the STS-trust configuration), and note the exact token endpoint URL and issuer.
Stage secrets in Secrets Manager, as JSON (requirement b): client credentials (if any) and the CA certificate(s).
Wire the network (requirement c): connectivity from Replicator’s subnets to your brokers and IdP, plus interface VPC endpoints for Secrets Manager and (secretless grants only) STS, both with private DNS.
[Optional but recommended]: Smoke-test the path from inside the VPC – IdP setup is often the part that takes the most iterations, and Replicator provisioning is a slow way to discover a misconfigured token endpoint or a missing TLS trust. Spin up a small EC2 instance in Replicator’s subnets, install a Kafka client, and run an end-to-end produce/consume against the external brokers using SASL/OAUTHBEARER (a client_credentials flow is simplest). This validates the three things most likely to be wrong (network reachability to the IdP and brokers, both TLS trusts for the broker CA and IdP CA, and token vending) while you can still fix them in seconds. Tear the instance down once the round trip works.
Enable STS Outbound Web Identity Federation (requirement e. Secretless grants only) and configure your IdP to trust the resulting issuer.
Build the SER (requirement d) with a trust policy the Replicator service can assume and the required permissions.
Create the replicator with the create-replicator request for your grant. Remember both TLS trust fields for a private-CA IdP (requirement a), and vpcConfig on the target entry only.
Verify – Produce to a topic on the external cluster and confirm the records land on the target (consume with IAM auth on the Amazon MSK side). Then watch the health signals:
- In the Amazon MSK console, the replicator should reach the RUNNING state.
- In Amazon CloudWatch, under the AWS/Kafka namespace, watch the replicator’s ReplicationLatency and MessageLag metrics. Both should be low and stable, and MessageLag should trend toward zero as it catches up.
- A healthy replicator commits offsets continuously. A steady “1 message per batch” with no producer activity is only the internal heartbeat topic, not a stall.

Handling an additional identity layer: the federation-proxy pattern

Who owns what – Before the details, the ownership line is simple and worth stating up front:

What Replicator guarantees: it calls the configured tokenEndpointUrl with the configured grant, includes the STS JWT, expects a standard {access_token, token_type, expires_in} response, and refreshes before expiry.
What you own: everything at and behind the proxy, including validating the STS JWT, the downstream token exchanges, claim mapping, and the availability and latency of the endpoint. The proxy runs in your VPC and is owned entirely by you.

So far we have assumed you can point Replicator at a single token endpoint. Some organizations can’t. Instead, they have an internal identity chain: several hops of token exchange and federation that a workload must traverse before it holds a token the Kafka brokers accept.

A representative example is a large financial institution whose chain has several hops: an AWS workload’s identity (a signed GetCallerIdentity request) is exchanged at an internal Token Exchange service for an intermediate JWT, which an internal IdP then consumes as a client_assertion to issue the final Bearer token the Kafka brokers accept.

Replicator connects to a single HTTPS token endpoint using one of the three grant types and expects a standard token response. When the identity flow spans multiple hops like this, you place a proxy in front of that chain so Replicator still sees a single endpoint.

The solution: a customer-owned proxy

You deploy a small proxy in your own VPC that collapses the chain behind a single endpoint. From Replicator’s perspective, this is an ordinary OAuth flow against one token endpoint. Everything behind that endpoint is opaque to Replicator and owned entirely by you.

The grant Replicator uses to reach the proxy is a separate choice from the exchanges happening behind it. We recommend a secretless grant (IAM_JWT_BEARER or CLIENT_CREDENTIALS_ASSERTION) so there is no long-lived secret between Replicator and the proxy. CLIENT_CREDENTIALS is also valid if you would rather the proxy authenticate Replicator with a client secret. The following walkthrough uses IAM_JWT_BEARER, where the proxy validates the STS JWT that Replicator presents.

How it works, end to end. The following sequence diagram traces the full token exchange, from Replicator’s request to the Bearer it finally presents to the external Kafka brokers.

Figure 6: Federation-proxy token flow. The proxy validates Replicator’s STS JWT, exchanges its own AWS identity at the Token Exchange service for an intermediate JWT, presents that to the internal IdP, and returns the resulting Bearer to Replicator.

Replicator to proxy – Replicator POSTs its STS JWT as assertion to the proxy’s token endpoint, a plain IAM_JWT_BEARER request (grant_type=jwt-bearer). Because the endpoint is private, Replicator reaches it through an execute-api interface VPC endpoint, the same private-connectivity approach used for Secrets Manager and STS. (Replicator first obtains the STS JWT by calling STS GetWebIdentityToken through the STS VPC endpoint.)
Proxy validates the STS JWT (signature against STS’s JWKS, plus iss/aud/exp/sub checks. The sub is the caller’s AWS ARN).
Proxy to Token Exchange service – The proxy exchanges its own AWS identity, presented as a signed GetCallerIdentity request, at the internal Token Exchange service.
Token Exchange service → proxy – It returns a signed intermediate JWT.
Proxy to internal IdP – The proxy makes a client_credentials request that carries the intermediate JWT as the client_assertion.
Internal IdP to proxy – The IdP issues the final Bearer access token.
Proxy to Replicator – The proxy returns the Bearer, and Replicator presents it to the external brokers over SASL/OAUTHBEARER. The brokers validate it against the final IdP’s JWKS, a completely ordinary OAuth handshake from their point of view.

Reference architecture

Here is the reference architecture for the end-to-end solution.

Figure 7: Federation-proxy reference architecture. Replicator ENIs in a private subnet call a customer-owned proxy (a Lambda behind a private API Gateway), which runs the on-premises identity chain over Direct Connect before Replicator replicates into the target Amazon MSK cluster.

Everything on the Replicator side runs in your VPC’s private subnets: the Replicator ENIs, the customer-owned proxy, and the target Amazon MSK cluster. The proxy here is an AWS Lambda function behind a private Amazon API Gateway, but it can run on any compute you prefer (EC2, ECS, or EKS) as long as it exposes a single private HTTPS token endpoint. Connectivity to the on-premises Token Exchange service, internal IdP, and Kafka brokers runs over AWS Direct Connect (a VPN or VPC peering works too).

The outer legs of this flow are exactly the base migration from Solution overview: step 1 (fetch the broker CA from Secrets Manager), step 1a (mint the STS JWT through STS), step 3 (present the Bearer to the brokers), and step 4 (replicate to the target with IAM). What’s new here is the proxy hop in the middle, which replaces the single “step 2” call to a token endpoint:

2. POST /token – Replicator sends the STS JWT as the assertion to the proxy’s private token endpoint, reached through the execute-api interface VPC endpoint. The proxy validates it against STS’s JWKS.
2a. Exchange AWS identity – The proxy presents its own AWS identity (a signed GetCallerIdentity request) to the internal Token Exchange service and gets back a signed intermediate JWT.
2b. Present as client_assertion – The proxy sends a client_credentials request to the internal IdP with the intermediate JWT as the client_assertion, and receives the final Bearer.
2c. Final Bearer token – The proxy returns the Bearer to Replicator, which then continues at step 3.

As in the base architecture, the dotted lines are prerequisite trust relationships, not runtime calls: the proxy trusts AWS STS as an issuer (validating the STS JWT against STS’s JWKS), and the Kafka brokers validate the final Bearer against the internal IdP’s JWKS.

One subtlety worth calling out is the split of TLS trust. Replicator connects directly only to the private API Gateway (which uses a publicly trusted certificate) and to the Kafka brokers, so the only certificate it fetches from Secrets Manager is the broker CA. The internal IdP’s CA is the proxy’s concern: the proxy terminates TLS to the Token Exchange service and internal IdP, so it carries their CA material, not Replicator.

The same single-endpoint pattern handles other “extra layer” scenarios without any Replicator change: claim enrichment (the proxy intercepts and augments), rate-limited IdPs (the proxy caches tokens), IdPs requiring mTLS (the proxy terminates Replicator’s HTTPS and initiates mTLS onward), and IdP migrations (swap the proxy’s target without touching Replicator config).

A working reference implementation of this customer-owned proxy is available at GitHub.

Conclusion

In this post, we walked through how to migrate a self-managed, OAuth-authenticated Apache Kafka cluster to Amazon MSK using MSK Replicator: how the SASL/OAUTHBEARER handshake works, how to map your identity provider to one of the three supported grant types, the deployment architecture and prerequisites that the connection depends on, and how to handle identity providers that sit behind an additional federation layer. To get started, see the Amazon MSK Developer Guide and the Amazon MSK Replicator documentation. For the federation-proxy example, see the sample implementation on GitHub.

About the author

Announcing in-place ZooKeeper-to-KRaft cluster upgrades for Amazon MSK

Austin Groeneveld — Wed, 26 Aug 2026 18:59:47 +0000

Apache Kafka 4.0 officially removes ZooKeeper. If your Amazon Managed Streaming for Apache Kafka (Amazon MSK) Provisioned clusters still run in ZooKeeper metadata mode, now is the time to plan your migration. Amazon MSK now supports in-place upgrades from ZooKeeper to KRaft metadata mode, so you can modernize your existing cluster’s metadata management through the familiar version upgrade workflow.

For more than a decade, Apache ZooKeeper provided dependable metadata management for Kafka, including controller election, partition state, broker registration, and topic configuration. With Apache Kafka 4.0, ZooKeeper is officially removed in favor of KRaft, an embedded Raft-based consensus protocol that handles metadata management internally. It brings those responsibilities into Apache Kafka itself, creating a more streamlined foundation for the continued evolution of Kafka. Amazon MSK has supported KRaft-mode clusters since May 2024, and all Kafka 4.x versions on Amazon MSK use KRaft.

With the in-place upgrade, you can retain your cluster data and metadata while Amazon MSK manages the control-plane transition. Your cluster remains available for produce and consume traffic throughout the process, with no expected downtime if you’re following best practices. By using the existing version upgrade workflow, the move to KRaft becomes a natural step in your cluster’s lifecycle. This prepares your cluster for Kafka 4.x and future Kafka releases.

Prerequisites

Before initiating the upgrade, review the following requirements to confirm your cluster is ready for the transition.

Supported source versions

Clusters must be running Kafka 3.9.x in ZooKeeper mode to use the in-place upgrade. If your cluster is running an earlier version, such as 3.6.0, 3.7.x, or 3.8.x, first complete a standard in-place version upgrade to 3.9.x. You can then initiate the upgrade to 3.9.x.kraft.

Kafka 3.9 is the bridge release for this transition because it supports both ZooKeeper and KRaft modes. To support customers through this migration process, Amazon MSK provides extended support for 3.9.x for a minimum of 2 years from its April 2025 release.

Client compatibility

Requirement	Detail
Minimum client library	Apache Kafka client v3.0+
Recommended client version	v3.9 or above
Connection strings	Must use `bootstrap.servers` only. Any ZooKeeper connection strings (the `--zookeeper` flag) must be removed before upgrade.

The --zookeeper admin flag was deprecated in Kafka 2.5 and removed in 3.0. Before upgrading, update any remaining applications or tools that connect directly to ZooKeeper.

Pre-upgrade checklist

Before beginning the upgrade, confirm the following:

For Standard brokers, the cluster must be deployed across three Availability Zones. Express brokers provide this by default.
The cluster is running Kafka 3.9.x in ZooKeeper mode.
Standard brokers expose direct ZooKeeper access on ports 2181 (plaintext) and 2182 (TLS). Before upgrading, validate that you’ve disabled ZooKeeper access on the cluster and none of your applications rely on these connections.
Solutions using dynamic Kafka configurations that relied on ZooKeeper have been removed before attempting the upgrade operation.
- If you previously configured custom domain names on a ZooKeeper-based deployment using the dynamic override (kafka-configs.sh --alter on advertised.listeners), be aware that KRaft does not support this dynamic configuration. If you attempt to upgrade your MSK cluster to KRaft with altered advertised.listeners, the upgrade operation fails.
- If you’re implementing your custom domain name solution on MSK moving forward with KRaft, we recommend our coinciding MSK release for custom domain name support by statically configuring the custom.advertised.listeners property through the UpdateClusterConfiguration API.
The cluster has no under-replicated partitions.
The cluster is running within per-broker partition limits for standard or express broker clusters.
For clusters running above the KRaft brokers-per-cluster limit, you might need an additional quota increase. If you previously raised a quota increase for your ZooKeeper brokers-per-cluster, submit another quota increase for the KRaft limit before attempting the upgrade.
The cluster has enough reserve capacity to support rolling broker restarts while serving client traffic.
As a best practice, verify that monitoring is ready for the transition from ZooKeeper-specific metrics to KRaft controller metrics.
- After the migration, ZooKeeper-specific Amazon CloudWatch metrics such as ZookeeperRequestLatencyMsMean and ZookeeperSessionState are no longer available.
- If you use Open Monitoring, Kafka also stops publishing ZooKeeper metrics. Plan to update or retire related alerts and dashboards as part of your migration preparation.

How the upgrade works

When you initiate the upgrade, Amazon MSK performs a managed, multi-phase migration:

Controller quorum bootstrap: Amazon MSK provisions KRaft controller nodes alongside the existing ZooKeeper infrastructure. Both systems operate in parallel during this phase.
Metadata migration: The KRaft controller reads the cluster state from ZooKeeper and writes it to the internal KRaft metadata log.
Broker transition: Amazon MSK performs a rolling update and registers with the KRaft controller quorum. Data plane operations remain available during the transition.
Validation and bake period: Amazon MSK verifies cluster health under KRaft, including partition leadership, replication state, and controller responsiveness.
ZooKeeper decommissioning: After validation succeeds, Amazon MSK removes the ZooKeeper infrastructure and the cluster operates entirely in KRaft mode.

During the upgrade, the cluster enters UPDATING state. You can continue producing and consuming data, while Amazon MSK administrative API operations are temporarily unavailable until the cluster returns to ACTIVE.

Amazon MSK maintains a high bar for durability during the transition. It uses rigorous safety checks at each phase of the migration to protect customer metadata in both roll-forward and rollback scenarios.

Built-in recovery

Amazon MSK monitors cluster health throughout the upgrade. If it detects a condition that prevents the migration from completing, it automatically returns the cluster to its pre-migration state. No customer action is required during recovery.

The operation status changes to Reverting to pre-migration state while Amazon MSK restores the original Kafka version and reconnects ZooKeeper. After the cluster returns to ACTIVE, the describe-cluster-operation API provides error codes, failure reasons, and recommended remediation steps. You can use these to address the issue before starting the upgrade again.

How to perform the upgrade

The following steps walk you through the upgrade process using the Amazon MSK console. You can also perform these steps programmatically using the AWS Command Line Interface (AWS CLI) or SDK.

Step 1: Disable ZooKeeper access (standard brokers only)

Note: This step applies only to Standard broker clusters. Express broker clusters don’t expose direct ZooKeeper access and can skip directly to Step 2.

Standard brokers expose direct ZooKeeper access on ports 2181 (plaintext) and 2182 (TLS). Before upgrading, validate that none of your applications rely on these connections.

Navigate to your cluster’s Properties tab, choose Network settings, and then choose Edit ZooKeeper access.

Figure 1: Editing ZooKeeper access from the cluster network settings

In the pop-up window, verify that ZooKeeper access is set to Disabled, and then choose Save.

Figure 2: Confirming ZooKeeper access is disabled

Confirm that producers, consumers, and admin tooling continue operating normally without ZooKeeper connectivity. This step is fully reversible. Re-enable ZooKeeper access immediately if anything breaks.

Figure 3: Verifying client traffic continues without ZooKeeper access

Step 2: Initiate the version upgrade

In the Amazon MSK console, under Properties, choose Upgrade in the Apache Kafka version section.

Figure 4: Starting a version upgrade from the Apache Kafka version section

Select your cluster and start a version upgrade to 3.9.x with Target metadata mode set to KRaft. Choose Upgrade.

Figure 5: Selecting KRaft as the target metadata mode

You can monitor your upgrade progress on the cluster properties page.

Figure 6: Monitoring upgrade progress on the cluster properties page

Step 3: Monitor upgrade progress

Track progress on the Cluster operations tab in the Amazon MSK console or with the describe-cluster-operation API.

Figure 7: Tracking the upgrade on the Cluster operations tab

Step 4: Validate the KRaft cluster

After the cluster returns to ACTIVE state in KRaft mode:

Verify that topics, partitions, and consumer groups are present.
Confirm producer and consumer throughput aligns with pre-migration baselines.
Update or disable any ZooKeeper-specific monitoring alerts.
Update operational documentation and runbooks to reflect KRaft mode.

Figure 8: Cluster running in KRaft mode after the upgrade

After the upgrade completes, your cluster appears in an Active state with KRaft enabled as the metadata mode.

Get ready for the next generation of Kafka on Amazon MSK

The in-place ZooKeeper-to-KRaft mode upgrade makes it straightforward to prepare existing Amazon MSK clusters for the future of Apache Kafka. Beyond removing external metadata dependencies, KRaft delivers faster failover times and higher partition limits per cluster. Amazon MSK handles the entire metadata transition, rolling broker updates, validation, and recovery workflow for you. With the new in-place experience, you have a clear, streamlined path to upgrade on your schedule and unlock enhanced scalability and resilience.

For more details, see the Amazon MSK Developer Guide and the supported Kafka versions documentation.

About the authors

Amazon MSK Service 101: How many partitions does an Amazon MSK topic need?

Yashika Jain — Wed, 26 Aug 2026 15:56:42 +0000

Customers new to Amazon Managed Streaming for Apache Kafka (Amazon MSK) often ask how many partitions their topics need. Choosing the right partition count is one of the most impactful architectural decisions you make, because it directly affects throughput, scalability, and operational complexity.

In Apache Kafka, a topic is the fundamental unit for categorizing data streams, but to achieve high scalability and performance, Kafka divides topics into smaller, independent units called partitions.

In this post, we provide practical guidance for determining the ideal partition count for your use case.

Understanding Kafka partitions

In Apache Kafka, a partition is the unit of storage and parallelism. Each partition is an ordered, immutable log that can store records as they are produced to a topic. When you create a topic, Kafka distributes its partitions across the brokers in the cluster. Partitions allow Kafka to scale in three key ways:

Parallelism – Within a consumer group, each partition can be read by only one consumer at a time. Each partition maps to a dedicated log file in storage on the broker, and Kafka manages these logs through separate processing threads. This architecture allows more partitions to support more consumers processing data in parallel, with each partition’s log being independently managed for read and write operations.

The following diagram shows how Kafka distributes partition replicas across a three-broker cluster, with each broker serving as a leader for some partitions and a follower for others.

Figure 1: Partition replicas distributed across a three-broker cluster

The following diagram illustrates how producers append new records to the end of a partition log, while consumers read sequentially from their current offset position.

Figure 2: Producer writes and consumer offset positions in two partition logs

Throughput – Producers and consumers can read and write data in parallel across partitions, increasing overall throughput.
Scalability – Partitions allow Kafka to spread data and load across multiple brokers instead of concentrating it on a single node.

However, increasing partitions comes with trade-offs. Each partition adds metadata overhead, consumes memory, and requires file handles on the broker. While more partitions improve throughput and parallelism, they also increase the operational burden on the cluster. Too many partitions can lead to longer leader election times during broker failures, increased end-to-end latency, and higher memory consumption for both producers and consumers managing connections to multiple partitions.

Trade-offs when choosing partition count

Choosing a partition count is a balancing act between parallelism and resource utilization.

Benefits of more partitions

Using more partitions can significantly improve throughput by allowing Kafka to distribute read and write traffic across more brokers. This is particularly useful for high-volume ingestion pipelines and real-time analytics workloads. More partitions also allow consumer groups to scale horizontally, because the maximum number of active consumers in a group is limited by the number of partitions. In addition, choosing a partition count that is evenly divisible by the number of brokers helps provide balanced leadership and replica distribution, reducing the risk of uneven load.

Operational costs of more partitions

However, higher partition counts also come with costs. When a broker fails or undergoes maintenance, Kafka must perform recovery operations for each affected partition. During recovery, Kafka elects new leaders for partitions that were hosted on the unavailable broker and replicates data from the remaining in-sync replicas to newly assigned brokers. This process involves copying partition data across the network to restore the replication factor, which can be resource intensive. As the number of partitions increases, these recovery operations take longer because each partition requires its own leader election and data replication cycle.

You might encounter clusters with very high partition counts that experience extended recovery times during rolling upgrades, even when overall traffic volumes are modest. Amazon MSK Express brokers address this challenge by recovering 90x faster and providing 180x faster elasticity when scaling out clusters. This significantly reduces the operational impact of high partition counts during maintenance windows and failure scenarios.

Infrastructure cost implications

Beyond operational complexity, more partitions can directly increase infrastructure costs. Amazon MSK publishes partition-per-broker limits that vary by instance type. When the total partition count (including replicas) exceeds what the current broker fleet can support, you must add brokers to stay within recommended limits, even if throughput alone does not warrant the additional capacity.

Amazon MSK partition-per-broker guidelines

Amazon MSK publishes recommended partition-per-broker guidelines to help you operate clusters reliably. These values are strict limits. Exceeding them can lead to operational challenges, particularly during broker replacement or rolling upgrades, and can block cluster operations such as configuration updates or scaling down.

Express brokers support up to 5x more partitions per broker compared to Standard brokers. For example, the largest Standard broker (kafka.m7g.16xlarge) supports a recommended maximum of 4,000 partitions per broker. The equivalent Express broker (express.m7g.16xlarge) supports up to 20,000 recommended partitions per broker. This higher partition density means partition-bound workloads can be hosted on fewer brokers, improving price-performance by up to 50% for such workloads.

We recommend setting Amazon CloudWatch alarms on PartitionCount per-broker metrics to proactively monitor your partition distribution. When an alarm triggers, evaluate your partition strategy and consider rebalancing partitions across brokers, consolidating topics, or scaling out your cluster to stay within recommended limits. For detailed guidance, see Right-size your cluster: Number of partitions per Standard broker and Express broker partition quota.

Practical guidance for choosing a partition count

There is no single formula that works for every Kafka workload. In practice, you typically combine several considerations when sizing partitions.

Start with throughput requirements – The first step is to determine your per-partition throughput capacity, which then informs how many partitions you need.

For Express brokers, use the per-broker throughput capacity as the primary means for sizing your cluster. Express brokers feature a fully managed storage layer, so you do not need to separately account for storage I/O constraints. The published per-broker limits represent the effective capacity available to your workload.

For Standard brokers, the achievable throughput depends on additional factors beyond the broker instance size. These factors include provisioned EBS storage throughput, the number of consumer groups reading from the broker, and how much data is served from memory versus disk. Storage I/O is consumed when producers write, when data replicates between brokers, and when consumers read data that is not in memory. For this reason, validate the effective per-partition throughput for Standard brokers through load testing in your environment.

Once you know your per-partition throughput, calculate the required number of partitions: Number of partitions = Peak throughput of the topic ÷ Throughput per partition

For example, if a topic must handle 40 MB/sec at peak and your testing shows each partition can sustain 5 MB/sec, you would need: 40 ÷ 5 = 8 partitions. Always validate these assumptions with load testing, as actual throughput varies based on your workload characteristics. For initial sizing estimates, refer to the Amazon MSK Sizing and Pricing worksheet and the Amazon MSK Best Practices documentation.

Consider your consumer parallelism needs – If you know the number of consumers required during peak processing times, use that as your partition count. We don’t recommend having more active consumers in a consumer group than partitions. For example, if you have 5 partitions, only 5 consumers can actively process data. Additional consumers remain idle. These idle consumers still maintain active TCP connections to the brokers, sending frequent heartbeats and group coordination requests. This might result in unnecessary overhead on broker resources and contribute to high CPU usage despite low egress traffic.

Figure 3: Idle consumers when a consumer group has more consumers than partitions

Producer throughput and partition keys – When sizing partitions, consider producer-side throughput in addition to consumer parallelism. If producers generate data faster than a single partition can handle, additional partitions can help distribute write traffic across brokers. Partition keys also play a critical role. Poorly distributed or low-cardinality keys can create hot partitions and limit throughput. In such cases, increasing the number of partitions alone does not improve throughput unless records are evenly distributed.
Plan for even distribution and future growth – Kafka works best when partitions can be spread evenly across brokers. Instead of focusing on specific numbers, aim for partition counts that divide reasonably well across your expected broker count. This reduces reassignment churn when brokers are added or replaced. But avoid excessive over-partitioning. It’s reasonable to leave some headroom for future growth. However, creating thousands of partitions “just in case” often causes more harm than good. Increasing partitions later is supported, but it can affect ordering guarantees and may require consumer changes. Start with a conservative number, monitor real traffic patterns, and scale gradually.

From an operational perspective, Amazon MSK provides recommended partition-per-broker guidelines based on broker instance type. Exceeding these guidelines increases operational risk and can block cluster operations such as version upgrades, scaling, or configuration changes. Large partition counts can also increase consumer group rebalance duration, temporarily pausing message processing and increasing end-to-end latency.

Keep in mind that partitioning improves scalability, but it does not address application-level bottlenecks such as slow consumers, inefficient processing logic, or downstream system constraints.

Conclusion

Determining the right number of partitions for an Amazon MSK topic is a foundational design decision. It affects throughput, scalability, failure recovery, and day-to-day operability of your Kafka cluster. Start by understanding your throughput and consumer parallelism needs, respect Amazon MSK partition-per-broker guidelines, avoid excessive over-partitioning, and validate assumptions through load testing. Most importantly, there is no universal “correct” number, only a number that fits your workload, operational goals, and cost.

For more information, see the Amazon MSK Developer Guide and Recommended best practices for Amazon MSK.

About the authors

AWS and DuckLabs: Building the future of analytics together

Mai-Lan Tomsen Bukovec — Wed, 26 Aug 2026 13:28:34 +0000

Today we are announcing that Amazon has signed a definitive agreement to acquire DuckLabs, the Amsterdam-based company behind the open-source analytical database DuckDB. We expect the transaction to close shortly, subject to customary closing conditions. Hannes Mühleisen and Mark Raasveldt, who created DuckDB and co-founded DuckLabs, will continue leading the team and the open-source project’s technical direction as part of AWS. The DuckDB open-source project will also continue to be driven by the DuckLabs team, remain open source under the independent Foundation (the non-profit that oversees DuckDB), and available under the MIT license as it does today (see DuckLabs blog).

Data has always been a core asset and differentiator for companies. That is true now more than ever, as organizations use their data to customize inference and build AI agents. For 20 years AWS has driven the frontier of data, starting with the launch of Amazon S3 to create data lakes for every business, the first cloud analytics service in Amazon EMR, the first cloud data warehouse with Amazon Redshift and the many capabilities that we have introduced with Athena, Glue ETL, etc. We continue innovating for AWS customers on the data frontier including providing Apache Iceberg capabilities directly in S3 Tables, vector storage in the data lake and our new optimized Graviton-based Redshift clusters.

DuckDB has also been at the forefront of changing how the world works with data. Hannes and Mark started DuckDB while at Centrum Wiskunde & Informatica (CWI), the national research institute in the Netherlands that also invented Python. The founders of DuckDB realized that older databases and analytics engines like Spark focused on performance for very large data processing but didn’t have an effective way to “scale down” to smaller size data queries that form the backbone of what most customers do with SQL analytics.

DuckDB set out to solve the problem of blazingly fast performance for the 90%+ of data queries in the world today, that often runs 1 terabyte of data or less as part of analysis and dashboarding. DuckDB’s architecture is based on that core premise of “make the everyday SQL query super fast” so DuckDB runs in-process to other applications which simplifies and speeds up data exchange with the application. DuckDB gets big performance gains from its vectorized execution because it does not require a heavy compiler to run simple statements like SELECT * FROM table. And what works for everyday queries also (unsurprisingly) works very well for agents because agents behave a lot like people when interacting with data. They poke. They experiment. They run exploratory analysis on small data sets before figuring out what they really want to do. DuckDB ends up being naturally optimized for AI agents to use. What started as an academic project is now widely adopted across data engineering, data science, analytics, and now AI agents, for its simplicity of use and raw performance. We plan to combine the superpower of DuckDB at everyday queries of a terabyte or less with the proven exabyte-plus enterprise scale of S3 and our AWS analytics services of Redshift, Athena, EMR, Glue-ETL, and SageMaker platform which power analytics across hundreds of terabytes to petabytes of data. Andy Warfield, Distinguished Engineer at AWS, talks about DuckDB and the Changing Physics of Analytics in Werner Vogel’s All Things Distributed blog.

Our customers use DuckDB today with AWS services and tell us how much they love it for its speed and simplicity. For example, DuckDB today executes SQL directly against external files, such as Parquet, CSV, and JSON, stored locally or on cloud storage like S3 for unparalleled performance and significantly lower cost.

David Feng, Executive Director, Scientific Computing at Allen Institute, said “The Allen Institute accelerates science for a healthier world by tackling the biggest questions in biology at a large scale, and that involves extensive analysis of large, multimodal data. We started using DuckDB to analyze terabytes of scientific data in 2025 and love it. We are storing data in S3 for realtime quality control and analysis of neurophysiology and behavior data, critical to driving the next data acquisition. Queries that took minutes now come back in less than a second, enabling completely new ways of interacting with data.” DuckDB can also run in-process to AWS Lambda functions.

We are excited to make DuckDB applications run best on AWS, and will continue to invest in deep integration between DuckDB and our building block services.

We are also using DuckDB in our own AWS infrastructure. When Amazon Quick wanted to augment the performance of their custom dashboarding engine, they picked DuckDB to query data in S3 Tables. The Quick team found that the DuckDB engine scales effortlessly with the number of CPUs, and its single library can easily plug into the internal Quick control plane subsystems. Since we launched Quick in October 2025, we have processed over 2.5B queries using our custom Quick query engine with the DuckDB integrations and optimizations. These DuckDB integrations and optimizations helped Amazon Quick reduce average query latency by 30%. We are going to look at how we can integrate DuckDB’s performance and simplicity in our other AWS services across data and analytics.

Stay tuned for more about how DuckLabs and AWS will reinvent the frontier of data together for applications, data engineers, and AI, meeting customers where they are today and giving them the benefits of DuckDB’s innovation within AWS.

About the author

PythonOperator and BashOperator Now Available on Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless

Pradeep Kumar Nalluri — Tue, 25 Aug 2026 19:28:00 +0000

If you run Apache Airflow workflows on Amazon MWAA Serverless, you can now use PythonOperator and BashOperator to run custom code directly in the serverless runtime. Previously, Amazon Managed Workflows for Apache Airflow (Amazon MWAA) Serverless only supported orchestration of AWS services through operators for scheduling tasks, managing dependencies, and handling retries. It did not support running your own Python functions or shell scripts natively. If you needed custom Python logic or shell commands, you had to wrap code in AWS Lambda functions, start Amazon Elastic Container Service (Amazon ECS) tasks, or use other AWS compute services. These alternatives add complexity, cost, and latency to your orchestration pipelines.

With this launch, you can run custom Python functions and shell scripts directly within the serverless task runtime, without requiring additional infrastructure. This means you can now use PythonOperator and BashOperator many data engineering teams rely on for ETL pipelines and data quality checks – without provisioning additional compute.

In this post, we walk through how this feature works and demonstrate a practical example: building a serverless pipeline that converts CSV files to JSON format using a PythonOperator, and verifies the output using a BashOperator. By the end, you will know how to:

Package a Python module with dependencies and upload it to an Amazon Simple Storage Service (Amazon S3) bucket as a code bundle
Define a multi-task workflow using the dag-factory compatible YAML
Create and run a workflow with the AWS Command Line Interface (AWS CLI)
Verify that your pipeline produced the expected output

How it works

With MWAA Serverless, you can package your custom code, upload it to an Amazon S3 bucket, and reference it when creating a workflow. The service snapshots your code at workflow creation time and uses that snapshot for all subsequent runs of the same workflow version.

Code bundles

A code bundle is the package that contains your custom logic. You package your Python modules or shell scripts and upload them to an Amazon S3 bucket. A code bundle can be:

A single .py file or .sh bash script (uploaded to an Amazon S3 bucket)
A ZIP archive containing multiple shell scripts, Python modules and dependencies (up to 250 MB)

Execution model

When you create or update a workflow, MWAA Serverless snapshots your code bundle from an Amazon S3 bucket provided and stores it on the service side. At task execution time, the service uses this snapshot – not the object currently residing in your Amazon S3 bucket – to run your code in an isolated runtime environment.

Python and Bash tasks do not have internet access. They can reach only Amazon S3, Amazon Elastic Container Registry (Amazon ECR), and Amazon CloudWatch, which are the services the runtime requires to operate. To have internet access, configure the workflow with Amazon VPC so that it can go through the provided VPC.

Supported operators

The following table describes the two operators now available in MWAA Serverless.

Operator	Description
PythonOperator	Executes a Python callable (function) from your code bundle
BashOperator	Runs shell commands or scripts

Security

AWS Key Management Service (AWS KMS) encrypts your code bundles at rest. IAM policies control who can create, update, and trigger the workflows. The execution role scopes what AWS resources your code can access at runtime.

Prerequisites

Before getting started, verify that you have the following resources and tools configured in your AWS account:

An AWS account with access to Amazon MWAA Serverless
AWS CLI v2 (latest version) installed and configured. To install or update, see Installing or updating to the latest version of the AWS CLI.
An Amazon S3 bucket for storing DAG definitions and code bundles
An IAM role that MWAA Serverless can assume (see the execution role setup below)

Walkthrough: Building a serverless CSV-to-JSON pipeline

In this walkthrough, we build a pipeline that converts CSV files to JSON format – a common data transformation for downstream APIs and analytics systems that consume JSON. The pipeline uses a PythonOperator for the conversion logic and a BashOperator to verify the output. Here is what the pipeline does:

Reads a CSV file from an Amazon S3 bucket
Converts it to JSON format with column type inference
Writes the JSON file back to an Amazon S3 bucket
Validates record counts match between source and output

Step 1: Create the execution role

Create an IAM role that your workflow assumes at runtime. The trust policy must allow the airflow-serverless.amazonaws.com service to assume the role:

cat > trust-policy.json << 'EOF'
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "airflow-serverless.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF

Create the role and attach an inline policy granting least-privilege access to your S3 bucket:

aws iam create-role \
  --role-name MWAAServerlessExecutionRole \
  --assume-role-policy-document file://trust-policy.json

aws iam put-role-policy \
  --role-name MWAAServerlessExecutionRole \
  --policy-name MWAAServerlessAccessPolicy \
  --policy-document '{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::amzn-s3-demo-mwaa-data",
        "arn:aws:s3:::amzn-s3-demo-mwaa-data/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams",
        "logs:GetLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:log-group:/aws/mwaa-serverless/*"
    }
  ]
}'

Step 2: Write the Python module

Create a file called csv_to_json.py with the conversion logic:

# csv_to_json.py
import csv
import json
import boto3
import io

def convert(**kwargs):
    """Read a CSV from S3 and write it back as JSON lines."""
    bucket = "amzn-s3-demo-mwaa-data"
    source_key = "raw/sales_data.csv"
    output_key = "processed/sales_data.json"

    s3 = boto3.client("s3")

    # Read source file
    response = s3.get_object(Bucket=bucket, Key=source_key)
    content = response["Body"].read().decode("utf-8")

    # Parse CSV
    reader = csv.DictReader(io.StringIO(content))
    rows = list(reader)

    # Type inference - convert numeric fields
    for row in rows:
        for key, value in row.items():
            try:
                row[key] = float(value)
            except (ValueError, TypeError):
                pass

    # Write as JSON lines
    output = "\n".join(json.dumps(row) for row in rows) + "\n"
    s3.put_object(Bucket=bucket, Key=output_key, Body=output.encode("utf-8"))

    print(f"Converted {len(rows)} rows to JSON lines")
    print(f"Output: s3://amzn-s3-demo-mwaa-data/{output_key}")
    return {"rows": len(rows), "output_key": output_key}

This function uses boto3 (which comes pre-installed with the MWAA Serverless execution environment) and Python’s built-in csv and json modules. The conversion reads the CSV, infers numeric types, and writes a JSON lines file back to the S3 bucket.

Step 3: Write the verification script

Create a file called verify_output.sh. This script validates the pipeline output by comparing the record count in the source CSV against the output JSON file. If the counts do not match, the task fails with a non-zero exit code, which causes the workflow run to fail.

#!/bin/bash
echo "=== Data Validation ==="

# Count source records (skip CSV header)
SOURCE_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/raw/sales_data.csv - | tail -n +2 | wc -l)
echo "Source CSV records: $SOURCE_COUNT"

# Count output records
OUTPUT_COUNT=$(python3 -m awscli s3 cp s3://amzn-s3-demo-mwaa-data/processed/sales_data.json - | wc -l)
echo "Output JSON records: $OUTPUT_COUNT"

# Validate counts match
if [ "$SOURCE_COUNT" -ne "$OUTPUT_COUNT" ]; then
    echo "FAILED: Record count mismatch (source=$SOURCE_COUNT, output=$OUTPUT_COUNT)"
    exit 1
fi

echo "PASSED: Record counts match ($OUTPUT_COUNT records)"
echo "Timestamp: $(date -u +%Y-%m-%dT%H:%M:%SZ)"

This script runs the AWS CLI, which is bundled as a dependency in the code package. The s3 cp streams the file content to stdout without writing to disk, allowing standard shell tools like wc -l and tail to process it. The execution role credentials are automatically available in the execution environment, so the CLI can access S3 without additional configuration.

Step 4: Package and upload the code to Amazon S3

Since the verification script uses the AWS CLI, bundle it as a dependency in the ZIP archive along with your Python module and shell script:

BUCKET="amzn-s3-demo-mwaa-data"
REGION="us-east-1"

# Install awscli into a package directory
pip install awscli \
  --target my_package/ \
  --platform manylinux2014_x86_64 \
  --python-version 3.12 \
  --only-binary=:all:

# Add your module
cp csv_to_json.py my_package/
cp verify_output.sh my_package/

# Create the ZIP archive
cd my_package && zip -r ../code_bundle.zip . && cd ..
# Upload to S3
aws s3 cp code_bundle.zip s3://$BUCKET/code/code_bundle.zip --region $REGION

Upload a sample CSV file for testing:

cat > sales_data.csv << 'EOF'
date,region,product,units,revenue
2026-07-01,us-east,widget-a,150,4500.00
2026-07-01,eu-west,widget-b,89,2670.00
2026-07-02,us-east,widget-a,203,6090.00
2026-07-02,ap-south,widget-c,67,1340.00
2026-07-03,us-east,widget-b,178,5340.00
EOF

aws s3 cp sales_data.csv s3://$BUCKET/raw/sales_data.csv --region $REGION

Step 5: Define the DAG (YAML)

MWAA Serverless uses a declarative YAML format for DAG definitions. Create a file called conversion_dag.yaml:

csv_to_json_pipeline:
  start_date: "2026-01-01"
  schedule: null
  tasks:
    convert_to_json:
      operator: airflow.operators.python.PythonOperator
      python_callable: csv_to_json.convert
    verify_output:
      operator: airflow.operators.bash.BashOperator
      bash_command: "verify_output.sh"
      dependencies:
        - convert_to_json

This DAG defines two tasks:

convert_to_json – Runs the convert function from the Python module to transform CSV to JSON lines.
verify_output – Runs a shell script that validates the pipeline output by comparing source and output record counts, failing the task if they do not match.

Upload the DAG definition to S3. Note: You can also run inline Bash commands directly without a shell script.

aws s3 cp conversion_dag.yaml s3://$BUCKET/dags/conversion_dag.yaml --region $REGION

Step 6: Create the workflow

Create the MWAA Serverless workflow, referencing the DAG definition and the code bundle:

ROLE_ARN="arn:aws:iam::<your-account-id>:role/MWAAServerlessExecutionRole"

aws mwaa-serverless create-workflow \
  --name csv-to-json-workflow \
  --definition-s3-location Bucket="$BUCKET",ObjectKey="dags/conversion_dag.yaml" \
  --code '{"S3Location": {"Bucket":"'"$BUCKET"'","ObjectKey":"code/code_bundle.zip"}}' \
  --role-arn $ROLE_ARN \
  --region $REGION

The response includes a WorkflowArn that you use to trigger runs:

{
  "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123",
  "CreatedAt": "2026-07-15T10:30:00.000000+00:00",
  "WorkflowVersion": "a1b2c3d4e5f6"
}

Step 7: Run the workflow

Trigger a workflow run:

WORKFLOW_ARN="arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123"

aws mwaa-serverless start-workflow-run \
  --workflow-arn $WORKFLOW_ARN \
  --region $REGION

The response confirms the run has started:

{
  "RunId": "6OZV9ABF9enHKXk",
  "Status": "STARTING"
}

Step 8: Monitor execution

Check the status of your run:

RUN_ID="6OZV9ABF9enHKXk"

aws mwaa-serverless get-workflow-run \
  --workflow-arn $WORKFLOW_ARN \
  --run-id $RUN_ID \
  --region $REGION

A successful run returns:

{
  "RunDetail": {
    "Duration": 45,
    "RunState": "SUCCESS",
    "TaskInstances": ["ex_abc123_convert_to_json_1", "ex_abc123_verify_output_1"]
  },
  "RunId": "6OZV9ABF9enHKXk",
  "RunType": "ON_DEMAND",
  "WorkflowArn": "arn:aws:airflow-serverless:us-east-1:123456789012:workflow/csv-to-json-workflow-abc123",
  "WorkflowVersion": "a1b2c3d4e5f6"
}

Step 9: Verify the output

Confirm the JSON file was written to the S3 bucket:

# List the output file
aws s3 ls s3://$BUCKET/processed/sales_data.json --region $REGION

You should see the JSON file:

2026-07-15 10:32:45 1847 sales_data.json

You can also verify task-level output in Amazon CloudWatch Logs. Open the log group for your workflow and find the convert_to_json task log stream:

Converted 5 rows to JSON lines
Output: s3://amzn-s3-demo-mwaa-data/processed/sales_data.json

Considerations and limits

When planning your workloads on MWAA Serverless with these operators, keep the following considerations in mind:

Code bundle size – ZIP archives must be under 250 MB per bundle.
Network access – Python and Bash tasks do not have internet access. They can reach a limited set of AWS services required for the runtime to function (Amazon S3, Amazon ECR, and Amazon CloudWatch) but cannot call other AWS services or external endpoints. If your workflow requires calls to external APIs, preprocess that data and store it in an Amazon S3 bucket before invoking the workflow.
Runtime dependencies – boto3 and the Python standard library are pre-installed. For additional packages (such as pandas or requests), bundle them in your ZIP archive following the Amazon MWAA Serverless packaging guidelines.
Execution timeout – Tasks are subject to the workflow’s configured timeout limits.
Python version – Check the Amazon MWAA Serverless documentation for the currently supported Python runtime version.
DAG format – MWAA Serverless uses YAML-based DAG definitions, not traditional Python DAG files. If you are migrating from MWAA Provisioned, you will need to convert your DAGs to the YAML format.
Operators not supported – Some Airflow community operators and custom plugins are not available in the Serverless runtime. Refer to the documentation for the full compatibility list.

Clean up

To avoid ongoing charges, delete the resources you created in this walkthrough. The following commands remove the workflow, S3 objects, and IAM role:

Note: $WORKFLOW_ARN is defined in Step 7.

# Delete the workflow
aws mwaa-serverless delete-workflow \
  --workflow-arn $WORKFLOW_ARN \
  --region $REGION

Note: $BUCKET is exported in Step 4. If appropriate, delete the bucket as well.

# Remove S3 objects
aws s3 rm s3://$BUCKET/code/code_bundle.zip
aws s3 rm s3://$BUCKET/dags/conversion_dag.yaml
aws s3 rm s3://$BUCKET/raw/sales_data.csv
aws s3 rm s3://$BUCKET/processed/sales_data.json

# Delete the IAM role
aws iam delete-role-policy \
  --role-name MWAAServerlessExecutionRole \
  --policy-name MWAAServerlessAccessPolicy

aws iam delete-role --role-name MWAAServerlessExecutionRole

Conclusion

With native support for PythonOperator and BashOperator, you can now run the custom code execution patterns that many data engineering teams rely on daily directly in MWAA Serverless. Run data transformations, format conversions, validations, and shell scripts in the serverless runtime – without provisioning additional compute or managing containers.

If you are running Airflow workloads on MWAA Provisioned or self-managed infrastructure, your existing PythonOperator and BashOperator logic requires minimal changes. Convert your Python DAG files to the YAML format, package your code as a bundle, and you are ready to run on MWAA Serverless.

To get started, visit the Amazon MWAA Serverless documentation and try the walkthrough earlier in this post with your own data. For pricing details, visit the Amazon MWAA pricing page. We look forward to your feedback.

About the authors

Enable cross-cloud analytics with Amazon S3 Tables and Google BigQuery, Part 1: IAM-based access control

Lakshmi Nair — Tue, 25 Aug 2026 16:38:49 +0000

Organizations running analytics workloads across multiple clouds often hit the same friction: the data lives on one cloud, but the engine querying it lives on another. Copying data across the boundary creates a second dataset that must be kept in sync, adding cost, latency, and reconciliation overhead. In this post, we address a specific instance of that pattern: your Google BigQuery users need to work with data that lives in Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), on AWS. The ideal outcome is a single, governed dataset that serves teams in both clouds without a standing replication pipeline between them.

With Amazon S3 Tables, you get managed Apache Iceberg tables with built-in compaction, snapshot management, and an integration with the AWS Glue Data Catalog. Because S3 Tables stores data in the open Iceberg format, supported external engines can read it directly if the right access path exists.

This two-part blog series demonstrates how you can connect Google BigQuery to Amazon S3 Tables using the cross-cloud lakehouse with AWS Glue. We cover two access control approaches:

AWS Identity and Access Management (IAM): You can define a single policy that uses IAM permissions to set up access to both table metadata and data.
AWS Lake Formation: You can use temporary vended credentials for data access, with metadata access managed by Lake Formation permissions.

This post focuses on the IAM-based approach. Part 2 covers the Lake Formation approach for organizations that need credential-vended access across multiple engines.

By the end, you will have BigQuery querying Iceberg tables stored on S3 Tables without data copy or duplication, providing live access to Iceberg data.

Cross-cloud analytics scenarios

There are several scenarios where organizations benefit from cross-cloud querying capabilities. Here are some of the common patterns this architecture addresses:

Schema evolution across cloud boundaries

When source schemas change frequently, streaming pipelines writing to BigQuery-managed store require coordinated DDL changes on the BigQuery table and downstream views. Teams often work around this challenge by storing payloads as untyped columns and parsing them later.

With Iceberg on S3 Tables, schema evolution is tracked in table metadata. When the writing engine adds a new column, BigQuery’s Lakehouse refresh picks up the updated schema automatically on the next sync cycle.

Multi-cloud analytics without data duplication

A company has its production data environment on AWS (data lakes, warehouses, streaming) but acquired a business unit that runs analytics exclusively on BigQuery. In-place querying from BigQuery keeps your data in Amazon S3 Tables, so you pay for one copy, work from live data, and avoid the operational overhead of a synchronized second store.

Cost optimization for infrequently queried datasets

An organization has hundreds of datasets on AWS, but only a fraction is queried daily from BigQuery. Replicating all of them to Google Cloud Storage drives unnecessary storage and transfer costs. With Lakehouse catalog federation, you keep your data on S3 Tables. BigQuery reads data only when queried, so you pay per query rather than per-copy storage.

Decoupled compute across engines

Data team wants storage on AWS with the flexibility for multiple engines to read the same data: BigQuery and Amazon Redshift for data warehousing use cases, Amazon Athena for interactive ad-hoc querying, Amazon SageMaker AI for machine learning (ML). With Apache Iceberg’s open format, you can use one storage layer, many compute engines, no data copies between them.

Solution overview

You use the AWS Glue Iceberg REST Catalog (IRC) as the bridge between BigQuery and S3 Tables. BigQuery’s cross-cloud Lakehouse creates a federated catalog that syncs metadata from the Glue IRC, then uses the synced metadata to read Iceberg data files directly.

Figure 1: Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

The key components in this architecture:

Amazon S3 Tables: With Amazon S3 Tables, you get a fully managed Apache Iceberg table experience in Amazon S3, optimized for analytics workloads. You can register table metadata in the AWS Glue Data Catalog for discovery and governance.
AWS Glue Data Catalog: With AWS Glue Data Catalog, you can access the federated s3tablescatalog catalog that maps S3 Tables resources (table buckets, namespaces, tables) into a catalog hierarchy from supported analytics engines. The standard Iceberg REST endpoint of Glue Data Catalog serves table metadata to external engines. BigQuery connects through this endpoint.
Google Cross-Cloud Lakehouse: With Google Cross-Cloud Lakehouse, you can connect BigQuery to external Iceberg catalogs. It assumes an AWS IAM role using OpenID Connect (OIDC), calls the Glue Iceberg REST endpoint, and syncs metadata on a configurable refresh interval.

Prerequisites

Before you begin, you need:

An AWS account with Amazon S3 Tables available in your AWS Region.
A Google Cloud project with billing enabled and the BigLake API activated.
AWS Command Line Interface (AWS CLI) and gcloud CLI installed and configured.
An S3 table bucket with at least one namespace and table containing data.

Setting up Amazon S3 Tables

If you already have S3 Tables with data, skip to the next section. Otherwise, create a table bucket, namespace, and populate a table.

Create a table bucket and namespace

Use the AWS CLI to create resources as follows:

# Create a Table bucket
aws s3tables create-table-bucket \
    --name <TABLE_BUCKET_NAME> \
    --region <REGION>

# Create a Namespace (Database)
aws s3tables create-namespace \
    --table-bucket-arn "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>" \
    --namespace <NAMESPACE> \
    --region <REGION>

Integrating S3 Tables with the Glue Data Catalog

For BigQuery to access S3 Tables, the tables must be discoverable through the Glue Data Catalog. S3 Tables integrates with Glue through a federated catalog called s3tablescatalog.

Set up S3 Tables integration with the Glue Data Catalog using IAM mode

Open the Amazon S3 console:

In the navigation pane, choose Table buckets.
Choose Enable integration, and then choose Enable integration again to confirm.

This creates the s3tablescatalog federated catalog in Glue, where access is controlled entirely by IAM policies on the calling role. This is a one-time setup per account and Region. After you enable it, the analytics integration applies to all table buckets in your account.

Figure 2: Enabling the S3 Tables integration in the Amazon S3 console

Alternatively, create the catalog using the AWS CLI:

aws glue create-catalog --region <REGION> --cli-input-json '{
  "Name": "s3tablescatalog",
  "CatalogInput": {
    "FederatedCatalog": {
      "Identifier": "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/*",
      "ConnectionName": "aws:s3tables"
    },
    "CreateDatabaseDefaultPermissions": [
      { "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }
    ],
    "CreateTableDefaultPermissions": [
      { "Principal": {"DataLakePrincipalIdentifier": "IAM_ALLOWED_PRINCIPALS"}, "Permissions": ["ALL"] }
    ]
  }
}'

Create a table and insert data

Now, to create the table and insert data, open the Amazon Athena console. In the query editor, select s3tablescatalog/<TABLE_BUCKET_NAME> as your data source and <NAMESPACE> as the database. Then run the following SQL statements one by one:

CREATE TABLE `<NAMESPACE>`.orders (
    order_id STRING,
    customer_id STRING,
    amount BIGINT,
    order_date DATE,
    region STRING
)
TBLPROPERTIES ('table_type' = 'iceberg');

INSERT INTO orders
VALUES
    ('ORD-001', 'C100', 4500, DATE '2024-06-01', 'EMEA'),
    ('ORD-002', 'C200', 8900, DATE '2024-06-01', 'EMEA'),
    ('ORD-003', 'C100', 3200, DATE '2024-06-02', 'NAMER'),
    ('ORD-004', 'C300', 12000, DATE '2024-06-02', 'NAMER'),
    ('ORD-005', 'C400', 6700, DATE '2024-06-03', 'APJ'),
    ('ORD-006', 'C200', 4100, DATE '2024-06-03', 'APJ'),
    ('ORD-007', 'C500', 9500, DATE '2024-06-04', 'EMEA'),
    ('ORD-008', 'C100', 2800, DATE '2024-06-04', 'LATAM'),
    ('ORD-009', 'C600', 15000, DATE '2024-06-05', 'NAMER'),
    ('ORD-010', 'C300', 7200, DATE '2024-06-05', 'LATAM');

Configuring cross-cloud access

BigQuery assumes an AWS IAM role via OIDC federation to access the Glue IRC. This section walks through creating the role, OIDC provider, and permissions.

Create the OIDC identity provider

Register Google as an OIDC identity provider in your AWS account. This allows AWS to validate tokens issued by Google’s identity service:

aws iam create-open-id-connect-provider \
    --url https://accounts.google.com \
    --client-id-list accounts.google.com \
    --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6

The –thumbprint-list parameter is optional. When omitted, IAM automatically retrieves the thumbprint from the OIDC provider’s certificate. See AWS documentation for details.

Create the cross-cloud IAM role

Login into AWS Console, and create the role with a placeholder trust policy. You will update it with the actual BigLake service account ID after you create the federated catalog in Google Cloud.

aws iam create-role \
    --role-name bigquery-cross-cloud-role \
    --max-session-duration 43200 \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["PLACEHOLDER"],
            "accounts.google.com:aud": ["PLACEHOLDER"]
          }
        }
      }]
    }'

The --max-session-duration 43200 allows sessions up to 12 hours, which is needed for long-running BigQuery queries.

Attach permissions

The permissions policy differs based on your access control approach. For the IAM-based approach, attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueRead",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases",
        "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog/<TABLE_BUCKET>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>/*"
      ]
    },
    {
      "Sid": "S3TablesRead",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:ListTableBuckets",
        "s3tables:ListNamespaces", "s3tables:GetNamespace",
        "s3tables:ListTables", "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation", "s3tables:GetTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>",
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>/*"
      ]
    },
    {
      "Sid": "S3TablesListBuckets",
      "Effect": "Allow",
      "Action": ["s3tables:ListTableBuckets"],
      "Resource": "*"
    }
  ]
}

Connecting BigQuery to S3 Tables

With the AWS side configured, create the federated catalog in Google Cloud that connects BigQuery to the Glue IRC.

Create the federated catalog

Authenticate to Google Cloud using gcloud auth login, or use Cloud Shell, which is pre-authenticated. Verify that the BigLake API is enabled:

gcloud services enable biglake.googleapis.com --project="<GCP_PROJECT_ID>"

For IAM mode:

gcloud alpha biglake iceberg catalogs create <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --catalog-type=federated \
    --federated-catalog-type=glue \
    --glue-aws-region=<AWS_REGION> \
    --glue-aws-role-arn=arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role \
    --glue-warehouse=<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET> \
    --primary-location=<GCP_REGION>

The --glue-warehouse parameter uses the format <AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>. This tells the Glue IRC to scope requests to your specific S3 Tables bucket within the federated catalog hierarchy.

The --primary-location refers to the Google Cloud region where the federated catalog metadata is stored. Use the AWS to Google Cloud region mapping to find the corresponding GCP region for your AWS Region. For example, AWS us-east-1 maps to GCP us-east4.

Retrieve the BigLake service account ID

After catalog creation, Google provisions a dedicated service account for your federated catalog. Retrieve its numeric ID:

BIGLAKE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --format="value(biglake-service-account-id)")
echo $BIGLAKE_SA_ID

Update the AWS trust policy

Back on AWS, replace the placeholder in the IAM role’s trust policy with the actual service account ID:

aws iam update-assume-role-policy \
    --role-name bigquery-cross-cloud-role \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["<BIGLAKE_SA_ID>"],
            "accounts.google.com:aud": ["<BIGLAKE_SA_ID>"]
          }
        }
      }]
    }'

Register the service account ID in the OIDC provider’s audience list. Without this step, AWS rejects the token because the aud claim doesn’t match any registered client:

aws iam add-client-id-to-open-id-connect-provider \
    --open-id-connect-provider-arn "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com" \
    --client-id "<BIGLAKE_SA_ID>"

Set up metadata sync

Wait 3–5 minutes for IAM changes to propagate globally, then set up background refresh:

gcloud alpha biglake iceberg catalogs update <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --refresh-interval=300s

The --refresh-interval (300 seconds in this example) determines how often BigQuery syncs metadata from the Glue IRC. New tables and schema changes appear in BigQuery within this interval.

Querying from BigQuery

After the catalog refresh completes, BigQuery automatically creates external datasets corresponding to the synced namespaces. No manual CREATE SCHEMA is required.

Verify the sync:

gcloud alpha biglake iceberg namespaces list \
    --catalog="<FEDERATED_CATALOG_NAME>" \
    --project="<GCP_PROJECT_ID>"

Run a query in BigQuery:

SELECT * FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders` LIMIT 1000

Sample Query Output:

SELECT
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spend
FROM `<PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders`
GROUP BY customer_id
ORDER BY total_spend DESC

Figure 3: BigQuery query results returned directly from the Amazon S3 Tables data

BigQuery reads the Iceberg metadata to identify which Parquet data files contain relevant data. It also applies partition pruning where applicable, and fetches only the necessary files from S3 Tables managed storage.

Schema evolution

When new columns are added to an Iceberg table on the AWS side (through Spark, Athena, or the Glue IRC), the schema change is captured in Iceberg’s metadata. On the next Lakehouse refresh cycle, BigQuery picks up the new columns automatically. No DDL changes are needed in BigQuery.

Metadata freshness

The s3tablescatalog in Glue is a federated catalog that resolves table metadata live from the S3 Tables service on each request. When a streaming job commits new data to an S3 Table, the latest metadata is immediately available through the AWS Glue IRC. BigQuery sees the update on its next refresh cycle (as configured by --refresh-interval).

OIDC identity federation

The trust relationship between Google Cloud and AWS uses OpenID Connect. When BigQuery Lakehouse needs to access your data, it presents a signed JWT token containing:

iss: accounts.google.com (the issuer)
sub: The BigLake service account ID (identifies which catalog is making the request)
aud: The same service account ID (the intended audience)

AWS validates this token against the registered OIDC provider and trust policy conditions before issuing temporary credentials. Each federated catalog receives a unique service account ID, providing per-catalog isolation and auditability through AWS CloudTrail.

Network path

By default, traffic between BigQuery and AWS travels over the public internet. For workloads requiring private connectivity, Google Cloud supports Cross-Cloud Interconnect or Partner Interconnect. This helps routing queries over a dedicated network path. Refer to the Google Cloud documentation for private interconnect configuration.

Clean up

To avoid ongoing charges, remove the resources created in this walkthrough.

On AWS:

# Delete the table (if created for this walkthrough)
aws s3tables delete-table \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace analytics --name orders --region <AWS_REGION>

# Delete namespace and table bucket
aws s3tables delete-namespace \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace <NAMESPACE> --region <AWS_REGION>

aws s3tables delete-table-bucket --name <TABLE_BUCKET> --region <AWS_REGION>

# Delete IAM role and OIDC provider (if no longer needed)
aws iam delete-role --role-name bigquery-cross-cloud-role

On Google Cloud:

gcloud alpha biglake iceberg catalogs delete <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" --location=<GCP_REGION>

Conclusion

This post demonstrated how to query Amazon S3 Tables from Google BigQuery using the open Apache Iceberg format and the AWS Glue Iceberg REST Catalog as the metadata bridge. Using Apache Iceberg’s open format, you can write data once on AWS and read it from supported engines that speak Iceberg, including BigQuery. We used IAM-based access control to govern access to both Glue Data Catalog metadata and the underlying Amazon S3 Tables data. This is the simpler configuration path with fewer components. In Part 2, we walk through configuring AWS Lake Formation to vend temporary, scoped credentials to BigQuery for data access.

To get started with this pattern in your environment:

Review the Amazon S3 Tables documentation for table bucket setup.
Follow the Google Cloud cross-cloud Lakehouse setup guide for the BigQuery federation configuration.
Explore the AWS Glue Iceberg REST Catalog APIs for programmatic access patterns.

About the authors

Enable cross-cloud analytics with Amazon S3 Tables and Google BigQuery, Part 2: access control with Lake Formation

Lakshmi Nair — Tue, 25 Aug 2026 16:38:46 +0000

In Part 1, we showed how to connect Google BigQuery to Amazon Simple Storage Service (Amazon S3) Tables, a capability of Amazon S3, using access control based on AWS Identity and Access Management (IAM). A single IAM policy governs both table metadata and data access. We also walked through common cross-cloud analytics scenarios where this pattern adds value. This post covers the approach using AWS Lake Formation. Instead of relying solely on IAM policies for data access, Lake Formation manages fine-grained permissions and vends temporary, scoped credentials to the requesting engine. This is a better fit when multiple engines need different levels of access to the same tables, or when you want to manage grants centrally without touching IAM policies every time a new consumer comes along.

Solution overview

Figure 1: Architecture diagram showing BigQuery connecting to Amazon S3 Tables through the AWS Glue Iceberg REST Catalog

The key components in this architecture:

Amazon S3 Tables: With Amazon S3 Tables, data is stored in table buckets, specifically designed for storing tables in the Apache Iceberg format. Table metadata is registered on AWS Glue Data Catalog for discovery and governance.
AWS Glue Data Catalog: With AWS Glue Data Catalog, you can access the federated s3tablescatalog catalog that maps S3 Tables resources (table buckets, namespaces, tables) into a catalog hierarchy from supported analytics engines. The standard Iceberg REST endpoint of Glue Data Catalog serves table metadata to external engines. BigQuery connects through this endpoint.
AWS Lake Formation: With AWS Lake Formation, you define access permissions at the catalog, database, and table level. Instead of granting broad IAM permissions for data access, Lake Formation evaluates permissions at query time and issues short-lived credentials limited to the resources the caller is authorized to read.
Google Cross-Cloud Lakehouse: With Google Cross-Cloud Lakehouse, you can connect BigQuery to external Iceberg catalogs. It assumes an IAM role using OpenID Connect (OIDC), calls the AWS Glue Iceberg REST endpoint, and syncs metadata on a configurable refresh interval.

Prerequisites

Before you begin, you need:

An AWS account with Amazon S3 Tables available in your AWS Region.
A Google Cloud project with billing enabled and the BigLake API activated.
AWS Command Line Interface (AWS CLI) and gcloud CLI installed and configured.
An S3 table bucket with at least one namespace and table containing data.

Setting up Amazon S3 Tables

If you already have S3 Tables with data, skip to the next section. Otherwise, create a table bucket, namespace, and populate a table.

Create a table bucket and namespace

Use AWS CLI to create resources as follows:

# Create a Table bucket
aws s3tables create-table-bucket \
    --name <TABLE_BUCKET_NAME> \
    --region <REGION>

# Create a Namespace (Database)
aws s3tables create-namespace \
    --table-bucket-arn "arn:aws:s3tables:<REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET_NAME>" \
    --namespace <NAMESPACE> \
    --region <REGION>

Set up S3 Tables integration with the Glue Data Catalog using Lake Formation mode

Lake Formation needs its own service role to interact with S3 Tables on your behalf. This is the role Lake Formation assumes internally when it reads or writes data on behalf of authorized callers.

Create a Lake Formation service IAM role named LakeFormationS3TablesServiceRole with the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationPermissionsForS3ListTableBucket",
      "Effect": "Allow",
      "Action": ["s3tables:ListTableBuckets"],
      "Resource": ["*"]
    },
    {
      "Sid": "LakeFormationDataAccessPermissionsForS3TableBucket",
      "Effect": "Allow",
      "Action": [
        "s3tables:CreateTableBucket", "s3tables:GetTableBucket",
        "s3tables:CreateNamespace", "s3tables:GetNamespace",
        "s3tables:ListNamespaces", "s3tables:DeleteNamespace",
        "s3tables:DeleteTableBucket", "s3tables:CreateTable",
        "s3tables:DeleteTable", "s3tables:GetTable",
        "s3tables:ListTables", "s3tables:RenameTable",
        "s3tables:UpdateTableMetadataLocation", "s3tables:GetTableMetadataLocation",
        "s3tables:GetTableData", "s3tables:PutTableData"
      ],
      "Resource": ["arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/*"]
    }
  ]
}

Attach the following trust relationship:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "LakeFormationDataAccessPolicy",
      "Effect": "Allow",
      "Principal": { "Service": "lakeformation.amazonaws.com" },
      "Action": ["sts:AssumeRole", "sts:SetContext", "sts:SetSourceIdentity"],
      "Condition": { "StringEquals": { "aws:SourceAccount": "<AWS_ACCOUNT_ID>" } }
    }
  ]
}

In the Lake Formation console, in the navigation pane, choose Catalogs, and then choose Enable S3 Table Integration.

Figure 2: Enabling the S3 Tables integration in the Lake Formation console

Choose the role you created earlier when prompted for an IAM role, and select Allow external engines to access data in Amazon S3 locations with full table access.

S3 Tables integration performs the following:

Registers the S3 Tables data location with Lake Formation.
Creates the s3tablescatalog federated catalog in Glue.

Important: Before enabling the integration, verify your Lake Formation data lake settings have empty default permissions to prevent IAMAllowedPrincipals from being auto-granted on the catalog:

aws lakeformation put-data-lake-settings \
    --data-lake-settings '{"DataLakeAdmins":[{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/<ADMIN_ROLE>"}],"CreateDatabaseDefaultPermissions":[],"CreateTableDefaultPermissions":[]}' \
    --region <AWS_REGION>

Figure 3: Selecting full table access for external engines during S3 Tables integration

When you select this option, you allow external engines to access data in Amazon S3 locations with full table access, and Lake Formation grants full table-level access to external engines. Column-level and row-level filtering are not enforced for external engine connections. Access is granted at the whole-table level.

Verify the integration by confirming the catalog in Lake Formation console.

Create a table and insert data

CREATE TABLE `<NAMESPACE>`.orders (
    order_id STRING,
    customer_id STRING,
    amount BIGINT,
    order_date DATE,
    region STRING
)
TBLPROPERTIES ('table_type' = 'iceberg');

INSERT INTO orders
VALUES
    ('ORD-001', 'C100', 4500, DATE '2024-06-01', 'EMEA'),
    ('ORD-002', 'C200', 8900, DATE '2024-06-01', 'EMEA'),
    ('ORD-003', 'C100', 3200, DATE '2024-06-02', 'NAMER'),
    ('ORD-004', 'C300', 12000, DATE '2024-06-02', 'NAMER'),
    ('ORD-005', 'C400', 6700, DATE '2024-06-03', 'APJ'),
    ('ORD-006', 'C200', 4100, DATE '2024-06-03', 'APJ'),
    ('ORD-007', 'C500', 9500, DATE '2024-06-04', 'EMEA'),
    ('ORD-008', 'C100', 2800, DATE '2024-06-04', 'LATAM'),
    ('ORD-009', 'C600', 15000, DATE '2024-06-05', 'NAMER'),
    ('ORD-010', 'C300', 7200, DATE '2024-06-05', 'LATAM');

Configuring cross-cloud access

BigQuery assumes an AWS IAM role using OIDC federation to access the AWS Glue IRC. This section walks through creating the role, OIDC provider, and permissions.

Create the OIDC identity provider

Register Google as an OIDC identity provider in your AWS account. This allows AWS to validate tokens issued by Google’s identity service:

aws iam create-open-id-connect-provider \
    --url https://accounts.google.com \
    --client-id-list accounts.google.com \
    --thumbprint-list 08745487e891c19e3078c1f2a07e452950ef36f6

The –thumbprint-list parameter is optional. When omitted, IAM automatically retrieves the thumbprint from the OIDC provider’s certificate. See AWS documentation for details.

Create the cross-cloud IAM role on AWS

Sign in to the AWS Management Console. Create the role with a placeholder trust policy. You will update it with the actual BigLake service account ID after you create the federated catalog in Google Cloud.

aws iam create-role \
    --role-name bigquery-cross-cloud-role \
    --max-session-duration 43200 \
    --assume-role-policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["PLACEHOLDER"],
            "accounts.google.com:aud": ["PLACEHOLDER"]
          }
        }
      }]
    }'

The --max-session-duration 43200 allows sessions up to 12 hours, which is needed for long-running BigQuery queries.

Attach permissions

The permissions policy differs based on your access control approach. For the Lake Formation approach, attach the following policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "GlueRead",
      "Effect": "Allow",
      "Action": [
        "glue:GetCatalog", "glue:GetDatabase", "glue:GetDatabases",
        "glue:GetTable", "glue:GetTables", "glue:GetPartition", "glue:GetPartitions"
      ],
      "Resource": [
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:catalog/s3tablescatalog/<TABLE_BUCKET>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:database/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>",
        "arn:aws:glue:<AWS_REGION>:<AWS_ACCOUNT_ID>:table/s3tablescatalog/<TABLE_BUCKET>/<NAMESPACE>/*"
      ]
    },
    {
      "Sid": "S3TablesRead",
      "Effect": "Allow",
      "Action": [
        "s3tables:GetTableBucket", "s3tables:ListTableBuckets",
        "s3tables:ListNamespaces", "s3tables:GetNamespace",
        "s3tables:ListTables", "s3tables:GetTable",
        "s3tables:GetTableMetadataLocation", "s3tables:GetTableData"
      ],
      "Resource": [
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>",
        "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>/*"
      ]
    },
    {
      "Sid": "LakeFormationCredentialVending",
      "Effect": "Allow",
      "Action": ["lakeformation:GetDataAccess"],
      "Resource": "*"
    }
  ]
}

Grant Lake Formation permissions

Lake Formation permissions work as a layered grant model: you grant access at each level of the catalog hierarchy, from catalog down to table. The cross-cloud role needs DESCRIBE on the catalog and database so it can discover what exists, and SELECT plus DESCRIBE on the table so it can read the actual data. Without grants at every level, Lake Formation denies access even if the IAM policy allows it.

If using Lake Formation, grant the bigquery-cross-cloud-role access to your tables:

Grant catalog permission: DESCRIBE.
Grant database permission: DESCRIBE.
Grant table permission: SELECT, DESCRIBE.

Grant Lake Formation permissions on the cross-cloud role (one-time).

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Catalog":{"Id":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Database":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","Name":"<NAMESPACE>"}}'     --permissions '["DESCRIBE"]'     --region <AWS_REGION>

aws lakeformation grant-permissions     --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<AWS_ACCOUNT_ID>:role/bigquery-cross-cloud-role"}'     --resource '{"Table":{"CatalogId":"<AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>","DatabaseName":"<NAMESPACE>","Name":"orders"}}'     --permissions '["SELECT","DESCRIBE"]'     --region <AWS_REGION>

Before granting Lake Formation permissions, revoke the default IAMAllowedPrincipals access. By default, Lake Formation grants IAMAllowedPrincipals full access to all databases and tables, so you first need to revoke this to enforce fine grain access. IAMAllowedPrincipals provides backward compatibility when you start using Lake Formation permissions to secure the Data Catalog resources that were earlier protected by IAM policies for AWS Glue.

Set up Lake Formation for external engines

For table metadata to sync from Glue to BigLake/BigQuery, the following Lake Formation settings are required. You might notice that a similar setting also appeared during the S3 Table integration setup. The first one registers the data location and enables external access at the catalog level, while this one enables the Lake Formation credential vending mechanism at the account level for all external engines. For a clean cross-cloud setup, we recommend that you enable both.

In the Lake Formation console, choose Administration, then Application integration settings, and then select Allow external engines to access data in Amazon S3 locations with full table access.

Figure 4: Enabling external-engine access in Lake Formation application integration settings

Connecting BigQuery to S3 Tables

With the AWS side configured, create the federated catalog in Google Cloud that connects BigQuery to the AWS Glue IRC.

Create the federated catalog

Authenticate to Google Cloud using gcloud auth login, or use Cloud Shell, which is pre-authenticated. Verify the BigLake API is enabled:

gcloud services enable biglake.googleapis.com --project="<GCP_PROJECT_ID>"

For Lake Formation mode (with credential vending):

gcloud alpha biglake iceberg catalogs create <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --catalog-type=federated \
    --federated-catalog-type=glue \
    --glue-aws-region=<AWS_REGION> \
    --glue-aws-role-arn=arn:aws:iam::<ACCOUNT_ID>:role/bigquery-cross-cloud-role \
    --glue-warehouse=<ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET> \
    --primary-location=<GCP_REGION> \
    --credential-mode=vended-credentials

The --glue-warehouse parameter uses the format <AWS_ACCOUNT_ID>:s3tablescatalog/<TABLE_BUCKET>. This tells the AWS Glue IRC to scope requests to your specific S3 Tables bucket within the federated catalog hierarchy.

The --credential-mode=vended-credentials flag (Lake Formation mode) instructs BigQuery Lakehouse to request scoped temporary credentials from Lake Formation rather than using the role’s IAM permissions directly for data access.

Retrieve the BigLake service account ID

After catalog creation, Google provisions a dedicated service account for your federated catalog. Retrieve its numeric ID:

BIGLAKE_SA_ID=$(gcloud alpha biglake iceberg catalogs describe <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --format="value(biglake-service-account-id)")
echo $BIGLAKE_SA_ID

Update the AWS trust policy

Back on AWS, replace the placeholder in the IAM role’s trust policy with the actual service account ID:

aws iam update-assume-role-policy \
    --role-name bigquery-cross-cloud-role \
    --policy-document '{
      "Version": "2012-10-17",
      "Statement": [{
        "Effect": "Allow",
        "Principal": {
          "Federated": "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com"
        },
        "Action": "sts:AssumeRoleWithWebIdentity",
        "Condition": {
          "StringEquals": {
            "accounts.google.com:sub": ["<BIGLAKE_SA_ID>"],
            "accounts.google.com:aud": ["<BIGLAKE_SA_ID>"]
          }
        }
      }]
    }'

Register the service account ID in the OIDC provider’s audience list. Without this step, AWS rejects the token because the aud claim doesn’t match any registered client:

aws iam add-client-id-to-open-id-connect-provider \
    --open-id-connect-provider-arn "arn:aws:iam::<AWS_ACCOUNT_ID>:oidc-provider/accounts.google.com" \
    --client-id "<BIGLAKE_SA_ID>"

Set up metadata sync

Wait 3–5 minutes for IAM changes to propagate globally, then set up background refresh:

gcloud alpha biglake iceberg catalogs update <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" \
    --refresh-interval=300s

The --refresh-interval (300 seconds in this example) determines how often BigQuery syncs metadata from the AWS Glue IRC. New tables and schema changes appear in BigQuery within this interval.

Querying from BigQuery

After the catalog refresh completes, BigQuery automatically creates external datasets corresponding to the synced namespaces. No manual CREATE SCHEMA is required.

Verify the sync:

gcloud alpha biglake iceberg namespaces list \
    --catalog="<FEDERATED_CATALOG_NAME>" \
    --project="<GCP_PROJECT_ID>"

Run a query in BigQuery:

SELECT * FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders` LIMIT 1000

Sample Query Output:

SELECT
    customer_id,
    COUNT(*) as order_count,
    SUM(amount) as total_spend
FROM `<GCP_PROJECT_ID>.<FEDERATED_CATALOG_NAME>.<NAMESPACE>.orders`
GROUP BY customer_id
ORDER BY total_spend DESC

Figure 5: BigQuery query results returned through Lake Formation credential vending

Schema evolution

When new columns are added to an Iceberg table on the AWS side (through Spark, Athena, or the AWS Glue IRC), the schema change is captured in Iceberg’s metadata. On the next Lakehouse refresh cycle, BigQuery picks up the new columns automatically. No DDL changes are needed in BigQuery.

Metadata freshness

The s3tablescatalog catalog in AWS Glue is a federated catalog that resolves table metadata live from the S3 Tables service on each request. When a streaming job commits new data to an S3 Table, the latest metadata is immediately available through the AWS Glue IRC. BigQuery sees the update on its next refresh cycle (as configured by --refresh-interval).

OIDC identity federation

The trust relationship between Google Cloud and AWS uses OpenID Connect. When BigQuery Lakehouse needs to access your data, it presents a signed JWT token containing:

iss: accounts.google.com (the issuer)
sub: The BigLake service account ID (identifies which catalog is making the request)
aud: The same service account ID (the intended audience)

Network path

Clean up

To avoid ongoing charges, remove the resources created in this walkthrough.

On AWS:

# Delete the table (if created for this walkthrough)
aws s3tables delete-table \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace analytics --name orders --region <AWS_REGION>

# Delete namespace and table bucket
aws s3tables delete-namespace \
    --table-bucket-arn "arn:aws:s3tables:<AWS_REGION>:<AWS_ACCOUNT_ID>:bucket/<TABLE_BUCKET>" \
    --namespace <NAMESPACE> --region <AWS_REGION>

aws s3tables delete-table-bucket --name <TABLE_BUCKET> --region <AWS_REGION>

# Delete IAM role and OIDC provider (if no longer needed)
aws iam delete-role --role-name bigquery-cross-cloud-role

On Google Cloud:

gcloud alpha biglake iceberg catalogs delete <FEDERATED_CATALOG_NAME> \
    --project="<GCP_PROJECT_ID>" --location=<GCP_REGION>

Conclusion

This post demonstrated how to query Amazon S3 Tables from Google BigQuery using AWS Lake Formation credential vending, where Lake Formation manages the permissions and issues temporary, scoped credentials for data access. With the open Iceberg format, you can write data once on AWS and read it from supported engines that speak Iceberg, including BigQuery.

Together with the IAM approach covered in Part 1, two access control modes provide flexibility: IAM for teams who want a straightforward setup and Lake Formation for organizations with complex governance requirements where multiple engines need centrally managed access to the same data.

To get started with this pattern in your environment:

Review the Amazon S3 Tables documentation for table bucket setup.
Follow the Google Cloud cross-cloud Lakehouse setup guide for the BigQuery federation configuration.
Explore the AWS Glue Iceberg REST Catalog APIs for programmatic access patterns.

About the authors

GPU-accelerated Apache Spark with Amazon EMR and NVIDIA RTX PRO 4500 on Amazon EC2 G7 instances runs up to 3.7x faster

McCall Peltier — Tue, 25 Aug 2026 16:17:45 +0000

For years, Apache Spark has been the backbone of large-scale data processing. However, as datasets grow and artificial intelligence and machine learning (AI/ML) pipelines become more complex, modern workloads demand more computational power. Feature engineering for machine learning models, large-scale extract, transform, and load (ETL) transformations, and real-time analytics workloads are computationally intensive by nature. GPU-accelerated instances improve performance and transform jobs that once took hours into minutes, so you can iterate on models faster and reduce operational costs. You can process larger datasets in single batches, make decisions in real time, and achieve strong performance without over-provisioning infrastructure.

We’re excited to share the benchmarking results on Amazon EMR with Amazon Elastic Compute Cloud (Amazon EC2) G7 instances, powered by NVIDIA RTX PRO 4500 Blackwell Server Edition GPUs. For data engineers and data scientists running Apache Spark workloads, this means faster pipelines, shorter iteration cycles, and more time spent on insights.

Amazon EMR on EKS natively supports the NVIDIA cuDF plugin for Apache Spark. This support is the result of joint engineering between AWS and NVIDIA to qualify the cuDF plugin for Amazon EMR, co-optimize Spark execution paths for RTX PRO 4500, and validate performance at scale through shared TPC-DS benchmarking on Amazon EC2 G7 instances. Now, Apache Spark workloads on Amazon EMR on EKS run up to 3.7x faster with Amazon EC2 G7 GPU instances than with comparable CPU instances, and require no changes to existing Spark code.

In the TPC-DS 3 TB benchmark, at the 64 GB memory tier, EC2 G7 instances with RTX PRO finished in 4.7 minutes. If you run large-scale data processing pipelines, you can cut job run times by more than two-thirds while maintaining full compatibility with the applications you already have in production.

The use cases that benefit most are those where speed directly unlocks business value. In AI/ML feature engineering, faster Spark jobs mean data science teams can iterate on features more quickly, reducing the time from raw data to trained model. In complex ETL pipelines, like financial transactions, clickstream aggregation, or supply chain data consolidation, GPU acceleration compresses multi-hour batch windows into near-real-time processing. For real-time analytics, teams running fraud detection, personalization engines, or operational dashboards can process larger volumes of data within tighter latency windows, without redesigning their architecture.

Beyond data analytics, the G7 instances will support a broad range of AI and graphics workloads, including conversational AI, content generation, recommender systems, and video streaming and rendering. Built on the AWS Nitro System, they deliver the security and resource efficiency that production AI, analytics, and graphics workloads demand.

The following sections walk through the cluster configuration, benchmark methodology, and performance results.

Cluster configuration

We benchmarked four instance types to measure the real-world performance of G7 GPU instances against comparable CPU instances for Spark SQL performance. The g7.4xlarge also provides 80 Gbps network bandwidth (compared to 15–17 Gbps on the CPU baselines) and uses RapidsShuffleManager. However, CPU runs showed no evidence of being network- or shuffle-bound at this cluster scale. All tests used Amazon EMR on EKS 7.12.0 with Apache Spark 3.5.6 and cuDF plugin 26.04.2, running the full TPC-DS benchmark at 3 TB scale across 103 queries. Each experiment ran 5 iterations. We report the median. Data was stored as Parquet on Amazon Simple Storage Service (Amazon S3) (same-region gateway endpoint). All instances were launched in a single Availability Zone.

Instance specifications

All four instance types share the same compute footprint of 16 vCPUs and 64 GB system RAM. The g7.4xlarge additionally includes an NVIDIA RTX PRO 4500 Blackwell GPU with 32 GB of dedicated video memory (VRAM), which the cuDF plugin uses to accelerate Spark SQL operations. The baseline for all speedup and cost comparisons is m9gd.4xlarge (Graviton), the lowest-cost CPU instance in the group.

.	g7.4xlarge	m9gd.4xlarge	m8id.4xlarge	m8a.4xlarge
Architecture	x86_64	arm64 (Graviton)	x86_64	x86_64
vCPU	16	16	16	16
RAM	64 GB	64 GB	64 GB	64 GB
GPU	1× RTX PRO 4500 Blackwell (32 GB VRAM)	–	–	–
NVMe	875 GB	950 GB	950 GB	EBS only (GP3 16k IOPS and 2000 MB/s throughput to match NVMe
Network	80 Gbps	Up to 17 Gbps	Up to 15 Gbps	Up to 15 Gbps

The g7.4xlarge uses the RTX PRO 4500 Blackwell Server Edition GPU. The CPU baselines cover all three major architectures: m8id.4xlarge (Intel x86), m8a.4xlarge (AMD x86), and m9gd.4xlarge (Graviton arm64).

Spark configuration

All instances used eight executor nodes with the following configuration:

Configuration	GPU instances	CPU instances
Amazon EMR release	emr-7.12.0-spark-rapids-latest	emr-7.12.0-latest
executor.cores	14	14
executor.instances	8	8
executor.memory	20G	20G
executor.memoryOverhead	30G	30G
spark.plugins	com.nvidia.spark.SQLPlugin	–
rapids.memory.pinnedPool.size	8G	–
rapids.sql.concurrentGpuTasks	3	–
shuffle.manager	RapidsShuffleManager	default (sort)
sql.adaptive.enabled	true	true
io.compression.codec	zstd	zstd

CPU instances use the same 30 GB memoryOverhead as GPU to make sure that the memory comparison is apples-to-apples. This setting reserves off-heap memory for shuffle and caching on both sides.

For GPU instances, the cuDF plugin offloads Spark SQL operations to the GPU automatically. No code changes are required. The executor.memoryOverhead value is set higher on GPU instances to accommodate GPU memory management and the RAPIDS shuffle manager.

The cuDF plugin automatically falls back to CPU execution for unsupported operators and user-defined functions (UDFs). Your job still completes, but those stages run without GPU acceleration. To identify which operations run on GPU compared to CPU, set spark.rapids.sql.explain=NOT_ON_GPU in your Spark configuration. For a pre-migration assessment of your workloads, use the NVIDIA cuDF tool to estimate GPU acceleration potential before moving to G7 instances.

To tune settings like concurrentGpuTasks and pinnedPool.size, use the Spark History Server on Amazon EMR on EKS, which provides per-stage execution details to identify CPU fallback and shuffle bottlenecks.

Getting started

Reference the Using cuDF Accelerator for Apache Spark with Amazon EMR on EKS for detailed setup instructions.

Prerequisites

Before running GPU-accelerated Spark on Amazon EMR on EKS, make sure the following are in place:

Amazon EMR on EKS release version 6.9.0 or later (this post uses emr-7.12.0-spark-rapids-latest).

The -spark-rapids release variant ships the NVIDIA cuDF plugin pre-installed.

Amazon Elastic Kubernetes Service (Amazon EKS) cluster with a GPU-enabled node group using G7 instances.
Node AMI: AL2023_x86_64_NVIDIA (Amazon EKS optimized accelerated AMI).

NVIDIA device plugin installed in the cluster to expose GPUs to Kubernetes pods:

kubectl apply -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.9.0/nvidia-device-plugin.yml

Amazon EMR on EKS virtual cluster registered to the EKS namespace.

To validate GPU availability on your nodes:

kubectl get nodes "-o=custom-columns=NAME:.metadata.name,GPU:.status.allocatable.nvidia\.com/gpu"

Note: Getting started with GPU-accelerated Spark on Amazon EMR is straightforward. To use the latest cuDF plugin, overlay the latest version (for example, 26.04.2 as of May 2026) onto the Amazon EMR RAPIDS image using an initContainer technique. This replaces the bundled cuDF JAR with a newer version while preserving all other Amazon EMR dependencies. We recommend using the latest Amazon EMR release to get the most up-to-date cuDF plugin for better performance. In our benchmarks, upgrading from cuDF plugin 25.08.0 to 26.04.2 reduced runtime by 36–38 percent. Download the latest cuDF plugin JAR from the NVIDIA repository. AWS Support covers Amazon EMR. For issues specific to a cuDF JAR, file a GitHub issue or contact NVIDIA at cudf-spark-support@nvidia.com.

Performance benchmarks and cost efficiency

We ran the full TPC-DS benchmark suite (103 queries) at 3 TB scale on 8-node clusters in us-east-1. The following table summarizes the results:

.	GPU instances	CPU instances
Cost per run	$2.06	$2.93–$3.18
Total time (103 queries)	281s (4.7 min)	1,010–1,043s (16.8–17.4 min)
Speedup compared to CPU instances	3.7×	baseline

Cost per run is the total cluster cost for the benchmark’s duration: Cluster $/hr × (median runtime ÷ 3,600). The hourly rate combines the EC2 On-Demand cost for all 8 nodes and the Amazon EMR on EKS charge for the vCPU and memory the Spark pods consume. Both are billed per second (one-minute minimum), so you pay only for what a job uses while it runs. All runs used Amazon EMR on EKS 7.12.0 in us-east-1, with 8 × 4xlarge nodes (128 vCPU) on both the GPU and CPU sides. The g7.4xlarge cluster runs at $26.35/hr (8 × $3.042 EC2 = $24.34, plus $2.01 for Amazon EMR on EKS) and finishes in 281 seconds, at $2.06 per run. The CPU clusters run at a lower hourly rate ($10.43–$10.99) but take 1,010–1,043 seconds, landing at $2.93–$3.18 per run. All prices reflect On-Demand pricing in us-east-1 as of May 2026. G7 instances are also eligible for EC2 Spot and Compute Savings Plans, which can further reduce costs for recurring batch workloads.

Cost-per-run calculations include EC2 and Amazon EMR charges only. They exclude the EKS control-plane fee, EBS volumes, S3 request and storage costs, and the driver pod.

Figure 1: Total runtime by instance type for all 103 TPC-DS queries at 3 TB scale. The g7.4xlarge with GPU acceleration completed the benchmark in 4.7 minutes, 3.7× faster than CPU instances (16.8-17.4 minutes)

Figure 2: Total cost per benchmark run, including both Amazon EC2 instance and Amazon EMR on EKS cost across all 8 nodes. Despite a ~2.5× higher hourly rate, the g7.4xlarge GPU instance costs up to 31% less per run than Graviton because it finishes the workload 3.7× faster

Where GPU acceleration excels

GPU acceleration completed the 103-query power run in 281s compared to 1,032s on CPU, an overall 3.7× speedup that saves 750 seconds per run. GPU was faster on 102 of 103 query executions.

GPU acceleration delivers the largest gains on the long-running, compute- and shuffle-heavy queries where kernel throughput outweighs launch overhead. The biggest absolute time savings:

Query	CPU time	GPU time	Speedup	Time saved
q24 (part 1+2)	81.6s	15.9s	~5.1×	65.7s
q23 (part 1+2)	79.4s	16.4s	~4.9×	63.1s
q93	63.7s	5.6s	11.4×	58.1s
q76	30.4s	3.4s	9.0×	27.0s
q64	35.4s	8.5s	4.2×	26.9s
q50	27.6s	3.5s	7.9×	24.0s

Speedup distribution across all 103 executions:

Speedup band	Queries
≥5×	15
4–5×	13
3–4×	21
2–3×	26
1–2×	27
<1× (CPU faster)	1

Median per-query speedup 2.94× (geomean 2.84×). The heaviest wins (q50, q76, q93) are aggregation- and shuffle-join-intensive queries that convert cleanly to GpuHashAggregate and GpuBroadcastHashJoin.

Where CPU wins

With RAPIDS 26.04.2, the following query showcases a workload pattern where CPU was faster:

Query	CPU time	GPU time	Ratio	Root cause
q16	0.96s	1.44s	CPU 1.5x faster	Trivial/near-empty scan. Sub-second runtime where GPU kernel-launch overhead is not amortized

Choosing the right instance

Instance	Best for	Summary
g7.4xlarge (RTX PRO GPU)	Fastest and most cost-effective	Up to 3.7× faster than comparable CPU instances and up to 31% cheaper per run. Completes in 4.7 min compared to 17.2 min. Best choice for both speed and cost efficiency.
CPU instances (m8a / m8id / m9gd)	Flexibility, availability, and always-on workloads	Multiple architecture options deliver similar Spark SQL performance. Choose CPU when GPUs are unavailable, when clusters need to remain running continuously (for example, overnight jobs ready for next-day analysis), or when workloads cannot use GPU acceleration. CPU instances offer broad availability and predictable capacity without startup delays.

G7 instances require a G-instance vCPU service quota in your account (default is often 0 for GPU types). Request a quota increase through the Service Quotas console, or use On-Demand Capacity Reservations (ODCRs) to guarantee availability for recurring batch jobs.

Based on these benchmark results, consider evaluating GPU acceleration for your own Apache Spark workloads. Start by identifying compute-intensive operations in your current pipelines, particularly those involving large-scale aggregations, joins, or machine learning feature engineering that could benefit from the performance improvements demonstrated here.

Conclusion

Amazon EMR on EKS with NVIDIA RTX PRO 4500 together provide a meaningful step forward for teams running data-intensive Spark workloads at scale. Whether you’re building ML pipelines that demand rapid feature iteration, running complex ETL transformations across massive datasets, or powering real-time analytics that can’t afford to wait on slow batch jobs, GPU-accelerated Spark on G7 delivers the performance and speed to do more. As data and AI workloads continue to evolve, GPU-accelerated analytics on Amazon EMR is becoming the foundation for data teams. Get started with GPU-accelerated Spark on Amazon EMR on EKS today by visiting Amazon EMR documentation to launch your first G7-powered cluster and see the performance gains for yourself.

About the authors

Introducing AWS Glue 6.0 for faster and more cost-effective data integration

Aarthi Srinivasan — Mon, 24 Aug 2026 19:06:39 +0000

Organizations running large data processing pipelines want lower costs, faster job runtimes, and dependable support for open table formats, without adding operational overhead. AWS Glue, a serverless, scalable data integration service that you can use to discover, prepare, move, and integrate data from multiple sources, has now launched AWS Glue 6.0, the new version of AWS Glue that addresses these needs. This version upgrade lowers AWS Glue pricing by 30% and improves performance with AWS optimized Apache Spark 4.1. It also augments developer experience with new features and adds support for Apache Iceberg V3 specifications that are suitable for enterprise adoption. The newly available AWS Glue 6.0 makes data processing workloads more manageable, faster to run, and easier to operate.

In this post, we cover the key capabilities of AWS Glue 6.0 and their performance benefits. We share code examples to help you take full advantage of the release, and we show you how to get started.

AWS Glue 6.0 highlights

AWS Glue 6.0 brings together four major improvements designed to transform how you build and run data integration workloads.

First, it upgrades the underlying runtime to Apache Spark 4.1.1, Python 3.13, Scala 2.13, and AWS SDK for Java 2.x, delivering performance improvements that can help with faster job completion times and lower costs.

Second, this release reduces current AWS Glue usage rate by 30%, and when combined with the performance improvements, you may realize an even lower effective cost.

Third, AWS Glue 6.0 introduces support for more capabilities of Apache Iceberg V3. This includes the VARIANT data type with automatic shredding, deletion vectors, row lineage tracking, nanosecond timestamps, and geo types. With these capabilities, you can build modern lakehouse architectures on the latest open table format standards.

Finally, new features like Spark Declarative Pipelines, Real-Time Mode for streaming and Python virtual environments with S3 caching are designed to further improve performance and developer experience. The following sections dive deeper into each of these areas.

Runtime upgrades

AWS Glue 6.0 upgrades the core runtime stack across the board, bringing newer versions of Apache Spark, Python, Scala, and the AWS SDK to your serverless data integration workloads.

Apache Spark 4.1.1 – AWS Glue 6.0 runs an AWS optimized build of Apache Spark 4.1.1, a major generational leap from Spark 3.5 on AWS Glue 5.1. This release introduces improvements focused on intent-driven data engineering, real-time streaming with sub-second latencies down to single-digit milliseconds for stateless tasks, faster PySpark performance, and expanded SQL features.
Python 3.13 – Supports Python 3.13, a stable release that brings interpreter changes, Python data model enhancements, standard library updates, and security updates.
Scala 2.13 – Upgrades to Scala 2.13 which includes a collections library overhaul, language and syntax feature changes, standard library additions, and compiler performance updates.

Reduced Pricing

AWS Glue 6.0 cuts current AWS Glue pricing by 30%. This means every job you run on AWS Glue 6.0 costs 30% less per DPU-hour compared to AWS Glue 5.1, with no changes required to your workload configuration. When you combine this pricing reduction with the performance improvements delivered by runtime upgrades, your effective cost savings can compound because jobs can complete faster and consume fewer DPU-hours on a lower price point. If you run large-scale Extract, Transform, and Load (ETL) pipelines or recurring batch workloads, this compounding effect can help reduce your monthly spend.

To quantify the comparison, we ran the industry-standard TPC-DS benchmark at 3 TB scale on Parquet data stored in Amazon Simple Storage Service (Amazon S3), using 30 G.2X workers on AWS Glue. The following table compares the results we obtained in our tests for AWS Glue 6.0 and AWS Glue 5.1. Thus, based on TPC-DS benchmark at 3 TB scale, AWS Glue 6.0 delivers up to 36% better price performance than AWS Glue 5.1.

.	AWS Glue 6.0	AWS Glue 5.1
Estimated Cost ($)	USD 5.61	USD 8.87

Table 1: 3TB TPC-DS benchmark comparison between AWS Glue 6.0 and AWS Glue 5.1

Updated Open Table Format (OTF) support

AWS Glue 6.0 ships with updated versions of all three major open table formats – Iceberg 1.11.0, Hudi 1.1.1, and Delta Lake 4.2.0 – providing better performance, improved merge-on-read capabilities, streamlined concurrency control, and expanded SQL compatibility.

Besides supporting the latest open table format versions, AWS Glue 6.0 delivers Apache Iceberg V3 specification that is suitable for enterprise use. The highlight is Variant shredding, which AWS Glue uses to automatically decompose semi-structured data into physically optimized, columnar sub-fields, which should result in faster query read performance. Combined with deletion vectors for efficient row-level updates, UNKNOWN column types, default column values, and richer data type support, AWS Glue 6.0 is designed to make your open data lakes faster, more flexible, and more cost-efficient. AWS Glue 6.0 also adds support for geospatial data types (Geometry and Geography) and nanosecond-precision timestamps from the Apache Iceberg V3 specification, neither of which are currently supported in open-source Apache Spark 4.1. Additional features like row lineage tracking round out the Apache Iceberg V3 capabilities available on AWS Glue 6.0.

In the following sections, we illustrate select capabilities from Apache Iceberg V3 specification on AWS Glue 6.0.

VARIANT column type

Apache Iceberg V3 introduces the Variant type to store semi-structured data (think JSON, XML, logs, and deeply nested event data) in a compact binary format. Variant shredding is designed to automatically decompose VARIANT columns into physically optimized, columnar sub-fields, facilitating predicate pushdowns and reducing scan overhead. It aims to provide simpler management of semi-structured data, without the need for complex flattening logic. With Variant type, you get the flexibility of embedding a JSON data type in your table columns while shredding is designed to help accelerate read queries and reduce costs.

UNKNOWN column type

The UNKNOWN type in Apache Iceberg V3 acts as a flexible placeholder for columns where the data type is not yet determined at the time of table creation or data ingestion. Tables can accept all-null data initially, and the column type can be upgraded later without breaking ingestion pipelines or consuming applications. This can simplify schema evolution for rapidly changing data sources. Apache Iceberg V3’s UNKNOWN column type maps to Spark 4.1’s VOID type.

DEFAULT column values

Apache Iceberg V3’s DEFAULT column values allow specifying a default value for a column in the table metadata. When you add a new column, the query engine is designed to automatically apply this default to older rows, without rewriting data or running manual backfill operations.

The following code demonstrates creating an Apache Iceberg V3 table that uses VARIANT and UNKNOWN types, and DEFAULT values for a column.

Prerequisites

To get started with this code example, make sure you have the following prerequisites.

An AWS account.
An AWS Identity and Access Management (IAM) role with permissions for AWS Glue, the AWS Glue Data Catalog, and Amazon S3. For more information, see Setting up IAM permissions for AWS Glue. This will be the AWS Glue job execution role.
An S3 bucket to store the Iceberg table data.

Steps

To create an AWS Glue 6.0 job, use the following steps.

Log in to your AWS account and open the AWS Glue console.
Create a new ETL job, with Script editor option.
1. Choose engine as Spark in the drop-down menu.
2. Start fresh, Create script and copy-paste the following code.
3. Replace the demo S3 bucket name with your bucket name in the code.

# Example pySpark script for testing few Iceberg v3's new data types
from pyspark.sql import SparkSession

CATALOG = "glue_catalog"
DATABASE = "sample_glue6_iceberg_db"
TABLE_NAME = "sample_glue6_table"
TABLE = f"{CATALOG}.{DATABASE}.{TABLE_NAME}"
TABLE_LOCATION = "s3://amzn-s3-demo-table-bucket/glue6blog-newdatatypes/"

# Configure Spark to use Apache Iceberg with the AWS Glue Data Catalog.
spark = (
    SparkSession.builder
    .appName("Glue6NewDataTypes")
    .config("spark.sql.extensions", "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions")
    .config(f"spark.sql.catalog.{CATALOG}", "org.apache.iceberg.spark.SparkCatalog")
    .config(f"spark.sql.catalog.{CATALOG}.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
    .config(f"spark.sql.catalog.{CATALOG}.io-impl", "org.apache.iceberg.aws.s3.S3FileIO")
    .config(f"spark.sql.catalog.{CATALOG}.warehouse", "s3://amzn-s3-demo-table-bucket/glue6blog-newdatatypes")
    .config("spark.sql.defaultColumn.enabled", "true")
    .getOrCreate()
)

spark.sql(f"CREATE DATABASE IF NOT EXISTS {CATALOG}.{DATABASE}")

# Create an Iceberg v3 table with VARIANT, unknown, and a default value.
# Spark's VOID type is stored as the Iceberg v3 unknown type.
spark.sql(
    f"""
    CREATE TABLE {TABLE} (
        record_id BIGINT,
        payload VARIANT,
        reserved_field VOID,
        status STRING DEFAULT 'active'
    )
    USING ICEBERG
    LOCATION '{TABLE_LOCATION}'
    TBLPROPERTIES ('format-version' = '3')
    """
)

# Insert two rows. The omitted columns use null and the declared default.
spark.sql(
    f"""
    INSERT INTO {TABLE} (record_id, payload)
    VALUES
    (1, parse_json('{{"event_type":"created","score":98.5}}')),
    (2, parse_json('{{"event_type":"processed","score":87.2}}'))
    """
)

# Query the row and extract values from the VARIANT column.
spark.sql(
    f"""
    SELECT
        record_id,
        variant_get(payload, '$.event_type', 'string') AS event_type,
        variant_get(payload, '$.score', 'double') AS score,
        reserved_field,
        status
    FROM {TABLE}
    """
).show(truncate=False)

spark.stop()

Provide the following details in the Job details tab.
1. A Name for the job.
2. The IAM role you have from Prerequisites (2) for the IAM role of the job.
3. Choose Glue 6.0 for the Glue version.
4. Leave the rest as defaults.The following screenshot shows the Job details tab with illustrated values in the AWS Glue console.
  
  Figure 1: Job details tab with the Glue version set to Glue 6.0
5. Scroll down. Under Advanced properties, for Job parameters, add the following additional Job parameter key-value pair:--datalake-formats=icebergThe following screenshot shows the Job parameters with the illustrated key-value pair in the AWS Glue console.
  
  Figure 2: Job parameters with the datalake-formats key set to iceberg
Save the job and choose Run.
After the job is completed successfully, from the Runs tab – Run details, you can inspect the Output logs that take you to the logs in the Amazon CloudWatch console. The following shows the sample output for the SELECT query in the script.

+---------+----------+-----+--------------+------+
|record_id|event_type|score|reserved_field|status|
+---------+----------+-----+--------------+------+
|1        |created   |98.5 |NULL          |active|
|2        |processed |87.2 |NULL          |active|
+---------+----------+-----+--------------+------+

Notice that we inserted two rows with values only in the record_id and the variant column. Variant column inserts were done using parse_json(). The reserved_field is of VOID type, hence returns NULL values. The status column is declared with a default active value and returns active, since the column was omitted during the insert operation.

Deletion Vectors
Apache Iceberg V3 replaces the traditional positional delete files used in Apache Iceberg V2 to deletion vectors. This change can help improve Merge-on-Read (MoR) performance. This shift replaces heavy, multi-file Parquet reads with highly compressed, direct binary bitmaps that can provide lower storage overhead and faster reads on delete-heavy tables. In scenarios with heavy table updates, such as streaming change data capture (CDC) from operational databases, Apache Iceberg V3 can offer read performance advantage over Apache Iceberg V2.

To validate the performance of deletion vectors, we created two identical AWS Glue streaming jobs and ingested the events into two different Iceberg tables, one in Apache Iceberg V2 and another in Apache Iceberg V3 format. The streaming CDC events were approximately 150,000 events per second, merge-on-read, update-heavy. Every micro-batch writes row-level deletes. We froze both tables at the same delete-heavy state and disabled compaction, leaving the tables with roughly 1.7 million rows in valid state out of the 26.4 million physical rows. The following table summarizes the read performance latency of the two Iceberg tables. We observed in this testing that reading from the delete-heavy Apache Iceberg V3 is at least 1.5 times faster than the reading from a similar Apache Iceberg V2 table. For larger enterprise scale Apache Iceberg V3 tables, the read performance could improve further.

Table 2 – Read latency comparison between Apache Iceberg V2 delete files and Apache Iceberg V3 deletion vectors

New ETL features

AWS Glue 6.0 introduces several additional capabilities designed to simplify how you build and manage data pipelines, some of which are discussed in the following list.

Spark Declarative Pipelines (SDP) where you define the outcomes you want for your entire data pipelines in a declarative fashion with SQL statements or Python decorators while AWS Glue handles execution flow, dependency resolution, parallelism, checkpointing, CDC, and recovery automatically. This helps you focus on business logic rather than orchestration plumbing.
Real-Time Mode (RTM) for streaming delivers continuous execution for Structured Streaming with sub-second latencies, down to single-digit milliseconds for stateless tasks. This can help support real-time use cases like fraud detection, live dashboards, and event-driven architectures without managing dedicated streaming infrastructure.
Arrow-Native UDFs/UDTFs execute Python functions directly on PyArrow batches without Pandas conversion overhead, which can result in faster performance for custom transformation logic at scale.
Recursive Common Table Expressions (CTE) adds WITH RECURSIVE queries natively, allowing graph traversals and hierarchical queries without workarounds or external libraries.
Python data source filter pushdown evaluates filters at the data source designed to minimize data movement, reduce the volume of data scanned, and improve job performance.
With Python virtual environments and S3 caching, you can provide pre-built Python dependencies, which should result in reduced startup latency for AWS Glue jobs by eliminating runtime dependency resolution. For existing jobs that use --additional-python-modules, no action is required. AWS Glue automatically handles the conversion to virtual environments when your job runs on AWS Glue 6.0.

Dependent library upgrades

The following table summarizes the key runtime and library version upgrades on AWS Glue 6.0.

Feature	Glue 6.0	Glue 5.1
Spark	4.1.1-amzn-0	3.5.6-amzn-1
Python	3.13.14	3.11.15
Scala	2.13.17	2.12.18
AWS SDK for Java	2.44.6 (Version 1.x removed)	2.35.5
Boto3	1.42.84	1.40.61
Java	17.0.20	17.0.19
Iceberg	1.11.0	1.10.0
Hudi	1.1.1	.0.2
Delta Lake	4.2.0	3.3.2
AWS Glue Data Catalog client	4.11.0	4.9.0
EMR DynamoDB connector	6.1.0	5.7.0
Arrow	18.3.0	2.0.1
Hive	2.3.10-amzn-1	2.3.9-amzn-4

Table 3: Runtime and library version comparison across AWS Glue 6.0 and AWS Glue 5.1

Getting started

To get started with AWS Glue 6.0, you can use one of the following methods.

Open the AWS Glue console and create a new ETL job, selecting Glue 6.0 for the Job version. You can also use AWS SDK, AWS Command Line Interface (CLI) and Amazon SageMaker Unified Studio for creating Glue jobs with version 6.0.
To get started with AWS Glue 6.0 interactively, use Glue Interactive Sessions within Glue Studio Notebooks (via the %glue_version magic) or connect with Spark Connect from your local IDE.
Migrate your existing AWS Glue jobs to AWS Glue 6.0 using the Spark Upgrade Agent.

Clean up

To avoid incurring costs, clean up the resources you created for this post.

Delete the Data Catalog database and the Iceberg table.
Delete the data and metadata folders of the Iceberg table from your S3 bucket.
Delete the AWS Glue job and the AWS Glue job execution IAM role.

Conclusion

AWS Glue 6.0 is designed to be faster, more cost-effective, and easier to use for building your open data lakehouse architectures and orchestrating your data pipelines. In this post, we discussed the key highlights of AWS Glue 6.0 and illustrated usage of Apache Iceberg V3 features with code samples. You can create new AWS Glue jobs on AWS Glue 6.0 or migrate your existing AWS Glue jobs to benefit from these improvements.

With Apache Spark 4.1.1, Apache Iceberg V3, Python 3.13, upgraded open table format libraries, and new streaming capabilities, AWS Glue 6.0 aims to help you build new data applications or to operate your existing data pipelines more efficiently and with less maintenance overhead.

We encourage you to test AWS Glue 6.0 in your development environment today. Check out this blog that talks about upgrading your AWS Glue jobs to AWS Glue 6.0. Also, in the coming days and weeks, look out for blogs on individual topics illustrating various features of Spark 4.1.1 and Apache Iceberg V3 on AWS Glue 6.0.

Acknowledgements: We thank the numerous engineers and leaders who helped build AWS Glue 6.0 to support customers with a highly performant Spark runtime and other value-added capabilities.

About the authors

Upgrade AWS Glue jobs to Glue 6.0 with AI-powered Spark upgrades

Prasad Nadig — Mon, 24 Aug 2026 19:06:16 +0000

Upgrading PySpark jobs to a new Apache Spark major version can introduce breaking changes. Removed configuration keys, stricter type casting, and Python library incompatibilities can cause runtime failures or silent behavior differences. With AWS Glue 6.0 now running Apache Spark 4.1 and Python 3.13, you need a reliable way to migrate your existing jobs while validating correctness.

In this post, we walk through upgrading a PySpark ETL job from AWS Glue 5.1 to AWS Glue 6.0. We use the generative AI upgrades for Apache Spark in the AWS Glue console. The upgrade analysis automatically identifies incompatibilities, iteratively resolves them, validates the result with data quality checks, and presents recommended changes for your review. AWS Glue 6.0 also delivers up to 36% better price performance* along with Iceberg v3, Spark Declarative Pipelines, Real-Time Mode, and Arrow-native Python UDFs.

What changes with AWS Glue 6.0

AWS Glue 6.0 runs Apache Spark 4.1, which introduces several behavioral changes from the Spark 3.5 runtime used in AWS Glue 5.1:

Behavior	Spark 3.5 (AWS Glue 5.1)	Spark 4.1 (AWS Glue 6.0)
ANSI SQL mode	Disabled by default	Enabled by default
Legacy Parquet datetime configs	Supported	Removed (renamed)
Python runtime	3.11	3.13

Beyond version compatibility, AWS Glue 6.0 also introduces:

Apache Iceberg v3 with VARIANT Shredding for efficient semi-structured data handling.
Spark Declarative Pipelines — agent-authorable ETL.
Real-Time Mode — single-digit millisecond streaming latency.
Arrow-native Python UDFs (PyArrow) for improved performance.
Built-in observability with structured metrics and enhanced Spark UI.
Up to 36% better price performance compared to AWS Glue 5.1*.

These runtime changes mean your existing AWS Glue jobs might encounter removed configuration keys, stricter type casting behavior, or Python package version incompatibilities when running on AWS Glue 6.0. Fixing these manually is time-consuming and error-prone. The following sections show how the generative upgrade analysis handles this automatically.

The sample job

Our example is a daily ecommerce order analytics pipeline running on AWS Glue 5.1:

What the job does:

Ingests 10,000 orders from Parquet files with INT96 timestamps (including pre-1900 historical dates from a legacy system migration).
Computes revenue metrics by casting string prices to numeric values and calculating line totals with discounts and tax.
Segments customers using recency, frequency, and monetary (RFM) scoring through mapInPandas with pandas and scikit-learn.
Writes enriched results back to Amazon Simple Storage Service (Amazon S3).

Job configuration (AWS Glue 5.1):

Glue version: 5.1
Worker type: G.1X
Workers: 10
Python modules: pandas==2.2.2, scikit-learn==1.5.0, numpy==1.26.4
Spark configs:
  spark.sql.legacy.parquet.datetimeRebaseModeInWrite=LEGACY
  spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY
  spark.sql.parquet.datetimeRebaseModeInRead=LEGACY
  spark.sql.parquet.int96RebaseModeInRead=LEGACY

This job runs successfully on AWS Glue 5.1. The following sections walk through how the upgrade analysis identifies and resolves incompatibilities when upgrading this job to AWS Glue 6.0. Before starting, confirm you have the prerequisites in place.

Prerequisites

An AWS account with access to the AWS Glue console.
An existing AWS Glue job on version 5.1 or earlier with at least one successful run.
An Amazon S3 path for storing the upgrade analysis results.

Running the upgrade analysis from the console

The following steps walk through the upgrade analysis workflow using the AWS Glue console.

Step 1: Select your job

Navigate to your job in the AWS Glue Studio console. Confirm the job has a successful run history on AWS Glue 5.1 before starting the upgrade analysis.

Figure 1: Job run status for the job on AWS Glue 5.1

Step 2: Start the upgrade analysis

From the job’s Actions menu, select Upgrade with generative AI. Configure the following:

Target AWS Glue version: 6.0.
Results S3 path: An S3 location where the analysis stores its artifacts and recommendations.

Figure 2: The Upgrade with generative AI option in the Actions menu

Configure the target AWS Glue version and the S3 results path, then choose Run.

Figure 3: The Upgrade with generative AI window for setting the target AWS Glue version and results path

Choose Run. The analysis begins by running your job on AWS Glue 5.1 to establish a baseline. It then iteratively tests the job on AWS Glue 6.0, identifies failures, applies recommended fixes, and validates the job. If the upgrade analysis cannot resolve an incompatibility within its attempt budget, the analysis stops and reports the unresolved issue for manual review. Your original job remains unchanged.

Note: The upgrade analysis executes your job multiple times (one baseline run plus one or more validation attempts), and each run consumes Data Processing Units (DPUs). For large or long-running jobs, consider using the run configuration option to specify fewer workers or a smaller dataset to optimize analysis cost.

Step 3: Monitor progress

The console displays the analysis progressing through multiple validation attempts. Each attempt either succeeds or fails with a specific error, and the upgrade analysis uses that error signal to determine and apply the appropriate fix for the next attempt.

Figure 4: Upgrade analysis progress across multiple validation attempts

What the upgrade analysis found and fixed

The analysis completed in four validation attempts, identifying and resolving three distinct incompatibilities. The upgrade uses deterministic migration rules for known config changes, and automated diagnosis for runtime or code errors.

Iteration 1: Removed Parquet legacy configuration

The analysis first sanitizes any Spark configurations that were removed in Spark 4.1. Our job used spark.sql.legacy.parquet.datetimeRebaseModeInWrite and spark.sql.legacy.parquet.int96RebaseModeInWrite, which no longer exist.

Migration rule applied: The SQL configs with the spark.sql.legacy prefix were removed in Spark 4.1. They have been renamed to their non-legacy equivalents, preserving the original values.

Recommended change:

Before:
spark.sql.legacy.parquet.datetimeRebaseModeInWrite=LEGACY
spark.sql.legacy.parquet.int96RebaseModeInWrite=LEGACY

After:
spark.sql.parquet.datetimeRebaseModeInWrite=LEGACY
spark.sql.parquet.int96RebaseModeInWrite=LEGACY

The read-side configs (datetimeRebaseModeInRead, int96RebaseModeInRead) already used the correct non-legacy names and required no changes.

However, with this fix applied, the validation run still failed because the Python module installation encountered an error on the AWS Glue 6.0 image.

Iteration 2: Python module version incompatibility

The pinned module versions (pandas==2.2.2, scikit-learn==1.5.0, numpy==1.26.4) could not be installed in the AWS Glue 6.0 Python 3.13 environment.

Error:

LAUNCH ERROR | Installation of Additional Python Modules failed

Recommended change: The upgrade analysis updated the version specifications from exact pins to minimum version constraints, allowing pip to resolve compatible versions for Python 3.13:

Before: pandas==2.2.2, scikit-learn==1.5.0, numpy==1.26.4
After:  pandas>=2.1.0, scikit-learn>=1.3.0, numpy>=1.24.0

With modules installing successfully, the job launched on AWS Glue 6.0 but encountered a runtime error.

Iteration 3: ANSI mode strict type casting

Spark 4.1 enables ANSI SQL mode by default (spark.sql.ansi.enabled=true). Our revenue calculation casts string prices to double, but approximately 1.8% of records contain non-numeric placeholder values such as “N/A”, “pending”, or “null” from the upstream system. From a business perspective, this meant 1.8% of revenue orders were silently excluded from revenue metrics. This data quality issue was invisible to the original pipeline.

On AWS Glue 5.1 (ANSI mode off), cast("N/A" as double) silently returns null. On AWS Glue 6.0 (ANSI mode on), this throws an exception:

Error:

NumberFormatException: [CAST_INVALID_INPUT] The value 'null' of the type
"STRING" cannot be cast to "DOUBLE" because it is malformed. Correct the
value as per the syntax, or change its target type. Use try_cast to
tolerate malformed input and return NULL instead. SQLSTATE: 22018

Migration rule applied: As of Spark 4.1, spark.sql.ansi.enabled is on by default. Casting a malformed value now raises CAST_INVALID_INPUT instead of returning NULL. The upgrade analysis resolved this by updating the script to use try_cast(), which safely returns NULL for malformed input while preserving ANSI mode protections for the rest of the job.

Recommended change:

Before (AWS Glue 5.1):

F.col("unit_price").cast("double")

After (AWS Glue 6.0, fixed by the upgrade analysis):

F.expr("try_cast(unit_price as double)")

This is a targeted fix that handles the known dirty data without disabling ANSI mode globally, keeping overflow detection and type safety active throughout the job.

Final validation and data quality check

After applying all three fixes, the analysis ran the job on AWS Glue 6.0 one final time and performed a data quality comparison between the AWS Glue 5.1 baseline output and the AWS Glue 6.0 output.

Result: The job completed successfully and all data validations passed with no mismatches detected between the source and target outputs.

Figure 5: Final analysis status with links to the results output path in Amazon S3

Reviewing the upgrade summary

The analysis produces a detailed summary stored in your S3 results path. This summary documents each validation attempt, the errors encountered, the migration rules applied, and the recommended configuration changes:

s3://amzn-s3-demo-bucket/scripts/auto-upgrade/ja-{analysis-id}/
    summary/
        summary.md                       # Full iteration-by-iteration report
        data_validation_summary.md       # Data quality comparison results
    artifact/
        attempt_N/
            script/main.py               # Recommended script (if modified)
            job_config_modifications.json  # Recommended parameter changes
            requirements.txt             # Updated dependency versions

The following is a snippet from the upgrade summary (summary.md) showing the recommended changes and validation attempt details:

The summary documents each validation attempt, the changes applied, and the data quality results.

Figure 6: Upgrade summary snippet showing validation attempt details, data quality, and analysis results

After reviewing the recommendations, accept the changes to upgrade your job to AWS Glue 6.0. This updates your job definition with the recommended configuration, including the renamed Spark configs, updated module versions, and any script modifications. Because the analysis has already validated the job on AWS Glue 6.0 and confirmed data quality parity with the original, your job is ready for production.

After reviewing the recommendations, you can apply the upgraded script to your job.

Figure 7: The option to apply the upgraded script to the job

Choose Apply to confirm the upgrade.

Figure 8: The Apply button that confirms upgrading the job to AWS Glue 6.0

After applying, the job definition reflects the new AWS Glue version.

Figure 9: The AWS Glue version for the job after applying the upgrade

Python virtual environments in AWS Glue 6.0

AWS Glue 6.0 introduces --python-virtual-env-storage-prefix, a service-managed virtual environment with S3 caching that simplifies Python dependency management.

For existing jobs that use --additional-python-modules, no action is required. AWS Glue automatically handles the conversion to virtual environments when your job runs on AWS Glue 6.0. Your jobs continue to work without any changes.

For new jobs on AWS Glue 6.0, we recommend using the virtual environment approach:

{
    "DefaultArguments": {
        "--python-virtual-env-storage-prefix": "s3://amzn-s3-demo-bucket/glue-venv-cache/",
        "--additional-python-modules": "pandas>=2.1.0,scikit-learn>=1.3.0,numpy>=1.24.0"
    }
}

How it works:

On the first run, AWS Glue installs your modules into a virtual environment, packages it, and caches the result to your specified S3 path (approximately 15–30 seconds of additional startup time).
On subsequent runs, AWS Glue downloads and extracts the cached virtual environment instead of running pip install.
The cache is automatically invalidated when your module list, versions, or AWS Glue version changes.

This approach provides faster cold starts after the first run, requires no Docker image management (unlike --python-virtual-env), and is entirely service-managed with no maintenance burden.

Conclusion

The generative upgrade analysis identified and resolved three distinct compatibility issues in our AWS Glue 5.1 job, so the job now runs successfully on AWS Glue 6.0 with Apache Spark 4.1:

The upgrade analysis renamed legacy Parquet datetime configuration keys (removed in Spark 4.1) to their current equivalents.
The upgrade analysis updated Python module version specifications that were incompatible with Python 3.13 to use flexible minimum version constraints.
The upgrade analysis addressed the new ANSI SQL mode default (which causes runtime failures on malformed data) with a targeted fix using try_cast() to safely handle non-numeric values while preserving ANSI mode protections.

The analysis validated that the upgraded job produces output consistent with the original, and presented all changes as recommendations for review before applying them to your job.

Next steps

Review the AWS Glue 6.0 documentation for the complete list of new features and changes. For a detailed walkthrough of what is new, see Introducing AWS Glue 6.0 for Apache Spark.
Read the Apache Spark 4.1 migration for additional behavioral changes.
Try the upgrade analysis on your AWS Glue jobs through the console.
See the Python virtual environment documentation for detailed setup guidance.
To reproduce this walkthrough, start with any existing AWS Glue job running on version 5.1 or earlier with a successful run history. No additional sample code or CloudFormation template is required.

After you have reviewed and accepted the upgrade changes, you can delete the analysis results stored in your S3 results path.

*Based on 3TB TPC-DS benchmark comparing AWS Glue 6.0 to AWS Glue 5.1.

About the authors

Long-term system tables retention in Amazon Redshift with Amazon S3 Tables

Nidhi Nayak — Thu, 20 Aug 2026 19:20:20 +0000

Amazon Redshift system tables capture a continuous stream of operational signals: every query that runs, every connection that is made. This data powers observability, performance analysis, and compliance auditing across your data warehouses. Until now, the system tables retained this critical data for only 7 days, making long-term compliance and auditing difficult without custom workarounds.

Amazon Redshift system table integration with Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), automatically delivers your system table logs data to Amazon S3 Tables and stores them in Apache Iceberg format. You can configure retention periods for Amazon Redshift system table beyond the current 7-day limit, giving you extended compliance, auditing, and cross-warehouse observability without custom ETL pipelines or cluster resource consumption. Your data is open, durable, and queryable from Amazon Redshift, Amazon Athena, AWS Glue, Amazon EMR, or other Apache Iceberg-compatible engines.

In this post, we walk through how the Amazon Redshift system table integration delivers log data to Amazon S3 Tables. This feature is supported on RA3 and RG provisioned clusters and Amazon Redshift Serverless workgroups.

The challenge

If you run Amazon Redshift, you often face operational challenges driven by the 7-day system table retention limit:

Limited query trend visibility: You want to compare how the same query performed 30 days ago compared to today. When performance shifts gradually, extended baselines enable data-driven root cause analysis rather than reactive troubleshooting.
Enable before-and-after comparisons: When you add a new workload, change instance type, or adjust Workload Management (WLM) queues, you want to measure the impact precisely. Extended retention preserves the baseline data you need.
Unlock seasonal capacity planning: Month-end spikes, quarter-close surges, and annual peaks require months of historical data to identify and plan. Extended retention reveals seasonal patterns across months and years.
Custom ETL pipeline overhead: To work around the retention limit, teams build custom pipelines that copy system table data hourly/daily into persistent tables within Amazon Redshift Managed Storage. These pipelines consume cluster resources, compete with production workloads, and require ongoing engineering maintenance. When Amazon Redshift updates system table schemas and data sharing configurations, these pipelines require manual intervention and create gaps in records.
Compliance requirements: Regulated industries are required to maintain audit trails spanning months or years. The 7-day limit requires custom infrastructure to meet these requirements. Amazon S3 Tables integration for Amazon Redshift system tables now addresses this.

How it works

Amazon Redshift system tables integration with Amazon S3 Tables is a fully managed capability that automatically writes Amazon Redshift system table data to Amazon S3 tables in Apache Iceberg format. AWS handles partitioning, compression, and retention management automatically. The log writing process runs in an isolated background process that alleviates resource contention with production workloads. AWS manages the pipelines for you.

The feature supports over 25 system views at launch – see the supported system views documentation.

Setting up

Follow these steps to enable system table integration with Amazon S3 Tables from the Amazon Redshift console:

Open the Amazon Redshift console and navigate to the System table integrations page. You can also access this from the detail page of your provisioned cluster or Serverless workgroup.
Choose Create System table integration. This launches the configuration wizard.
Select the Amazon Redshift Provisioned cluster or Amazon Redshift Serverless workgroup that you want to enable the feature on.

Figure 1: Selecting the Amazon Redshift data warehouse in the System table integration wizard
Choose the system views to publish from the Available system tables list. Select individual SYS_* views, or choose Select all supported system tables to publish all current and future supported views. If you select all, new views added in the future are automatically included without requiring a configuration change.

Figure 2: Choosing the system views to publish from the Available system tables list
Select the deployment model. Choose how data is organized in Amazon S3 Tables:
- Individual S3 table per system table per data warehouse to keep this warehouse’s data in its own set of tables.
- Shared S3 table per system table across data warehouses to consolidate data from multiple warehouses in the account into a shared set of tables.

Optionally configure encryption with an AWS Key Management Service (AWS KMS) customer managed key. By default, data is encrypted with Amazon S3-managed key (SSE-S3) encryption.
Save your changes. Amazon Redshift begins publishing the selected views to Amazon S3 Tables and continues adding new records on a fixed frequency.

To verify the integration is active:

Navigate to your cluster or workgroup detail page.
Check the integration status and the last ingestion time for each view.
You can also view the published data from the Amazon S3 Tables console.

After it’s enabled, Amazon Redshift writes log data to Amazon S3 tables periodically through an isolated background process, separate from production workloads. To start querying the retained logs, you will need to perform a one-time setup that connects your Amazon Redshift environment to Amazon S3 Tables data through AWS Glue Catalog. Complete the following steps:

Set up an AWS Identity and Access Management (IAM) role with the necessary permissions for AWS Glue Data Catalog and Amazon S3 Tables access, then associate it with your Amazon Redshift cluster or Amazon Redshift serverless namespace.
In AWS Glue Data Catalog, create a resource link that points to the Amazon S3 Tables database where your logs reside.

In Amazon Redshift, create an external schema that references the resource link:

CREATE EXTERNAL SCHEMA <schema_name>
FROM DATA CATALOG
DATABASE '<resource_link_database>'
IAM_ROLE '<iam_role_arn>';

With this in place, you can query your historical system table data using familiar 2-part notation:
```
SELECT * FROM <schema_name>.<table_name>;
```

Because access to Amazon S3 Tables is read-only, the integrity of your audit trails is inherently preserved.

For detailed setup instructions including IAM policy examples, see Registering the S3 Tables bucket with AWS Glue Data Catalog.

Your data is now in Apache Iceberg

Your system table data is stored in Apache Iceberg, an open table format, so you have the freedom to choose a compatible query engine. Your observability and auditing data works with the tool you already use.

You can analyze your operational data using:

Amazon Redshift: After the S3 table bucket is integrated with AWS Glue Data Catalog, create an external schema in Amazon Redshift pointing at the resource link to query the retained tables.
Amazon Athena: Run serverless SQL queries against historical logs with zero infrastructure provisioning.
AWS Glue: Build automated data processing and transformation jobs on top of your operational data.
Amazon EMR: Run Spark-based analytics at scale for complex cross-warehouse analysis.

Because the data is stored in open Apache Iceberg format in Amazon S3 Tables, you can query it with Amazon Redshift, Amazon Athena, AI agent skills for natural-language queries, Amazon SageMaker Unified Studio, an Iceberg-compatible engine, business intelligence (BI) tools, and observability systems.

Cost efficiency

Log delivery from Amazon Redshift to Amazon S3 Tables incurs no additional cost. You only pay for Amazon S3 Tables storage, maintenance, and querying the data with the engine of your choice.

Solution overview

The following scenarios illustrate how Amazon Redshift system tables integration with Amazon S3 Tables addresses common operational, compliance, and observability challenges across your Amazon Redshift environment. We also built a dedicated skill, querying-aws-redshift, for this feature and embedded it into the AWS MCP Server so you can query Amazon Redshift system tables from Amazon S3 Tables.

Scenario 1: Track query trends over time

With months or years of SYS_QUERY_HISTORY data retained, you can trace how individual queries perform over extended periods. You can compare execution time, queue time, and resource consumption for a query across days, weeks, or months.

You can pinpoint exactly when performance started degrading and correlate it with what changed: a new schema, a spike in data volume, or an additional concurrent workload. Extended retention turns troubleshooting into proactive, data-driven root cause analysis.

Scenario 2: Assess workload impact before and after changes

Every workload change affects your system: a new ETL pipeline, an instance type change, a Workload Management (WLM) queue adjustment, or a new team of analysts running ad hoc queries. The question is always: how did this change affect performance?

With Amazon S3 Tables integration for Amazon Redshift system table, you can make data-driven decisions with confidence. Query SYS_QUERY_HISTORY to compare execution times, queue wait durations, and concurrency scaling events from the weeks before a change versus the weeks after. If you onboarded a new reporting workload two weeks ago and want to understand its effect on existing queries, the data to confirm that is already there, with zero custom pipeline required.

Scenario 3: Build observability dashboards

Your system table data is stored in Apache Iceberg and cataloged in AWS Glue, which means an observability or business intelligence (BI) tool that reads Apache Iceberg can connect directly to it. Visualize workload distribution trends in Amazon Quick Sight for executive reporting. Use Amazon SageMaker Unified Studio for deeper analytical exploration or to power AI-driven insights from your operational data. Beyond AWS services, connect your preferred third-party observability systems and BI tools to track query volumes, monitor connection patterns, set up alerts for anomalies, or correlate Amazon Redshift operational data alongside application-level logs.

Your observability and auditing data works with tools that you already use. Direct access to durable, structured operational data, with a tool you prefer.

Scenario 4: Plan capacity with seasonal context

Workload demand varies throughout the year. Month-end close, quarter-end reporting, annual planning cycles, and promotional events all create predictable usage spikes, but only if you have enough historical data to see the pattern.

With extended retention, you can analyze utilization trends across multiple business cycles. Identify when you consistently approach capacity limits, measure how demand shifts quarter over quarter, and validate whether your provisioned resources align with actual usage.

Scenario 5: Maintain compliance audit trails

For regulated industries, extended retention delivers a fully managed audit trail with built-in integrity.

SYS_CONNECTION_LOG records every authentication attempt. SYS_USERLOG captures user account changes. SYS_QUERY_HISTORY documents every query executed against your warehouse.

Configure retention to match your organization’s data retention policies: whether that is 90 days, one year, or multiple years. The read-only access policy helps prevent records from being altered after they are written, including by administrators.

Scenario 6: Centralize fleet observability across your warehouse

If you run multiple Amazon Redshift warehouses, you benefit from a unified view of operational data. The feature supports two deployment patterns to match your organizational structure:

Individual tables per warehouse: Each warehouse writes to its own dedicated Amazon S3 tables, providing complete data isolation for compliance-sensitive environments. To query multiple warehouses, a UNION operation is required.
Shared tables: Warehouses across the same account and same AWS Region write to a single shared set of Amazon S3 tables, with data distinguished by the warehouse_name column. Filter by warehouse for instant cross-cluster analysis.

Best practices

Identify warehouses with logs requiring isolation for privacy reasons and select the individual table per warehouse option for those. For the remaining warehouses, use the Shared tables (consolidated) option for ease of management.
Align retention duration with your compliance requirements. Configure the minimum retention period that satisfies your compliance requirements to reduce storage costs.
When querying retained system tables, filter on metadata columns such as warehouse_account_id, warehouse_region_name, warehouse_namespace_arn, warehouse_name, and s3_tables_ingestion_time to reduce scan scope and improve performance. This is particularly important when querying large volumes of historical data across multiple warehouses.
Rely on the built-in read-only access for audit trail integrity. Use the Amazon S3 Tables configuration APIs to manage retention and encryption settings.
Plan your encryption strategy early. Choose your encryption key carefully at setup, as changes require recreating the integration. If you anticipate consolidating warehouses in the future, choose a shared AWS KMS key from the start.

Conclusion

Amazon Redshift system table integration with Amazon S3 Tables replaces custom ETL pipelines with a fully managed solution to preserve your Amazon Redshift operational data. With automatic Apache Iceberg-based storage, open format queryability, and built-in audit integrity, you get months or years of observability data, fully managed. You can enable it through the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDKs.

To learn more, visit the Amazon Redshift system tables documentation.

About the authors

Track SageMaker Unified Studio project costs with custom tags and AWS CUR

Nisha Gambhir — Thu, 20 Aug 2026 16:19:24 +0000

Organizations running machine learning (ML), analytics, and generative AI workloads on Amazon SageMaker Unified Studio domains and projects face a common cost governance challenge. System tags (AmazonDataZoneDomainId and AmazonDataZoneProject) are automatically propagated to all underlying project resources. However, custom tags such as CostCenter, Team, or Environment are not propagated to dynamic resources created through the Studio UI. This creates a gap when you need to report project costs grouped by custom tags.

In this post, we walk through a serverless solution that bridges this gap by enriching AWS Cost and Usage Report (CUR) data with custom project tags. By the end of this post, you can build an Amazon Quick Sight dashboard to filter and analyze Amazon SageMaker Unified Studio project costs by any custom tag dimension that you define. This gives your team the visibility to make informed spending decisions.

Solution overview

The solution consists of three automated subsystems:

Event-driven tag lookup management – An Amazon EventBridge rule captures Amazon DataZone project lifecycle events (Create, Update, Delete) and triggers an AWS Lambda function. The function maintains an Amazon DynamoDB lookup table that maps each project’s DomainId and ProjectId to its custom tags.
CUR enrichment pipeline – An AWS Glue extract, transform, and load (ETL) job reads CUR 2.0 Parquet data from Amazon Simple Storage Service (Amazon S3). The job joins each billing line item with the DynamoDB lookup table using the system tags (DomainId, ProjectId), appends the custom tag values as new columns, and writes the enriched data back to Amazon S3.
Cost visualization – An Amazon Quick Sight dashboard backed by a custom SQL dataset over Amazon Athena provides interactive cost and consumption analytics filtered by custom tags.

Architecture

The following diagram shows the end-to-end architecture:

Figure 1: SageMaker Unified Studio project custom tag cost reporting

The workflow is as follows:

An Amazon SageMaker Unified Studio administrator creates or updates a project with custom tags.
AWS CloudTrail captures the API call.
Amazon EventBridge matches the event.
The Lambda orchestrator writes the tag mapping to DynamoDB.
Separately, AWS Data Exports delivers CUR data to Amazon S3.
The AWS Glue ETL job enriches CUR line items with custom tags from DynamoDB.
The AWS Glue Crawler catalogs the enriched data.
Amazon Quick Sight visualizes costs by custom tags.

Prerequisites

Before deploying this solution, you need:

An Amazon SageMaker Unified Studio domain (you create projects after deployment).
AWS Cloud Development Kit (AWS CDK) CLI installed.
Python 3.12+.
Amazon Quick Sight Enterprise edition enabled in your account.
An AWS Identity and Access Management (IAM) user or role with permissions to deploy AWS CloudFormation stacks.

Step 1: Configure custom tags on your project profile

You configure custom tags on project profiles through the Amazon DataZone API. First, enable custom tags on your project profile:

aws datazone update-project-profile \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_PROFILE_ID \
  --region $REGION \
  --allow-custom-project-resource-tags \
  --project-resource-tags '[
  {"key": "CostCenter", "value": "default", "isValueEditable": true},
  {"key": "Team", "value": "default", "isValueEditable": true},
  {"key": "Environment", "value": "default", "isValueEditable": true}
]'

When creating or updating a project, set the tag values:

aws datazone update-project \
  --domain-identifier $DOMAIN_ID \
  --identifier $PROJECT_ID \
  --project-profile-version latest \
  --region $REGION \
  --resource-tags '{"CostCenter": "CC-100", "Team": "ML-Platform", "Environment": "Production"}'

Important: The AmazonSageMakerProvisioning-<domainAccountId> role needs an inline policy that permits your custom tag keys. Without this, project environment deployment fails.

The following is the inline policy that’s used for the custom tags shared in this post:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowCustomTagKeys",
      "Effect": "Allow",
      "Action": [
        "sagemaker:AddTags",
        "sagemaker:DeleteTags",
        "cloudformation:TagResource",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack"
      ],
      "Resource": "*",
      "Condition": {
        "ForAnyValue:StringLike": {
          "aws:TagKeys": [
            "AmazonDataZone*",
            "CostCenter",
            "Team",
            "Environment"
          ]
        }
      }
    }
  ]
}

Step 2: Activate cost allocation tags

Activate the SageMaker Unified Studio system tags as cost allocation tags so they appear in CUR data:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Active"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Active"}
]'

These tags take up to 24 hours to start appearing in CUR reports after activation.

Step 3: Deploy the infrastructure

The solution is packaged as a CDK application. Clone the GitHub repository and deploy:

# Install dependencies
pip install -r requirements.txt

# Bootstrap CDK (first time only)
cdk bootstrap aws://$ACCOUNT_ID/$REGION

# Deploy
cdk deploy

This creates the following resources:

DynamoDB table (smus-project-tag-lookup) – stores project-to-tag mappings.
Lambda function (smus-orchestrator) – processes project lifecycle events.
Amazon EventBridge rule – matches Amazon DataZone CreateProject/UpdateProject/DeleteProject events.
S3 buckets – for raw CUR and enriched CUR data.
AWS Glue ETL job (smus-cur-enrichment) – enriches CUR with custom tags.
AWS Glue Crawler – catalogs enriched data.
Amazon Simple Notification Service (Amazon SNS) topic – pipeline failure alerts.

Note: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight), so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Step 4: Configure CUR delivery

Create a CUR 2.0 export through AWS Data Exports that delivers Parquet files to the CUR S3 bucket created by the stack. The following screenshots show the complete configuration process in the AWS Billing and Cost Management console.

To create the export, follow these steps:

Go to AWS Billing and Cost Management and then choose Data Exports.
Choose Create in the upper right corner of the Exports and dashboards page. The Data Exports console shows any existing exports, their status, export type, data table, and last refresh date.
On the Create export page, under Export details, select Standard data export and enter an export name. Under Data table content settings, select CUR 2.0.
Under Data table configurations, set Time granularity to Hourly. The configuration page also lets you choose additional export content options such as including resource IDs, split cost allocation data, caller identity allocation data, and capacity reservation columns.
Under Data export delivery options, set Compression type and file format to Parquet. Under Data export storage settings, configure the S3 bucket to: smus-cur-report-{account-id}-{region} and set the S3 path prefix as needed. Choose Create to finish.

Figure 2: Data Exports page listing existing exports

Figure 3: Create export page with Standard data export and CUR 2.0 selected

Figure 4: Data table configurations with time granularity set to Hourly

Figure 5: Data export delivery options with Parquet format and S3 storage settings

Step 5: How the event-driven tag capture works

When a project is created or updated in Amazon SageMaker Unified Studio (through the Studio UI or API), the following happens automatically:

CloudTrail logs the Amazon DataZone API call.
Amazon EventBridge matches the event.
Amazon EventBridge invokes the Lambda function.
The Lambda extracts custom tags from the CloudTrail event payload.
The Lambda writes a record to DynamoDB with the DomainId, ProjectId, and all custom tag key-value pairs.

The Lambda function reads tags directly from the responseElements.resourceTags field of the CloudTrail event rather than making a separate GetProject API call. This avoids a race condition where GetProject might return empty tags while the project is in the UPDATING state.

def _extract_tags_from_event(detail):
    tags = {}
    response_elements = detail.get("responseElements") or {}
    for tag_entry in response_elements.get("resourceTags", []):
        if isinstance(tag_entry, dict) and "key" in tag_entry:
            tags[tag_entry["key"]] = tag_entry["value"]
    request_params = detail.get("requestParameters") or {}
    req_tags = request_params.get("resourceTags", {})
    if isinstance(req_tags, dict):
        tags.update(req_tags)
    return tags

Step 6: How the CUR enrichment works

The AWS Glue ETL job runs on a schedule (after each CUR delivery):

Reads CUR Parquet files from the CUR S3 bucket.
Reads all records from the DynamoDB lookup table.
Performs a left outer join on DomainId and ProjectId.
Appends custom tag columns (CostCenter, Team, Environment, and so on) to each CUR line item.
Writes enriched Parquet to the enriched S3 bucket.

Line items without a matching project in the lookup table retain all original columns with NULL custom tag values. No data is dropped.

joined_df = cur_df.join(
    lookup_df,
    on=(
        (cur_df[DOMAIN_COL] == lookup_df["domainId"])
        & (cur_df[PROJECT_COL] == lookup_df["projectId"])
    ),
    how="left_outer",
)

Step 7: Set up the Amazon Quick Sight dashboard

After the first ETL run and crawler execution, set up the Amazon Quick Sight dashboard:

python scripts/setup_quicksight.py \
  --account-id $ACCOUNT_ID \
  --region $REGION \
  --quicksight-user $QUICKSIGHT_USER_ARN

This creates a dashboard with five visuals:

Cost by Custom Tag (CostCenter) – horizontal bar chart.
Cost by Project – horizontal bar chart.
Daily Cost Trend – line chart.
Cost by Service per Project – stacked bar chart.
Usage by Project & Service – summary table.

And six interactive list filters: Domain, Project, CostCenter, Team, Environment, Service.

The custom SQL includes a CASE statement for service categorization:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_usage_amount,
  line_item_unblended_cost,
  resource_tags_user_amazondatazone_domain_id AS domain_id,
  resource_tags_user_amazondatazone_project AS project_id,
  costcenter, team, environment,
  CASE
    WHEN line_item_product_code = 'AmazonSageMaker' THEN 'SageMaker'
    WHEN line_item_product_code = 'AmazonS3' THEN 'S3'
    WHEN line_item_product_code = 'AWSGlue' THEN 'Glue'
    ELSE line_item_product_code
  END AS service_category
FROM "smus_cost_reporting"."enriched_cur"
WHERE line_item_unblended_cost > 0

Step 8: Verifying the solution

After deploying the infrastructure and setting up the dashboard, verify that each component of the pipeline is functioning correctly.

8.1 Verify Amazon EventBridge is capturing project events

Open the Amazon EventBridge console.
In the navigation pane, choose Rules.
Select the rule created by the CDK stack (for example, SmusCostReporting-ProjectTagRule).
Choose the Monitoring tab.
Confirm that the invocations are being recorded in the metrics.

Create or update an Amazon SageMaker Unified Studio project with custom tags using the following command:

aws datazone update-project \
  --domain-identifier <domain-id> \
  --identifier <project-id> \
  --custom-tags CostCenter=Engineering Team=DataPlatform Environment=Production

Within a few seconds, the Amazon EventBridge rule should show a new invocation in its metrics.

8.2 Verify DynamoDB schema and tag mappings

The DynamoDB lookup table uses a simple key schema:

Attribute	Type	Role
domainId	String	Partition Key
projectId	String	Sort Key
CostCenter	String	Custom tag
Team	String	Custom tag
Environment	String	Custom tag

Custom tags are stored as dynamic attributes. Any tag key set on a project becomes a column in the table.

8.2.1 Verify DynamoDB table contains tag mappings

Open the DynamoDB console.
Navigate to the table created by the stack (for example, SmusCostReporting-ProjectTagsTable).
Choose Explore table items.

Scan for your project with the following keys:

Partition key (domainId): <your-domain-id>
Sort key (projectId): <your-project-id>

Confirm the item contains the expected custom tag attributes (CostCenter, Team, Environment) with the values you assigned.

Alternatively, use the AWS CLI:

aws dynamodb get-item \
  --table-name SmusCostReporting-ProjectTagsTable \
  --key '{"domainId": {"S": "<domain-id>"}, "projectId": {"S": "<project-id>"}}'

8.3 Verify the AWS Glue ETL job enriches CUR data

Wait for the next CUR delivery (hourly if configured as described in Step 4).
Wait for the subsequent AWS Glue job execution.
Open the AWS Glue console.
In the navigation pane, choose ETL Jobs.
Confirm the job completed successfully (status: Succeeded).

Query the enriched data in Amazon Athena to confirm custom tag columns are populated:

SELECT
  line_item_usage_start_date,
  line_item_product_code,
  line_item_unblended_cost,
  costcenter,
  team,
  environment
FROM "smus_cost_reporting"."enriched_cur"
WHERE costcenter IS NOT NULL
LIMIT 10;

You should see rows with your custom tag values populated in the costcenter, team, and environment columns.

8.4 Verify the Amazon Quick Sight dashboard displays enriched data

Open the Amazon Quick Sight console and navigate to the dashboard created by the setup script.
Confirm that:
- The Cost by Custom Tag (CostCenter) bar chart displays cost data grouped by your CostCenter values.
- The list filters for CostCenter, Team, and Environment contain selectable values.
- Selecting a filter value correctly narrows the displayed data.
If the dashboard shows no data, verify that:
- The AWS Glue Crawler has run after the ETL job (check the crawler’s last run status in the AWS Glue console).
- The SPICE dataset has been refreshed. In the Amazon Quick Sight console, navigate to Datasets, select the dataset, and then choose Refresh now.

Figure 6 shows the Amazon Quick Sight dashboard with two side-by-side horizontal bar charts: Cost by Cost Center and Cost by Project. Domain Name and Project Name list filters appear at the top.

Figure 6: Amazon Quick Sight dashboard showing cost data by custom tags, including Cost by Cost Center and Cost by Project bar charts with Domain Name and Project Name filters

Note: The first end-to-end cycle can take up to 48 hours depending on CUR delivery timing. After the initial cycle completes, subsequent updates will flow automatically on the configured schedule.

Operational considerations

Monitoring: The Amazon SNS topic smus-cost-reporting-alerts receives notifications when the AWS Glue ETL job fails or the Lambda orchestrator encounters repeated errors. Subscribe an email address or Slack webhook to stay informed. For instructions on how to create a subscription, see Subscribing to an Amazon SNS topic.

Cost: The solution uses serverless components (Lambda, DynamoDB on-demand, AWS Glue, Amazon Quick Sight, SPICE) so you only pay for what you use. The primary cost drivers are AWS Glue ETL job execution time and Amazon Quick Sight SPICE storage.

Scaling: The DynamoDB table uses on-demand capacity and can scale to accommodate your projects. You can scale the AWS Glue ETL job by increasing the number of workers for larger CUR datasets. For more information, see Managing throughput capacity automatically with DynamoDB auto scaling.

New tag keys: When you add new custom tag keys to projects, the ETL automatically picks them up as new columns. The AWS Glue Crawler’s UPDATE_IN_DATABASE policy adds new columns to the catalog table without manual intervention.

Cleanup

Warning: The following cleanup steps will permanently delete all CUR data, project tag mappings, and Amazon Quick Sight dashboards.

To remove all resources:

# Delete Amazon Quick Sight resources
python scripts/setup_quicksight.py --account-id $ACCOUNT_ID --region $REGION --quicksight-user $QS_USER --clean

# Delete CDK stack
cdk destroy

Go to AWS Billing and Cost Management, and then choose Data Exports and delete the CUR 2.0 export created in Step 4.

Deactivate the cost allocation tags that were activated in Step 2:

aws ce update-cost-allocation-tags-status \
  --cost-allocation-tags-status '[
  {"TagKey": "AWSDataZoneProject", "Status": "Inactive"},
  {"TagKey": "AmazonDataZoneDomainId", "Status": "Inactive"}
]'

Conclusion

In this post, we showed how to build an end-to-end cost reporting solution for Amazon SageMaker Unified Studio projects using custom tags. This solution combines tag capture driven by Amazon EventBridge, CUR enrichment through AWS Glue ETL, and visualization in Amazon Quick Sight. With it, organizations can track and attribute costs by CostCenter, Team, Environment, or any custom dimension. This works even for resources created through the Studio UI that don’t receive custom tag propagation.

This solution serves as an extension to the custom tag propagation feature and reports cost for all project resources. The architecture is fully serverless, automated, and can be deployed to any AWS account using the provided CDK application.

To start building your custom tag cost reporting pipeline, visit the GitHub repository. To learn more about the underlying services, visit the Amazon SageMaker Unified Studio service page. For a related approach to custom tag governance, see Use Amazon SageMaker custom tags for project resource governance and cost tracking

References

About the authors

Secure SageMaker Unified Studio access with SAML and conditional policies

Manos Samatas — Wed, 19 Aug 2026 20:35:54 +0000

Amazon SageMaker Unified Studio is a single data and AI development environment that brings together data preparation, analytics, and machine learning (ML) development in one place. By unifying these workflows, it saves teams from managing multiple tools and makes it straightforward for data scientists, analysts, and developers to build, train, and deploy ML models while collaborating. In Amazon SageMaker Unified Studio, a domain is the organizing entity for connecting your assets, users, and their projects. With Amazon SageMaker unified domains, you have the flexibility to reflect the data and analytics needs of your organizational structure. You can create a single unified domain for your enterprise or multiple domains for different business units.

Some enterprises, especially those in regulated industries, might require limiting access to trusted networks (such as VPN CIDRs) or to managed devices that meet compliance standards through device attestation.

In this post, we demonstrate how to integrate SageMaker Unified Studio as a custom SAML application and apply conditional access policies for enforcing device compliance, IP-based restrictions, or multi-factor authentication (MFA). For this post, we use Okta as the identity provider (IdP).

Solution overview

This solution demonstrates how to integrate Amazon SageMaker Unified Studio (SMUS) with external SAML identity providers such as Okta. The integration enforces enterprise security controls, including trusted network access, device compliance, and multi-factor authentication. With this integration, organizations in regulated industries can maintain strict access controls while providing single sign-on for their data science and AI development teams. By using SAML 2.0 federation with conditional access policies, you can help make sure that only authenticated users on compliant devices from trusted networks gain access. This access applies to your SageMaker Unified Studio domains and the associated data and AI workloads.

Authentication flow for accessing SageMaker Unified Studio through SAML

The architecture diagram illustrates the secure authentication flow for accessing SageMaker Unified Studio through SAML integration:

Users typically initiate access from corporate-managed devices through VPN or trusted network connections.
The IdP authenticates the user and evaluates conditional access policies defined by your organization. Based on these policies, it checks for trusted devices, approved source IP ranges, and MFA completion. If any policy fails, the login is rejected. Otherwise, authentication proceeds.
Upon successful authentication and policy validation, the IdP generates a digitally signed SAML assertion containing user attributes and group memberships, securely delivering it to the user’s browser through HTTP POST binding.
The client browser automatically posts the SAML assertion to the AWS Security Token Service (AWS STS) sign-in endpoint. There, the AWS IAM Identity Provider validates the trust relationship with your corporate IdP through pre-configured SAML federation settings.
AWS STS validates the SAML assertion signature and authenticity. It then maps the user attributes to a specifically configured IAM role with SageMaker Unified Studio permissions, including the datazone:GetIamPortalLoginUrl permission required for domain access.
AWS STS confirms successful role assumption and generates temporary AWS credentials with a defined session duration. It then issues an HTTP redirect that returns the browser to the SageMaker Unified Studio domain with authenticated session tokens.
Users gain access to the unified environment for data preparation, analytics, and machine learning development. All activities are governed by the assumed IAM role permissions and logged for comprehensive audit trails.

Walkthrough

In this walkthrough, you create a SAML application in Okta, connect it to AWS, and configure a SageMaker Unified Studio domain to use it for authentication.

Prerequisites

Before you get started, make sure you have the following:

Familiarity with Amazon SageMaker Unified Studio.
A basic understanding of SAML 2.0.
AWS Identity and Access Management (IAM) permissions to create a domain in Amazon SageMaker Unified Studio.
Access to your SAML IdP (such as Okta or Entra ID) to create and configure a SAML application.

Step 1: Create an application in Okta

The first step is to set up a new SAML application in Okta that manages authentication for SMUS.

In Okta, go to Applications → Create App Integration, and choose SAML 2.0.
Provide an App name.
Set the Single sign-on URL to https://signin.aws.amazon.com/saml.
Set Name ID format to Persistent.
Set the Audience URI (SP Entity ID) to https://signin.aws.amazon.com/saml.
Choose Next, and finish creating the application.
Once created, copy the Metadata URL and Sign On URL. You need these in later steps.

Step 2: Create an identity provider in IAM

Now, let’s connect Okta to AWS by creating an IAM identity provider. This allows AWS to trust authentication responses from Okta.

Open the IAM console.
Go to Identity providers → Add provider.
Select SAML as the provider type.
Provide a Provider name.
In Okta, go to your application’s Sign On tab, choose Identity Provider metadata, and save the XML file. Upload it here.
Choose Add provider.
Copy the ARN of this provider. You need it when you create the role.

Step 3: Create an IAM role for Okta

Next, create an IAM role that Okta can assume. This role defines what access users have when they sign in through Okta.

In IAM, go to Roles → Create role.
Use the following trust policy (replace both instances of “{Replace with Identity provider ARN}” with the ARN you copied in Step 2):

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "{Replace with Identity provider ARN}"
            },
            "Action": "sts:AssumeRoleWithSAML",
            "Condition": {
                "StringEquals": {
                    "SAML:aud": "https://signin.aws.amazon.com/saml"
                }
            }
        },
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "{Replace with Identity provider ARN}"
            },
            "Action": "sts:TagSession",
            "Condition": {
                "StringLike": {
                    "aws:RequestTag/Email": "*"
                }
            }
        }
    ]
}

Attach a permission policy. For example:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": "datazone:GetIamPortalLoginUrl",
            "Resource": "arn:aws:datazone:<REGION>:<ACCOUNT-ID>:domain/<DOMAIN-ID>"
        }
    ]
}

Replace <REGION>, <ACCOUNT-ID>, and <DOMAIN-ID> with the corresponding values from your SageMaker Unified Studio domain ARN (arn:aws:sagemaker:<REGION>:<ACCOUNT-ID>:domain/<DOMAIN-ID>). You can find the domain ARN in the SageMaker console under Domains.

Step 4: Configure SAML assertions

To make sure AWS understands who is signing in, configure the SAML assertions in Okta.

Open your application in Okta.
Go to General → SAML Settings → Edit.
Choose Next until you reach Attribute Statements.
Add the following mappings:
- https://aws.amazon.com/SAML/Attributes/PrincipalTag:Email → user.email.
- https://aws.amazon.com/SAML/Attributes/Role → {IAMROLEARN,IdentityProviderARN}.
- https://aws.amazon.com/SAML/Attributes/RoleSessionName → user.email.

Step 5: Create an SMUS domain

Finally, let’s set up the SMUS domain and tie it all together.

Note: Creating a SageMaker Unified Studio domain incurs charges. For pricing details, see the Amazon SageMaker pricing page.

Open the Amazon SageMaker console.
Choose Create domain.
Choose Manual setup (this allows for SAML integration).
Enter a domain name, then choose Create.
In Configure SSO user access, select SAML, then choose Next.
Set the IdP SSO URL to the Sign On URL from Step 1.
Select Do not require assignments. (Access is instead managed by your IdP team through Okta or Entra.)
Choose Next, then choose Save.

To verify the integration works, open your SMUS domain and choose Sign in with SSO. You are redirected to Okta, and conditional access policies such as VPN, device attestation, or MFA apply automatically.

Open your SMUS domain URL in a browser.
Choose Sign in with SSO.
Confirm that you are redirected to Okta for authentication.
Sign in with your Okta credentials.
Verify that you are redirected back to the SMUS domain with access to your projects.

Step 6: Assign users to the Okta application

Before users can authenticate through Okta to access SMUS, you must assign them to the application.

In Okta, navigate to your SAML application.
Go to the Assignments tab.
Choose Assign, and select Assign to People or Assign to Groups.
Select the users or groups who need access to SMUS.
Choose Save and Go Back, then choose Done.

Step 7: Apply conditional access policies

Up to Step 5, we configured SMUS with an external SAML IdP. At this point, anyone assigned to the new application in your IdP can sign in and access the SMUS domain.

This is where conditional access policies come into play. Based on your organization’s governance model, you can add policies in your IdP to further control how and when users gain access. For example:

Restricting access to specific corporate IP address ranges (for example, only through VPN).
Enforcing device compliance so that only managed or secure devices can connect.
Adding MFA requirements for sensitive actions.
Applying device attestation to help assess whether the endpoint conforms to security baselines.

Most major IdPs, including Okta and Entra ID, support conditional access. You can find more details in their documentation:

These policies allow you to enforce the right level of protection, from something as simple as requiring users to connect through corporate networks to something as advanced as verifying device attestation across your fleet.

Clean up

To avoid incurring ongoing charges, delete the resources you created during this walkthrough:

Delete the Amazon SageMaker Unified Studio domain from the SageMaker console.
Delete the IAM role you created for Okta.
Delete the IAM identity provider.
Delete the SAML application in Okta.

Important: Deleting the SMUS domain permanently removes all projects, assets, and data within it. Back up any important work before proceeding.

Conclusion

By integrating SMUS with an external IdP through SAML, you can help enforce modern access controls based on your organization’s security requirements. This post walked through how to configure SMUS with a custom SAML application and pointed you toward resources for setting up conditional access policies.

With conditional access in place, you can decide, based on your organization’s needs, whether access should be limited to trusted users on trusted networks, trusted devices, or both. This approach can help provide a more secure and compliant login experience that aligns SMUS access with your company’s broader identity and security strategy.

About the authors

Querying raw log data using SQL and PPL with the optimized engine in Amazon OpenSearch Service

Kaushik Krishnan — Tue, 18 Aug 2026 18:32:05 +0000

In this post, you learn how to run fast analytical queries directly against raw log and trace data in Amazon OpenSearch Service using PPL and SQL.

Amazon OpenSearch Service is a fully managed service that helps you deploy, scale, and operate OpenSearch, the open source suite for search, analytics, and observability in the AWS Cloud. OpenSearch Service powers search and real-time analytics workloads, from lexical and hybrid search to log analytics and observability. This post focuses on log analytics, and on a practical question: how much analytical work can you do directly against raw log and trace data, without moving it or reshaping it first?

The new optimized engine in OpenSearch Service answers that question: you can point Piped Processing Language (PPL) and Structured Query Language (SQL) queries at raw log and trace data. The engine returns aggregations, filters, and scans over billions of events on the data exactly as you ingested it. In this post, you follow a single incident investigation, one query at a time. You see how the engine answers each new question, from multi-dimensional breakdowns and latency distributions to error rates and fleet sizing. No precomputed structure sits behind the results.

How the optimized engine queries raw data

The optimized engine stores data in the columnar Apache Parquet format and runs queries through Apache DataFusion, a vectorized execution engine, with Apache Calcite planning each query. Because the engine stores data in columns, an analytical query reads only the columns it touches and processes their values in batches, instead of reading each matching document in full. Alongside the columnar format, the engine also keeps an inverted index on the same data, so the query planner routes each operation to the path that serves it best: the columnar engine for aggregations and analytical scans, and the inverted index for selective search and filtering.

You ingest your logs and traces through the same Bulk API and clients you use today, and you write PPL or SQL against them as they land.

An investigation, one query at a time

The following walkthrough traces a common observability use case, root-cause analysis during a live incident, from the perspective of a site reliability engineer (SRE). The engineer notices elevated latency and a handful of error alerts, with nothing that points to a clear cause. No existing dashboard covers this particular shape of problem, so the engineer opens Amazon OpenSearch Service and starts asking questions of the raw trace data, letting each answer decide the next one. PPL suits this work well. Each command transforms the data and passes it to the next, so the engineer reads a query left to right the same way they think through the investigation.

The walkthrough uses generated OpenTelemetry (OTEL) data from a synthetic load generator, at billion-document scale. The focus is the query capability, that is, what the engineer can express and retrieve directly from raw spans, rather than the specific values in each result.

Step 1: Assess the scope

The first question in any investigation is how widespread the signal is. The engineer breaks errors down across service, HTTP method, and cloud Region in a single pass over roughly 1.1 billion spans.

source=otel-traces
| where @timestamp >= timestamp("2026-05-15 00:00:00") and @timestamp < timestamp("2026-05-18 00:00:00")
| eval e = if(status_code = 2, 1, 0)
| stats sum(e) as errors, avg(durationInNanos) as avg_ns, count() as total_count
  by serviceName, http_method, cloud_region
| sort - errors
| head 8

In plain terms, this query answers the engineer’s first question: where are the failures happening? It counts the error spans and breaks them down by service, HTTP method, and AWS Region in a single pass. Rather than guessing which service to open first, the engineer gets a ranked list of the hardest-hit combinations to investigate.

errors	total_count	avg_ns	serviceName	http_method	cloud_region
730	112,436	41,246,806	export-service	GET	us-west-2
722	111,215	41,000,227	catalog-service	PUT	eu-central-1
704	112,051	41,295,539	image-service	PATCH	us-west-2
612	93,214	41,451,145	healthcheck-service	PUT	us-east-1
609	94,314	41,418,897	auth-service	POST	us-east-1
609	94,414	41,447,444	email-service	PATCH	ap-northeast-1
593	89,726	41,047,114	payment-service	PUT	eu-central-1
581	89,854	41,195,643	file-service	PUT	ap-northeast-1

The errors spread across services, methods, and Regions, which points to a systemic pattern rather than a single misbehaving service.

Step 2: Check whether one host concentrates the failures

The spread could still reflect one saturated node or a fleet-wide condition. To tell the two apart, the engineer groups failures by exception type, service, and host across the entire index, with no time filter to narrow the scan.

source=otel-traces
| where isnotnull(exception_type)
| stats count() as total_count by exception_type, serviceName, host_name
| sort - total_count
| head 8

total_count	exception_type	serviceName	host_name
6	DeadlockDetectedException	notification-service	ip-10-0-16-34
6	IllegalStateException	api-gateway	ip-10-0-180-234
6	FileNotFoundException	cart-service	ip-10-0-90-162
6	ConnectionRefusedException	feature-flag-service	ip-10-0-8-123
5	TimeoutException	auth-service	ip-10-0-97-78
5	ConcurrentModificationException	order-service	ip-10-0-165-15
5	TimeoutException	coupon-service	ip-10-0-158-25

In this sample the counts are low and every row lands on a different host, so no single node stands out. This points to a fleet-wide pattern rather than one bad machine. On production data the same query makes the distinction directly: a code-level bug shows up across many hosts, whereas a single failing node concentrates its errors on one host_name.

Step 3: Quantify the latency distribution per service

Next, the engineer pulls a latency profile for each service. This includes count, average, minimum, and maximum duration, to see how each one behaves and how wide the spread runs.

source=otel-traces
| where @timestamp >= timestamp("2026-05-15 00:00:00") and @timestamp < timestamp("2026-05-18 00:00:00")
| stats count() as total_count, avg(durationInNanos) as avg_ns, min(durationInNanos) as min_ns, max(durationInNanos) as max_ns
  by serviceName
| sort - total_count
| head 8

serviceName	total_count	avg (ns)	min (ns)	max (ns)
event-bus	11,087,263	41,249,552	26,113	9,304,132,159
scheduler-service	9,175,964	41,251,927	21,919	13,432,040,933
cdn-service	9,173,572	41,225,385	23,468	13,768,293,306
ml-inference	9,036,753	41,289,101	40,410	14,625,084,517
compliance-service	8,274,635	41,294,694	41,915	7,462,983,016
metrics-collector	7,804,234	41,334,728	16,535	23,228,217,669
notification-service	7,688,714	41,204,635	51,562	8,695,311,374
image-service	7,674,406	41,248,069	47,473	15,350,500,299

This gives the engineer a latency fingerprint for each service: the averages sit near 41 milliseconds. But the multi-second maxima reveal a long tail consistent with requests queuing behind a slow dependency.

Step 4: Measure the error rate per service

To track a service-level objective, the engineer computes the error rate (errors against total requests) per service. The query uses an inline conditional, followed by a grouped sum and count, and a final division to produce the error rate.

source=otel-traces
| eval is_err = if(status_code = 2, 1, 0)
| stats sum(is_err) as errors, count() as total_count by serviceName
| eval error_pct = round(100.0 * errors / total_count, 2)
| sort - error_pct
| head 8

errors	total_count	error_pct	serviceName
699,358	22,415,308	3.12	payment-service
647,811	26,880,140	2.41	checkout-service
562,811	30,096,860	1.87	auth-service
316,192	24,510,990	1.29	cart-service
288,314	30,671,704	0.94	order-service
202,612	28,140,552	0.72	search-service
186,012	33,820,415	0.55	catalog-service
134,722	35,453,247	0.38	image-service

The engineer defines the error-rate metric in the query itself, and the engine computes it across the full index. The busiest paths, payment and checkout, run near 3 percent, whereas some services stay below 1 percent.

Step 5: Size the fleet footprint with SQL

Finally, the engineer sizes how much of the fleet each service spans, a capacity and impact question, and switches from PPL to SQL to express it.

SELECT serviceName,
       COUNT(*) AS total_count,
       COUNT(DISTINCT host_name) AS hosts
FROM otel-traces
GROUP BY serviceName
ORDER BY total_count DESC
LIMIT 8

serviceName	total_count	hosts
ml-inference	35,481,688	2,535
image-service	35,453,247	2,491
email-service	35,443,569	2,517
shipping-service	30,700,372	2,438
translation-service	30,490,570	2,502
auth-service	30,096,860	2,466
chat-service	25,564,111	2,449
recommendation-service	25,366,844	2,483

The query runs a COUNT(DISTINCT) over a high-cardinality field at billion-row scale, and switching languages mid-investigation costs the engineer nothing more than writing SQL instead of PPL. The host counts cluster in the approximately 2,400–2,540 range, so each service runs across a broad slice of the fleet. That confirms the earlier finding: the errors reflect a fleet-wide pattern, not a single node.

The engineer asked five questions and ran five queries, and each answer shaped the next. The optimized engine served every query directly from raw trace data, across both PPL and SQL, without a rollup table or precomputed summary behind any result.

Run these queries where you already work

You don’t need a separate tool to run the queries in this walkthrough.

Figure 1: Investigation queries and results grid in Query Workbench

Query Workbench in OpenSearch Dashboards UI gives you a dedicated editor for PPL and SQL. You write a query, run it, and read the results in a grid, using the same queries shown throughout this post. When you want to move from a written query to interactive exploration, Discover runs the same PPL and SQL against your indexes. In Discover, you can filter, expand fields, and drill into individual documents without leaving the page. The same query language works in both places, so you can start an investigation in Discover and carry it into Query Workbench, or the reverse, without rewriting anything.

Figure 2: PPL query and field list in Discover

Keep all your data and query it as it is

Querying raw data directly only helps if you can afford to keep the raw data. The optimized engine compresses observability data up to 70 percent more efficiently than the default General Purpose engine. That compression turns “keep everything and query it directly” into a practical default. You retain full-fidelity data for the questions you cannot predict in advance. You also pay less to store it than you would to store the raw JSON.

Get started

To try the optimized engine, create an Amazon OpenSearch Service domain running OpenSearch 3.5 or later. Then select the Observability use case during setup, which provisions the domain with the optimized engine.

To learn more about configuring and using the optimized engine, see Optimized for Log Analytics in the Amazon OpenSearch Service documentation. For an overview of the service, visit Amazon OpenSearch Service Log Analytics.

For more information, see the blog post Run log analytics for a fraction of the cost with the new engine for Amazon OpenSearch Service.

Give it a try and send feedback to AWS re:Post for Amazon OpenSearch Service or through your usual AWS Support contacts.

About the authors

Fresher insights, faster decisions: talabat’s near-real-time analytics across AWS and Google Cloud

Harish Ramesh — Tue, 18 Aug 2026 16:56:53 +0000

talabat is the leading everyday app in the Middle East and North Africa (MENA) region, offering customers a convenient and personalized way to order food, groceries, and other everyday essentials from a wide selection of restaurants and retailers. Founded in Kuwait in 2004, talabat has expanded its operations to the United Arab Emirates, Oman, Qatar, Bahrain, Jordan, Iraq, and Egypt, serving over seven million monthly active customers as of December 2025. talabat is headquartered in Dubai, United Arab Emirates, and in December 2024 successfully completed its initial public offering on the Dubai Financial Market (DFM). As a subsidiary of Delivery Hero SE, talabat uses global expertise to continuously enhance its service, expand its landscape, and drive innovation. With a strong network of partners and riders, talabat connects customers to what they need, when they need it – powering everyday convenience across the region.

In this post, we show how talabat built a hybrid, multi-cloud lakehouse that keeps a single Apache Iceberg copy of streaming data on AWS while enabling governed, near-real-time analytics from Google Cloud Platform (GCP).

Data at talabat

Data is the nervous system of talabat’s business. From the moment a customer hits “order” to the second their doorbell rings, talabat’s systems make split-second, data-driven decisions, instantaneously optimizing pricing, dispatch, routing, and order security. Over the years, talabat’s application grew into a landscape spanning two public clouds. Our transactional and operational backbone matured on AWS, where the engineering teams build and operate services. In parallel, a large population of analysts, data scientists, and analytics-engineering pipelines standardized on the Google Cloud Platform warehouse, Google BigQuery.

Both investments are deep, and both deliver value. So the strategic question wasn’t “which cloud do we consolidate on,” but rather “how do we make our data flow cleanly across the boundary between them.” That framing shaped everything that follows. The challenge isn’t only cross-cloud but cross-Region as well, with AWS services hosted in the EU region and the data in the GCP US region.

The following diagram shows how talabat’s data flows between the operational plane on AWS and the analytics plane on GCP.

Figure 1: Data flow between the operational plane on AWS and the analytics plane on Google Cloud

Historically, the data engineering team orchestrated the data movement between the two clouds, mandating a physical movement from AWS to GCP, EU to US. Moving this using conventional extract, transform, and load (ETL) tools and frameworks delayed and duplicated the data through multiple hops: Amazon Relational Database Service (Amazon RDS) to Amazon Simple Storage Service (Amazon S3) EU AWS Region, Amazon S3 EU to Amazon S3 US Region, and finally Amazon S3 US to BigQuery US.

Each hop was a copy, and every copy compounded risk: multiple failure points, compounding latency, redundant compute and storage, type fidelity, and most importantly, cross-Region and cross-cloud egress cost.

In short, the old design paid in dollars, latency, and reliability to solve a problem it had created for itself: it moved data so that BigQuery could read it. A classic data warehouse bottleneck. Could we use an open data lake instead? Yes. But the analytics usage is heavy on BigQuery, which limits access through an open source data lake layer. So the redesign started from the opposite premise: keep one copy on AWS and let BigQuery read it in place. That is what the rest of this post describes: a lakehouse for talabat.

Challenges

Operational systems emit a continuous stream of business events like order lifecycle changes, vendor, menu, logistics and rider signals, and payments information published to Apache Kafka on Amazon Managed Streaming for Apache Kafka (Amazon MSK). These events are encoded as Protocol Buffers and governed by backward-compatible schemas registered in Confluent Schema Registry, so producers and consumers can evolve safely over time.

The requirement on the analytics side is straightforward to state and hard to meet: make these events queryable, correctly typed, within minutes of being produced, and make them queryable from the tools each team already uses.

It’s tempting to view a two-cloud footprint as technical debt. For a real-time business like talabat, it’s simply the terrain, and each side plays to a genuine strength:

The event backbone lives on AWS. Our transactional and streaming systems publish to Amazon MSK. The lowest-latency, lowest-risk place to consume and process those events is next to them, in the same AWS Region.
The analytics estate lives on Google Cloud. Thousands of downstream models and dashboards, and the people who build them, assume BigQuery as the query surface.

Consolidating either side would mean a multi-year migration and a significant regression in capability for one group of users, all to remove a seam between ingestion and analytics. Data engineers decided to engineer the seam instead. The design goal became a single sentence: keep one physical copy of the data on AWS, and read it natively from both clouds. A hybrid data lakehouse makes the “which cloud” question an access-path detail rather than an architectural fork.

What we tried first: Cross-cloud writes on the hot path

Our first attempt inverted the flow we eventually shipped. Raw (also called Bronze) layer data was written from AWS directly into BigQuery-managed Iceberg tables on Google Cloud Storage. On paper, this placed the data closest to the largest consumer base. In practice, writing across clouds on an always-on streaming path introduced a class of problems we did not want to live with:

A cross-cloud dependency on the ingestion path. Every micro-batch was coupled to the availability and latency of a remote cloud’s write API.
Streaming write-API failures surfaced as ingestion incidents. The remote write became the fragile link, turning read-side concerns into write-side outages, the worst place to absorb them.
Preview-gated capabilities constrained the physical layout. Certain partitioning behaviors and features were not generally available, limiting how we could organize the data for cost and performance.

The lesson was clear: Shift left. The write path should be short, local, and straightforward. The cross-cloud concern belongs on the read path, where it can be made read-only, cached, and retried without affecting the ingestion. That reframing led directly to the architecture we run today.

Choosing how BigQuery would read AWS resident data

With the flow inverted (raw data on AWS, read from Google Cloud), we evaluated three ways for BigQuery to read tables that physically live on AWS. We assessed each against four criteria:

No data movement.
An open table format.
A governable trust model.
Minimal operational surface.

Approach	Assessment
Cross-cloud write to Google Cloud Storage	Continue writing bronze into BigQuery-managed Iceberg on Google Cloud Storage. We rejected this for the preceding reasons: it puts a cross-cloud dependency and cross-Region latency on the ingestion hot path.
BigQuery Omni	Query AWS resident data through the managed cross-cloud compute of BigQuery Omni. This introduced more managed surface and more constraints than we needed for a read-only bronze layer, and we wanted to own the catalog and trust model directly.
Lakehouse federated Apache Iceberg REST catalog (authenticated by IAM)	Let BigQuery read data in Amazon S3 Tables, a capability of Amazon S3 that provides managed Apache Iceberg tables, through a federated catalog that synchronizes AWS Glue Data Catalog metadata, with access authenticated by cross-cloud IAM trust. This met all four criteria, and we chose it.

The deciding properties were that the raw data doesn’t leave AWS, the format is open Apache Iceberg (so Amazon Athena, Spark, and Iceberg-compatible engines read the same tables), and the cross-cloud relationship is expressed as identity and trust rather than as a recurring copy job.

Why Amazon S3 Tables

With the architecture settled on a single Iceberg copy living on AWS, we needed a storage layer purpose-built for Iceberg at scale. Amazon S3 Tables met the requirements without adding operational surface. Table maintenance (compaction, snapshot expiration, and unreferenced file removal) runs automatically as a service-managed policy, avoiding the need for external orchestration jobs that would otherwise grow linearly with table count. Equally important, every table is an Amazon Resource Name (ARN)-addressable resource. That means IAM policies can grant or deny access for individual tables, the same least-privilege model we apply to any other AWS resource, and AWS CloudTrail records every access decision. For a cross-cloud design where the trust boundary is expressed entirely through IAM, having tables that are first-class IAM resources isn’t a convenience but a prerequisite. S3 Tables gave us managed Iceberg housekeeping and fine-grained, auditable access control in a single construct, so the engineering team could focus on the streaming logic rather than the storage plumbing beneath it.

Solution overview

The system has two halves that meet at an open table format:

A short, local write path on AWS.
A read-only cross-cloud handshake that lets BigQuery consume the data.

The single source of truth is Apache Iceberg data in Amazon S3 Tables. Every consumer reads that one physical copy.

The following diagram shows the end-to-end architecture, from event ingestion through storage to consumption paths.

Figure 2: End-to-end architecture from event ingestion through storage to consumption paths

The write path: Short, local, and reliable

We run one Amazon EMR Serverless Spark Structured Streaming job per Kafka topic (with a prebaked Docker image, emr-7.13.0 on ARM64/Graviton) in the same AWS Region (eu-west-2) as Amazon MSK. Co-locating compute with the event backbone minimizes the data transferred per micro-batch, saving cost and latency. Each job runs the Spark foreachBatch operation with a trigger interval of roughly one to five minutes and at-least-once delivery. Every micro-batch performs five steps:

Consume from Kafka.
Decode Protocol Buffers using the registered schema.
Transform to the target Iceberg schema.
Append to the Iceberg table in Amazon S3 Tables.
Commit offsets.

The cycle repeats without interruption.

This path touches only AWS. There is no cross-cloud dependency, only one deliberate cross-Region hop: compute in the Europe (London) Region (eu-west-2), storage in the US East (N. Virginia) Region (us-east-1). This incurs standard AWS inter-Region data transfer cost, a deliberate choice so that the cross-cloud read from BigQuery stays within the same Region.

Bad records don’t block the stream. They land in a dedicated dead-letter queue (DLQ) table (<table>_dlq) in a separate S3 Tables bucket, storing the raw payload (raw_value_b64) and a skip_reason. Nothing is silently dropped. The DLQ tables are registered with the AWS Glue Data Catalog through Lakehouse, so engineers can inspect failures from Amazon Athena or BigQuery.

From this point on, Amazon S3 Tables is the source of truth.

The crux: Cross-cloud handshake

This is the heart of the design. BigQuery reads the S3 Tables Iceberg data through a Lakehouse federated Apache Iceberg REST catalog, a read-only catalog on the Google Cloud side that points at the AWS resident tables. Three mechanisms make it work.

An open catalog contract (Iceberg REST).

Amazon S3 Tables exposes an Apache Iceberg REST catalog interface, and Google Lakehouse speaks that same standard. Because both sides agree on the Iceberg on-disk format and REST catalog protocol, no translation layer or data copy is required. BigQuery reads the identical Iceberg data files that Athena and Spark read.

On the Google Cloud side this is a single Lakehouse federated catalog. A table surfaces to analysts as talabat-data.s3tables-glue.catalog.orders.

Cross-cloud identity and trust (IAM and OIDC).

The Lakehouse catalog authenticates to AWS as a Google-managed service identity (the Lakehouse REST-catalog service account) that an AWS Identity and Access Management (IAM) role trusts through OpenID Connect (OIDC) federation with accounts.google.com, using sts:AssumeRoleWithWebIdentity with the service account’s numeric ID pinned in the role’s trust policy. Requests to the S3 Tables Iceberg endpoint are SigV4-signed. It’s the same AWS request-signing scheme that any AWS SDK uses, scoped to the S3 Tables service. In other words, the handshake isn’t a proprietary connector. It’s standard AWS request signing performed by a trusted external identity.

The trust is codified as infrastructure as code (IaC) on the AWS side: granted least-privilege, and revocable at any time. The following diagram shows this authentication sequence.

Figure 3: Cross-cloud authentication sequence between the Lakehouse catalog and AWS IAM

For a step-by-step walkthrough of this trust relationship, creating the IAM role, validating the token’s audience and subject, and pinning the Lakehouse service-account identity in the trust policy, see Create and manage AWS Glue federated datasets and Set up cross-cloud Lakehouse for AWS Glue.

Metadata synchronization (approximately five-minute refresh).

The federated catalog periodically synchronizes table metadata from the AWS Glue Data Catalog that fronts S3 Tables. Newly created tables and new data become visible to BigQuery on a short refresh cycle (approximately 300 seconds). Reads are served against the live Iceberg data. Only the catalog pointers are synchronized.

The result is that a table written once on AWS appears in BigQuery as an ordinary catalog object and can be queried with standard SQL, while the bytes don’t leave AWS and the format stays open.

Infrastructure as code: The cross-cloud trust surface

The following section explains the authentication handshake shown in the architecture diagram. The Lakehouse catalog service account presents a Google OIDC JSON Web Token (JWT), which AWS validates through the IAM OIDC provider, returning short-lived credentials scoped to read-only S3 Tables access.

resource "aws_iam_openid_connect_provider" "google" {
  url = "https://accounts.google.com"
  client_id_list = [var.lakehouse_sa_audience] #Lakehouse REST-catalog serviceaccount
}

Pin the trust to exactly that one identity. This is the security crux. The role can only be assumed through a Google-signed token whose subject matches our service account. A condition on the sub claim closes the door to every other principal:

data "aws_iam_policy_document" "trust" {
  statement {
    actions = ["sts:AssumeRoleWithWebIdentity"]
    principals {
      type = "Federated"
      identifiers = [aws_iam_openid_connect_provider.google.arn]
    }
    condition {
      test = "StringEquals"
      variable = "accounts.google.com:sub"
      values = [var.lakehouse_sa_subject_id] # nobody else can assume the role
    }
  }
}

resource "aws_iam_role" "lakehouse_read" {
  name = "bq-lakehouse-read"
  assume_role_policy = data.aws_iam_policy_document.trust.json
  max_session_duration = 43200 # 12-hour sessions, then re-issued
}

Grant read-only, least privilege. The assumed role carries only enough to read the catalog metadata through AWS Glue and access the Iceberg data through S3 Tables, secured entirely by IAM policy and nothing writable:

statement {
  actions = [
    "glue:Get*",
    "s3tables:GetTable", "s3tables:GetTableData", "s3tables:ListTables", "s3tables:ListTableBuckets", "s3tables:GetTableMetadataLocation", "s3tables:ListNamespaces", "s3tables:GetNamespace","s3tables:GetTableBucket"
  ]
  resources = [var.s3tables_bucket_arn, "${var.s3tables_bucket_arn}/*"]
}

The Google-side catalog is bound to this role. The Lakehouse federated catalog itself is created out of band (a one-time gcloud call), pointed at the preceding role so that every read presents that trusted identity. No AWS keys ever live in Google Cloud:
```
gcloud iceberg catalogs create s3tables-glue \
  --federated-catalog-type=GLUE --glue-aws-region=us-east-1 \
  --glue-aws-role-arn=arn:aws:iam::<account>:role/bq-lakehouse-read
```

Together these four steps are the whole handshake: a trusted issuer, a role that only our service account can assume, a least-privilege read grant, and a catalog bound to that role.

Operational lessons: Metadata as a first-class concern

Operating an open, federated catalog across clouds taught us to treat table metadata as a first-class operational concern. In practice this means:

Snapshot retention: Keeping Iceberg snapshot retention short so that per-table metadata stays compact and synchronizes reliably.
Compaction: Standardizing table maintenance (compaction and snapshot expiry) as a uniform, service-managed policy through the S3 Tables built-in maintenance configuration.
Schema evolution: When a Protobuf schema evolves (backward-compatible additions), the Spark job appends or removes columns in the Iceberg schema in S3 Tables. The federated catalog picks up the change on its next sync cycle, and BigQuery reflects the changes without manual intervention.

These are small, well-understood settings once we know how to set them, and they are the difference between a catalog that simply works and one that drifts.

Consuming the data is a choice of engine, not a choice of copy

After a source is live, the same Iceberg table is available three ways over one physical dataset.

A BigQuery user queries it in standard SQL and joins it to the rest of the Google Cloud warehouse.
An infrastructure engineer runs the identical query in Amazon Athena for ad hoc checks and continuous integration (CI) validation.
A data scientist reads the table directly with Spark, with no BigQuery or Athena in the path.

Nobody waits for a nightly export, and nobody reconciles three divergent copies. There is only one.

Performance and cost impact

The qualitative benefits are already clear:

Minutes-fresh raw data for near-real-time analytics. The previous architecture’s latency was not a volume problem. It was a design constraint. Ingestion ran every five minutes, but a downstream hourly batch job gated end-to-end freshness to 60–90 minutes. With catalog federation, that same data is queryable within minutes of being produced: under five minutes for 95 percent of events, with the option to tune the pipeline to cover 100% of events for latency-sensitive or mission-critical workloads.
One storage copy in S3 Tables, three compute engines. BigQuery, Athena, and Spark or another Iceberg-compatible engine read a single physical Iceberg dataset in Amazon S3 Tables, avoiding duplicate storage and the reconciliation tax of keeping copies in sync.
No cross-cloud egress on the hot path. Ingestion is local to AWS. The only cross-cloud traffic is read-time metadata synchronization and query reads, not a continuous write stream. Based on an internal comparison of monthly AWS and Google Cloud data-transfer charges, orchestration overhead, multi-layered ETL workflow costs, and storage backup charges, talabat reduced data-movement costs by approximately 40 percent for comparable data volumes. The comparison spanned a two-month period before and after removing the continuous replication pipeline, and the change eliminated hundreds of terabytes of recurring cross-Region and cross-cloud data transfer per month.
Open table format, no lock-in. Because the raw bronze data layer is Apache Iceberg in Amazon S3 Tables, the data isn’t captive to any single query engine or cloud. New consumers adopt it by speaking Iceberg, not by requesting an export.
Governable cross-cloud access. The cross-cloud boundary is secured by an IAM trust relationship (least-privilege, auditable, and revocable) rather than a standing data pipeline. End-user access control within BigQuery is managed separately through the native role-based access control (RBAC) in GCP and fine-grained access controls on the federated catalog.

Future enhancements

Looking ahead, we plan to broaden source coverage by onboarding the remaining high-value event streams and batch stores onto a hybrid one-configuration pattern. We’re formalizing end-to-end freshness objectives and the observability around them: batch-level metrics, dead-letter monitoring, and catalog-synchronization health. We will continue tuning snapshot retention and compaction so the cross-cloud catalog stays fast and reliable as the number of tables grows. More broadly, we intend to make “written once, read by any engine” the default for new datasets beyond the bronze layer, leaning further into open table formats as the connective tissue between cloud service providers.

Conclusion

Being on two clouds is often framed as a problem to migrate away from. It’s simply the terrain for talabat. The event backbone is prominent on AWS, and the analytics community operates on BigQuery. By making Amazon S3 Tables with Apache Iceberg the single source of truth on AWS and letting BigQuery consume it read-only through a Lakehouse federated Iceberg REST catalog secured by cross-cloud IAM trust, we turned a two-cloud constraint into a single governed dataset that engines can read within minutes. The write path stays short, local, and reliable. The cross-cloud concern lives on the read path, where it belongs, expressed as open standards and identity, not as data movement.

That is the handshake: one copy of the data on AWS, an open catalog contract, and a signed, trusted, revocable identity reaching across the cloud boundary to read it.

This post focuses on reading AWS resident data from BigQuery. For the broader multi-cloud Lakehouse pattern, including federating catalogs from other systems into the AWS Glue Data Catalog, see Multi-cloud Lakehouse architecture on AWS for agentic AI.

About the authors

Powering agentic AI with real-time streaming data on AWS

Mazrim Mehrtens — Tue, 18 Aug 2026 16:42:15 +0000

Two years ago, the conversation about streaming data and generative AI centered on a straightforward question: how do you feed real-time context into a large language model (LLM) so it can answer questions using fresh data? We explored that question in our 2024 blog post, “Exploring real-time streaming for generative AI applications,” which introduced patterns for connecting streaming pipelines to foundation models.

The landscape has shifted. Today’s generative AI systems don’t only answer questions. They observe, reason, and act. Agentic AI applications have moved from research prototype to production reality. Agentic AI-powered data pipelines now monitor streaming telemetry, detect anomalies, decide on remediation strategies, and execute actions without human intervention. They maintain memory across sessions, query live data sources on demand, and coordinate with other agents to solve complex problems.

This shift demands a fundamentally different relationship between streaming infrastructure and AI. It’s no longer enough to inject context into a prompt. You need architectures where streaming data continuously powers autonomous agent action and keeps a real-time lakehouse fresh for training and retrieval. That data also flows into multiple consumption patterns, such as generative business intelligence (BI) for humans, standardized protocols for agent queries, and proactive memory hydration for low-latency agent context.

This post introduces three architectural patterns that together form a unified streaming backbone for the agentic AI era:

Streaming feature engineering → real-time inference → action: Continuous data flows build features, invoke AI models, and act in a single pipeline.
Event-driven agent invocation: Streaming pipelines detect patterns across millions of events and trigger agentic workflows with full context already assembled.
Real-time context synchronization: Change data capture (CDC) and streaming pipelines keep agents’ memory current, so agents can respond instantly rather than making expensive external calls.

The following sections explore each pattern in depth.

Pattern 1: Streaming feature engineering → real-time inference → action

You’re watching a live football match. As a striker receives the ball in the box, AI-generated commentary appears on screen: “This is Smith’s third touch in the penalty area in the last 3 minutes. His conversion rate from this zone is 34% this season.” That insight was computed from streaming event data, passed through a feature pipeline, and fed to a generative AI model. All of this happened within the time it takes the striker to turn and shoot.

This pattern combines two capabilities that are often treated separately: using real-time data to continuously improve AI models, and using real-time data to invoke those models for immediate action. The streaming pipeline does both: it builds the features that train the model and the features that drive inference.

Streaming events (user interactions, sensor readings, game events, and transaction records) flow into Amazon Managed Streaming for Apache Kafka (Amazon MSK) or Amazon Kinesis Data Streams. Amazon Managed Service for Apache Flink processes these events through windowed aggregations (tumbling windows, sliding windows, or session windows) to produce features: rolling averages, counts, ratios, behavioral sequences, or other derived signals relevant to your use case.

These features serve two paths simultaneously:

The inference path: At the end of each window (or on each event, depending on your latency requirements), features are passed to a generative AI or machine learning (ML) inference endpoint: Amazon Bedrock for generative output, or Amazon SageMaker for custom models. The model produces a result (commentary, a recommendation, a personalization decision, or a risk score) and the pipeline acts: posting content to a user, updating a recommendation feed, sending a notification, or writing to a downstream system.

The training path: The same streaming features are continuously written to a real-time data warehouse or lakehouse such as Apache Iceberg tables on Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), that keeps training datasets fresh. Amazon SageMaker lakehouse architecture provides unified access for training jobs and fine-tuning pipelines. As new data streams in, your models can be retrained or fine-tuned on data that’s minutes old rather than days old. This matters for domains where patterns shift quickly, such as fraud detection, personalization, and industry dynamics.

Amazon S3 Tables handles the Iceberg table management automatically, including compaction, snapshot management, and metadata optimization. Your team focuses on feature logic rather than storage operations. The AWS Glue Data Catalog makes these tables discoverable across training jobs, inference pipelines, and analytics consumers. Glue Data Catalog supports business context and semantic search. This context helps models discover and select the right data asset for any given task.

Scenarios

Real-time sports commentary: Streaming game events (passes, shots, player positions) flow through Apache Flink on Managed Service for Apache Flink, which computes rolling features (possession percentage, shot frequency by zone, player heat maps). These features feed a generative AI model through Amazon Bedrock that produces natural-language commentary and statistical insights in real time. Simultaneously, the features are written to S3 Tables to improve the model’s understanding of game patterns over time.

Streaming personalization: User clickstream data flows through Managed Service for Apache Flink, which computes behavioral features (session duration, category affinity scores, recency-weighted purchase history). These features invoke a personalization model that updates the user’s experience in real time by reranking product recommendations, adjusting content feeds, or triggering targeted offers. The same features feed the lakehouse to retrain the personalization model nightly.

Figure 1: Streaming feature engineering feeding a real-time inference path and a continuous training path

Pattern 2: Event-driven agent invocation

At 2:47 AM, a pressure sensor on a manufacturing line begins drifting. Within seconds, a streaming pipeline detects the anomaly, assembles full context (device history, maintenance schedule, correlated sensor readings), and invokes an agent that opens a maintenance work order, adjusts the device’s sampling rate, and notifies the on-call engineer. All of this happens before a human sees an alert.

Pattern 1 invokes inference on every window or event. It runs continuously. Pattern 2 adds to this approach: the streaming pipeline continuously analyzes data and invokes an agentic workflow when specific conditions are met or a pattern is detected. The pipeline is the sensor. The agent is the responder. Dynamic rules are the bridge between them.

The key distinction is that the events and triggers are dynamic. They’re defined by rules programmed into the streaming pipeline or traditional ML models for prediction or detection. The pipeline determines when and how the agent is triggered, making the system fluid and adaptive. You can update detection logic without redeploying the agent. You can add new anomaly patterns without changing the response logic.

Streaming telemetry flows into Amazon MSK or Amazon Kinesis Data Streams. Managed Service for Apache Flink runs continuous anomaly-detection logic, such as statistical models, windowed aggregations, threshold-based rules, or ML-based scoring. Critically, when Flink detects an anomaly, it doesn’t only publish a raw alert. It assembles a context package: the anomaly details, relevant historical data, correlated signals from other streams, and metadata the agent needs to act immediately.

This context package is published to a downstream topic and consumed by an Amazon Bedrock AgentCore agent. Because the pipeline has already assembled full context, the agent doesn’t waste time gathering information. It can reason and act immediately. AgentCore Runtime hosts the agent, AgentCore Observability provides tracing and logging, and AgentCore Memory maintains state across invocations (so the agent knows, for example, that this is the third anomaly from this device this week).

The benefit of this pattern over a polling-based or scheduled approach is twofold:

Latency: The agent is invoked within seconds of the anomaly, not at the next polling interval.
Context richness: The pipeline has already done the work of correlating signals and assembling context. A polling-based agent would need to make multiple queries to reconstruct what the pipeline already knows.

The rules that trigger invocation are a powerful abstraction. They can be simple thresholds (“temperature exceeds 95°C”), statistical (“value deviates more than 3σ from the rolling mean”), or ML-based (“anomaly score from an embedded model exceeds 0.85”). You can update these rules dynamically by adding new detection patterns, adjusting sensitivity, or routing different anomaly types to different agents.

Figure 2: Event-driven agent invocation triggered by anomaly detection in the streaming pipeline

Pattern 3: Real-time agent context

A customer messages their bank: “Was that $847 charge at the airport legitimate?” The agent responds in under two seconds with full context (the customer’s recent travel pattern, the merchant’s fraud-risk score, and the transaction details) because all of this was already loaded into the agent’s context layer through streaming CDC. A reactive agent without this synchronization would need to make five separate API calls across three systems, taking 8–12 seconds and risking timeout failures.

This pattern addresses a fundamental question: how proactive should your agent be about gathering context?

A proactive agent has the full context, continuously synchronized with the state of the world. When a user asks a question, the agent already has the relevant knowledge from context. It responds from memory rather than making expensive external calls. A reactive agent starts cold. It knows nothing until it queries for information, making multiple calls across security boundaries, handling authentication, and stitching together data from disparate sources. For latency-sensitive use cases, where a user sends a prompt and expects a fast response, this difference is critical.

Real-time context synchronization uses CDC and streaming pipelines to keep agent memory current. The agent’s knowledge graph becomes a synchronized replica of the distributed systems it needs to reason about.

No agent is purely proactive or purely reactive. The design decision is: what data should be pre-loaded, and what should be fetched on demand? This is a spectrum, and where you land depends on three factors:

Latency sensitivity: If users expect fast, contextually relevant responses, pre-load the data the agent needs most frequently.
Data volume: Synchronizing everything is impractical. An efficient, fast search that still produces accurate results matters more than exhaustive pre-loading. Be selective about what you push.
Data freshness requirements: Some data changes every second (stock prices, session state). Other data changes rarely (customer preferences, account configuration). Load what changes frequently and matters immediately.

Streaming pipelines (Managed Flink reading from Amazon MSK, Kinesis Data Streams, or CDC streams from operational databases) continuously process events and write aggregated results to the agent’s knowledge graph, or the context layer. These stores can take multiple forms depending on your access patterns:

AWS Context automatically maps relationships across your existing data into a knowledge graph and supports agentic search so AI agents can access governed data relationships, business rules, and domain knowledge at runtime. Data stewards manage the graph through an intuitive console, reviewing inferred relationships, promoting them to production, and attaching domain-specific knowledge like business definitions and usage rules.
Amazon Bedrock AgentCore Memory for structured agent context that persists across sessions.
Amazon DynamoDB for low-latency key-value lookups (customer profiles, account state).
Amazon OpenSearch Serverless for semantic search over unstructured context (past conversations, documents).
Amazon Neptune for relationship-rich data (knowledge graph).
Amazon S3 Tables fully managed Apache Iceberg tables in Amazon S3, for interoperability between multiple query engines.

For data that isn’t pre-loaded, the agent falls back to on-demand retrieval. This applies when the data is too large, changes too rarely to justify streaming, or is needed only in edge cases. The Model Context Protocol (MCP) provides a standardized interface for this. MCP servers expose heterogeneous data sources through a uniform protocol. The agent queries MCP when it needs context that isn’t in its synchronized memory.

This same real-time context synchronization pattern serves different consumers:

AI agents access fresh context through a real-time knowledge graph or a context layer, and MCP servers (pull tier), as in the preceding sections.

Human analysts and executives access the same context layer, which can directly query Apache Iceberg tables on S3 Tables through its direct query mode. Amazon Quick chat provides natural-language access to real-time lakehouse data. No intermediate warehouse is required. This is the generative BI expression of the same underlying pattern: streaming data keeps the lakehouse current, and Amazon Quick gives humans conversational access to it.

Training and fine-tuning pipelines access the synchronized lakehouse through Amazon SageMaker Lakehouse, keeping models fresh (as described in Pattern 1).

The underlying principle is the same across consumers: streaming pipelines synchronize distributed data into accessible stores, and each consumer accesses those stores through the interface that fits their needs.

Figure 3: Real-time context synchronization serving agents, analysts, and training pipelines from shared stores

Bringing it together

The three patterns in this post form a unified architecture built on a single streaming backbone:

Pattern 1 uses streaming pipelines to build features that simultaneously drive real-time inference and keep training data fresh. Your models improve continuously while serving predictions in real time.

Pattern 2 uses streaming pipelines as intelligent sensors that detect anomalies and invoke agents with full context already assembled. This separates detection logic from response logic for maximum flexibility.

Pattern 3 uses streaming pipelines to synchronize distributed system state into the agent’s context layer, making agents more proactive and serving multiple consumers (agents, humans, and training jobs) from the same pre-loaded data.

The streaming infrastructure you build (Amazon MSK, Amazon Kinesis Data Streams, Amazon Managed Service for Apache Flink, and Amazon S3 Tables) serves all three patterns simultaneously. A Flink application can compute features for inference (Pattern 1), detect anomalies that trigger agents (Pattern 2), and synchronize state into agent memory (Pattern 3).

To get hands on with the patterns described in this post, refer to Agentic AI-Powered anomaly detection: Spotting anomalies in real-time.

You don’t need to implement all three patterns at once. Start with the one that addresses your most pressing need. But design your streaming infrastructure knowing it will serve multiple patterns. In the agentic AI era, every stream is a potential input to an agent, a model, and a human decision-maker.

About the authors

Amazon MSK simplifies configuring custom domain names

Ali Alemi — Tue, 18 Aug 2026 16:35:06 +0000

Previously, you had to manually override the advertised listener on each broker and repeat it every time a broker was added. This approach was operationally heavy and could not be implemented on a cluster in KRaft mode. With Amazon Managed Streaming for Apache Kafka (Amazon MSK), you can now configure custom domain names for your Provisioned clusters using a single property. This works for clusters in both ZooKeeper and KRaft mode. Now you define the domain once and Amazon MSK applies it across every broker, so custom domain names keep working through scaling of the MSK cluster.

Custom domain names on Amazon MSK

Amazon MSK is a fully managed service for building and running applications that use Apache Kafka to process streaming data. By default, Amazon MSK brokers advertise addresses that AWS generates (for example, b-1.cluster-name.kafka.us-east-1.amazonaws.com) to connecting clients. These addresses are unique to each cluster and change when a cluster is recreated.

Many organizations need a static, customer-controlled endpoint that stays the same regardless of the underlying cluster. They achieve this with a custom domain name, so that they can:

Route traffic through Network Load Balancers (NLBs) when IP exhaustion forces clusters into non-routable subnets.
Avoid client reconfiguration during cluster migrations, so clients keep the same endpoint even when the underlying cluster changes.
Simplify disaster recovery (DR) failover, where the same domain fronts both primary and standby clusters.
Align with organizational naming, security, and compliance conventions.

Until now, the only way to do this was to override the advertised.listeners on each broker using the kafka-configs.sh --alter tool. It required carefully preserving every internal listener and re-running that override every time a broker was added. This works, but it accepts any string with no validation. A single typo can cause an outage. It requires manual, per-broker steps with no cluster-wide mechanism. It cannot be managed through infrastructure as code, and it could not be implemented on Amazon MSK brokers in KRaft mode. This blocked customers who rely on custom domain names from using them on KRaft-based clusters. With this launch, a single configuration property replaces all of that.

What you set up, and what Amazon MSK manages

A working custom domain name has two parts, and understanding this split up front helps the rest of this post make sense. You own the client connectivity and trust layer. Amazon MSK owns the cluster-side advertised listener configuration. The following diagram shows the client connectivity and trust layer.

Figure 1: The client connectivity and trust layer (left) is a prerequisite you own and manage. The advertised listener configuration on the cluster (right) is what Amazon MSK manages for you

Important: When you apply custom.advertised.listeners, your custom domain name replaces the default addresses that clients use to connect to broker nodes. If the networking and trust layer is not already in place, resolvable, reachable, and trusted from the client, the client cannot reconnect, even though it was connected moments earlier.

Part 1: The client connectivity and trust layer (you manage)

The Prerequisites section below shows the key requirements. You can find the detailed setup in an existing post, Configure a custom domain name for your Amazon MSK cluster, which includes a diagrammed walkthrough of the NLB, Amazon Route 53, and AWS Certificate Manager (ACM) topology.

Part 2: The advertised listener configuration (Amazon MSK managed)

After the connectivity layer exists, you tell the brokers which custom address to advertise to clients. This is the part that used to require a per-broker CLI override, and it is what this launch simplifies. This next section describes how it works.

Prerequisites

Before a client can reach your brokers through a custom domain, the connectivity and trust path must exist. You create and manage this layer. It covers three things:

Networking: A network gateway, like a Network Load Balancer (NLB), TLS certificate, DNS records, and security groups that route traffic from your custom domain to your broker IPs.
Certificate trust: The client’s truststore must include certificate authorities in the path (the load balancer’s custom-domain cert).
DNS resolution: Clients must resolve the custom domain to your NLB, typically through an Amazon Route 53 private hosted zone associated with the client virtual private cloud (VPC).

This layer must be in place for custom domain names to function. It is a prerequisite for this feature to work.

How it works

You add a property to your Amazon MSK configuration. The value takes the form:

custom.advertised.listeners=<LISTENER>://<hostname>:<port>

where <LISTENER> is one of your cluster’s client listeners and <hostname>:<port> is the custom address pattern. For example, on an IAM cluster:

custom.advertised.listeners=CLIENT_IAM://b-{broker_id}.example.com:9000+{broker_id}

The property specifies two things:

Each listener corresponds to an authentication type on your cluster. Custom advertised endpoints can be set only for client listeners: CLIENT, CLIENT_SECURE, CLIENT_SECURE_PUBLIC, CLIENT_SASL_SCRAM, CLIENT_SASL_SCRAM_PUBLIC, CLIENT_IAM, and CLIENT_IAM_PUBLIC. Internal listeners (REPLICATION, CONTROLLER) are not supported and are rejected at validation. The listener you specify must also be bound (active) on your cluster. For example, if your cluster uses only IAM authentication, specifying CLIENT_SECURE is rejected, and the error message lists the valid client listeners for your cluster.
A custom hostname:port pattern that includes the {broker_id} template variable. Each broker resolves to a unique address. In this pattern, the {broker_id} template variable is replaced with each broker’s numeric ID. The port number 9000+{broker_id} means the broker ID is added to the base port 9000, so broker 1 resolves to 9001, broker 2 to 9002, broker 10 to 9010, and so on. The base port 9000 is only an example. You can use any base port, as long as the resulting ports match the TLS listeners you provisioned on your NLB.

{broker_id} can appear in the hostname, the port, or both, as long as each broker’s resolved host:port is unique. Placing it in the port alone is valid, so a shared hostname with a per-broker port also works:
custom.advertised.listeners=CLIENT_IAM://example.com:9000+{broker_id}

Before you begin, you need an MSK configuration to hold this property. You create one with the CreateConfiguration API (or the AWS Management Console), passing your server properties as the configuration body. MSK returns a configuration ARN and a revision number, which together identify the exact configuration you apply to the cluster.

custom.advertised.listeners does not need its own standalone configuration. You can include it alongside any other broker-level properties MSK already supports, such as auto.create.topics.enable, num.partitions, or log-retention settings, within a single configuration revision. If you already manage an MSK configuration for your cluster, add custom.advertised.listeners to it and create a new revision using the UpdateConfiguration API. No separate configuration is needed.

You then apply the configuration to your cluster with the UpdateClusterConfiguration API. Amazon MSK then performs three actions:

Validates the configuration.
Resolves the pattern for each broker.
Applies it through a rolling restart across the cluster.

These safeguards prevent you from accidentally removing or modifying the internal listeners that Amazon MSK manages. Validation is synchronous. The listener must be a client-facing listener, the pattern must include {broker_id}, and each broker’s resolved host:port must be unique. If any check fails, the API returns a descriptive error and makes no change.

The override affects only the advertised address of the named listener. Replication, authentication, multi-VPC (CLIENT_IAM_VPCE), and AWS PrivateLink connectivity remain unaffected. The change is also fully reversible: remove the custom.advertised.listeners property and re-apply the configuration, and Amazon MSK reverts the listener to its original address.

You can track progress with the DescribeOperation API, which shows state transitions from UPDATE_IN_PROGRESS to UPDATE_COMPLETE or UPDATE_FAILED. If a broker fails to start, the rollout halts at that broker, the remaining brokers keep their previous configuration, and you can fix the property and re-apply to recover.

Setting up a custom domain name end to end

When you apply custom.advertised.listeners, your custom domain name replaces the default addresses that clients use to connect to broker nodes. If the networking and trust layer is not already in place, resolvable, reachable, and trusted from the client, the client cannot reconnect, even though it was connected moments earlier.

The networking layer, the Network Load Balancer (NLB), DNS, and TLS certificate that route traffic from your custom domain to your broker IPs, is a prerequisite you own. It is not specific to this launch. The existing post Configure a custom domain name for your Amazon MSK cluster covers it in detail, with a diagrammed walkthrough of the NLB, Route 53, and ACM topology. With the networking in place, the following steps cover the cluster-side setup this launch introduces.

Step 1: Add the custom domain to your Amazon MSK configuration

Create or update an Amazon MSK configuration that includes the custom.advertised.listeners property, matching the hostnames and ports you provisioned on the NLB. For a three-broker IAM cluster fronted by an NLB with ports 9001–9003, put the property in a file:

custom.advertised.listeners=CLIENT_IAM://b-{broker_id}.example.com:9000+{broker_id}

Then create the configuration, passing the file as the server properties:

aws kafka create-configuration \
    --name "custom-domain-iam" \
    --description "Custom advertised listeners for CLIENT_IAM" \
    --server-properties fileb://custom-domain-config.txt

Use fileb:// (not file://) so the CLI reads the file as bytes and base64-encodes it. Passing the value inline is fragile because of the {broker_id} braces. Leave {broker_id} literal in the file. Amazon MSK resolves it per broker at apply time. The response returns the configuration ARN and LatestRevision.Revision, which you use in the next step.

Step 2: Apply the configuration

Apply the configuration to your cluster with UpdateClusterConfiguration, using the console, AWS Command Line Interface (AWS CLI), AWS CloudFormation, CDK, or Terraform. This is the same workflow you already use for broker configuration changes.

aws kafka update-cluster-configuration \
    --cluster-arn <your-cluster-arn> \
    --configuration-info arn=<configuration-arn>,revision=<revision> \
    --current-version <current-cluster-version>

If the configuration fails to apply, review the errors. For details, see the troubleshooting section in the Amazon MSK Developer Guide.

Step 3: Track the rollout

aws kafka describe-cluster-operation-v2 \
    --cluster-operation-arn <operation-arn>

After the configuration is accepted, Amazon MSK applies it through a rolling restart. Wait until the operation reports SUCCESS. If it reports FAILED, a broker could not apply the change. The rollout halts at that broker, the remaining brokers keep their previous configuration, and you can fix the configuration and re-apply to recover.

Step 4: Verify

Confirm clients can connect through the custom domain:

kafka-topics.sh --list --bootstrap-server b-1.example.com:9001

If your topic list is returned, clients are successfully connecting through your custom domain. If the operation reported SUCCESS but clients cannot connect, the cluster-side configuration is correct, but your networking layer likely needs attention.

Client connectivity during rollout

This step is important. Clients can be disconnected if the networking is not ready. Kafka clients do not keep using the original address they bootstrapped with. On a periodic metadata refresh, each client learns the broker’s advertised listener. The client uses that address for all subsequent connections. When you apply a custom domain name, that advertised address changes from the default name that Amazon MSK generates to your custom domain, so at the next metadata refresh every client connects over the custom domain. For this reason, the connectivity and trust layer described in What you set up, and what Amazon MSK manages is a prerequisite, not a follow-up task.

The safe sequence, which is also how customers move from Amazon DNS to a custom domain today, is two phases:

Build the networking path first: Stand up the NLB, DNS, and certificate, and point your clients at the custom bootstrap endpoint, but do not set the advertised listener yet. Clients bootstrap through the custom endpoint while still connecting to brokers over the addresses that Amazon MSK generates.
Configure the advertised listener: With the path already in place, applying custom.advertised.listeners changes what the brokers advertise. At the next metadata refresh, clients pick up the custom domain and cut over to it automatically.

Because the path already exists, this cutover is transparent: as Amazon MSK applies the change broker by broker, clients reconnect on their own, with no restart or reconfiguration.

Scaling and replacement of brokers

When you scale the cluster or a broker is replaced during automated healing, Amazon MSK automatically applies the configuration to the new broker, resolving {broker_id} for its ID, with no manual steps required on the cluster side. Remember to add the corresponding NLB listener, target group, and DNS record for any new broker, because the networking layer does not auto-scale.

Conclusion

Custom domain name configuration turns a per-broker CLI workaround into a single, validated, cluster-wide Amazon MSK configuration property. It works identically on ZooKeeper and KRaft, persists through scaling and failover, and flows through your existing Terraform, CloudFormation, and CLI workflows. If you rely on custom domain names, we recommend adopting the static configuration now.

This capability is available on all Amazon MSK Provisioned clusters with Standard and Express brokers, in all AWS Regions where Amazon MSK Provisioned is available. To get started, see the Amazon MSK Developer Guide and the end-to-end networking walkthrough in Configure a custom domain name for your Amazon MSK cluster.