AWS Big Data Blog

Resolving concurrent write conflicts for Apache Iceberg tables in Amazon Redshift

Sanjay Devireddy — Tue, 13 Oct 2026 20:16:48 +0000

In Getting started with Apache Iceberg write support in Amazon Redshift – Part 1, you learned how to create Apache Iceberg tables and write to them with INSERT. In Part 2, you modified data at the row level with DELETE, UPDATE, and MERGE. Both posts assumed your write had the table to itself. In production, that is rarely true: streaming jobs, scheduled pipelines, and one-time transformations frequently write to the same table at the same time. Amazon Redshift now resolves many of these concurrent writes automatically, committing changes that previously failed and were forced to retry. In this post, we explain why concurrent writes to Iceberg tables conflict, how Amazon Redshift resolves the conflicts that don’t actually overlap, how the engine escalates its checks from operations to partitions to individual files, and the throughput improvement it provides.

The challenge of concurrent writes

Apache Iceberg uses optimistic concurrency control (OCC) to keep a table consistent when more than one writer commits at the same time. Each writer prepares its changes against the table’s current version and commits by atomically advancing the table version in the catalog, on the condition that no other writer advanced it first. If another writer commits in the meantime, the condition fails and the entire write is discarded. This strictness maintains snapshot isolation semantics: if the later commit were applied anyway, it would replace the table version without the earlier writer’s changes, silently losing a committed write.

Readers don’t encounter this problem. An Iceberg scan always reads a committed snapshot, so a read is never invalidated by a concurrent write. That leaves write-write conflicts as the only case the engine must resolve, and resolving them correctly is what preserves snapshot isolation semantics for the table.

The problem is the bluntness of this check. The condition is evaluated against the table version, not against the rows or partitions each writer touched. At that grain, its answer is correct in that the version really did change. However, it says nothing about whether the two writes overlapped in the data underneath. Two writers that modified entirely separate partitions, or that independently inserted new data, conflict exactly as hard as two that rewrote the same rows. This happens whenever their query-start-to-commit windows overlap.

Consider a retail analytics platform with a customer_orders Iceberg table. A streaming pipeline might land new orders continuously with hundreds of inserts a minute. A nightly compliance job deletes records older than seven years, and takes about an hour to run. The two writes never touch the same data: the stream only adds recent orders, and the compliance job only removes old ones. Under basic optimistic concurrency, that doesn’t matter. If a single streaming insert commits at any point during that hour, the compliance job loses the race, its hour of work is discarded, and the next attempt re-scans terabytes of data only to re-enter the same race. The table version conflict was real even though the data conflict was not.

The common ways to avoid this are all manual, and each carries a cost. You can serialize your writers so that only one commits at a time, which gives up concurrency entirely. You can split a single logical table into several physical tables so that no two writers ever share one. This trades a clean data model and the ability to query across them for the ability to write in parallel. Or you can wrap every write in an application-side retry loop and absorb the cost of re-running statements that never actually conflicted. In each case, the effort goes toward working around the commit protocol rather than toward your data.

How Amazon Redshift resolves write conflicts

When a commit loses the race, Amazon Redshift now asks a more precise question before giving up: Did the writers that committed ahead of it actually change the same partitions or files or rows it changed? When the answer is no, there’s no reason to discard the work. Instead of aborting, Amazon Redshift rebases the change (similar to git rebase), replaying the already-prepared write onto the latest version of the table and committing that result. Both writes land, with no re-scan, no re-execution, and no retry loop in your application.

In the retail example, the compliance delete previously failed the moment a streaming insert committed inside its execution window. Now Amazon Redshift examines both writes, sees that the inserts only added data files while the delete only removed others, and confirms they never touched the same data. Because the changes are provably independent, Amazon Redshift rebases the delete onto the updated table version and commits it. Both writes succeed, the hour of work is preserved, and downstream consumers see both changes.

This behavior is automatic for eligible writes, with no clustering keys to choose, no repartitioning, and no setting to enable. When two writes genuinely conflict, the behavior is unchanged from today: the conflict surfaces and the later writer fails and must retry.

Two principles govern how the resolution works. The first is that a conflict is a matter of granularity. A failed race isn’t automatically a conflict. It’s a question that can be asked at different resolutions. The engine can ask whether any writer removed data at all or whether they only added it, and if data was removed, whether it came from the same partition, the same file, or the same rows that the current write touched. The finer the granularity examined, the more apparent conflicts turn out to be independent changes that can safely coexist. But finer checks also cost more: taken to the limit, the engine could intersect two change sets row by row, and that is expensive and usually unnecessary. So Amazon Redshift evaluates the least expensive level first and examines a finer level only when it must.

The second principle is that correctness takes precedence over concurrency. Conflict resolution works entirely from table metadata and never re-scans data or re-plans a query. When the engine can’t prove that two writes are independent, it doesn’t guess. It declines to resolve the conflict and the write retries, exactly as it would today. An unnecessary retry costs a small amount of time, whereas an incorrect resolution risks losing data, so the system is deliberately biased toward the safe outcome. Resolution operates under snapshot isolation, the same model Amazon Redshift already uses for Iceberg reads. Engines such as Apache Spark, Amazon EMR, and Amazon Athena honor it through the table’s isolation-level property, so a table shared across engines retains consistent semantics. For tables whose write isolation property is set to serializable, Amazon Redshift preserves the stricter contract and surfaces every conflict rather than resolving it. This is the Iceberg table property write.*.isolation-level. It’s unrelated to the Amazon Redshift database isolation level set with ALTER DATABASE … ISOLATION LEVEL, which governs Amazon Redshift transactions and has no effect on Iceberg commit conflict resolution.

Resolving at the right granularity

When a commit loses the race, the engine escalates through a sequence of checks, from the cheapest to the most precise. Each level is a complete resolution on its own and if a check proves the writes independent, the rebase commits and no finer check runs. The following diagram shows this escalation sequence: an operation-level check first, then partition-level, then file-level. Each level either proves the writes independent or escalates to the next, and a genuine conflict surfaces only after the last.

Figure 1: The conflict-resolution escalation from operation-level to partition-level to file-level checks

The first check is by operation. A write that only added data, for example an INSERT consisting of new data landing from a streaming job, can’t have removed anything another writer relied on, so any write can be rebased against it. This single rule resolves every combination in which at least one of the two writes only appended. A DELETE, UPDATE, or MERGE commits cleanly against a stream of concurrent inserts, and an INSERT commits cleanly over any other concurrent change, including a compaction.

Next, when both writes removed data, the engine compares the partitions they each touched. Writes that modified disjoint partitions, for example batch jobs updating different dates or per-Region pipelines maintaining their own slices of one table, are independent by construction and both can commit cleanly.

Finally, when both writes touch the same partition, the engine compares the individual files that each one changed. Two jobs deleting different rows of the same date partition usually land in different files, and when they do, both can commit cleanly. Only when two writes modified the same file does the conflict surface and the statement re-run. The protocol additionally has headroom for one further level: distinguishing changes to different rows within a single file.

The following table summarizes the outcomes by operation pair. Read down for the concurrent write that committed first, and across for your write.

	Your write ↓ / Concurrent write →	`INSERT`	`DELETE` / `UPDATE` / `MERGE`	Compaction	`ALTER`
1	`INSERT`	Resolves	Resolves	Resolves	Re-run required
2	`DELETE` / `UPDATE` / `MERGE`	Resolves	Resolves based on partitions or files touched. A same-file change surfaces as a conflict	Resolves based on partitions or files touched. A same-file change surfaces as a conflict	Re-run required
3	`ALTER`	Re-run required	Re-run required	Re-run required	Re-run required

Performance

To measure the effect, we ran an increasing number of concurrent writers committing to a single Iceberg table over a fixed window. We compared basic optimistic concurrency mode, where a losing commit is aborted and re-runs, against automatic conflict resolution mode being enabled. Every writer issued the same single-statement INSERT, so the only variable across the various runs was contention rather than workload. We scaled 1–16 concurrent writers and recorded, for each configuration, how many commits succeeded and how many were rejected as conflicts.

Setup: one unpartitioned Iceberg table registered in the AWS Glue Data Catalog in the US East (N. Virginia) AWS Region (us-east-1). Each writer is an independent session issuing autocommit INSERT statements in a loop over a 60-second period. We report the commits completed and the attempts rejected as conflicts for each configuration.

The first figure shows write throughput as writers are added, normalized to the single-writer case so the comparison doesn’t depend on instance type or table size.

Figure 2: Write throughput as concurrent writers increase, with and without automatic conflict resolution

Under basic optimistic concurrency, adding writers makes writing to the table slower. At 16 writers, throughput falls to 0.17× of a single writer, because nearly every writer’s work is discarded and redone rather than committed. With automatic conflict resolution the same workload scales to roughly 2×, and no commit is rejected in any measured configuration. At 16 concurrent writers, 12× more commits complete in the same window. The second figure shows why.

Figure 3: Commit failure rate as concurrent writers increase, with and without automatic conflict resolution

The baseline’s failure rate is almost exactly (N−1)/N, for example 50 percent at two writers, 75 percent at four writers, and 94 percent at 16 writers. This illustrates the table-level check that occurred previously: only one writer wins each commit round and every other writer in flight is rejected. Additional writers translate almost entirely into repeated work rather than progress. With conflict resolution enabled, the same runs record no conflicts at all, because every write in this workload is an append, and an append never removes data another writer relied on.

This benchmark isolates the most basic append case deliberately: a single unpartitioned table and insert-only writers, which is the pattern behind continuous ingestion. Workloads that remove data exercise the partition and file checks described earlier, where the outcome depends on whether the two writes touched the same partition and the same files.

Fleet Results

Before conflict resolution, concurrent-write conflicts were the single largest cause of failed Iceberg writes in Amazon Redshift, accounting for roughly one in five failed Iceberg write queries each week. We rolled out this feature in stages, beginning with the operation-level check that covers concurrent appends. As that stage reached full fleet deployment in early September 2026, conflict errors from concurrent writes dropped to nearly zero, even as Iceberg write volume on the fleet more than doubled. One residual spike shown below came from a workload issuing concurrent UPDATE statements, a conflict class that subsequent partition and file level checks will resolve as they deploy across the fleet.

Figure 3: Conflict errors as a share of failed Iceberg writes

Observing conflict resolution

Every Iceberg commit attempt is recorded in the stl_iceberg_commit system table, including the outcome, the number of attempts, the concurrent operations the write rebased over, and the time spent rebasing. To see how often resolution is engaging on an example table:

SELECT outcome, count(*), avg(attempt_number)
FROM stl_iceberg_commit
WHERE trim(table_name) = 'customer_orders'
GROUP BY outcome;

Conclusion

Concurrent writes are a fact of production data lakes, and until now, they forced you to serialize writers, partition around the problem, or retry. With automatic conflict resolution, Amazon Redshift commits concurrent writes that don’t actually overlap. These include any mix of writes alongside concurrent appends, and row-level writes that changed different partitions or different files of the same table. To get started, write to your Iceberg tables from multiple jobs on patch 203 or later. Non-conflicting commits start to resolve on their own. To learn more about Iceberg write support in Amazon Redshift, see Part 1 and Part 2.

About the authors

Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality

Mert Hocanin — Mon, 12 Oct 2026 23:40:26 +0000

Bad data rarely announces itself. It slips into an Amazon EMR pipeline quietly. A null where there should be a value. A timestamp in the wrong format. A referential key that no longer exists. It travels downstream until it surfaces as a wrong number in a dashboard, a failed downstream job, or a business decision made on data nobody verified.

For teams running large-scale data processing on Amazon EMR, choosing a data quality strategy has meant picking between two trade-offs. You can build your own quality checks in Apache Spark using an open source library like Deequ. With this approach, you can customize checks to your exact pipeline, but it adds custom code your team must maintain, with no centralized ruleset management and no consistent quality catalog across pipelines. Or you can route your data through AWS Glue to access its managed data quality capabilities. This approach minimizes the code you need to maintain but turns data quality validation into a separate step and adds another engine to your pipeline.

Today, there is a third option.

Amazon EMR Data Quality built on AWS Glue is now available natively on Amazon EMR, bundled in AWS runtime for Apache Spark (emr-spark-8.1.0) and later versions. You can write data quality rules using the same Data Quality Definition Language (DQDL) used in AWS Glue. Run them directly inside your Amazon EMR Spark jobs, across each of the three Amazon EMR deployment modes: Amazon EMR on EC2, Amazon EMR Serverless, and Amazon EMR on EKS.

This post walks through two patterns for using AWS Glue Data Quality, a capability of AWS Glue, on Amazon EMR. The first shows a lightweight ad hoc validation workflow, useful for exploratory processing and pipeline-local checks. The second builds on the Apache Iceberg Write-Audit-Publish (WAP) pattern. An extract, transform, and load (ETL) job writes data to an Iceberg branch, retrieves a managed ruleset from the AWS Glue Data Catalog, and validates the branch. Only if the data passes your quality bar does it merge to the main table. By the end of this post, you will have working examples of both patterns ready to run on any mode of Amazon EMR.

What is Amazon EMR Data Quality?

With Amazon EMR Data Quality, you can define and run data quality rules against your datasets. At its core is the Data Quality Definition Language (DQDL), a declarative rule syntax. With DQDL, you can express constraints like completeness, uniqueness, value ranges, referential integrity, and statistical distributions in plain, readable statements.

DQDL also includes machine learning (ML)-based anomaly detection. Rather than setting explicit thresholds, DetectAnomalies rules learn the statistical baseline of your data across runs and flag deviations automatically. This is useful for metrics that shift over time, like daily record counts or revenue totals, where a fixed range rule would require constant maintenance.

You can define rules inline within a job or store them as managed rulesets in the AWS Glue Data Catalog, a centralized metadata repository for table definitions and data quality rulesets. After the rules are stored, you can version these rulesets, share them across teams, and reuse them across multiple pipelines. The evaluator returns per-rule pass or fail results and an overall quality score. Your job can inspect the score to take action: fail the pipeline, write results to Amazon Simple Storage Service (Amazon S3), trigger an alert, or conditionally merge data.

You can now run this rule engine directly inside your Spark jobs on Amazon EMR. The same DQDL rulesets you author for AWS Glue work without modification.

For a full reference on DQDL syntax and rule types, see the AWS Glue Data Quality documentation.

Prerequisites and setup

Amazon EMR is a managed service for running Apache Spark and other open source big data frameworks on AWS. It offers three deployment modes: Amazon EMR on EC2, Amazon EMR Serverless, and Amazon EMR on EKS.

Before running the examples in this post, make sure you have the following in place.

Amazon EMR release: Amazon EMR Data Quality is bundled in AWS runtime for Apache Spark (emr-spark-8.1.0) and later versions across Amazon EMR on EC2, Amazon EMR on EKS, and Amazon EMR Serverless. No bootstrap action or additional installation is required. The library is available as part of the release.

AWS Identity and Access Management (IAM) permissions: The execution role for your Amazon EMR job needs the following permissions:

glue:AccessDataQualityRuntimeConfiguration to publish data quality results to the AWS Glue Data Catalog.
glue:CreateDataQualityRuleset to create managed rulesets in the AWS Glue Data Catalog.
glue:GetDataQualityRuleset to retrieve managed rulesets from the AWS Glue Data Catalog.
glue:GetDataQualityResult to read prior quality results.
glue:PublishDataQuality to submit data quality results to the AWS Glue Data Catalog.
glue:GetDatabase and glue:GetTable for catalog access.
Standard Amazon S3 read/write permissions for your data and output locations.

Pattern 1: Ad hoc data quality validation

This pattern defines data quality rules directly inside your Amazon EMR Spark job, with no externally managed ruleset to maintain.

The use case

Your team ingests a raw CSV file from a third-party vendor and needs to validate it before loading it into your data lake. The rules are straightforward: check that the record ID is never null, that revenue values are non-negative, and that country codes conform to an allowed list. This is a one-off check that lives entirely within the job. There is no centralized ruleset to manage, and the goal is to validate data quality before the load.

This pattern works well for:

Exploratory or one time processing pipelines.
Early-stage pipelines where rules are still evolving.
Lightweight validation that does not require centralized governance.

How it works

You define your DQDL rules inline as a string, pass your DataFrame and the rules to EvaluateDataQuality, and inspect the results to decide how the job should proceed.

To validate data quality inline using PySpark

To implement this pattern using PySpark, see the following code sample, which can be run on all three Amazon EMR deployment modes:

import os
from pyspark.sql import SparkSession
from awsdataquality.transforms import EvaluateDataQuality

spark = SparkSession.builder \
    .appName("dq-ruleset-catalog") \
    .enableHiveSupport() \
    .getOrCreate()

# Read new incoming data
input_df = spark.read.parquet("s3://amzn-s3-demo-sales-data/sales_data/incoming_data/")

# Contains our DQDL rules and analyzers. Analyzers submit statistics to the AWS Glue service for inspection. This can help create new rules.
ruleset = """
Rules = [
    RowCount > 0,
    IsPrimaryKey "order_id",
    Completeness "customer_email" >= 0.95,
    ColumnValues "order_status" in ["PENDING","SHIPPED","DELIVERED","RETURNED"],
    ColumnValues "net_amount" between 0.0 and 25000.0
]
Analyzers = [
    AllStatistics "net_amount",
    DistinctValuesCount "country",
    RowCount
]
"""

# Publishing options for the results of the data quality evaluation
publishing_opts = {
    "dataQualityEvaluationContext": "blog-post-dq-evaluation-context",
    "enableDataQualityResultsPublishing": "true"
}

# AWS Glue Data Quality uses the additional_options parameter to configure advanced evaluation behaviors, output formats, anomaly detection and performance tuning for data quality checks.
additional_opts = {"compositeRuleEvaluation.method": "ROW"}

# This set tells the Data Quality Evaluator if there are other data sources that
# we need to pull in for checks such as cross table reference checking.
additional_sources = {}

# Evaluate the ruleset against the data frame created by reading the parquet files and save
# the results in dq_results as a DataFrame.
dq_results = EvaluateDataQuality.apply(input_df, ruleset, publishing_opts, additional_opts,
    additional_sources)

print(f"{dq_results.count()} rule result row(s)")  # forces evaluation

# Iterate through the DQ results for any failed rules and if there are any failed
# rules, throw an exception to stop the job.
failures = []
for row in dq_results.collect():
    if row["Outcome"] == "Failed":
        failures.append(f"Rule: {row['Rule']} | Reason: {row['FailureReason']}")

if failures:
    failure_summary = "\n".join(failures)
    raise RuntimeError(f"Data quality checks failed:\n{failure_summary}")

# Write out data into the Iceberg table sales_data table
input_df.writeTo("company_sales.sales_data").append()
print(f"all rules passed; appending {input_df.count()} row(s) to company_sales.sales_data")

The preceding script passes the incoming DataFrame, an inline ruleset, and publishing options to EvaluateDataQuality, which runs the ruleset and returns a results DataFrame. The results DataFrame lists each rule with a Passed or Failed outcome and, for failed rules, the reason. The publishing options control if and where the results get published. The script raises an error and stops if any rule fails. If every rule passes, it appends the incoming data to the company_sales.sales_data table.

What the data quality results look like

The EvaluateDataQuality.apply() function returns the results as a DataFrame containing each rule, whether it passed or failed, and the evaluated rule. Your job can read this DataFrame and branch its logic accordingly: writing clean records to Amazon S3, routing failed records to a quarantine path, or halting the pipeline entirely. Following is sample output of the code earlier:

Rule	Outcome	FailureReason	EvaluatedMetrics
RowCount > 0	Passed	NULL	{RowCount -> 50000.0}
IsPrimaryKey “order_id”	Passed	NULL	{Uniqueness -> 1.0, Completeness -> 1.0}
Completeness “customer_email” >= 0.95	Passed	NULL	{Completeness -> 0.97924}
ColumnValues “order_status” in […]	Failed	Value: 0.99598 does not meet …	{ColumnValues.Compliance -> 0.99598}
ColumnValues “net_amount” between 0.0 and 25000.0	Failed	Value: -2228.87 does not meet …	{Minimum -> -2228.87, Maximum -> 3394.51}

RuntimeError: Data quality checks failed:
Rule: ColumnValues "order_status" in ["PENDING","SHIPPED","DELIVERED","RETURNED"] | Reason: Value: 0.99598 ...
Rule: ColumnValues "net_amount" between 0.0 and 25000.0 | Reason: Value: -2228.87 ...

Pattern 2: Iceberg Write-Audit-Publish with managed rulesets

This pattern gates data promotion on a centrally managed ruleset, merging new data into your main table only after it passes your quality checks.

The use case

Your team runs a nightly ETL job that transforms raw transactional data and writes it to an Apache Iceberg table. Apache Iceberg is an open table format for large analytic datasets that supports schema evolution, time travel, and branching on Amazon S3. Rather than writing directly to the production table and discovering quality issues after the fact, you want a safety gate. Write to a staging branch, validate it against your organization’s data quality standards, and only promote the data to the main table if it passes.

This is the Write-Audit-Publish (WAP) pattern, a well-established practice in Iceberg-based architectures. For a detailed walkthrough of this pattern using AWS Glue, see Build Write-Audit-Publish pattern with Apache Iceberg branching and AWS Glue Data Quality. WAP gives you atomic promotion: either all the new data meets your quality bar and gets merged, or none of it does. Readers of the main table should not see partial or failed writes.

When combining AWS Glue Data Quality with WAP, you define your quality rules once in the AWS Glue Data Catalog and have your data governance team own them. You enforce them automatically at promotion time, without touching the pipeline code. Combined with managed rulesets in the AWS Glue Data Catalog, you get quality enforcement that is consistent and auditable. The data quality rules are also decoupled from the job code and changing them does not require touching the pipeline.

This pattern works well for:

Production ETL pipelines where bad data reaching the main table has downstream consequences.
Organizations with centralized data governance teams that own and version quality standards.
Apache Iceberg-based lakehouses where the WAP pattern is already in use.

How it works

The flow has four steps:

Retrieve the ruleset – Call the AWS Glue Data Catalog to fetch the managed ruleset for this table. The ruleset is owned by your governance team and versioned in the catalog.
Write to a branch – Before writing the ETL output, create an Apache Iceberg branch on the target table. Write all transformed records to this branch rather than to main.
Evaluate – Run EvaluateDataQuality against the branch data using the retrieved ruleset.
Merge or discard – If the quality score meets your threshold, merge the branch into main. If not, discard the branch and trigger your alerting workflow.

Creating a managed ruleset: To create a ruleset in the AWS Glue Data Catalog that targets the same database and table, you can create one using the AWS Command Line Interface (AWS CLI). For example:

aws glue create-data-quality-ruleset \
    --name "sales-orders-ruleset" \
    --ruleset "Rules = [ RowCount > 0, IsPrimaryKey \"order_id\", Completeness \"customer_email\" >= 0.95, ColumnValues \"order_status\" in [\"PENDING\",\"SHIPPED\",\"DELIVERED\",\"RETURNED\"], ColumnValues \"net_amount\" between 0.0 and 25000.0 ]" \
    --target-table '{"TableName": "sales_data", "DatabaseName": "company_sales"}'

The --target-table values must match the ICEBERG_DATABASE and ICEBERG_TABLE environment variables used in the Pattern 2 job. The job discovers the ruleset at runtime using list_data_quality_rulesets filtered by that target table.

To validate data quality inline using PySpark

To implement this pattern, see the following code sample which can be run on all three Amazon EMR deployment modes:

import os
import boto3
from pyspark.sql import SparkSession
from awsdataquality.transforms import EvaluateDataQuality

spark = (SparkSession.builder.appName("blog-wap-dq")
    .config("spark.sql.catalog.iceberg", "org.apache.iceberg.spark.SparkCatalog")
    .config("spark.sql.catalog.iceberg.warehouse", "s3://amzn-s3-demo-bucket/warehouse/")
    .config("spark.sql.catalog.iceberg.catalog-impl", "org.apache.iceberg.aws.glue.GlueCatalog")
    .enableHiveSupport().getOrCreate())

database = os.environ["ICEBERG_DATABASE"]
table = os.environ["ICEBERG_TABLE"]
input_path = os.environ["WAP_INPUT_PATH"]
threshold = float(os.environ.get("WAP_SCORE_THRESHOLD", "1.0"))
region = os.environ.get("AWS_REGION", "us-east-1")
ruleset_name = os.environ.get("GLUE_DQ_RULESET_NAME")

# Branch name carries the Spark application id so concurrent nightly runs, and reruns
# of a failed one, never collide on the same branch.
branch = os.environ.get("WAP_BRANCH") or f"audit_{spark.sparkContext.applicationId}"
branch = branch.replace("-", "_")
fqtn = f"iceberg.`{database}`.`{table}`"
branch_fqtn = f"iceberg.`{database}`.`{table}`.branch_{branch}"

glue = boto3.client("glue", region_name=region)

# ---------------------------------------------------------- 1. DQ RULESET
# We will then get the data quality rules.
ruleset = glue.get_data_quality_ruleset(Name=ruleset_name)["Ruleset"]
print(f"auditing against catalog ruleset '{ruleset_name}':\n{ruleset}")

# ---------------------------------------------------------- 2. WRITE
main_before = spark.table(fqtn).count()
print(f"main has {main_before} row(s) before this batch")

# CREATE BRANCH (not CREATE OR REPLACE): if the branch somehow already exists, that is
# a concurrent or half-finished run and silently replacing it would discard its state.
spark.sql(f"ALTER TABLE {fqtn} CREATE BRANCH {branch}")
print(f"created branch {branch}")

# Write out our batch to our new branch
batch = spark.read.parquet(input_path)
batch.writeTo(branch_fqtn).append()
print(f"wrote {batch.count()} row(s) to branch {branch}")

# Reread the branch and create a data frame to perform our data quality checks.
branch_df = spark.read.option("branch", branch).table(f"iceberg.`{database}`.`{table}`")

# ---------------------------------------------------------- 3. AUDIT
# Will now evaluate the ruleset against the branch
publishing_opts = {
    "dataQualityEvaluationContext": ruleset_name,
    "publishToDataCatalogTarget": {
        "DatabaseName": database,
        "TableName": table
    },
    "enableDataQualityResultsPublishing": "true"
}

# See the AWS Glue Data Quality additional_options reference for advanced configuration:
# https://docs.aws.amazon.com/glue/latest/dg/data-quality-additional-options.html
additional_opts = {}
additional_sources = {}

# Run EvaluateDataQuality.
dq_results = EvaluateDataQuality.apply(branch_df, ruleset, publishing_opts,
    additional_opts, additional_sources)

# ---------------------------------------------------------- 4. PUBLISH or discard
# Iterate through the DQ results for any failed rules and if there are any failed
# rules, throw an exception to stop the job.
failures = []
for row in dq_results.collect():
    if row["Outcome"] == "Failed":
        failures.append(f"Rule: {row['Rule']} | Reason: {row['FailureReason']}")

# If there are any failures of any rule, we will drop the branch, and throw an error
if failures:
    # Drop first, then raise: an exception alone would leave the rejected branch (and
    # its data files) behind on the table for every failed night.
    spark.sql(f"ALTER TABLE {fqtn} DROP BRANCH {branch}")
    main_after = spark.table(fqtn).count()
    print(f"REJECTED: branch {branch} dropped; main unchanged at {main_after} row(s)")
    failure_summary = "\n".join(r for r in failures)
    raise RuntimeError(f"Data quality checks failed:\n{failure_summary}")

# fast_forward moves main's pointer to the branch's snapshot. It requires main to
# be an ancestor of the branch, which holds here because the branch was cut from
# main and only appended to --- no rewrite of the promoted data, no second copy.
spark.sql(f"CALL iceberg.system.fast_forward("
    f"table => '{database}.{table}', branch => 'main', to => '{branch}')")
spark.sql(f"ALTER TABLE {fqtn} DROP BRANCH {branch}")
main_after = spark.table(fqtn).count()
print(f"PUBLISHED: main fast-forwarded to {branch} "
    f"({main_before} -> {main_after} row(s)); branch dropped")

spark.stop()

This code demonstrates a sample WAP approach using Apache Iceberg. We start by retrieving the ruleset from the AWS Glue service that will run against the table. We then create a branch within the Apache Iceberg table and apply new data against it. We then run our data quality checks using EvaluateDataQuality with the ruleset that we earlier obtained. If any of the data quality rules fail, a branch that holds the new version of the table gets dropped. Otherwise, it will be committed as the new main branch and made available for all consumers.

In this example, you will notice that the publishing_opts (publishing options) look different. For the dataQualityEvaluationContext, we set the ruleset name. We also fill in the publishToDataCatalogTarget with the database and table name within the AWS Glue Data Catalog. This tags the data quality run and its results to that table for easier lookups if you want to view the history of data quality runs on the table.

Clean up

To avoid ongoing charges, remove the resources that you no longer need:

Shut down Amazon EMR on EC2 clusters, Amazon EMR Serverless applications, or Amazon EMR on EKS.
Delete the tables that you created. The table mentioned in this blog post is company_sales.sales_data.
Delete the data quality ruleset created in Pattern 2.
Delete any Amazon S3 data that was generated for the purposes of this blog post.

Conclusion

Data quality is a concern for most production data pipelines. Until now, teams running on Amazon EMR had to build that quality layer themselves or bring in AWS Glue as a separate processing engine to get managed quality capabilities.

With AWS Glue Data Quality now bundled in AWS runtime for Apache Spark (emr-spark-8.1.0) and later versions, that gap is closed. You write your DQDL rules once and store them in the AWS Glue Data Catalog. From there, run them across each of the three Amazon EMR deployment modes.

The patterns in this post represent two points on that spectrum: a lightweight inline check for ad hoc workloads, and a full WAP workflow for production pipelines that need atomic, auditable data promotion.

To learn more:

AWS Glue Data Quality documentation – Full DQDL rule syntax and rule type reference.
Amazon EMR Serverless documentation – Getting started with Amazon EMR Serverless.
Apache Iceberg branching and Write-Audit-Publish – Full reference for the WAP pattern used in Pattern 2.
Amazon EMR release 8.1 notes – Bundled libraries and changes in release 8.1.

Have questions or feedback on this post? Leave a comment below or reach out on the AWS re:Post community.

About the authors

Build cross-account streaming pipelines with the Amazon Kinesis Producer Library and Kinesis Client Library

Charishma Makineni — Mon, 12 Oct 2026 15:48:07 +0000

Organizations use real-time streaming to act on data the moment it arrives. They detect fraud, personalize customer experiences, and respond to infrastructure events, all within milliseconds. Amazon Kinesis Data Streams ingests and processes millions of records per second with sub-second latency. With multi-account deployment, you can enforce strict security boundaries, satisfy regulatory requirements, and contain scope using Amazon Kinesis Data Streams. This approach still delivers the high-throughput performance that streaming pipelines require. Realizing these benefits, however, requires configuring producers and consumers to operate across account boundaries. This involves coordinating AWS Identity and Access Management (IAM) policies, encryption keys, and monitoring across accounts, which can extend delivery timelines.

This post addresses that complexity directly. We first examine the common architectural patterns for producing and consuming data from Amazon Kinesis Data Streams across multiple AWS accounts. This post has a detailed walkthrough of the most isolated configuration, in which the producer, the stream, and the consumer each reside in a separate AWS account. This illustrates the best practices for enterprise-scale streaming with maximum security isolation.

Amazon Kinesis Data Streams integrates directly with services such as AWS Lambda, Amazon Data Firehose, and Amazon Managed Service for Apache Flink. Granting cross-account access to other service is usually done through Kinesis resource policies. However, many enterprise workloads run on custom applications. Two open source libraries let these applications produce and consume streams:

Amazon Kinesis Producer Library (KPL) – Writes records to a Kinesis data stream at high throughput. KPL aggregates multiple user records into a single Kinesis record to maximize shard utilization. It retries failed records with exponential backoff. It also publishes per-instance Amazon CloudWatch metrics for operational visibility.

Kinesis Client Library (KCL) – Consumes and processes records from a Kinesis data stream. KCL 3.5 discovers shards automatically and coordinates leases through Amazon DynamoDB. It checkpoints progress so that a failed worker resumes from the last saved position. It performs graceful lease handover during resharding. It also balances shard assignments across workers to distribute load evenly.

Both libraries support Java. KCL also supports other languages through the MultiLangDaemon. This walkthrough uses both libraries.

Challenges enterprises face with multi-account streaming

Large enterprises commonly encounter the following challenges when building cross-account streaming architectures:

Account boundary enforcement: Producers (app teams) and consumers (analytics teams) sit in separate accounts for isolation and cost attribution. Access is denied by default, so each path is granted explicitly. The producer needs write access to a stream it does not own, the consumer needs read access to that foreign stream, and encrypted streams add AWS Key Management Service (AWS KMS) key access on both sides.
Cross-account access has policy complexity and requires coordinating three policies across accounts: a resource-based policy on the stream, identity-based policies on producers and consumers, and AWS KMS key sharing for encrypted streams.
Encryption across boundaries: Streams encrypted with AWS managed keys cannot be shared cross-account. Customer managed AWS KMS keys are required.
Monitoring across accounts: Amazon CloudWatch metrics for producer throughput, stream utilization, and consumer lag live in different accounts, requiring cross-account observability.

Cross-account deployment patterns

The following table illustrates common permutations of KPL, Kinesis data stream, and KCL placement across accounts:

Pattern	KPL (Producer)	Kinesis data stream	KCL (Consumer)
All-in-one account	Account A	Account A	Account A
Cross-account Consumer	Account A	Account A	Account B
Cross-account Producer	Account A	Account B	Account B
Full separation	Account A	Account B	Account C

Each pattern requires specific IAM configurations. The full-separation pattern (Account A to Account B to Account C) demonstrates the most complex scenario and is what we walk through in the following sections.

Solution architecture

Our reference implementation deploys three AWS accounts:

Account A (Producer): Runs the KPL application on Amazon Elastic Compute Cloud (Amazon EC2). Assumes a cross-account IAM role in Account B to write records.
Account B (Stream): Owns the Kinesis data stream. Hosts a cross-account IAM role for the producer (Account A) and a resource-based policy granting read access to the consumer (Account C). Also pre-registers an Enhanced Fan-Out consumer shared with Account C through a consumer resource policy. The stream is encrypted with a customer managed AWS KMS key shared with both accounts.
Account C (Consumer): Runs the KCL 3.5 application with its own DynamoDB lease table and CloudWatch metrics. It reads the cross-account stream directly through a resource-based policy (no role assumption). The policy authorizes GetRecords, GetShardIterator, DescribeStream, and ListShards. Enhanced Fan-Out (EFO) is supported for dedicated 2 MB/s per-shard throughput and ~70 ms latency. It uses SubscribeToShard, which must be authorized separately, and Account B must pre-register the EFO consumer and share the consumer ARN.

How the cross-account access works

Producer path (Account A to B):

The KPL app runs with KPLProducerRole (Account A EC2 instance profile).
At startup it calls sts:AssumeRole on KinesisProducerCrossAccountRole in Account B (with external ID).
The assumed temporary credentials are used for Kinesis PutRecords calls.
CloudWatch metrics are published to Account A using local instance profile credentials.

Consumer path (Account C to B):

The KCL app runs with KCLConsumerRole (Account C EC2 instance profile).
It calls Kinesis GetRecords using its local credentials (KCLConsumerRole).
The stream’s resource-based policy in Account B authorizes the request.
DynamoDB lease tables and CloudWatch metrics use the local KCLConsumerRole credentials (Account C).

The following figure shows how cross-account access works.

Figure 1: Cross-account streaming architecture across the producer, stream, and consumer accounts

Deployment walkthrough

The solution is deployed using AWS CloudFormation templates across three accounts. Clone the repository from GitHub and follow these steps:

Prerequisites

Three AWS accounts (Producer, Stream, Consumer) with AWS Command Line Interface (AWS CLI) configured.
AWS CLI v2 installed with named profiles for each account.
Java 11+ and Maven installed (for building KPL and KCL applications).
Clone the repository: git clone https://github.com/aws-samples/sample-kinesis-cross-account-kpl-kcl.git.

Step 1: Deploy the stream stack (Account B: Stream Account)

Deploy the CloudFormation template that creates the Kinesis data stream, customer managed AWS KMS key, cross-account producer role, resource-based consumer policy, and pre-registered EFO consumer. Navigate to the cloudformation/ directory.

Deploy:

aws cloudformation deploy \
  --template-file cloudformation/stream-account-stack.yaml \
  --stack-name kinesis-cross-account-stream \
  --parameter-overrides \
  ProducerAccountId=<ACCOUNT_A_ID> \
  ConsumerAccountId=<ACCOUNT_C_ID> \
  --capabilities CAPABILITY_NAMED_IAM \
  --profile <account-b-profile>

Note the stack outputs: StreamArn, KMSKeyArn, ProducerCrossAccountRoleArn, and EFOConsumerArn. These are needed for the next steps.

The following screenshot shows the CloudFormation output for the deployed stream stack.

Figure 2: CloudFormation outputs for the stream stack

Step 2: Deploy the producer stack (Account A: Producer Account)

The KPL handles the following for your producer application:

Aggregation – Packs multiple user records into one Kinesis record (up to 1 MB) to maximize shard utilization.
Retries – Retries failed puts with exponential backoff to absorb transient throttling.
Rate limiting – Tracks shard limits and adjusts send rates to prevent ProvisionedThroughputExceededException.
Metrics – Publishes per-instance CloudWatch metrics (buffering time, record age, errors).

Deploy the CloudFormation template that creates the KPL application infrastructure and cross-account IAM role:

Deploy:

aws cloudformation deploy \
  --template-file cloudformation/producer-account-stack.yaml \
  --stack-name kinesis-cross-account-producer \
  --parameter-overrides \
  ProducerCrossAccountRoleArn=<ProducerCrossAccountRoleArn from Step 1> \
  StreamArn=<StreamArn from Step 1> \
  KMSKeyArn=<KMSKeyArn from Step 1> \
  --capabilities CAPABILITY_NAMED_IAM \
  --profile <account-a-profile>

Connect to the EC2 instance through AWS Systems Manager, clone the repository, then build and run.

The following screenshot shows the output for the producer CloudFormation stack.

Figure 3: CloudFormation outputs for the producer stack

Step 3: Deploy the consumer stack (Account C: Consumer Account)

KCL 3.5 handles the following for your consumer application:

Graceful lease handover – Transfers shard leases during resharding without duplicate processing.
Load balancing – Distributes shards evenly across workers and rebalances when workers join or leave.
Checkpointing – Persists progress in DynamoDB so a failed worker resumes from its last checkpoint.
Enhanced Fan-Out – Supports SubscribeToShard for dedicated per-shard throughput when you pass an EFO consumer ARN.

Deploy the CloudFormation template that creates the KCL application infrastructure, DynamoDB lease table permissions, and Kinesis read permissions:

Deploy:

aws cloudformation deploy \
  --template-file cloudformation/consumer-account-stack.yaml \
  --stack-name kinesis-cross-account-consumer \
  --parameter-overrides \
  StreamArn=<StreamArn from Step 1> \
  EFOConsumerArn=<EFOConsumerArn from Step 1> \
  KMSKeyArn=<KMSKeyArn from Step 1> \
  --capabilities CAPABILITY_NAMED_IAM \
  --profile <account-c-profile>

Connect to the EC2 instance through Systems Manager, clone the repository, then build and run.

The following screenshot shows the output for the consumer CloudFormation stack.

Figure 4: CloudFormation outputs for the consumer stack

Testing end-to-end data flow

After all three stacks are deployed, start the applications to verify end-to-end cross-account streaming:

Start the KPL producer (Account A)

Connect to the producer EC2 instance through Session Manager, then run CrossAccountKPLProducer. The application assumes the cross-account role, connects to the Kinesis data stream in Account B, and begins publishing aggregated records:

aws ssm start-session --target <ProducerInstanceId> --profile <account-a-profile>

# On the instance:
git clone https://github.com/aws-samples/sample-kinesis-cross-account-kpl-kcl.git
cd sample-kinesis-cross-account-kpl-kcl/producer
mvn clean package -q
java -jar target/cross-account-kpl-producer-1.0.jar <StreamArn> <region> <ProducerCrossAccountRoleArn>

Figure 5: KPL producer publishing aggregated records to the cross-account stream

Start the KCL consumer (Account C)

Connect to the consumer EC2 instance through Session Manager, then run CrossAccountKCLConsumer. The KCL application uses its local credentials, and the stream’s resource-based policy authorizes read access without role assumption:

aws ssm start-session --target <ConsumerInstanceId> --profile <account-c-profile>

# On the instance:
git clone https://github.com/aws-samples/sample-kinesis-cross-account-kpl-kcl.git
cd sample-kinesis-cross-account-kpl-kcl/consumer
mvn clean package -q
java -jar target/cross-account-kcl-consumer-1.0.jar <StreamArn> <region> StockTradeConsumer

To use Enhanced Fan-Out (dedicated 2 MB/s per consumer, approximately 70 ms latency), pass --efo with the pre-registered consumer ARN:

java -jar target/cross-account-kcl-consumer-1.0.jar <StreamArn> <region> StockTradeConsumerEFO --efo <EFOConsumerArn>

Note: With resource-based policies, KCL cannot auto-register stream consumers cross-account. The stream owner (Account B) pre-registers the consumer through CloudFormation, and you pass the resulting EFOConsumerArn.

The following screenshot shows the KCL consumer logs after the consumer has started.

Figure 6: KCL consumer startup logs

Verify in CloudWatch (Account B: Stream metrics)

In the stream-owning account, verify the IncomingRecords and GetRecords.Success metrics in Amazon CloudWatch to confirm records are flowing end to end across all three accounts.

The following screenshot shows the CloudWatch metrics for the stream.

Figure 7: CloudWatch metrics confirming records flow end to end across all three accounts

Key considerations

Use stream ARN, not stream name: Cross-account KCL configuration must reference the full stream ARN. Stream names resolve only within the owning account.
AWS KMS key sharing is mandatory for encrypted streams: AWS managed keys can’t be shared. Create a customer managed key and add cross-account key policies.
Minimum library versions: KCL 2.5.3+ or KCL 1.15.0+ is required for resource-based policy support. For best performance and cost, use KCL 3.5.
Enhanced Fan-Out works cross-account: The stream owner must pre-register the consumer and attach a resource policy to the consumer ARN (SubscribeToShard, DescribeStreamConsumer). Pass the consumer ARN to KCL with --efo. Auto-registration does not work cross-account with resource policies.

Clean up

To avoid ongoing charges, delete the CloudFormation stacks in reverse deployment order:

Consumer Account (Account C): Delete the kinesis-cross-account-consumer stack.
Producer Account (Account A): Delete the kinesis-cross-account-producer stack.
Stream Account (Account B): Delete the kinesis-cross-account-stream stack.

This removes all CloudFormation-managed resources across the three accounts, including the Kinesis data stream, customer managed AWS KMS key, IAM roles, and EC2 instances. Stack deletion typically completes within 5–10 minutes per account.

Note: The DynamoDB tables created by KCL at runtime (StockTradeConsumer, StockTradeConsumer-WorkerMetricStats, StockTradeConsumer-CoordinatorState, and the corresponding StockTradeConsumerEFO-* tables if you ran the EFO consumer) are not managed by CloudFormation and must be deleted manually from Account C.

Common errors in cross-account

AccessDenied on AssumeRole (producer): Account B’s role trust policy is missing Account A’s principal or the external ID doesn’t match.
AccessDenied on reads despite the resource policy (consumer): Granted on the stream but not on the consumer’s own identity policy. Cross-account needs both.
AWS KMS AccessDenied: The key policy in the stream account does not grant the foreign producer or consumer GenerateDataKey/Decrypt.
EFO failures: Resource policy missing SubscribeToShard, or the EFO consumer was not pre-registered or shared.
Expired temporary credentials (producer): Long-running KPL must refresh assumed-role credentials, not assume once at startup.

What’s new in KPL and KCL

Kinesis Client Library (KCL) 3.5: The newest minor version of KCL introduces a streamlined single-table format and leader-driven migration paths for high-scale stream processing applications. KCL 3.5 consolidates all internal tracking (including worker leases, coordinator states, and resource utilization metrics) into a single unified Amazon DynamoDB table, rather than spreading data across multiple tables. This translates directly into lower baseline AWS database costs, reduced operational complexity, and simpler IAM cross-account permission models for enterprise stream consumer fleets.

Kinesis Client Library (KCL) 3.0: The latest major version of KCL introduces graceful load balancing and reduced compute costs for stream processing applications. KCL 3.0 dynamically assigns shards based on actual CPU and memory utilization rather than distributing them evenly across workers. As a result, you can process the same streaming volume with fewer compute instances. This translates directly into lower infrastructure spend for cross-account architectures where consumer fleets must be right-sized.

Kinesis Client Library (KCL) 2.5.3+: Added native support for cross-account access through resource-based policies, so KCL consumers can read from streams in other accounts without complex IAM role chaining. For KCL 1.x users, version 1.15.0 or higher is required for cross-account support.

Kinesis Producer Library (KPL) 1.x: KPL continues to provide high-throughput, batched record aggregation with automatic retries. When used cross-account, KPL assumes an IAM role in the stream-owning account to publish records, using AWS Security Token Service (AWS STS) for temporary credential exchange.

Conclusion

In this post, we demonstrated how to build a cross-account streaming pipeline using Amazon Kinesis Data Streams with KPL and KCL deployed across three separate AWS accounts. With this pattern, you can maintain security isolation, cost attribution, and operational independence while processing real-time data at scale. The resource-based policy model introduced for Kinesis Data Streams simplifies cross-account access without complex IAM role chaining, and KCL’s intelligent load balancing helps you right-size consumer compute regardless of deployment topology.

The solution code is available in the sample-kinesis-cross-account-kpl-kcl repository on GitHub. Clone it, customize it for your account structure, and deploy a cross-account streaming pipeline with maximum security isolation.

About the authors

Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service

Anupa Bhattacharyya — Tue, 06 Oct 2026 16:17:46 +0000

When your MWAA-orchestrated extract, transform, and load (ETL) pipeline spans multiple AWS services, troubleshooting a failure becomes a scavenger hunt. AWS Glue jobs transform data, custom scripts run on Amazon Elastic Compute Cloud (Amazon EC2), and directed acyclic graphs (DAGs) in Amazon Managed Workflows for Apache Airflow (Amazon MWAA) coordinate the workflow, but each service writes logs to its own Amazon CloudWatch log group. When something breaks at 2 AM, your team spends valuable time locating the right log stream before they can even begin diagnosing the root cause.

These observability challenges aren’t unique to any one team. DevOps engineers routinely juggle multiple tools and must analyze numerous logs to identify and resolve an issue. Every pivot between tools or logs costs minutes during an outage and directly inflates mean time to resolution. Interpreting logs is another challenge. Even when engineers locate the right log stream, parsing the output requires deep familiarity with each service’s logging conventions. A single failed task can scatter relevant context across dozens of verbose, interleaved log entries that obscure the root cause rather than reveal it.

This solution helps you remediate errors in an analytics pipeline by using an AI agent to speed up root cause identification, interpret relevant logs, and recommend how to resolve the issue. In this post, you learn how to implement this analytics observability solution. The target audience is data engineers, DevOps engineers, and cloud engineers.

You deploy a set of provided AWS CloudFormation templates and an Amazon SageMaker AI notebook to implement a sample architecture and create a set of demo ETL jobs orchestrated by Amazon MWAA. The CloudFormation templates deploy the architectural components, and the SageMaker AI notebook contains code to configure the components and interact with the MCP server.

Solution overview

This solution uses CloudWatch real-time streaming to centralize the logs in Amazon OpenSearch Service. With an OpenSearch MCP server running on Amazon Bedrock AgentCore, engineers can identify issues and receive recommendations through the ETL analysis agent.

Figure 1: Solution architecture that streams ETL logs into Amazon OpenSearch Service and queries them through an MCP server on Amazon Bedrock AgentCore

Prerequisites

Before deploying this solution, make sure you have the following in place:

AWS account and AWS Region

An active AWS account with access to the US East (N. Virginia) us-east-1 Region. CloudFormation stacks must be deployed in us-east-1.

IAM permissions

An AWS Identity and Access Management (IAM) user or role with permissions to create and manage the following AWS resources:

Amazon OpenSearch Service (domain creation, fine-grained access control).
Amazon MWAA (environment creation, DAG execution).
Amazon MWAA Serverless (workflow creation and execution, a versioned S3 bucket for the workflow definition, and a workflow execution role that requires iam:PassRole).
AWS Glue (job creation and execution).
Amazon EC2 (instance launch, security groups).
Amazon Simple Storage Service (Amazon S3) (bucket creation, object management).
AWS Lambda (function creation and execution).
Amazon CloudWatch Logs (log group creation, subscription filters).
Amazon SageMaker AI (notebook instance creation).
Amazon Bedrock (model access, and the AgentCore runtime, a capability of Amazon Bedrock AgentCore).
Amazon Cognito (user pool creation).
Amazon Elastic Container Registry (Amazon ECR) (repository creation).
AWS CodeBuild (project creation).
AWS CloudFormation (stack creation with IAM resources).
AWS Secrets Manager (secret creation).
IAM (role and policy creation).

Amazon Bedrock model access

Enable access to the Anthropic Claude Sonnet model in the Amazon Bedrock console. Navigate to Model access in the Amazon Bedrock console and request access if it isn’t already enabled.

CloudFormation templates

Download the three CloudFormation template files (opensearch_cfn.yaml, etl.yaml, and agentcore-mcp-server.yaml) from the provided GitHub repository before beginning deployment.

Networking

The ETL stack creates a new virtual private cloud (VPC) (CIDR 10.192.0.0/16 by default). Check that this CIDR range doesn’t conflict with existing VPCs in your account if you plan to set up VPC peering or connectivity.

Architecture

The architecture uses Amazon MWAA (provisioned and serverless) as the orchestration layer. As a managed Apache Airflow service, Amazon MWAA lets teams author complex, dependency-aware pipelines as code and schedule, retry, and monitor them without provisioning or operating any Airflow infrastructure. An Airflow DAG defines the pipeline workflow, triggering AWS Glue ETL jobs and Python scripts running on Amazon EC2 instances. Each of these components generates logs that flow into Amazon CloudWatch Logs: Amazon MWAA through its native integration, AWS Glue through its default log group configuration, and Amazon EC2 through the CloudWatch agent.

CloudWatch subscription filters provide the bridge between log storage and analysis. When configured, these filters immediately start streaming real-time log data from selected log groups to Amazon OpenSearch Service. This approach means that data doesn’t need to be copied or duplicated. The subscription filter creates a real-time streaming pipeline that indexes logs as they arrive.

Within OpenSearch, the ML Connector framework integrates with Amazon Bedrock to provide large language model (LLM)-based inference over the log indices. The OpenSearch MCP (Model Context Protocol) server then exposes these capabilities to AI assistants, so users can query their pipeline logs using natural language to identify errors, understand failure patterns, and receive contextual remediation suggestions.

Orchestration and log generation

The architecture begins with Amazon MWAA as the orchestration layer. An Airflow DAG defines the pipeline workflow, triggering AWS Glue ETL jobs and Python scripts running on Amazon EC2 instances. Each of these components generates logs that flow into Amazon CloudWatch Logs.

Workflow

The Amazon MWAA DAG triggers the ETL workflow on a scheduled or event-driven basis. AWS Glue jobs run Spark-based transformations, and logs flow automatically to the /aws-glue/ CloudWatch log group. In parallel, Amazon EC2 Python scripts run custom processing logic and send their logs to CloudWatch. Amazon MWAA task logs automatically land in /airflow/{env}/ log groups. Finally, the CloudWatch unified agent ships logs to the designated log group, where they can be queried through the OpenSearch MCP server.

Log integration methods

Component	Integration	Log group
Amazon MWAA	Native integration	/airflow/{env}/task
AWS Glue	Default log configuration	/aws-glue/jobs/output
Amazon EC2 scripts	CloudWatch unified agent	/ec2/etl-scripts

Figure 2: Log generation and integration across Amazon MWAA, AWS Glue, and Amazon EC2 components

Real-time log streaming

Process

Subscription filters are configured on each CloudWatch log group to match incoming log events.
An AWS Lambda function decompresses gzip-encoded log data, parses and enriches records, and formats them for the OpenSearch Bulk API.
Transformed data is bulk-indexed into Amazon OpenSearch Service domain indices (airflow-logs-*, glue-logs-*, ec2-logs-*, unified-etl-*).

Figure 3: Real-time log streaming from CloudWatch through AWS Lambda into Amazon OpenSearch Service indices

AI-powered analysis and MCP interface

Within OpenSearch, the ML Connector framework integrates with Amazon Bedrock to provide LLM-based inference over the log indices. The OpenSearch MCP (Model Context Protocol) server then exposes these capabilities to AI assistants, so users can query their pipeline logs using natural language to identify errors, understand failure patterns, and receive contextual remediation suggestions.

Process

A user submits a natural language query through the AI assistant (for example, “Why did the Glue job fail at 3 AM?”).
The MCP server translates the query into OpenSearch DSL with ML-enhanced ranking and semantic search.
The ML Connector invokes Amazon Bedrock (Claude) for semantic understanding, log summarization, and pattern detection.
The AI assistant returns root cause analysis with specific remediation steps to the user.

Figure 4: AI-powered log analysis flow from a natural language query to root cause and remediation guidance

Key architectural benefits

Zero data duplication: Subscription filters stream data directly without batch exports, S3 staging, or data copying.
Near real-time: Logs are indexed in OpenSearch shortly after generation in source systems.
Natural language: Users query logs conversationally through MCP, with no need to write OpenSearch DSL manually.
Reduced mean time to resolution: Root cause and remediation recommendations are delivered in a single query, reducing mean time to resolution.
Unified view: ETL components are observable through a single search interface.
Managed services: There’s no infrastructure to provision, patch, or maintain.

CloudFormation stacks

The solution is split into three CloudFormation stacks. Each template provides a distinct layer of the pipeline.

Stack	Template file	Deploy time	Purpose
opensearch-cfn	`opensearch_cfn.yaml`	~15–20 min	OpenSearch domain, SageMaker AI notebook, IAM roles
etl	`etl.yaml`	~25–30 min	VPC, Amazon MWAA, AWS Glue, Amazon EC2, log streaming pipeline
agentcore-mcp-server	`agentcore-mcp-server.yaml`	~8–12 min	Amazon Bedrock AgentCore MCP Server with Amazon Cognito authentication

Stack 1: opensearch-cfn (opensearch_cfn.yaml)

This stack provisions the foundational OpenSearch domain along with a classic SageMaker AI notebook instance for interactive exploration. This stack creates:

An Amazon OpenSearch Service domain (OpenSearch 3.5) with fine-grained access control.
A SageMaker AI notebook instance preloaded with workshop lab notebooks.
S3 buckets for ETL data, DAGs, and more.
IAM roles for notebook and Amazon Bedrock access.
A Secrets Manager secret for OpenSearch credentials.

Parameter	Default	Description
OpenSearchUsername	admin	Admin username for the OpenSearch cluster
OpenSearchPassword	(secure)	Admin password (8–32 chars, letters + numbers + symbols)

Stack 2: etl (etl.yaml)

This stack provisions the ETL resources: the networking, orchestration, compute, and log streaming pipeline that feeds OpenSearch. This stack creates:

A VPC with private subnets and a NAT gateway for Amazon MWAA networking.
An Amazon MWAA environment running an Airflow DAG.
An AWS Glue ETL job (reads XLSX, drops a column, writes JSON).
An Amazon EC2 instance running a parallel Python ETL script.
An Amazon MWAA Serverless workflow for aggregation.
S3 buckets for DAG storage and ETL data.

Parameter	Default	Description
EC2InstanceType	t3.micro	EC2 instance type for the ETL script
VpcCIDR	10.192.0.0/16	CIDR block for the Amazon MWAA VPC
OpenSearchStackName	opensearch-cfn	Name of the OpenSearch stack (for cross-stack imports)

Stack 3: agentcore-mcp-server (agentcore-mcp-server.yaml)

This stack deploys an Amazon Bedrock AgentCore MCP Server that exposes OpenSearch tools (ListIndexTool, IndexMappingTool, SearchIndexTool) for natural language log queries. It includes Amazon Cognito authentication and a containerized MCP server built through CodeBuild. This stack creates:

An Amazon Bedrock AgentCore runtime hosting the OpenSearch MCP server.
An Amazon Cognito user pool for OAuth authentication.
An Amazon ECR repository for the MCP server container.
A CodeBuild project to build and deploy the container.

Parameter	Default	Description
MultimodalStackName	opensearch-cfn	OpenSearch stack name (for importing domain endpoint)
AgentCoreMCPServerName	opensearch_mcp_server	MCP server name (max 35 chars, appended with unique ID)
AmazonOpenSearchEndpoint	(auto-import)	Leave blank to auto-import from opensearch-cfn stack
ExecutionRole	(auto-create)	Leave blank to create a new role
ECRRepository	(auto-create)	Leave blank to create a new ECR repo
OAuthDiscoveryURL	(auto-create Cognito)	Leave blank to create a new Amazon Cognito user pool

Deployment order: opensearch-cfn, followed by etl and agentcore-mcp-server. The etl and agentcore-mcp-server CloudFormation templates use outputs from opensearch-cfn.

Implementation

Deploy each CloudFormation stack in sequence

Deploy the OpenSearch infrastructure for opensearch-cfn.
Go to the AWS CloudFormation console, confirm you are in us-east-1, and confirm that Amazon Bedrock is available.

Figure 5: Confirming the us-east-1 Region in the AWS CloudFormation console
Create the stack. Choose Create stack (with new resources), and then choose an existing template. Select Upload a template file, choose the opensearch-cfn YAML file, and choose Next.

Figure 6: Uploading the opensearch-cfn template on the Create stack page
Configure stack options. For stack name, enter opensearch-cfn, leave the parameters as their defaults, and choose Next.

Figure 7: Entering the stack name on the Configure stack options page
Review and deploy. Review the parameter summary, then scroll to the bottom and select I acknowledge that AWS CloudFormation might create IAM resources with custom names. Choose Next, and then choose Submit.

Figure 8: Acknowledging IAM resource creation on the review page

Figure 9: Reviewing and submitting the stack

Wait for the stack to complete until its status changes to CREATE_COMPLETE.
Deploy the ETL stack the same way. Set the stack name to etl, leave the parameters as their defaults, and wait for the status to change to CREATE_COMPLETE.
Deploy the agentcore-mcp-server stack the same way. Set the stack name to agentcore-mcp-server, leave the parameters as their defaults, and wait for the status to change to CREATE_COMPLETE.

Work with the SageMaker AI notebook

Open the SageMaker AI notebook. Go to AWS CloudFormation on the AWS Management Console and select the opensearch-cfn stack.
Select the Outputs tab, scroll down, and open the URL for the SageMaker AI notebook.
After SageMaker AI has loaded, select Lab-OpenSearch-Observability.ipnyb from the left side menu to open the notebook.
Run each cell in order. To do this, select the cell and then choose the play button (the right-facing triangle).
Complete the prerequisites. Section 1 of the notebook loads the required Python modules to run the code in this notebook. It also retrieves resource metadata for the resources created by the CloudFormation stacks. These cells need to run before you move to section 2.

Figure 10: Load the required libraries and import the Python modules

Figure 11: Load the CloudFormation stack outputs
Section 2: Connect to OpenSearch.

Figure 12: Authenticate by retrieving the OpenSearch admin credentials from Secrets Manager

Figure 13: Grant access to the notebook role and the AgentCore execution role

Figure 14: Switch to IAM-based authentication

Figure 15: Persist connection variables for use in subsequent cells and notebooks
Section 3: Stream CloudWatch Logs into OpenSearch.

Figure 16: Create the Lambda function and CloudWatch subscription filters that stream ETL-related log groups into OpenSearch indices

Figure 17: Create the IAM role for the Lambda function

Figure 18: Create the Lambda function

Figure 19: Grant CloudWatch Logs permission to invoke the Lambda function

Figure 20: Map the Lambda execution role to the OpenSearch all_access role
Section 4: Trigger the ETL DAG.
Trigger the ETL DAG and generate logs through the USE_SERVERLESS flag to select your preferred runtime environment.

When set to False (the default), the observability_etl_dag DAG runs on provisioned Amazon MWAA, running AWS Glue and Amazon EC2 tasks in parallel. When set to True, the observability_blog_aggregation DAG runs on Amazon MWAA Serverless, running an AWS Glue aggregation job.

Figure 21: Trigger the ETL DAG

Figure 22: Verify that the DAG has stopped running

Figure 23: Verify log ingestion
Section 5: Register the Claude LLM connector in OpenSearch.
1. Create an ML connector in OpenSearch that calls the Anthropic Claude model on Amazon Bedrock (us.anthropic.claude-sonnet-4-20250514-v1:0) through the Converse API, using SigV4 authentication and an assumed IAM role.
2. Register and deploy the model so that OpenSearch can use it for ML-powered features such as Retrieval Augmented Generation (RAG) and conversational search.
Figure 24: Create the machine learning connector to the Claude model

Figure 25: Register and deploy the model in OpenSearch
Section 6: Register the AI agent with the OpenSearch MCP server.
1. Install the agent libraries (mcp, strands-agents, uv).
2. Load the Amazon Cognito credentials for authenticating with the AgentCore MCP Server.
3. Choose a deployment mode. Option A (local) runs the OpenSearch MCP server as a subprocess through uvx for development.
Figure 26: Run the OpenSearch MCP server locally as a subprocess

Option B (AgentCore) connects to a production MCP server on Amazon Bedrock AgentCore using OAuth 2.0 client credentials.

Figure 27: Connect to the production MCP server on Amazon Bedrock AgentCore

d. Create the ETL analysis agent.

Figure 28: Create the ETL analysis agent

Results

In section 7 of the notebook, you can ask questions about your ETL pipeline. A sample question is included in the notebook: “What errors do you see in the logs”. Try asking additional questions about the ETL pipeline and related services. The agent autonomously searches indices, correlates events, and returns an analysis of the error with remediation guidance.

Example natural language queries

What you want to find	Example prompt
Errors across the sources	Show me the ERROR level logs
AWS Glue job success logs	Find successful AWS Glue ETL job completions
Amazon EC2 script failures	Show me Amazon EC2 ETL script errors with stack traces
Amazon MWAA task failures	Find failed Amazon MWAA DAG tasks
Amazon MWAA Serverless workflow logs	Show me logs from the Amazon MWAA Serverless aggregation workflow
Recent activity	Show me the last 20 log entries from any source
Specific time range	Show me logs from the last 30 minutes

The following screenshot shows a natural language query being sent to the search_agent through the MCP client, with the agent using multiple tools (ListIndexTool, IndexMappingTool, SearchIndexTool) to discover indices, understand intent, and return structured findings from the pipeline-logs index.

Figure 29: Example natural language query and the agent’s structured findings

Outcome

After a single CloudFormation deployment and five steps, you have a pipeline that processes tabular data through parallel ETL paths, aggregates the results, and consolidates operational logs into one searchable index. When something breaks, you open one dashboard, type what you are looking for, and get your answer. There is no tab-hopping, no timestamp-matching, and no guessing which service threw the error.

The combination of Amazon MWAA for orchestration, OpenSearch for log aggregation, and Amazon Bedrock for natural language access gives you an observability layer that your team actually uses, because it is faster than the alternative.

Clean up

To remove the services used in this solution, delete the three stacks using the AWS CloudFormation console, or run the following command in the AWS CLI:

aws cloudformation delete-stack --stack-name <stack name>

This removes the provisioned resources, including the OpenSearch domain, Amazon MWAA environment, AWS Glue jobs, Amazon EC2 instance, and associated IAM roles.

Conclusion

Observability for a multi-service ETL pipeline doesn’t need to mean stitching together multiple CloudWatch log groups by hand. By streaming every component’s logs into a single OpenSearch index and putting an Amazon Bedrock model in front of it, you turn “I need to find the right log group and write a filter expression” into “show me errors from AWS Glue in the last hour”.

Where to go from here:

Add alerting: configure OpenSearch alerting rules to notify your team through Amazon Simple Notification Service (Amazon SNS) when ERROR-level logs exceed a threshold.
Expand the index: add logs from other components (AWS Step Functions, AWS Lambda, and additional AWS Glue jobs) by creating new CloudWatch subscription filters.
Build dashboards: use the OpenSearch UI for visualizations, such as error rate over time, log volume by source, and latency between DAG trigger and job completion.
Fine-tune the Amazon Bedrock model: adjust the prompt template in the ML connector to include your index mapping, which improves query accuracy for domain-specific questions.
Use OpenSearch Ask AI directly from the OpenSearch UI.

The goal is straightforward: when your pipeline fails, you should spend your time fixing the problem, not finding it. Try the solution in your own environment and tell us what you think in the comments.

About the authors

Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue

Bezuayehu Wate — Tue, 06 Oct 2026 16:13:02 +0000

Many data integration jobs are SQL-centric: they filter, join, and aggregate data on a schedule, and they run frequently enough that fast startup matters. For this shape of work, teams want to match the engine to the job and run it quickly and cost-effectively, without standing up and tuning separate infrastructure.

AWS Glue is the serverless data integration service that customers use to run extract, transform, and load (ETL) jobs at any scale, without managing infrastructure. With AWS Glue, you can run DuckDB, an embedded, in-process, vectorized SQL engine, inside a standard AWS Glue job. DuckDB reads Parquet files from Amazon Simple Storage Service (Amazon S3) and writes Apache Iceberg tables directly to Amazon S3 Tables, a capability of Amazon S3. DuckDB is an open source, in-process, vectorized analytical SQL engine that runs embedded in your application, with no separate server or cluster to manage. It reads and writes cloud data formats such as Parquet and Apache Iceberg natively. AWS Glue 6.0 is the latest version, running on a modernized runtime with a 30 percent price reduction over previous versions. DuckDB reads Amazon S3 Parquet through its httpfs extension and commits Iceberg snapshots to Amazon S3 Tables through the Iceberg REST endpoint, so no separate catalog synchronization is required. Running DuckDB in AWS Glue is well suited to SQL-centric transformations such as filters, joins, and aggregations. It also fits frequent, scheduled jobs such as hourly or daily aggregations, incremental loads, and rollups that benefit from fast startup. This pattern complements Apache Spark on AWS Glue rather than replacing it: when a workload needs distributed processing, the same job type runs PySpark with no change to your infrastructure, IAM, or triggers.

This post walks through the pattern with a concrete ETL use case and provides complete, runnable code. It also compares measured cost and runtime against a Spark job performing the same work on the same AWS Glue 6.0 runtime.

When to use this pattern

This pattern is a complement to Spark on AWS Glue, not a replacement. The following table summarizes when each approach yields the best results.

Signal	DuckDB on AWS Glue 6.0	Apache Spark on AWS Glue 6.0
Dataset size per run	Scales with worker size	Scales horizontally across multiple nodes for datasets of any size
Parallelism requirement	Single-node, in-process execution	Distributed processing across a managed cluster
SQL complexity	Aggregations, joins, window functions	Complex graph operations, custom UDFs, ML pipelines
Cost priority	Minimize per-run cost and duration	Maximize throughput at scale
Iceberg writes	DuckDB iceberg extension to S3 Tables	Native Spark Iceberg integration

For workloads that need distributed processing, the same glueetl job type runs PySpark with no change to your infrastructure, AWS Identity and Access Management (IAM) configuration, or triggers. You choose the engine that fits each workload.

How DuckDB runs on AWS Glue 6.0

Running DuckDB in an AWS Glue job comes down to two things working together: a runtime modern enough to load DuckDB and its native extensions, and the capabilities DuckDB brings to ETL once it does.

What the AWS Glue 6.0 runtime provides

Modern runtime compatibility. AWS Glue 6.0 runs on Amazon Linux 2023 with glibc 2.34 and Python 3.13. DuckDB 1.5.x and its native C++ extension binaries (httpfs, aws, iceberg) install through pip and load without workarounds. The DuckDB extension binaries require a modern glibc (2.28 or later), which the AWS Glue 6.0 runtime provides.

AWS Glue 6.0 resolves this compatibility requirement. You can add DuckDB 1.5.x to an AWS Glue 6.0 job in two ways. The first is the --additional-python-modules job parameter (duckdb==1.5.1), which pip-installs the package at job startup and loads all extensions without additional steps. Alternatively, you can package the dependencies as a Python virtual environment, upload it to Amazon S3, and reference it using the --python-virtual-env parameter. On AWS Glue 6.0, you can also add --python-virtual-env-storage-prefix to have AWS Glue build and cache the virtual environment automatically. For more information, see Using Python virtual environments with AWS Glue.

What DuckDB provides

DuckDB is an open source, in-process analytical SQL engine. It runs inside an AWS Glue job as a single process, with no separate cluster or coordinator. The following capabilities make it a practical fit for ETL on the AWS Glue 6.0 runtime.

Single-node vectorized execution. DuckDB runs inside a single AWS Glue job. For a couple of gigabytes, there is no shuffle, no executor scheduling, and no inter-node network I/O. The work happens in a single vectorized pass over columnar memory.
Native Amazon S3 and Parquet access. The httpfs extension reads and writes Amazon S3 objects directly, using the IAM role of the AWS Glue job automatically through CREDENTIAL_CHAIN.
Native Amazon S3 Tables writes. The iceberg extension connects to the Amazon S3 Tables Iceberg REST endpoint (ENDPOINT_TYPE s3_tables) and commits standard Iceberg snapshots. With AWS Glue 6.0, you can use two capabilities that matured independently: Amazon S3 Tables and DuckDB Iceberg writes.
Larger-than-memory operators. Sort, join, and aggregate spill to /tmp, so datasets larger than available RAM still process without code changes.

The output is a standard Apache Iceberg table in Amazon S3 Tables. It is queryable by Amazon Athena, Amazon Redshift, and Amazon EMR, and other Iceberg-compatible engines that support the Iceberg REST Catalog API.

Sizing guidance. DuckDB runs within a single AWS Glue worker, so its available memory and disk scale with the worker type. This walkthrough uses the minimum glueetl configuration of 2 workers (2 data processing units, or DPUs) with worker type G.1X: each G.1X worker provides 4 vCPUs and 16 GB of memory. DuckDB runs on the driver and processes data in memory, spilling to local disk when a dataset or intermediate result exceeds available RAM. For larger inputs, choose a bigger worker: G.2X provides 8 vCPUs and 32 GB of memory, and the G.4X and G.8X types scale higher. Size the worker to your input volume and the memory footprint of your aggregations and joins. For current specifications, see AWS Glue worker types.

Architecture

The following image shows the architecture described in this post.

The pipeline consists of the following managed components:

Layer	Role	AWS Service
Source	Raw Parquet files, partitioned by date	Amazon S3
Compute	DuckDB SQL engine running on the AWS Glue 6.0 runtime	AWS Glue 6.0 (`glueetl`)
Destination	Iceberg analytical tables, queryable by any engine	Amazon S3 Tables
Query	Analytics and business intelligence (BI) on the output tables	Amazon Athena, Amazon QuickSight

Raw Parquet files land in Amazon S3 on a schedule. An AWS Glue 6.0 job runs DuckDB. DuckDB reads the files, applies SQL transformations in memory, and writes the aggregated result as an Iceberg table to Amazon S3 Tables through the Iceberg REST catalog. Amazon Athena and Amazon QuickSight can query the output immediately. No separate catalog synchronization is required.

You can trigger the job several ways:

With a native AWS Glue trigger, which can be scheduled, on-demand, or conditional on another job or crawler completing.
With Amazon EventBridge Scheduler calling StartJobRun on a fixed schedule.
In response to new data through an Amazon S3 event notification that invokes an AWS Lambda function calling StartJobRun.
As part of an AWS Glue Workflow.

Walkthrough: eCommerce daily order summary

This section walks through a daily ETL pipeline for an eCommerce application. The pipeline reads raw transaction files from Amazon S3, cleanses and aggregates them, and writes a query-ready summary to Amazon S3 Tables.

Step	Operation	Detail
1. Source	Read raw Parquet from S3	`s3://<amzn-s3-demo-source-bucket>/orders/year=2026/month=08/*.parquet`
2. Filter	`status IN (‘completed’,‘processing’)`	Drop canceled and test orders
3. Enrich	`net_revenue`, `avg_order_value`	Derived columns via SQL expressions
4. Aggregate	`GROUP BY order_day, region, category`	Daily revenue, order count, unique customers
5. Write	`INSERT` into an Amazon S3 Tables Iceberg table	Idempotent per-day reload

Prerequisites

An AWS account with permissions for AWS Glue, Amazon S3, Amazon S3 Tables, and AWS Identity and Access Management (IAM).
An Amazon S3 bucket containing raw Parquet files (referred to as <amzn-s3-demo-source-bucket> in this post).
An Amazon S3 Tables table bucket (referred to as <amzn-s3-demo-table-bucket> in this post). See the Create the S3 Tables table bucket section.
An IAM role for the AWS Glue job with:
- Amazon S3 read access on <amzn-s3-demo-source-bucket>.
- Amazon S3 Tables permissions on the table bucket ARN: s3tables:GetTableBucket, s3tables:ListNamespaces, s3tables:GetNamespace, s3tables:CreateNamespace, s3tables:ListTables, s3tables:GetTable, s3tables:CreateTable, s3tables:GetTableMetadataLocation, s3tables:UpdateTableMetadataLocation, s3tables:GetTableData, and s3tables:PutTableData
- AWS Glue job execution permissions.
An AWS Glue 6.0 job (glueetl) with:
- --additional-python-modules: duckdb==1.5.1.
- Minimum worker configuration: 2 workers, type G.1X.

Note on DuckDB versions. DuckDB support for writing Apache Iceberg tables through a REST catalog, including Amazon S3 Tables, requires version 1.4.0 or later. This walkthrough uses duckdb==1.5.1. On the AWS Glue 6.0 runtime (Amazon Linux 2023), it installs and all native extensions load without additional configuration.

Create the S3 Tables table bucket

If you don’t already have an Amazon S3 Tables table bucket, create one using the AWS Command Line Interface (AWS CLI):

aws s3tables create-table-bucket \
    --name <amzn-s3-demo-table-bucket> \
    --region <YOUR-REGION>

Note the table bucket Amazon Resource Name (ARN) from the output. It follows the format:

arn:aws:s3tables:<YOUR-REGION>:<YOUR-ACCOUNT-ID>:bucket/<amzn-s3-demo-table-bucket>

Turn on integration with AWS analytics services so the table is discoverable by Amazon Athena, Amazon Redshift, and Amazon EMR. Complete the integration by creating the s3tablescatalog catalog in the AWS Glue Data Catalog using the AWS CLI. For the steps, see Integrating Amazon S3 Tables with AWS analytics services.

Generate sample data

This walkthrough uses a synthetic eCommerce dataset. Run the following Python script locally or in AWS CloudShell to generate Parquet files that match the schema used in the transform. It produces roughly 8.4 million rows across 12 files (about 94 MB on disk as Parquet, roughly 1.2 GB uncompressed in memory).

import pandas as pd
import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import os

np.random.seed(42)

N = 8_400_000  # ~8.4M rows
NUM_FILES = 12  # split across 12 files to mimic a partitioned landing zone

df = pd.DataFrame({
    "order_date": pd.date_range("2026-08-01", periods=N, freq="s"),
    "region": np.random.choice(["US", "EU", "APAC"], N),
    "category": np.random.choice(["electronics", "books", "home", "clothing"], N),
    "status": np.random.choice(
        ["completed", "processing", "cancelled"], N, p=[0.6, 0.3, 0.1]
    ),
    "quantity": np.random.randint(1, 10, N),
    "unit_price": np.round(np.random.uniform(5.0, 200.0, N), 2),
    "customer_id": np.random.randint(1000, 9999, N),
})

os.makedirs("sample_orders", exist_ok=True)

for i, chunk in enumerate(np.array_split(df, NUM_FILES)):
    path = f"sample_orders/orders_part_{i}.parquet"
    pq.write_table(pa.Table.from_pandas(chunk), path)
    print(f"Wrote {path} ({os.path.getsize(path):,} bytes)")

Upload the generated files to your source bucket:

aws s3 cp sample_orders/ \
    s3://<amzn-s3-demo-source-bucket>/orders/ \
    --recursive

Note. The CLI commands and code examples in this walkthrough use angle-bracket placeholders such as <amzn-s3-demo-source-bucket> and <amzn-s3-demo-table-bucket>. Replace these with your own values before running.

Step 1: Configure DuckDB in the AWS Glue 6.0 job

The AWS Glue filesystem is read-only except for /tmp, so DuckDB uses /tmp as a writable home directory for its extension cache and spill files. The job loads DuckDB extensions: httpfs reads and writes Amazon S3 objects directly, aws handles AWS credential resolution, refresh, and AWS Region detection, and iceberg connects to the Amazon S3 Tables REST catalog. The CREDENTIAL_CHAIN provider (from the aws extension) tells DuckDB to use the standard AWS credential provider chain, which automatically picks up the IAM role attached to the AWS Glue job. No access keys or secrets appear in the code.

import os
import duckdb

os.makedirs('/tmp/.duckdb/extensions', exist_ok=True)

con = duckdb.connect(':memory:')
con.execute("SET home_directory='/tmp';")
con.execute("SET extension_directory='/tmp/.duckdb/extensions';")

con.execute("INSTALL httpfs; LOAD httpfs;")
con.execute("INSTALL aws; LOAD aws;")
con.execute("INSTALL iceberg; LOAD iceberg;")

con.execute("CREATE SECRET (TYPE s3, PROVIDER credential_chain);")

The home_directory setting must be applied before loading any extensions. Without it, DuckDB attempts to write to /.duckdb/ and fails with IOError: Permission denied.

Step 2: Read and transform with DuckDB SQL

DuckDB reads Amazon S3 Parquet files directly through the httpfs extension. No local download is required. The read_parquet() function accepts Amazon S3 glob patterns, reading multiple files as a single relation.

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ['s3_input_path'])
S3_INPUT = args['s3_input_path']

con.execute(f"""
CREATE OR REPLACE TEMP TABLE _batch AS
SELECT date_trunc('day', order_date) AS order_day,
region, category,
COUNT(*) AS total_orders,
SUM(quantity * unit_price) AS gross_revenue,
SUM(CASE WHEN status = 'completed'
THEN quantity * unit_price ELSE 0 END) AS net_revenue,
COUNT(DISTINCT customer_id) AS unique_customers,
ROUND(AVG(quantity * unit_price), 2) AS avg_order_value,
SUM(CASE WHEN quantity * unit_price > 500
THEN 1 ELSE 0 END) AS high_value_orders
FROM read_parquet('{S3_INPUT}')
WHERE status IN ('completed', 'processing')
GROUP BY ALL
ORDER BY order_day DESC, gross_revenue DESC
""")

GROUP BY ALL is a DuckDB SQL extension that groups by every non-aggregate column in the SELECT list. It’s a convenience feature rather than standard SQL, and support varies across query engines. If you adapt this query for another engine, check whether it supports GROUP BY ALL or list the grouping columns explicitly (GROUP BY order_day, region, category).

The WHERE clause retains both completed and processing orders. The gross_revenue column reflects all in-flight revenue, while net_revenue counts only completed orders. A partition containing only processing orders shows net_revenue = 0. This is by design: the two columns serve different reporting purposes.

Step 3: Write to Amazon S3 Tables

DuckDB attaches the S3 Tables bucket as an Iceberg REST catalog using the ENDPOINT_TYPE s3_tables option. DuckDB commits each write as a new Iceberg snapshot through the catalog.

The write uses an idempotent per-day reload pattern: create the table if it does not exist, delete any existing rows for the batch’s date range, then insert. This way, re-runs don’t produce duplicate rows.

Note: The DELETE and INSERT are not committed atomically. If the job fails between them, the affected partition is left empty. For mitigations, see Error handling for production.

import sys
from awsglue.utils import getResolvedOptions

args = getResolvedOptions(sys.argv, ['s3t_arn'])
S3T_ARN = args['s3t_arn']

con.execute(f"ATTACH '{S3T_ARN}' AS s3t (TYPE iceberg, ENDPOINT_TYPE s3_tables);")
con.execute("CREATE SCHEMA IF NOT EXISTS s3t.analytics;")
con.execute("""
CREATE TABLE IF NOT EXISTS s3t.analytics.daily_order_summary (
order_day DATE,
region VARCHAR,
category VARCHAR,
total_orders BIGINT,
gross_revenue DOUBLE,
net_revenue DOUBLE,
unique_customers BIGINT,
avg_order_value DOUBLE,
high_value_orders BIGINT
);
""")

con.execute("""
DELETE FROM s3t.analytics.daily_order_summary
WHERE order_day IN (SELECT DISTINCT order_day FROM _batch);
""")
con.execute("""
INSERT INTO s3t.analytics.daily_order_summary BY NAME
SELECT * FROM _batch;
""")

count = con.execute(
"SELECT COUNT(*) FROM s3t.analytics.daily_order_summary"
).fetchone()[0]
print(f"S3 Tables now holds {count} rows in analytics.daily_order_summary")

The resulting Iceberg table is immediately readable by Amazon Athena, Amazon Redshift, and Amazon EMR through the S3 Tables REST catalog. Amazon S3 Tables handles compaction, snapshot expiration, and orphan-file cleanup automatically.

Complete AWS Glue 6.0 job script

The following script combines all three steps with structured logging, error handling, and AWS Glue job parameter parsing. It can be used directly as the script for an AWS Glue 6.0 glueetl job.

import os, sys, logging, duckdb
from awsglue.utils import getResolvedOptions

logging.basicConfig(level=logging.INFO,
                    format='%(asctime)s %(levelname)s %(message)s')
logger = logging.getLogger(__name__)

TARGET = 's3t.analytics.daily_order_summary'

DDL = """
CREATE TABLE IF NOT EXISTS s3t.analytics.daily_order_summary (
order_day DATE, region VARCHAR, category VARCHAR,
total_orders BIGINT, gross_revenue DOUBLE, net_revenue DOUBLE,
unique_customers BIGINT, avg_order_value DOUBLE, high_value_orders BIGINT
);
"""

TRANSFORM = """
SELECT date_trunc('day', order_date) AS order_day,
region, category,
COUNT(*) AS total_orders,
SUM(quantity * unit_price) AS gross_revenue,
SUM(CASE WHEN status = 'completed' THEN quantity * unit_price ELSE 0 END) AS net_revenue,
COUNT(DISTINCT customer_id) AS unique_customers,
ROUND(AVG(quantity * unit_price), 2) AS avg_order_value,
SUM(CASE WHEN quantity * unit_price > 500 THEN 1 ELSE 0 END) AS high_value_orders
FROM read_parquet('{s3_input}')
WHERE status IN ('completed', 'processing')
GROUP BY ALL
ORDER BY order_day DESC, gross_revenue DESC
"""

def setup_duckdb():
    os.makedirs('/tmp/.duckdb/extensions', exist_ok=True)
    con = duckdb.connect(':memory:')
    con.execute("SET home_directory='/tmp';")
    con.execute("SET extension_directory='/tmp/.duckdb/extensions';")
    con.execute("INSTALL httpfs; LOAD httpfs;")
    con.execute("INSTALL aws; LOAD aws;")
    con.execute("INSTALL iceberg; LOAD iceberg;")
    con.execute("CREATE SECRET (TYPE s3, PROVIDER credential_chain);")
    logger.info("DuckDB %s initialized with httpfs, aws, and iceberg extensions",
                duckdb.__version__)
    return con

def transform_orders(con, s3_path):
    logger.info("Reading source data: %s", s3_path)
    con.execute("CREATE OR REPLACE TEMP TABLE _batch AS " +
                TRANSFORM.format(s3_input=s3_path))
    return con.execute("SELECT COUNT(*) FROM _batch").fetchone()[0]

def write_to_s3_tables(con, s3t_arn):
    con.execute(f"ATTACH '{s3t_arn}' AS s3t (TYPE iceberg, ENDPOINT_TYPE s3_tables);")
    con.execute("CREATE SCHEMA IF NOT EXISTS s3t.analytics;")
    con.execute(DDL)
    con.execute(f"""
DELETE FROM {TARGET}
WHERE order_day IN (SELECT DISTINCT order_day FROM _batch);
""")
    con.execute(f"INSERT INTO {TARGET} BY NAME SELECT * FROM _batch;")
    count = con.execute(f"SELECT COUNT(*) FROM {TARGET}").fetchone()[0]
    logger.info("Write complete. %s now holds %s rows.", TARGET, count)
    return count

def main():
    args = getResolvedOptions(sys.argv, ['s3_input_path', 's3t_arn'])
    con = setup_duckdb()
    n = transform_orders(con, args['s3_input_path'])
    logger.info("Transformed %s summary rows", n)
    total = write_to_s3_tables(con, args['s3t_arn'])
    logger.info("ETL complete. Table holds %s total rows.", total)

if __name__ == '__main__':
    main()

Create the job using the AWS CLI:

aws glue create-job \
    --name duckdb-order-summary \
    --role <YOUR-AWS-GLUE-ROLE> \
    --glue-version "6.0" \
    --number-of-workers 2 --worker-type G.1X \
    --command '{"Name":"glueetl","ScriptLocation":"s3://<amzn-s3-demo-source-bucket>/scripts/duckdb_job.py","PythonVersion":"3"}' \
    --default-arguments '{
    "--additional-python-modules": "duckdb==1.5.1",
    "--s3_input_path": "s3://<YOUR-SOURCE-BUCKET>/orders/year=2026/month=08/*.parquet",
    "--s3t_arn": "arn:aws:s3tables:<YOUR-REGION>:<YOUR-ACCOUNT-ID>:bucket/<amzn-s3-demo-table-bucket>"
}'

Note. Replace the angle-bracket placeholders (<amzn-s3-demo-source-bucket>, <amzn-s3-demo-table-bucket>, <YOUR-REGION>, <YOUR-ACCOUNT-ID>, <YOUR-AWS-GLUE-ROLE>) with your own values before running.

Error handling for production

For production use, plan for three failure modes:

Catalog access. If ATTACH to Amazon S3 Tables returns an access-denied error, verify that the job’s IAM role has the s3tables actions listed in the prerequisites on the table bucket ARN.
Partial writes. The DELETE and INSERT are not committed atomically, so a failure between them can leave a partition empty. Set MaxRetries to 1 so the idempotent reload re-runs automatically, or write to a staging table and swap on success.
Timeouts. Set the job Timeout higher than the expected run time to stop hung runs.

Monitoring

DuckDB runs inside a standard AWS Glue job, so you can track each run from the AWS Glue console or the GetJobRun API, including run state, duration, and DPU usage.

To observe DuckDB itself, use structured logging. The production script uses logger.info to record the rows transformed and rows written, and those lines appear in the job’s Amazon CloudWatch Logs stream. For finer-grained timing, add logger.info statements around each stage. To track values over time, such as rows written per run, publish them as custom metrics with the CloudWatch PutMetricData API.

Measured results

The measurements in this section were collected on AWS Glue 6.0 with DuckDB 1.5.1 writing to Amazon S3 Tables in the US East (N. Virginia) Region (us-east-1). Output tables were verified by querying them in Amazon Athena. Both jobs produced identical output: 1,176 summary rows.

The dataset consisted of 8.4 million rows across 12 Parquet files (approximately 94 MB compressed on disk, approximately 1.2 GB uncompressed). One job ran DuckDB on the AWS Glue 6.0 runtime. The other ran Apache Spark on AWS Glue 6.0 with the equivalent transform and a native Iceberg write.

Metric	DuckDB on AWS Glue 6.0	Spark on AWS Glue 6.0
Compute configuration	2 DPU (2x G.1X)	2 DPU (2x G.1X)
Job Duration	~56 seconds	~117 seconds
Billed duration	1 minute (minimum)	2 minutes
Cost per run	$0.0103	$0.0205
Output rows (Athena-verified)	1,176	1,176

On the same AWS Glue 6.0 runtime and the same 2 DPU configuration, DuckDB completed in approximately half the time at approximately half the cost of Spark for this workload.

Cost is calculated at $0.308 per DPU-hour (AWS Glue 6.0 rate). AWS Glue bills in 1-second increments with a 1-minute minimum per run. Verify against the current AWS Glue pricing page for your Region. Results scale with dataset size, query complexity, and Region.

At 20 runs per day, this job costs approximately $75 per year with DuckDB, compared to $150 per year with Spark. Beyond the cost savings, this pattern keeps SQL-centric work quick to iterate on: you express the transformation in SQL, and DuckDB runs it in-process on the AWS Glue worker.

Clean up

To avoid ongoing charges, delete the resources created during this walkthrough:

Delete the AWS Glue job (duckdb-order-summary).
Remove the sample data from your Amazon S3 bucket (s3://<amzn-s3-demo-source-bucket>/orders/).
Drop the Iceberg table in Amazon Athena: DROP TABLE analytics.daily_order_summary;
Delete the Amazon S3 Tables table bucket if it was created for this walkthrough.
Remove the IAM role if no longer needed.

Conclusion

In this post, we demonstrated how to run DuckDB inside an AWS Glue 6.0 job to read Amazon S3 Parquet, transform it with SQL, and write Apache Iceberg tables directly to Amazon S3 Tables. AWS Glue 6.0 modernized the runtime environment to Amazon Linux 2023, Python 3.13, and Apache Spark 4.1. With that modernization, you can run embedded SQL in the AWS Glue job and write Iceberg tables directly to Amazon S3 Tables. For ETL jobs where the data fits in memory on a single worker, this pattern completed the same work in approximately half the time and half the cost of Spark. The Measured results section describes these measurements. The job uses the same glueetl job type, IAM configuration, and triggering mechanisms as any Spark job on AWS Glue. When a workload outgrows single-worker processing, switching the script back to PySpark requires no infrastructure changes. The result is the ability to match the engine to each job: a scheduled SQL transformation and a large distributed workload can run on one platform, and you pick the engine per job without managing separate systems.

To get started, create an AWS Glue 6.0 job, add duckdb==1.5.1 through the --additional-python-modules parameter, and point it at your Amazon S3 source data and an Amazon S3 Tables bucket. The complete script in this post is a working starting point you can adapt to your own datasets and schedules. For more information, see the AWS Glue Developer Guide and the Amazon S3 Tables user guide. For a complementary pattern that uses DuckDB to read and query data in Amazon S3 Tables, see Streamlining access to tabular datasets stored in Amazon S3 Tables with DuckDB.

About the authors

Query across accounts and table formats with multi-catalog in Amazon EMR

Suthan Phillips — Tue, 06 Oct 2026 16:05:27 +0000

Analytics teams on AWS often store data in more than one open table format, and that data frequently lives in more than one AWS account. Two problems follow: querying across table formats without catalog-level complexity, and joining data across accounts without copying it. This is common in a data mesh, where domain teams own data in separate AWS accounts while analytics workloads run centrally. Multi-catalog support in Amazon EMR 8.1.0 addresses both problems.

Amazon EMR release 8.1.0 addresses both challenges with multi-catalog support. The RedirectingSessionCatalog (RSC) is an opt-in catalog that you set as the Spark default. It automatically detects each table’s format from AWS Glue metadata, routes queries to the correct format handler, and supports multiple AWS Glue Data Catalogs across AWS accounts. With multi-catalog support, you can query Iceberg, Hudi, Delta Lake, and Hive tables through a single unified catalog. You can also join tables across AWS accounts without copying data and discover remote catalogs dynamically at query time.

In this post, we show how to put these capabilities into practice using Amazon EMR Serverless.

The Spark single-catalog constraint

Many formats. The Spark default catalog (spark_catalog) accepts only one CatalogExtension at a time: SparkSessionCatalog for Iceberg, DeltaCatalog for Delta Lake, or HoodieCatalog for Hudi. You configure one, and queries against tables in other formats fail unless those tables are registered in a separate, format-specific catalog.

Many accounts. The Spark V1 metastore (ExternalCatalog) is a singleton bound to one AWS Glue Data Catalog in one account. The Spark V2 catalog API supports named catalogs, but only Iceberg uses it. Delta Lake, Hudi, and Hive tables still rely on V1. As a result, cross-account access for those formats required complex multi-step workarounds. These included AWS Lake Formation grants, AWS Resource Access Manager (AWS RAM) shares, resource links, and per-table permissions.

Amazon EMR 8.1.0 addresses both constraints with the RedirectingSessionCatalog, described in the following section.

The RedirectingSessionCatalog

The RedirectingSessionCatalog provides three opt-in, backward-compatible capabilities. Set RSC as the default catalog to run multi-format queries without format-specific prefixes. Declare a named RSC catalog to join across accounts without data copies. Turn on the AWS Glue Data Catalog resolver to discover and register remote catalogs at query time, with no upfront spark.sql.catalog.* configuration. The following sections cover each capability.

Multi-format support

In Amazon EMR 8.1.0, you set the default catalog to the RedirectingSessionCatalog (RSC). On table resolution, RSC calls the AWS Glue Data Catalog, reads the table’s format from its metadata, caches the result, and delegates to the matching format-specific catalog (Iceberg, Delta Lake, Hudi, or Hive).

Without this feature, you had to register a separate catalog for each format and prefix every table reference:

# One catalog per format, all pointing at the same Glue metastore
spark.sql.catalog.spark_catalog = org.apache.iceberg.spark.SparkSessionCatalog
spark.sql.catalog.delta_catalog = org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql.catalog.hudi_catalog = org.apache.spark.sql.hudi.catalog.HoodieCatalog

# Queries must use format-specific catalog prefixes
SELECT * FROM spark_catalog.db.iceberg_table;
SELECT * FROM delta_catalog.db.delta_table;
SELECT * FROM hudi_catalog.db.hudi_table;

With multi-format support, this reduces to a single catalog property:

What you set:

# Replace the default catalog with RedirectingSessionCatalog.
# RSC auto-detects each table's format from Glue metadata.
spark.sql.catalog.spark_catalog = org.apache.spark.sql.connector.catalog.\
redirecting.RedirectingSessionCatalog

How you query:

-- No format prefixes needed. RSC resolves the format at query time.
SELECT * FROM db.iceberg_table;
SELECT * FROM db.delta_table;
SELECT * FROM db.hudi_table;

-- Cross-format joins work in a single statement.
SELECT i.id, d.val, h.val
FROM db.iceberg_table i
JOIN db.delta_table d ON i.id = d.id
JOIN db.hudi_table h ON i.id = h.id;

On table resolution, RSC:

Calls the AWS Glue Data Catalog to read table metadata.
Inspects the table’s Parameters map to determine the format (Iceberg, Delta Lake, Hudi, or Hive/Parquet).
Delegates the operation to the appropriate format-specific catalog implementation.

The following diagram shows how a single query flows through the RedirectingSessionCatalog to the correct format handler.

Figure 1: Multi-format routing. A single query enters the RedirectingSessionCatalog, which calls the AWS Glue Data Catalog to read each table’s format, then routes the table to the matching format-specific handler (Iceberg, Delta Lake, Hudi, or Hive) so results return through one catalog

As the diagram illustrates, the query enters through spark_catalog (the RSC). The RSC reads each table’s format from the AWS Glue Data Catalog and routes the operation to the matching engine: Iceberg, Delta Lake, Hudi, or Hive/Parquet. The four format handlers read the underlying data files from Amazon Simple Storage Service (Amazon S3). The caller issues one query with no format-specific catalog prefixes.

Multi-catalog: Cross-account access

When production data lives in a separate AWS account from your analytics compute, you can declare a named RSC catalog that points at that account’s AWS Glue Data Catalog. RSC resolves tables in the remote account the same way it resolves local tables, so a single query can join across accounts without copying data.

Previously, cross-account table resolution worked only for Iceberg (V2 catalog). Hive, Delta Lake, and Hudi tables in another account required manual Lake Formation and AWS RAM configuration rather than catalog-level resolution. With Amazon EMR 8.1.0, you declare a named RSC catalog for the remote account:

Configuration:

# Declare a named catalog pointing at the remote account's Glue catalog.
# "prod" is any name you choose for this catalog.
spark.sql.catalog.prod = org.apache.spark.sql.connector.catalog.\
redirecting.RedirectingSessionCatalog

# Tell it to use Glue as the metastore backend.
spark.sql.catalog.prod.metastore.type = glue

# Point it at the remote account's Glue catalog ID.
spark.sql.catalog.prod.metastore.hadoop.hive.metastore.glue.catalogid = 111122223333

Query:

-- Join local and remote tables directly. No data copy.
SELECT o.order_id, f.status
FROM spark_catalog.analytics.orders o -- local Iceberg table
JOIN prod.salesdb.fulfillment f -- remote Hudi table (account 111122223333)
ON o.order_id = f.order_id;

Each named RSC instance creates its own V1 metastore delegate and registers it with the global SessionCatalog, removing the singleton limitation.

The following diagram shows how a single query joins tables across two AWS accounts through named catalogs.

Figure 2: Cross-account access. A query in the local account references a named RSC catalog that points at a second account’s AWS Glue Data Catalog, so the local and remote tables join in one query without copying data between accounts

As the diagram illustrates, the analytics account uses spark_catalog (the RSC) for its local Glue Data Catalog, while a named catalog (prod) points at the production account’s Glue Data Catalog. The query joins a local table to a remote table in a single statement, shown by the JOIN between the two accounts. Each account keeps its own Glue Data Catalog, and no data is copied between them.

Auto-wiring

The preceding multi-catalog setup requires you to pre-declare each remote catalog in spark.sql.catalog.* properties. Auto-wiring in Amazon EMR 8.1.0 removes this requirement at two levels.

Before Amazon EMR 8.1.0, you declared the resolver, a handler per format, and each handler’s delegate class explicitly:

# Declare the resolver, a per-format handler, and each handler's delegate catalog
spark.sql.catalog.spark_catalog.table-format-resolver = \
com.amazonaws.glue.catalog.redirecting.GlueTableFormatResolver
spark.sql.catalog.spark_catalog.handler.iceberg = \
org.apache.spark.sql.connector.catalog.redirecting.DefaultTableFormatCatalogHandler
spark.sql.catalog.spark_catalog.handler.delta = \
org.apache.spark.sql.connector.catalog.redirecting.DefaultTableFormatCatalogHandler
spark.sql.catalog.spark_catalog.handler.hudi = \
org.apache.spark.sql.connector.catalog.redirecting.DefaultTableFormatCatalogHandler
spark.sql.catalog.spark_catalog.iceberg.delegate-class = org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.spark_catalog.iceberg.catalog-impl = org.apache.iceberg.aws.glue.GlueCatalog
spark.sql.catalog.spark_catalog.delta.delegate-class = \
org.apache.spark.sql.delta.catalog.DeltaCatalog
spark.sql.catalog.spark_catalog.hudi.delegate-class = \
org.apache.spark.sql.hudi.catalog.HoodieCatalog

# Manually declare every remote catalog you might query
spark.sql.catalog.prod = ...RedirectingSessionCatalog
spark.sql.catalog.prod.metastore.type = glue
spark.sql.catalog.prod.metastore.hadoop.hive.metastore.glue.catalogid = 111122223333
spark.sql.catalog.staging = ...RedirectingSessionCatalog
spark.sql.catalog.staging.metastore.type = glue
spark.sql.catalog.staging.metastore.hadoop.hive.metastore.glue.catalogid = 444455556666

Auto-wiring reduces this to:

Configuration:

# Set the default catalog. EMR auto-registers Iceberg, Delta, Hudi handlers.
# No handler.* properties needed.
spark.sql.catalog.spark_catalog = org.apache.spark.sql.connector.catalog.\
redirecting.RedirectingSessionCatalog

# Enable catalog discovery. The resolver inspects the connection type and
# registers the catalog at query time, with no upfront spark.sql.catalog.* config.
spark.sql.catalogResolver = com.amazonaws.glue.catalog.\
redirecting.GlueCatalogResolver

# Optional. Falls back to the SDK default region chain. Set this to query Glue catalogs in a specific region.
spark.sql.catalogResolver.region = us-east-1

Query:

-- Reference a remote catalog by its account ID. No prior declaration exists.
-- EMR calls Glue GetCatalog, determines the type, registers it on the spot.
SELECT o.order_id, f.status
FROM spark_catalog.analytics.orders o
JOIN `111122223333`.sales.fulfillment f
ON o.order_id = f.order_id;

The AWS Glue Data Catalog resolver is opt-in. When enabled, Amazon EMR issues an AWS Glue GetCatalog API call each time a query references a catalog that hasn’t been registered. This isn’t enabled by default to avoid unintended API calls for catalog names that don’t exist.

At the handler level, setting the default catalog to RedirectingSessionCatalog is enough. Amazon EMR fills in the Iceberg, Delta, and Hudi handlers automatically, so you don’t need to write handler.* properties.

At the catalog level, when you enable the Glue catalog resolver, Amazon EMR discovers new catalogs on demand. The first time a query references an undeclared catalog, Amazon EMR calls the AWS Glue GetCatalog API, inspects the connection type, and registers the catalog at query time.

Dynamic discovery is functional in Amazon EMR 8.1.0 for three catalog types. For standard cross-account AWS Glue catalogs, the resolver registers a redirecting catalog and multi-format routing applies. For Amazon S3 Tables, a capability of Amazon S3, the resolver reads the federated AWS Glue metadata and routes through Iceberg for both reads and writes. It also supports Amazon Redshift Managed Storage.

The following diagram shows how the GlueCatalogResolver discovers and registers a catalog the first time a query references it.

Figure 3: Auto-wiring. When a query references an undeclared catalog, the Glue catalog resolver calls the AWS Glue GetCatalog API, inspects the connection type, and registers the catalog at query time, so no upfront catalog configuration is required

As the diagram illustrates, a query references a catalog that has not been declared, which raises a catalog-not-found condition. The GlueCatalogResolver intercepts it and calls the AWS Glue Data Catalog through the GetCatalog API. Based on the connection type, the resolver registers the appropriate catalog: a redirecting catalog for a cross-account AWS Glue Data Catalog, a push-down catalog for Redshift Managed Storage, or a Spark catalog for Amazon S3 Tables. Registration happens at query time, with no upfront configuration.

Quick start

Follow these steps to enable multi-catalog support on an existing Amazon EMR 8.1.0 application:

1. Set spark_catalog to the RedirectingSessionCatalog:

spark.sql.catalog.spark_catalog = org.apache.spark.sql.connector.catalog.\
redirecting.RedirectingSessionCatalog

Note: If using Delta Lake or Hudi, also add open table format (OTF) session extensions. Hudi additionally requires KryoSerializer.

Note: To query a different AWS account, add a named catalog pointing to that account’s AWS Glue Data Catalog.

2. (Optional) Enable the Glue catalog resolver:

spark.sql.catalogResolver = com.amazonaws.glue.catalog.\
redirecting.GlueCatalogResolver

Step 1 is all you need for Iceberg-only multi-format queries. The notes call out additional configuration for Delta Lake, Hudi, or cross-account scenarios. Step 2 removes the need to pre-declare catalogs by resolving them at query time.

Try it yourself

The following walkthrough creates four tables (one per format), runs a cross-format join, and extends to a cross-account query. The accompanying sample scripts handle resource creation, job submission, and cleanup. The accompanying code is in the aws-emr-utilities repository.

Prerequisites

An AWS account with permissions for Amazon EMR Serverless, AWS Glue Data Catalog, and Amazon S3.
An Amazon EMR Serverless application running release emr-spark-8.1.0 (Spark). The multi-catalog features also work on Amazon EMR on EC2 and Amazon EMR on EKS.
An Amazon EMR Serverless job execution role scoped to the specific AWS Glue databases and S3 prefixes.
An S3 bucket for scripts and output (this post uses s3://amzn-s3-demo-bucket/multicatalog/).
(For cross-account) A producer account with Lake Formation grants, AWS Glue resource policy, Amazon S3 bucket policy, and AWS Key Management Service (AWS KMS) key policy configured.

Step-by-step implementation

Step 1: Clone the repository and configure

git clone https://github.com/aws-samples/aws-emr-utilities.git
cd aws-emr-utilities/examples/emr-multi-catalog
cp env.template .env
# Edit .env with your application ID, role ARN, bucket, and region

The repository contains two phases: single-account multi-format and cross-account. The .env file stores resource identifiers referenced by all scripts.

Step 2: Bootstrap the environment

./scripts/bootstrap.sh

This script creates the AWS Glue database, uploads PySpark scripts to S3, and verifies that your Amazon EMR Serverless application is in CREATED state. Note the application ID from the output if you have not set it in .env.

Step 3: Create tables across four formats

./scripts/run_demo.sh --phase setup

The setup phase submits a PySpark job that creates one table per format (Iceberg, Delta Lake, Hudi, Hive/Parquet) in a single AWS Glue database with a shared id/val schema. Each CREATE TABLE uses a different USING clause but all go through the same spark_catalog. RSC routes each to the correct engine.

The job configuration includes the RedirectingSessionCatalog, OTF session extensions for Delta and Hudi, and KryoSerializer for Hudi. If your workload is Iceberg-only, you can omit the extensions and serializer.

Step 4: Run the cross-format join

./scripts/run_demo.sh --phase query

This submits a query that references four tables by database.table only, with no format prefix. The query joins all four through a single catalog with no format-specific configuration:

SELECT i.id, i.val AS iceberg, d.val AS delta, h.val AS hudi, p.val AS hive
FROM salesdb.orders_iceberg i
JOIN salesdb.returns_delta d ON i.id = d.id
JOIN salesdb.shipments_hudi h ON i.id = h.id
JOIN salesdb.products_hive p ON i.id = p.id;

Expected output:

+---+---------+-------+-------+------+
| id| iceberg | delta | hudi | hive |
+---+---------+-------+-------+------+
| 1| ice-1 | dl-1 | hu-1 | hv-1 |
| 2| ice-2 | dl-2 | hu-2 | hv-2 |
| 3| ice-3 | dl-3 | hu-3 | hv-3 |
+---+---------+-------+-------+------+

Each column came from a different table format, joined in one query with no format-specific catalog configuration.

Step 5: (Optional) Extend to cross-account

Cross-account access requires grants from the account that owns the data. Run one bootstrap in each account:

# In the consumer account (where EMR runs):
./scripts/bootstrap_consumer.sh --region us-east-1 \
--producer-account 111122223333

# In the producer account (which owns the data):
./scripts/bootstrap_producer.sh \
--consumer-role <ROLE_ARN from previous step>

Then, back in the consumer account, run the cross-account phase:

./scripts/run_demo.sh --phase xacct-autowire \
--producer-account 111122223333 --producer-table fulfillment

The result joins the producer account’s table to a local Iceberg table in a single query, with no data copy.

For the full cross-account policy setup (Lake Formation grants, AWS Glue resource policy, S3 bucket policy, and AWS KMS key policy), see the Cross-account setup section.

Tip: Add –dry-run to either bootstrap script to preview every action it would take (buckets, roles, policies, applications) without creating anything.

Clean up

To avoid ongoing charges, run the clean-up script or follow the steps in the repository README:

./scripts/cleanup.sh

This removes S3 data, AWS Glue databases and tables, Lake Formation permissions, and the Amazon EMR Serverless application.

Cross-account setup

Cross-account access requires configuration across four services. The following example shows the AWS Glue resource policy. The accompanying bootstrap_producer.sh script configures all four, so you don’t need to author each policy by hand.

1. AWS Lake Formation: Grant permissions on the database and tables to the consumer principal.

2. AWS Glue resource policy: Allow cross-account access to catalog metadata.

3. Amazon S3 bucket policy: Allow access to the underlying data files.

4. AWS KMS key policy: Use a customer managed key (the aws/glue managed key doesn’t support cross-account grants).

Example: AWS Glue resource policy

{ "Version": "2012-10-17", "Statement": [{
"Effect": "Allow",
"Principal": {"AWS": "arn:aws:iam::444455556666:role/EMRExecutionRole"},
"Action": ["glue:GetCatalog","glue:GetDatabase","glue:GetDatabases",
"glue:GetTable","glue:GetTables","glue:GetPartition","glue:GetPartitions"],
"Resource": ["arn:aws:glue:us-east-1:111122223333:catalog",
"arn:aws:glue:us-east-1:111122223333:database/*",
"arn:aws:glue:us-east-1:111122223333:table/*/*"]
}]}

Choosing the right configuration

Your situation	What to set
Lake has Iceberg, Delta, and Hudi tables	`spark.sql.catalog.spark_catalog` = RedirectingSessionCatalog (Delta and Hudi require additional `spark.sql.extensions`. See Quick start)
Analytics in one account, data in another	`spark.sql.catalog.<name>` pointing at the remote Glue account (see Multi-catalog section)
Many accounts, want zero upfront config	`spark.sql.catalogResolver` = GlueCatalogResolver
All of the above	All three. They compose.

Note: Multi-catalog is query-time resolution. It doesn’t copy data between accounts, replicate tables, or grant access. Cross-account reads still require Lake Formation grants, AWS Glue resource policies, Amazon S3 bucket policies, and (if encrypted) AWS KMS key policies.

Conclusion

In this post, we configured the RedirectingSessionCatalog as the default Spark catalog on Amazon EMR 8.1.0. With a single configuration property, the RSC resolved Iceberg, Delta Lake, Hudi, and Hive tables through one catalog without format-specific prefixes. We then declared a named catalog to join tables across two AWS accounts, and enabled the GlueCatalogResolver to discover remote catalogs at query time without pre-declared spark.sql.catalog.* properties.

Multi-catalog support is available on Amazon EMR 8.1.0 across all deployment models: Amazon EMR Serverless, Amazon EMR on EC2, and Amazon EMR on EKS. To reproduce the walkthrough, clone the aws-samples/aws-emr-utilities repository and follow the steps in the Quick start section.

To learn more and get started, explore the following resources:

Amazon EMR
Amazon EMR documentation
aws-emr-utilities repository with the accompanying walkthrough code
Get a quick start with Apache Hudi, Apache Iceberg, and Delta Lake with Amazon EMR on EKS
Enforce fine-grained access control on open table formats via Amazon EMR integrated with AWS Lake Formation

About the authors

Materialize once, query anywhere: Introducing Iceberg materialized views in Amazon Redshift

Sudipta Bagchi — Mon, 05 Oct 2026 21:41:49 +0000

Amazon Redshift has progressively deepened its integration with Apache Iceberg. Earlier this year we launched Amazon Redshift RG, powered by AWS Graviton, with a purpose-built, integrated vectorized query engine designed from the ground up for data lakes. Instead of sending scans to a separate fleet, RG runs them natively on the cluster using vectorized Parquet scans, a smart-prefetch I/O subsystem, partition- and file-level pruning, improved bloom filters, and automatic Iceberg statistics collection through JIT Analyze for better query plans. Together, these deliver up to 2.4x faster Apache Iceberg queries than RA3, at 30 percent lower cost per vCPU and with no per-terabyte scan charges on data lake queries. On top of that performance foundation, you can write directly to Iceberg tables with full ACID alignment using INSERT, CTAS, UPDATE, DELETE, and MERGE. You can govern access with AWS Identity and Access Management (IAM) permissions through the external schema’s IAM role or with AWS Lake Formation for fine-grained, cross-engine control.

Amazon Redshift now also supports creating and refreshing Iceberg materialized views. A materialized view (MV) pre-computes expensive joins and aggregations once and stores the result as a standard Apache Iceberg table in Amazon Simple Storage Service (Amazon S3) or Amazon S3 Table Buckets, registered in the AWS Glue Data Catalog. You create one using familiar SQL (CREATE MATERIALIZED VIEW ... USING ICEBERG), and the result is instantly queryable by Iceberg-compatible engines, including Amazon Athena, Apache Spark on Amazon EMR, and AWS Glue. Amazon Redshift keeps it current with incremental refresh, and because the result is an ordinary Iceberg table in the AWS Glue Data Catalog, it is governed and discovered like any other catalog table.

Consider a team that runs its analytics on Amazon Redshift. Their transformations are already written in Amazon Redshift SQL, their staff know Amazon Redshift, and they’ve invested in its query engine. What they don’t have is a way to share their most expensive pre-computed results with the other engines in their organization, such as a data science group on Spark or an ad-hoc reporting team on Athena, without exporting copies or standing up a second transformation stack. The gap for this team is that they want interoperability and acceleration from the engine they already run.

Now they can create this materialized view in Amazon Redshift, in the SQL they already write, and Amazon Redshift stores the pre-computed result as an open Iceberg table. The Spark and Athena teams read that same result directly, without maintaining copies or separate pipelines. As new data lands, incremental refresh recomputes only what changed. The team gets a single, consistent source of truth for its most expensive queries that every engine shares. The Amazon Redshift team can run an end-to-end transformation pipeline in one engine, using materialized views as the building block between raw, cleaned, and serving layers without stitching multiple engines together stage by stage.

And you don’t need to choose between open and fast: for your most latency-sensitive dashboards, you can still load these Iceberg materialized views into Amazon Redshift Managed Storage (RMS) as native RMS materialized views.

When to use Iceberg MVs compared to Amazon Redshift (RMS) materialized views

Iceberg materialized views don’t replace the standard materialized views of Amazon Redshift. They serve a different need. Amazon Redshift materialized views store their results in Amazon Redshift Managed Storage (RMS), which is highly optimized for fast reads from Amazon Redshift. Iceberg materialized views store their results as open Iceberg tables in your Amazon S3, readable by your choice of engine. Choose based on where and how the result is consumed:

Use Amazon Redshift (RMS) materialized views when:

You query only from Amazon Redshift.
You need the lowest read latency. For interactive dashboards and sub-second lookups, reading from RMS is significantly faster than reading an Iceberg table from Amazon S3.
You want the most straightforward option for an Amazon Redshift-only workload.

Use Iceberg materialized views when:

You want the pre-computed result readable by engines beyond Amazon Redshift (Athena, Spark, Amazon SageMaker AI, third-party engines) without copying data.
You’re standardizing on Apache Iceberg for interoperability and don’t want acceleration tied to an Amazon Redshift-only storage format.
You want to run an end-to-end pipeline in a single engine and have every downstream consumer share the same open result.

They’re complementary. A common pattern is to build and transform data as Iceberg materialized views for openness and cross-engine access, then load the most performance-sensitive results into an RMS materialized view for your hottest interactive dashboards. This keeps your data open by default and fast where it counts.

In this post, you will:

Understand why Iceberg materialized views matter and their key use cases.
Learn how incremental refresh and cross-engine access work.
Set up prerequisites (IAM, Amazon S3, AWS Glue).
Create your first Iceberg materialized view.
Verify cross-engine access from Amazon Athena and PyIceberg.

This solution uses the following AWS services:

Amazon Redshift (Serverless or RG provisioned).
AWS Glue Data Catalog.
Amazon S3 (general purpose buckets or Amazon S3 Tables).
AWS Identity and Access Management (IAM).
AWS Lake Formation (optional, for governed access).

Solution overview

With Iceberg materialized views, you can compute aggregations once in Amazon Redshift and store the results as standard Apache Iceberg tables in Amazon S3 or Amazon S3 Table buckets. Iceberg-compatible engines can then query these pre-computed results directly.

Figure 1: Iceberg-compatible engines query the pre-computed materialized view directly from Amazon S3

Powered by Amazon Redshift Serverless and Amazon Redshift RG

Iceberg materialized views are supported on:

Amazon Redshift Serverless – Fully managed, auto scaling compute. Recommended for variable workloads where MV refreshes run alongside one-time queries without capacity planning.

Amazon Redshift RG (provisioned instances powered by AWS Graviton) – Provisioned clusters running on AWS Graviton processors with a custom-built integrated vectorized query engine. Up to 2.4x better performance for data lake workloads at 30% lower price per vCPU compared to RA3 instances.

Note: Amazon Redshift RA3 and DC2 instance types don’t support Iceberg materialized views.

Amazon Redshift does the heavy computation once on Serverless or Provisioned RG instances. Every Iceberg-compatible engine (Athena, Spark, SageMaker, and AWS Glue) consumes the pre-computed Iceberg MV from Amazon S3 or Amazon S3 Tables at standard Amazon S3 read cost. No additional compute charges on the consumer side.

Use cases

Iceberg materialized views support several patterns across analytics, cost optimization, and AI workloads.

1. Medallion architecture with shared optimization

The problem: In Bronze→Silver→Gold architectures, optimizations at silver/gold layers benefit only the engine that computed them.

With Iceberg MVs: Amazon Redshift RG computes silver and gold layers as Iceberg MVs with incremental refresh. Output is standard Iceberg on Amazon S3, so every consumer benefits without additional compute.

2. Empowering agentic AI, feature stores, and generative AI workloads

The problem: AI agents, machine learning (ML) pipelines, and generative AI applications need pre-computed features, such as rolling averages, customer lifetime value, and engagement scores, in a format frameworks can consume without direct warehouse connectivity.

With Iceberg MVs: The heavy computation (complex joins, window functions, statistical aggregations) runs once on Amazon Redshift Serverless or RG. Materialized views that use window functions or aggregations beyond COUNT and SUM are fully recomputed on each refresh rather than incrementally updated. Once the materialized view is computed and stored in Amazon S3 as a standard Iceberg table, it can be accessed by different consumers natively:

Amazon SageMaker notebooks and training jobs read features directly from Amazon S3 through PyIceberg, with no JDBC driver needed.
Amazon Bedrock agents access pre-computed analytics as structured data for Retrieval Augmented Generation (RAG).
Apache Spark on Amazon EMR consumes features through spark.table() for large-scale ML training pipelines.
Amazon Athena provides serverless SQL access to materialized features for ad-hoc analysis and dashboarding.

Incremental refresh keeps features fresh. For incremental refresh eligibility, see Materialized views stored as Apache Iceberg tables.

3. Cost optimization through compute consolidation

The problem: When the same aggregation is re-executed independently across multiple engines (Amazon Redshift, Athena, Spark, third-party tools), organizations pay for redundant compute on each engine, which multiplies cost linearly with the number of consumers.

With Iceberg MVs: One Amazon Redshift Serverless or RG refresh computes the aggregation once. Consumers read the pre-computed result directly from Amazon S3 at standard storage read cost, alleviating redundant compute across engines. The cost reduction can scale with the number of consuming engines you consolidate.

The problem: Sharing analytics across teams requires data copying or engine-specific sharing mechanisms.

With Iceberg MVs: Output is governed by AWS Lake Formation. Grant access with a single permission model. Consumers bring their preferred engine.

5. Single source of truth across analytics engines

The problem: Multiple teams recompute the same metrics independently across Spark, Amazon Redshift, Athena, and custom tools, producing inconsistent numbers.

With Iceberg MVs: One CREATE MATERIALIZED VIEW ... USING ICEBERG computes the metric once on Amazon Redshift Serverless or RG. Every engine reads the same Iceberg table from Amazon S3, with the same numbers, the same snapshot, and zero reconciliation.

How it works

Iceberg MVs extend the native materialized view capability of Amazon Redshift with the USING ICEBERG clause:

CREATE MATERIALIZED VIEW awsdatacatalog.analytics.daily_revenue
USING ICEBERG
LOCATION 's3://amzn-s3-demo-analytics/daily_revenue/'
PARTITIONED BY (day(order_date))
AS
SELECT order_date, region,
       SUM(amount) AS total_revenue, COUNT(*) AS transaction_count
FROM awsdatacatalog.source.transactions
GROUP BY 1, 2;

The MV can also be stored in Amazon S3 Table Buckets. If you omit the LOCATION clause, Amazon S3 Tables manages storage automatically.

Incremental refresh

Amazon Redshift tracks Iceberg snapshot IDs across refreshes. On REFRESH MATERIALIZED VIEW, it identifies changed source partitions and recomputes only the delta.

Patterns supporting incremental refresh:

SUM and COUNT aggregates with GROUP BY.
Non-aggregated queries (row-level delta tracking).
Inner JOINs between Iceberg tables.

Constructs that use full refresh (still supported):

DISTINCT, outer JOINs, window functions, subqueries.
Set operations (UNION ALL, UNION, INTERSECT, EXCEPT).
MIN, MAX, AVG, COUNT(DISTINCT), SUM(DISTINCT).
GROUPING SETS, ROLLUP, CUBE.

Cross-cluster refresh

The MV isn’t tied to the creating cluster. Amazon Redshift clusters or Serverless workgroups with the appropriate IAM role can refresh it. When multiple clusters attempt to refresh the same MV concurrently, Amazon Redshift coordinates through the AWS Glue Data Catalog to make sure that only one refresh succeeds at a time, helping prevent conflicts automatically. For more details on concurrency handling, see the Amazon Redshift Iceberg materialized views documentation.

Cross-engine access

The result is a standard Iceberg table that needs no special drivers. This materialized view can be read from different engines, as shown in the following examples:

Amazon Athena:

SELECT * FROM analytics.daily_revenue WHERE region = 'us-east';

Apache Spark on Amazon EMR:

spark.table("analytics.daily_revenue").filter(col("region") == "us-east")

Amazon SageMaker / PyIceberg:

from pyiceberg.catalog import load_catalog
catalog = load_catalog("glue", **{"type": "glue"})
df = catalog.load_table("analytics.daily_revenue").scan().to_pandas()

Prerequisites

Setting up Iceberg MVs requires IAM, Amazon S3, and AWS Glue configuration. Follow these steps to prepare your environment.

For a complete walkthrough with console screenshots, see Getting started with Iceberg materialized views in the Amazon Redshift documentation.

The following table summarizes the resources you will configure:

Resource	Purpose	Created in Step
IAM Role (IcebergMvDefiner)	Definer role for MV operations (2-service trust policy)	Steps 1–2
S3 Bucket	Stores Iceberg MV data (Parquet files)	Step 3
AWS Glue database	Catalogs MV metadata in AWS Glue Data Catalog	Step 4
Cluster Role Association	Grants the Amazon Redshift cluster permission to assume the definer role	Step 5

Step 1: Create the IAM role

Create an IAM role named IcebergMvDefiner with the following trust policy. Note that two service principals are required:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": [
                    "redshift.amazonaws.com",
                    "glue.amazonaws.com"
                ]
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Why are these two principals? Amazon Redshift needs to assume the role to perform materialized view operations. AWS Glue needs to check base table permissions on behalf of the materialized view definer role.

Step 2: Attach IAM policies

Attach the following scoped inline policies to the IcebergMvDefiner role. These provide the minimum permissions required for Iceberg materialized view operations.

S3 access (scoped to your bucket):

Create an inline policy named s3-mv-access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:PutObject",
                "s3:DeleteObject",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<<your-bucket>>",
                "arn:aws:s3:::<<your-bucket>>/*"
            ]
        }
    ]
}

AWS Glue Data Catalog access policy (scoped to your database):

Create an inline policy named glue-mv-access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetTables",
                "glue:CreateTable",
                "glue:UpdateTable",
                "glue:DeleteTable",
                "glue:GetPartitions",
                "glue:BatchGetPartition"
            ],
            "Resource": [
                "arn:aws:glue:<<your-region>>:<<your-account-id>>:catalog",
                "arn:aws:glue:<<your-region>>:<<your-account-id>>:database/<<your-glue-db>>",
                "arn:aws:glue:<<your-region>>:<<your-account-id>>:table/<<your-glue-db>>/*"
            ]
        }
    ]
}

IAM PassRole policy (scoped to the definer role):

Create an inline policy named mv-access:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "arn:aws:iam::<<your-account-id>>:role/IcebergMvDefiner"
        }
    ]
}

Step 3: Create S3 bucket

Create an S3 bucket for MV storage. We recommend the naming convention iceberg-mv-. Enable default encryption (SSE-S3) and block all public access.

Step 4: Create AWS Glue database

Create a database named iceberg_mv in the AWS Glue Data Catalog. Use a plain create-database command. The database inherits IAM_ALLOWED_PRINCIPALS by default, which allows cross-engine access from Amazon Athena and other engines.

Step 5: Associate role with Redshift

Associate the IcebergMvDefiner role with your Amazon Redshift cluster or Serverless namespace:

aws redshift modify-cluster-iam-roles --cluster-identifier --add-iam-roles arn:aws:iam:::role/IcebergMvDefiner

Step 6: Set case sensitivity

Connect to your Amazon Redshift cluster and run:

SET enable_case_sensitive_identifier TO FALSE;

Creating Your First Iceberg MV

With prerequisites in place, you can now create an external schema, a base Iceberg table, and your first materialized view.

Step 7: Create external schema

CREATE EXTERNAL SCHEMA iceberg_schema FROM DATA CATALOG DATABASE 'iceberg_mv' REGION '<<your-region>>' IAM_ROLE 'arn:aws:iam:::role/IcebergMvDefiner';

Step 8: Create Iceberg base table with sample data

CREATE TABLE iceberg_schema.orders USING ICEBERG LOCATION 's3://<<your-bucket>>/iceberg_mv_blog/orders' AS SELECT 1 AS id, 'us' AS region, 100 AS amount UNION ALL SELECT 2, 'eu', 200 UNION ALL SELECT 3, 'jp', 150;

Step 9: Create the Iceberg materialized view

CREATE MATERIALIZED VIEW iceberg_schema.sales_by_region USING ICEBERG LOCATION 's3://<<your-bucket>>/iceberg_mv_blog/sales_by_region' AS SELECT region, SUM(amount) AS total, COUNT(*) AS num_orders FROM iceberg_schema.orders GROUP BY region;

Step 10: Verify MV contents

SELECT * FROM iceberg_schema.sales_by_region;

Figure 2: Initial materialized view query results aggregated by region

Step 11: Test incremental refresh

Insert new rows into the base table and refresh the MV:

INSERT INTO iceberg_schema.orders VALUES (4, 'us', 300), (5, 'eu', 50);
REFRESH MATERIALIZED VIEW iceberg_schema.sales_by_region;

SELECT * FROM iceberg_schema.sales_by_region;

Figure 3: Materialized view query results after inserting new rows and refreshing

Cross-engine verification

The materialized view is now a standard Iceberg table in the AWS Glue Data Catalog, accessible from compatible engines without an Amazon Redshift connection.

Amazon Athena:

SELECT * FROM iceberg_mv.sales_by_region WHERE region = 'us';

Figure 4: Querying the materialized view from Amazon Athena

Amazon SageMaker / PyIceberg:

from pyiceberg.catalog import load_catalog
catalog = load_catalog("glue", **{"type": "glue"})
df = catalog.load_table("iceberg_mv.sales_by_region").scan().to_pandas()
print(df)

Apache Spark on Amazon EMR:

spark.table("iceberg_mv.sales_by_region").filter(col("region") == "us").show()

The business case

The following table illustrates a representative scenario where a common aggregation is computed across multiple engines:

Dimension	Traditional (siloed)	Iceberg MVs on Serverless/RG
Compute cost	~$7,500/month (4 engines)	~$1,500/month (1 refresh)
Metric consistency	3–4 versions	1 version
Time to new metric	Days (per engine)	Hours (one definition)
Governance	Per-engine ACLs	IAM + optional Lake Formation

Cost estimate assumes a mid-size aggregation (1 TB input, 100 GB output) running daily across Athena ($5/TB scan), Spark on Amazon EMR ($0.096/hr × 4 nodes), Amazon Redshift Serverless (8 RPU), and a third-party engine. Actual savings vary by workload.

Current limitations

For the current list of supported SQL constructs, incremental refresh eligibility, and known limitations, see Materialized views stored as Apache Iceberg tables in the Amazon Redshift documentation.

(Optional) Add Lake Formation governance

If your organization requires centralized access control across engines, you can layer AWS Lake Formation governance on top of the IAM-only setup. Note that Lake Formation permissions for Iceberg MVs are coarse-grained (database and table level). Fine-grained access control (row filters, column filters) isn’t supported on Iceberg materialized views. The following additional steps were validated in the same environment used in this walkthrough:

Add lakeformation.amazonaws.com to the IAM role trust policy (in addition to redshift.amazonaws.com and glue.amazonaws.com).
Add lakeformation:GetDataAccess to the role’s inline policy.

aws lakeformation register-resource --resource-arn arn:aws:s3:::<your-bucket> --role-arn arn:aws:iam::<account-id>:role/IcebergMvDefiner --region <region>

Recreate the AWS Glue database with empty CreateTableDefaultPermissions (this makes Lake Formation authoritative for table-level access):

aws glue delete-database --name iceberg_mv --region <region>
aws glue create-database --region <region> --database-input '{"Name":"iceberg_mv","CreateTableDefaultPermissions":[]}'

Grant Lake Formation permissions to the definer role: DATA_LOCATION_ACCESS on the S3 bucket, CREATE_TABLE/DESCRIBE/ALTER/DROP on the database, and ALL on tables (with grant option).

For a complete Lake Formation walkthrough, see How to use streamlined permissions for Amazon S3 Tables and Iceberg materialized views.

Clean up

To avoid incurring ongoing charges, remove the resources created in this walkthrough:

DROP MATERIALIZED VIEW iceberg_schema.sales_by_region;
DROP TABLE iceberg_schema.orders;
DROP SCHEMA iceberg_schema;

Note: DROP MATERIALIZED VIEW removes the AWS Glue catalog entry but does not delete the underlying data in Amazon S3. To remove the data, delete the Amazon S3 prefix manually:

aws s3 rm s3://<<your-bucket>>/iceberg_mv_blog/ --recursive

Conclusion

Iceberg materialized views take the open lakehouse promise further: optimization itself becomes portable. Amazon Redshift, whether running as Serverless or on RG instances powered by AWS Graviton, does the heavy computation once. Every other engine and ML pipeline benefits without additional compute. Start with one MV. Watch the numbers match across engines for the first time. Then scale from there.

Resources

Getting started with Iceberg materialized views (Amazon Redshift documentation)

About the authors

Run an automated operational review with the Amazon Redshift MCP server

Sidhanth Muralidhar — Mon, 05 Oct 2026 15:44:45 +0000

It’s the end of a strong quarter, and your Amazon Redshift workloads have grown with the business. Data volumes are up, new pipelines have shipped, and more teams are querying than when you first sized the cluster. Nothing is broken, but this is exactly when a periodic operational review pays off. It confirms the cluster is still tuned for how it’s used today, and surfaces ways to optimize cost and performance as you scale.

Amazon Redshift already automates significant operational complexity. Autonomics features such as automatic table optimization, automatic workload management, and automatic vacuum handle much of the complex work, so you can focus on writing queries rather than managing infrastructure. Amazon Redshift Serverless goes further, using AI-driven scaling that adapts compute to workload demand.

Even so, some decisions still benefit from human judgment. For example, table design choices may not have accounted for common join patterns, or query patterns may have shifted since the tables were built. Both are worth revisiting. Likewise, as workloads increase, it’s worth deciding whether the current deployment model is still correctly sized.

Many organizations have turned this into a recurring business process: monitoring dashboards and key performance indicators (KPIs), narrowing down long-running queries, collaborating across teams to resolve them, and coaching users on efficient query patterns. This isn’t unique to Amazon Redshift. It’s a best practice for any production data system.

AWS Enterprise Support helps customers with these reviews. But even with specialist assistance, the process typically consumes 4–8 hours of focused effort per cluster: assembling diagnostic queries, interpreting results, cross-referencing documentation, and compiling findings into a prioritized report. Most teams recognize the value, but consistently deprioritize it in favor of feature delivery and day-to-day operations.

In a previous post, we demonstrated how to query Amazon Redshift using natural language with Kiro and the Amazon Redshift Model Context Protocol (MCP) server. That approach replaced manual schema navigation and hand-written SQL with conversational analytics. This post takes the next step: from asking questions about your data to asking questions about your cluster’s operational health.

The review_cluster tool in the Amazon Redshift MCP server makes the entire diagnostic process available from a single natural-language request. It evaluates 12 diagnostic areas, identifies potential issues, and returns prioritized recommendations linked directly to AWS documentation, covering both provisioned clusters and serverless workgroups in one invocation. What previously required hours of specialist effort now completes in a few minutes. This makes it practical to review your cluster weekly, after significant schema changes, or before peak traffic events.

In this post, you learn how to:

Run an automated operational review of your Amazon Redshift cluster using natural language.
Interpret the structured findings and recommendations.
Act on specific findings conversationally, turning diagnostics into remediation without leaving Kiro, Claude, or any MCP-compatible client.
Incorporate periodic reviews into your operational workflow.

What is the review_cluster tool?

A single natural-language request triggers a comprehensive diagnostic assessment. It complements the server’s discovery and query tools: those give you conversational access to your data, while review_cluster gives you the same for your cluster’s health. The tool executes a curated set of queries against Amazon Redshift system views, evaluates the results against known best-practice thresholds, and returns structured findings with prioritized recommendations. Every recommendation includes direct links to the relevant AWS documentation, so you can move from identification to remediation without searching. The entire process is read-only. No data is modified, no configuration is changed, and no resources are created. The tool observes and reports. Remediation decisions remain with you.

The tool returns a structured result containing:

Signals evaluated: The total number of diagnostic checks executed.
Findings: A list of triggered conditions, each with a name, the number of affected objects (tables, queries, nodes), and linked recommendation IDs.
Recommendations: A deduplicated list of corrective actions, ordered by effort, with documentation links.

What it inspects

The review currently evaluates your cluster across 12 diagnostic areas:

Automatic Table Optimization (ATO): Whether the automated tuning actions of Amazon Redshift (encoding, sort keys, distribution styles) are completing successfully or not.
Table design recommendations: Amazon Redshift Advisor recommendations for encoding, sort keys, and distribution that have not yet been applied.
COPY and data ingestion performance: File sizing, parallelism relative to slice count, and ingestion throughput patterns.
External query (Spectrum) performance: Partition pruning effectiveness, file sizes, and scan efficiency for queries against external tables.
Materialized view health: Staleness, auto-refresh status, and maintenance overhead of materialized views.
Node and storage utilization: Disk usage, node type currency, and whether the cluster would benefit from migration to the latest instance generation.
Table-level statistics: Vacuum status, stale statistics, sort key effectiveness, distribution skew, and compression encoding coverage.
Query performance: The longest-running queries, nested loop joins, and disk spill patterns.
Workload usage patterns: How intensively the cluster is used throughout the day and whether the workload suits the current deployment model.
Workload Management (WLM) configuration: Queue setup, concurrency scaling, short query acceleration, priority settings, and query monitoring rules.
Workload evaluation: Whether the cluster’s utilization pattern suggests it could benefit from a different deployment model such as serverless.
Serverless scaling: Whether a serverless workgroup’s observed compute range falls within the AI-driven scaling window.

Not every area applies to every deployment, because some checks target configuration that only exists in one model. Four checks are provisioned-only: COPY parallelism relative to slice count, node and storage utilization, WLM configuration, and workload evaluation. These tune constructs that Amazon Redshift Serverless manages for you. Serverless has no nodes or slices to size, and it always uses automatic WLM rather than user-configured queues, so there is nothing for those checks to act on. The serverless-scaling check is the reverse: it evaluates AI-driven Redshift Processing Unit (RPU) scaling, which is specific to serverless and has no equivalent on a provisioned cluster. As a result, a provisioned cluster evaluates 11 of the areas and a serverless workgroup evaluates 8, and the tool reports the number actually run as signals evaluated.

Running your first review

The following steps cover what you need in place and how to start a review.

Prerequisites

Before running a review, make sure that you have:

An MCP-compatible client configured with the Amazon Redshift MCP server (for example, Kiro). See the Amazon Redshift MCP server README for installation and configuration steps.
Valid AWS credentials with the AWS Identity and Access Management (IAM) permissions the server requires.
The sys:monitor role granted to the connecting database user, the one additional database permission review_cluster needs.

To grant this access, a database administrator with superuser privileges runs:

GRANT ROLE sys:monitor TO "IAMR:YourRoleName";

The database user name for IAM identities follows the format IAMR:RoleName or IAM:UserName. Confirm yours with SELECT current_user in the Amazon Redshift Query Editor. The double quotes in the GRANT statement are required for IAM identity names.

With prerequisites in place, the review is a single prompt:

Run a review of analytics-cluster

The agent identifies the target cluster, connects to the database, and executes the diagnostic assessment.

The following example shows the output from a review of a serverless workgroup. Because four provisioned-only areas don’t apply, the tool evaluated 8 of the 12 diagnostic areas, identified 8 findings, and mapped them to 6 distinct recommendations:

Cluster Review: analytics-cluster (Serverless, eu-west-2 / London)
Ran the diagnostic --- result: 8 signals evaluated, 8 findings, 6 recommendations.
Findings
- Missing statistics --- 16 queries hit tables that have never been analyzed
- Dist/Broadcast alerts --- 10 queries reshuffle data across nodes
- Nested loop joins / cross-joins --- 5 long-running queries
- Small tables without ALL distribution --- 4 tables
- Small tables with a manual sort key --- 4 tables
- Large table with data skew --- 1 table (slice imbalance ≥4x)
- Large table distributed by date/datetime --- 1 table
- Variable serverless workload in the AI-scaling range --- 1 workgroup
Recommendations
1. REC_003 --- Analyze tables (fix missing stats). Effort: Low. Run or schedule ANALYZE on the 16 affected tables. Lowest-effort, highest-impact item; also improves the poor distribution/plan decisions the optimizer is making. Docs: https://docs.aws.amazon.com/redshift/latest/dg/t_Analyzing_tables.html
2. REC_007 --- Right-size sort keys. Effort: Low. The 4 small tables carry manual sort keys that add overhead for little benefit --- switch them to ALTER TABLE [table] ALTER SORTKEY AUTO. Docs: https://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-sort-key.html
3. REC_008 --- Choose the right distribution style. Effort: Low--Medium. Small tables → DISTSTYLE AUTO/ALL; the skewed large table → pick a better DISTKEY (skew often from NULLs in the key); the date-distributed large table → reconsider the key or move to DISTSTYLE AUTO. This also resolves the 10 Dist/Broadcast query alerts. Docs: https://docs.aws.amazon.com/redshift/latest/dg/c_best-practices-best-dist-key.html
4. REC_030 --- Tune serverless AI-driven scaling. Effort: Low. Review your price-performance target (Optimizes for cost / Balanced / Optimizes for performance) and pair it with Max capacity + Max RPU-hours caps to keep cost predictable. Adapts over 1--3 days. Docs: https://docs.aws.amazon.com/redshift/latest/mgmt/serverless-capacity.html
5. REC_009 --- Remove nested loop joins (cross-joins). Effort: Medium. Add join conditions to the 5 cross-join queries. Docs: https://docs.aws.amazon.com/redshift/latest/dg/query-performance-improvement-opportunities.html#nested-loop
6. REC_019 --- Add QMR rules. Effort: Medium. Add query monitoring rules to catch nested loops and long runtimes going forward (log/hop/abort actions). Docs: https://docs.aws.amazon.com/redshift/latest/dg/cm-c-wlm-query-monitoring-rules.html
Suggested starting point
Start with REC_003 (ANALYZE) --- cheapest, highest-impact, and likely a root cause behind several slow-query findings. Then apply the AUTO table-optimization changes in REC_007 and REC_008, which are mostly one-line ALTER TABLE statements.
Want me to pull the specific table names behind each finding and draft the exact ANALYZE / ALTER TABLE remediation statements?

Each finding represents an independent diagnostic condition that was triggered. The affected_row_count indicates how many objects match that specific condition. These counts describe different dimensions of the cluster and are not additive across findings.

When the tool returns zero findings, the cluster is operating within best-practice thresholds across the evaluated areas.

From findings to fixes

The review output isn’t a static report. It’s a starting point for an interactive conversation. Each finding identifies a specific condition, and each recommendation provides a clear remediation path with documentation links. From here, you can continue working within the same Kiro session to plan and prepare your next steps.

Recommendations fall into four broad categories:

Quick configuration changes: Actions such as enabling concurrency scaling or short query acceleration require a single parameter change in the Amazon Redshift console or a brief API call. These are low-risk, high-impact adjustments that can often be applied immediately.
Batch table operations: Findings related to table design (distribution style, sort keys, compression encoding) typically affect multiple tables. You can ask Kiro to list the specific tables involved and generate the corresponding ALTER TABLE statements. Review the generated SQL, then apply it through the Amazon Redshift Query Editor or your preferred SQL client.
Data ingestion optimization: Findings related to COPY performance identify inefficiencies in how data is loaded. For example, source files might be too small, or file counts might not align with the cluster’s slice count. Addressing these requires changes upstream in your extract, transform, and load (ETL) pipeline or Amazon Simple Storage Service (Amazon S3) staging process rather than within Amazon Redshift itself.
Architectural decisions: Recommendations such as migrating to Amazon Redshift Serverless or resizing to RG instances require broader evaluation. These are not single-command fixes. They involve capacity planning, workload testing, and potentially migration steps. The recommendation text and linked documentation provide the context needed to begin that planning.

After presenting findings, Kiro proposes the next best step to act upon, typically starting with the lowest-effort, highest-impact recommendation. Alternatively, you can direct the conversation yourself. For example:

“Which tables are affected by the distribution style finding?”
“What would the ALTER TABLE statements look like for those tables?”
“Explain what concurrency scaling does and how to enable it.”
“What are the trade-offs of migrating this workload to serverless?”

Kiro retrieves the relevant details, generates SQL where applicable, and references the documentation. The tool diagnoses and recommends, but does not modify your cluster. The decision to apply changes remains with you.

Best practices

Tips for getting the most out of review_cluster.

Embed reviews in your DataOps practice. Regular table and cluster maintenance is a foundational best practice for any production data warehouse, and its value comes from consistency. Operational reviews are one component of a broader DataOps discipline: the practice of maintaining data systems that are clean, reliable, governed, and always available. Define KPIs for your cluster, such as query latency percentiles, disk spill frequency, and WLM queue wait times. Then use periodic review_cluster runs to track your query and workload optimization progress against them. Over time, this creates a feedback loop: findings inform remediation, KPIs measure impact, and the next review validates improvement.
Run reviews on a regular cadence. Treat operational reviews like testing: the more routinely you run them, the sooner you catch drift before it affects users. Consider running a review weekly, after significant schema changes, after major data loads, or before anticipated peak traffic events.
Follow the documentation links. Every recommendation includes direct links to the relevant AWS documentation. These pages provide detailed guidance, edge cases, and configuration examples that go beyond what the recommendation text can cover. Use them as your primary reference when planning remediation.
Start with low-effort wins. The recommendations are ordered by effort. Begin with quick configuration changes (concurrency scaling, short query acceleration, query monitoring rules) before moving to structural changes that require broader planning. Early wins build confidence and often improve cluster performance enough to create headroom for larger changes.
Use the review as a baseline. Run a review before and after significant changes to measure their effect. For example, after applying table design recommendations, a follow-up review should show fewer table-related findings. This before-and-after pattern helps validate that your changes had the intended impact.

Conclusion

In this post, you learned how to run an automated operational review of your Amazon Redshift cluster using natural language with Kiro. The review_cluster tool evaluates 12 diagnostic areas, from table design and workload management to node utilization and data ingestion performance. It returns prioritized, actionable recommendations linked directly to AWS documentation.

What previously required a specialist to assemble diagnostic scripts, interpret system view outputs, and compile findings over several hours now completes in a single request. This shift makes it practical to incorporate operational reviews into your regular workflow rather than treating them as an infrequent, resource-intensive exercise.

To get started:

Make sure the Amazon Redshift MCP server is configured in Kiro (see the setup post).
Grant the sys:monitor role to your database user.
Ask Kiro to review your cluster.

To go further:

Visit the Amazon Redshift MCP server documentation for the full tool reference.
Read the MCP protocol to understand how AI agents integrate with external tools.
Explore Kiro for additional capabilities including steering files, hooks, and agent automation.

About the authors

Optimize consumer rebalancing on Amazon MSK with next generation protocol

Yashika Jain — Thu, 01 Oct 2026 16:43:54 +0000

If you run large consumer groups on Apache Kafka and Amazon Managed Streaming for Apache Kafka (Amazon MSK), you’ve likely experienced the pain of slow rebalances: processing stalls across all consumers, “rebalance storms” triggered by routine scaling events, and prolonged recovery times that impact downstream applications. With the classic rebalance protocol, even a single consumer joining or leaving the group forces a global synchronization barrier, pausing every consumer regardless of whether its partition assignments changed.

The KIP-848 consumer protocol, introduced in Apache Kafka 4.0, fundamentally redesigns how consumer group rebalancing works. Also referred to as “the Next Generation Consumer Rebalance Protocol”, KIP-848 shifts coordination logic from the client to the broker-side group coordinator. This supports fully incremental, server-driven rebalancing that significantly improves performance for large consumer groups. You can use the consumer protocol on Amazon MSK on all 4.x Apache Kafka versions on both MSK Standard and Express brokers.

In this post, we explain how the consumer protocol works, how to enable it on Amazon MSK, and how to diagnose and resolve slow rebalancing issues to help improve performance.

The classic protocol compared to the consumer protocol

The classic protocol relied on client-side rebalance logic with a global synchronization barrier. Every rebalance caused all consumers in the group to pause processing simultaneously regardless of whether their partition assignments were changing. This led to “rebalance storms” in large consumer groups where cascading rebalances could take minutes to resolve. The CooperativeStickyAssignor is a client-side partition assignment strategy that supports incremental, cooperative rebalancing. It significantly improves rebalance performance and minimizes disruption to groups during rebalance events. However, it still suffers from bottlenecks as consumer group size and partition count increase. For large workloads, client-side rebalancing behavior can result in longer rebalancing times and require significant client tuning and monitoring during rebalances.

The consumer protocol addresses these limitations by moving all rebalancing logic to the server. The broker handles coordination using a continuous heartbeat mechanism and server-driven reconciliation process. Only affected partitions move during a rebalance, and consumers with unchanged assignments continue processing uninterrupted. This results in faster recovery compared to the classic protocol, and improved scalability as workloads grow.

The following table compares the classic and next generation protocols across key dimensions.

Aspect	Classic Protocol	Next Generation Protocol (KIP-848)
Rebalance logic	Client-side	Fully server-driven
Consumer impact	Depends on assignor, all consumers pause, or rebalance is limited by group size	Only affected consumers impacted, scales effectively as groups grow
Mechanism	Client-side algorithm and cross-group coordination	Incremental, async reconciliation
Commit processing	Paused during rebalance	Able to progress during rebalance
Scalability	Complex, fragile at scale	Resilient, broker-driven
Rebalance storms	Common in large groups	Eliminated

Server-side configuration

With the consumer protocol, key parameters are now configured on the server rather than the client:

group.consumer.heartbeat.interval.ms – Controls the consumer heartbeat interval (server-side).
group.consumer.session.timeout.ms – Controls the session timeout (server-side).
group.consumer.assignors – Specifies available assignors (uniform and range by default).

In Amazon MSK Express brokers, these configurations are read-only and cannot be modified. In Amazon MSK Standard brokers, these configurations are managed with broker configurations. To update these configurations in Amazon MSK Standard brokers, refer to Update the configuration of an Amazon MSK cluster.

When to use the consumer protocol

The consumer protocol provides the most benefit to workloads with the following requirements:

Large consumer groups: Groups with many consumers and partitions see the most significant improvements because of the elimination of global synchronization barriers.
High-availability applications: Applications that cannot afford processing interruptions benefit from continuous message processing during rebalances. Financial services, real-time analytics, and fraud detection systems are ideal candidates.
Frequently rebalancing environments: Automatic scaling deployments, Kubernetes with frequent pod restarts, or continuous integration and continuous delivery (CI/CD) environments experience significantly less disruption.
Dynamic partition scaling: Workloads that regularly add partitions or topics benefit from the incremental, server-driven approach.

Prerequisites

Before you begin, make sure that you have the following:

An Amazon MSK cluster running Apache Kafka version 4.0 or later (both MSK Standard and Express brokers are supported).
A Kafka client library that supports the KIP-848 consumer protocol (see Step 4 for supported versions).
Basic familiarity with Apache Kafka consumer groups and partition assignment.
An AWS account with appropriate permissions to manage your MSK cluster.

Enabling the consumer protocol on Amazon MSK

The following steps walk you through verifying your cluster version, configuring your consumer client, removing deprecated configurations, and confirming client library support.

Step 1: Verify cluster version

The consumer protocol requires Apache Kafka 4.0 or later. To use the consumer protocol on Amazon MSK, verify that your cluster is running Apache Kafka version 4.0.x or later. You can verify your cluster’s Apache Kafka version using the AWS Management Console, AWS Command Line Interface (AWS CLI), or AWS SDKs:

aws kafka describe-cluster-v2 --cluster-arn <your-cluster-arn> \
    --query "ClusterInfo.Provisioned.CurrentBrokerSoftwareInfo.KafkaVersion"

If your cluster is running Apache Kafka 4.0.x or later, the consumer protocol is automatically enabled on the server and ready to use. No additional server-side feature flag verification is needed.

Step 2: Configure consumer client

Set group.protocol=consumer in your consumer configuration. The protocol is not enabled by default:

# confluent-kafka-python example
config = {
    'bootstrap.servers': bootstrap_servers,
    'group.id': group_id,
    'group.protocol': 'consumer',  # Required — defaults to 'classic' if omitted
    'auto.offset.reset': 'earliest'
}

The consumer protocol can be changed in-place for existing consumer groups. When you update the group.protocol, perform a rolling restart of your consumers. The broker-side group coordinator automatically handles the upgrade to the consumer protocol and handles classic protocol requests from old clients alongside the upgraded clients.

Step 3: Remove deprecated client configurations

When the consumer protocol is enabled, the following client-side configurations are no longer supported because they are controlled by the brokers:

heartbeat.interval.ms.
session.timeout.ms.
partition.assignment.strategy.

Step 4: Verify client library support

Verify that your Kafka client version supports the consumer protocol:

Java clients: Generally available (GA) in Apache Kafka 4.0+.
confluent-kafka-python: Version 2.12.0+ (GA support for KIP-848). See the release notes.
librdkafka-based clients (Go, .NET, C/C++): Based on librdkafka 2.12.0+.

Note: For other Kafka client libraries, verify your client library’s documentation for group.protocol=consumer support before enabling the next generation protocol. If your client doesn’t support KIP-848, it will continue to use the classic protocol.

Diagnosing slow consumer group rebalancing with the consumer protocol

Even after enabling the consumer protocol, you may encounter situations where consumer group rebalancing takes longer than expected. The following sections help you diagnose and resolve these issues.

Common symptoms

Consumer group rebalancing takes longer than expected despite setting group.protocol=consumer.
Consumers pause processing during rebalances.
Broker logs show “member session expired” or “fenced” messages.
Frequent rebalances triggered during rolling deployments or pod restarts.

Step 1: Confirm the consumer protocol is active using broker logs

Before troubleshooting performance, verify which protocol your consumers are actually using. Check broker logs in Amazon CloudWatch Logs Insights. The log patterns differ significantly between protocols.

Consumer protocol expected logs:

Key indicators: “consumer protocol”, “epoch” terminology, “target assignment” with server-side assignor, “fenced” for member removal.

[GroupCoordinator id=X] [GroupId <group-id>] Member <member-id> joins the consumer group using the consumer protocol.
[GroupCoordinator id=X] [GroupId <group-id>] Bumped group epoch to 309 with metadata hash 4064309670987706693.
[GroupCoordinator id=X] [GroupId <group-id>] Computed a new target assignment for epoch 309 with 'uniform' assignor in 0ms.
[GroupCoordinator id=X] [GroupId <group-id>] Member <member-id> fenced from the group because the member session expired.

Classic protocol expected logs:

Key indicators: “PreparingRebalance” state, “old generation” terminology, “Assignment received from leader”.

If you see classic protocol logs, the consumer protocol is not active. Proceed to Step 2 to troubleshoot why.

[GroupCoordinator id=X] Preparing to rebalance group <group-id> in state PreparingRebalance with old generation X
[GroupCoordinator id=X] Stabilized group <group-id> with X members
[GroupCoordinator id=X] Assignment received from leader for group <group-id>

Step 2: Troubleshoot why the consumer protocol is not active

Verify that your client configuration, client library versions, and cluster versions support the consumer protocol, as described in the preceding Step 1 through Step 4.

Step 3: Resolve slow rebalancing when KIP-848 is active

After you verify the consumer protocol is active but rebalancing is still slow, investigate the following causes:

A. Consumer session timeout causing premature member removal

With the consumer protocol, session timeout is server-controlled through group.consumer.session.timeout.ms (default: 45 seconds). The diagnostic path depends on whether you are using static group membership. The following table outlines the diagnostic path and recommended actions for each scenario.

Scenario Symptom Root cause Recommended action

With static group membership (group.instance.id configured) Slow rebalancing when a static member terminates without calling consumer.close() The coordinator waits for the full session timeout before reassigning partitions. This is the most common cause of slow rebalancing in containerized environments. MSK Standard: Implement graceful shutdown to trigger an immediate leave-group request, or increase the session timeout: group.consumer.session.timeout.ms=60000 (default is 45000). MSK Express: This configuration is not editable in Amazon MSK Express clusters. For Amazon MSK Express, optimize your client’s cold starts to allow members to restart within the 45 second consumer session timeout.

Without static group membership

Session timeouts expiring during normal operations

Your consumer is freezing or becoming unresponsive, which prevents heartbeats from reaching the coordinator.

Investigate long-running message processing, garbage collection pauses, network connectivity issues, or resource exhaustion on the consumer host. Look for this in broker logs:

[GroupCoordinator id=X] [GroupId <group-id>] Member <member-id> has timed out

B. Missing graceful shutdown handling

When consumers terminate without calling consumer.close(), the coordinator waits for the full session timeout before removing the member. This is the most common cause of slow rebalancing in containerized environments.

Resolution: Implement proper SIGTERM handling to trigger an immediate leave-group:

import signal
import sys
from confluent_kafka import Consumer

class GracefulKafkaConsumer:
    def __init__(self, config):
        self.running = True
        self.consumer = Consumer(config)
        signal.signal(signal.SIGTERM, self.shutdown_handler)
        signal.signal(signal.SIGINT, self.shutdown_handler)

    def shutdown_handler(self, signum, frame):
        print(f"Received signal {signum}, initiating graceful shutdown...")
        self.running = False

    def consume_loop(self):
        self.consumer.subscribe(['your-topic'])
        while self.running:
            msg = self.consumer.poll(timeout=1.0)
            if msg is None:
                continue
            # Process message

        print("Closing consumer gracefully...")
        self.consumer.close()  # Sends LeaveGroup — triggers immediate rebalance
        sys.exit(0)

For Kubernetes, verify that terminationGracePeriodSeconds allows time for consumer.close() to complete:

spec:
  terminationGracePeriodSeconds: 60
  containers:
    - name: kafka-consumer

C. Frequent rebalances from unstable consumers

If consumers repeatedly join and leave (out-of-memory (OOM) kills, CrashLoopBackOff, or short-lived tasks), each event triggers a new rebalance epoch.

Resolution: Use static membership by assigning a unique group.instance.id:

config = {
    'bootstrap.servers': bootstrap_servers,
    'group.id': 'my-group',
    'group.protocol': 'consumer',
    'group.instance.id': f'consumer-{unique_identifier}'  # Unique per consumer
}

With static membership:

Short restarts within the session timeout don’t trigger rebalances.
The consumer rejoins with the same partition assignment.
Scaling up (adding new consumers) still works. New group.instance.id values trigger assignment of unassigned partitions only.

Monitoring and validation

After applying changes, confirm the improvement:

Check broker logs: Confirm that “member session expired” messages no longer appear during normal operations or deployments.
Monitor consumer lag: Use the SumOffsetLag and EstimatedMaxTimeLag Amazon CloudWatch metrics to verify that lag returns to zero quickly after a rebalance.
Describe consumer group: Use kafka-consumer-groups.sh --describe to verify that all members are active and stable.

Conclusion

After implementing the consumer protocol, you should observe the following behavior for consumer group rebalances:

Consistently faster rebalance times compared to the classic protocol.
Fewer session timeout-related rebalances.
More stable consumer group membership.
Smooth scaling operations without disrupting existing consumers.
Fewer unnecessary rebalances during consumer restarts when using static membership.
Clean consumer departures without waiting for timeout expiration when using graceful shutdown.

To get started, try the consumer protocol in your non-production workloads and observe the rebalance improvements as you scale your workload up and down.

To learn more about Amazon MSK and the consumer rebalance protocol, see the following resources:

About the authors

Building an LLM-powered DAG failure analysis plugin for Amazon MWAA

Sushant Samantaray — Tue, 29 Sep 2026 16:17:09 +0000

Apache Airflow has become the orchestration backbone for data pipelines across industries. But as those pipelines grow to hundreds of directed acyclic graphs (DAGs) spanning services like AWS Glue, Amazon EMR, Amazon Athena, and Amazon Redshift, debugging a single task failure turns into a significant operational challenge. When a task fails, data engineers sift through logs, cross-reference DAG configurations, and analyze error messages to find the root cause, delaying pipeline service level agreements (SLAs) and impacting team productivity.

In this post, we show you how to build a custom Apache Airflow plugin that integrates with Amazon Bedrock to automatically analyze DAG task failures and provide actionable diagnostic insights. The plugin deploys to Amazon Managed Workflows for Apache Airflow (Amazon MWAA) and provides AI-powered root cause analysis on demand.

The complete source code for this solution is available in the sample-aws-mwaa-llm-powered-plugin GitHub repository. Clone the repository and follow along as we explain the design decisions throughout this post.

Solution overview

Apache Airflow is a widely adopted open source platform for programmatically authoring, scheduling, and monitoring complex data pipelines. Teams use Airflow to orchestrate extract, transform, and load (ETL) processes, machine learning workflows, and data lake management across industries.

Amazon MWAA is a managed service that makes it straightforward to run Apache Airflow on AWS without the operational burden of managing the underlying infrastructure. With Amazon MWAA, you can focus on authoring workflows and business logic while AWS handles provisioning, patching, scaling, and securing your Airflow environments.

The solution uses the following AWS services:

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) – Hosts the Airflow environment and plugin.
Amazon Bedrock – Provides foundation model (FM) inference (Anthropic Claude) for failure analysis.
Amazon Simple Storage Service (Amazon S3) – Stores plugin artifacts, DAG files, and operator scripts.

The plugin adds an analysis view directly into your Airflow UI. At a high level, when a task fails and you trigger an analysis, the plugin automatically does the following:

Retrieves the failed task instance metadata from the Airflow metadata database.
Collects comprehensive context including task logs, DAG source code, and operator-specific scripts.
Sends the enriched context to Amazon Bedrock for analysis.
Returns a structured diagnostic report with root cause identification, step-by-step resolution, and prevention recommendations.

How it works

The preceding four steps happen behind a single Analyze Task action. The following diagram and pipeline show the high-level architecture and how the plugin carries them out.

Figure 1: High-level architecture of the LLM-powered task analyzer plugin on Amazon MWAA

The plugin follows a multi-step analysis pipeline:

User triggers analysis – From the Airflow UI, you select a failed task and choose Analyze Task.
Context collection – The plugin retrieves task metadata, execution logs, and DAG source code from the Airflow metadata database and Amazon S3.
Operator-aware enrichment – Based on the operator type, the plugin fetches the actual code or query that failed (for example, a PySpark script from AWS Glue or a SQL query from Amazon Athena).
Foundation model analysis – The enriched context is sent to Amazon Bedrock, which returns a structured diagnostic report.
Results presentation – The analysis displays in the Airflow UI with actionable recommendations.

All AWS API calls (Amazon Bedrock, Amazon S3, and AWS Glue) are authenticated through the aws_default Airflow connection. By default on Amazon MWAA, this connection has no static credentials, so boto3 falls back to the environment’s execution role. This means there are no keys to manage or rotate. If you need to call Amazon Bedrock or fetch scripts using a different identity, you can supply those credentials in the aws_default connection. This can be a dedicated IAM role or a cross-account principal, used instead of the execution role.

Operator-aware context collection

A key differentiator of this solution is its ability to understand different Airflow operator types and automatically fetch the associated code or queries. Unlike generic log analyzers, the plugin retrieves the actual code that failed, not just the error message.

The following table summarizes what the plugin fetches for each operator type:

Operator type	What the plugin fetches	Source
GlueJobOperator	PySpark or Python script	Amazon S3 (from the AWS Glue job definition)
EmrAddStepsOperator	Spark or Python script	Amazon S3 (from step arguments)
EmrServerlessStartJobOperator	Spark script	Amazon S3 (from job driver)
AthenaOperator	SQL query	Inline (from operator parameters)
RedshiftDataOperator	SQL query	Inline (from operator parameters)
BashOperator	Bash command	Inline (from operator parameters)
PythonOperator	Python function	DAG source code

This approach means the foundation model can analyze the actual logic that failed, correlating error messages with specific lines in your code for precise root cause identification.

Prerequisites

Before you begin, make sure that you have the following:

An Amazon MWAA environment running Apache Airflow 3.x (this walkthrough uses Airflow 3.2). The plugin registers its UI through the FastAPI-based plugin interface (fastapi_apps) introduced in Airflow 3.x. For setup instructions, see Get started with Amazon MWAA.
Access to Amazon Bedrock with the Anthropic Claude model family enabled in your AWS Region. This walkthrough uses Anthropic Claude, but you can adapt the plugin to work with Amazon Nova or other foundation models by modifying the prompt payload format in prompts.py. See Model access.
An AWS Identity and Access Management (IAM) execution role for Amazon MWAA with bedrock:InvokeModel and s3:GetObject permissions.
An Amazon S3 bucket backing your Amazon MWAA environment with bucket versioning enabled. See Create an Amazon S3 bucket for Amazon MWAA.
Python 3.10 or later installed locally.
The AWS Command Line Interface (AWS CLI) configured with appropriate permissions.

Note: In most Regions, you invoke Claude through an inference profile ID (for example, us.anthropic.claude-sonnet-4-5-20250929-v1:0) rather than a bare on-demand model ID. Run aws bedrock list-inference-profiles to confirm a model is ACTIVE before configuring it.

Plugin design

In this section, we explain the plugin design and its key components. The next section walks through deploying it to your Amazon MWAA environment.

Plugin structure

The plugin follows the standard Apache Airflow plugin architecture. The repository is organized as follows:

plugins/
├── task_analyzer_plugin.py    # Main plugin: FastAPI app, endpoints, registration
└── task_analyzer/
    ├── __init__.py
    ├── prompts.py             # Bedrock model configuration and prompt templates
    ├── script_utils.py        # Operator-specific script fetching logic
    ├── templates/
    │   └── index.html
    └── static/
        ├── css/
        │   └── styles.css
        └── js/
            ├── app.jsx
            ├── components.jsx
            ├── config.js
            ├── template.jsx
            └── utils.jsx

The repository also includes example DAGs that simulate various failure scenarios across different operator types.

Plugin registration

In Apache Airflow 3.x, the web component of a plugin is registered as a FastAPI application through the fastapi_apps attribute. In task_analyzer_plugin.py, the TaskAnalyzerPlugin class registers the FastAPI app under /task-analyzer and adds a view to the task instance page:

class TaskAnalyzerPlugin(AirflowPlugin):
    name = "task_analyzer_plugin"

    fastapi_apps = [
        {
            "app": app,
            "url_prefix": "/task-analyzer",
            "name": "Task Analyzer",
        }
    ]

    external_views = [
        {
            "name": "Analyze Task",
            "href": "/task-analyzer/",
            "url_route": "task_analyzer_view",
            "destination": "task_instance",
        }
    ]

Airflow automatically discovers any AirflowPlugin subclass in the plugins folder. No registration call or configuration change is needed. On Amazon MWAA, the file is delivered inside plugins.zip and extracted to /usr/local/airflow/plugins/.

Analysis engine

The analysis engine is the POST /api/analyze-task endpoint in task_analyzer_plugin.py. When you trigger an analysis, the endpoint performs the following steps:

Retrieves AWS credentials from the aws_default Airflow connection. To override, edit the aws_default connection in the Airflow UI (Admin > Connections).
Assembles a context dictionary from the request (task metadata, logs, DAG source).
Enriches the context with an operator-specific script through fetch_and_add_operator_script.
Builds the prompt using the template in prompts.py.
Invokes Amazon Bedrock and returns the structured analysis.

Operator script fetching

The process_operator_script function in script_utils.py routes script retrieval based on operator type:

External scripts (AWS Glue, Amazon EMR) – The plugin calls the AWS Glue API to look up the job definition, then reads the PySpark script from Amazon S3. Amazon EMR handlers follow the same pattern, extracting the script path from the step configuration or job driver.
Inline scripts (Amazon Athena, Amazon Redshift, BashOperator, PythonOperator, DBTOperator) – The plugin reads the query or command directly from the task’s rendered template fields with no external API call.

The plugin implements smart fetching: for external scripts, it only makes the Amazon S3 API call when the error message contains code-relevant patterns (such as SyntaxError, TypeError, or data type mismatch). Infrastructure errors like timeouts skip the script fetch entirely, minimizing unnecessary API calls.

Prompt engineering

The prompt template in prompts.py provides the foundation model with:

Task metadata (DAG ID, task ID, run ID, state).
Error message and execution logs.
DAG source code.
Operator-specific script (when available).

The model produces a structured diagnostic report with root cause identification, step-by-step resolution, and prevention recommendations. Model IDs are configurable through Airflow Variables, so you can switch between Claude Sonnet and Claude Opus without redeploying the plugin.

Security measures

Before sending content to Amazon Bedrock, the plugin applies the following safeguards:

Credential redaction – The sanitize_script function removes sensitive patterns (passwords, tokens, access keys) from scripts and logs.
Content truncation – The truncate_script function caps content size to stay within model context windows.
Path traversal prevention – The read_allowlisted_file function resolves canonical paths and verifies they reside within allowed base directories before reading any file.

For the full implementation, see script_utils.py.

Optional: PII detection and redaction. The built-in sanitize_script function targets credential patterns. If your logs or scripts might contain personally identifiable information (PII), consider adding a detection pass with Amazon Comprehend before invoking Amazon Bedrock. The DetectPiiEntities API returns the entity types (such as names, email addresses, or account numbers) and their character offsets. You can use these offsets to mask or obfuscate the spans before the context leaves your environment. This adds one API call and cost per analysis, so add it where your compliance requirements call for it. For guidance, see Detecting PII entities.

Deploy the plugin

Follow these steps to deploy the plugin to your Amazon MWAA environment.

Step 1: Clone the repository

git clone https://github.com/aws-samples/sample-aws-mwaa-llm-powered-plugin.git
cd sample-aws-mwaa-llm-powered-plugin

Step 2: Package and upload to Amazon S3

Create the plugins.zip archive from the plugins/ directory and upload it to your Amazon MWAA S3 bucket:

cd plugins
zip -r ../plugins.zip .
cd ..

aws s3 cp plugins.zip s3://<amzn-s3-demo-bucket>/plugins.zip

aws s3api head-object \
  --bucket <amzn-s3-demo-bucket> \
  --key plugins.zip \
  --query VersionId --output text

Note the VersionId returned. You need it in the next step.

Note: This plugin requires only fastapi and Boto3, both pre-installed on Amazon MWAA for Airflow 3.x. You don’t need a requirements.txt file. Skipping the requirements file avoids package resolution conflicts that are a common cause of failed Amazon MWAA environment updates.

Step 3: Update the Amazon MWAA environment

Update your environment to use the new plugin archive:

aws mwaa update-environment \
  --name <your-environment-name> \
  --plugins-s3-path plugins.zip \
  --plugins-s3-object-version <version-id-from-step-2>

The environment restarts automatically. This process typically takes 10–30 minutes. Monitor the status with:

aws mwaa get-environment \
  --name <your-environment-name> \
  --query "Environment.{Status:Status,Plugins:PluginsS3Path}" --output json

Step 4: Configure the Amazon Bedrock connection

On Amazon MWAA, the aws_default connection exists by default and resolves to your environment’s execution role. In most cases, no action is needed.

To override the Region, edit the aws_default connection in the Airflow UI (Admin > Connections) and set the Extra field to:

{"region_name": "us-east-1"}

Leave login and password empty so the execution role is used.

Step 5: Verify the deployment

After the environment finishes updating, navigate to Admin > Plugins in the Airflow UI. Verify that task_analyzer_plugin appears in the list. The Analyze Task entry is now available from any task instance view.

Test the solution

The repository includes example DAGs that simulate failure scenarios across different operator types. To validate the deployment:

Copy the dags/ directory contents to your Amazon MWAA S3 bucket’s DAGs folder:
```
aws s3 cp dags/ s3://<amzn-s3-demo-bucket>/dags/ --recursive
```
Wait for Amazon MWAA to sync the DAGs (typically 1–2 minutes).
In the Airflow UI, trigger one of the test DAGs (for example, test_aws_sql_operators) and let the intentional failure occur.
Navigate to the failed task instance.
Choose Analyze Task in the task instance view.
Review the generated analysis, which includes:
- Root cause identification with file and line references.
- Step-by-step resolution with code examples.
- Prevention recommendations and monitoring suggestions.

The analysis typically completes within 5–10 seconds.

Cost considerations

The primary cost driver for this solution is Amazon Bedrock inference, which is billed by the number of input and output tokens each analysis consumes. Input tokens come from the task logs, DAG source, and operator script sent to the model. Output tokens come from the diagnostic report the model returns. Larger logs and scripts increase input tokens, and the model you select affects the per-token rate. For current per-model rates, see Amazon Bedrock pricing.

To help control cost, the plugin includes a caching mechanism that stores results keyed by a hash of the error context. Repeated analyses of the same failure pattern return cached results without invoking Amazon Bedrock again.

Best practices

When you deploy this solution in production, consider the following:

IAM least privilege – Grant only bedrock:InvokeModel for your chosen model IDs and scope s3:GetObject to specific bucket paths where your operator scripts reside. For guidance, see Amazon MWAA execution role.
Data sanitization – The plugin redacts credentials and truncates content before sending data to Amazon Bedrock. Store configuration values in AWS Secrets Manager rather than hardcoding them in DAG source files.
Access control – The plugin’s endpoints are protected by Airflow’s built-in authentication. For DAG-level access management at scale, see Automated tag-based DAG permission management in Amazon MWAA.
Operational resilience – Add retry logic and circuit breaker patterns around the Amazon Bedrock API call. Use Amazon CloudWatch to monitor plugin performance and set alarms on failure rates.

Extending the solution

You can extend this solution in the following ways:

Proactive notifications – Integrate with Amazon Simple Notification Service (Amazon SNS) or Slack to deliver analyses automatically when failures occur.
Knowledge base integration – Build a knowledge base of past analyses using Amazon Bedrock Knowledge Bases for Retrieval Augmented Generation (RAG) powered recommendations that learn from your organization’s historical failures.
Additional operator support – Add handlers for custom operators specific to your organization, such as proprietary data connectors or internal platform integrations.
Automated remediation – For well-understood failure patterns, trigger automated fixes such as restarting tasks with adjusted resource configurations.

Clean up

To remove the plugin from your environment:

Delete the plugin archive from Amazon S3:

aws s3 rm s3://<amzn-s3-demo-bucket>/plugins.zip

Update your Amazon MWAA environment to remove the plugin reference, then wait for the environment to restart.
Optionally, remove the Amazon Bedrock permissions from your execution role if they are no longer needed.

Conclusion

In this post, we showed you how to deploy an LLM-powered DAG failure analysis plugin for Amazon MWAA using Amazon Bedrock. The operator-aware context collection differentiates this approach from generic log analyzers. By fetching the actual code from AWS Glue, Amazon EMR, and other services, the foundation model provides precise, actionable recommendations with specific line references.

To get started, clone the sample-aws-mwaa-llm-powered-plugin repository, deploy it to a development Amazon MWAA environment, and test with the included example DAGs. As your team builds confidence in the analysis quality, roll it out to production environments where it serves as the first line of investigation for any pipeline failure.

About the authors

Announcing Spark Connect on Amazon EMR on EKS: Interactive PySpark development, anywhere

Amit Maindola — Tue, 29 Sep 2026 16:12:17 +0000

Today, we’re announcing support for Spark Connect on Amazon EMR on EKS, starting from EMR release 7.14 (Apache Spark 3.5.8) and emr-spark-8.1 (Apache Spark 4.1.1). You can now build, test, and debug Spark applications from your preferred tools, such as VS Code, PyCharm, Jupyter notebooks, Amazon SageMaker Unified Studio. At the same time, your full-scale Spark operations run on Amazon Elastic Kubernetes Service (Amazon EKS).

Deploying Spark applications from a local development environment to a remote Amazon EKS cluster often means dealing with environment differences, dependency conflicts, and performance gaps at scale. Spark Connect removes this friction. It separates your application client from the Spark server, so you develop and debug locally while Spark Connect routes your operations to a scalable Spark cluster running on Amazon EKS.

This client-server architecture supports a range of use cases, including interactive development from notebooks and IDEs, embedded Spark in web services, and continuous integration and continuous delivery (CI/CD) data-quality tests. All of these run on your existing EKS infrastructure. Each Spark Connect session uses its own AWS Identity and Access Management (IAM) execution role, custom tags, and cost tracking. For more information, see the Amazon EMR on EKS documentation.

Here are two demonstrations of using Spark Connect in Amazon SageMaker Unified Studio Notebooks and in a VS Code local IDE:

Amazon SageMaker Unified Studio Notebooks demo:

Local IDE demo:

For a runnable end-to-end example in an IDE, try the Spark Connect sample notebook in the aws-emr-utilities repository. It includes a client wrapper solution, built by AWS architects, for simplified connectivity:

How Spark Connect works on Amazon EMR on EKS

Spark Connect uses a client-server architecture that separates application code from the Spark engine:

Client – A lightweight PySpark library running in your environment (such as an IDE or notebook). It doesn’t need Spark installed, direct access to data, or resources sized for the workload.
Connection (EMR managed endpoint) – The client sends Spark operations over a secure gRPC/TLS channel to the Spark Connect server.
Server – Runs Spark pods in your Amazon EMR on EKS namespace, starting from a minimum of two executors (adjustable) with autoscaling. The server performs Spark operations using the EKS compute resources and accesses data stores, such as an Amazon Simple Storage Service (Amazon S3) bucket, through job execution roles.
Results – The server streams query results back to the client through gRPC as Apache Arrow-encoded row batches.

Figure 1: Spark Connect’s client-server architecture

On endpoint creation, Amazon EMR on EKS launches the Spark Connect server as pods on EKS and returns an Elastic Load Balancing (ELB)-backed endpoint and a short-lived token. You don’t need to provision any server or networking manually. Because the Spark Connect server runs on the EKS cluster you already operate, it inherits the node types, container images, and Spark configurations. What you see while developing Spark applications on the client side is what runs in the EKS environment at scale.

To provide a secure, simplified experience, Amazon EMR on EKS provisions two additional components on first use of Spark Connect on the EKS cluster:

Figure 2: Shared Envoy router and Secret Agent service on the EKS cluster

Managed authentication-proxy router – a shared Envoy router with three replicas by default (adjustable), fronted by a Network Load Balancer (NLB). It routes client traffic to the correct server pods, terminates TLS, and validates the session token. One router serves Spark Connect endpoints on the EKS cluster.
Secret Agent service – a lightweight, long-running pod that manages the short-lived credentials for session authentication. One service per EMR security configuration.

These components are long-running and shared across endpoints. Amazon EMR on EKS creates them automatically with the first endpoint on the cluster. Because the router is cluster-scoped and Secret Agent is namespace-scoped, deleting a managed endpoint doesn’t remove them. They keep running so that new endpoints can start within a minute. The router’s replica count is tunable. Scale down for non-production environments to reduce cost or scale up for higher throughput.

To fully remove these components:

Terminate all active managed endpoints and their virtual cluster that reference the Secret Agent’s security configuration, then delete the security configuration.
Once the last session-enabled virtual cluster is deleted, the authentication-proxy router and its underly resources, including the NLB and VPC endpoint, are removed automatically.
Alternatively, delete the EKS cluster to remove all in-cluster components at once.

Why use Spark Connect on Amazon EMR on EKS

With Amazon EMR on EKS, teams can run Spark alongside other applications on shared Kubernetes clusters with existing infrastructure, operational tooling, and system expertise. Spark Connect extends that value to interactive, embedded, and self-service Spark workloads. Your client stays lightweight while Spark code runs in governed, scalable server pods on EKS.

Interactive development on shared Kubernetes clusters

Data engineers and scientists iterate on Spark code cell-by-cell in notebooks or local IDEs. The Spark engine runs remotely on EKS, so validation runs on the same engine as your batch workloads. After validation on the Spark Connect client, the same Spark code deploys as a batch StartJobRun with no changes.

Spark Connect sessions run as pods on your existing cluster. They reuse your EKS RBAC, network policies, node autoscaling, and observability stack (Prometheus, Grafana, Amazon CloudWatch Container Insights). There are no separate compute and monitoring layers to operate.

Embedded Spark in applications and services

The Spark Connect client is a compact PySpark library. Teams can embed Spark operations directly into Python applications such as web services, dashboards, automation scripts, or backend APIs. The heavy processing runs on EKS while the application stays lightweight.

Teams can also expose Spark Connect as a self-service capability on their internal application. Business users submit Spark SQL scripts from a web UI. The compute runs on Spark Connect server on EKS, so the team manages capacity, security, and upgrades centrally.

Multi-tenant data exploration with governance

Each Spark Connect session uses the data user’s IAM permissions that you configure, limiting their access to authorized AWS services, data lake tables, and S3 paths. Every session carries tags with user, project, endpoint and virtual cluster IDs, feeding directly into billing and compliance reports. Meanwhile, data producers maintain guardrails on source data without blocking self-service exploration.

To manage resource consumption across teams, Amazon EMR on EKS virtual clusters provide namespace-level isolation. Each tenant binds their Spark Connect endpoints to a virtual cluster (a namespace) with independent IAM roles. Using resource quotas and limit ranges on EKS, you can protect each virtual cluster by controlling the compute resources that Spark Connect sessions can consume. Importantly, activating EKS split-cost allocation tags helps with chargeback reporting in a multi-tenant environment.

Reusable container images and scalable deployment

Teams often maintain custom container images with proprietary libraries, including internal feature stores, compliance toolkits, UDFs, or machine learning (ML) frameworks. With Spark Connect on Amazon EMR on EKS, teams reuse those same images as the Spark runtime for interactive sessions. No separate dependency lists needed. The same image works for both batch jobs and Spark Connect sessions.

Beyond the image itself, you can control Spark pod scheduling in Amazon EMR on EKS through pod templates and managed endpoint APIs, scaling across your environment. For example, you can:

Pin server pods to specific node types through pod templates. For example, Spot for cost savings.
Apply Spark Dynamic Resource allocation (DRA) to right-size each interactive session.
Use GPU node pools for accelerated Spark RAPIDS or ML.

cat > /tmp/spark-connect-endpoint.json << EOF
{
  "name": "spark-connect-custom-config",
  "virtualClusterId": "$VC_ID",
  "type": "SPARK_CONNECT",
  "releaseLabel": "emr-7.14.0-latest",
  "executionRoleArn": "$ROLE_ARN",
  "configurationOverrides": {
    "applicationConfiguration": [{
      "classification": "spark-defaults",
      "properties": {
        "spark.kubernetes.container.image": "${CUSTOM_IMAGE_URI}",
        "spark.kubernetes.executor.podTemplateFile": "s3://$S3BUCKET/exec-pod-template.yaml",
        "spark.kubernetes.node.selector.karpenter.sh/nodepool": "gpu-pool",
        "spark.dynamicAllocation.enabled": "true",
        "spark.dynamicAllocation.minExecutors": "0"
      }
    }]
  }
}
EOF

aws emr-containers create-managed-endpoint \
--cli-input-json file:///tmp/spark-connect-endpoint.json

Multi-cluster, multi-Region, and hybrid architectures

Enterprises running EKS clusters across multiple AWS accounts, AWS Regions, or hybrid environments with on-premises Kubernetes can use Spark Connect to query data wherever it’s processed. The lightweight client only needs to reach the Spark Connect endpoint, not the underlying S3 buckets or AWS Glue data catalogs. This means no VPC peering or direct network paths to every data store.

The client-server split is the core architectural advantage of Spark Connect on Amazon EMR on EKS. A developer on a laptop behind a VPN, a CI/CD deployment pipeline in a centralized service account, or an Airflow DAG orchestrating across Regions can all connect to a remote Spark server on EKS. This works regardless of where the client itself runs. This decoupling simplifies cross-Region or cross-account analytics without duplicating data or requiring direct access to each data store.

Getting started

To create a Spark Connect endpoint on Amazon EMR on EKS, complete the following steps:

Create EMR namespaces on EKS.
Create an EMR security configuration.
Create a virtual cluster with the security configuration.
Create a Spark Connect managed endpoint.
Obtain a session token.
Connect from your application.

Prerequisites

To proceed with this post, make sure you have the following:

An active AWS account with permissions to create Amazon EMR on EKS resources.
An Amazon EKS cluster
An AWS Load Balancer Controller installed on your EKS cluster.
AWS Command Line Interface (AWS CLI) 2.x >=2.35.23, boto3 >=1.43.48.
pyspark[connect]==3.5.8 in Python 3.8+ environment (client library for EMR 7.14).
Or pyspark[connect]==4.1.1 in Python 3.10+ environment (client library for emr-spark-8.1).
A job execution IAM role.

Step 1: Create EMR namespaces

# set environment variables
export EKS_CLUSTER_NAME=my-eks-cluster
export USER_NAMESPACE=spark-connect-1
export SYS_NAMESPACE=spark-connect-1-system
export AWS_REGION=us-west-2
# connect to your EKS cluster
aws eks update-kubeconfig --name $EKS_CLUSTER_NAME --region $AWS_REGION
kubectl create namespace $USER_NAMESPACE
kubectl create namespace $SYS_NAMESPACE

Step 2: Create a security configuration

cat > /tmp/sec-config.json << EOF
{
  "name": "spark-connect-1-sc",
  "securityConfigurationData": {
    "authenticationConfiguration": {
      "identityCenterConfiguration": { "enableIdentityCenter": false }
    }
  },
  "containerProvider": {
    "type": "EKS",
    "id": "$EKS_CLUSTER_NAME",
    "info": { "eksInfo": { "namespace": "$SYS_NAMESPACE" } }
  }
}
EOF

SEC_CONFIG_ID=$(aws emr-containers create-security-configuration \
--region $AWS_REGION \
--cli-input-json file:///tmp/sec-config.json \
--query id \
--output text)
echo "Security Configuration ID: $SEC_CONFIG_ID"

Step 3: Create a virtual cluster with the security configuration

cat > /tmp/vc.json << EOF
{
  "name": "spark-connect-demo",
  "containerProvider": {
    "id": "$EKS_CLUSTER_NAME",
    "type": "EKS",
    "info": {"eksInfo": {"namespace": "$USER_NAMESPACE"}}
  },
  "securityConfigurationId": "$SEC_CONFIG_ID",
  "sessionEnabled": true
}
EOF

VC_ID=$(aws emr-containers create-virtual-cluster \
--region $AWS_REGION \
--cli-input-json file:///tmp/vc.json \
--query 'id' \
--output text)
# validate the virtual cluster
echo "Virtual Cluster ID: $VC_ID"
aws emr-containers describe-virtual-cluster --region $AWS_REGION --id $VC_ID

Step 4: Create a Spark Connect managed endpoint

Start an interactive session on your virtual cluster. Provide a job execution role that grants the session access to your data sources.

# reuse an existing execution role
ROLE_ARN="arn:aws:iam::YOUR_ACCOUNT_ID:role/EMRonEKSExecutionRole"
cat > /tmp/spark-connect-endpoint.json << EOF
{
  "name": "spark-connect-demo",
  "virtualClusterId": "$VC_ID",
  "type": "SPARK_CONNECT",
  "releaseLabel": "emr-7.14.0-latest",
  "executionRoleArn": "$ROLE_ARN",
  "sessionIdleTimeoutInMinutes": 1440,
  "configurationOverrides": {
    "applicationConfiguration": [{
      "classification": "spark-defaults",
      "properties": {
        "spark.dynamicAllocation.enabled": "true",
        "spark.dynamicAllocation.minExecutors": "0",
        "spark.dynamicAllocation.maxExecutors": "2"
      }
    }]
  }
}
EOF

EP_ID=$(aws emr-containers create-managed-endpoint \
--region $AWS_REGION \
--cli-input-json file:///tmp/spark-connect-endpoint.json \
--query 'id' \
--output text)
# Validate
echo "Endpoint ID: $EP_ID"
export EP_URL=$(aws emr-containers describe-managed-endpoint \
--region $AWS_REGION \
--virtual-cluster-id $VC_ID \
--id $EP_ID \
--query 'endpoint.authProxyUrl' \
--output text)
echo "Endpoint URL: $EP_URL"

Figure 3: Managed endpoint creation output with the endpoint ID and URL

You can optionally pass some custom configuration overrides and tags:

aws emr-containers create-managed-endpoint \
--type SPARK_CONNECT \
--virtual-cluster-id $VC_ID \
--name more-endpoint \
--execution-role-arn $ROLE_ARN \
--release-label emr-7.14.0-latest \
--configuration-overrides '{
"applicationConfiguration": [{
"classification": "spark-defaults",
"properties": {
"spark.executor.instances": "1",
"spark.executor.memory": "4g",
"spark.executor.cores": "1",
"spark.sql.extensions": "org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions"
}
}]
}' \
--tags '{
"team": "data-engineering",
"project": "customer-analytics"
}'

Step 5: Obtain a session token

Request a session token after the managed endpoint is active:

# get a session token with a 12-hour expiry (adjustable)
export TOKEN=$(aws emr-containers get-managed-endpoint-session-credentials \
--region $AWS_REGION \
--virtual-cluster-identifier $VC_ID \
--endpoint-identifier $EP_ID \
--execution-role-arn $ROLE_ARN \
--credential-type TOKEN \
--duration-in-seconds 43200 \
--query 'credentials.token' \
--output text)
echo "Session Token: $TOKEN"

Security note: Communication between your environment and the Spark Connect server is encrypted using TLS. The authentication token is time-limited (15 minutes by default). For long-running sessions, refresh the token periodically by calling get-managed-endpoint-session-credentials again. Consider using AWS Secrets Manager to store and retrieve tokens programmatically.

Step 6: Connect from your application

Use the returned endpoint URL and token to connect from a PySpark-compatible environment. The following Python code shows how to establish a Spark Connect session:

import os
from pyspark.sql import SparkSession
session_endpoint = os.environ["EP_URL"]
auth_token = os.environ["TOKEN"]
spark_conn_url = (f"{session_endpoint};use_ssl=true;x-aws-proxy-auth={auth_token}")
spark = SparkSession.builder
.remote(spark_conn_url)
.getOrCreate()
# verify the connection
print(f"Connected remotely! Spark version: {spark.version}")
# query data through the AWS Glue Data Catalog
df = spark.sql("SELECT * FROM my_catalog.my_database.my_table LIMIT 10")
df.show()

After you’re connected, you can:

Debug interactively – Set breakpoints, inspect DataFrames, and step through Spark code in your IDE or notebook while the operations run remotely on EKS.
Combine local and remote processing – Pull query results back to the client as a pandas or PyArrow DataFrame for local analysis, visualization, or ML (scikit-learn, notebook widgets), then push further Spark operations back to the server in the same session. Heavy processing stays on Amazon EMR on EKS. Only the results you request cross the wire.
Reconnect without losing state – A managed endpoint runs independently of single clients for a configurable idle timeout (default: 60 minutes). Your Spark session, cached data, and temporary views are preserved on the server between connections. When a session token expires (default: 15 minutes, configurable up to 12 hours), request a new token and reconnect to the same endpoint to resume where you left off.
Reuse across workload types – The same client connection pattern works everywhere Python runs: notebooks, IDEs, batch scripts, Airflow operators, or web services. One endpoint, one connection pattern, many workload types.

Validation

After you create the endpoint, verify that the Spark Connect server is running and reachable through Amazon EMR on EKS API and standard Kubernetes tooling:

# get endpoint status
aws emr-containers describe-managed-endpoint --virtual-cluster-id $VC_ID --id $EP_ID
# inspect the server pods (driver + executors) in your namespace
kubectl get pods -n $USER_NAMESPACE -l "emr-containers.amazonaws.com/managed-endpoint-id=$EP_ID"
# View driver logs
kubectl logs -n $USER_NAMESPACE <driver-pod-name> -c spark-kubernetes-driver

Figure 4: Endpoint status and the running driver and executor pods

# to view the live Spark UI, port-forward your driver pod:
DRIVER_POD=$(kubectl get pods -n $USER_NAMESPACE \
-l "emr-containers.amazonaws.com/managed-endpoint-id=$EP_ID,emr-containers.amazonaws.com/component=driver" \
-o name)
kubectl port-forward -n $USER_NAMESPACE "$DRIVER_POD" 4040:4040
# Open http://localhost:4040 in your browser

Figure 5: Live Spark UI for the Spark Connect session

Spark Connect endpoints run as pods on your EKS cluster. The existing Kubernetes observability stack, such as CloudWatch Container Insights, Prometheus, and Grafana, captures Spark Connect endpoint metrics alongside other cluster workloads.

Clean up resources

Terminate your session when you’re done to avoid ongoing costs:

# (OPTIONAL) Endpoints are auto-deleted after the idle timeout (default: 60 minutes).
aws emr-containers delete-managed-endpoint \
--virtual-cluster-id $VC_ID \
--id $EP_ID
# Delete the virtual cluster only when no active endpoints remain
aws emr-containers delete-virtual-cluster --id $VC_ID
# Delete Security Configuration
aws emr-containers delete-security-configuration --id $SEC_CONFIG_ID
# remove the remaining EKS namespaces
kubectl delete namespace $USER_NAMESPACE $SYS_NAMESPACE spark-connect-router

Deleting or timing out a managed endpoint automatically removes its corresponding driver and executor pods. The Envoy router and Secret Agent service are shared across endpoints on the EKS cluster and remain running when individual endpoints are terminated. To fully remove these shared components, delete the virtual cluster to remove its corresponding Secret Agent service. Before doing so, ensure that no managed endpoints in the virtual cluster are active. Terminating the last session-enabled virtual cluster automatically removes the Envoy router from the EKS cluster.

Availability and pricing

Spark Connect on Amazon EMR on EKS is available with EMR release 7.14 (Apache Spark 3.5) and emr-spark-8.1 (Apache Spark 4.1), in all AWS Regions where Amazon EMR on EKS is available, except the AWS GovCloud (US) Regions and the China Regions. The Amazon SageMaker Unified Studio experience is available in supported Regions.

There is no additional charge for Spark Connect managed endpoints beyond the standard Amazon EMR on EKS pricing. You pay for underlying Amazon EKS resources such as EC2 and ELB. For timed-out or terminated managed endpoints, EMR automatically removes their Spark pods from the EKS cluster.

Recommendations for cost efficiency:

Use Karpenter (or Cluster Autoscaler) to right-size cluster capacity to session workload demand. This provisions nodes when endpoints need them and removes them when idle, which keeps cost aligned to actual usage.
Schedule interactive session pods on On-Demand instances for persistent compute.
Use AWS Graviton processors for better performance on Spark workloads.
Activate Amazon EMR on EKS Cost Allocation tags to track per-team and per-project spending at granular level.
Keep a single, shared Envoy router and NLB serving all Spark Connect endpoints (the default) on the cluster. Right-size the router replica count (three by default) for your availability requirements.

Considerations and limitations

Before you build on Spark Connect for Amazon EMR on EKS, review the Considerations and limitations in the Amazon EMR on EKS documentation.

Conclusion

In this post, we showed how, with Spark Connect on Amazon EMR on EKS, you can build, test, and debug Spark applications from the tools you already use: IDEs, notebooks, Amazon SageMaker Unified Studio or Airflow. Your workloads run at scale on your existing Kubernetes clusters, with no application code changes.

For teams already running Amazon EMR on EKS, Spark Connect extends your virtual clusters to interactive and embedded workloads. The same virtual cluster that runs your batch StartJobRun jobs now also serves Spark Connect sessions. Each session runs as pods on your EKS cluster, inheriting your node groups, container images, and Spark configurations. Each session also carries its own IAM execution role and cost tags. This extends the security, multi-tenancy, and observability of your Amazon EMR on EKS investment to a broader set of users and use cases.

To get started, visit the Spark Connect on Amazon EMR on EKS documentation, try the Amazon SageMaker Unified Studio Getting Started guide, and review the Amazon EMR on EKS release notes for EMR 7.14.

About the authors

Aurora PostgreSQL zero-ETL integration with Amazon SageMaker

Apurwa Pawar — Tue, 29 Sep 2026 16:10:14 +0000

When you need quick insights from your Amazon Aurora PostgreSQL operational data, traditional analytics approaches force you to build complex extract, transform, and load (ETL) pipelines. These pipelines introduce latency, operational overhead, and data silos, which slow down decision making and increase cost. AWS introduced the support for Amazon Aurora PostgreSQL zero-ETL integration with Amazon SageMaker, providing near real-time data availability for analytics workloads.

The zero-ETL integration automatically replicates the data from your Amazon Aurora PostgreSQL database into a target AWS Glue managed catalog, where it’s available as Apache Iceberg tables. You can then analyze this data through Amazon SageMaker alongside data from other sources using your preferred analytics and machine learning (ML) tools. The data is compatible with Apache Iceberg open standards, so you can use SQL, Apache Spark, business intelligence, and artificial intelligence and machine learning (AI/ML) tools.

In this post, you explore the benefits of this integration, the architectural concepts, and the underlying change data capture (CDC) mechanics. You also go through the setup process and learn how to query your Aurora PostgreSQL data in Amazon SageMaker AI.

Zero-ETL in the lakehouse architecture

The lakehouse architecture of Amazon SageMaker AI brings together data across Amazon Simple Storage Service (Amazon S3) data lakes and Amazon Redshift data warehouses. Because it’s built on open standards, you can build analytics and AI/ML applications on a single copy of data, without moving it between systems.

Amazon SageMaker AI uses AWS Glue Data Catalog and AWS Lake Formation to provide integrated access controls across S3 data lakes and Amazon Redshift data warehouses from a single governance plane.

Understanding change data capture mechanics

At its core, Aurora PostgreSQL zero-ETL integration is powered by CDC. CDC continuously monitors the database transaction log and streams every insert, update, and delete to a downstream target in near real time.

Aurora PostgreSQL uses enhanced logical replication as its CDC engine. Standard PostgreSQL logical replication publishes row-level changes from the write-ahead log (WAL). The enhanced logical replication in Aurora offers added capabilities that make it well-suited for zero-ETL integrations, including automatic DDL propagation and continuous streaming of transactional changes.

Solution overview

With Amazon Aurora PostgreSQL zero-ETL integration with Amazon SageMaker AI, you can:

Remove ETL complexity – Automatically replicate data without building custom ETL pipelines.
Near real-time analytics – Access operational data in Amazon SageMaker AI within seconds of changes in Aurora PostgreSQL.
Unify data analysis – Combine Aurora PostgreSQL data with data from other sources in a single lakehouse architecture.
Reduce costs – Minimize operational overhead and infrastructure costs associated with maintaining ETL pipelines.
Accelerate insights – Query data using familiar SQL tools and integrate with ML workflows in Amazon SageMaker AI.

The following diagram illustrates the architecture of this solution:

Figure 1: Architecture of the Aurora PostgreSQL zero-ETL integration with Amazon SageMaker

The workflow includes the following steps:

Your application writes data to an Amazon Aurora PostgreSQL database cluster.
The zero-ETL integration automatically captures changes from the Aurora PostgreSQL database.
Data is replicated to the target AWS Glue managed catalog in near real time.
You can query and analyze the data using Amazon Athena, Amazon Redshift, or other analytics tools integrated with Amazon SageMaker AI.
Data scientists can build and train ML models using Amazon SageMaker AI with direct access to the Apache Iceberg tables in the target AWS Glue managed catalog.

Prerequisites

Before setting up the zero-ETL integration, verify that you have the following:

An Amazon Virtual Private Cloud (Amazon VPC) setup with the proper networking configurations for database connectivity.
An Amazon Elastic Compute Cloud (Amazon EC2) security group set up and an allowed DB instance port connection to the source and target DB instances.
AWS Command Line Interface (AWS CLI v2) installed and configured with the appropriate AWS Identity and Access Management (IAM) credentials and permissions to interact with Amazon Aurora and SageMaker AI.
Sufficient AWS service quotas for Aurora PostgreSQL resources.

Configure the source PostgreSQL database for zero-ETL integration

When you have all the prerequisites in place, you can configure the source PostgreSQL database for zero-ETL integration.

Create a custom Aurora PostgreSQL cluster parameter group

Your Aurora PostgreSQL database needs to have parameters configured for real-time replication. In this section, you will create the DB cluster parameter group and configure parameters. For more information, see Getting started with Aurora zero-ETL integrations.

Use the following AWS CLI command to create an Aurora PostgreSQL cluster parameter group:

aws rds create-db-cluster-parameter-group \
    --db-cluster-parameter-group-name aurora-pgsql-zetl-cluster-pg \
    --db-parameter-group-family aurora-postgresql16 \
    --description "Aurora PostgreSQL with enhanced logical replication" \
    --region us-east-1 --output json

Now set the parameters by modifying the parameter group:

aws rds modify-db-cluster-parameter-group --db-cluster-parameter-group-name <aurora-pgsql-zetl-cluster-pg> \
    --parameters \
    ParameterName=rds.logical_replication,ParameterValue=1,ApplyMethod=pending-reboot \
    ParameterName=aurora.enhanced_logical_replication,ParameterValue=1,ApplyMethod=pending-reboot \
    ParameterName=aurora.logical_replication_backup,ParameterValue=0,ApplyMethod=pending-reboot \
    ParameterName=aurora.logical_replication_globaldb,ParameterValue=0,ApplyMethod=pending-reboot \
    --region us-east-1 --output json

The parameter group is now fully configured and ready to be applied to your Aurora PostgreSQL cluster.

Select or create a source Aurora PostgreSQL cluster

If you already have an Aurora PostgreSQL cluster, you can use it, or you can create a new Aurora PostgreSQL cluster.

Note: Your source DB cluster must be running a supported version of Aurora PostgreSQL. For a list of supported versions, see Regions and database engines supported for Aurora zero-ETL integrations.

While creating an Aurora PostgreSQL cluster, use the parameter group (aurora-pgsql-zetl-cluster-pg) you created earlier:

Note: Throughout this post, make sure to replace the with your own information.

aws rds create-db-cluster \
    --db-cluster-identifier <aurora-pgsql-zetl> \
    --engine aurora-postgresql \
    --engine-version 16 \
    --master-username <admin> \
    --master-user-password <password> \
    --database-name <my_db> \
    --db-cluster-parameter-group-name <aurora-pgsql-zetl-cluster-pg> \
    --storage-encrypted \
    --kms-key-id alias/aws/rds \
    --backup-retention-period 7 \
    --db-subnet-group-name <dbsubnet> \
    --vpc-security-group-ids <sg-c14219ba> \
    --region <us-east-1> \
    --output json

aws rds create-db-instance \
    --db-instance-identifier <aurora-pgsql-zetl-instance-1> \
    --db-instance-class db.r5.large \
    --engine aurora-postgresql \
    --db-cluster-identifier <aurora-pgsql-zetl> \
    --region us-east-1 \
    --output json

If you’re creating a new Aurora PostgreSQL cluster, wait for your DB instance(s) to be in an “Available” status. You can verify DB instance status by using the describe-db-instances API call:

aws rds describe-db-instances --filters 'Name=db-cluster-id,Values=<aurora-pgsql-zetl>' --output json | grep -o '"DBInstanceStatus": "[^"]*"'

"DBInstanceStatus": "available"

Reboot the cluster to apply parameter changes

A cluster reboot is needed before zero-ETL integration can function correctly:

aws rds reboot-db-instance \
    --db-instance-identifier <aurora-pgsql-zetl-instance-1> \
    --region <us-east-1>

Wait until the cluster and the primary instance are back in Available status. For more information, see reboot-db-instance.

Create a target AWS Glue managed catalog

With your source PostgreSQL database configured for enhanced logical replication, the next step is setting up your target Amazon SageMaker AI. Zero-ETL integration uses AWS Glue Data Catalog backed by Amazon Redshift managed storage as its target. To have this functionality, you need to create a managed catalog, configure IAM permissions for Amazon SageMaker AI to access and query the managed catalog, and set up authorization for incoming integration requests from your source database.

Create an AWS Glue managed catalog

You must create a new catalog (if it doesn’t exist already) managed by AWS Glue to store table metadata and serve as the landing zone for your replicated datasets. Zero-ETL integration streams the data into Amazon Redshift managed storage, and AWS Glue keeps track of table definitions so that tools such as SageMaker AI, Athena, and Amazon Redshift Spectrum can query the data.

Create an IAM role for AWS Glue and Amazon Redshift to access the AWS Glue managed catalog

Now, use the following command to create an IAM role so that AWS Glue and Amazon Redshift can interact with the catalog. This role serves two key functions: It allows AWS Glue and Amazon Redshift to perform catalog operations, and it authorizes incoming integration requests from your source database.

aws iam create-role \
    --role-name <GlueDataCatalogDataTransferRole> \
    --assume-role-policy-document '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Effect": "Allow",
                "Principal": {
                    "Service": [
                        "glue.amazonaws.com",
                        "redshift.amazonaws.com"
                    ]
                },
                "Action": "sts:AssumeRole"
            }
        ]
    }'

Next, attach a policy to this IAM role that provides the minimum required permissions for AWS Glue and Amazon Redshift. This policy should also include the necessary permissions for encryption key actions to help maintain secure data handling throughout the integration process:

aws iam put-role-policy \
    --role-name <GlueDataCatalogDataTransferRole> \
    --policy-name <GlueDataTransferPolicy> \
    --policy-document '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "DataTransferRolePolicy",
                "Effect": "Allow",
                "Action": [
                    "kms:GenerateDataKey",
                    "kms:Decrypt",
                    "glue:GetDatabase",
                    "glue:GetCatalog"
                ],
                "Resource": ["*"]
            }
        ]
    }'

Set up AWS Lake Formation access

Before using the managed catalog for zero-ETL integration, you must configure data lake administrators in AWS Lake Formation who have administrative or read-only permissions on the managed resources. Additionally, you need to grant ReadOnlyAdmin permissions to the Amazon Redshift service-linked role, AWSServiceRoleForRedshift, in your account. If this role doesn’t exist in your account or you need to verify its permissions, see Using service-linked roles for Amazon Redshift.

aws lakeformation put-data-lake-settings \
    --region <us-east-1> \
    --cli-input-json '{
        "DataLakeSettings": {
            "DataLakeAdmins": [
                {
                    "DataLakePrincipalIdentifier": "<arn:aws:iam::111122223333:role/Admin>"
                }
            ],
            "ReadOnlyAdmins": [
                {
                    "DataLakePrincipalIdentifier": "<arn:aws:iam::111122223333:role/aws-service-role/redshift.amazonaws.com/AWSServiceRoleForRedshift>"
                }
            ],
            "CreateDatabaseDefaultPermissions": [],
            "CreateTableDefaultPermissions": [],
            "Parameters": {
                "CROSS_ACCOUNT_VERSION": "4",
                "SET_CONTEXT": "TRUE"
            },
            "AllowExternalDataFiltering": false,
            "ExternalDataFilteringAllowList": []
        }
    }'

Create the AWS Glue managed catalog backed by Amazon Redshift managed storage

Because you have configured IAM permissions and Lake Formation settings, you can now create the AWS Glue managed catalog.

aws glue create-catalog \
    --region <us-east-1> \
    --cli-input-json '{
        "Name": "<zetl-catalog>",
        "CatalogInput": {
            "Description": "A Glue Data Catalog backed by Redshift Managed Storage",
            "CreateDatabaseDefaultPermissions": [],
            "CreateTableDefaultPermissions": [],
            "CatalogProperties": {
                "DataLakeAccessProperties": {
                    "DataLakeAccess": true,
                    "DataTransferRole": "<arn:aws:iam::111122223333:role/GlueDataCatalogDataTransferRole>",
                    "CatalogType": "aws:redshift"
                }
            }
        }
    }'

Register the catalog as a zero-ETL integration target

To prepare your target AWS Glue managed catalog for zero-ETL integration, use the create-integration-resource-property command with these required parameters:

The –resource-arn parameter specifies the Amazon Resource Name (ARN) of your AWS Glue managed catalog that will serve as the integration target.
The –target-processing-properties parameter requires the ARN of an IAM role that has describe permissions on the target AWS Glue managed catalog.

You can use the GlueDataCatalogDataTransferRole created in the earlier step because it already includes the minimal describe permissions needed for this integration. Alternatively, you can create a new IAM role specifically for this purpose and attach the necessary minimal permissions to meet your company’s security requirements.

aws glue create-integration-resource-property \
    --region <us-east-1> \
    <arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog> \
    '{"RoleArn": "<arn:aws:iam::111122223333:role/GlueDataCatalogDataTransferRole>"}'

Example output:

{
    "ResourceArn": "arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog",
    "TargetProcessingProperties": {
        "RoleArn": "arn:aws:iam::111122223333:role/GlueDataCatalogDataTransferRole"
    }
}

Configure authorization for inbound integration requests

The last step in creating a target managed catalog is to define a resource-based access policy that authorizes zero-ETL integration to push data into your catalog. This policy grants AWS Glue the necessary permissions to create and authorize incoming integration requests from your source database. Apply this resource policy by using the AWS Glue put-resource-policy API call to complete the catalog configuration for your zero-ETL integration:

aws glue put-resource-policy \
    --region <us-east-1> \
    --policy-in-json '{
        "Version": "2012-10-17",
        "Statement": [
            {
                "Principal": {
                    "AWS": [
                        "111122223333"
                    ]
                },
                "Effect": "Allow",
                "Action": [
                    "glue:CreateInboundIntegration"
                ],
                "Resource": [
                    "arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:SourceArn": "arn:aws:rds:us-east-1:111122223333:cluster:aurora-pgsql-zetl"
                    }
                }
            },
            {
                "Principal": {
                    "Service": [
                        "glue.amazonaws.com"
                    ]
                },
                "Effect": "Allow",
                "Action": [
                    "glue:AuthorizeInboundIntegration"
                ],
                "Resource": [
                    "arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog"
                ],
                "Condition": {
                    "StringEquals": {
                        "aws:SourceArn": "arn:aws:rds:us-east-1:111122223333:cluster:aurora-pgsql-zetl"
                    }
                }
            }
        ]
    }'

Your AWS Glue managed catalog is now ready to receive data from the zero-ETL integration.

Load data in the source Aurora PostgreSQL database

Now that your Aurora PostgreSQL database is configured and ready, you must populate it with sample data that serves as the historical baseline for your zero-ETL integration. This first dataset provides the foundation for testing and demonstrating the integration capabilities. After you set up the zero-ETL integration, subsequent database changes stream automatically in near real time to your target AWS Glue managed catalog.

Connect to the source Aurora PostgreSQL cluster

Use the following commands to create a connection to your source Aurora PostgreSQL cluster:

psql --host aurora-pgsql-zetl-xxxxx.us-east-1.rds.amazonaws.com --username admin --port 5432 --dbname my_db --password

Create a database and table

Create a table named products to store product information:

CREATE TABLE products (product_id SERIAL PRIMARY KEY,product_name VARCHAR(100) NOT NULL, description TEXT,category VARCHAR(50),price NUMERIC(10,2) NOT NULL,stock_quantity INTEGER DEFAULT 0,is_active BOOLEAN DEFAULT TRUE,created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP );

Insert historical data

Use the following code to insert a row:

INSERT INTO products (product_name, description, category, price, stock_quantity) VALUES ('Laptop', 'High-performance laptop with 16GB RAM and 512GB SSD', 'Electronics', 1299.99, 50);

This table serves as a representative dataset to demonstrate the data capture and streaming capabilities of the zero-ETL integration. After your zero-ETL integration is active, all database changes, including inserts, updates, and deletes, are automatically captured and streamed to your AWS Glue managed catalog. This creates a data pipeline from your Aurora PostgreSQL database to your Amazon SageMaker for real-time analytics on your operational data.

Create a zero-ETL integration

Because your Aurora PostgreSQL database is now populated with historical data, you can set up the zero-ETL integration that continuously streams database changes to your AWS Glue managed catalog backed by Amazon Redshift managed storage.

Create the integration

Create the integration between your source PostgreSQL database and target AWS Glue catalog by using the aws rds create-integration AWS CLI command. You can customize the integration by specifying added configurations, such as data filters, to control which data gets replicated to your target environment:

aws rds create-integration \
    --source-arn <arn:aws:rds:us-east-1:111122223333:cluster:aurora-pgsql-zetl> \
    --target-arn <arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog> \
    --integration-name <zetl-test-integration> \
    --data-filter "include: *.*" \
    --region us-east-1

When you run the command, the zero-ETL integration begins provisioning and enters a ‘creating’ state. The AWS CLI response provides key details about the integration configuration.

Example CLI output:

{
    "SourceArn": "<arn:aws:rds:us-east-1:111122223333:cluster:aurora-pgsql-zetl>",
    "TargetArn": "<arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog>",
    "IntegrationName": "<zetl-test-integration>",
    "IntegrationArn": "<arn:aws:rds:us-east-1:111122223333:integration:4c4d81b9-xxxx>",
    "KMSKeyId": "<arn:aws:kms:us-east-1:111122223333:key/b9130ae1-exxxx>",
    "Status": "creating",
    "Tags": [],
    "CreateTime": "2025-07-04T05:42:56.841000+00:00",
    "DataFilter": "include: *.*"
}

When the integration status changes to “active”, your zero-ETL integration pipeline is fully operational.

Monitor the integration

Before generating new live data, verify that the integration has reached an “active” state by running the describe-integrations AWS CLI command. This monitoring step is important to confirm that changes from your source Aurora cluster are successfully streaming to the AWS Glue managed catalog without errors:

aws rds describe-integrations

{
    "Integrations": [
        {
            "SourceArn": "<arn:aws:rds:us-east-1:111122223333:cluster:aurora-pgsql-zetl>",
            "TargetArn": "<arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog>",
            "IntegrationName": "<zetl-test-integration>",
            "IntegrationArn": "<arn:aws:rds:us-east-1:111122223333:integration:4c4d81b9-xxxx>",
            "KMSKeyId": "<arn:aws:kms:us-east-1:111122223333:key/b9130ae1-xxxx>",
            "Status": "active",
            "Tags": [],
            "CreateTime": "2025-07-04T05:42:56.841000+00:00",
            "DataFilter": "include: *.*"
        }
    ]
}

Verify the zero-ETL integration

Now that your historical data is loaded and the zero-ETL integration is “active”, you must confirm that the data has been successfully replicated.

Grant Lake Formation permissions

Before you can query the AWS Glue managed catalog by using the Amazon Redshift Data API, you must make sure the IAM user or role has the right permissions to create and manage tables within the catalog. Use the Lake Formation grant-permissions API to provide these necessary permissions so that Amazon Redshift can access your AWS Glue managed catalog for the zero-ETL integration. For more information, see Creating an Amazon Redshift managed catalog in the AWS Glue Data Catalog.

aws lakeformation grant-permissions \
    --region <us-east-1> \
    --cli-input-json '{
        "Principal": {
            "DataLakePrincipalIdentifier": "<arn:aws:iam::111122223333:role/Admin>"
        },
        "Resource": {
            "Table": {
                "DatabaseName": "my_db",
                "CatalogId": "<111122223333:zetl-catalog/zetl_0dff6d97-xxxx>",
            },
            "Permissions": [
                "CREATE_CATALOG",
                "DESCRIBE",
                "CREATE_DATABASE",
                "DROP",
                "ALTER"
            ],
            "PermissionsWithGrantOption": [
                "CREATE_CATALOG",
                "DESCRIBE",
                "CREATE_DATABASE",
                "DROP",
                "ALTER"
            ]
        }'

These permissions allow for query execution and metadata inspection on the managed catalog.

Query historical data by using the Amazon Redshift Data API

With the necessary permissions in place, you can now verify your historical data by querying the AWS Glue managed catalog through the Amazon Redshift execute-statement Data API. Begin this verification process by running a SELECT statement against the catalog:

aws redshift-data execute-statement --sql 'SELECT * FROM "zetl_0dff6d97-xxxx@zetl-catalog"."my_db"."products" LIMIT 10;' --database "<arn:aws:glue:us-east-1:111122223333:catalog/pg-zetl-catalog>"

The following command returns a unique query ID that you can use to monitor the execution status and retrieve results from your query:

{
    "CreatedAt": "2025-07-15T00:31:48.778000+00:00",
    "Database": "arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog",
    "DbUser": "IAMR:Admin",
    "Id": "ce1ff0el-xxxx",
}

Monitor your query’s progress by using the describe-statement API with the query ID. Continue checking until the status shows that your query has completed successfully:

//use the Id to make the describe-statement API call to verify execution status is Started
aws redshift-data describe-statement --id <ce1ff0el-xxxx>

{
    "CreatedAt": "2025-07-15T00:31:48.778000+00:00",
    "Database": "arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog",
    "DbUser": "IAMR:Admin",
    "Duration": 6238051060,
    "HasResultSet": true,
    "Id": "2ca8fedf-xxxx",
    "QueryString": "SELECT * FROM \"zetl_0dff6d97-xxxx_zeroetl@pg-zetl-catalog\".\"zetl_default\".\"products\" LIMIT 10;",
    "RedshiftPid": 1073791309,
    "RedshiftQueryId": 1018598,
    "ResultFormat": "json",
    "ResultRows": 1,
    "ResultSize": 149,
    "Status": "FINISHED",
    "UpdatedAt": "2025-07-15T00:31:55.491000+00:00"
}

To complete the verification process and view your historical data now available in Amazon SageMaker AI, retrieve the query results by using the get-statement-result API call:

aws redshift-data get-statement-result --id <ce1ff0el-xxxx>

{
    "Records": [
        [
            {
                "longValue": 1
            },
            {
                "stringValue": "Laptop"
            },
            {
                "stringValue": "High-performance laptop with 16GB RAM and 512GB SSD"
            },
            {
                "stringValue": "Electronics"
            },
            {
                "stringValue": "1299.99"
            },
            {
                "longValue": 50
            },
            {
                "booleanValue": true
            },
            {
                "stringValue": "2026-02-27 13:32:02.697117"
            },
            {
                "stringValue": "2026-02-27 13:32:02.697117"
            }
        ]
    ],
    "ColumnMetadata": [
        ....
        //Skipping metadata
    ],
    "TotalNumRows": 1
}

With your zero-ETL integration now active, you can demonstrate real-time data streaming by adding new data to your source Aurora PostgreSQL instance. Run the following INSERT query to add a new row, which shows how changes are automatically replicated in near real time:

INSERT INTO products (product_name, description, category, price, stock_quantity)
VALUES ('Wireless Mouse', 'Ergonomic wireless mouse with USB receiver and long battery life', 'Electronics', 29.99, 150);

You can verify that the recent changes from your source database have been replicated to the target environment within seconds. Use the same Amazon Redshift Data API workflow you used earlier to confirm the real-time replication:

aws redshift-data execute-statement --sql 'SELECT * FROM "zetl_0dff6d97-xxxx@zetl-catalog"."my_db"."products" LIMIT 10;' --database "<arn:aws:glue:us-east-1:111122223333:catalog/pg-zetl-catalog>"

{
    "CreatedAt": "2025-07-15T00:31:48.778000+00:00",
    "Database": "arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog",
    "DbUser": "IAMR:Admin",
    "Id": "2ca8fedf-a604-4c87-a183-3a553d62354c",
}

Use the describe-statement API call to monitor the query execution and confirm that the status shows ‘FINISHED’ before proceeding to retrieve the results:

aws redshift-data describe-statement --id <ce1ff0ef-xxxx>

{
    "CreatedAt": "2025-07-15T00:31:48.778000+00:00",
    "Database": "arn:aws:glue:us-east-1:111122223333:catalog/zetl-catalog",
    "DbUser": "IAMR:Admin",
    "Duration": 6238051060,
    "HasResultSet": true,
    "Id": "2ca8fedf-xxxx",
    "QueryString": "SELECT * FROM \"zetl_0dff6d97-xxxx_zeroetl@pg-zetl-catalog\".\"zetl_default\".\"products\" LIMIT 10;",
    "RedshiftPid": 1073791309,
    "RedshiftQueryId": 1018598,
    "ResultFormat": "json",
    "ResultRows": 2,
    "ResultSize": 317,
    "Status": "FINISHED",
    "UpdatedAt": "2025-07-15T00:31:55.491000+00:00"
}

Finally, retrieve the query results by using the get-statement-result API call:

aws redshift-data get-statement-result --id <ce1ff0ef-xxxx>

{
    "Records": [
        [
            {
                "longValue": 1
            },
            {
                "stringValue": "Laptop"
            },
            {
                "stringValue": "High-performance laptop with 16GB RAM and 512GB SSD"
            },
            {
                "stringValue": "Electronics"
            },
            {
                "stringValue": "1299.99"
            },
            {
                "longValue": 50
            },
            {
                "booleanValue": true
            },
            {
                "stringValue": "2026-02-27 13:32:02.697117"
            },
            {
                "stringValue": "2026-02-27 13:32:02.697117"
            }
        ],
        [
            {
                "longValue": 2
            },
            {
                "stringValue": "Wireless Mouse"
            },
            {
                "stringValue": "Ergonomic wireless mouse with USB receiver and long battery life"
            },
            {
                "stringValue": "Electronics"
            },
            {
                "stringValue": "29.99"
            },
            {
                "longValue": 150
            },
            {
                "booleanValue": true
            },
            {
                "stringValue": "2026-02-27 15:41:00.206273"
            },
            {
                "stringValue": "2026-02-27 15:41:00.206273"
            }
        ]
    ],
    "ColumnMetadata": [
        ....
        //Skipping metadata
    ],
    "TotalNumRows": 2
}

This verification process confirms that your zero-ETL integration from Aurora PostgreSQL to Amazon SageMaker AI is working and continuously replicating both historical and real-time data. Although zero-ETL integration significantly simplifies data replication, it’s important to understand certain limitations on supported data types, schema change handling, and data filtering capabilities. For more details about these considerations and best practices, see Aurora zero-ETL integrations and Amazon RDS zero-ETL integrations.

Clean up

This section guides you through the cleanup process to remove the resources and components you created during this walkthrough. When you delete a zero-ETL integration, Amazon Aurora removes it from the source Aurora DB cluster. Your transactional data isn’t removed from Amazon Aurora or the analytics destination, but Aurora doesn’t send new data to Amazon SageMaker AI.

Delete the zero-ETL integration: Begin the cleanup process by removing the integration between your source Amazon Relational Database Service (Amazon RDS) database and the AWS Glue managed catalog. Run the following command to delete the integration:

aws rds delete-integration --integration-identifier <arn:aws:rds:us-east-1:111122223333:integration:4c4d81b9-af2a-4b09-b922-007636ba7f66>

Delete the AWS Glue managed catalog: After you successfully delete the integration, delete the AWS Glue managed catalog that served as your zero-ETL target destination. Use the following command to remove the catalog:

aws glue delete-catalog --catalog-id <111122223333:zetl-catalog>

This permanently removes all associated table metadata and Amazon Redshift managed storage references.

Delete the Aurora DB cluster: If you created the source Aurora DB cluster for this demonstration and you no longer need it, you can complete the cleanup by deleting the entire DB cluster. By skipping the final snapshot option, you avoid retaining any test data and confirm complete resource removal:

aws rds delete-db-instance --db-instance-identifier <aurora-pgsql-zetl-instance-1> --skip-final-snapshot --region <us-east-1>

aws rds delete-db-cluster --db-cluster-identifier <aurora-pgsql-zetl> --skip-final-snapshot --region <us-east-1>

aws rds delete-db-cluster-parameter-group --db-cluster-parameter-group-name <aurora-zetl-cluster-pg> --region <us-east-1>

Conclusion

In this post, you learned how to configure zero-ETL integration between Aurora PostgreSQL and your Amazon SageMaker AI using AWS CLI. This integration automatically replicates your PostgreSQL data to a lakehouse in near real time, removing the need for custom ETL pipelines.

As you move forward, consider expanding this zero-ETL approach to more supported data sources, such as Amazon RDS for MySQL and Amazon DynamoDB. This creates a centralized data access strategy across your company. You can also explore advanced analytics scenarios by combining zero-ETL integrations with Amazon Redshift capabilities. These include large-scale SQL analytics, Amazon Redshift ML for in-database ML, and federated queries that span multiple data lakes and warehouses. These integrations provide the foundation for building a near real-time data platform that scales with your business needs.

To get started, see the AWS zero-ETL documentation for setup guidance, supported configurations, troubleshooting integrations, and architectural best practices.

About the authors

Getting started with Apache Iceberg write support in Amazon Redshift – Part 3

Raghu Kuppala — Mon, 28 Sep 2026 15:48:35 +0000

Production data is always evolving. Tables gain and lose columns, outgrow their data types, and get re-partitioned as query patterns shift. Multiple engines often need to read the same data. These changes used to mean expensive data rewrites or rebuilt pipelines. Apache Iceberg makes them metadata-only operations, and Amazon Redshift now supports evolving schemas and partitioning layouts through ALTER statements, with no data rewrites and no pipeline rebuilds. You can also create AWS Lake Formation resource links in the catalog of Amazon S3 Tables, a capability of Amazon Simple Storage Service (Amazon S3), for centralized cross-engine governance.

In Part 1, you created Apache Iceberg tables and wrote data directly from Amazon Redshift to your data lake, setting up external schemas, creating tables in both Amazon Simple Storage Service (Amazon S3) and Amazon S3 Tables, and performing INSERT operations with full ACID (Atomicity, Consistency, Isolation, Durability) compliance. In Part 2, you performed DELETE, UPDATE, and MERGE operations to modify data at the row level and synchronize staging and production tables.

In this post, you use the customer and orders datasets from the previous posts to evolve Iceberg table schemas and partitioning with ALTER operations. You also create an AWS Lake Formation resource link in the S3 Tables catalog to share tables with other analytics engines under a single, centralized permission model.

Solution overview

This solution demonstrates ALTER operations for Apache Iceberg tables in Amazon Redshift and Lake Formation resource link creation for the S3 Tables catalog. The walkthrough includes the following key operations:

ALTER TABLE RENAME COLUMN – Rename existing columns without changing data types or partition specs.
ALTER TABLE ADD/DROP COLUMN – Add new columns or remove existing columns as metadata-only operations.
ALTER TABLE ALTER COLUMN – Widen column data types (for example, INT to BIGINT) without rewriting data.
ALTER TABLE SET TABLE PROPERTIES – Change compression type for future writes.
ALTER TABLE ADD/DROP/REPLACE PARTITION FIELD – Evolve partition specs without re-partitioning existing data.
Lake Formation resource link – Create a resource link in the S3 Tables catalog for centralized access governance.

The following diagram shows the end-to-end architecture:

Figure 1: Architecture showing Amazon Redshift performing ALTER operations on Iceberg tables in S3 Tables, with Lake Formation resource links providing access from Amazon Athena and other engines

Prerequisites

Complete the setup from Part 1 and Part 2, including:

An Amazon Redshift data warehouse (provisioned or Serverless) on patch 201 or higher.
The AWS Identity and Access Management (IAM) role (RedshifticebergRole) with permissions for Amazon S3, AWS Glue Data Catalog, and Lake Formation.
The customer table in a standard Amazon S3 bucket (AWS Glue catalog: customer_db).
The orders table in an Amazon S3 table bucket (iceberg-write-blog@s3tablescatalog).
Access to an IAM role that is a Lake Formation data lake administrator.
AWS Glue Data Catalog integrated with S3 Tables (s3tablescatalog exists).

Schema evolution with ALTER TABLE

With ALTER TABLE, you can change Iceberg table definitions, including schema, partition specs, and properties, without rewriting stored data. Each operation updates only metadata. The table structure changes instantly while existing data files remain untouched. This helps make schema evolution, partition adjustments, and property updates safe to run on production tables.

Add a column

You can add a new column to an Iceberg table using ALTER TABLE. Each new column is added with a unique field ID that Iceberg uses for column tracking across schema evolution. Existing rows return NULL for the newly added column.

Verify the current schema:

SHOW TABLE dev.demo_iceberg.customer;

Figure 2: SHOW TABLE output showing the current customer table schema

Add the column:

-- Add a loyalty_tier column to the customer table
ALTER TABLE dev.demo_iceberg.customer
ADD COLUMN loyalty_tier VARCHAR;

Verify the schema change:

SHOW TABLE dev.demo_iceberg.customer;

Figure 3: SHOW TABLE output showing the loyalty_tier column added to the schema

The following output shows the new loyalty_tier column as NULL for existing rows:

SELECT customer_id, customer_name, city, loyalty_tier
FROM dev.demo_iceberg.customer
ORDER BY customer_id;

Figure 4: Query results showing loyalty_tier as NULL for existing rows

Populate the new column by aggregating order totals from the orders table in S3 Tables:

-- Set loyalty_tier based on total spend from orders
UPDATE dev.demo_iceberg.customer
SET loyalty_tier = CASE
WHEN a.total_spend > 300 THEN 'Gold'
ELSE 'Silver'
END
FROM (
SELECT customer_id, SUM(total_order_amt) AS total_spend
FROM "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
GROUP BY customer_id
) AS a
WHERE dev.demo_iceberg.customer.customer_id = a.customer_id;

The following output shows customer loyalty tiers after the update:

SELECT customer_id, customer_name, loyalty_tier
FROM dev.demo_iceberg.customer
ORDER BY customer_id;

Figure 5: Customer table showing Gold and Silver loyalty tiers

Note: Customer IDs 11, 13, and 15 show NULL for loyalty_tier because they have no matching orders in the orders table.

Drop a column

Remove columns that are no longer needed. The column is removed from the current schema, but data in existing files remains untouched and simply becomes invisible to queries.

Verify the current schema:

SHOW TABLE dev.demo_iceberg.customer;

Figure 6: SHOW TABLE output showing the current customer table schema before dropping loyalty_tier

Drop the column:

-- Drop the loyalty_tier column
ALTER TABLE dev.demo_iceberg.customer
DROP COLUMN loyalty_tier;

Verify the schema change:

SHOW TABLE dev.demo_iceberg.customer;

Figure 7: SHOW TABLE output showing the customer table schema after loyalty_tier is dropped

Verify the column is dropped:

SELECT * FROM dev.demo_iceberg.customer
ORDER BY customer_id;

Figure 8: Query results confirming the loyalty_tier column has been dropped

Note: To drop a column used in the current partition spec, first drop or replace the partition field, then drop the column.

Rename a column

Rename a column without affecting data types or partition specs:

-- Rename city to location
ALTER TABLE dev.demo_iceberg.customer
RENAME COLUMN city TO location;

The following output confirms the column has been renamed to location:

SELECT customer_id, customer_name, location
FROM dev.demo_iceberg.customer
ORDER BY customer_id;

Figure 9: Query results showing the renamed column location

Widen a column type

Widen a column’s data type without rewriting data. This is useful when your data outgrows the original precision, for example when order amounts exceed the original decimal range.

Verify the current column type:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 10: SHOW TABLE output showing total_order_amt as DECIMAL(10,2)

Now run the ALTER to widen the column:

-- Widen total_order_amt to support larger order values
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
ALTER COLUMN total_order_amt TYPE DECIMAL(18,2);

Verify the updated column type:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 11: SHOW TABLE output confirming total_order_amt widened to DECIMAL(18,2)

Note: Amazon Redshift supports safe type promotions (for example, INT to BIGINT, FLOAT to DOUBLE, DECIMAL(10,2) to DECIMAL(18,2)). Plan column types accordingly for future growth.

Set table properties

Change the compression type for future writes:

Verify the current compression type:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 12: SHOW TABLE output showing the current compression type before the update

Now run the ALTER to change the compression type:

-- Switch to zstd compression for better ratios
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
SET TABLE PROPERTIES ('compression_type'='zstd');

The following SHOW TABLE output confirms the updated compression setting:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 13: SHOW TABLE output showing compression_type set to zstd

Note: This affects only future writes. Existing data files retain their original compression.

Partition evolution

A powerful feature of Iceberg is partition evolution, the ability to change how a table is partitioned without rewriting existing data. Amazon Redshift writes new data with the updated partition scheme, while existing data remains in the old layout. Query engines handle both layouts transparently.

Adding a partition field

The orders table from Part 1 is partitioned by DAY(order_date). Add an additional bucket partition to distribute data across hash buckets:

Verify the current partition spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 14: SHOW TABLE output showing the current partition spec before adding a partition field

Add the partition field:

-- Add bucket partitioning on customer_id
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
ADD PARTITION FIELD bucket(16, customer_id);

After this change, new data is partitioned by both DAY(order_date) and bucket(16, customer_id), while existing data remains in the original day-only layout.

Verify the updated spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 15: SHOW TABLE output showing the updated partition spec with DAY(order_date) and bucket(16, customer_id)

Replacing a partition field

Instead of separately dropping and adding, use REPLACE PARTITION FIELD as a single atomic operation. This is the recommended approach when swapping one transform for another on the same source column, because it makes the intent explicit and avoids a transient state where the table is unpartitioned between operations.

Verify the current partition spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 16: SHOW TABLE output showing the current partition spec with DAY(order_date) and bucket(16, customer_id)

Replace the partition field:

-- Replace daily partitioning with monthly
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
REPLACE PARTITION FIELD DAY(order_date) WITH MONTH(order_date);

After this change:

Existing data remains in day-based partition folders.
Amazon Redshift writes new data into month-based partition folders.
The query engine reads both layouts transparently.

Confirm the new partition spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 17: SHOW TABLE output confirming the partition field replaced with MONTH(order_date)

Insert new data and verify that both partition layouts are queryable:

-- New data follows monthly partitioning
INSERT INTO "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
(order_date, order_id, customer_id, total_order_amt, total_order_tax_amt,
tax_pct, order_created_at_tz, is_active_ind)
VALUES
('2025-01-15', 1018, 3, 210.00, 16.80, 0.08, '2025-01-15 09:00:00-06:00', true);
-- Query spans both old (daily) and new (monthly) layouts transparently
SELECT order_id, order_date, total_order_amt
FROM "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
WHERE order_date >= '2024-11-01'
ORDER BY order_date;

Figure 18: Query results spanning both partition layouts

Converting to a multi-level partition

Iceberg supports multi-level (composite) partition specs, where data is organized by more than one partition field. You can evolve an existing single-level spec into a multi-level spec by adding partition fields one at a time. Each ADD PARTITION FIELD is a lightweight metadata operation, and no data is rewritten.

The orders table is currently partitioned by MONTH(order_date) and bucket(16, customer_id). Add one more partition field to create a three-level spec:

Verify the current partition spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 19: SHOW TABLE output showing the current two-level partition spec of MONTH(order_date) and bucket(16, customer_id)

Add partition field to build the three-level spec:

-- Add a day-level partition field on order_created_at_tz
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
ADD PARTITION FIELD day(order_created_at_tz);

Verify the new multi-level partition spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 20: SHOW TABLE output showing the three-level partition spec of MONTH(order_date), bucket(16, customer_id), and day(order_created_at_tz)

After these changes:

Existing data remains in the original single-level layout (month-based folders).
Amazon Redshift writes new data into the multi-level layout (month, then bucket, then day folders).
The query engine reads both layouts transparently.

Dropping partition fields from a multi-level partition

You can also evolve in the other direction by removing partition fields from a multi-level spec to simplify the partition layout. Like adding fields, dropping a partition field is a metadata-only operation and removes one field per statement.

Verify the current multi-level partition spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 21: SHOW TABLE output showing the three-level partition spec before dropping fields

Drop the partition fields one at a time:

-- Drop the bucket partition field
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
DROP PARTITION FIELD bucket(16, customer_id);
-- Drop the day-level partition field
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
DROP PARTITION FIELD day(order_created_at_tz);

Verify the table is back to its original single-level spec:

SHOW TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders;

Figure 22: SHOW TABLE output confirming the table is back to a single-level MONTH(order_date) partition spec

After dropping a partition field:

Data written under the dropped field’s layout stays in place and remains queryable.
Amazon Redshift writes new data using only the remaining partition fields.
Queries that filtered on the dropped field still work, but they no longer benefit from partition pruning on that field for newly written data.

Supported partition transforms

The following table lists the partition transforms available for Iceberg tables in Amazon Redshift:

Partition transform	Syntax example	What it does
Year	`year(order_date)`	Groups data into yearly partitions based on a date or timestamp column.
Month	`month(order_date)`	Groups data into monthly partitions based on a date or timestamp column.
Day	`day(order_date)`	Groups data into daily partitions based on a date or timestamp column.
Hour	`hour(event_ts)`	Groups data into hourly partitions based on a timestamp column.
Bucket	`bucket(16, customer_id)`	Distributes data across N hash buckets for even distribution on high-cardinality columns.
Truncate	`truncate(3, zip_code)`	Truncates column values to a fixed width W for grouping similar values together.
Identity	`identity(region)`	Partitions by the exact column value with no transformation applied.

Note: A column that is already part of an existing partition field can’t be used in a new partition field. Drop or replace the existing field first.

Accessing S3 Tables with external schemas

Lake Formation resource links provide cross-engine access to your S3 Tables through centralized governance. You create a resource link in the default AWS Glue Data Catalog that points to your S3 Tables database. Amazon Redshift, Amazon Athena, Amazon EMR, and other engines can then discover and query the tables using a single permission model.

Figure 23: S3 Tables integration with AWS Glue Data Catalog and Lake Formation

For the complete setup walkthrough, including Lake Formation prerequisites, resource link creation, and permission grants, see Optimize Amazon S3 Tables queries with Amazon Redshift. For conceptual details on resource links and S3 Tables catalog integration, see About resource links and Creating an S3 Tables catalog.

The following steps show how to query S3 Tables through a resource link after completing the setup from the referenced blog.

Grant access to the resource link

In the Lake Formation console, the resource link appears as a database named iceberg_write_blog_rl (type: Resource link). To grant access to the resource link:

In the Lake Formation console, choose Databases.
Locate iceberg_write_blog_rl (type: Resource link).
Choose Actions, then Grant.
Grant DESCRIBE permission to RedshiftIcebergRole.

Create an external schema

With the resource link in place, create an external schema in Amazon Redshift for two-part notation access.

For IAM federated users:

CREATE EXTERNAL SCHEMA s3tables_iceberg
FROM DATA CATALOG
DATABASE 'iceberg_write_blog_rl'
CATALOG_ID '<ACCOUNT_ID>'
IAM_ROLE 'SESSION';

For database users and business intelligence (BI) tools:

CREATE EXTERNAL SCHEMA s3tables_iceberg
FROM DATA CATALOG
DATABASE 'iceberg_write_blog_rl'
IAM_ROLE 'arn:aws:iam::<ACCOUNT>:role/RedshifticebergRole';

Grant access to specific users or roles:

-- Grant to the IAM role used in this walkthrough
GRANT USAGE ON SCHEMA s3tables_iceberg TO "IAMR:RedshifticebergRole";

Figure 24: S3 Tables available through an external schema

Query with two-part notation

With the external schema created, query S3 Tables using two-part notation:

-- Instead of: "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
SELECT * FROM s3tables_iceberg.orders;

Figure 25: Query results from S3 Tables through an external schema using two-part notation

Access methods comparison

The following table compares the available methods for accessing Iceberg tables in Amazon Redshift:

Access method	Query syntax	Authentication	Best for
S3 Tables three-part notation	`"bucket@s3tablescatalog".namespace.table`	IAM federated identity only	Interactive queries in Query Editor v2 with direct catalog access.
External schema through resource link	`schema_name.table`	Any (IAM role defined in schema)	BI tools, Data API, JDBC/ODBC applications, and shared team access.
awsdatacatalog	`awsdatacatalog.database.table`	IAM federated identity only	Multi-database access in a single session without creating external schemas.

Bringing it together

Combine schema evolution with cross-engine access in a single workflow. The following example adds a column to the orders table and immediately queries it through the external schema:

-- 1. Add a column to the S3 Tables orders table
ALTER TABLE s3tables_iceberg.orders
ADD COLUMN fulfillment_status VARCHAR;
-- 2. Update the new column
UPDATE s3tables_iceberg.orders
SET fulfillment_status = 'shipped'
WHERE order_date < '2024-11-01';
UPDATE s3tables_iceberg.orders
SET fulfillment_status = 'pending'
WHERE order_date >= '2024-11-01';
-- 3. Query immediately via the external schema (no schema recreation needed)
SELECT o.order_id, o.order_date, o.fulfillment_status, c.customer_name
FROM s3tables_iceberg.orders o JOIN demo_iceberg.customer c
ON o.customer_id = c.customer_id
ORDER BY o.order_date DESC;

Figure 26: Cross-catalog join showing the evolved schema immediately visible through the external schema

The new column is visible through both the three-part notation and the external schema without any additional configuration, because the schema evolution in Iceberg propagates automatically.

Best practices

Test ALTER operations in non-production first. While metadata-only, schema changes affect all readers immediately.
Use REPLACE PARTITION FIELD instead of DROP + ADD. The atomic operation avoids a transient unpartitioned state.
Monitor partition spec changes with SHOW TABLE. Verify the current spec after any partition evolution.
Choose partition transforms based on query patterns. Use month() or day() for time-range filters. Use bucket() for high-cardinality join keys.
Set table properties before bulk loads. Change compression type (zstd for better ratios, snappy for speed) before large INSERT operations.
Run table maintenance after mutations. After performing multiple UPDATE, DELETE, or MERGE operations, run AWS Glue table optimizers to compact deletion files and improve read performance.
Use Lake Formation for fine-grained access. Column-level and row-level security can be applied through Lake Formation on tables accessed through resource links.
Grant schema access to specific users or roles. Avoid granting to PUBLIC. Use named IAM roles or database users for least-privilege access.
Monitor query performance. Use Amazon Redshift query monitoring features to track performance of write operations and optimize partitioning strategies as needed.

Considerations

Keep the following in mind when working with ALTER TABLE and partition evolution on Iceberg tables:

Plan for metadata-only behavior. ALTER TABLE operations update metadata instantly, and existing data files remain unchanged. All readers see the new schema immediately after the operation completes.
Drop partition fields before dropping partitioned columns. To remove a column used in the current partition spec, first drop or replace the partition field, then drop the column.
Use safe type promotions for ALTER COLUMN TYPE. Amazon Redshift supports widening within compatible families (INT to BIGINT, FLOAT to DOUBLE, DECIMAL(10,2) to DECIMAL(18,2)). Plan column types with future growth in mind.
Account for mixed partition layouts after evolution. Partition evolution doesn’t re-partition existing data. Old files remain in their original layout, and the query engine reads both layouts transparently.
Use external schemas for database user access. The auto-mounted three-part notation ("bucket@s3tablescatalog") requires IAM federated authentication. For database users and BI tools, create an external schema with an explicit IAM role.
Use full three-part notation with awsdatacatalog. The USE statement isn’t supported with awsdatacatalog, so always specify the full path.
Clean up S3 data separately after dropping tables. Dropping an Iceberg table removes only the catalog entry from AWS Glue Data Catalog. Delete the underlying S3 data files separately, or use AWS Glue table optimizers to remove orphaned files.

Clean up

To avoid ongoing charges, run the following:

-- Drop the fulfillment_status column added during testing
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
DROP COLUMN fulfillment_status;
-- Restore original partition spec (if changed)
ALTER TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
REPLACE PARTITION FIELD MONTH(order_date) WITH DAY(order_date);
-- Drop external schema
DROP SCHEMA IF EXISTS s3tables_iceberg;

Conclusion

In this post, you evolved Apache Iceberg table schemas using ALTER TABLE operations. You added, dropped, and renamed columns, widened data types, changed compression, and evolved partition specs, all as metadata-only operations without rewriting data. You also created Lake Formation resource links to provide governed cross-engine access to S3 Tables, and simplified query syntax with external schemas.

This concludes the three-part series on getting started with Apache Iceberg write support in Amazon Redshift:

Part 1: Create Iceberg tables and perform INSERT operations.
Part 2: Run DELETE, UPDATE, and MERGE for row-level modifications.
Part 3: Evolve schemas with ALTER TABLE and add cross-engine access with Lake Formation resource links.

If you have questions or feedback about this series, leave a comment on this post.

Additional resources

Amazon Redshift Iceberg integration – Complete syntax reference.
Writing to Apache Iceberg tables – Detailed examples.
ALTER TABLE for Iceberg – Full ALTER reference.
Amazon S3 Tables – Managed Iceberg storage.
AWS Lake Formation – Centralized data governance.
Optimize S3 Tables queries with Amazon Redshift – Resource links and performance tuning.

About the authors

Enforce IAM permissions boundaries for Amazon SageMaker Unified Studio Tooling blueprints

Sanjana Sekar — Mon, 28 Sep 2026 15:45:42 +0000

Amazon SageMaker Unified Studio now supports custom permissions boundaries for IAM roles created by the Tooling blueprint. Organizations that enforce Service Control Policies (SCPs) requiring permissions boundaries on all AWS Identity and Access Management (IAM) roles can now adopt Amazon SageMaker Unified Studio without modifying their security posture.

Amazon SageMaker Unified Studio is a unified development environment that brings together data engineering, machine learning, and analytics tools into a single workspace. In Amazon SageMaker Unified Studio, a project is a collaborative workspace that bundles people, tools, and access permissions together. It builds every project from a project profile, which defines a list of blueprints. Blueprints are pre-configured infrastructure templates that provision AWS resources at project creation time or on demand, along with their default parameters. The Tooling blueprint is the only mandatory one. Amazon SageMaker Unified Studio deploys it with every project, creating foundational resources such as the project IAM role and security groups.

In this post, you learn how to create a permissions boundary that restricts AI agent capabilities. You then configure it on the Tooling blueprint using the AWS Command Line Interface (AWS CLI). Finally, you validate that the boundary is enforced on all provisioned roles.

The problem

Enterprises in regulated industries use SCPs to require that every IAM role in an account carries a permissions boundary. A well-scoped boundary prevents privilege escalation and verifies no role exceeds the maximum permissions defined by the organization’s security team. Before this feature, Amazon SageMaker Unified Studio Tooling blueprints created IAM roles without permissions boundaries. When an SCP enforced permissions boundaries, project creation failed with an explicit deny:

User: arn:aws:sts::<account-id>:assumed-role/AmazonSageMakerProvisioning-<account-id>/AmazonDataZoneEnvironmentDeployer-<account-id> is not authorized to perform: iam:CreateRole on resource: arn:aws:iam::<account-id>:role/AmazonBedrockServiceRole-<project-id>-<env-id> with an explicit deny in a service control policy

Amazon SageMaker Unified Studio surfaces the blocked role creation as a Tooling environment provisioning failure, as shown in Figure 1.

Figure 1: Project creation fails when the SCP requires a permissions boundary that is not attached

The project is marked as failed because its Tooling environment couldn’t deploy in the US East (N. Virginia) AWS Region (us-east-1). The details show a 403 permissions error, while the preceding IAM message identifies the underlying iam:CreateRole SCP denial. This blocked adoption for any organization with SCP-enforced permissions boundaries. The AWS CloudFormation event for the Tooling stack exposes the IAM failure behind the project-level error, as shown in Figure 2.

Figure 2: Detailed error showing the SCP denial in the Tooling blueprint AWS CloudFormation stack

The AmazonBedrockServiceRole resource entered CREATE_FAILED because iam:CreateRole was explicitly denied by the SCP, even though AWS CloudFormation surfaced the wrapper error as UnauthorizedTaggingOperation.

Granular control using a permissions boundary: Example use case

Beyond satisfying SCP requirements, permissions boundaries give administrators granular control over what the Tooling blueprint roles can do. For instance, some organizations have SecOps policies that require disabling Data Agent and Data Notebook capabilities across their accounts. These organizations want project members to access data connections and run SQL queries directly, but must block conversational AI, code generation, and notebook cell execution through the agent.

When PermissionsBoundaryArn is configured on the Tooling blueprint, SageMaker Unified Studio attaches the specified customer-managed permissions boundary to all IAM roles provisioned by that blueprint. If your governance requires a boundary, configure it explicitly and verify the resulting roles.

The following permissions boundary policy scopes the roles to the AWS services that Amazon SageMaker Unified Studio uses and then explicitly denies the Amazon DataZone actions that power the AI agent. This permissions boundary is provided for illustrative purposes only and isn’t a recommendation or reference for environment configuration. You should tailor your permissions boundaries to your specific workloads in accordance with the principle of least privilege.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowSmusServiceScope",
      "Effect": "Allow",
      "Action": [
        "datazone:*",
        "sagemaker:*",
        "glue:*",
        "s3:*",
        "lakeformation:*",
        "redshift:*",
        "redshift-data:*",
        "redshift-serverless:*",
        "athena:*",
        "q:*",
        "elasticmapreduce:*",
        "bedrock:*",
        "lambda:*",
        "kms:*",
        "secretsmanager:*",
        "codecommit:*",
        "logs:*",
        "cloudwatch:*",
        "sts:AssumeRole",
        "iam:PassRole",
        "ec2:Describe*",
        "ec2:CreateNetworkInterface",
        "ec2:DeleteNetworkInterface",
        "ec2:CreateNetworkInterfacePermission",
        "ec2:DeleteNetworkInterfacePermission"
      ],
      "Resource": "*"
    },
    {
      "Sid": "DenyDataNotebookAndDataAgent",
      "Effect": "Deny",
      "Action": [
        "datazone:*Notebook*",
        "datazone:*Cell*",
        "datazone:*Conversation*",
        "datazone:SendMessage",
        "datazone:GenerateCode",
        "datazone:CancelMessage"
      ],
      "Resource": "*"
    }
  ]
}

Warning: validate before using in production. This example scopes the roles to the service namespaces Amazon SageMaker Unified Studio uses, but it is still coarse (it allows each listed service in full) and is provided only for illustration. Because a permissions boundary is a ceiling, it must remain a superset of everything the three Tooling roles (datazone_usr_role, AmazonBedrockServiceRole, and AmazonBedrockLambdaExecutionRole) actually need. If Amazon SageMaker Unified Studio adds a dependency that isn’t listed, provisioning or in-console actions will fail with an access denied error. Validate in a non-production domain first.

With this boundary attached, the Tooling blueprint provisions normally, project members can access data connections and run SQL queries. However, any attempt to invoke the AI assistant or execute notebook cells through the agent returns an access denied error. The boundary acts as a ceiling that no policy attached to the role can override.

How it works

The custom permissions boundary feature operates at the blueprint configuration level. An administrator sets a PermissionsBoundaryArn in the Tooling blueprint’s regional parameters. When a user creates a new project that includes the Tooling blueprint, Amazon SageMaker Unified Studio provisions an AWS CloudFormation stack that creates three IAM roles and attaches the specified boundary to each:

datazone_usr_role – the role that all project members assume to access data and resources in that project.
AmazonBedrockServiceRole – for Amazon Bedrock operations.
AmazonBedrockLambdaExecutionRole – for Amazon Bedrock-related AWS Lambda functions.

Because the boundary is set at the blueprint level, it applies to every project created under that blueprint. No per-project configuration is needed.

Prerequisites

Before you begin, make sure that you have:

An AWS account with a Amazon SageMaker Unified Studio Identity Center-based domain created.
The Tooling blueprint enabled in the domain.
AWS CLI v2 configured with administrator access.

If your organization uses AWS Organizations with SCPs that require permissions boundaries, you will also need an organization with the target account as a member and permissions to create and attach SCPs in the management account.

Setting up the SCP (optional)

This section provides instructions to create an SCP and attach it to your AWS Organizations organizational unit or accounts. If your organization already enforces permissions boundaries through SCPs, skip this section. Otherwise, create an SCP in your AWS Organizations management account that denies IAM role creation unless an approved permissions boundary is attached. This also prevents the boundary from being removed, swapped, or weakened afterward:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyRoleWithoutApprovedBoundary",
      "Effect": "Deny",
      "Action": [
        "iam:CreateRole",
        "iam:PutRolePermissionsBoundary"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "iam:PermissionsBoundary": "arn:aws:iam::${aws:PrincipalAccount}:policy/SMUSToolingBoundary"
        }
      }
    },
    {
      "Sid": "DenyRemovingBoundary",
      "Effect": "Deny",
      "Action": "iam:DeleteRolePermissionsBoundary",
      "Resource": "*"
    },
    {
      "Sid": "ProtectBoundaryPolicy",
      "Effect": "Deny",
      "Action": [
        "iam:DeletePolicy",
        "iam:CreatePolicyVersion",
        "iam:SetDefaultPolicyVersion"
      ],
      "Resource": "arn:aws:iam::${aws:PrincipalAccount}:policy/SMUSToolingBoundary"
    }
  ]
}

This policy does three things:

DenyRoleWithoutApprovedBoundary blocks creating a role, or attaching a boundary to an existing role, with anything other than the approved boundary ARN. Denying iam:PutRolePermissionsBoundary in addition to iam:CreateRole stops a privileged principal from swapping in a weaker boundary after the role exists.
DenyRemovingBoundary blocks iam:DeleteRolePermissionsBoundary outright, so the boundary cannot be stripped off. (This action doesn’t support the iam:PermissionsBoundary condition key, so it must be denied unconditionally.)
ProtectBoundaryPolicy prevents tampering with the boundary policy itself. Deleting it, or publishing and defaulting a new version that quietly widens what it allows.

Note: Scope these denies so you don’t lock yourself out. A broad deny on iam:PutRolePermissionsBoundary and iam:DeleteRolePermissionsBoundary also applies to your own administrators. Add an exception for a break-glass or IAM-admin role (for example, an aws:PrincipalArn StringNotLike condition) so a trusted principal can still manage boundaries.

To create the SCP, sign in to the AWS Organizations console with your management account and go to AWS Organizations → Policies → Service control policies. If SCPs aren’t enabled for your organization yet, choose Enable service control policies first. Choose Create policy, give it a name (for example, test_scp), and replace the default content in the policy editor with the JSON above substituting <account-id> with your account ID. Choose Create policy to save it.

After creating the SCP in the management account, verify its content before attaching it. Figure 3 shows the core create-role control. The full example above adds controls that prevent replacing or removing the boundary and modifying the protected policy.

Figure 3: Service Control Policy defined in the AWS Organizations management account

The AWS Organizations Content tab displays the customer-managed test_scp policy. Its visible statement denies iam:CreateRole unless the request uses the SMUSToolingBoundary policy.

Attach this SCP to the organizational unit or account where your Amazon SageMaker Unified Studio domain and domain-associated accounts reside. To do so, open the test_scp service control policy, choose the Targets tab, and choose Attach. The AWS organization structure appears; select the OU or account where the SCP should apply, then choose Attach policy.

Figure 4 identifies the member account that must inherit the SCP in this example organization. The target member account, datazone-account2, resides under OU2, while datazone-account1 is the organization’s management account. Attaching the SCP to the target account or a parent organizational unit enforces it there.

Figure 4: AWS Organizations account structure showing the management account and the target member account

After attaching the policy, verify the association on the SCP’s Targets tab, as shown in Figure 5.

Figure 5: Service Control Policy attached to the target account where it should be enforced

The Targets tab lists datazone-account2 as an ACCOUNT target, confirming that test_scp is enforced directly on the intended member account.

Configuring the permissions boundary

In this section you will execute the required steps to create the permissions boundary and enable it in the Tooling blueprint. The example in this walkthrough uses us-east-1. Change it to the Region where your Amazon SageMaker Unified Studio domain is deployed. You must execute the configuration in the account where you plan to create your project. This can be your Amazon SageMaker Unified Studio domain account or accounts associated to your Amazon SageMaker Unified Studio domain.

Step 1: Create the permissions boundary policy

If you haven’t already created the boundary policy, save the following JSON document as a boundary-policy.json file on your workstation:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowSmusServiceScope",
            "Effect": "Allow",
            "Action": [
                "datazone:*",
                "sagemaker:*",
                "glue:*",
                "s3:*",
                "lakeformation:*",
                "redshift:*",
                "redshift-data:*",
                "redshift-serverless:*",
                "athena:*",
                "q:*",
                "elasticmapreduce:*",
                "bedrock:*",
                "lambda:*",
                "kms:*",
                "secretsmanager:*",
                "codecommit:*",
                "logs:*",
                "cloudwatch:*",
                "sts:AssumeRole",
                "iam:PassRole",
                "ec2:Describe*",
                "ec2:CreateNetworkInterface",
                "ec2:DeleteNetworkInterface",
                "ec2:CreateNetworkInterfacePermission",
                "ec2:DeleteNetworkInterfacePermission",
                "iam:GetRole",
                "sqlworkbench:*"
            ],
            "Resource": "*"
        },
        {
            "Sid": "DenyDataNotebookAndDataAgent",
            "Effect": "Deny",
            "Action": [
                "datazone:*Notebook*",
                "datazone:*Cell*",
                "datazone:*Conversation*",
                "datazone:SendMessage",
                "datazone:GenerateCode",
                "datazone:CancelMessage"
            ],
            "Resource": "*"
        }
    ]
}

As noted previously, this illustrative policy is scoped to the services Amazon SageMaker Unified Studio uses but is still coarse, and its allow list must stay a superset of what all three Tooling roles need.

Then create the policy using the following command:

aws iam create-policy \
--policy-name SMUSToolingBoundary \
--policy-document file://boundary-policy.json \
--description "Permissions boundary for SMUS Tooling roles - denies Data Agent and Data Notebook capabilities"

Note the policy ARN from the output, because it will be used later in the procedure.

Step 2: Retrieve the ID of your domain

Retrieve the ID of your domain by running the following command. Replace <YOUR_DOMAIN_NAME> with the name of your SageMaker Unified Studio domain.

aws datazone list-domains \
--region us-east-1 \
--query "items[?name=='<YOUR_DOMAIN_NAME>'].id | [0]" \
--output text

Note the returned ID, because it will be used later in the procedure.

Step 3: Identify the Tooling blueprint

Retrieve the Tooling blueprint ID by running the following command. Replace <domain-id> with the ID you noted in Step 2.

aws datazone list-environment-blueprints \
  --domain-identifier <domain-id> \
  --managed \
  --region eu-west-1 \
  --query "items[?name=='Tooling'].id" \
  --output json | jq -r '.[0]'

Note the returned ID, because it will be used later in the procedure.

Step 4: Read the current configuration

Retrieve the current Tooling blueprint configuration by executing the following command. Replace <domain-id> with the ID from Step 2 and <tooling-bp-id> with the ID from Step 3.

aws datazone get-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--region us-east-1 | tee tooling-bp-config-backup.json

Important: Back up the output of get-environment-blueprint-configuration before making any changes. The command above pipes the response to tooling-bp-config-backup.json so you have a restore point if you need to revert.

Note the values of provisioningRoleArn, manageAccessRoleArn, enabledRegions, and all fields inside regionalParameters (AZs, S3Location, Subnets, VpcId). You will need all of these in the next step.

Step 5: Set the permissions boundary

Update the blueprint configuration to include PermissionsBoundaryArn in the regional parameters using the following command.

Important: The put-environment-blueprint-configuration API operates in overwrite mode, it replaces the entire configuration with what you provide. You must include all existing values from the previous step’s output. The only new addition is PermissionsBoundaryArn inside the regional parameters. Omitting any existing parameter removes it.

Make sure to replace <domain-id> with the ID you noted in Step 2, <tooling-bp-id> with the ID you noted in Step 3, and all other <placeholder> values with the corresponding values from Step 4’s output.

aws datazone put-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--enabled-regions '<enabledRegions>' \
--provisioning-role-arn "<provisioningRoleArn>" \
--manage-access-role-arn "<manageAccessRoleArn>" \
--regional-parameters '{
  "<region>": {
    "AZs": "<AZs>",
    "S3Location": "<S3Location>",
    "Subnets": "<Subnets>",
    "VpcId": "<VpcId>",
    "PermissionsBoundaryArn": "arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"
  }
}' \
--region <region>

The following anonymized example is based on an existing Tooling blueprint configuration. Its S3Location reflects the bucket naming pattern used in that environment. Copy the exact S3Location returned in Step 4. Don’t use the following illustrative value. Here’s an example:

aws datazone put-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--enabled-regions '["us-east-1"]' \
--provisioning-role-arn "arn:aws:iam::<account-id>:role/service-role/AmazonSageMakerProvisioning-<account-id>" \
--manage-access-role-arn "arn:aws:iam::<account-id>:role/service-role/AmazonSageMakerManageAccess-us-east-1-<domain-id>" \
--regional-parameters '{
  "us-east-1": {
    "AZs": "us-east-1a,us-east-1b,us-east-1c,us-east-1d",
    "S3Location": "s3://amazon-sagemaker-<account-id>-us-east-1-<suffix>",
    "Subnets": "<subnet-1>,<subnet-2>,<subnet-3>,<subnet-4>",
    "VpcId": "<vpc-id>",
    "PermissionsBoundaryArn": "arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"
  }
}' \
--region us-east-1

Step 6: Verify the configuration was applied

Confirm the permissions boundary ARN is now set in the blueprint configuration using the following command. Make sure to replace <domain-id> with the ID you noted in Step 2 and <tooling-bp-id> with the ID you noted in Step 3.

aws datazone get-environment-blueprint-configuration \
--domain-identifier <domain-id> \
--environment-blueprint-identifier <tooling-bp-id> \
--region us-east-1 \
--query "regionalParameters.\"us-east-1\".PermissionsBoundaryArn"

The output should return your boundary policy ARN:

"arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"

Validating the configuration

After configuring the permissions boundary, in this section you will get instructions to create a new project to verify it works end to end and that the IAM roles created with the project actually include the permissions boundary.

Step 1: Select a project profile in enabled state

Use the following command to list project profiles configured in your domain. Make sure to replace <domain-id> with the ID you noted in Step 2 of the “Configuring the permissions boundary” section.

aws datazone list-project-profiles \
--domain-identifier <domain-id> \
--region us-east-1

Choose a project profile that has "status": "ENABLED". Note the id of any project profile returned in the previous command.

Step 2: Create a test project

Create a new project using the following command. Make sure to replace <domain-id> with the ID you noted in Step 2 of the “Configuring the permissions boundary” section and to replace <profile-id> with the project profile ID noted in Step 1 of this section.

aws datazone create-project \
--domain-identifier <domain-id> \
--name "PB-Validation-$(date +%Y%m%d-%H%M%S)" \
--project-profile-id <profile-id> \
--region us-east-1

Note the id (project ID) returned in the response. Wait for the Tooling blueprint to provision. This typically takes a minute or two. After provisioning completes, confirm that the validation project reaches the Active state, as shown in Figure 6.

Figure 6: Project created successfully with the permissions boundary configured

The Projects list shows the timestamped PB-Validation-* project with an Active status, confirming that project creation succeeded with the custom boundary configured.

Step 3: Verify the roles have the boundary attached

In this section you check that the IAM roles created with the project have the permissions boundary attached. Use the following commands to get the configuration for the IAM roles created with the project you just created. Replace <domain-id> and <project-id> with the values from the previous steps.

# Get the environment ID
ENV_ID=$(aws datazone list-environments \
--domain-identifier <domain-id> \
--project-identifier <project-id> \
--region us-east-1 \
--query "items[?name=='Tooling'].id" --output text)

# List IAM roles in the AWS CloudFormation stack
aws cloudformation describe-stack-resources \
--stack-name "DataZone-Env-${ENV_ID}" \
--region us-east-1 \
--query "StackResources[?ResourceType=='AWS::IAM::Role'].PhysicalResourceId" \
--output table

# Verify each role has the boundary
aws iam get-role \
--role-name "<role-name>" \
--query 'Role.PermissionsBoundary'

All three roles should return a response showing the permissions boundary ARN:

{
  "PermissionsBoundaryType": "Policy",
  "PermissionsBoundaryArn": "arn:aws:iam::<account-id>:policy/SMUSToolingBoundary"
}

You can also verify each role in the IAM console. Figure 7 shows the permissions boundary for the project user role.

Figure 7: IAM console showing the permissions boundary attached to the datazone_usr_role

The datazone_usr_role Permissions tab displays SMUSToolingBoundary as its customer-managed permissions boundary.

Figure 8 confirms that the same boundary is attached to the Amazon Bedrock service role.

Figure 8: IAM console showing the permissions boundary attached to the AmazonBedrockServiceRole

The AmazonBedrockServiceRole also displays SMUSToolingBoundary as its customer-managed permissions boundary.

Figure 9 verifies the boundary on the third Tooling role, the Bedrock Lambda execution role.

Figure 9: IAM console showing the permissions boundary attached to the AmazonBedrockLambdaExecutionRole

The AmazonBedrockLambdaExecutionRole likewise displays SMUSToolingBoundary, confirming that all three provisioned roles carry the boundary.

Step 4: Verify the boundary denies AI agent actions

In this section you verify the boundary actually denies AI agent actions. If you configured the boundary from the use case section earlier, the boundary blocks Data Notebooks and messages to the Data Agent, such as the Query Editor assistant. Any such attempt returns an access denied error. The project user role has the boundary attached, so even if the role’s identity policies grant the relevant APIs, the boundary’s explicit deny takes precedence.

To confirm, navigate to your project in SageMaker Unified Studio and test the following actions:

Attempt to create a notebook – In the left sidebar, select Notebooks. Select Create notebook. The operation will fail because the permissions boundary prevents the datazone:CreateNotebook action (Figure 10).

Figure 10: Permissions boundary preventing creation of Data Notebooks

After the create action, Amazon SageMaker Unified Studio reports Failed to create notebook and identifies datazone:CreateNotebook as explicitly denied by SMUSToolingBoundary.

Attempt to use Data Agent in the Query Editor – In the left sidebar, select Query Editor, then select the Chat with AI icon. The agent chat will fail to load because the permissions boundary blocks the APIs required by Data Agent (Figure 11).

Figure 11: Permissions boundary preventing using Data Agent on Query Editor

The Query Editor remains available, but the Agent panel reports “You don’t have access to Data Agent“. In this configured test, that message is the user-visible result of denying the Data Agent APIs. The screenshot itself doesn’t display the denied API or boundary ARN.

Important considerations

Immutable after project creation – The permissions boundary is set at provisioning time. Changing the boundary ARN on the blueprint configuration only affects new projects. Existing projects retain their original boundary.
Applies to all Tooling-provisioned roles – When PermissionsBoundaryArn is configured on the Tooling blueprint, SageMaker Unified Studio attaches the specified customer-managed permissions boundary to all three IAM roles created by that blueprint. It’s applied uniformly — you can’t selectively apply it to individual roles. No boundary is attached unless you configure one, so if your governance requires a boundary, set it explicitly and verify the resulting roles rather than assuming one is present by default.
Policy must exist – The IAM policy referenced by PermissionsBoundaryArn must exist in the account before project creation. If the policy is deleted or the ARN is invalid, provisioning will fail.
Tooling blueprint only – Among Amazon SageMaker Unified Studio provided blueprints, only the Tooling blueprint supports custom permissions boundaries. Other provided blueprints that create IAM roles (for example, the EmrOnEc2 blueprint) don’t currently support this feature. If your organization requires permissions boundaries on roles created by additional blueprints, you can build custom blueprints that include a permissions boundary configuration so you can extend this security control across your entire project infrastructure.

Clean up

To remove test resources, delete the test project from the SageMaker Unified Studio UI. On the project’s Overview page, choose the ⋮ (more actions) menu in the top-right and choose Delete project.

Figure 12: Deleting the test project from the project Overview page.

In the Delete project dialog, type confirm in the text box to acknowledge that the action is final, then choose Delete project. This permanently deletes the project and its underlying resources, and triggers an asynchronous AWS CloudFormation stack deletion.

Figure 13: Confirming project deletion.

To remove the boundary from future projects, re-run the put-environment-blueprint-configuration command from Step 5: Set the permissions boundary, but omit the PermissionsBoundaryArn field from the regional parameters. Because you backed up the original configuration in Step 4: Read the current configuration (tooling-bp-config-backup.json), you can reuse the exact same provisioningRoleArn, manageAccessRoleArn, enabledRegions, and regionalParameters values (AZs, S3Location, Subnets, VpcId) — just without PermissionsBoundaryArn — so the blueprint returns to provisioning roles with no permissions boundary.

Conclusion

With the custom permissions boundary feature for Amazon SageMaker Unified Studio, organizations can adopt Amazon SageMaker Unified Studio Tooling blueprints without compromising their IAM governance posture. By configuring a single parameter on the Tooling blueprint, all IAM roles provisioned by future projects automatically carry the specified permissions boundary. This satisfies SCPs that mandate a boundary on every role and gives administrators granular control over what the Tooling roles can do, for example disabling AI agent and notebook capabilities. Remember that the example boundary in this post is illustrative, because it scopes to the services SageMaker Unified Studio uses but is still coarse.

“I just updated the EnvironmentBlueprintConfiguration for the Tooling blueprint to include the new PermissionsBoundaryArn param. After that the blueprint provisioned successfully with the required permissions boundary attached to all the IAM roles, in line with our security policies. In the end it was a one-line change.”

— Nat Noordanus, Data Tech Lead at Nexthink

To get started, create your permissions boundary policy, configure it on the Tooling blueprint using the CLI, and create a project to verify the boundary is attached.

For more information, see the documentation for Amazon SageMaker Unified Studio, IAM permissions boundaries, and Service Control Policies.

About the authors

How Delivery Hero rebuilt real-time ad measurement with Apache Flink

Kirill Tishenkov — Thu, 24 Sep 2026 16:09:12 +0000

This post is co-written with Kirill Tishenkov, Alexandru Pisarenco, Upendra Kambhampati, and Sabariesh Ganesan from Delivery Hero.

Real-time ad measurement is one of the harder streaming problems in advertising. Every impression and click has to be accurate enough to bill a vendor for, and fresh enough for the ad server to act on. In this post, we describe how Delivery Hero moved its ad measurement pipeline from hourly batch processing to real time on Amazon Managed Service for Apache Flink. Delivery Hero, based in Berlin, Germany, is one of the world’s leading local delivery platforms, operating across Asia, Europe, Latin America, the Middle East, and North Africa. Working with more than 1.5 million restaurant partners and local vendors in around 65 countries, Delivery Hero handles millions of orders for food, groceries, and everyday essentials daily.

At the center of Delivery Hero’s business sits an advertising platform that connects vendors and brands with millions of active consumers. The platform handles tens of thousands of messages per second and processes billions of ad events per day, supporting an advertising revenue stream that reached almost EUR 1.5 billion in 2025. Every impression served and every click recorded must satisfy two requirements at once. The data must be accurate enough to bill vendors fairly, and fresh enough for the ad server to act on in real time. Delivery Hero replaced its batch-oriented measurement system with a fully real-time pipeline built on Amazon Managed Service for Apache Flink. The new pipeline cut infrastructure costs by more than half and reached a level of data quality the previous system could not.

Challenges with the legacy system

The legacy ads measurement system consumed impression, click, and order events from message queues. It enriched them through synchronous API calls for campaign metadata and product lookups, then wrote hourly aggregated metrics to a reporting database. This design worked at a modest scale, but five structural problems emerged as traffic grew.

No event-time semantics, and slow processing. The pipeline bucketed events by the time it processed them rather than the time they occurred, because most events arrived without a usable event timestamp. Results were internally consistent, but they skewed whenever ingestion lagged or events arrived out of order. That widened the error bar on every time-sensitive metric, including return on ad spend (ROAS). The bigger cost was speed. Metrics were assembled in hourly batches, so the average gap between when an event occurred and when it was recorded was 61 minutes. The platform was reacting to clicks and impressions up to an hour after the fact, far too late for budget pacing or ad serving.

Synchronous enrichment capped how far the system could scale. Enrichment is the step that attaches business context to a raw ad event: which campaign it belongs to, which vendor owns it, and which product was advertised. In the legacy system, every event triggered a chain of blocking external API calls to fetch that context. During traffic spikes, such as a flash sale or a back-to-school surge, exhausted connection pools cascaded into billing, ad serving, and reporting simultaneously. There was no back-pressure mechanism and no way to scale enrichment independently of event ingestion.

The database behind the pipeline was built for a very different access pattern. The pipeline kept its working data in a NoSQL document database: deduplication keys, attribution history, and running totals. The platform inherited that database from its pre-streaming era, when ad measurement looked like document storage and retrieval. The workload then evolved into continuous deduplication, multi-day attribution lookups, and rolling aggregation. Every event ended up triggering a full document read and write against a database designed for occasional access, not per-event mutation. Read/write amplification stored far more data than the logic needed, every write triggered index updates and collection scans, and storage costs grew in lockstep with query latency. At peak load, this often tipped into production outages.

Reprocessing was a project, not a capability. Recovery from a bug, a traffic spike, or a corrupted upstream batch required different tooling for every consuming system. Billing replay was a hand-rolled combination of Google Cloud BigQuery tables, Pub/Sub topics, and custom CLI scripts. Reporting replay ran as a separate daily Airflow job with a one-hour-per-day cost and a six-month horizon. Campaigns and credits events had no replay path at all. Every recovery was a coordination exercise across teams. Every event type that could not be replayed was a class of problems that could only be patched manually after the fact.

Incomplete event context corrupted downstream data quality. Enrichment was synchronous and best-effort, so the pipeline still wrote through events that failed a lookup or arrived malformed, leaving their fields blank. The pipeline had no mechanism to recover the missing context later. Three gaps mattered most:

Missing session rate: the share of events that landed without a usable session ID, leaving the interaction unattached to the user browsing session it belonged to. At 30–40 percent, roughly a third of all events could not be tied back to a session, breaking any session-scoped analysis or feature.
Missing customer identifiers (IDs): the share of events with no customer ID, severing the link between an ad interaction and the customer who generated it and weakening attribution and personalization.
Missing impression timestamps: the share of impression events lacking a reliable event-time timestamp (the same root cause as the processing-time fallback described earlier). At 91 percent, most impressions had no trustworthy event time, forcing the processing-time approximation and widening the error bar on every time-based metric.

These omissions propagated silently into the reporting metrics and into the session-scoped features consumed by machine learning (ML) models for campaign ranking, conversion-rate estimation, and anomaly detection.

Why Amazon Managed Service for Apache Flink

The team set three non-negotiable requirements. First, fault-tolerant data processing, to eliminate data loss. Second, stateful stream processing that could hold multiple days of interaction history in low-cost, low-latency storage. Third, fully managed infrastructure, so engineers could focus on application logic rather than cluster operations.

The team selected Apache Flink because it satisfies all three requirements natively, without bolting on external systems. Its event-time watermark model helps place out-of-order events in the correct time window even when they arrive late. Its RocksDB state backend holds large keyed state on disk without Java Virtual Machine (JVM) heap pressure.

The team chose Amazon Managed Service for Apache Flink over self-hosted Flink on Amazon Elastic Kubernetes Service (Amazon EKS) to eliminate the operational burden of managing JobManagers, TaskManagers, and checkpoint storage. Amazon Kinesis Data Streams serves as the upstream event bus, with two streams: one for user event actions (impressions and clicks) and one for orders. The team chose Kinesis Data Streams over Amazon Managed Streaming for Apache Kafka (Amazon MSK) for cost efficiency at this topology.

Amazon DynamoDB holds campaign and product reference data, queried through Flink’s Async I/O API to enrich events without blocking the processing pipeline. AWS Secrets Manager stores ad event decryption keys, retrieved once at job startup. Amazon Simple Storage Service (Amazon S3) stores granular event logs in Avro format and serves as the incremental checkpoint store for Flink state. Amazon EventBridge Pipes bridged Amazon Simple Queue Service (Amazon SQS) to Kinesis in the minimum viable product (MVP) phase without any custom code, cutting time-to-production by two weeks.

Solution architecture

The following diagram shows the end-to-end pipeline.

Figure 1: End-to-end architecture of the real-time ad measurement pipeline

Two Amazon Simple Notification Service (Amazon SNS) topics ingest events: one receives user event actions (compressed, encrypted ad tokens containing campaign, vendor, and placement metadata), the other receives order events. Amazon SQS buffers both before Amazon EventBridge Pipes (MVP) or an AWS Fargate service (production) forwards them into Kinesis.

Amazon Managed Service for Apache Flink runs a five-stage Java pipeline:

Decompress and decrypt. The pipeline decrypts the ad event token using keys from AWS Secrets Manager.
Deduplicate. The pipeline keys events on a composite of entity, ad, event, and customer identifiers. Flink’s RocksDB state tracks seen events over a 30-hour window (approximately 20 GB of state), filtering duplicates while preserving them in Amazon S3 for audit.
Enrich. Flink’s Async I/O API queries Amazon DynamoDB concurrently for campaign metadata and product master codes, populated continuously from upstream Kafka topics by an AWS Fargate consumer.
Attribute. A multi-day keyed interval join matches user event actions to subsequent orders on entity, customer, vendor, and campaign dimensions (approximately 100 GB of state). This stage emits attributed orders to Amazon S3.
Aggregate. The pipeline accumulates impression, click, order, revenue, and ad spend metrics in RocksDB state, then batch-upserts them to the reporting database every 5 minutes.

The pipeline emits billing events (cost per mille (CPM) impressions and valid cost per click (CPC) clicks) to Apache Kafka topics. The ad server and budget service consume those topics in real time. Flink checkpoints all state incrementally to Amazon S3, so the job restores from the last checkpoint after a failure. Kinesis Data Streams and the upstream sources deliver at-least-once, and the deduplication stage in step 2 drops any event replayed during recovery. Billing is therefore effectively exactly-once, even though the transport underneath it is at-least-once.

Results and impact

The redesigned architecture achieved quantifiable performance gains across data fidelity, processing throughput, and operational expenditure, while introducing capabilities that were not feasible under the legacy model.

Processing latency: From hourly windows to real time

The average gap between when an event was published and when it was recorded dropped from 61 minutes to 1.2 seconds. Budget pacing and aggregated metrics now reflect activity within seconds rather than the following hour. Downstream ad serving and budget pacing systems act on real-time signals instead of reconciling after the fact.

Cost efficiency

The migration reduced monthly operational costs by approximately 57 percent, which more than halves the annual run rate for the pipeline. The saving came alongside stronger reliability, not at its expense.

System reliability

Durable attribution window. The multi-day attribution window lives in RocksDB-backed keyed state, roughly 100 GB on local TaskManager disks, checkpointed incrementally to Amazon S3. Per-key lookups stay in the low-millisecond range regardless of state size, and a crash or shard rebalance restores state from the last checkpoint rather than triggering a reconciliation job.

Elasticity replacing fragility. Async I/O against DynamoDB removed the synchronous enrichment chain that previously gated every event. The pipeline sustains 20,000 messages per second at peak without back-pressure leaking into ad serving or billing, and enrichment scales independently of ingestion. Flash sales and seasonal surges no longer threaten upstream systems.

Replayable history. The pipeline persists every raw event to Amazon S3 in Avro format the moment it lands, and Kinesis Data Streams retains the source stream for up to 7 days. When a logic bug surfaces or a downstream contract changes, the team reprocesses the affected time range deterministically against the original inputs. There is no bespoke backfill job and no reconciliation against external systems. Past data is a first-class input, not a frozen artifact.

Data quality at the source

The following table compares the three data quality gaps before and after the migration.

Metric	Before	After
Missing session rate	30–40%	0%
Missing customer IDs	5%	0.8%
Missing impression timestamps	91%	0.2%

Downstream applications now receive fully enriched transactional and session context. Machine learning models use session-scoped features for campaign ranking, conversion-rate estimation, and anomaly detection. The pipeline now computes those features from a complete event stream, rather than one in which roughly a third of events were missing session context and 91 percent of impressions were missing a reliable timestamp.

What’s next

The pipeline described here is the first of several planned migrations to Amazon Managed Service for Apache Flink. The team is extending the same architecture to additional ad formats, and connecting real-time Flink aggregations directly to the ad serving layer for sub-second budget pacing. The real-time data layer built for measurement also serves as the foundation for AI-driven use cases. The team plans to explore live user interaction streams feeding personalization ranking models and grounded large language model (LLM) recommendations, which were impractical with batch-oriented infrastructure.

Conclusion

Delivery Hero’s migration to Amazon Managed Service for Apache Flink shows that effectively exactly-once billing, multi-day stateful attribution, and manageable operational complexity are not competing goals. The combination that made it work: Kinesis Data Streams for ingestion, DynamoDB for low-latency enrichment, Amazon S3 for event storage and checkpointing, and Amazon EventBridge Pipes for rapid MVP delivery. Together they produced a system that is more accurate, more resilient, and less expensive than the one it replaced. For advertising platforms where billing accuracy and attribution correctness are commercial imperatives, this architecture offers a replicable path from batch approximation to real-time measurement.

To get started with Apache Flink on AWS, see the Amazon Managed Service for Apache Flink Developer Guide.

Additional resources

About the authors

Configure domain-level VPC networking in Amazon SageMaker Unified Studio

Prasad Nadig — Wed, 23 Sep 2026 18:36:56 +0000

Enterprise operations teams that run domain-level VPC networking in Amazon SageMaker Unified Studio often support dozens of projects spanning data engineering, analytics, and machine learning (ML) teams. Each project requires private connectivity to internal databases, Amazon Simple Storage Service (Amazon S3) buckets, and AWS services. Without a domain-level Amazon Virtual Private Cloud (Amazon VPC) configuration, project owners coordinate with the networking team individually. This piecemeal approach leads to inconsistent subnet choices, missing VPC endpoints, connectivity failures that are hard to troubleshoot, and a network posture that is difficult to audit.

With domain-level VPC networking, you configure the network once, and all new projects get the right network immediately upon creation. In this post, you learn how to:

Configure SageMaker Unified Studio domain-level VPC networking.
Select subnets and security groups that provide multi-Availability Zone (multi-AZ) resilience.
Update projects that have no VPC to inherit the domain VPC, and understand when a project must be recreated instead.
Validate network connectivity from within a project.

In this post, you learn how to configure VPC networking for a SageMaker Unified Studio domain that uses AWS Identity and Access Management (IAM)-based authentication. You see how network components map to domain and project resources, and how to plan a configuration that balances security, connectivity, and operational simplicity.

Solution overview

Domain-level VPC networking provides a single network configuration that applies to all new projects in the domain. Projects automatically inherit the VPC settings, including subnets, security groups, and connectivity to AWS services through VPC endpoints. Existing projects are an exception and are handled separately (see Step 3).

The following diagram shows a single VPC with private subnets across two Availability Zones configured at the domain level, with data engineering, analytics, and ML projects all inheriting that configuration.

Figure 1: Domain-level VPC configuration in Amazon SageMaker Unified Studio. A single VPC with private subnets across two Availability Zones is configured at the domain level. All projects (data engineering, analytics, ML) inherit this configuration automatically

Key benefits of this approach:

Configure once, apply across projects: New projects inherit the domain VPC without manual intervention.
Consistent security posture: A single network boundary covers all data, analytics, and ML workloads.
Simplified auditing: One VPC to audit rather than one per project. Turn on VPC Flow Logs and review AWS CloudTrail events for network-level auditing.
Reduced operational overhead: Project teams start working immediately without submitting networking requests.

The following AWS services are used in this solution:

Amazon SageMaker Unified Studio – The unified development environment for data, analytics, and AI.
Amazon Virtual Private Cloud (Amazon VPC) – Provides network isolation with private subnets and VPC endpoints.
IAM – Controls domain administrator permissions.

Prerequisites

Before configuring domain-level VPC networking, verify you have the following:

Domain administrator permissions for Amazon SageMaker Unified Studio.
An existing VPC with the following requirements:
- At least two private subnets in different Availability Zones.
- DNS hostnames and DNS support enabled.
- At least five available IP addresses per expected Amazon SageMaker Unified Studio project. This is a baseline minimum. Workloads using AWS Glue, Amazon EMR, or Amazon Redshift Serverless consume additional elastic network interfaces (ENIs) per worker or node. We recommend /24 or larger subnets for production domains and forward-looking capacity planning based on your expected users and compute types. For detailed guidance, see How to set up a network-isolated VPC for Amazon SageMaker Unified Studio.
VPC endpoints configured for the AWS services your projects access (for example, Amazon S3, AWS Glue, Amazon SageMaker AI).
Private DNS enabled on all interface VPC endpoints (you must enable this so that service DNS names resolve to private IPs). If you use centralized VPC endpoints through AWS Resource Access Manager (AWS RAM) or AWS Transit Gateway, configure Amazon Route 53 Resolver inbound endpoints instead.
S3 gateway endpoint route table associations configured for all selected private subnets (without this, S3 access fails in subnets whose route table lacks the prefix-list route).
A security group (optional), if not provided, SageMaker Unified Studio creates one automatically.
The SageMakerStudioAdminIAMConsolePolicy managed policy (or equivalent permissions including ec2:Describe*, ec2:CreateSecurityGroup, and datazone:* actions) attached to the domain administrator IAM role. See SageMakerStudioAdminIAMConsolePolicy in the AWS Managed Policy Reference for the full permission set.

Note: The VPC must be in the same AWS Region as the domain.

For detailed guidance on VPC networking configuration, see Configure VPC networking for IAM-based domains in the SageMaker Unified Studio Administrator Guide.

VPC endpoint requirements

Because your subnets are private (no internet gateway route), compute resources access AWS services through VPC endpoints. At a minimum, configure the following interface and gateway endpoints (add Amazon Athena, AWS Lake Formation, or Amazon Redshift endpoints if you use them in your projects):

Endpoint	Type	Purpose
com.amazonaws.region.s3	Gateway	S3 access for data storage
com.amazonaws.region.glue	Interface	AWS Glue job connectivity
com.amazonaws.region.sagemaker.api	Interface	SageMaker API calls
com.amazonaws.region.sagemaker.runtime	Interface	Model inference
com.amazonaws.region.logs	Interface	Amazon CloudWatch Logs
com.amazonaws.region.monitoring	Interface	Amazon CloudWatch metrics
com.amazonaws.region.sts	Interface	IAM role assumption
com.amazonaws.region.datazone	Interface	Amazon SageMaker Unified Studio service connectivity
com.amazonaws.region.ecr.api	Interface	ECR API calls (container image metadata)
com.amazonaws.region.ecr.dkr	Interface	ECR image layer pulls (Docker registry)
com.amazonaws.region.kms	Interface	AWS Key Management Service (AWS KMS) encryption/decryption operations

Note: Interface endpoints incur an hourly charge per Availability Zone plus data processing fees. Gateway endpoints (such as S3) have no hourly charge. Factor endpoint count and AZ spread into your cost estimate.

For a comprehensive list of all mandatory and optional VPC endpoints for a fully network-isolated setup, see How to set up a network-isolated VPC for Amazon SageMaker Unified Studio. For current pricing details, see AWS PrivateLink pricing.

Note: Review your account’s service quotas for interface VPC endpoints per VPC (default 50) and ENIs per Region before scaling. Request increases through Service Quotas if needed.

Solution walkthrough

The following steps walk you through configuring the domain VPC and validating it, from signing in to the console through confirming private connectivity from a project.

Sign in to the AWS Management Console as your Amazon SageMaker Unified Studio domain administrator (the IAM role designated as the domain login role).
Open the Amazon SageMaker console.
Use the Region selector in the top navigation bar to select the Region where your domain exists.
On the Amazon SageMaker Unified Studio landing page, choose Open to launch your IAM-based domain.

The following screenshot shows the Amazon SageMaker Unified Studio landing page, where you choose Open to launch the domain.

Figure 2: Amazon SageMaker Unified Studio landing page with the Open button to launch the IAM-based domain

From the navigation pane, choose Domain management.

The following screenshot shows Domain management in the navigation pane.

Figure 3: Domain management on navigation pane

Note: Access to the domain administration page is restricted to the IAM role specified as the domain login role during domain creation.

Step 2: Add VPC configuration

In the navigation pane, choose Settings. In the Networking in this account section, choose Add VPC.

The following screenshot shows the Networking in this account section with the Add VPC button.

Figure 4: Domain management Settings page showing the Networking in this account section to add a VPC

For VPC, select the VPC with connectivity to your compute, database, and storage resources. If no VPC exists, choose Create VPC to provision one using AWS CloudFormation.
For Subnets, select a minimum of two private subnets in different Availability Zones.
(Optional) For Security group, select a security group to control inbound and outbound traffic. If you don’t choose one, SageMaker Unified Studio creates one automatically.
Choose Save.
Verify the VPC configuration status shows Ready in the Networking in this account section.

The following screenshots show the Add VPC dialog and the resulting Ready status in the Networking in this account section.

Figure 5: Add VPC dialog with fields for VPC, subnets, and security group selection

Figure 6: VPC configuration status showing Ready in the Networking in this account section

Note: IAM-based domains support only one VPC configuration at a time. AWS IAM Identity Center-based domains can have a VPC per Region. For details, see Configure VPC networking for IAM-based domains in the SageMaker Unified Studio Administrator Guide.

New projects created in the domain now automatically use the saved VPC configuration. Existing projects are an exception. See Step 3 to update them.

Step 3: Update existing projects

Existing projects don’t automatically inherit the domain VPC configuration. How you apply the new settings depends on the project’s current state:

Projects with no VPC configured – Update in place to adopt the domain VPC. See the following steps.

Projects that already have a VPC – These can’t be switched to a different VPC configuration. To adopt the domain VPC:

Create a new project (which inherits the domain VPC automatically).
Recreate connections in the new project.
Migrate assets from the old project.
Back up any data you need, then delete the original project.

Because recreation can disrupt in-progress work and doesn’t migrate project data automatically, schedule this as a planned maintenance window.

To update a project that currently has no VPC configured:

From the domain administration page, choose Projects in the navigation pane.
Choose the project you want to update.
On the project detail page, a banner appears: “Configurations have changed. Please update this project to access the latest configuration.”
In the banner, choose Update.
Confirm the update when prompted.

Repeat this process for each existing project that should use the domain VPC. The following screenshot shows the project detail page with the configuration update banner.

Figure 7: Project detail page showing the configuration update banner

Step 4: Validate connectivity

After configuring the domain VPC and updating your projects, verify connectivity. Compute resources should have private connectivity to AWS services through the VPC, without any additional project-level network configuration.

Create a notebook in one of your projects as shown in the following figure and run the following code:

Figure 8: Creating a notebook in a SageMaker Unified Studio project to validate VPC connectivity

Requirements: Python 3.8+, Boto3 1.26 or later. Run in a notebook within your SageMaker Unified Studio project.

import boto3
import socket
import ipaddress

def validate_vpc_connectivity():
    """Validate that the project has private connectivity to AWS services
    through the domain-level VPC configuration."""

    results = {}
    region = boto3.session.Session().region_name
    if not region:
        raise RuntimeError('Could not determine AWS Region. Run this notebook inside a SageMaker Unified Studio project.')

    # Test Amazon S3 access via VPC endpoint
    try:
        s3 = boto3.client('s3')
        response = s3.list_buckets()
        results['S3'] = f"[PASS] Accessible ({len(response['Buckets'])} buckets)"
    except Exception as e:
        results['S3'] = f"[FAIL] Failed: {e}"

    # Test AWS Glue access via VPC endpoint
    try:
        glue = boto3.client('glue')
        dbs = glue.get_databases()
        results['Glue'] = f"[PASS] Accessible ({len(dbs['DatabaseList'])} databases)"
    except Exception as e:
        results['Glue'] = f"[FAIL] Failed: {e}"

    # Test STS (role assumption through VPC endpoint)
    try:
        sts = boto3.client('sts')
        identity = sts.get_caller_identity()
        results['STS'] = f"[PASS] Accessible (Account: {identity['Account']})"
    except Exception as e:
        results['STS'] = f"[FAIL] Failed: {e}"

    # Verify interface endpoint resolves to private IP
    try:
        sts_endpoint = f"sts.{region}.amazonaws.com"
        addr_info = socket.getaddrinfo(sts_endpoint, 443, family=socket.AF_INET)
        ip = addr_info[0][4][0]
        is_private = ipaddress.ip_address(ip).is_private
        if is_private:
            results['DNS Resolution'] = f"[PASS] Private IP ({ip}) (traffic stays on AWS network)"
        else:
            results['DNS Resolution'] = f"[WARN] Public IP ({ip}) - check VPC endpoint config"
    except Exception as e:
        results['DNS Resolution'] = f"[FAIL] Failed: {e}"

    # Print results
    print("-" * 40)
    print("Domain VPC Connectivity Validation")
    print("-" * 40)
    for service, status in results.items():
        print(f" {service}: {status}")
    print("-" * 40)
    print(f"\n Region: {region}")

    # Check if all tests passed
    all_passed = all("[PASS]" in status for status in results.values())
    has_warn = any("[WARN]" in status for status in results.values())
    if all_passed:
        print(f"\n [PASS] All services accessible via private VPC endpoints.")
        print(f" This project inherited its network configuration")
        print(f" from the domain without per-project setup.")
    elif has_warn and all("[PASS]" in s or "[WARN]" in s for s in results.values()):
        print(f"\n [WARN] Services are reachable, but DNS resolves to public IPs.")
        print(f" Verify that Private DNS is enabled on your interface VPC endpoints.")
    else:
        print(f"\n [FAIL] Some services are not reachable.")
        print(f" Check that VPC endpoints are configured and security")
        print(f" groups allow outbound traffic on port 443.")

validate_vpc_connectivity()

Expected output when VPC is correctly configured:

Figure 9: Successful validation output showing all services accessible through private VPC endpoints

If any service shows a failure, one common cause is security groups preventing traffic on port 443 to the VPC endpoint. Other causes include missing VPC endpoints, incorrect route table entries, or DNS resolution issues. For more information, see Configure VPC networking for IAM-based domains in the SageMaker Unified Studio Administrator Guide.

Note: An AccessDenied error indicates the request reached the service. Connectivity is working, but IAM permissions need adjustment (for example, the S3 test requires s3:ListAllMyBuckets, which some project roles lack). A timeout or connection error points to a networking problem (missing endpoint, route, or security group rule). The following screenshot shows the validation output when VPC endpoints are missing, where the affected services report timeout errors.

Figure 10: Validation output when VPC endpoints are not configured. Timeout errors indicate missing endpoints

The security group applied at the domain level controls network access for all projects. To review or tighten the rules:

Navigate to the Amazon VPC console.
Choose Security groups and choose the security group shown in your domain’s Networking settings.
Review the Inbound rules and Outbound rules tabs.

By default, the auto-created security group allows all outbound traffic on port 443 (HTTPS) to reach AWS services through VPC endpoints. Consider restricting outbound rules to only the specific VPC endpoint security groups for least-privilege access. Additionally, make sure your VPC endpoint security groups allow inbound TCP 443 from the domain security group or subnet CIDRs. For distributed compute services (AWS Glue, Amazon EMR), add a self-referencing inbound rule to allow worker-to-worker communication.

Updating VPC configuration

After the initial setup, you can modify the VPC configuration to change the VPC, subnets, or security group:

From the domain administration page, choose Settings in the navigation pane.
In the Networking in this account section, under the Actions column, choose Update.
Update the VPC, subnets, or security group as needed.
Choose Update.

The following screenshot shows the Update VPC dialog, where you modify the VPC, subnets, or security group.

Figure 11: Update VPC dialog showing the option to modify VPC, subnets, or security group for the domain

Important: Updating the VPC does not affect already provisioned resources. Newly created resources in projects use the updated VPC. Existing projects that already have a VPC keep their original settings and must be recreated to adopt the change. Projects with no VPC can be updated in place (see Step 3).

Clean up

To remove the VPC configuration from your domain:

From the domain administration page, choose Settings in the navigation pane.
In the Networking in this account section, choose the Actions menu (⋮) and choose Remove.

The following screenshot shows the Actions menu with the Remove option.

Figure 12: Actions menu in the Networking in this account section showing the Remove option

If you created a dedicated VPC for this walkthrough and no longer need it:

Delete the VPC and associated resources (subnets, VPC endpoints, security groups) from the Amazon VPC console. Before deleting, remove the domain VPC configuration and make sure all project resources are terminated. Active projects create ENIs that block VPC and subnet deletion.
If you used an AWS CloudFormation template to create the VPC, delete the stack to remove all resources cleanly. Open the AWS CloudFormation console and delete the stack.

Note: Removing the domain VPC configuration does not retroactively change projects that already have VPC applied. Those projects retain their existing network configuration. New projects created after removal do not have a VPC configured.

Conclusion

In this post, we showed how to configure domain-level VPC networking in Amazon SageMaker Unified Studio. A single domain-level VPC eliminates per-project networking overhead, enforces a consistent security posture, and simplifies compliance auditing.

Key takeaways:

Domain-level VPC is a one-time configuration that automatically applies to all new projects.
Projects with no VPC can be updated in place. Projects that already have a VPC must be recreated to adopt a changed configuration.
Private subnets with VPC endpoints provide secure, private connectivity to AWS services without traversing the public internet.

As next steps, consider:

Reviewing your auto-created security group rules and tightening them for least-privilege access.
Adding VPC endpoints for additional AWS services as your projects’ needs evolve.
Monitoring subnet IP address utilization to plan capacity as you add more projects. Use the AvailableIpAddressCount Amazon CloudWatch metric for your subnets to track utilization and set alarms.

For more information, see Configure VPC networking for IAM-based domains in the Amazon SageMaker Unified Studio Administrator Guide.

About the authors

Announcing Spark Connect on Amazon EMR on EC2: Interactive PySpark anywhere

Al MS — Wed, 23 Sep 2026 18:28:32 +0000

Today, we’re announcing support for Spark Connect on Amazon EMR on EC2 with the AWS runtime for Apache Spark (emr-spark-8.0, Apache Spark 4.0.2 and later). You can now develop and debug PySpark interactively from Amazon SageMaker Unified Studio Data Notebooks or your own IDE, such as Visual Studio Code, PyCharm, Kiro, or Jupyter. Spark runs on a dedicated Amazon EMR on EC2 cluster while your Python runs locally, so you can set breakpoints and inspect a DataFrame against full-size data from your IDE. In SageMaker Unified Studio Data Notebooks, you connect to your cluster, catalog, and AI tools. Production-scale PySpark and SQL run without leaving the studio. Because each session is isolated with its own permissions, your whole team can share one cluster at the same time. This post shows you how to get started with both SageMaker Unified Studio Data Notebooks and your own IDE.

Previously, developing Spark for an Amazon EMR on EC2 cluster meant working in a notebook tied to that cluster, or packaging your code as a job and submitting it before you could see a result. Local code often behaved differently on the cluster because of version and dependency mismatches, and the slow deploy-and-check loop made those differences hard to find. There was no way to attach your own IDE and debugger and inspect a DataFrame mid-transformation. Spark Connect closes that gap: your code runs against the cluster’s own Spark engine while you develop locally, so the environment you debug in is the one that runs your data.

How Spark Connect works on Amazon EMR on EC2

Spark Connect uses a client-server architecture that separates your application code from the Spark engine. The client is a lightweight PySpark library that runs in your notebook or IDE, and it sends DataFrame and SQL operations over a gRPC/TLS connection to a Spark Connect Server on your cluster. The server runs those operations and returns the results to your local session. Your machine does not need Spark installed and does not need to be sized for the workload.

Figure 1: Spark Connect client-server architecture on Amazon EMR on EC2

When you start a session, Amazon EMR launches the Spark Connect Server as a YARN application on your cluster and hands back an endpoint and a short-lived token. There’s no server for you to stand up or manage. Because that server runs on a cluster you already operate, your session inherits the instance types, libraries, bootstrap actions, and Spark configuration you use in production. What you see while debugging is what runs when the same code is scheduled as a batch job, since both use the same cluster and its configuration.

Now that you can start sessions, a single dedicated cluster can serve your whole team, because each session is a separate resource with its own execution role, tags, and lifecycle. A single cluster supports up to 1,000 concurrent sessions and 1,000 concurrent execution roles. These values are service maximums, not sizing targets. Actual concurrency depends on cluster size and per-session workload. Because interactive sessions are bursty and rarely all active at once, one cluster typically serves a team larger than its peak concurrent-session count. Enable Amazon EMR managed scaling so that capacity tracks demand. If peak concurrency approaches these maximums, or to isolate cost and data access by group, use multiple clusters—for example, one per team, business unit, or environment. Sharing one cluster gives you:

On-demand Spark without extra clusters — Developers get interactive sessions without provisioning a cluster apiece, which keeps utilization high and removes the cost of idle per-person clusters.
Consistent environments — Everyone runs the same Spark version, libraries, and security configuration, so results stay consistent, and your platform team patches and monitors one cluster.
Isolation and attribution — Per-session execution roles and tags keep each person’s work separate, so you can scope data access by session, track cost by user, and stop one session without disturbing anyone else.
Full visibility and control — View active sessions in the Spark UI, review finished ones in the Spark History Server, and manage them from the Amazon EMR console, API, CLI, or SDK.

Getting started

Getting started with Spark Connect on Amazon EMR on EC2 takes three steps: Create an Amazon EMR cluster with Spark Connect session enabled, start a session, and connect from your IDE or SageMaker Unified Studio Data Notebooks.

Note: In SageMaker Unified Studio, on-demand cluster creation is available for domains that use AWS IAM Identity Center. For domains that use AWS Identity and Access Management (IAM), attach an existing cluster. If your cluster runs in a private subnet, make sure that your network configuration allows connectivity between SageMaker Unified Studio and the cluster endpoint.

Prerequisites

You must have the following prerequisites in place.

An Amazon EMR cluster running release emr-spark-8.0.0 or later with SessionEnabled set to true.
The Spark application is installed on the cluster.
Python 3.9 or later with pyspark[connect] installed locally. The PySpark version must match the Spark version on your cluster.
For clusters in private subnets, the Amazon EMR service role must include the AmazonEMRServicePolicyForSessions managed policy, which grants permissions to create Network Load Balancers and virtual private cloud (VPC) endpoint services in your account.
To use Spark Connect sessions, you need permissions to start and list sessions on the cluster (elasticmapreduce:StartSession, ListSessions), get session details and endpoints and terminate sessions (elasticmapreduce:GetSession, GetSessionEndpoint, TerminateSession), and pass the execution role to the Amazon EMR service (iam:PassRole).

Working with interactive sessions

To create a session-enabled cluster and connect to it, follow these steps.

To start a Spark Connect session

Create a cluster with sessions enabled, running emr-spark-8.0.0 or later. The following is a sample command that you can modify for your needs, such as the instance types and counts:

aws emr create-cluster \
  --name "spark-connect-cluster" \
  --release-label emr-spark-8.0.0 \
  --applications Name=Spark \
  --service-role EMR_DefaultRole \
  --ec2-attributes InstanceProfile=EMR_EC2_DefaultRole,SubnetId=subnet-id \
  --instance-groups '[
    {"InstanceCount":1,"InstanceGroupType":"MASTER","InstanceType":"m8g.xlarge"},
    {"InstanceCount":2,"InstanceGroupType":"CORE","InstanceType":"m8g.xlarge"}
  ]' \
  --session-enabled \
  --tags Key=for-use-with-amazon-emr-managed-policies,Value=true

Note: The following steps use the AWS Command Line Interface (AWS CLI) directly. If you develop in SageMaker Unified Studio (Option 1), cluster attachment and session creation are handled for you, so you can skip steps 2 through 6.

After the cluster reaches the WAITING state, start a session and wait for it to reach IDLE:
```
aws emr start-session --cluster-id j-XXXXXXXXXXXXX --name "my-session"
aws emr get-session --cluster-id j-XXXXXXXXXXXXX --session-id is-XXXXXXXXXXXXX
```
Note: For runtime role sessions, add the --execution-role-arn parameter to the start-session command.

Retrieve the endpoint and token, and build your connection string from the returned Endpoint value rather than hardcoding a host:

aws emr get-session-endpoint --cluster-id j-XXXXXXXXXXXXX --session-id is-XXXXXXXXXXXXX

The response includes the endpoint URL and an authentication token:

{
  "Endpoint": "https://session-id.emr-spark-connect.region.amazonaws.com",
  "AuthToken": "v2.local.xxx...",
  "AuthTokenExpirationTime": "2026-01-01T01:00:00Z"
}

Install the matching PySpark client and connect. GetSessionEndpoint returns an https:// URL with no port. Build the connection string by converting it to the sc:// scheme and appending :443. Without the port, the PySpark client defaults to 15002, which isn’t reachable. Your Python code runs locally. The SQL and DataFrame operations run on the cluster:

pip install 'pyspark[connect]==4.0.2' boto3

from pyspark.sql import SparkSession

session_id = "is-XXXXXXXXXXXXX"
auth_token = "<AuthToken from get-session-endpoint>"
host = "<Endpoint from get-session-endpoint, without https://>"

url = f"sc://{host}:443/;use_ssl=true;x-aws-proxy-auth={auth_token};authorization={session_id}"
spark = SparkSession.builder.remote(url).getOrCreate()
spark.sql("SELECT 'Hello from EMR on EC2' AS message").show()

Run a transformation against full-size data. This groups a DataFrame, writes the result to Amazon Simple Storage Service (Amazon S3), and reads it back:

import pyspark.sql.functions as F

df = spark.range(0, 1000).withColumn(
    "category", F.when(F.col("id") % 2 == 0, "even").otherwise("odd")
)
df.groupBy("category").count().show()
df.write.mode("overwrite").parquet("s3://amzn-s3-demo-bucket/demo/")
spark.read.parquet("s3://amzn-s3-demo-bucket/demo/").filter("id < 50").orderBy("id").show()

When you finish, terminate the session to release cluster resources. Calling spark.stop() only closes the local connection. The session keeps running until you terminate it or it reaches the idle timeout:
```
aws emr terminate-session --cluster-id j-XXXXXXXXXXXXX --session-id is-XXXXXXXXXXXXX
```
When you’re done with the walkthrough, terminate the cluster you created in step 1 so it stops incurring charges. Terminating the cluster also ends any sessions still running on it:
```
aws emr terminate-clusters --cluster-ids j-XXXXXXXXXXXXX
```

You can start a Spark Connect session in two ways: from SageMaker Unified Studio or from your own IDE client.

Option 1: Develop in SageMaker Unified Studio Data Notebooks

Amazon SageMaker Unified Studio brings your data, catalogs, and analytics and AI tools into one place, and Amazon EMR on EC2 is now one of the Spark runtimes a Data Notebook can use. When you choose that cluster as the notebook runtime, SageMaker Unified Studio connects to it over Spark Connect. The same runtime then drives both your PySpark and SQL cells, so a single notebook can query the AWS Glue Data Catalog and transform the data without switching tools. The built-in AI assistant generates code and execution plans from natural-language prompts, and the Spark UI shows running work alongside your other runtimes.

To start a session from SageMaker Unified Studio:

Open a Data Notebook in SageMaker Unified Studio.
In the Compute panel, do one of the following:
1. To create a new cluster, choose Create cluster and configure an Amazon EMR on EC2 cluster.
2. To use an existing cluster, choose Attach cluster and select a running Amazon EMR on EC2 cluster.
Select the cluster as the notebook’s runtime.
Begin writing PySpark or SQL code in the notebook cells.

For a complete example, open the SageMaker Unified Studio Spark Connect example notebook , which connects a Data Notebook to an Amazon EMR on EC2 cluster and runs PySpark and SQL cells against the AWS Glue Data Catalog.

Watch a walkthrough: Develop in a SageMaker Unified Studio Data Notebook. The preceding steps cover the same workflow, so you can complete it from the notebook without the video.

Option 2: Develop in your own IDE

Use the IDE of your choice, such as Visual Studio Code, PyCharm, Kiro, or a local Jupyter notebook. You debug Spark the way you debug any Python program: set a breakpoint, inspect a variable, and step through your code, all while the Spark work runs on the cluster. Your libraries, source control, and continuous integration and continuous delivery (CI/CD) stay on your local machine, and only your Spark operations are sent to the cluster.

To see this end to end, the following example attaches an IDE to a Spark Connect session and steps through a breakpoint against cluster data.

Open the local IDE Spark Connect example notebook then use the connection steps in the preceding Getting started section to attach your client.

Watch a walkthrough: Develop your own IDE with Spark Connect. The written connection steps in Getting started cover the same workflow, so you can complete it without the video.

Use cases

Spark Connect on Amazon EMR on EC2 supports the following interactive workflows:

Interactive extract, transform, and load (ETL) development: Build and test pipelines against full-size data on the cluster, then schedule the same transformations as a Spark step on that cluster, where the Spark version, libraries, and configuration already match what you validated.
Exploratory data analysis and feature engineering: Analyze production-scale data from your notebook or IDE instead of sampled subsets, so you catch data quality issues earlier.
Notebook-driven analytics in SageMaker Unified Studio: Run PySpark and SQL next to your catalogs and AI tools, switching runtimes per notebook.
Apache Iceberg lakehouse analytics: Query and manage Iceberg tables through the AWS Glue Data Catalog, with time travel, schema evolution, and partition management.
Compute standardization: Point interactive development at the same clusters that run your production batch jobs, so development and production share one engine and configuration.

Release information

Spark Connect on Amazon EMR on EC2 is available with the AWS runtime for Apache Spark (emr-spark-8.0, Apache Spark 4.0.2) and later. It’s available in all AWS Regions where Amazon EMR is available, except the AWS GovCloud (US) Regions and the China Regions. The SageMaker Unified Studio experience is available in its supported Regions. There’s no additional charge for Spark Connect. You pay for the Amazon Elastic Compute Cloud (Amazon EC2) instances in your cluster. Because these sessions run on your own clusters, they use the Amazon EMR on EC2 capabilities you already rely on, including AWS Graviton processors for price-performance and your choice of On-Demand, Reserved, AWS Savings Plans, or Spot capacity.

Considerations for the release are as follows:

The PySpark version that you install locally must match the Apache Spark version on your cluster.
Spark Connect supports the DataFrame and SQL APIs. RDD-based APIs aren’t supported.
Authentication tokens expire after 1 hour, and sessions end after a configurable idle timeout (60 minutes by default, up to 24 hours).
High-availability clusters with multiple primary nodes, Trusted Identity Propagation, and fine-grained access control through AWS Lake Formation aren’t supported for Spark Connect sessions in this release.

Conclusion

Spark Connect on Amazon EMR on EC2 brings interactive, debuggable PySpark development to the clusters you already run. Develop on a SageMaker Unified Studio Data Notebook or in your own IDE, debug against full-size data while the cluster runs the work and share a single cluster across your whole team. To get started, see the Interactive sessions with Spark Connect guide or open a Data Notebook in Amazon SageMaker Unified Studio. To learn more about the service, see the Amazon EMR detail page.

About the authors

Query unstructured data in Amazon SageMaker Catalog using generative AI

Nishchai JM — Wed, 23 Sep 2026 18:22:00 +0000

Each day, businesses generate massive amounts of unstructured data, such as PDFs, images, email, customer feedback, and medical reports. But knowing data exists isn’t enough. You need to find it, access it, and extract answers from it fast. In Part 1 of this series, you saw how to set up the producer side of the pipeline: using Amazon Textract and Anthropic Claude on Amazon Bedrock to extract and enrich metadata, and then publish those enriched assets to Amazon SageMaker Catalog so your organization can discover them.

In this post, you take the next step: the consumer side. You sign in as a data consumer, search for and subscribe to the enriched unstructured data assets, and then query them using two approaches. The first is a no-code chat agent for natural language queries. The second is Amazon Bedrock model inference for programmatic access. By the end of this post, you will know how to unlock the business knowledge inside your unstructured data and make it available to analysts and application engineers alike.

Solution overview

This post continues the two-part series architecture, where Amazon SageMaker Catalog acts as the central hub connecting data producers and consumers through a publish-subscribe model.

The consumer workflow picks up after the producer has enriched and published the unstructured data assets. As a consumer, you will:

Sign in to your SageMaker Unified Studio consumer project and search the catalog using keywords from the enriched metadata README.
Subscribe to the published Amazon Simple Storage Service (Amazon S3) asset and get the subscription approved by the producer.
Interact with the subscribed data through two options:
- Option 1 – A no-code chat agent for natural language queries (NLQs), ideal for data analysts and business users.
- Option 2 – Amazon Bedrock model inference for programmatic NLQ integration, suited for application engineers building data-driven applications.

The following diagram illustrates the consumer workflow in this solution. The consumer (1) signs in to SageMaker Unified Studio, (2) searches the Amazon SageMaker Catalog for enriched unstructured data assets using keywords from the AI-generated metadata, (3) subscribes to the S3 data asset and receives approval from the producer, and then (4) queries the data using either the Amazon Bedrock chat agent app (Option 1) or Amazon Bedrock model inference through a Jupyter notebook (Option 2).

With both a no-code and a programmatic path, consumers across different roles, from analysts to engineers, can query data in the way that fits their workflow, while the SageMaker Catalog approval workflow maintains governed access throughout.

Figure 1: Consumer workflow for the publish-subscribe solution

Prerequisites

Before you begin, make sure you have completed all steps in Part 1 of this series, including:

An AWS account.
An Amazon Simple Storage Service (Amazon S3) bucket with the enriched documents and results.txt file uploaded.
An Amazon SageMaker Unified Studio domain with two projects:
- Producer project (unstructured-producer-project) – with the asset published to SageMaker Catalog.
- Consumer project (unstructured-consumer-project) – where you will subscribe and query the data.
Note: You can use custom blueprints to tailor your Amazon SageMaker Unified Studio project environments to your specific needs. For more information, see Tailor Amazon SageMaker Unified Studio project environments to your needs using custom blueprints.
The AWS Identity and Access Management (IAM) permissions added to the producer project role in Part 1.

Consume published data from the consumer project

In this section, you sign in as a consumer user in the SageMaker Unified Studio consumer project. You then subscribe to the S3 bucket by searching for a keyword that is part of the README published in Part 1.

Sign in to the consumer project and search for the keyword emergency, which was added to the README file during publishing. The search returns the enriched asset that the producer published in Part 1.

Figure 2: Catalog search results for the emergency keyword
Choose the asset from the results to view its details, including the AI-generated business metadata, glossary terms, and README content. Then choose Subscribe.

Figure 3: Asset details with AI-generated metadata and the Subscribe option
Enter analysis as the Reason for request in the Comment section, then choose Request.

Figure 4: Subscription request with the reason for request entered
Sign back in to the producer project (unstructured-producer-project) to approve the subscription request.
After approval, return to the consumer project and confirm that the subscribed asset now appears under Manage, Assets, Subscribed assets.

Figure 5: Approved subscription under the Subscribed assets tab

With the subscription approved, you can now access the enriched unstructured data through two approaches.

Option 1: As a data or business analyst, you can use the Amazon Bedrock chat agent app for natural language queries.

Option 2: As an application engineer, you can use Amazon Bedrock model inference for programmatic natural language queries.

Let’s explore both options.

Option 1: Amazon Bedrock chat agent app

The Amazon Bedrock chat agent app gives you a no-code, conversational interface to query your enriched unstructured data using natural language. As a data analyst or business user, you can ask questions in plain English. You get answers grounded in the documents your organization has ingested, without writing any code. For production workloads, especially in sensitive domains such as healthcare, you can apply Amazon Bedrock Guardrails to add content filtering and grounding validation to your model responses.

Data scientists and application engineers can also extend these capabilities by integrating the chat agent app APIs into custom applications, so users can interact with unstructured Amazon S3 data programmatically.

To set up the Amazon Bedrock chat agent app on your subscribed dataset, complete the following steps.

Prerequisite: Add the S3 data location.

Before creating the chat agent app, you need to add the S3 location of your subscribed data as a registered location in your project.

Choose the Data tab in Overview.
Choose the S3 bucket, and then choose Add to add the S3 location.

Figure 6: Adding the S3 location from the Data tab
On the S3 location page, provide the following details:
- Add a name: producerprojectdata.
- Add the producer’s S3 path as a new S3 location: s3://amzn-sagemaker-bucket-<domain-id>-<project-id>/medical/.
Note: You can get the S3 location details from the technical name of your subscribed asset.
- Choose the AWS Region, and then choose Add data to add this as a new location.
Note: Make sure the AWS Region you select supports the Amazon Bedrock foundation models used later in this post. For a list of available models by Region, see Supported Regions and models for Amazon Bedrock.

Figure 7: S3 location details and AWS Region selection

Note: Make sure to select only the PDF files within the S3 path for the data source.

Figure 8: Selecting the PDF files as the data source

After the location is added, it appears as a selectable S3 location when creating a knowledge base in AI Apps.

Complete the following steps to configure the chat agent app:

In the left navigation pane, under Generative AI, choose AI Apps.
In the Build section of the page, choose Chat agent.

Figure 9: Choosing Chat agent in the AI Apps Build section
Expand the Data tab to create a knowledge base with your S3 bucket. On the Create a new knowledge base page, enter the following:
- Add a name: MedicalKB.
- Add a description: Knowledge base built from subscribed medical S3 data assets. Contains medical documents used to provide grounded, context-aware responses to medical domain queries.
- Choose the data source. You will see the S3 bucket that you added in the previous step.
Figure 10: Creating the MedicalKB knowledge base from the S3 data source
Choose your embedding model. You can leave the default settings and choose Create. It might take 10–15 minutes to create the knowledge base, depending on file sizes.
After the knowledge base is created, on the Chat agent page:
- Choose your preferred model from the Model menu (you can switch between different large language models as needed).
- Under Data, choose your published S3 bucket as the knowledge base.
- Begin interacting with the agent by entering questions in the Enter prompt field.
Figure 11: Chat agent page with the model and knowledge base selected

For example, entering “Which age groups had the highest rates of emergency department visits for tooth disorders?” returns an answer grounded in the enriched dental dataset published in Part 1.

The chat agent uses the enriched README metadata along with the underlying documents to surface contextually relevant answers. Analysts can explore unstructured content without needing to know where the data lives or how it’s structured.

Option 2: Natural language queries using Amazon Bedrock model inference

This option demonstrates how to use Amazon Bedrock model inference to query subscribed data using natural language. You can integrate this capability with external chat applications so users can run natural language queries through Amazon Bedrock.

In your consumer project, choose Manage, Assets from the bottom of the left navigation pane. On the Subscribed tab, choose your subscribed S3 asset. Under Actions, choose Open JupyterLab notebook.

Figure 12: Opening the JupyterLab notebook from the subscribed asset
This opens the JupyterLab notebook environment. Upload the s3_document_consumer_v2.ipynb notebook and run all the cells. You can download the notebook from s3_document_consumer_v2.ipynb.Note: The project role requires permissions for Amazon S3, Amazon Textract, and Amazon Bedrock. If you followed Part 1, you might already have these policies attached. For details on the required policies and guidance, see the prerequisites in Part 1.
Review the notebook cells.

Figure 13: Sample question answered by Amazon Bedrock in the notebook

In the final cell, you find a sample question that Amazon Bedrock answers: “Which primary payer types (Medicare, Medicaid, private insurance, and so on) account for the highest proportion of dental-related emergency department visits?”

Amazon Bedrock processes the question against the enriched content in the S3 bucket and returns a grounded answer. You can replace this sample question with any query relevant to your documents.

The Amazon Bedrock model inference approach gives you programmatic control, making it possible to embed natural language query capabilities directly into your existing data applications and business intelligence tools.

Clean up

To avoid ongoing charges, make sure to delete the resources used in this solution immediately after completing the walkthrough. The primary cost drivers are SageMaker Unified Studio notebook instances, Amazon Bedrock model inference calls, and Amazon S3 storage.

Stop SageMaker Unified Studio resources:
- Close running notebooks.
- Stop running notebook instances.
- Shut down unused kernels.
Note: Running notebook instances continue to incur charges even when not in use.
Clean Amazon S3 storage:
- Delete temporary files created during processing.
- Remove uploaded test documents that are no longer needed.
Note: Although Amazon S3 costs are minimal, large volumes of data can accumulate significant charges, so it’s best to remove unneeded data.

Conclusion

In this post, you saw how to consume and query the enriched unstructured data assets published in Part 1 of this series. By subscribing to assets through the Amazon SageMaker Catalog publish-subscribe model, you can discover, access, and interact with your organization’s unstructured data, whether through the no-code chat agent or Amazon Bedrock model inference.

Together, both parts of this series show you how to build a comprehensive pipeline that transforms raw unstructured documents into governed, queryable knowledge assets. The combination of Amazon Textract for extraction, Amazon Bedrock for intelligent summarization and NLQ, and Amazon SageMaker Catalog for governance and discoverability means your teams can focus on extracting business insights rather than managing infrastructure.

To continue your Amazon SageMaker journey, see the following resources:

Explore the Amazon SageMaker Developer Guide.
Find complete setup guides in the Amazon SageMaker Unified Studio documentation.
Get started with these new integrations through the Amazon SageMaker Unified Studio console.

About the authors

Best practices for scaling large consumer groups on Amazon MSK

Pallavi Jha — Tue, 22 Sep 2026 15:16:06 +0000

When scaling large consumer groups on Amazon Managed Streaming for Apache Kafka (Amazon MSK), a common challenge is managing the size of internal metadata records. During rebalances, Kafka persists a metadata record to the internal __consumer_offsets topic. If the consumer group is large enough, this record can exceed the default 1 MB limit. This causes a RecordTooLargeException and a rebalance retry loop.

In this post, we explain how consumer group metadata grows and how to estimate your metadata size. We provide a step-by-step walkthrough for increasing the topic-level size limit (the most common remediation), along with guidance on three complementary strategies: splitting groups, right-sizing partitions, and optimizing naming conventions. We also discuss capacity planning, monitoring, and how KIP-848 in Apache Kafka 4.0 addresses this constraint at the protocol level.

Prerequisites: This post assumes familiarity with Apache Kafka consumer groups, rebalance protocols, and Amazon MSK cluster configuration. You should have access to the kafka-configs.sh CLI tool or the Amazon MSK console. The configuration approaches described apply to Amazon MSK Provisioned clusters with Standard brokers. The KIP-848 section covers a forward-looking protocol change in Apache Kafka 4.0 that applies across deployment types.

How consumer group metadata grows

The following diagram illustrates how consumer group metadata flows through the system during a rebalance:

Figure 1: Consumer group metadata flow during a rebalance

During a rebalance, the Group Coordinator serializes a GroupMetadata record containing information about every member in the group and persists it to the __consumer_offsets topic. This record must fit within the topic’s max.message.bytes limit. Follower brokers must also replicate it, constrained by replica.fetch.max.bytes. For each member, the record includes:

Subscription topics – The list of topics the member subscribes to.
Owned partitions – Partitions currently held by the member.
Assignment – The new partition assignment after rebalancing.
Client ID – The configured client.id.

Apache Kafka’s serialization format repeats topic names multiple times per member: once in subscription, once in ownedPartitions, and once in assignment. The client.id adds further per-member overhead. The broker stores these metadata records uncompressed in __consumer_offsets.

Estimating your metadata record size

You can approximate your consumer group’s metadata record size with the following formula:

record_size ≈ member_count × (3 × avg_topic_name_bytes + 2 × client_id_bytes + ~200 bytes overhead)

Example: A group with 1,000 members, a 50-byte topic name, and a 40-byte client ID:

1,000 × (150 + 80 + 200) = ~430 KB

At 1,500 members with the same parameters: ~645 KB. With multiple topic subscriptions or longer naming conventions, the record can exceed 1 MB well before 2,000 members.

Approaches to handle large consumer group metadata

The following sections describe four strategies for managing large consumer group metadata, starting with the most direct remediation.

Increase max.message.bytes on __consumer_offsets

If your consumer group metadata exceeds 1 MB, you can increase the maximum record size on the internal topic. This is the most direct path to help unblock consumer groups that have already scaled beyond the default. Note that max.message.bytes is the topic-level configuration name, while message.max.bytes is the equivalent broker-level default.

1. Update the topic-level configuration:

kafka-configs.sh --bootstrap-server <bootstrap-server> \
    --entity-type topics \
    --entity-name __consumer_offsets \
    --alter \
    --add-config max.message.bytes=2097152 # topic-level config for max record size

2. Update replica.fetch.max.bytes at the cluster level:

This broker-level setting controls the maximum fetch size for inter-broker replication. Set it equal to or greater than max.message.bytes on __consumer_offsets so follower brokers can replicate large metadata records.

replica.fetch.max.bytes=2097152

You can apply this through the Amazon MSK console under Cluster configuration or using the AWS Command Line Interface (AWS CLI) with update-cluster-configuration.

Figure 2: Setting replica.fetch.max.bytes in the Amazon MSK cluster configuration console

Important: Always make sure that replica.fetch.max.bytes ≥ max.message.bytes for __consumer_offsets. Without this, you might observe UnderReplicatedPartitions on the internal topic.

3. Test in a non-production environment first:

Trigger a consumer group rebalance (restart consumers or scale the group).
Verify no RecordTooLargeException in broker logs.
Confirm broker heap usage and replication lag remain healthy.

Split large consumer groups

Breaking a single large consumer group into multiple smaller groups reduces the per-group metadata record size proportionally. To split a group, deploy multiple connector or consumer instances, each with a distinct group.id, subscribing to the same topic but consuming from a subset of partitions. The system preserves offset tracking within each sub-group independently.

When to use: Consumer group membership is growing unboundedly through Auto Scaling, and you want to keep each group’s metadata well within limits without modifying internal topic configuration.

Trade-off: Increases operational complexity. You have multiple groups to monitor and manage instead of one.

Right-size partition count and auto scaling bounds

Over-partitioned topics require more consumers to fully parallelize, which inflates group membership. Unbounded auto scaling policies can grow consumer groups beyond what was originally planned.

Review whether your topic’s partition count matches your actual throughput requirements.
Configure auto scaling policies with an upper bound on consumer replicas (for example, Horizontal Pod Autoscaler on Amazon Elastic Kubernetes Service (Amazon EKS)).
Align partition count with the maximum number of consumers you intend to support.

This is a proactive measure, best applied during topic design and capacity planning to prevent the metadata size issue from occurring in the first place.

Optimize naming conventions

Consumer group names and client IDs contribute to the overall metadata size. Shorter, standardized naming reduces per-member overhead.

Considerations: Changing an active consumer group’s name means the new group starts with no committed offsets and all tracking history is lost. For this reason, naming optimization is most practical for new deployments rather than existing production groups.

Capacity planning for larger metadata records

When you increase max.message.bytes on __consumer_offsets, larger metadata records consume more broker heap during rebalance processing. Proper capacity planning helps you select the right broker instance type and configuration value before hitting production issues.

Planning steps:

Calculate your current record size using: member_count × (3 × topic_name_bytes + 2 × client_id_bytes + ~200).
Project peak membership based on your auto scaling upper bound (maximum consumer replicas × number of tasks per connector, if using Kafka Connect).
Apply a 2× safety margin to account for protocol overhead, multi-topic subscriptions, and burst scaling events.
Select your max.message.bytes value from the following guidance table.
Choose your broker instance type based on heap requirements. Larger metadata records increase heap pressure during rebalances. For groups exceeding 1,000 members with 2+ MB metadata records, use kafka.m5.xlarge or larger to provide sufficient heap headroom.
Validate in non-production by running a consumer group at projected peak membership and monitoring HeapMemoryAfterGC during rebalances.

The following table provides sizing guidance based on consumer group size:

Consumer Group Size	Guidance
< 500 members	Default 1 MB is typically sufficient. `kafka.m5.large` or larger.
500–1,000 members	Monitor metadata size. Consider increasing to 2 MB. `kafka.m5.xlarge` or larger.
1,000–2,000 members	Increase to 2–5 MB. `kafka.m5.2xlarge` or larger for adequate heap headroom.
> 2,000 members	Combine increased limit with group splitting. `kafka.m5.2xlarge` minimum. Consider `kafka.m5.4xlarge` for high rebalance frequency.

Key metrics to monitor

The following Amazon CloudWatch metrics help you track consumer group metadata health:

Metric	What it tells you
`HeapMemoryAfterGC` (Amazon CloudWatch)	Percentage of heap memory in use after garbage collection. Indicates memory pressure from larger metadata records during rebalances.
`UnderReplicatedPartitions` (Amazon CloudWatch)	Replication health. Non-zero may indicate `replica.fetch.max.bytes` is too low.
GC pause duration (broker logs)	Prolonged GC can trigger session timeouts and cascading rebalances.
Consumer group rebalance rate	Stable groups should not rebalance frequently after configuration changes.
Consumer lag	Confirms consumers are making progress after rebalances complete.

We recommend creating two Amazon CloudWatch alarms for HeapMemoryAfterGC. Set a warning alarm at 60% to indicate potential performance degradation. Set a critical alarm at 80 percent, at which point you should scale brokers or reduce consumer group size. For UnderReplicatedPartitions, alarm at any value> 0 sustained for more than 5 minutes after a configuration change.

Looking ahead: KIP-848 and Apache Kafka 4.0

Apache Kafka 4.0 (released March 2025) adopted KIP-848 as the default consumer protocol. The broker now computes partition assignments server-side rather than delegating to a consumer group leader. Because each member no longer carries full subscription and assignment data on the wire, the new protocol reduces per-member metadata size. For details on the protocol changes that achieve this reduction, see the KIP-848 design document. KIP-848 also introduces incremental rebalances.

Newer Apache Kafka versions on Amazon MSK bring smaller metadata records by default. The following steps help you prepare for KIP-848 adoption:

Track Amazon MSK version support for Apache Kafka 4.0+.
Verify your Kafka client libraries support the new consumer protocol.
Test the new protocol in a non-production environment before migrating production consumer groups.
Plan for a phased rollout, starting with non-critical consumer groups.

Conclusion

The following table summarizes when to apply each approach. The max.message.bytes increase (covered step-by-step earlier) is the primary remediation. The other strategies are complementary guidance you can adapt to your environment:

Situation	Recommended approach
Already hitting `RecordTooLargeException` in production	Increase `max.message.bytes` on `__consumer_offsets` + set `replica.fetch.max.bytes` accordingly
Planning for growth	Right-size partitions, set auto scaling bounds, monitor metadata size
Naming overhead is significant	Optimize naming conventions for new deployments
Operating at very large scale (2,000+ members)	Combine increased limits with consumer group splitting
Long-term architecture	Plan migration path to KIP-848 (Apache Kafka 4.0)

Test configuration changes in non-production first, monitor broker metrics during and after rebalances, and scale incrementally. With these practices in place, you can operate consumer groups at the scale your streaming workloads require.

To get started:

Review the Amazon MSK Developer Guide for cluster configuration steps.
Use the sizing formula in this post to estimate your current metadata record size.
Set up Amazon CloudWatch alarms on HeapMemoryAfterGC to monitor broker health proactively.

About the authors

How Moeve standardized dbt runs across data lakes with Amazon Athena

Rubén Romero Córdoba — Tue, 22 Sep 2026 15:13:31 +0000

As organizations grow, data processing often becomes fragmented across teams, environments, and orchestration tools. This fragmentation leads to inconsistent patterns, duplicated logic, limited cost visibility, and operational overhead.

At Moeve we were no exception. Our analytics teams build their transformations with dbt, an open source tool that defines transformations as SQL models, resolves the references between them, and works out the order in which they run. dbt describes what to transform, but it does not define where or how a project runs. We left that decision to each team, and as the number of projects grew we found fragmented pipelines, inconsistent compute engines, and limited cost visibility slowing every project down. Standardizing our dbt runs on Amazon Athena was how we worked our way out of that.

This post describes the architecture of the centralized, serverless solution we built on Athena, which reduced onboarding for a new dbt project from days to about 15 minutes. The solution centralizes how dbt runs across our data lakes while staying loosely coupled from orchestration. It uses Amazon Athena as the default processing engine, a centralized dbt launcher, and a shared event bus for downstream orchestration.

In the sections that follow we explain how we decoupled dbt runs from orchestration using AWS Step Functions and AWS Fargate, why Athena fits our workloads from a cost and operational perspective, how storing run parameters in Amazon DynamoDB rather than in pipeline code removed the infrastructure deployment step from onboarding, and how publishing results to Amazon EventBridge lets our run and orchestration layers evolve independently.

The challenge of running dbt at scale

Before the dbt launcher, our dbt runs had grown in different directions. Run logic was embedded in project-specific pipelines, orchestration and processing were tightly coupled, teams selected different compute engines for comparable workloads, and we had no consistent governance over run parameters and retries.

As the number of dbt projects increased, this made it difficult to enforce consistent standards and to evolve the solution without touching every pipeline. We needed a way to standardize dbt runs across our data lakes, decouple running a project from deciding what to run, improve cost control and observability, and deliver faster and safer continuous integration and continuous delivery (CI/CD) iterations.

Why Amazon Athena as the default dbt engine

Choosing the processing engine for dbt is a foundational architectural decision, so we made it first.

Serverless processing

Athena is fully serverless. There are no clusters to provision, scale, or maintain. Our teams run their queries with the default Athena pricing, which charges for the data a query scans and gives us elasticity with no capacity planning.

Because each data lake lives in its own account, each team makes its own decision about Athena payment. A team whose workload grows into continuous, high-concurrency usage can move to Athena capacity reservations with no change to the launcher, to their dbt profiles, or to their project configuration. None of our teams have needed to do so yet, and the architecture keeps that choice independent per team.

Our workload is predictable but not continuous. Each dbt project runs for a few minutes when its schedule fires or when its upstream data lands, then stays idle until the next trigger. That shape is what made serverless the right fit for us.

Why Athena fit our solution

For Moeve, the decision to standardize on dbt and Amazon Athena was driven by our goal of creating a common transformation solution that could be adopted across multiple teams and AWS accounts while keeping operations lightweight.

Our data was already stored in Amazon S3 and registered in the AWS Glue Data Catalog, making Athena a natural processing layer. Athena allowed us to run transformations without managing clusters, capacity, or infrastructure, which was particularly important for a small central team supporting multiple domains.

We evaluated alternative processing engines, but for our workload profile and data volumes, Athena provided the best balance between scalability, operational simplicity, and maintainability. Adapter maturity was another important factor. The dbt Athena adapter offered strong integration with testing, CI/CD workflows, and the broader dbt ecosystem, reducing the operational risk of maintaining custom solutions.

Athena also aligned naturally with our architecture. Transformations run in the AWS account that owns the data through cross-account role assumption, while orchestration remains centralized. As a result, we standardized how projects run across teams while keeping compute close to the data.

Finally, the Apache Iceberg support in Athena underpins the idempotent incremental processing model described in this post, so incremental loads and historical reprocessing follow the same path with minimal operational overhead.

Optimized incremental processing

Our largest cost was not reading source data. It was merging into it.

Our fact tables are Apache Iceberg tables in the data lake, and most of our models are incremental. Each run brings in new or corrected records and merges them into a target that can hold several years of history. A merge has to locate the rows it is about to update, and without a predicate on the target the query reads far more of the table than the incoming data can affect. The common approach is a static filter such as the last 30 days, which is wrong in both directions: too wide for an ordinary daily load, and too narrow as soon as a correction arrives for an older partition.

Instead of a fixed window, the platform derives the predicate from the data. Before the merge runs, it reads the distinct values of the partition column present in the incoming dataset and builds the target predicate from them. For a single partition it applies an equality predicate, for a small set an IN list, and for a larger set a bounded range. The merge then reads only the partitions the incoming data can affect.

The same principle applies on the source side. The launcher builds the source filter from the parameters given for that run: an explicit range, an arbitrary SQL condition, or, when neither is supplied, a default window taken from the project configuration. Input is therefore bounded to the subset each run needs.

Two results mattered to us. Because Athena charges for the data a query scans, narrowing both ends of the merge reduces cost without any team hand-tuning individual models. More importantly, a daily load and a full historical reprocess became the same operation with different inputs. Every run is idempotent, so the same input always produces the same result regardless of how many times it runs. That removed the distinction between processing and reprocessing from our runbooks and simplified incident response.

Table design is what makes this pruning possible. Partitioning, columnar formats, and compression all contribute, and the AWS Big Data Blog post Top 10 performance tuning tips for Amazon Athena covers the general techniques. We have deliberately not published a before and after figure here, because the saving depends so heavily on partition design and data distribution that a single number would mislead without extensive context.

Architecture overview

Moeve built a centralized dbt launcher that runs dbt jobs in a uniform way, regardless of the project or the target data lake.

Architecture diagram

Figure 1: Centralized dbt launcher and cross-account run flow

At a high level, the architecture consists of:

AWS Step Functions to control the run lifecycle.
AWS Fargate to run dbt in an isolated, ephemeral container.
Amazon Athena as the default dbt processing engine.
Amazon DynamoDB to store dbt project configuration.
Amazon EventBridge to publish run results.

Every dbt run follows the same contract, which gives us consistency and reduces the operational surface we must maintain.

Cross-account processing model

The solution operates in a centralized account while running transformations in domain-specific data lake accounts: corporate, marketing, and manufacturing. The Fargate container assumes a dbt-child IAM role in the target account, so the container processes data where it lives while governance stays centralized.

Each data lake account keeps control of its own IAM permissions and manages its own storage and catalog without affecting the solution. This also puts costs in the right account. We could have attributed Athena spend using Athena workgroups, but Athena is only part of what a query costs. The Amazon Simple Storage Service (Amazon S3) requests it makes and the AWS Key Management Service (AWS KMS) operations it triggers are real costs as well, and running in the owning account attributes all of them to the team that owns the data, per project and per run.

Centralizing dbt runs with the dbt launcher

To stop every team inventing its own way of running dbt, we built a single launcher that all of them go through. Instead of dbt logic living inside multiple pipelines, every run is triggered through one well-defined path.

Run lifecycle

The launcher is an AWS Step Functions state machine. It receives a run request with its parameters, starts an AWS Fargate task from our dbt container image, and the container assumes the IAM role of the target data lake account. dbt then runs its SQL transformations in Athena, reading the source tables and materializing the targets. Alongside the run, Elementary, an open source dbt package, records model-level results and data quality test outcomes. When the run finishes, the launcher publishes a completion event to Amazon EventBridge.

The state machine can be started in different ways depending on the scenario. Some projects run on a schedule, others are triggered when upstream data lands, and teams can request a run on demand. Those decisions are made by our orchestration layer, which submits a standardized run request to the launcher. The launcher therefore stays focused on running dbt projects, regardless of how the run was initiated.

This sequence is identical for all projects and environments. The launcher is responsible only for running the project it was asked to run. It doesn’t decide what should run next, and that responsibility is intentionally delegated to downstream consumers through Amazon EventBridge.

Configuration-driven runs with Amazon DynamoDB

We had to decide where a project’s run parameters would live. In the pipeline definition, changing a timeout would be a code change, a review, a build, and a deployment, for a value we sometimes need to change while an incident is open. We put the parameters in DynamoDB instead, keyed by project, and the launcher reads them at the start of every run. That lets us decouple code deployment from run behavior, update parameters without redeploying services, and enforce consistent defaults across all dbt projects.

The parameters themselves are modest. They cover the target environment and AWS Identity and Access Management (IAM) role, the default processing engine, how long to allow a run to take, how many times to retry on failure, and how many days of data to process by default. These values change for operational reasons rather than logical ones, which is why we did not want them coupled to a release cycle.

The effect on onboarding was larger than we expected. Deploying a new dbt project is now a merge of the dbt models and one configuration entry. There is no Terraform change, no infrastructure review, and nothing to provision, because the compute the project needs already exists and is shared. What used to be a multi-step infrastructure pipeline is now a single CI workflow that validates the project’s SQL and lineage locally, then merges and registers it. We run that local validation with DuckDB, which returns feedback in under two minutes without consuming cloud resources.

Governance did not weaken as a result. Who may change a configuration entry is controlled the same way as any other production change. What changed is that the change no longer has to travel through an infrastructure deployment to take effect.

CI/CD pipeline diagram

Figure 2: CI workflow for onboarding and updating a dbt project

Athena remains the engine of record. Local validation catches Jinja errors, unresolved references, and obvious SQL mistakes, but Athena-specific behavior, cross-account permissions, and AWS Glue Data Catalog interactions are only proven in the target environment. It’s important to be explicit about that boundary with the teams, so that a green CI run is not read as a guarantee.

Publishing run results with Amazon EventBridge

After a dbt run finishes, the launcher publishes a structured event to a central Amazon EventBridge bus recording whether the run succeeded, which project and source it covered, when it started and finished, how long it took, and which datasets it updated. The launcher does not know which consumers are subscribed.

This event-driven approach gives us loose coupling between running a project and orchestrating what comes next, multiple downstream consumers for the same signal, and independent evolution of both layers.

At Moeve the main consumer is our orchestration layer, which models the dependencies between datasets as a graph. Each completion event tells it that a node is now up to date, so it can determine which downstream projects have all their inputs ready and start them. That consumer has no special status. An AWS Lambda function, a monitoring dashboard, or a notification integration can subscribe to the same events without any change to the launcher.

Validation and observability

Because every project follows the same lifecycle, we get validation and observability in one place instead of per pipeline. Step Functions shows the state of any run and the step at which it failed, and error handling and retries are defined once. Fargate logs carry the container runtime detail. dbt and Elementary report model-level results and data quality test outcomes. The completion event on the bus is the auditable record that a project finished and what it produced. Together these layers give us operational visibility without coupling the components to each other.

Cost control and resource cleanup

The platform is serverless end to end, which keeps idle cost close to zero and removes a class of operational mistake. Fargate tasks are created for a run and destroyed when it ends, so no long-running container needs maintenance. Athena has no persistent compute. An idle Step Functions state machine costs nothing. Nothing is left running unintentionally, which matters when the number of projects on the platform keeps growing.

Results

The metrics in the following table are the ones our teams notice day to day, and the reason the solution is maintainable by a small central team.

Metric	Before	After
Onboarding time for a new dbt project	Days, including pipeline and infrastructure setup	About 15 minutes, configuration only
Run consistency	Varied by team	Same lifecycle and contract for every project
CI feedback time	8 to 10 minutes on Jenkins	Under 2 minutes with local validation
Cost visibility	Per-account aggregate	Per-project and per-run attribution
Operational overhead	One pipeline per project	One solution for all projects

Conclusion

Standardizing how we run dbt turned out to depend less on dbt than on defining two boundaries clearly.

The first is the contract of the launcher: parameters in, event out. We defined that interface before building the internals, and it has stayed stable while the implementation changed several times. The second is the separation between running a project and deciding what to run next. Making the launcher publish events without knowing its consumers is why our orchestration layer could be rebuilt while the launcher stayed as it was, and the launcher has never been modified to accommodate a new orchestration requirement.

Two smaller decisions carried more weight than we expected. Keeping run parameters in DynamoDB rather than in pipeline definitions means timeouts, retries, and engine selection can be changed without a deployment, which is valuable during incident response. Making every run idempotent removed the distinction between processing and reprocessing, so a daily load and a full historical reprocess are the same operation with different inputs.

Athena is serverless, so our teams did not need to set up infrastructure of their own. That is what made it practical for everyone to run dbt in a standardized way, and why onboarding a new project went from days to about 15 minutes.

If your organization runs dbt across multiple accounts and teams, pair Amazon Athena with a clear contract for the component that runs your projects: fixed parameters in, a published event out.

AWS Big Data Blog

Resolving concurrent write conflicts for Apache Iceberg tables in Amazon Redshift

The challenge of concurrent writes

How Amazon Redshift resolves write conflicts

Resolving at the right granularity

Performance

Fleet Results

Observing conflict resolution

Conclusion

About the authors

Bring data quality to your Amazon EMR workloads with Amazon EMR Data Quality built on AWS Glue Data Quality

What is Amazon EMR Data Quality?

Prerequisites and setup

Pattern 1: Ad hoc data quality validation

The use case

How it works

To validate data quality inline using PySpark

What the data quality results look like

Pattern 2: Iceberg Write-Audit-Publish with managed rulesets

The use case

How it works

To validate data quality inline using PySpark

Clean up

Conclusion

About the authors

Build cross-account streaming pipelines with the Amazon Kinesis Producer Library and Kinesis Client Library

Challenges enterprises face with multi-account streaming

Cross-account deployment patterns

Solution architecture

How the cross-account access works

Deployment walkthrough

Prerequisites

Step 1: Deploy the stream stack (Account B: Stream Account)

Step 2: Deploy the producer stack (Account A: Producer Account)

Step 3: Deploy the consumer stack (Account C: Consumer Account)

Testing end-to-end data flow

Start the KPL producer (Account A)

Start the KCL consumer (Account C)

Verify in CloudWatch (Account B: Stream metrics)

Key considerations

Clean up

Common errors in cross-account

What’s new in KPL and KCL

Conclusion

About the authors

Monitoring MWAA-orchestrated ETL pipelines with Amazon OpenSearch Service

Solution overview

Prerequisites

Architecture

Orchestration and log generation

Workflow

Log integration methods

Real-time log streaming

Process

AI-powered analysis and MCP interface

Process

Key architectural benefits

CloudFormation stacks

Stack 1: opensearch-cfn (opensearch_cfn.yaml)

Stack 2: etl (etl.yaml)

Stack 3: agentcore-mcp-server (agentcore-mcp-server.yaml)

Implementation

Deploy each CloudFormation stack in sequence

Work with the SageMaker AI notebook

Results

Outcome

Clean up

Conclusion

About the authors

Cost-effective ETL with DuckDB and Amazon S3 Tables on AWS Glue

When to use this pattern

Signal

DuckDB on AWS Glue 6.0

Apache Spark on AWS Glue 6.0

How DuckDB runs on AWS Glue 6.0

What the AWS Glue 6.0 runtime provides

What DuckDB provides

Architecture

Layer

Role