AWS Big Data Blog

Unified observability in Amazon OpenSearch Service: metrics, traces, and AI agent debugging in a single interface

Muthu Pitchaimani — Tue, 28 Apr 2026 17:29:01 +0000

Amazon OpenSearch Service now brings application monitoring, native Amazon Managed Service for Prometheus integration, and AI agent tracing together in OpenSearch UI‘s observability workspace. You can query Prometheus metrics with PromQL alongside logs and traces stored in Amazon OpenSearch Service, trace an AI agent’s full reasoning chain down to the failing tool call, and drill from a service-level health view to the exact span that caused a checkout failure, all without leaving the interface.

In this post, we walk through two real-world scenarios using the OpenTelemetry sample app: a multi-agent travel planner facing slow processing, and a checkout flow quietly failing on one microservice. We chase each one to its root cause using these new capabilities.

Scenario 1: An underperforming AI agent

Your multi-agent travel planner is live and users start reporting slow responses. With the new AI agent tracing capability in Amazon OpenSearch Service, you can trace the agent’s full processing path to pinpoint exactly where things went wrong.

In any observability workspace in OpenSearch UI, navigate to Application Map in the left navigation pane.

You can see the full topology of your system including the travel agent and the sub-agents it calls. The travel agent node shows elevated latency and occasional errors. Select it, and the side panel confirms that latency is up but the latency chart shows intermittent spikes rather than consistent degradation.

The application map tells you something is wrong, but understanding why an AI agent is underperforming requires seeing its reasoning chain. Select Agent Traces in the left navigation pane, then filter by service name and time range.

Select one of the traces to see the trace tree. Unlike a traditional span waterfall, this view organizes around the agent’s reasoning chain: the root agent span, the LLM calls it made, the tools it invoked, and how they nested each step color-coded by type. The trace map provides a visual directed graph of the same execution. You can see which model was called, how many input and output tokens were consumed, and the actual messages sent to and received from the model.

A tool call inside the weather agent errored out. The agent then spent additional time reasoning about the failure before returning a partial response explaining the intermittent latency spikes and occasional faults.

Why this matters for AI agents

Agents make autonomous decisions based on LLM responses, tool results, and chained reasoning. Unlike traditional microservices with deterministic code paths, agent behavior varies across executions. Without semantic tracing that captures these AI-specific signals, root-cause analysis is guesswork. The trace tree surfaced the model name, token counts, and failing tool call because the travel planner was instrumented with OpenTelemetry’s generative AI semantic conventions. The next section describes how.

Instrumenting AI agents

OpenTelemetry auto-instrumentation enriches spans with well-known attributes for HTTP, database, and gRPC calls. AI agents need a different set of attributes such as which LLM was called, what tokens were consumed, which tools were invoked, that standard instrumentation doesn’t cover.

The OpenTelemetry gen_ai semantic conventions define standard attributes for these signals, including gen_ai.operation.name, gen_ai.usage.input_tokens, gen_ai.request.model, and gen_ai.tool.name. When Amazon OpenSearch Service receives spans with these attributes, it categorizes them by operation type (agent, LLM, tool, embeddings, retrieval) and renders the agent trace tree and trace map views.

The Python SDK provides one way to generate these spans. To send traces to Amazon OpenSearch Ingestion, configure the SDK with AWS Signature Version 4 (SigV4) authentication. The AWSSigV4OTLPExporter cryptographically signs each HTTP request to help prevent unauthorized data ingestion. The calling identity needs an IAM policy that grants osis:Ingest on your pipeline’s ARN. Credentials are resolved through the standard AWS credential provider chain.

from opensearch_genai_observability_sdk_py import register, AWSSigV4OTLPExporter

exporter = AWSSigV4OTLPExporter(
    endpoint="https://pipeline.us-east-1.osis.amazonaws.com/v1/traces",
    service="osis",
    region="us-east-1",
)

register(service_name="my-agent", exporter=exporter)

Use the @observe decorator to trace agent functions and enrich() to add model metadata:

@observe(op=Op.EXECUTE_TOOL)
def get_weather(city: str) -> dict:
    return {"city": city, "temp": 22, "condition": "sunny"}

@observe(op=Op.INVOKE_AGENT)
def assistant(query: str) -> str:
    enrich(model="gpt-4o", provider="openai")
    data = get_weather("Paris")
    return f"{data['condition']}, {data['temp']}C"

result = assistant("What's the weather?")

The SDK also supports auto-instrumentation for OpenAI, Anthropic, Amazon Bedrock, LangChain, LlamaIndex, and others. Because the instrumentation is built on OpenTelemetry standards, any agent framework that emits spans with gen_ai.* attributes is compatible with OpenSearch UI.

Scenario 2: Investigating a microservice issue

AI agents are only one part of most production environments. The same interface surfaces telemetry from conventional microservices, where the troubleshooting workflow follows a more familiar path.

Your ecommerce checkout begins paging during a busy traffic window. From OpenSearch UI, navigate to APM Services in the left navigation pane. Every instrumented service is listed alongside its health indicators. The checkout service shows an elevated error rate.

Select the affected service. The detail view shows Request, Error, and Duration (RED) metrics: request rate is climbing, fault rate has spiked in the last 15 minutes, and p99 duration has doubled. You can see exactly when the degradation started.

Drill into the correlated spans for the affected time window. The span list shows multiple failed requests, all hitting the same endpoint. Select one to see the full trace waterfall. The checkout service called prepareOrder, which failed trying to retrieve a product from the catalog. The error message in the span details tells you exactly what went wrong, that’s your root cause.

Checking the infrastructure with PromQL

In both scenarios, the natural next question is whether the problem originates in the application or in the infrastructure beneath it. With the new Amazon Managed Service for Prometheus integration, you can answer that question without leaving OpenSearch UI.

Prometheus metrics are now queryable directly from the same workspace using native PromQL syntax, alongside the logs and traces you’ve already been navigating.

For the database timeout in Scenario 2, run a PromQL query to check the database instance’s read/write throughput for the same time window. For the agent latency issue in Scenario 1, check the LLM endpoint’s response time metrics to see if the slowness originates from the model provider.

This is a key architectural decision: metrics continue to live in Amazon Managed Service for Prometheus, logs and traces continue to live in Amazon OpenSearch Service, and neither signal is copied or warehoused into a second store. Each backend remains the single store for the data type it’s purpose-built to handle, while OpenSearch UI federates queries across both at runtime. The cost, retention, and operational model of each store stay intact while the troubleshooting workflow collapses into a single interface.

To configure the OpenTelemetry Collector and OpenSearch Ingestion pipelines that route metrics into Amazon Managed Service for Prometheus, see Ingesting application telemetry.

How it’s wired together

The following diagram shows the end-to-end architecture. Applications instrumented with OpenTelemetry send traces, logs, and metrics over OTLP to Amazon OpenSearch Ingestion. OpenSearch Ingestion routes each signal to the appropriate store: traces and logs land in Amazon OpenSearch Service, while metrics flow into Amazon Managed Service for Prometheus. OpenSearch UI then queries both stores to render the Application Map, Services catalog, Agent Traces, and Metrics views.

The entire experience rests on open-source foundations, Prometheus for metrics, OpenSearch for logs and traces, and OpenTelemetry for instrumentation, so teams already running an OpenTelemetry collector can adopt it by updating the collector’s export configuration to point at Amazon OpenSearch Ingestion, with no proprietary agents or rewritten instrumentation required.

Getting started

To enable these capabilities, log in to OpenSearch UI’s observability workspace, select the Gear icon in the bottom left corner to open Settings and setup, and verify that the Observability:apmEnabled toggle is on under the Observability section. OpenSearch UI is available at no additional charge for Amazon OpenSearch Service customers.

Explore locally first. The OpenSearch Observability Stack gives you a fully configured environment including application monitoring, agent tracing, and Prometheus integration, running on your machine with a single install command. It ships with sample instrumented services, including a multi-agent travel planner, so you can explore the full workflow with real telemetry data out of the box.

For AI agent development. Agent Health is an open-source, evaluation-driven observability tool designed for local development. It gives you execution flow graphs, token tracking, and tool invocation visibility right in your development loop, before you push to production.

For production. The Python SDK provides one-line setup and decorator-based tracing with gen_ai semantic conventions, with auto-instrumentation support for OpenAI, Anthropic, Amazon Bedrock, LangChain, LlamaIndex, and others. See the Amazon OpenSearch Service documentation and the Amazon Managed Service for Prometheus integration guide for the full managed experience.

About the authors

Migrate to Apache Flink 2.2 on Amazon Managed Service for Apache Flink

Francisco Morillo — Mon, 27 Apr 2026 17:57:34 +0000

Migrating to Apache Flink 2.2 on Amazon Managed Service for Apache Flink gives you access to Java 17 runtime, faster checkpoints and recovery through RocksDB 8.10.0, and SQL-native artificial intelligence and machine learning (AI/ML) inference. If you run Flink 1.x today, you might be dealing with an aging Java 11 runtime that will no longer receive standard support by the end of this year, slower state backend performance, and a fragmented API surface split across DataSet, DataStream, and legacy connector interfaces. Flink 2.2 addresses these gaps in a single major version upgrade.

Apache Flink is an open source distributed processing engine for stream and batch data, with first-class support for stateful processing and event-time semantics. Amazon Managed Service for Apache Flink removes the operational overhead of running Flink. You provide your application code, and the service provisions, scales, checkpoints, and patches the infrastructure for you.

In this post, we explain what’s new in Amazon Managed Service for Apache Flink 2.2, provide a guided migration using CLI commands, console instructions, and code examples, and show you how to monitor the upgrade and roll back if needed.

Before you upgrade: Flink 2.2 removes the DataSet API, drops Java 11 support, and replaces legacy connector interfaces. We recommend reviewing the Upgrading to Flink 2.2: Complete Guide and the State Compatibility Guide for Flink 2.2 Upgrades before upgrading production applications.

What’s new in Amazon Managed Service for Apache Flink 2.2

This release spans runtime upgrades, SQL, and Table API capabilities. The following sections break down each area.

Runtime and performance

These changes improve application performance and bring your runtime up to current standards.

Java 17 runtime – Flink 2.2 requires Java 17. Build your application code with JDK 17 for better garbage collection, a more secure runtime, and modern language features like sealed classes and records. Java 11 is no longer supported.
Python 3.12 – Flink 2.2 requires Python 3.9+, with Python 3.12 as the default. Python 3.8 is no longer supported.
RocksDB 8.10.0 – Your stateful applications benefit from improved I/O performance with the upgraded state backend, resulting in faster checkpoints and recovery.
Dedicated collection serializers – Improved serializers for Map, List, and Set types reduce serialization overhead, which lowers checkpoint sizes for applications that use these data structures frequently.
Kryo 5.6 – Kryo upgrades from version 2.24–5.6. This has state compatibility implications covered in the migration section.

SQL and Table API highlights

With Flink 2.2, you can:

Call Machine Leaning (ML) models directly from SQL using ML_PREDICT and CREATE MODEL
Work with semistructured data through the native VARIANT type
Build stateful event-driven logic in SQL with ProcessTableFunction
Run more efficient streaming joins with StreamingMultiJoinOperator and Delta Join

For details on these features, see the Apache Flink 2.2 release documentation.

Migrating from Flink 1.x to 2.2

In-place version upgrades

You can upgrade a running Flink 1.x application to 2.2 using the UpdateApplication API, the AWS Management Console, AWS CloudFormation, the AWS SDK, and Terraform Modules. The upgrade preserves your application configuration, logs, metrics, tags, and, if your state and binaries are compatible.

Auto-rollback

With auto-rollback turned on, binary incompatibilities detected during job startup trigger an automatic revert to the previous Flink version within minutes, with no manual intervention required. For state incompatibilities that surface as restart loops after a successful upgrade, invoke the Rollback API to return to your previous version and state.

Unsupported open source features

The following Flink 2.2 features aren’t currently supported in Amazon Managed Service for Apache Flink because they’re still considered experimental: Materialized Tables, ForSt State Backend (disaggregated state storage), Java 21, and custom metric reporters/telemetry configurations. We continue to evaluate these features as they mature in the Apache Flink project and will share updates on availability. You can have a closer look to which features are supported in Apache Flink 2.2 features supported

Now that you know what’s changed, the next section walks through the migration process.

Prerequisites

Before starting the migration, confirm that you have the following in place:

An existing Apache Flink 1.x application running on Amazon Managed Service for Apache Flink.
JDK 17 installed in your local build environment.
The AWS Command Line Interface (AWS CLI) installed and configured with permissions to call the kinesisanalyticsv2 APIs (UpdateApplication, CreateApplicationSnapshot, DescribeApplication, RollbackApplication).
An Amazon Simple Storage Service (Amazon S3) bucket to upload your updated application JAR.

We recommend testing each phase on a non-production replica of your application before applying the same steps to production.

Step 1: Update your application code

Start by updating your Flink dependencies to version 2.2.0 and replacing deprecated APIs. The following sections show the most common changes.

Update your pom.xml:

<properties>
    <flink.version>2.2.0</flink.version>
    <java.version>17</java.version>
</properties>

Replace legacy Kinesis connectors:

Flink 2.2 removes the FlinkKinesisConsumer and FlinkKinesisProducer classes. The following example shows how to migrate to the FLIP-27 based KinesisStreamsSource.Before (Flink 1.x):

FlinkKinesisConsumer<String> consumer = new FlinkKinesisConsumer<>(
    "my-stream",
    new SimpleStringSchema(),
    consumerConfig);
env.addSource(consumer);

After (Flink 2.2):

KinesisStreamsSource<String> source = KinesisStreamsSource.<String>builder()
    .setStreamArn("arn:aws:kinesis:us-east-1:123456789012:stream/my-stream")
    .setDeserializationSchema(new SimpleStringSchema())
    .build();
env.fromSource(source, WatermarkStrategy.noWatermarks(), "Kinesis Source");

Update connector dependencies:

The following AWS connectors have Flink 2.x-compatible releases:

Connector	Flink 2.x Artifact	Version
Apache Kafka	flink-connector-kafka	4.0.0-2.0
Amazon Kinesis Data Streams	flink-connector-aws-kinesis-streams	6.0.0-2.0
Amazon Data Firehose	flink-connector-aws-kinesis-firehose	6.0.0-2.0
Amazon DynamoDB	flink-connector-dynamodb	6.0.0-2.0
Amazon Simple Queue Service (Amazon SQS)	flink-connector-sqs	6.0.0-2.0

During writing, the JDBC, OpenSearch, and Prometheus connectors don’t yet have Flink 2.x-compatible releases. For the latest versions, see the Amazon Managed Service for Apache Flink connector documentation.

Beyond connector updates, make the following code changes:

Replace DataSet API usage with the DataStream API or Table API/SQL.
Replace Scala API usage with the Java API.
Verify that your build targets JDK 17.

Build your updated application JAR and upload it to Amazon S3 with a different file name than your current JAR (for example, my-app-flink-2.2.jar).

Step 2: Check state compatibility

Before upgrading, assess whether your application state is compatible with Flink 2.2. The Kryo upgrade from version 2.24 to 5.6 changes the binary format of serialized state. Applications using POJOs with Java collections (HashMap, ArrayList, HashSet) are the most common source of incompatibility.

Quick compatibility check:

Serialization type	Compatible?
Avro (SpecificRecord, GenericRecord)	Yes
Protobuf	Yes
POJOs without collections	Yes
Custom TypeSerializers (no Kryo delegation)	Yes
POJOs with Java collections	No
Scala case classes	No
Types using Kryo fallback	No

Check your logs for Kryo fallback:

Search your application logs for this pattern, which indicates a type is falling back to Kryo serialization:Class class <className> cannot be used as a POJO type

Step 3: Turn on auto-rollback and automatic snapshots

Turn on auto-rollback so the service automatically reverts to the previous version if the upgrade fails. Also, verify that automatic snapshots are turned on. The service takes a snapshot before the upgrade that serves as your rollback point.

Check current settings:

aws kinesisanalyticsv2 describe-application \
    --application-name MyApplication \
    --query 'ApplicationDetail.ApplicationConfigurationDescription.{
        AutoRollback: ApplicationSystemRollbackConfigurationDescription.RollbackEnabled,
        AutoSnapshots: ApplicationSnapshotConfigurationDescription.SnapshotsEnabled
    }'

Turn on both if they’re not already active:

aws kinesisanalyticsv2 update-application \
    --application-name MyApplication \
    --current-application-version-id <version-id> \
    --application-configuration-update '{
        "ApplicationSystemRollbackConfigurationUpdate": {
            "RollbackEnabledUpdate": true
        },
        "ApplicationSnapshotConfigurationUpdate": {
            "SnapshotsEnabledUpdate": true
        }
    }'

Step 4: Take a manual snapshot (recommended)

Although the upgrade process takes an automatic snapshot, taking a manual snapshot gives you a named restore point that you can quickly identify.

aws kinesisanalyticsv2 create-application-snapshot \
    --application-name MyApplication \
    --snapshot-name pre-flink-2.2-upgrade

Verify that the snapshot is ready before proceeding:

aws kinesisanalyticsv2 describe-application-snapshot \
    --application-name MyApplication \
    --snapshot-name pre-flink-2.2-upgrade

Wait until SnapshotStatus is READY.

Step 5: Run the upgrade

Run the upgrade while the application is in RUNNING or READY (stopped) state. The following example upgrades a running application and points to the new JAR.

AWS CLI:

aws kinesisanalyticsv2 update-application \
    --application-name MyApplication \
    --current-application-version-id <version-id> \
    --runtime-environment-update FLINK-2_2 \
    --application-configuration-update '{
        "ApplicationCodeConfigurationUpdate": {
            "CodeContentUpdate": {
                "S3ContentLocationUpdate": {
                    "FileKeyUpdate": "my-app-flink-2.2.jar"
                }
            }
        }
    }'

AWS Management Console:

To upgrade from the console, follow these steps:

Navigate to your application in the Amazon Managed Service for Apache Flink console.
Choose Configure.
Select the Flink 2.2 runtime.
Point to your new application JAR on Amazon S3.
Select the snapshot to restore from (use Latest to start from the most recent snapshot).
Choose Update.

AWS CloudFormation:

Update the RuntimeEnvironment field in your template. AWS CloudFormation now performs an in-place update instead of deleting and recreating the application.

Terraform:

If you manage your Flink application with Terraform, you can perform the same in-place upgrade by updating the runtime_environment and code reference in your aws_kinesisanalyticsv2_application resource. Note: Terraform support for FLINK-2_2 requires AWS provider version 6.40.0 or later (released April 8, 2026). Earlier provider versions don’t recognize this runtime value. First, update your provider version constraint:

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 6.40.0"
    }
  }
}

Then run terraform init -upgrade to pull the new provider.Next, update your application resource. Change runtime_environment from “FLINK-1_20” to “FLINK-2_2” and point to your new JAR:

resource "aws_kinesisanalyticsv2_application" "my_app" {
  name                   = "MyApplication"
  runtime_environment    = "FLINK-2_2"
  service_execution_role = aws_iam_role.flink.arn
  application_configuration {
    application_code_configuration {
      code_content_type = "ZIPFILE"
      code_content {
        s3_content_location {
          bucket_arn = aws_s3_bucket.app_code.arn
          file_key   = "my-app-flink-2.2.jar"
        }
      }
    }
    application_snapshot_configuration {
      snapshots_enabled = true
    }
    flink_application_configuration {
      checkpoint_configuration {
        configuration_type = "DEFAULT"
      }
      monitoring_configuration {
        configuration_type = "CUSTOM"
        log_level          = "INFO"
        metrics_level      = "APPLICATION"
      }
      parallelism_configuration {
        auto_scaling_enabled = true
        configuration_type   = "CUSTOM"
        parallelism          = 4
        parallelism_per_kpu  = 1
      }
    }
  }
}

Run the upgrade:

terraform plan    # Review the in-place update
terraform apply   # Apply the runtime change

Terraform will perform an in-place update of the application, changing the runtime version and code location. The application will restart with the new Flink 2.2 runtime. To roll back with Terraform, revert runtime_environment to “FLINK-1_20”, point file_key back to your original JAR, and run terraform apply again. Note that you cannot restore a Flink 2.2 snapshot on Flink 1.x, so the rollback will start from the last Flink 1.x snapshot.

Important Terraform considerations:

Auto-rollback and the RollbackApplication API aren’t directly exposed as Terraform resource attributes. If you need auto-rollback during the upgrade, enable it using the AWS CLI (Step 3) before running terraform apply, or use a provisioner/null_resource to call the CLI.
Always take a manual snapshot (Step 4) before running terraform apply for the upgrade. Terraform doesn’t automatically snapshot before updating the runtime.

Step 6: Monitor the upgrade

After initiating the upgrade, monitor the application to verify that it completes successfully.

Check application status:

The application should transition through RUNNING → UPDATING → RUNNING. Confirm the runtime version changed to 2.2:

aws kinesisanalyticsv2 describe-application \
    --application-name MyApplication \
    --query 'ApplicationDetail.RuntimeEnvironment'

What to watch for:

Scenario	What happens	Action
Binary incompatibility	Upgrade operation fails. Auto-rollback reverts to the previous version automatically.	Check operation logs for the exception, fix your code, and retry.
State incompatibility	Upgrade appears to succeed but the application enters restart loops.	Monitor `numRestarts` metric. If restarts are continuous, invoke the Rollback API manually. Review the [State Compatibility Guide].
Successful upgrade	`numRestarts` is zero, uptime is increasing, checkpoints are completing.	Proceed to validation.

Key CloudWatch metrics to monitor:

numRestarts: should be zero after upgrade
lastCheckpointDuration: should be similar to pre-upgrade values
numberOfFailedCheckpoints: should remain at zero
uptime: should be steadily increasing

Step 7: Validate application behavior

After the application is running on Flink 2.2:

Confirm that data is being read from sources and written to sinks.
Compare the output with your pre-upgrade baseline.
Monitor latency, throughput, checkpoint duration, and resource utilization.
Run for at least 24 hours to confirm stable behavior: no memory leaks, no unexpected restarts, consistent checkpoint sizes.

Step 8: Rollback (if needed)

If the application is running but is unhealthy after the upgrade, invoke the Rollback API:

AWS CLI:

aws kinesisanalyticsv2 rollback-application \
    --application-name MyApplication \
    --current-application-version-id <version-id>

AWS Management Console:

Navigate to your application.
Choose Actions, Roll back.
Confirm the rollback.

During rollback, the application stops, reverts to the previous Flink version and application code, and restarts from the snapshot taken before the upgrade.

Important: You can’t restore a Flink 2.2 snapshot on Flink 1.x. Rollback uses the snapshot taken before the upgrade. This is why Steps 3 and 4 are critical.

Next steps

Your path depends on where you are today:

If you’re new to Apache Flink: Start with the guide to choosing the right API and language, the Amazon Managed Service for Apache Flink getting started guide, and the Amazon Managed Service for Apache Flink workshop.
If you’re running Flink 1.x in production: Follow the migration steps in this post on a non-production replica first, then apply to production. For the complete reference, see the Upgrading to Flink 2.2: Complete Guide and the State Compatibility Guide for Flink 2.2 Upgrades.
If you’re evaluating Flink 2.2 features: Launch a new application on the Flink 2.2 runtime to explore SQL/ML capabilities, the VARIANT data type, and the new join operators. See the Amazon Managed Service for Apache Flink sample applications on GitHub for reference architectures.
If you need help with your migration: Use the Kiro Power and Agent Skill for Amazon Managed Service for Apache Flink to identify compatibility issues in your existing codebase and receive guidance on refactoring steps. You can also open a case through AWS Support, post a question on AWS re:Post for Amazon Managed Service for Apache Flink, or reach out through the Apache Flink community.

For the Apache Flink 2.2 documentation, see nightlies.apache.org/flink/flink-docs-release-2.2. For Amazon Managed Service for Apache Flink documentation, see the Developer Guide. For pricing, see the pricing page.

Conclusion

With Apache Flink 2.2 on Amazon Managed Service for Apache Flink, you get a modern Java 17 runtime, SQL-native AI/ML inference, improved state management performance, and a streamlined API surface. In-place upgrades with state preservation and auto-rollback make the migration straightforward. Test on a replica, follow the steps in this post, and start building on Flink 2.2.

About the authors

Using Apache Sedona with AWS Glue to process billions of daily points from a geospatial dataset

Ruan Roloff — Wed, 22 Apr 2026 15:42:28 +0000

Data strategy can use geospatial data to provide organizations with insights for decision-making and operational optimization. By incorporating geospatial data (such as GPS coordinates, points, polygons and geographic boundaries), businesses can uncover patterns, trends, and relationships that might otherwise remain hidden across multiple industries, from aviation and transportation to environmental studies and urban planning. Processing and analyzing this geospatial data at scale can be challenging, especially when dealing with billions of daily observations.

In this post, we explore how to use Apache Sedona with AWS Glue to process and analyze massive geospatial datasets.

Introduction to geospatial data

Geospatial data is information that has a geographic component. It describes objects, events, or phenomena along with their location on the Earth’s surface. This data includes coordinates (latitude and longitude), shapes (points, lines, polygons), and associated attributes (such as the name of a city or the type of road).

Key types of geospatial geometries (and examples of each in parentheses) include:

Point – Represents a single coordinate (a weather station).
MultiPoint – A collection of points (bus stops in a city).
LineString – A series of points connected in a line (a river or a flight path).
MultiLineString – Multiple lines (multiple flight routes).
Polygon – A closed area (the boundary of a city).
MultiPolygon – Multiple polygons (national parks in a country).

Geospatial datasets come in different formats, each designed to store and represent different types of geographic information. Common formats for geospatial data are vector formats (Shapefile, GeoJSON), raster formats (GeoTIFF, ESRI Grid), GPS formats (GPX, NMEA), web formats (WMS, GeoRSS) among others.

Core concepts of Apache Sedona

Apache Sedona is an open-source computing framework for processing large-scale geospatial data. Built on top of Apache Spark, Sedona extends Spark’s capabilities to handle spatial operations efficiently. At its core, Sedona introduces several key concepts that enable distributed spatial processing. These include Spatial Resilient Distributed Datasets (SRDDs), which allow for the distribution of spatial data across a cluster, and Spatial SQL, which provides a familiar SQL-like interface for spatial queries. Some of the core capabilities of Apache Sedona are:

Efficient spatial data types like points, lines and polygons.
Spatial operations and functions such as ST_Contains (check if point is inside of a polygon), ST_Intersects (check if point is inside of a polygon), ST_H3CellIDs (geospatial indexing system developed by Uber, return the H3 cell ID(s) that contain the given point at the specified resolution).
Spatial joins to combine different spatial datasets.
Integration with Spark SQL (geospatial functions to run spatial SQL queries).
Spatial indexing techniques, such as quad-trees and R-trees, to optimize query performance.

For more information about the functions available in Apache Sedona, visit the official Sedona Functions documentation.

Use case

This use case consists of a global air traffic visualization and analysis platform that processes and displays real-time or historical aircraft tracking data on an interactive world map. Using unique aircraft identifiers from the International Civic Aviation Organization (ICAO), the system ingests trajectory records containing information such as geographic position (latitude and longitude), altitude, speed, and flight direction, then transforms this raw data into two complementary visual layers. The Flight Tracks Layer plots the routes traveled by each aircraft individually, allowing for the analysis of specific trajectories and navigation patterns. The Flight Density Layer uses hexagonal spatial indexing (H3) to aggregate and identify regions of higher air traffic concentration worldwide, revealing busy air corridors, aviation hubs, and high-density flight zones.

The dataset used for this use case is historical flight tracker data from ADSB.lol. ADSB.lol provides unfiltered flight tracker with a focus on open data. Data is also freely available via the API. The data contains a file per aircraft, a JSON gzip file containing the data for that aircraft for the day.

This is a JSON trace file format sample:

{
    icao: "0123ac", // hex id of the aircraft
    timestamp: 1609275898.495, // unix timestamp in seconds since epoch (1970)
    trace: [
        [ seconds after timestamp,
            lat,
            lon,
            altitude in ft or "ground" or null,
            ground speed in knots or null,
            track in degrees or null, (if altitude == "ground", this will be true heading instead of track)
            flags as a bitfield: (use bitwise and to extract data)
                (flags & 1 > 0): position is stale (no position received for 20 seconds before this one)
                (flags & 2 > 0): start of a new leg (tries to detect a separation point between landing and takeoff that separates flights)
                (flags & 4 > 0): vertical rate is geometric and not barometric
                (flags & 8 > 0): altitude is geometric and not barometric
             ,
            vertical rate in fpm or null,
            aircraft object with extra details or null,
            type / source of this position or null,
            geometric altitude or null,
            geometric vertical rate or null,
            indicated airspeed or null,
            roll angle or null
        ],
    ]
}

For this use case, this is a simplified schema of the dataset after processing:

icao - Unique aircraft identifier
timestamp - Epoch timestamp of the observation (converted to readable format)
trace.lat / trace.lon - Latitude and longitude of the aircraft
trace.altitude - Aircraft altitude
trace.ground_speed - Ground speed
geometry - Geospatial geometry of the observation point (Point)

Solution overview

This solution enables aircraft tracking and analysis. The data can be visualized on maps and used for aviation management and safety applications. The process begins with data acquisition, extracting the compressed JSON files from TAR archives, then transforms this raw data into geospatial objects, aggregating them into H3 cells for efficient analysis. The processed data schema includes ICAO aircraft identifiers, timestamps, latitude/longitude coordinates, and derived fields such as H3 cell identifiers and point counts per cell. This structure allows detailed tracking of individual flights and aggregate analysis of traffic patterns. For visualization, you can generate density maps using the H3 grid system and create visual representations of individual flight tracks. The architecture data flow is as follows:

Data ingestion – Aircraft observation data stored as JSON compressed files in Amazon Simple Storage Service (Amazon S3).
Data processing – AWS Glue jobs using Apache Sedona for geospatial processing.
Data visualization – Spark SQL with Sedona’s spatial functions to extract insights and export data to visualize the information in a map on Kepler.gl.

The following figure illustrates this solution.

Prerequisites

You will need the following for this solution:

An AWS Account and a user with AWS Console access.
Access to a Linux terminal and the AWS Command Line Interface (AWS CLI).
An IAM role for AWS Glue with list, read, and write permissions for Amazon S3 buckets.
An Amazon S3 Bucket for flight files. For this example, name the bucket blog-sedona-nessie-<account_number>-<aws_region>, using your account number and region.
An Amazon S3 bucket for artifacts and Sedona libraries. For this example, name the bucket blog-sedona-artifacts-<account_number>-<aws_region>, using your account number and region.
Download a day of historical data from ADSB.lol. In our examples, we used v2025.05.29-planes-readsb-prod-0tmp.tar.aa and v2025.05.29-planes-readsb-prod-0tmp.tar.ab.
Download the Apache Sedona libraries. The example was created using sedona-spark-shaded-3.5_2.12-1.7.1.jar and geotools-wrapper-1.7.1-28.5.jar.
Download the AWS Glue script from AWS Sample to process the geospatial data.
Review the AWS Glue security best practices, especially IAM least-privilege, encryption for sensitive data at rest and in transit, and configuring VPC Endpoints to prevent data from routing through the public internet.

Solution walkthrough

From now on, executing the next steps will incur costs on AWS. This step-by-step walkthrough demonstrates an approach to processing and analyzing large-scale geospatial flight data using Apache Sedona and Uber’s H3 spatial indexing system, using AWS Glue for distributed processing and Apache Sedona for efficient geospatial computations. It explains how to ingest raw flight data, transform it using Sedona’s geospatial functions, and index it with H3 for optimized spatial queries. Finally, it also demonstrates how to visualize the data using Kepler.gl. For data processing, it is possible to use both Glue scripts and Glue notebooks. In this post, we will focus only on Glue scripts.

Upload the Apache Sedona libraries to Amazon S3

Open your OS terminal command line.

Create a folder to download the Sedona libraries and name it jar.


	# Create a directory for the Sedona libraries (JARs files)
	mkdir jar
	# Go to the folder JARs folder
	cd jar

Download the Apache Sedona libraries.


	# Download required Sedona libraries (JARs files)
	wget https://repo1.maven.org/maven2/org/apache/sedona/sedona-spark-shaded-3.5_2.12/1.7.1/sedona-spark-shaded-3.5_2.12-1.7.1.jar
	wget https://repo1.maven.org/maven2/org/datasyslab/geotools-wrapper/1.7.1-28.5/geotools-wrapper-1.7.1-28.5.jar

Upload the Sedona libraries (JARs files) to Amazon S3. In this example, we use the S3 path s3://aws-blog-post-sedona-artifacts/jar/.


	# Upload the JARs files to Amazon S3 bucket
	aws s3 cp . s3://blog-sedona-artifacts-<account_number>-<aws_region>/jar/ --recursive

Your Amazon S3 folder should now look similar to the following image:

Download and upload the geospatial data to Amazon S3

Open your OS terminal command line.

Create a folder to download the flight files and name it adsb_dataset.

		# Create a directory for download the geospatial flight files
		mkdir adsb_dataset
		# Go to the folder for geospatial flight files
		cd adsb_dataset

Download the flight files data from adsblol GitHub repository.

	# Download the geospatial flight files in the folder created
	wget https://github.com/adsblol/globe_history_2025/releases/download/v2025.05.29-planes-readsb-prod-0tmp/v2025.05.29-planes-readsb-prod-0tmp.tar.aa
	wget https://github.com/adsblol/globe_history_2025/releases/download/v2025.05.29-planes-readsb-prod-0tmp/v2025.05.29-planes-readsb-prod-0tmp.tar.ab

Extract the flight files.

	# Combine the two the tar files together
	cat v2025.05.29* >> combined.tar
	# Extract the json flight files from the tar file
	tar xf combined.tar

Copy the flight files to Amazon S3. In this case, we are using the S3 folder: s3://blog-sedona-nessie-<account_number>-<aws_region>/raw/adsb-2025-05-28/traces/.

	# Copy the json flight files to Amazon S3
	aws s3 cp ./traces/ s3://blog-sedona-nessie-<account_number>-<aws_region>/raw/adsb-2025-05-28/traces/ --recursive

Your Amazon S3 folder should now look similar to the following image.

Create an AWS Glue job and set up the job

Now, we are ready to define the AWS Glue job using Apache Sedona to read the geospatial data files. To create a Glue job:

Open the AWS Glue console.
On the Notebooks page, choose Script editor.

On the Script screen, for the engine, choose Spark, then select the option Upload script.
Choose Choose file. Find the process_sedona_geo_track.py file, then choose Create script.

Rename the job from Untitled to process_sedona_geo_track.
Choose Save.
Now, let’s set up the AWS Glue job. Choose Job Details.
Choose the IAM Role created to be used with Glue. For this example, we use blog-glue.
Set the Glue version to Glue 5.0 and the Worker type as needed. For this example, G.1X is sufficient, but we use G.2X to speed up processing.

Now, let’s import the libraries for Apache Sedona.
In the Dependent JARs path, type the path of the JAR files for Apache Sedona that you uploaded in the preceding steps. For this example, we used s3://blog-sedona-artifacts-<account_number>-<aws_region>/jar/sedona-spark-shaded-3.5_2.12-1.7.1.jar,s3://blog-sedona-artifacts-<account_number>-<aws_region>/jar/geotools-wrapper-1.7.1-28.5.jar
In Additional Python modules path, enter the modules for Apache Sedona: apache-sedona==1.7.1,geopandas==0.13.2,shapely==2.0.1,pyproj==3.6.0,fiona==1.9.5,rtree==1.2.0

In the Job parameters section, in the Key field, type —BUCKET_NAME. For its Value, enter your bucket name. In this example, ours is blog-sedona-nessie-<account_number>-<aws_region>.

Choose Save.

Processing the geospatial flights data

Before we run the job, let’s understand how the code works. First, import the Apache Sedona libraries:

import json 
import gzip 
from sedona.spark import SedonaContext

Next, initialize the Sedona context using an existing Spark session:

sedona = SedonaContext.create(spark)

After that, create a function for handling compressed JSON data:

def parse_gzip_json(byte_content):
        try:
            decompressed = gzip.decompress(byte_content)
            return json.loads(decompressed.decode('utf-8'))
        except Exception as e:
            print(f"Error during gzip parse: {str(e)}")
            return None

Add a function to transform raw tracking data into a structured format suitable for a valid coordinates process:

def flatten_records(json_obj):
    records = []
    if "trace" in json_obj and isinstance(json_obj["trace"], list):
        for point in json_obj["trace"]:
            if len(point) >= 3:
                lat, lon = float(point[1]), float(point[2])
                if -90 <= lat <= 90 and -180 <= lon <= 180:
                    records.append(Row(
                        icao=json_obj.get("icao", None),
                        timestamp=json_obj.get("timestamp", None),
                        lat=lat,
                        lon=lon
                    ))
    return records

The flat_rdd variable applies these functions to the structured data from the original gzipped JSON. Each element in this RDD is a Row object representing a single data point from an aircraft’s trace, with fields for ICAO, timestamp, latitude, and longitude.

flat_rdd = raw_rdd.map(lambda x: parse_gzip_json(x[1])).filter(lambda x: x is not None).flatMap(flatten_records)

The ADSB trace files contain a deeply nested JSON structure where the trace field holds an array of mixed-type arrays, compressed in Gzip format. For this specific case, developing a UDF represented one of the most practical and efficient solutions. Since Gzip is a non-splittable format, Spark is unable to parallelize processing, constraining both methods to a single worker per file and processing the data multiple times across JVM decompression, full JSON parsing, and subsequent re-parsing operations. The UDF bypasses all of this by reading raw bytes and doing everything in a single Python pass: decompress → parse → extract → validate, returning only the small set of needed fields directly to Spark.

The Spark SQL query processes geographic trace data using the H3 hexagonal grid system, converting point data into a regularized hexagonal grid that can help identify areas of high point density. A resolution of 5 was adopted, producing hexagons of approximately 253 km² (roughly the same size as the city of Edinburgh, Scotland, which is approximately 264 km²), for its ability to effectively capture route density patterns at the city and metropolitan level.

h3_traces_df = spark.sql("""
WITH base_h3 AS (
    SELECT
        ST_H3CellIDs(geometry, 5, false)[0] AS h3_index,
        lat,
        lon
    FROM traces
)
SELECT
    COUNT(*) AS num, -- Count points in each H3 cell
    h3_index,
    AVG(lon) AS center_lon,
    AVG(lat) AS center_lat
FROM base_h3
GROUP BY h3_index
""")

Finally, this code prepares the datasets for visualization purposes. The first dataset is based on the aircraft unique identifier. The complete dataset for a single day can contain more than 80 million data points. A random sampling rate of 0.1% was applied, which proves sufficient to illustrate route density patterns without overwhelming the Kepler.gl browser renderer. The second dataset aggregates trace points into hexagonal spatial cells (result from the query above).

points_viz_sampled = df_points.select(
    col("icao"), # Aircraft unique identifier (24-bit address)
    col("timestamp").cast("double").alias("timestamp"),
    col("lat").cast("double").alias("lat"),
    col("lon").cast("double").alias("lon")
).sample(False, 0.001)

h3_viz_csv = h3_traces_df.select(
    col("num").alias("point_count"),
    col("h3_index").cast("string").alias("h3_index"),
    col("center_lon"),
    col("center_lat")
)

Now that we understand the code, let’s run it.

Open the AWS Glue console.
On the ETL jobs >> Notebooks page, choose the job name process_sedona_geo_track.
Choose Run.

Now, it is possible to monitor the job by choosing the Runs tab.
It may take a few minutes to run the entire job. It took nearly 8 minutes to process approximately 2.50 GB (67,540 compressed files) with 20 DPUs. After the job is processed, you should see your job with the status Succeeded.

Now your data should be saved for a preview visualization demo in a folder named s3://blog-sedona-nessie-<account_number>-<aws_region>/visualization/.

Performance insights

The workload characterization of this job reveals a CPU-intensive profile, primarily because of the processing of small binary files with GZIP compression and subsequent JSON parsing. Given the inherent nature of this pipeline, which includes Python UDF serialization and partial single-partition write stages, linear scaling does not yield proportional performance gains. The following table presents an analysis of AWS Glue configurations, evaluating the trade-off between computational capacity, execution duration, and associated costs:

Duration	Capacity (DPUs)	Worker type	Glue version	Estimated Cost*
10 m 7 s	32 DPUs	G.1X	5	$2.34
11 m 50 s	10 DPUs	G.1X	5	$0.88
19 m 7 s	4 DPUs	G.1X	5	$0.59
8 m 19 s	20 DPUs	G.2X	5	$1.32

*Estimated Cost = DPUs x Duration (hours) x $0.44 per DPU-hour (us-east-1)

Visualizing and analyzing geospatial data with Kepler.gl

Kepler.gl is an open-source geospatial analysis tool developed by Uber with code available at Github. Kepler.gl is designed for large-scale data exploration and visualization, offering multiple map layers, including point, arc, heatmap, and 3D hexagon. It supports various file formats like CSV, GeoJSON, and KML. In this use case, we will use Kepler.gl to present interactive visualizations that illustrate flight patterns, routes, and densities across global airspace.

Downloading the geospatial files

Before we can view the graph, we will need to download the flight files to our local machine, unzip them, and rename them (to make it easier to identify the files).

Open your OS terminal command line.

Create the folders to download the data processed in the steps before. In this case, we create kepler and kepler_csv.

	#create kepler folders: first folder is to download the files,
	#second folder is to organize the files to use in the next step
	mkdir kepler
	mkdir kepler_csv

Replace the bracketed variables with your account and directory information, then download all the CSV files.

	#copy the files from Amazon S3 to local machine
	aws s3 cp s3://blog-sedona-nessie-<account_number>-<aws_region>/visualization/ /<user_directory>/kepler --recursive

Extract the files, rename them, and move them to another folder.

	# Extract the files processed by Spark and Sedona
	gzip -d ./kepler/kepler_h3_density/*.gz
	gzip -d ./kepler/kepler_track_points_sample/*.gz
	
	# Rename the Spark output files to more readable names
	cd ./kepler/kepler_h3_density/
	ls
	mv part-00000-*.csv kepler_h3_density.csv
	cd ..
	
	cd ./kepler/kepler_track_points_sample/
	ls
	mv part-00000-*.csv kepler_track_points_sample.csv
	cd ..
	
	# Ensure the output folder exists
	mkdir -p ../kepler_csv
	
	# Copy the renamed CSV files to the folder that will be used as input in kepler.gl
	cp ./kepler/kepler_h3_density/*.csv ../kepler_csv
	cp ./kepler/kepler_track_points_sample/*.csv ../kepler_csv

Your kepler_csv folder should look similar to the return of the command below.

	#list the files in the kepler_csv directory
	ls -l
	total 11684
	-rw-rw-r-- 1 ec2-user ec2-user 8630110 Jun 12 14:47 kepler_h3_density.csv
	-rw-rw-r-- 1 ec2-user ec2-user 3331763 Jun 12 14:47 kepler_track_points_sample.csv

Visualizing the data in a graph

Now that you have saved the data to your local machine, you can analyze the flight data through interactive map graphics. To import the data into the Kepler.gl web visualization tool:

Open the Kepler.gl Demo web application.
Load data into Kepler.gl:
1. Choose Add Data in the left panel.
2. Drag and drop both CSV files (flight_points and h3_density) into the upload area.
3. Confirm that both datasets are loaded successfully.
Delete all layers.
Create the Flight Density Layer:
1. Choose Add Layer in the left panel.
2. In Basic, choose H3 as the layer type, then add the following configuration:
  1. Layer Name: Flight Density
  2. Data Source: kepler_h3_density.csv
  3. Hex ID: h3_index
3. In the Fill Color section:
  1. Color: point_count
  2. Color Scale: Quantile.
  3. Color Range: Choose a blue/green gradient.
4. Set Opacity to 0.7.
5. In the Coverage section, set it to 0.9.
Create the Flight Tracks Layer:
1. Choose Add Layer in the left panel.
2. In Basic, choose Point as the layer type, then add the following configuration:
  1. Layer Name: Flight Tracks
  2. Data Source: kepler_track_points_sample.csv
  3. Columns:
    1. Latitude: lat
    2. Longitude: lon
3. In the Fill Color section:
  1. Solid Color: Orange
  2. Opacity: 0.3
4. Set the Point’s Radius to 1
The layers should look similar to the following figure.

The graph visualization should now show flight density through color-coded hexagons, with individual flight tracks visible as orange points:

There you go! Now that you have knowledge about geospatial data and have created your first use case, take the opportunity to do some analysis and learn some interesting facts about flight patterns.

It is possible to experiment with other interesting types of analysis in Kepler.gl, such as Time Playback.

Clean up

To clean up your resources, complete the following tasks:

Delete the AWS Glue job process_sedona_geo_track.
Delete content from the Amazon S3 buckets: blog-sedona-artifacts-<account_number>-<aws_region> and blog-sedona-nessie-<account_number>-<aws_region>.

Conclusion

In this post, we showed how processing geospatial data can present significant challenges due to its complex nature (from big data to data structure format). For this use case of flight trackers, it involves vast amounts of information across multiple dimensions such as time, location, altitude, and flight paths, however, the combination of Spark’s distributed computing capabilities and Sedona’s optimized geospatial functions helps overcome those challenges. The spatial partitioning and indexing features of Sedona, coupled with Spark’s framework, enable us to perform complex spatial joins and proximity analyses efficiently, simplifying the overall data processing workflow.

The serverless nature of AWS Glue eliminates the need for managing infrastructure while automatically scaling resources based on workload demands, making it an ideal platform for processing growing volumes of flight data. As the volume of flight data grows or as processing requirements fluctuate, with AWS Glue, you can quickly adjust resources to meet demand, ensuring optimal performance without the need for cluster management.

By converting the processed results into CSV format and visualizing them in Kepler.gl, it is possible to create interactive visualizations that reveal patterns in flight paths, and you can efficiently analyze air traffic patterns, routes, and other insights. This end-to-end solution demonstrates how a modern data strategy in AWS with the support of open-source tools can transform raw geospatial data into actionable insights.

About the authors

Analyzing your data catalog: Query SageMaker Catalog metadata with SQL

Ramesh H Singh — Wed, 22 Apr 2026 15:37:38 +0000

As your data and machine learning (ML) assets grow, tracking which assets lack documentation or monitoring asset registration trends becomes challenging without custom reporting infrastructure. You need visibility into your catalog’s health, without the overhead of managing ETL jobs. The metadata feature of Amazon SageMaker provides this capability to users. Converting catalog asset metadata into Apache Iceberg tables stored in Amazon S3 Tables removes the need to build and maintain custom ETL pipelines. Your team can then query asset metadata directly using standard SQL tools. You can now answer governance questions like asset registration trends, classification status, and metadata completeness using standard SQL queries through tools like Amazon Athena, Amazon SageMaker Unified Studio notebooks, and BIsystems.

This automated approach reduces ETL development time and gives your team visibility into catalog health, compliance gaps, and asset lifecycle patterns. The exported tables include technical metadata, business metadata, project ownership details, and timestamps, partitioned by snapshot date to enable time travel queries and historical analysis. Teams can use this capability to proactively monitor catalog health, identify gaps in documentation, track asset lifecycle patterns, and make sure that governance policies are consistently applied.

How metadata export works

After you enable the metadata export feature, it runs automatically on a daily schedule:

SageMaker Catalog creates the infrastructure — An Amazon Simple Storage Service (Amazon S3) table bucket named aws-sagemaker-catalog is created with an asset_metadata namespace and an empty asset table.
Daily snapshots are captured — A scheduled job runs once per day around midnight (local time per AWS Region) to export updated asset metadata.
Metadata is structured and partitioned — The export captures technical metadata (resource_id, resource_type), business metadata (asset_name, business_description), project ownership details, and timestamps, partitioned by snapshot_date for query performance.
Data becomes queryable — Within 24 hours, the asset table appears in Amazon SageMaker Unified Studio under the aws-sagemaker-catalog bucket and becomes accessible through Amazon Athena, Studio notebooks, or external BI tools.
Teams query using standard SQL — Data teams can now answer questions like “How many assets were registered last month?” or “Which assets lack business descriptions?” without building custom ETL pipelines.

The export evaluates catalog assets and their metadata properties in the domain, converting them into Apache Iceberg table format. The data flows into downstream analytics operations immediately, with no separate ETL or batch processes to maintain. The exported metadata becomes part of a queryable data lake that supports time-travel queries and historical analysis.

In this post, we demonstrate how to use the metadata export capability in Amazon SageMaker Catalog and perform analytics on these tables. We explore the following specific use-cases.

Audit historical changes to investigate what an asset looked like at a specific point in time.
Monitor asset growth view how the data catalog has grown over the last 30 days.
Track metadata improvements to see which assets gained descriptions or ownership over time.

Solution overview

Figure 1 – SageMaker catalog export to S3 Tables

The architecture consists of three key components:

Amazon SageMaker Catalog exports asset metadata daily to Amazon S3.
S3 Tables stores metadata as Apache Iceberg tables in the aws-sagemaker-catalog bucket with ACID compliance and time travel.
Query engines (Amazon Athena, Amazon Redshift, and Apache Spark) access metadata using standard SQL from the asset_metadata.asset table.

What metadata is exposed?

SageMaker Catalog exports metadata in the asset_metadata.asset table:

Metadata Type	Fields	Description
Technical metadata	`resource_id`, `resource_type_enum`, `account_id`, `region`	Resource identifiers (ARN), types (`GlueTable`, `RedshiftTable`, `S3Collection`), and location
Namespace hierarchy	`catalog`, `namespace`, `resource_name`	Organizational structure for assets
Business metadata	`asset_name`, `business_description`	Human-readable names and descriptions
Ownership	`extended_metadata['owningEntityId']`	Asset ownership information
Timestamps	`asset_created_time`, `asset_updated_time`, `snapshot_time`	Creation
Custom metadata	`extended_metadata['form-name.field-name']`	User-defined metadata forms as key-value pairs

The snapshot_time column supports point-in-time analysis and query of historical catalog states.

Prerequisites

To follow along with this post, you must have the following:

An Amazon SageMaker Unified Studio domain set up with a domain owner or domain unit owner permissions.
- A SageMaker Unified Studio domain identifier
AWS Identity and Access Management (IAM) permissions for configuring metadata export.
Grant catalog, database, and table Select and Describe permissions with AWS Lake Formation.
AWS Command Line Interface (AWS CLI) version 2.33.0 or later installed and configured
An Amazon SageMaker project for publishing assets.

For SageMaker Unified Studio domain setup instructions, refer to the SageMaker Unified Studio Getting started guide.

After you complete the prerequisites, complete the following steps.

Add this policy to our IAM user or role to enable metadata export. If using SageMaker Unified Studio to query the catalog, add this policy to the AmazonSageMakerAdminIAMExecutionRole managed role.

{ "Version": "2012-10-17", 
"Statement": [ 
{
 "Effect": "Allow",
 "Action": [ "datazone:GetDataExportConfiguration",
 "datazone:PutDataExportConfiguration"
 ],
 "Resource": "*"
 },
 {
 "Effect": "Allow",
 "Action": [
 "s3tables:CreateTableBucket",
 "s3tables:PutTableBucketPolicy"
 ],
 "Resource": "arn:aws:s3tables:*:*:bucket/aws-sagemaker-catalog" 
} 
]
}

Grant describe and select permissions for SageMaker Catalog with AWS Lake Formation. This step can be performed in the AWS Lake Formation console.
1. Select Permissions -> Data permissions and choose Grant.
  
  Figure 2 – AWS Lake Formation grant permission
2. Under Principal type, select Principals, IAM users and roles and the AWS managed AmazonSageMakerAdminIAMExecutionRole execution role.
3. Choose Named Data Catalog resources.
4. Under Catalogs, search for and select <account-id>:s3tablecatalog/aws-sagemaker-catalog.
5. Under Databases, select asset_metadata database.
  
  Figure 3 – AWS Lake Formation catalog, database, and table
  
  Figure 4 – AWS Lake Formation grant permission
6. For Table, select asset.
7. Under Table permissions, check Select and Describe.
8. Choose Grant to save the permissions.

Enable data export using the AWS CLI

Configure metadata export using the PutDataExportConfiguration API. The Amazon DataZone service automatically creates an S3 table bucket named aws-sagemaker-catalog with an asset_metadata namespace, and schedules a daily export job. Asset metadata is exported once daily around midnight local time per AWS Region.

The SageMaker Domain identifier is available on domain detail page in the AWS Management Console. Accessing the asset table through the S3 Tables console or the Data tab in SageMaker Unified Studio can require up to 24 hours.

AWS CLI command to enable SageMaker catalog export:

aws datazone put-data-export-configuration --domain-identifier <domain-id> --region <region> --enable-export

Use this AWS CLI command to validate the configuration is enabled:

aws datazone get-data-export-configuration --domain-identifier <domain-id> --region <region>
{
    "isExportEnabled": true,
    "status": "COMPLETED",
    "s3TableBucketArn": "arn:aws:s3tables:<region>:<account-id>:bucket/aws-sagemaker-catalog",
    "createdAt": "2025-11-26T18:24:02.150000+00:00",
    "updatedAt": "2026-02-23T19:33:40.987000+00:00"
}

Access the exported asset table

Navigate to Amazon SageMaker Domains in the AWS Management Console.
Select your domain and select Open.

Figure 5 – Open Amazon SageMaker Unified Studio
In SageMaker Unified Studio, choose a project from the Select a project dropdown list.
To query SageMaker catalog data, select Build in the menu bar and then choose Query Editor. To create a new project, follow the instructions in the Amazon SageMaker Unified Studio User Guide.

Figure 6 – Open SageMaker Unified Studio Query Editor

The asset_metadata.asset table is available in Data explorer. Use Data explorer to view the schema and query data to perform analytics from.

Expand Catalogs in Data explorer. Then, select and expand s3tablecatalog, aws-sagemaker-catalog, asset_metadata, and asset.
Test querying the catalog with SELECT * FROM asset_metadata.asset LIMIT 10;.

Figure 7 – Query SageMaker catalog

Queries for observability and analytics

With setup complete, execute queries to gain insights on catalog usage and changes. To monitor asset growth, and view how the data catalog has grown over the last five days:

SELECT 
    DATE (snapshot_time) as date,
    COUNT (*) as total_assets
FROM asset_metadata.asset
WHERE 
     DATE (snapshot_time) >= CURRENT_DATE - INTERVAL '5' DAY
GROUP BY DATE (snapshot_time)
ORDER BY date DESC;

Figure 8 – Query asset growth

Use the catalog to track metadata changes to determine which assets gained descriptions or ownership over time. Use this query to identify assets that gained business descriptions over the past five days by comparing today’s snapshot with the earlier snapshot.

SELECT
    t.asset_id,
    t.resource_name,
    p.business_description as description_before,
    t.business_description as description_now
FROM asset_metadata.asset t
JOIN asset_metadata.asset p ON t.asset_id = p.asset_id
WHERE DATE(t.snapshot_time) = CURRENT_DATE
    AND DATE(p.snapshot_time) = CURRENT_DATE - INTERVAL '5' DAY
    AND p.business_description IS NULL
    AND t.business_description IS NOT NULL;

Investigate asset values at a specific point in time using this query to retrieve metadata from any snapshot date.

SELECT
     asset_id,
     resource_name,
     business_description,
     extended_metadata['owningEntityId'] as owner,
     snapshot_time
FROM asset_metadata.asset
WHERE asset_id = 'your-asset-id'
     AND DATE(snapshot_time) = DATE('2025-11-26');

Clean up resources

To avoid ongoing charges, clean up the resources created in this walkthrough:

Disable metadata export:

Disable the daily metadata export to stop new snapshots:

aws datazone put-data-export-configuration \
  --domain-identifier <domain-id. \
  --no-enable-export \
  --region <region>

Delete S3 Tables resources:

Optionally, delete the S3 Tables namespace containing the exported metadata to remove historical snapshots and stop storage charges. For instructions on how to delete S3 tables, see Deleting an Amazon S3 table in the Amazon Simple Storage Service User Guide.

Conclusion

In this post, you enabled the metadata export feature of SageMaker Catalog and used SQL queries to gain visibility into your asset inventory. The feature converts asset metadata into Apache Iceberg tables partitioned by snapshot date, so you can perform time-travel queries, monitor catalog growth, track metadata completeness, and audit historical asset states. This provides a repeatable, low-overhead way to maintain catalog health and meet governance requirements over time.

To learn more about Amazon SageMaker Catalog, see the Amazon SageMaker Catalog documentation. To explore Apache Iceberg table formats and time-travel queries, see the Amazon S3 Tables documentation.

About the Authors

Configure a custom domain name for your Amazon MSK cluster enabled with IAM authentication

Mazrim Mehrtens — Tue, 21 Apr 2026 16:33:29 +0000

Most Amazon Managed Streaming for Apache Kafka (Amazon MSK) customers are simplifying and standardizing access control to Kafka resources using AWS Identity and Access Management (IAM) authentication. This adoption is also accelerated as Amazon MSK now supports IAM authentication in popular languages including Java, Python, Go, JavaScript, and .NET.

In the first part of Configure a custom domain name for your Amazon MSK cluster, we discussed about why custom domain names are important and provided details on how to configure a custom domain name in Amazon MSK when using SASL_SCRAM authentication. In this post, we discuss how to configure a custom domain name in Amazon MSK when using IAM authentication. We recommend you read the first part of this blog as it captures solution details implementation steps.

Solution overview

IAM authentication for Amazon MSK uses TLS to encrypt the Kafka protocol traffic between the client and Kafka broker. To use a custom domain name, the Kafka broker needs to present a server certificate that matches the custom domain name. To achieve this, this solution uses an Network Load Balancers (NLBs) with Amazon Certificate Manager to provide a custom certificate on behalf of the MSK brokers, and a Route 53 Private Hosted Zone to provide DNS for the custom domain name.

The following diagram shows all components used by the solution.

Certificate management

For clients to perform TLS communication with the MSK cluster the cluster needs to provide a certificate with hostnames matching the custom domain name. This solution uses a certificate in AWS Certificate Manager (ACM) signed with a Private Certificate Authority (PCA) for TLS with the custom domain name. This solution uses a certificate with bootstrap.example.com as the Common Name (CN) so that the certificate is valid for the bootstrap address, and Subject Alternative Names (SANs) are set for all broker DNS names (such as b-1.example.com). Since this solution uses a private certificate authority, the CA chain must be imported into the client trust stores.

This solution works with any server certificate, whether certificates are signed by a public or private Certificate Authority (CA). You can import existing certificates into ACM to be used with this solution. Certificates must provide a common name and/or subject alternative names that match the bootstrap DNS address as well as the individual broker DNS addresses. If the certificate is issued by a private CA, clients need to import the root and intermediate CA certificates to the client trust store. If the certificate is issued by a public CA, the root and intermediate CA certificates will be in the default trust store.

Network Load Balancer

The NLB provides the ability to use a TLS listener. The ACM certificate is associated with the listeners and enables TLS negotiation between the client and the NLB. The NLB performs a separate TLS negotiation between itself and the MSK brokers. In addition to the above architecture, this solution also allows using AWS Private Link to connect the cluster to external VPCs. This allows secure access to MSK between VPCs while using a custom domain name.

The following diagram illustrates the NLB port and target configuration. A TLS listener with port 9000 is used for bootstrap connections with all MSK brokers set as targets. IAM authentication is configured to run on port 9098 of the MSK brokers using a TLS target type. A TLS listener port is used to represent each broker in the MSK cluster. In this post, there are three brokers in the MSK cluster starting with port 9001, representing broker 1 and up to port 9003, representing broker 3.

Domain Name System (DNS)

For the client to resolve DNS queries for the custom domain, we use an Amazon Route 53 private hosted zone to host the DNS records, and associate it with the client’s VPC to enable DNS resolution from the Route 53 VPC resolver. This solution uses a private MSK cluster and private DNS. For publicly accessible MSK clusters a public NLB and DNS provider such as a Route53 public hosted zone can be used.

Amazon MSK

Finally, each broker needs to have its advertised listeners configuration (advertised.listeners) updated to match the custom domain name and NLB ports. Advertised listeners is a configuration option used by Kafka clients to connect to the brokers. By default, an advertised listener is not set. Once set, Kafka clients use the advertised listener instead of listeners to obtain the connection information for brokers. MSK brokers use the listener configuration to tell clients the DNS names and ports to use to connect to the individual brokers for each authentication type enabled. Advertised listeners are unique to each broker; and the cluster won’t start if multiple brokers have the same advertised listener address. For this reason, this solution uses a unique custom DNS name for each broker (such as, b-1.example.com).

Solution Deployment

To deploy the solution, use the CloudFormation template from the GitHub repository.

This template deploys a VPC, NLB, PCA, ACM certificate, MSK cluster, and an Amazon EC2 instance for cluster connectivity. The EC2 instance includes a script to handle updating the broker advertised.listeners settings to match the custom domain name. For more information on deploying a CloudFormation template, refer to Create a stack from the CloudFormation console.

After deploying the CloudFormation template, run the script to update advertised listeners as follows:

Retrieve the MSKClusterARN and CertificateAuthorityARN from the CloudFormation outputs for your stack as they will be used in subsequent steps.
Navigate to the EC2 console and identify the KafkaClientInstance. Choose Connect to connect to the instance using AWS Systems Manager Session Manager.
Session Manager starts a session in shell. Start a bash session with the command:
```
bash -l
```
The Kafka client SDKs have already been installed in the EC2 instance. You can update the advertised.listeners configuration as follows, replacing CLUSTER_ARN with the ARN of your MSK cluster retrieved from CloudFormation in step 1:
```
./update_advertised_listeners.sh --region us-east-1 --cluster-arn CLUSTER_ARN
```
Note that once this script completes, the brokers will have new advertised listeners configurations. Connections using the standard IAM address for the MSK service will not work until we complete the next steps, as the brokers will redirect connections over this address back to the custom domain name and TLS will fail.
Next, we need to create a truststore with the certificate for our AWS Private Certificate Authority (PCA) to allow TLS with the NLB. In the following command, replace PCA_ARN with the ARN of the PCA retrieved from CloudFormation in step 1:
We’re using the default Java truststore which uses the password changeit.When asked “Trust this certificate?” enter “yes”.
```
export PCA_ARN=<<PCA_ARN>>
export REGION=<<REGION>>

cp /etc/pki/java/cacerts . && chmod 600 cacerts
aws acm-pca get-certificate-authority-certificate --certificate-authority-arn $PCA_ARN --region $REGION | jq -r '.Certificate' > pca.pem
keytool -import -file pca.pem -alias AWSPCA -keystore cacerts
```

Create a new properties file to allow IAM authentication with our custom truststore:

cat <<EOF >> /home/ssm-user/client-iam.properties
ssl.truststore.location=/home/ssm-user/cacerts
ssl.truststore.password=changeit
EOF

Verify you can connect to the cluster using IAM authentication using our new custom domain name, replacing bootstrap.example.com with your own custom domain name if you used a different one in CloudFormation:
```
bin/kafka-topics.sh --list --command-config client-iam.properties --bootstrap-server bootstrap.example.com:9000
```

Cleanup

To stop incurring costs navigate to CloudFormation and delete the CloudFormation stack to remove all resources provisioned by CloudFormation.

Frequently Asked Question about Custom Domain Name

Customers have asked a few questions about implementing custom domain names with MSK. You can find answers to some of the most popular questions here.

Are there any limitations for this solution on MSK?

The advertised.listeners setting was removed as a dynamic broker in KRaft-based Kafka clusters. Therefore, this solution is only supported in Zookeeper-based MSK clusters. Additionally, this solution is only applicable to SASL/SCRAM and IAM-authentication based MSK clusters.

How the custom domain name solution scales when we add new brokers?

When using the NLB for broker connectivity (option 2 in the configure a custom domain name for your Amazon MSK cluster blog post), you will need to add an additional listener for each additional broker created.

For TLS, if using Subject Alternative Name (SAN) to list individual broker DNS hostnames, you will need to create a new certificate that includes the names of the additional brokers. One option is to create a certificate with SANs for more brokers than needed to allow for growth.If a wildcard certificate is used, you do not need to modify certificates when adding brokers.

What changes are required when we remove brokers?

Amazon MSK supports scale-in by removing brokers from the cluster. Brokers are removed from each availability zones (AZ). So a 6 broker Amazon MSK cluster deployed in 3 AZ can be reduced to 3 broker cluster deployed in 3 AZ. When brokers are removed, you can remove the NLB listeners for the removed broker along with the Route53 DNS endpoints. However, you can also leave them as is, or just remove the target IP from the broker numbers target group. The NLB will mark the targets as unhealthy and stop directing traffic to them. If you ever plan to scale-out the number of brokers, you can re-use the existing NLB listeners and Route 53 DNS entries and would only need to update the target IPs used in the broker numbers target group.

Is there any change in configuration required if there is any broker failure?

No. When a broker fails, Amazon MSK replaces the failed broker with a new broker instance keeping the configuration of the broker exactly the same. So, there would be no change in the advertised listener of the broker. Once the broker is healthy, the broker can accept new connections and read/write traffic.

Can you use Amazon MSK Replicator between MSK clusters in multiple AWS Regions when using the custom domain name solution?

The Amazon MSK Replicator can be used when using the custom domain name solution, either in an active-passive or active-active setup. The same process can be followed to set the custom domain name.

You then follow build multi-Region resilient Apache Kafka applications with identical topic names using Amazon MSK and Amazon MSK Replicator post to configure MSK Replicator.

The following diagram shows an active-active AWS multi-Region MSK setup using the custom domain name solution:

Can I use a global bootstrap DNS name to connect to Amazon MSK clusters deployed across multiple AWS regions when IAM authentication is enabled?

No, it is not possible to use a global bootstrap reference to represent MSK clusters deployed in multiple AWS Regions, unless the client is aware of the cluster’s region when connecting. To use IAM authentication, the correct AWS Region must be included in the IAM authentication request for a given cluster. This is because the AWS Region is a part of the Sigv4 authentication protocol used by IAM. This scope prevents the IAM authorization being used to talk to a resource in another AWS Region. You can provide the AWS Region in one of two ways– with region-specific bootstrap URLs or by explicitly configuring the region.

For example, if the bootstrap string is bootstrap.us-east-1.example.com, then msk-iam-auth library will to extract the AWS Region from the broker connection string and use us-east-1 in its IAM requests. If the bootstrap string is simply bootstrap.example.com, then the client must explicitly configure AWS_REGION=us-east-1 to connect to the cluster if it is in us-east-1, or us-west-2 if it is in us-west-2.

Note that this is a limitation for IAM authentication, but not for SASL/SCRAM authentication. With SASL/SCRAM authentication, if the client’s credentials are applied to both clusters the global endpoint can point to either cluster and the client will be able to connect. The AWS Region is not used in SASL/SCRAM authentication, so it does not restrict the authentication scope.

How to allow public access to a private MSK cluster using the custom domain name solution?

To provide public access to a MSK cluster using the custom domain solution, you will need to do the following:

Create an Internet-facing NLB, and associate public subnets (subnets that have a route to the Internet Gateway attached to the VPC).
Create ingress rules in both the NLB and MSK security groups permitting the required public addresses. Note: the port will be 9098 for the MSK security group, and the ports you are using on the NLB listeners.
Provide public DNS resolution for the Kafka clients, by using a Route 53 public zone, or an alternative public DNS resolver.
The client needs have IAM credentials, with permission, to talk to the MSK brokers, using an IAM role, IAM access keys, IAM Roles Anywhere, or another mechanism that uses the AWS Security Token Service (AWS STS) to create and provide trusted users with temporary security credentials.

In the first part of the blog, two patterns have been highlighted. How to decide which pattern to use and why?

Option 1: Only bootstrap connection through NLB

If the Kafka clients have direct access to the broker, then you can use custom domain name for the bootstrap connection while the clients can still connect to the MSK Brokers with broker DNS. This is the simplest option, as it does not require custom TLS certificates or TLS listeners.Note that this option is not necessary when using MSK Express brokers, as MSK Express brokers already manages bootstrapping via a broker-agnostic connection string. For MSK Express, this option does not add value other than configuring a custom domain name for appearances / simplicity of client configuration. For MSK Standard brokers, this can improve client connectivity by making connection strings broker agnostic.

Option 2: All connections through NLB

When Kafka clients don’t have direct access to Amazon MSK Brokers, routing all connections through the NLB can be preferred. This can occur when a client is deployed in a different VPC than Amazon MSK VPC or the client is external, and when Amazon MSK Multi VPC Connectivity is not an option. In general, Amazon MSK Multi VPC Connectivity is preferred as this is a simpler pattern for most organizations to manage MSK Connectivity across accounts and VPCs.When Multi VPC Connectivity is not an option, NLB can be used to provide connectivity with Transit Gateway or PrivateLink, and the solution mentioned in the blog should be used.

Here is an example architecture how Kafka client and Amazon MSK cluster deployed in two separate VPCs but connected via AWS Private Link.

Is Amazon Route 53 required to use a custom domain name with Amazon MSK?

You can use an alternative DNS resolver service, and do not require Amazon Route 53 to use a custom domain name with Amazon MSK. The only requirement is that your clients can resolve against your DNS resolver service. The only change required, is to use a CNAME for the DNS records, referencing the NLBs DNS record, in place of the Alias records, as this is record type is only available in Amazon Route 53.

We don’t use Amazon Certificate Manager (ACM), can NLB integrate with other 3rd party certificate managers?

NLB only supports ACM to bind a certificate to a TLS listener. You can import a certificate created using your 3rd party certificate manager into ACM, and do not need to create a certificate using ACM.

Getting connection to node terminated during authentication after setting `advertised.listeners` , what could be the issue?

As the issue started to occur after changing the advertised.listeners configuration, the issue is unlikely to be related to permissions. The following can cause this issue:

The NLB and/or client’s Security Group does not permit access to the listener ports on the NLB from the client.
A firewall appliance between the NLB and client does not permit the client to talk to the NLB using the listener ports.
The advertised.listeners configuration has an error causing the client to receive invalid details, such as a typo in the name. If this is the case, use a client in the same VPC as the MSK broker that has IAM permissions to talk to the MSK broker, and Security Group rules permitting connectivity, you then use the following command to delete the advertised.listeners configuration.

/home/ec2-user/kafka/bin/kafka-configs.sh --alter \
         --bootstrap-server  \
         --entity-type brokers \
         --entity-name  \
         --command-config ~/kafka/config/client_iam.properties \
         --delete-config advertised.listeners

BROKERS_AMAZON_DNS_NAME such as b-1.clustername.xxxxxx.yy.kafka.region.amazonaws.com:9098.

Getting “unexpected broker id, expected 2 or empty string, but received 1”, what is causing this error?

This error is typically presented when the advertised.listeners configuration for one of the brokers has the port used by another broker set. For example broker 2 has port 9001 set for IAM, but this port is used to connect to broker 1, so broker 1 is responding with an error to say you presented broker id 2, but I am broker 1.

To correct this, you will need to update the broker with the incorrect advertised.listeners configuration to use the correct port. To gain access to the broker to make the change, you will need to use the following command to delete the incorrect configuration:

/home/ec2-user/kafka/bin/kafka-configs.sh --alter \
         --bootstrap-server \
         --entity-type brokers \
         --entity-name  \
         --command-config ~/kafka/config/client_iam.properties \
         --delete-config advertised.listeners

BROKERS_AMAZON_DNS_NAME such as b-2.clustername.xxxxxx.yy.kafka.region.amazonaws.com:9098.

You then need to use the following command to set the advertised.listeners configuration for that broker:

Note: The advertised.listeners configuration in the below assumes only IAM is used for authentication. If you are using additional authentication options, you will need to include them.

MSKDOMAIN=
broker_id=
Domain=

/home/ec2-user/kafka/bin/kafka-configs.sh --alter \
         --bootstrap-server  \
         --entity-type brokers \
         --entity-name "$broker_id" \
         --command-config ~/kafka/config/client_iam.properties \
         --add-config "advertised.listeners=[CLIENT_IAM://b-$broker_id.$Domain:900$broker_id,REPLICATION://b-$broker_id-internal.$MSKDOMAIN:9093,REPLICATION_SECURE://b-$broker_id-internal.$MSKDOMAIN:9095]"

Summary

In this post, we explained how you can use an NLB, Route 53, and the advertised listener configuration option in Amazon MSK to support custom domain names with MSK clusters when using IAM authentication. You can use this solution to keep your existing Kafka bootstrap DNS name and reduce or remove the need to change client applications because of a migration, recovery process, or to use a DNS name in line with your organization’s naming convention (for example, msk.prod.example.com).

Try the solution out for yourself, and leave your questions and feedback in the comments section.

About the authors

Migrate third-party and self-managed Apache Kafka clusters to Amazon MSK Express brokers with Amazon MSK Replicator

Ankita Mishra — Mon, 20 Apr 2026 20:00:27 +0000

Migrating Apache Kafka workloads to the cloud often involves managing complex replication infrastructure, coordinating application cutovers with extended downtime windows, and maintaining deep expertise in open-source tools like Apache Kafka’s MirrorMaker 2 (MM2). These challenges slow down migrations and increase operational risk. Amazon MSK Replicator addresses these challenges, enabling you to migrate your Kafka deployments (referred to as “external” Kafka clusters) to Amazon MSK Express brokers with minimal operational overhead and reduced downtime. MSK Replicator supports data migration from Kafka deployments (version 2.8.1 or later) that have SASL/SCRAM authentication enabled – including Kafka clusters running on-premises, on AWS, or other cloud providers, as well as Kafka-protocol-compatible services like Confluent Platform, Avien, RedPanda, WarpStream, or AutoMQ when configured with SASL/SCRAM authentication.

In this post, we walk you through how to replicate Apache Kafka data from your external Apache Kafka deployments to Amazon MSK Express brokers using MSK Replicator. You will learn how to configure authentication on your external cluster, establish network connectivity, set up bidirectional replication, and monitor replication health to achieve a low-downtime migration.

How it works

MSK Replicator is a fully managed serverless service that replicates topics, configurations, and offsets from cluster to cluster. It alleviates the need to manage complex infrastructure or configure open-source tools.

Before MSK Replicator, customers used tools like MM2 for migrations. These tools lack bi-directional topic replication when using the same topic names, creating complex application architectures to consume different topics on different clusters. Custom replication policies in MM2 can allow identical topic names, but MM2 still lacks bidirectional offset replication because the MM2 architecture requires producers and consumers to run on the same cluster to replicate offsets. This created complex migrations that required either migrating consumers before producers or big-bang migrations migrating all applications at once. When customers run into issues during the migration, the rollback process is error-prone and introduces large amounts of duplicate message processing due to the lack of consumer group offset synchronization. These approaches create risk and complexity for customers that make migrations difficult to manage.

MSK Replicator addresses these problems by supporting bidirectional replication of data and enhanced consumer group offset synchronization. MSK Replicator copies topics and offsets from an external Kafka cluster to MSK, allowing you to preserve the same topic and consumer group names on both clusters. MSK Replicator also supports creating a second Replicator instance for bidirectional replication of both data and enhanced offset synchronization, allowing producers and consumers to run independently on different Kafka clusters. Data published or consumed on the Amazon MSK cluster will be replicated back to the external cluster by the second Replicator. This feature works when producers and consumers are migrated regardless of order without worrying about dependencies between applications.

Because MSK Replicator provides bidirectional data replication and enhanced consumer group offset synchronization, you can move producers and consumers at your own pace without data loss. This reduces migration complexity, allowing you to migrate applications between your external Kafka cluster and Amazon MSK regardless of order. If you run into problems during the migration, enhanced offset synchronization allows you to roll back changes by moving applications back to the external Kafka cluster, where they restart from the latest checkpoint from the Amazon MSK cluster.

For example, consider three applications:

The “Orders” application, which accepts incoming orders and writes them to the orders Kafka topic
The “Order status” application, which reads from the “orders” Kafka topic and writes status updates to the order_status topic
The “Customer notification” application, which reads from the order_status topic and notifies customers when status changes

MSK Replicator enables these applications to be migrated between an on-premises Apache Kafka cluster and an Amazon MSK Express cluster with low downtime and no data loss, regardless of order. The “Order status” application can migrate first, receive orders from the on-premises “Orders” application, and send status updates to the on-premises “Customer notification” application. If issues arise during the migration, the “Order status” application can roll back to the on-premises cluster and its consumer group offsets for the orders topic will be ready for it to pick up from where it left off on the Amazon MSK cluster.

MSK Replicator supports data distribution across hybrid and multi-cloud environments for analytics, compliance, and business continuity. It is also configured for disaster recovery scenarios where Amazon MSK Express serves as a resilient target for your external Kafka clusters.

If you are currently using MM2 for replication, see Amazon MSK Replicator and MirrorMaker2: Choosing the right replication strategy for Apache Kafka disaster recovery and migrations to understand which solution best fits your use case.

Solution overview

MSK Replicator supports Kafka deployments running version 2.8.1 or later as a source, including 3rd party managed Kafka services, self-managed Kafka, and on-premises or third-party cloud-hosted Kafka. MSK Replicator automatically handles data transfer, uses SASL/SCRAM authentication with SSL encryption, and maintains consumer group positions across both clusters. If you do not use SASL/SCRAM today, this can be configured as a new listener used for MSK Replicator allowing current clients to use their existing authentication mechanisms alongside MSK Replicator.

Prerequisites

To follow along with this walkthrough, you need the following resources in place:

A source Kafka cluster using Kafka version 2.8.1 or above
Network connectivity between your external Kafka cluster and AWS (for example, using AWS Direct Connect, Site-to-Site VPN, or Amazon Virtual Private Cloud (VPC) peering or AWS Transit Gateway for connections between AWS VPCs) so that MSK Replicator can reach your source brokers
SASL/SCRAM authentication configured on your external cluster (SHA-256 or SHA-512), which MSK Replicator uses to authenticate with external clusters
An admin user configured on your external cluster with permissions to describe the external cluster and create and modify users/ACLs
An Amazon MSK Express cluster with IAM authentication enabled to serve as your target
AWS Secrets Manager configured to store your SASL/SCRAM credentials for the external cluster so that MSK Replicator can securely retrieve them at runtime
An Amazon CloudWatch log group for MSK Replicator logs
Appropriate IAM permissions for creating and managing MSK Replicator resources

Setting up replication

Step 1: Configure network connectivity

You can set up network connectivity between your external Kafka cluster and your AWS VPC using methods such as AWS Direct Connect for dedicated network connections, AWS Site-to-Site VPN for encrypted connections over the internet, and AWS VPC peering or AWS Transit Gateway for connections between AWS VPCs. Verify that IP routing and DNS resolution are properly configured between your external cluster and AWS.

To verify IP routing and DNS resolution, connect to your external Kafka cluster from inside of your VPC by using the Kafka CLI to list topics on the external cluster. If you can list topics from your VPC using the Kafka CLI, this means DNS resolution and IP routing are working successfully. If it fails, work with your network admins to troubleshoot network connectivity issues.

Step 2: Configure external cluster

In this step, you will set up authentication on your external Kafka cluster and store the credentials in AWS Secrets Manager so that MSK Replicator can connect securely.

Configure authentication

Using the external cluster admin user, configure SASL/SCRAM authentication for MSK Replicator using SHA-256 or 512 on your external Kafka cluster. Create a SASL/SCRAM user for MSK Replicator and give the user the following ACL permissions:

Topic operations – Alter, AlterConfigs, Create, Describe, DescribeConfigs, Read, Write
Group operations – Read, Describe
Cluster operations – Create, ClusterAction, Describe, DescribeConfigs

Configure SecretsManager

AWS Secrets Manager stores your SASL/SCRAM credentials securely so that MSK Replicator can retrieve them at runtime. The secret must use JSON format and have the following keys:

username – The SCRAM username that you configured in the authentication step above
password – The SCRAM password that you configured in the authentication step above
certificate – The public root CA certificate (the top-level certificate authority that issued your cluster’s TLS certificate) and the intermediate CA chain (intermediate certificates between the root and your cluster’s certificate), used for SSL handshakes with the external cluster

Optionally, you may create separate secrets for SCRAM credentials and the SSL certificate. This approach is useful when secrets for SCRAM credentials and certificates are provisioned in different stages, such as in Infrastructure as Code (IaC) pipelines.

Retrieve the cluster ID

As the admin user, use the Kafka CLI tools to retrieve the cluster ID of your external cluster. Run the following command, replacing your-broker-host:9096 with the address of one of your external cluster’s bootstrap servers:

bin/kafka-cluster.sh cluster-id --bootstrap-server your-broker-host:9096 --config admin.properties

The command returns a cluster ID string such as lkc-abc123. Take note of this value because you will need it when creating the replicator in Step 4.

Step 3: Create your MSK Express target cluster

With your external cluster configured, you can now set up the target. Create an Amazon MSK Express cluster with IAM authentication enabled. Make sure that the cluster is in subnets that have access to AWS Secrets Manager endpoints. See Get started using Amazon MSK for more information on creating an MSK cluster.

Step 4: Create the replicator

Now that both clusters are ready, you can connect them by setting up the MSK Replicator with the appropriate IAM role and replication configuration.

Set up an IAM role for MSK Replicator

MSK Replicator needs an IAM role to interact with your MSK Express cluster and retrieve secrets. Set up a service execution IAM role with a trust policy allowing kafka.amazonaws.com and attach the AWSMSKReplicatorExecutionRole permissions policy. Take note of the role ARN for creating the replicator.

Create and attach a policy for accessing your Secrets Manager secrets and reading/writing data in your MSK cluster. See Creating roles and attaching policies (console) for more information on creating IAM roles and policies.

The following is an example policy for reading and writing data to your MSK cluster and reading KMS-encrypted Secrets Manager secrets:

{ 
    "Version": "2012-10-17", 
    "Statement": [ 
        { 
            "Sid": "SecretsManagerAccess", 
            "Effect": "Allow", 
            "Action": [ 
                "secretsmanager:GetSecretValue", 
                "secretsmanager:DescribeSecret" 
            ], 
            "Resource": [ 
                "<SCRAM_SECRET_ARN>", 
                "<CERT_SECRET_ARN>" 
            ] 
        }, 
        { 
            "Sid": "KMSDecrypt", 
            "Effect": "Allow", 
            "Action": "kms:Decrypt", 
            "Resource": "<SECRETSMANAGER_KMS_KEY_ARN>" 
        }, 
        { 
            "Sid": "TargetClusterAccess", 
            "Effect": "Allow", 
            "Action": [ 
                "kafka-cluster:Connect", 
                "kafka-cluster:DescribeCluster", 
                "kafka-cluster:AlterCluster", 
                "kafka-cluster:DescribeClusterDynamicConfiguration", 
                "kafka-cluster:AlterClusterDynamicConfiguration", 
                "kafka-cluster:DescribeTopic", 
                "kafka-cluster:CreateTopic", 
                "kafka-cluster:AlterTopic", 
                "kafka-cluster:DescribeTopicDynamicConfiguration", 
                "kafka-cluster:AlterTopicDynamicConfiguration", 
                "kafka-cluster:WriteData", 
                "kafka-cluster:WriteDataIdempotently", 
                "kafka-cluster:ReadData", 
                "kafka-cluster:DescribeGroup", 
                "kafka-cluster:AlterGroup" 
            ], 
            "Resource": [ 
                "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:cluster/<MSK_CLUSTER_NAME>*/*", 
                "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:topic/<MSK_CLUSTER_NAME>/*", 
                "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:group/<MSK_CLUSTER_NAME>*/*" 
            ] 
        }, 
        { 
            "Sid": "CloudWatchLogsAccess", 
            "Effect": "Allow", 
            "Action": [ 
                "logs:CreateLogStream", 
                "logs:PutLogEvents", 
                "logs:DescribeLogStreams" 
            ], 
            "Resource": "<MSK_REPLICATOR_LOG_GROUP_ARN>" 
        } 
    ] 
}

Create the replicator for external to MSK replication

Use the AWS CLI, API, or Console to create your replicator. Here’s an example using the AWS CLI:

aws kafka create-replicator \
  --replicator-name external-to-msk \
  --service-execution-role-arn "arn:aws:iam::123456789012:role/MSKReplicatorRole" \
  --kafka-clusters file://./kafka-clusters.json \
  --replication-info-list file://./replication-info.json \
  --log-delivery file://./log-delivery.json \
  --region us-east-1

The kafka-clusters.json file defines the source and target Kafka cluster connection information, replication-info.json specifies which topics to replicate and how to handle consumer group offset synchronization, and log-delivery.json specifies the CloudWatch logging configuration. The following tables describe the required parameters:

CLI inputs:

CLI Parameter	Description	Example
replicator-name	The name of the replicator	external-to-msk
service-execution-role-arn	The ARN for the service execution IAM role you created	arn:aws:iam::123456789012:role/MSKReplicatorRole
kafka-clusters	The Kafka cluster connection info	See below
replication-info-list	The replication configuration	See below
log-delivery	The logging configuration	See below

Key kafka-clusters.json inputs:

CLI Parameter	Description	Example
ApacheKafkaClusterId	The cluster ID retrieved in Step 2	lkc-abc123
RootCaCertificate	The Secrets Manager ARN containing the public CA certificate and intermediate CA chain	arn:aws:secretsmanager:<REGION>:<ACCOUNT_ID>:secret:my-cert
MskClusterArn	The ARN for the MSK Express cluster	arn:aws:kafka:<REGION>:<ACCOUNT_ID>:cluster/my-cluster/abc-123
SecretArn	The Secrets Manager ARN containing the SASL/SCRAM username and password	arn:aws:secretsmanager:<REGION>:<ACCOUNT_ID>:secret:my-creds
SecurityGroupIds	The security group IDs for MSK Replicator	sg-0123456789abcdef0

Key replication-info.json inputs:

CLI Parameter	Description	Example
TargetCompressionType	The compression type to use for replicating data	LZ4
TopicsToReplicate	The list of topics to replicate (use [“.*”] for all topics)	[“my-topic”]
ConsumerGroupsToReplicate	The list of consumer groups to replicate	[“my-group”]
StartingPosition	The point in the Kafka topics to begin replication from (either EARLIEST or LATEST)	EARLIEST
ConsumerGroupOffsetSyncMode	Whether or not to use enhanced bidirectional consumer group offset synchronization	ENHANCED

Note that startingPosition is set to EARLIEST in the configuration below, which means the replicator begins reading from the oldest available offset on each topic. This is the recommended setting for migrations to avoid data loss.

Key log-delivery.json inputs:

CLI Parameter	Description	Example
Enabled	Allows you to enable CloudWatch logging	true
LogGroup	The CloudWatch logs log group name to log to	/msk/replicator/my-replicator

Additional log delivery methods for Amazon S3 and Amazon Data Firehose are supported. In this post, we use CloudWatch logging.

The configs should look like the following for external to MSK replication.

kafka-clusters.json:

[ 
  { 
    "ApacheKafkaCluster": { 
      "ApacheKafkaClusterId": "lkc-abc123", 
      "BootstrapBrokerString": "broker1.example.com:9096" 
    }, 
    "ClientAuthentication": { 
      "SaslScram": { 
        "Mechanism": "SHA512", 
        "SecretArn": "arn:aws:secretsmanager:<REGION>:<ACCOUNT_ID>:secret:my-creds" 
      } 
    }, 
    "EncryptionInTransit": { 
      "EncryptionType": "TLS", 
      "RootCaCertificate": "arn:aws:secretsmanager:<REGION>:<ACCOUNT_ID>:secret:my-cert" 
    } 
  }, 
  { 
    "AmazonMskCluster": { 
      "MskClusterArn": "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:cluster/my-cluster/abc-123" 
    }, 
    "VpcConfig": { 
      "SecurityGroupIds": ["sg-0123456789abcdef0"], 
      "SubnetIds": ["subnet-abc123", "subnet-abc124", "subnet-abc125"] 
    } 
  } 
]

replication-info.json:

[ 
  { 
    "SourceKafkaClusterId": "lkc-abc123", 
    "TargetKafkaClusterArn": "arn:aws:kafka:<REGION>:<ACCOUNT_ID>:cluster/my-cluster/abc-123", 
    "TargetCompressionType": "LZ4", 
    "TopicReplication": { 
      "TopicsToReplicate": ["my-topic"], 
      "CopyTopicConfigurations": true, 
      "CopyAccessControlListsForTopics": true, 
      "DetectAndCopyNewTopics": true, 
      "StartingPosition": {"Type": "EARLIEST"}, 
      "TopicNameConfiguration": {"Type": "IDENTICAL"} 
    }, 
    "ConsumerGroupReplication": { 
      "ConsumerGroupsToReplicate": ["my-group"], 
      "SynchroniseConsumerGroupOffsets": true, 
      "DetectAndCopyNewConsumerGroups": true, 
      "ConsumerGroupOffsetSyncMode": "ENHANCED" 
    } 
  } 
]

log-delivery.json:

{ 
  "ReplicatorLogDelivery": {
     "CloudWatchLogs": {
       "Enabled": true, 
       "LogGroup": "<LOG_GROUP_NAME>"
     }
  } 
}

Configure bidirectional replication from MSK to the external cluster

To enable bidirectional replication, create a second replicator that replicates in the opposite direction. Use the same IAM role and network configuration from Step 4, but swap the source and target. Replace SourceKafkaClusterId with TargetKafkaClusterId and TargetKafkaClusterArn with SourceKafkaClusterArn in a new msk-to-external-replication-info.json file:

aws kafka create-replicator \
  --replicator-name msk-to-external \
  --service-execution-role-arn "arn:aws:iam::123456789012:role/MSKReplicatorRole" \
  --kafka-clusters file:///./kafka-clusters.json \
  --replication-info-list file:///./msk-to-external-replication-info.json \
  --log-delivery file:///./log-delivery.json \
  --region us-east-1

Monitoring replication health

Monitor your replication using Amazon CloudWatch metrics. Three key metrics to understand are MessageLag, SumOffsetLag, and ReplicationLatency. MessageLag measures how far behind the replicator is from the external cluster in terms of messages not yet replicated, while SumOffsetLag measures how far behind a consumer group is from the latest message in a topic. ReplicationLatency is the amount of latency between the source and target clusters in data replication. When the three reach a sustained low level, your clusters are fully synchronized for both data and consumer group offsets.

To troubleshoot MSK Replicator replication or errors, use the CloudWatch logs to get more details about the health of the replicator. MSK Replicator logs status and troubleshooting information which can be helpful in diagnosing issues like connectivity, authentication, and SSL errors.

Note that the replication is asynchronous, so there will be some lag during replication. The lag will reach zero once a client is shut down during migration to the target cluster. This takes about 30 seconds under normal operations, allowing a low downtime migration without data loss. If your lag is continually increasing or does not reach a sustained low level, this indicates that you have insufficient partitions for high-throughput replication. Refer to Troubleshoot MSK Replicator for more information on troubleshooting replication throughput and lag.

Key metrics include:

MessageLag – Monitors the sync between the MSK Replicator and the source cluster. MessageLag indicates the lag between the messages produced to the source cluster and messages consumed by the replicator. It is not the lag between the source and target cluster.
ReplicationLatency – Time taken for records to replicate from source to target cluster (ms)
ReplicatorThroughput – Average number of bytes replicated per second
ReplicatorFailure – Number of failures the replicator is experiencing
KafkaClusterPingSuccessCount – Connection health indicator (1 = healthy, 0 = unhealthy)
ConsumerGroupCount – Total consumer groups being synchronized
ConsumerGroupOffsetSyncFailure – Failures during offset synchronization
AuthError – Number of connections with failed authentication per second, by cluster
ThrottleTime – Average time in ms a request was throttled by brokers, by cluster
SumOffsetLag – Aggregated offset lag across partitions for a consumer group on a topic (MSK cluster-level metric)

For more details on these metrics, see the MSK Replicator metrics documentation.

Your applications are ready to migrate when the following conditions are met. For most workloads, you should expect these metrics to stabilize within a few hours of starting replication. High-throughput clusters may take longer depending on topic volume and partition count.

ReplicatorFailure = 0
ConsumerGroupOffsetSyncFailure = 0
KafkaClusterPingSuccessCount = 1 for both source and target clusters
MessageLag < 1,000
- Your sustained lag may be lower or higher depending on your throughput per partition, message size, and other factors
- Sustained high message lag usually indicates insufficient partitions for high-throughput replication
ReplicationLatency < 90 seconds
- Your sustained latency may be lower or higher depending on your throughput per partition, message size, and other factors
- Sustained high latency usually indicates insufficient partitions for high-throughput replication
SumOffsetLag is at a sustained low level on both clusters
- Offset values on the two clusters may not be numerically identical.
- MSK Replicator translates offsets between clusters so that consumers resume from the correct position, but the raw offset numbers can differ due to how offset translation works. What matters is that SumOffsetLag is at a sustained low level.
ConsumerGroupCount (MSK) = Expected count (external cluster)
- If ConsumerGroupCount is zero or does not match the expected count, then there is an issue in the Replicator configuration or a permissions issue preventing consumer group synchronization

Migrating your applications

With bidirectional consumer offset synchronization, you can migrate your producers and consumers regardless of order. Start by monitoring replication metrics until they reach the target values described in the previous section. Then migrate your applications (producers or consumers) to use the MSK Express cluster endpoints and verify that they are producing and consuming as expected. If you encounter issues, you can roll back by switching applications back to the external cluster. The consumer offset synchronization makes sure that your applications resume from their last committed position regardless of which cluster they connect to.

For a comprehensive, hands-on walkthrough of the end-to-end migration process, explore the MSK Migration Workshop, which provides step-by-step guidance for migrating your Kafka workloads to Amazon MSK.

Security considerations

MSK Replicator uses SASL/SCRAM authentication with SSL encryption for secure data transfer between your external cluster and AWS. The solution supports both publicly trusted certificates and private or self-signed certificates. Credentials are stored securely in AWS Secrets Manager, and the target MSK Express cluster uses IAM authentication for access control.

When configuring security, keep the following in mind:

Make sure that the IAM role you create in Step 4 follows the principle of least privileges. Only attach AWSMSKReplicatorExecutionRole and an IAM policy for Secrets Manager with least-privileges access to read secret values and avoid adding broader permissions.
Verify that your Secrets Manager secret is encrypted with an AWS KMS key that the MSK Replicator service execution role has permission to decrypt.
Confirm that the security groups assigned to MSK Replicator allow outbound traffic to your external cluster’s broker ports (typically 9096 for SASL/SCRAM with TLS) and to the MSK Express cluster.
Rotate your SASL/SCRAM credentials periodically and update the corresponding Secrets Manager secret. MSK Replicator picks up the new credentials automatically on the next connection attempt.

Under the AWS shared responsibility model, AWS is responsible for securing the underlying infrastructure that runs MSK Replicator, including the compute, storage, and networking resources. You are responsible for configuring authentication mechanisms (SASL/SCRAM), managing credentials in AWS Secrets Manager, configuring network security (security groups and VPC settings), implementing IAM policies following least privilege, and rotating credentials. For more information, see Security in Amazon MSK in the Amazon MSK Developer Guide.

Cleanup

To avoid ongoing charges, delete the resources you created during this walkthrough. Start by deleting the replicators first, because they depend on the other resources:

aws kafka delete-replicator --replicator-arn <replicator-arn>

After both replicators are deleted, you can remove the following resources if they were created solely for this walkthrough:

The MSK Express cluster (deleting a cluster also removes its stored data, so verify that your applications have fully migrated before proceeding)
The Secrets Manager secrets containing your SASL/SCRAM credentials and certificates
The IAM role and policies created for MSK Replicator

You can verify that a replicator has been fully deleted by running aws kafka list-replicators and confirming it no longer appears in the output.

Conclusion

Amazon MSK Replicator simplifies the process of migrating to Amazon MSK Express brokers and establishes hybrid Kafka architectures. The fully managed service alleviates the operational complexity of managing replication while bidirectional consumer offset synchronization enables flexible, low-risk application migration.

Next Steps

To get started using MSK Replicator to migrate applications to MSK Express brokers, use the MSK Migration Workshop for a hands-on, end-to-end migration walkthrough. The Amazon MSK Replicator documentation includes detailed configuration details to help configure MSK Replicator for your use case. From there, use MSK Replicator to migrate your Apache Kafka workloads to MSK Express broker.

Once your migration is complete, consider exploring multi-region replication patterns for disaster recovery, or integrating your MSK Express cluster with AWS analytics services such as Amazon Data Firehose and Amazon Athena. If you need help planning your migration, reach out to your AWS account team, AWS Support or AWS Professional Services.

About the authors

Building unified data pipelines with Apache Iceberg and Apache Flink

Nikhil Jha — Mon, 20 Apr 2026 16:59:46 +0000

You can process real-time data from your data lake with Amazon Managed Service for Apache Flink without maintaining two separate pipelines. Yet many teams do exactly that, and the cost adds up fast. In this post, you build a unified pipeline using Apache Iceberg and Amazon Managed Service for Apache Flink that replaces the dual-pipeline approach. This walkthrough is for intermediate AWS users who are comfortable with Amazon Simple Storage Service (Amazon S3) and AWS Glue Data Catalog but new to streaming from Apache Iceberg tables.

The dual-pipeline problem

This dual-pipeline approach creates three problems:

Double the infrastructure costs. You run and pay for two separate compute environments, two storage layers, and two sets of monitoring. For example, if you’re spending $10,000/month on separate streaming and batch infrastructure, a meaningful portion of that spend is pure duplication.
Data synchronization issues. Your batch and streaming consumers read from different copies of the data, processed at different times. When a transaction shows up in your real-time dashboard but not in your batch report (or vice versa), debugging the inconsistency takes hours.
Operational complexity. Two pipelines mean two deployment processes, two failure modes to monitor, and two sets of schema evolution to manage. Your team spends time reconciling systems instead of building features.

Where this pattern fits

Before diving into the implementation, consider whether streaming from your data lake is the right approach for your use case.

Streaming from Apache Iceberg tables works well when you need data available within seconds to minutes and you query recent data frequently, multiple times per hour. Common scenarios include:

Operational data stores — Stream customer profile updates to serve downstream applications like recommendation engines. When a customer updates their preferences, those changes reach your operational data store within seconds.
Fraud detection — Stream transactions for immediate analysis. Start with a 3-second monitor interval and adjust based on your detection accuracy needs.
Live dashboards — Power real-time analytics directly from your lake. This is the strongest starting point if you’re evaluating the approach for the first time, because the feedback loop is immediate and straightforward to validate.
Event-driven architectures — Trigger downstream processes based on data changes in your Apache Iceberg tables.

Batch processing remains more cost-effective when you process data once per day or less, or you primarily query historical data. Batch queries on Apache Iceberg tables cost less because they don’t require a continuous Apache Flink runtime.

How Apache Iceberg solves this

Apache Iceberg’s snapshot-based architecture removes the need for a separate streaming pipeline. Think of snapshots like Git commits for your data. Each time you write data to your Iceberg table, Iceberg creates a new snapshot that points to the new data files while preserving references to existing files. Apache Flink reads only the changes between snapshots (the new files that arrived after the last checkpoint), rather than scanning the entire table. Atomicity, Consistency, Isolation, Durability (ACID) transactions prevent your concurrent reads and writes from producing partial or inconsistent results. For example, if your batch extract, transform, and load (ETL) job is writing 10,000 records while your Flink application is reading, ACID transactions mean that your streaming query sees either the complete batch of 10,000 records or none of them, not a partial set that could skew your analytics.

The result is a single pipeline that handles both real-time and batch access from the same data, through the same storage layer, with the same schema.

Solution architecture

Your architecture uses four AWS services and one open source table format working together. The following diagram shows how these components connect, replacing the dual-pipeline pattern shown earlier with a single unified flow.

Your source data lands in Amazon S3 as Apache Iceberg table files. AWS Glue Data Catalog tracks the metadata and schema. When new data arrives, Apache Iceberg creates a new snapshot that your application detects. Your Flink application monitors these snapshots and processes new records incrementally, reading only the files that arrived after the last checkpoint, not the entire table.

You use four main components:

Amazon S3 — Foundational storage layer for your data lake
Data Catalog — Metadata and schema management for Apache Iceberg tables
Apache Iceberg — Table format with snapshot-based streaming capabilities
Amazon Managed Service for Apache Flink — Stream processing and incremental consumption

Important notices

Before implementing this solution, evaluate these risks for your environment:

Data security: Streaming from data lakes exposes data to additional processing systems. Classify your data before implementation—customer profile updates and transaction data typically contain personally identifiable information (PII) and treat them as confidential. Apply encryption at rest and in transit for confidential data. Key risks include unauthorized data access through misconfigured Amazon S3 bucket policies or overly permissive IAM roles. Mitigations: use the resource-scoped IAM policy and TLS-enforcing bucket policy provided in the Security section.
Data integrity: Misconfigured checkpoints or schema changes during streaming can lead to data inconsistency. Mitigations: enable exactly-once processing semantics and test schema evolution in a non-production environment first.
Compliance: Verify that real-time data processing meets your regulatory requirements. For workloads subject to HIPAA, confirm that you use HIPAA Eligible Services and have a Business Associate Agreement (BAA) with AWS. For PCI-DSS or GDPR workloads, review the relevant compliance documentation on the AWS Compliance page. Implement data retention policies that comply with your regulatory framework.
Cost: Nearly continuous streaming incurs ongoing compute costs. Monitor usage to avoid unexpected charges. Cost estimates in this post are based on pricing as of March 2026 and might change. Verify current pricing on the relevant AWS service pricing pages.
Operational: Pipeline failures might impact downstream systems. Implement monitoring and alerting before running in production.

Prerequisites

Before you begin, make sure that you have the following in place. This walkthrough assumes intermediate Python skills (comfortable with functions, error handling, and environment variables), basic Apache Flink concepts (streaming compared to batch processing), and basic AWS Identity and Access Management (AWS IAM) knowledge (creating roles and attaching policies). Plan for approximately 90–120 minutes, including setup, implementation, and testing. First-time setup might take longer as you download dependencies and configure AWS resources. Expected AWS costs: approximately $5–10 if you complete the walkthrough within 2 hours and clean up resources immediately afterward. The primary cost driver is Amazon Managed Service for Apache Flink runtime ($0.11/hour per Kinesis Processing Unit (KPU)). You can minimize costs by stopping your application when not in use.

An AWS account with AWS IAM permissions for: s3:GetObject, s3:PutObject, s3:ListBucket on your data bucket; glue:GetDatabase, glue:GetTable for catalog access; and flink:CreateApplication, flink:StartApplication for Amazon Managed Service for Apache Flink
An existing Amazon S3 bucket for your data lake
An AWS Glue Data Catalog database configured
Apache Flink 1.19.1 installed locally
Python 3.8 or later
Java 11 or a more recent version
AWS Command Line Interface (AWS CLI) configured with credentials (aws configure)

Required Java Archive (JAR) dependencies

You need multiple JAR files because your Flink application coordinates between different systems—Amazon S3 for storage, AWS Glue for metadata, Hadoop for file operations, and Apache Iceberg for the table format. Each JAR handles a specific part of this integration. Missing even one causes ClassNotFoundException errors at runtime.

iceberg-flink-runtime-1.19-1.6.1.jar — Core Apache Iceberg integration with Apache Flink
iceberg-aws-bundle-1.6.1.jar — AWS-specific Apache Iceberg functionality for Amazon S3 and AWS Glue
flink-s3-fs-hadoop-1.19.1.jar — Provides Apache Flink read and write access to Amazon S3
flink-sql-connector-hive-3.1.3_2.12-1.19.1.jar — Hive metastore connector for catalog compatibility
hadoop-common-3.4.0.jar — Core Hadoop libraries required by Apache Iceberg
flink-shaded-hadoop-2-uber-2.8.3-10.0.jar — Repackaged Hadoop dependencies that avoid version conflicts with Apache Flink
hadoop-hdfs-client-3.4.0.jar — Hadoop Distributed File System (HDFS) client libraries for file system operations
flink-json-1.19.1.jar — JSON format support for Apache Flink
hadoop-aws-3.4.0.jar — Hadoop integration with AWS services
hadoop-client-3.4.0.jar — Hadoop client libraries
aws-java-sdk-bundle-1.12.261.jar — AWS SDK for authentication and service access

jars = [
    "flink-s3-fs-hadoop-1.19.1.jar",
    "flink-sql-connector-hive-3.1.3_2.12-1.19.1.jar",
    "hadoop-common-3.4.0.jar",
    "flink-shaded-hadoop-2-uber-2.8.3-10.0.jar",
    "iceberg-flink-runtime-1.19-1.6.1.jar",
    "iceberg-aws-bundle-1.6.1.jar",
    "hadoop-hdfs-client-3.4.0.jar",
    "flink-json-1.19.1.jar",
    "hadoop-aws-3.4.0.jar",
    "hadoop-client-3.4.0.jar",
    "aws-java-sdk-bundle-1.12.261.jar"
]

Technical implementation

The sample code in this post is available under the MIT-0 license.This section walks you through building the streaming pipeline step by step. You create a single Python file, iceberg_streaming.py, with three functions that run in sequence. Your main() function calls them in order: set up the Apache Flink environment, register the Data Catalog, then start the streaming query.

Set up your Apache Flink environment

To prepare your Apache Flink environment:

Download the required JAR files listed in the prerequisites section.
Place the JAR files in a lib directory in your project folder.
Configure your HADOOP_CLASSPATH environment variable to point to the lib directory.
Create your streaming execution environment by adding the following function to iceberg_streaming.py:

def setup_environment():
    """Configure the Flink streaming runtime."""
    try:
        os.environ['HADOOP_CLASSPATH'] = os.path.join(os.getcwd(), 'lib', '*')
        env = StreamExecutionEnvironment.get_execution_environment()
        env.set_parallelism(1)
        settings = EnvironmentSettings.new_instance().in_streaming_mode().build()
        t_env = StreamTableEnvironment.create(env, settings)
        return t_env
    except Exception as e:
        print(f"Failed to initialize Flink environment: {e}")
        raise

Verify your environment by running flink –version. If the command isn’t found, confirm that Apache Flink 1.19.1 is installed and that your PATH includes the Flink bin directory.

Configure AWS Glue Data Catalog

To connect your Flink application to Data Catalog:

Open your iceberg_streaming.py file.
Add the create_iceberg_source() function shown in the following section.
Replace the placeholder values with your actual AWS resources before running. These values are static configuration strings, not user input — do not construct them from external or untrusted sources at runtime.
Save the file.

def create_iceberg_source(t_env):
    """Register the AWS Glue Data Catalog as an Iceberg catalog."""
    try:
        catalog_sql = """
        CREATE CATALOG glue_catalog WITH (
            'type'='iceberg',
            'catalog-impl'='org.apache.iceberg.aws.glue.GlueCatalog',
            'warehouse'='s3://<example-data-lake-bucket>',
            'io-impl'='org.apache.iceberg.aws.s3.S3FileIO',
            'aws.region'='us-east-1',
            'hadoop-conf.fs.s3a.aws.credentials.provider'=
                'com.amazonaws.auth.DefaultAWSCredentialsProviderChain',
            'hadoop-conf.fs.s3a.endpoint'='s3.amazonaws.com',
            'property-version'='1'
        )
        """
        t_env.execute_sql(catalog_sql)
        t_env.use_catalog("glue_catalog")
        t_env.use_database("streaming_db")
    except Exception as e:
        print(f"Failed to configure Iceberg catalog: {e}")
        raise

Set up streaming logic

This function configures Apache Flink to monitor your Apache Iceberg table continuously and process new records as they arrive. Checkpointing runs every 10 seconds to track progress—if the job restarts, it resumes from the last checkpoint rather than reprocessing the entire table.Notice the monitor-interval parameter, it controls how frequently Apache Flink checks for new Apache Iceberg snapshots. A 3-second interval provides near real-time processing but generates approximately 1,200 Amazon S3 LIST API calls per hour (at $0.005 per 1,000 requests, roughly $0.04/month per table based on pricing as of March 2026). For less time-sensitive workloads, increase this to 30s to reduce API costs by 90%.Replace customer_events with the name of your Apache Iceberg table in Data Catalog:

def process_record(row):
    """Validate and process each record from the stream."""
    try:
        if row is None:
            raise ValueError("Received null row")
        required_fields = ["event_type", "timestamp"]
        for field in required_fields:
            if field not in row:
                raise ValueError(f"Missing required field: {field}")
        # Validate field types and content
        if not isinstance(row.get("event_type"), str) or len(row["event_type"]) > 256:
            raise ValueError("event_type must be a string under 256 characters")
        if not isinstance(row.get("timestamp"), (str, int)):
            raise ValueError("timestamp must be a string or integer")
        # Replace with your business logic
        print(f"Processing record: {row}")
    except ValueError as e:
        print(f"Validation error for record {row}: {e}")
    except Exception as e:
        print(f"Error processing record {row}: {e}")
def stream_data(t_env):
    """Start the streaming query and process results."""
    try:
        configuration = t_env.get_config().get_configuration()
        configuration.set_string("table.dynamic-table-options.enabled", "true")
        configuration.set_string("execution.checkpointing.interval", "10000")
        query = """
        SELECT * FROM customer_events /*+ OPTIONS(
            'streaming'='true',
            'monitor-interval'='3s',
            'table.exec.iceberg.cell-based-snapshot'='true'
        ) */
        """
        table_result = t_env.execute_sql(query)
        with table_result.collect() as results:
            for row in results:
                process_record(row)
    except Exception as e:
        print(f"Streaming query failed: {e}")
        raise

Putting it together

Your main() function calls the three steps in order:

def main():
    try:
        t_env = setup_environment()
        create_iceberg_source(t_env)
        stream_data(t_env)
    except Exception as e:
        print(f"Pipeline failed: {e}")
        raise
if __name__ == "__main__":
    main()

Run the pipeline locally:python iceberg_streaming.pyPackage the application and submit it to Amazon Managed Service for Apache Flink using the console or the AWS Command Line Interface (AWS CLI).

Running in production

Moving from a local test to a production deployment requires tuning four areas: performance, monitoring, cost, and security. This section covers the key decisions for each.

Performance tuning

Determine your latency requirements before tuning. For fraud detection, you need subsecond processing. For daily reporting dashboards, you can tolerate minutes of delay.

Partition pruning reduces the amount of data scanned per query. Proper partitioning can significantly reduce query times for time series data partitioned by date. To implement, create your Apache Iceberg table with partition columns (PARTITIONED BY (date_column) in your CREATE TABLE statement), then include partition filters in your WHERE clause: WHERE date_column >= CURRENT_DATE - INTERVAL '7' DAY.

Parallel processing matches your data volume and throughput requirements. For most workloads under 10,000 records per second, a parallelism of 1–4 is sufficient. Scale up incrementally and monitor backpressure metrics (indicators that data arrives faster than your pipeline processes it, causing queuing) to find the right setting.

Checkpoint tuning balances reliability and latency. Consider how much data you can afford to reprocess after a failure. If you process 1,000 records per second with 10-second checkpoints, a failure means reprocessing up to 10,000 records. When that’s acceptable, 10 seconds works well. For faster recovery or higher volumes, reduce to 5 seconds.

Resource allocation — Right-size your Apache Flink cluster to avoid over-provisioning. Monitor CPU and memory utilization during your initial runs and adjust task manager resources accordingly.

Monitoring

Configure your production deployment with the following checkpoint settings. These work well for moderate data volumes (up to 10,000 records per second), providing exactly-once processing semantics. This means that the pipeline processes each record exactly once, even if your application restarts. Adjust the checkpoint interval based on your latency requirements. Add this to your setup_environment() function after creating the table environment.

config_dict = {
    "execution.checkpointing.interval": "30000",
    "execution.checkpointing.mode": "EXACTLY_ONCE",
    "execution.checkpointing.timeout": "600000",
    "state.backend": "filesystem",
    "state.checkpoints.dir": "s3://<example-data-lake-bucket>/checkpoints"
}

Use Amazon CloudWatch to track checkpoint duration, records processed per second, and backpressure metrics. A 10-second checkpoint interval means writing state to Amazon S3 360 times per hour. For a 1 MB state size, that’s approximately 8.6 GB per day in checkpoint storage—at Amazon S3 Standard pricing of $0.023/GB, roughly $0.20/day or $6/month per application based on current pricing. If the checkpoint duration exceeds 50% of your interval, increase the interval or add parallelism.

Cost management

Use Amazon S3 Intelligent-Tiering for your Apache Iceberg data files, which typically have predictable access patterns after initial processing. Configure Apache Iceberg’s table expiration to automatically clean up early snapshots. This can reduce storage costs by an estimated 20–30%, though your results vary depending on write frequency and retention policies.

Right-size your Apache Flink resources based on actual throughput needs. Start with a minimal configuration and scale up based on observed backpressure and checkpoint duration metrics. Use Amazon Elastic Compute Cloud (Amazon EC2) Spot Instances where workload interruptions are acceptable, for example, in development and testing environments.

Set data retention policies on both your Apache Iceberg tables and checkpoint storage to avoid storing data longer than necessary.

Security

Security is a shared responsibility between you and AWS. AWS is responsible for the security of the cloud, including the hardware, software, networking, and facilities that run AWS services. You are responsible for security in the cloud, configuring access controls, encrypting data, and managing your application security. Apply these controls in priority order.

AWS IAM roles — Use AWS IAM roles with least-privilege access, scoped to specific resources. The following example policy restricts permissions to your data lake bucket and AWS Glue catalog:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::<example-data-lake-bucket>/*"
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::<example-data-lake-bucket>",
      "Condition": {
        "StringEquals": {
          "aws:SourceVpce": "<your-vpc-endpoint-id>"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": ["glue:GetDatabase", "glue:GetTable"],
      "Resource": [
        "arn:aws:glue:us-east-1:<account-id>:catalog",
        "arn:aws:glue:us-east-1:<account-id>:database/streaming_db",
        "arn:aws:glue:us-east-1:<account-id>:table/streaming_db/*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:<account-id>:key/<your-kms-key-id>"
    }
  ]
}

Scoping permissions to specific Amazon S3 buckets, AWS Glue databases, and AWS Key Management Service (AWS KMS) keys restrict access to only the resources your pipeline requires. Review IAM policies quarterly using the IAM Access Analyzer to identify and remove unused permissions.

Encryption — Configure server-side encryption with AWS Key Management Service (AWS KMS) customer managed keys (SSE-KMS) for your Amazon S3 buckets. Using customer managed keys requires additional review from your security team. Confirm your key management policies, rotation procedures, and access controls before implementation. Enable automatic key rotation annually. For encryption in transit, enforce TLS by adding a bucket policy that denies non-HTTPS access:

{
  "Effect": "Deny",
  "Principal": "*",
  "Action": ["s3:GetObject", "s3:PutObject", "s3:ListBucket"],
  "Resource": [
    "arn:aws:s3:::<example-data-lake-bucket>/*",
    "arn:aws:s3:::<example-data-lake-bucket>"
  ],
  "Condition": {
    "Bool": { "aws:SecureTransport": "false" }
  }
}

Amazon S3 bucket hardening — Enable Block Public Access on your buckets to prevent accidental public exposure:

aws s3api put-public-access-block \
  --bucket <example-data-lake-bucket> \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Enable versioning on buckets that store critical data and checkpoints to protect against accidental deletion. For production environments with sensitive data, consider enabling MFA Delete on versioned buckets. Enable S3 server access logging to track requests for security auditing.

Amazon Virtual Private Cloud (Amazon VPC) –Use Amazon VPC endpoints for private communication between your Apache Flink cluster and AWS services, removing public internet routing by keeping traffic within the AWS network.

Access logging – Enable AWS CloudTrail data events to log Amazon S3 object-level API calls (GetObject, PutObject) and Data Catalog API calls. Store logs in a separate Amazon S3 bucket with restricted access and enable log file integrity validation. Run regular compliance checks using AWS Config.

Operational practices

Set up a continuous integration and continuous deployment (CI/CD) pipeline to automate deployment and testing. Use version control to track schema and code changes. With Apache Iceberg’s schema evolution support, you can add columns without rewriting existing data files. Establish rollback procedures using Apache Iceberg’s snapshot-based architecture, so you can roll back to a previous table state if a bad write corrupts your data.

Troubleshooting

If you run into issues during setup or execution, use the following table to diagnose common errors.

Error	Cause	Solution
ClassNotFoundException	Missing JAR files	Check the dependencies in your lib directory and confirm `HADOOP_CLASSPATH` points to the correct path
Table not found	Database name mismatch	Check that the database name in `t_env.use_database()` matches the AWS Glue database where you registered your table
Checkpoint failures	Amazon S3 permissions	Check that your Amazon S3 bucket policy grants `s3:PutObject` for the checkpoint location
AWS credential errors	Missing AWS IAM configuration	Check that the AWS IAM role attached to your Apache Flink application has `glue:GetTable`, `glue:GetDatabase`, and `s3:GetObject` permissions on the relevant resources
Snapshot not found	Table modified during query	Increase monitor-interval or implement retry logic in your `process_record()` function
Schema mismatch	Table schema changed between snapshots	Review Apache Iceberg schema evolution settings and confirm backward compatibility

Clean up

To avoid ongoing charges, delete the resources that you created during this walkthrough.

Stop your Amazon Managed Service for Apache Flink application. Open the Amazon Managed Service for Apache Flink console, choose your application name, choose Stop, and confirm the action. Or use the AWS CLI:

aws kinesisanalyticsv2 stop-application --application-name your-app-name

Delete the Amazon S3 buckets that you created for data storage and checkpoints. For instructions, see Deleting a bucket in the Amazon S3 User Guide.
Remove the Apache Iceberg tables from your Data Catalog.
Delete the AWS IAM roles and policies created specifically for this walkthrough.
If you created an Amazon VPC or Amazon VPC endpoints for testing, delete those resources.

Conclusion

Maintaining separate streaming and batch pipelines doubles your infrastructure costs, creates data synchronization issues, and adds operational complexity that slows your team down. In this post, you replaced that dual-pipeline architecture with a single system built on Apache Iceberg and Amazon Managed Service for Apache Flink. You configured a Flink environment with the required JAR dependencies, connected it to Data Catalog, and implemented streaming queries that read new records incrementally with exactly-once processing semantics. The same data, the same storage layer, the same schema—accessible to both your real-time and batch consumers.

To extend this solution, try these next steps based on your use case:

If you’re processing high volumes (>10,000 records/sec): Start with partition pruning. Add PARTITIONED BY (date_column) to your table definition, this typically reduces query times by 60–80%.
If you need production monitoring: Implement custom Amazon CloudWatch metrics. Track checkpoint duration, records processed per second, and backpressure to catch issues before they impact your pipeline.
If you have variable workloads: Configure auto scaling for your Apache Flink cluster. See the Amazon Managed Service for Apache Flink Developer Guide for detailed guidance.

Share your implementation experience in the comments, your use case, data volumes, latency improvements, and cost reductions help other readers calibrate their expectations. To get started, try the Amazon Managed Service for Apache Flink Developer Guide and the Apache Iceberg documentation on the Apache Iceberg website.

About the authors

Securely connecting on-premises data systems to Amazon Redshift with IAM Roles Anywhere

Zainab Syeda — Mon, 20 Apr 2026 14:59:23 +0000

Securely connecting on-premises data systems to Amazon Redshift requires removing static credentials while preserving seamless access for your data teams. This solution extends connectivity from your on-premises data centers to Amazon Redshift by using short-lived, auditable credentials. All traffic remains within trusted, private channels.

Developers and data engineers need a process to run ingestion pipelines, Extract, Transform, Load (ETL) jobs, and analytics queries without managing static credentials or complex authentication flows. You can use AWS Identity and Access Management (IAM) Roles Anywhere to obtain temporary security credentials in IAM. This service extends the short-term credential model of AWS beyond the cloud and allows on-premises workloads to authenticate with IAM using X.509 certificates from an existing certificate authority. This approach removes static IAM access keys and applies least-privilege access through IAM policies. Every request is recorded in AWS CloudTrail. Paired with private Domain Name System (DNS) and Amazon Virtual Private Cloud (Amazon VPC) endpoints for Amazon Redshift, it keeps authentication and data flows inside private networks without traversing the public internet.

In this post, you will learn how to use AWS IAM Roles Anywhere with Amazon Redshift for secure, private connections. This removes the need to expose traffic to the public internet or manage long-lived access keys.

The challenge

Organizations connecting on-premises data systems to Amazon Redshift typically choose from several established security patterns, each with tradeoffs in risk, complexity, and operational overhead. Static IAM access keys are straightforward to adopt but require ongoing rotation, secure distribution, and storage across systems. Their long-lived nature increases the impact of accidental exposure in code, configuration files, or logs. Shared database or service credentials can streamline setup but often reduce auditability, weaken least-privilege controls, and create accountability challenges across teams. VPN or private network connections improve network isolation, yet they still require strong application-layer authentication and add infrastructure management burdens. Custom secret-management or credential-brokering solutions can reduce reliance on long-lived credentials, but they introduce additional components that must be built, integrated, and maintained. As organizations scale, these patterns often force tradeoffs between strong security controls and the developer productivity needed to build and operate data pipelines efficiently.

Solution overview

The solution integrates on-premises workloads with Amazon Redshift using IAM Roles Anywhere and the built-in IAM authentication of Amazon Redshift. The core idea is that on-premises workloads use X.509 certificates to obtain short-term IAM credentials, then exchange them for temporary Amazon Redshift database credentials. Both provisioned clusters and serverless workgroups are supported. The architecture consists of these main components:

Amazon Redshift Service Endpoint – Handles secure API calls such as GetClusterCredentials, GetCredentials, and GetClusterCredentialsWithIAM. The on-premises workload uses these API endpoints to request temporary database credentials.
Amazon Redshift Cluster Endpoint – Provides the connection point for database operations on provisioned Amazon Redshift clusters. After obtaining temporary credentials, applications and tools like JDBC/ODBC drivers or psql connect to the cluster endpoint. They use this connection to execute SQL queries, load data, and perform analytics tasks.
Amazon Redshift Serverless Workgroup Endpoint – Serves the same function as the cluster endpoint but for serverless deployments. After temporary credentials are retrieved through the GetCredentials API, applications connect to this endpoint using standard database drivers (JDBC/ODBC) or command line tools like psql to run queries and load data.
Certificate authority – For this post, we use AWS Private Certificate Authority (AWS Private CA) as the certificate authority (CA) source. Alternatively, you can integrate with an external CA. For more details, see IAM Roles Anywhere with an external certificate authority.
X.509 Certificate – We use a sample private certificate stored in AWS Certificate Manager (ACM) and issued by AWS Private CA.
IAM Roles Anywhere – Issues short-term AWS credentials to on-premises processes based on X.509 certificates from an organization’s certificate authority. These temporary credentials allow the workload to assume an IAM role that grants access to Amazon Redshift APIs.

To retrieve temporary credentials using IAM Role Anywhere, we use the credential_process parameter in AWS Command Line Interface (AWS CLI) profile configurations to trigger an external process that generates or retrieves credentials. This post uses X.509 certificates to authenticate and return temporary IAM credentials through IAM Roles Anywhere. The AWS IAM Roles Anywhere Credential Helper is executed to handle the signing process for the CreateSession API, returning credentials in a JSON format that applications and tools can consume.

Amazon Redshift provides several APIs that work together to support temporary, IAM-based authentication for different deployment scenarios. When connecting to a provisioned Amazon Redshift cluster, applications typically use the GetClusterCredentials API, which returns short-term database credentials tied to an IAM role’s permissions. For organizations with fully IAM-managed identities, GetClusterCredentialsWithIAM streamlines this process by automatically mapping the IAM identity to a database user, removing the need to specify usernames manually. In serverless deployments, the GetCredentials API performs the same function, issuing temporary credentials for Amazon Redshift Serverless workgroups based on IAM permissions. Collectively, these APIs keep static credentials from being stored or distributed while offering flexible integration paths for both provisioned and serverless Amazon Redshift architectures.

Flow overview

An on-premises ETL job begins by initiating a request and authenticates with AWS using IAM Roles Anywhere to assume an IAM role securely. After obtaining temporary security credentials, the workload calls the Amazon Redshift service endpoint to execute the GetClusterCredentials API, which returns short-term database credentials. These credentials allow the workload to connect to the Amazon Redshift cluster endpoint through a VPC endpoint. This enables running SQL queries or loading data into the cluster as part of the ETL process.

Prerequisites

You must have the following prerequisites to follow along with this post.

AWS account requirements

An AWS account with permissions to deploy AWS CloudFormation templates.
Access to AWS CloudShell for exporting a sample private certificate that we create using AWS CloudFormation in a later step.

Remote environment

Network Connectivity requirements

Establish secure connectivity between your on-premises environment and AWS using AWS Site-to-Site VPN, AWS Direct Connect, or AWS Client VPN.

Deploy AWS resources with AWS CloudFormation

Navigate to the AWS CloudFormation console.
Choose Create Stack.
Download the redshift-iamra-template template.
For Specify template, choose Upload a template file and upload redshift-iamra-template.
Choose Next.
Enter a unique name for Stack name. The default value is redshift-test.
Configure the stack parameters. The following table provides default values.

Parameter name	Default value	Description
`VPCCIDR`	10.0.0.0/16	CIDR block for the VPC
`PrivateSubnet1CIDR`	10.0.1.0/24	CIDR block for the first private subnet
`PrivateSubnet2CIDR`	10.0.2.0/24	CIDR block for the second private subnet
`CACommonName`	redshift-ca.example.com	Common Name for the Certificate
`CAOrganization`	Example Corp	Organization for the Certificate Authority
`CACountry`	US	Country for the Certificate Authority
`CAValidityInDays`	1826	Validity period in days for the CA Certificate (5 years)
`RedshiftClusterIdentifier`	`my-redshift-cluster`	Identifier for the Amazon Redshift cluster
`RedshiftDatabaseName`	`dev`	Name of the initial database in the Amazon Redshift cluster
`RedshiftMasterUsername`	`admin`	Main username for the Amazon Redshift cluster
`RedshiftNodeType`	`ra3.xlplus`	Node type for the Amazon Redshift cluster
`ServerlessNamespace`	`my-serverless-namespace`	Namespace identifier for Amazon Redshift Serverless
`ServerlessWorkgroup`	`my-serverless-workgroup`	Workgroup identifier for Amazon Redshift Serverless

Select the acknowledgement checkbox and choose Create Stack. Stack deployment takes about 10 minutes to complete.

When stack creation is complete, navigate to the Outputs tab on the AWS CloudFormation console and note down the values for the resources that the stack created.

The following table shows a summarized view of the output values.

Output	Description	Example value
`CertificateAuthorityArn`	Amazon Resource Name (ARN) of the Private Certificate Authority	`arn:aws:acm-pca:aa-example-1:111122223333:certificate-authority/a1b2c3d4-5678-90ab-cdef-EXAMPLE22222`
`ClientCertificateArn`	ARN of the sample client certificate	`arn:aws:acm:aa-example-1:111122223333:certificate/a1b2c3d4-5678-90ab-cdef-EXAMPLE11111`
`ProfileArn`	ARN of the IAM Roles Anywhere profile	`arn:aws:rolesanywhere:aa-example-1:111122223333:profile/a1b2c3d4-5678-90ab-cdef-EXAMPLE44444`
`RedshiftAccessRoleArn`	ARN of the Amazon Redshift Access role	`arn:aws:iam::1222345677:role/Redshift-test-RedshiftAccessRole`
`TrustAnchorArn`	ARN of the IAM Roles Anywhere profile. You will use this value for configuring `credential_process` for IAM Roles Anywhere in a later step.	`arn:aws:rolesanywhere:aa-example-1:111122223333:trust-anchor/a1b2c3d4-5678-90ab-cdef-EXAMPLE33333`
`RedshiftClusterEndpoint`	Private endpoint of the Amazon Redshift Cluster	`my-redshift-cluster-123456789012.aa-example-1.redshift.amazonaws.com`
`RedshiftClusterPort`	Port of the Amazon Redshift Cluster	`5439`
`ServerlessWorkgroupEndpoint`	Private endpoint of Amazon Redshift Serverless Workgroup	`my-serverless-workgroup-123456789012.aa-example-1.redshift.serverless.amazonaws.com`

Export a sample private certificate using CloudShell

To export a sample private certificate using CloudShell, complete the following steps.

Open CloudShell. For more details, see Navigating the AWS CloudShell interface.
Export the certificate ARN from the CloudFormation outputs. If you changed the stack name in the previous step, use that value for <stack-name>. Otherwise, use the default value redshift-public-iam-roles-anywhere.

export CERT_ARN=$(aws cloudformation describe-stacks \
    --stack-name <stack-name> \
    --query 'Stacks[0].Outputs[?OutputKey==`ClientCertificateArn`].OutputValue' \
    --output text)

Extract the certificate and private key files:

# Generate and save the passphrase
export PASSPHRASE=$(openssl rand -base64 32)
# Export certificate using environment variables
aws acm export-certificate \
    --certificate-arn $CERT_ARN \
    --passphrase $(echo -n "$PASSPHRASE" | base64) \
    > cert_export.json
# Extract components to separate files
jq -r '.Certificate' cert_export.json > certificate.pem
jq -r '.PrivateKey' cert_export.json > encrypted_private_key.pem
# Decrypt the private key
openssl rsa -in encrypted_private_key.pem -out private_key.pem -passin pass:"$PASSPHRASE"
# Clear environment variables
unset PASSPHRASE CERT_ARN

Download the extracted certificate and private key files from CloudShell:

/home/cloudshell-user/certificate.pem
/home/cloudshell-user/private_key.pem

Secure the private key on your local workstation.

After downloading the files, restrict file permissions to prevent unauthorized access:

chmod 400 private_key.pem chmod 400 certificate.pem

For production workloads, consider storing private keys in your operating system’s keychain (macOS Keychain, Windows Certificate Store), a hardware security module (HSM), or a secrets management tool rather than as files on disk.

Configure an AWS CLI profile

These are the steps to configure an AWS CLI profile on your system:

Store the downloaded certificate and private key to your environment. For an automated approach to generate and rotate certificates, see Set up AWS Private Certificate Authority to issue certificates for use with IAM Roles Anywhere.
Create a new profile named onprem-redshift. This invokes the credential process. Replace the placeholders with your specific values. Find the values for trusted-anchor-arn, profile-arn, and role-arn in your CloudFormation stack outputs.

aws configure set profile.onprem-redshift.credential_process "</path/to/aws_signing_helper> credential-process \
      --certificate </path/to/certificate.pem> \
      --private-key </path/to/private_key.pem> \
      --trust-anchor-arn <trusted-anchor-arn> \
      --profile-arn <profile-arn> \
      --role-arn < role-arn>"

Verify your configuration. Open the ~/.aws/config file and confirm that it contains a profile.

[profile onprem-redshift]
credential_process = </path/to/aws_signing_helper> credential-process       
--certificate </path/to/certificate.pem>       
--private-key </path/to/private_key.pem>       
--trust-anchor-arn <trusted-anchor-arn>       
--profile-arn <profile-arn>       
--role-arn <role-arn>

Test the solution

Follow these steps to validate your setup for provisioned clusters to confirm end-to-end connectivity:

Verify network connectivity

Before testing authentication, confirm that your on-premises environment can reach the Amazon Redshift cluster endpoint:

telnet my-redshift-cluster.abc123.us-east-1.redshift.amazonaws.com 5439

If the connection succeeds, you should see a response indicating the port is open. If it fails, verify your VPN/Direct Connect configuration and security group rules.

Create database user

If you haven’t already created a user, connect to your Amazon Redshift as the main user and create a dedicated user for testing:

CREATE USER analytics_user PASSWORD '[PASSWORD]';

Retrieve Amazon Redshift database credentials

With the configuration in place, request temporary database credentials from Amazon Redshift:

aws redshift get-cluster-credentials \
  --db-user analytics_user \
  --cluster-identifier my-redshift-cluster \
  --region us-east-1 \
  --profile onprem-redshift

This call returns a short-lived username and password that’s valid for connecting to the cluster. By default, the temporary credentials expire in 900 seconds. You can optionally specify a duration between 900–3600 seconds (15–60 minutes).

Connect using JDBC/ODBC or psql

Use the issued credentials in your connection string. For JDBC:

jdbc:redshift://my-redshift-cluster.abc123.redshift.amazonaws.com:5439/dev?ssl=true&UID=analytics_user&PWD=<temporary_password>

For psql:

PGPASSWORD=<temporary_password> psql \
  -h my-redshift-cluster.abc123.redshift.amazonaws.com \
  -p 5439 \
  -U analytics_user \
  -d dev \
  --set=sslmode=verify-full

Validate and monitor

Test authentication flows end-to-end using your ETL jobs.
Review AWS CloudTrail logs to validate. It records role assumptions and Amazon Redshift API calls.
Monitor session expiration to help workloads handle credential refresh seamlessly.

Testing end-to-end connectivity for Amazon Redshift Serverless

The testing process for Amazon Redshift Serverless follows a similar pattern to provisioned clusters, with minor differences in the API calls and connection parameters. These steps validate connectivity to your serverless workgroup.

Verify network connectivity

telnet my-serverless-workgroup.abc123.us-east-1.redshift.amazonaws.com 5439

Retrieve Amazon Redshift Serverless database credentials

aws redshift-serverless get-credentials \
  --workgroup-name my-serverless-workgroup \
  --db-name dev \
  --region us-east-1 \
  --profile onprem-redshift

Connect using JDBC/ODBC or psql

PGPASSWORD="<password_from_get_credentials>" psql \
  -h my-serverless-workgroup.abc12.us-east-1.redshift-serverless.amazonaws.com \
  -p 5439 \
  -U "IAMR:Redshift-IAMRA-RedshiftAccessRole" \
  -d dev \
  --set=sslmode=verify-full

Clean up

To avoid future charges, remove the deployed resources:

Delete the CloudFormation stack.
Remove the generated files from CloudShell:

rm cert_export.json encrypted_private_key.pem certificate.pem private_key.pem

Conclusion

In this post, we showed how to implement IAM Roles Anywhere with Amazon Redshift so that enterprises can securely connect on-premises data systems to their cloud data warehouse without relying on static credentials or public internet access. This architecture provides short-lived, auditable credentials, integrates with existing certificate authorities, and helps ensure authentication and data flows remain private and trusted.

With this approach, data engineers and developers can run ingestion pipelines, ETL jobs, and analytics queries, while security teams maintain full control through IAM governance and CloudTrail auditing. You can remove manual credential rotation tasks, allow your data engineers to connect to Amazon Redshift without managing static keys, and achieve complete audit trails through CloudTrail integration for your hybrid analytics environments.

To get started, deploy the solution using the CloudFormation template and follow the steps in this post. To learn more about the services used, see the following resources:

About the authors

Enhancing Identity Intelligence with Babel Street Match and Amazon OpenSearch

Kunal Sharma — Thu, 16 Apr 2026 18:21:52 +0000

This post is co-authored with Gil Irizarry, Mae Wells-Kress and Craig Harmon from Babel Street.

Can your system tell “John Smith” apart from “John Smith”?

Organizations requiring identity intelligence increasingly face challenges due to complexity of matching names and entities across vast, multilingual, and constantly evolving datasets. Whether helping border security, combating financial crimes, or maintaining regulatory compliance, the accuracy of identity and entity resolution directly determines whether threats are detected, investigations succeed, and regulatory requirements are met. Yet, linguistic diversity, transliterations, inconsistent data formats, and legacy system limitations continue to create friction, leading to false positives, missed matches, and costly manual reviews. As customers ingest and analyze petabytes of unstructured and structured data in Amazon OpenSearch Service, the need for intelligent, scalable, and multilingual matching becomes increasingly important. This is where the integration of Babel Street (an AWS Partner) with OpenSearch Service provides a solution that helps organizations enhance precision, reduce noise, and accelerate insights from their high-volume data environments.

This post explores how combining Babel Street Match with OpenSearch Service provides a solution that helps your organization to handle large-scale, multilingual data.

The growing complexity of identity and entity resolution

As organizations ingest and analyze massive volumes of multilingual and inconsistently formatted data, accurately matching names and entities becomes increasingly difficult. Variations in spelling, transliterations, semantic differences, cultural naming conventions, and incomplete or noisy records can contribute to mismatches. These challenges are compounded by legacy systems, fragmented data pipelines, operational inefficiencies, and evolving regulatory requirements—especially in sectors where precision is a requirement.

Evaluating and enhancing identity in high-volume enterprise environments

Amazon OpenSearch Service is a fully managed, scalable search and analytics service that enables organizations to ingest, search, visualize, and analyze massive volumes of data in near real time. Built to handle structured and unstructured information from diverse sources, it powers use cases ranging from security analytics and log monitoring to enterprise search and advanced analytical applications.

Babel Street delivers risk intelligence trusted by organizations across government, defense, and the private sector. The offering combines access to vast volumes of multilingual data with advanced analytics to uncover hidden identities, secure vendor networks, and identify emerging risks with precision, speed, and scale. From national security to regulatory compliance and enterprise resilience, Babel Street provides the strategic advantage needed to stay ahead of risk, safeguard operations, and protect missions.

Babel Street Match, an offering from Babel Street incorporates advanced identity risk intelligence capabilities, which enhance the precision and reliability of screening processes. This advanced solution uses sophisticated matching techniques to verify identities and identify variations in personal data—including aliases, alternate spellings, and differences in biographical details, helping organizations separate legitimate individuals from potential threats. The ability to screen names, addresses, dates, and other identifiers across different scripts and languages helps reduce false positives and negatives, helps accurately detect critical risks with transparent scoring to meet compliance and audit requirements. Further, Babel Street Match streamlines screening workflows, reduces the burden of manual reviews, and elevates the accuracy of threat detection.

The following diagram shows the details of OpenSearch Service and Babel Street Match Plugin integration.

Babel Street Match integrates directly with the OpenSearch Service domain through a lightweight plugin that runs inside your own AWS account where you have full control of your data. The Match plugin sends encrypted match requests to Babel Street’s fully managed Match engine, where the core matching engine performs the entity-resolution logic. The results return to you in real time, enhancing your existing OpenSearch Service workflows with advanced name- and entity-matching capabilities. Meanwhile, Babel Street’s control plane handles licensing, monitoring, and AWS Marketplace integration behind the scenes, provides continuous validation, automated updates, and a seamless operational experience.

Example use cases

The solution combines enterprise-scale search and analytics with AI-powered, multilingual identity intelligence. This section showcases example use cases where integration has enhanced organizations’ capabilities.

Border Screening: Help agencies identify high-risk travelers, cargo, and networks to strengthen point-of-entry security with faster, automated risk assessment.
Financial Services Compliance: Help Financial institutions and the FinTechs that serve them by offering AI-driven solutions for name screening, adverse media monitoring, and know your customer (KYC)/know your vendor (KYV) due diligence.
Identity and Organization Screening: Help businesses needing identity and organization screening by providing AI, analytics, and advanced matching technologies to assist in addressing complex screening challenges.
Customer and Vendor Onboarding: Help governments and financial institutions by providing research, analytics, and advanced matching technologies needed to quickly and confidently onboard customers and vendors at scale.

Customer Success Stories

Here’s how leading organizations are leveraging Babel Street Match and Amazon OpenSearch Service to solve real-world identity challenges:

A European online brokerage faced AML (anti-money laundering) compliance challenges with its outdated name-matching system, which produced excessive false positives and couldn’t process longer multilingual names. After implementing Babel Street Match on OpenSearch Service, the firm achieved up to 70% better accuracy across 25 languages—significantly reducing manual work and speeding customer payments.
Babel Street Match Improves FI’s Name-Matching Accuracy by Up to 70% on OpenSearch
A major border agency struggled with an outdated screening system that flagged 15% of travelers as potential watchlist matches—overwhelming agents and creating long queues. After implementing Babel Street Match, false positives dropped dramatically (from 80,000 to just 100 in one test), hardware needs fell by 70%, and travelers with common names can now pass through faster. As one stakeholder put it: “Name matching is not our biggest problem anymore.”
Enabling Stronger, Safer Borders with AI-powered Screening by Babel Street Match

Getting Started with Babel Street Match for Amazon OpenSearch Service

Amazon OpenSearch Service supports third-party plugins like Babel Street Match for OpenSearch. This plugin is supported on OpenSearch version 2.15 or higher and licenses can be obtained through AWS Marketplace.

Installing Babel Street Match for Amazon OpenSearch Service

Prerequisites: Obtain the license file from Babel Street and upload it to an S3 bucket in the same AWS Region as your OpenSearch domain.

Installation Steps:

Create packages – In the OpenSearch Service console, create a package for your license file and select the Babel Street Match plugin from the available options
Associate packages – Link both the license and plugin packages to your OpenSearch domain
Verify – Monitor the domain update and confirm the plugin is active

For details, refer to AWS documentation “Installing third-party plugins in Amazon OpenSearch Service” and Babel Street installation guide which provides detailed guidance on pre-requisites, installation and using the plugin.

Conclusion

Together, Babel Street Match and OpenSearch Service help organizations cut through false positives and catch true matches faster. The result? Greater precision, efficiency, and speed—whether protecting entities, maintaining compliance, or securing supply chains. That’s business-critical identity intelligence in action.

Explore how Babel Street Match on Amazon OpenSearch Service can elevate your organization’s identity intelligence capabilities and transform the screening operations through an interactive or customized demo on Babel Street’s website.

Portions of this content describing Babel Street products and services are provided by Babel Street. AWS is not responsible for the accuracy of third-party product information.

About the Authors

Getting started with Apache Iceberg write support in Amazon Redshift – Part 2

Sanket Hase — Wed, 15 Apr 2026 21:29:36 +0000

In Getting started with Apache Iceberg write support in Amazon Redshift – part 1, you learned how to create Apache Iceberg tables and write data directly from Amazon Redshift to your data lake. You set up external schemas, created tables in both Amazon Simple Storage Service (Amazon S3) and S3 Tables, and performed INSERT operations while maintaining ACID (Atomicity, Consistency, Isolation, Durability) compliance.

Amazon Redshift now supports DELETE, UPDATE, and MERGE operations for Apache Iceberg tables stored in Amazon S3 and Amazon S3 table buckets. With these operations, you can modify data at the row level, implement upsert patterns, and manage the data lifecycle while maintaining transactional consistency using familiar SQL syntax. You can run complex transformations in Amazon Redshift and write results to Apache Iceberg tables that other analytics engines like Amazon EMR or Amazon Athena can immediately query.

In this post, you work with customer and orders datasets that were created and used in the previously mentioned post to demonstrate these capabilities in a data synchronization scenario.

Solution overview

This solution demonstrates DELETE, UPDATE, and MERGE operations for Apache Iceberg tables in Amazon Redshift using a common data synchronization pattern: maintaining customer records and orders data across staging and production tables. The workflow includes three key operations:

DELETE – Remove customer records based on opt-out requests
UPDATE – Modify existing customer information
MERGE – Synchronize order data between staging and production tables using upsert patterns

Figure 1: solution overview

The solution uses a staging table (orders_stg) stored in an S3 table bucket for incoming data and reference tables (customer_opt_out) in Amazon Redshift for managing data lifecycle operations. With this architecture, you can process changes efficiently while maintaining ACID compliance across both storage types.

Prerequisites

For this walkthrough, you should have completed the setup steps from Getting started with Apache Iceberg write support in Amazon Redshift – part 1, including:

Create an Amazon Redshift data warehouse (provisioned or Serverless)
Set up the required IAM role (RedshifticebergRole) with appropriate permissions
Create an Amazon S3 bucket and S3 Table bucket
Configure AWS Glue Data Catalog database and setting up access
Set up AWS Lake Formation permissions
Create the customer Apache Iceberg table in Amazon S3 standard buckets with sample customer data
Create the orders Apache Iceberg table in Amazon S3 Table buckets with sample order data
Amazon Redshift data warehouse on p200 version or higher

Data preparation

In this section, you set up the sample data needed to demonstrate MERGE, UPDATE, and DELETE operations. To prepare your data, complete the following steps:

Log in to Amazon Redshift using Query Editor V2 with the Federated user option.
Create the orders_stg and customer_opt_out tables with sample data:

CREATE TABLE "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders_stg
(
customer_id BIGINT,
order_id BIGINT,
Total_order_amt DECIMAL(10,2),
Total_order_tax_amt REAL,
tax_pct DOUBLE PRECISION,
order_date DATE,
order_created_at_tz TIMESTAMPTZ,
is_active_ind BOOLEAN
)
USING ICEBERG;
INSERT INTO "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders_stg
(order_date, order_id, customer_id, total_order_amt, total_order_tax_amt, tax_pct, order_created_at_tz, is_active_ind)
VALUES
('2024-11-11', 1016, 10, 167.45, 13.40, 0.08, '2024-11-11 06:55:00-06:00', true),
('2024-11-12', 1017, 15, 34.99, 2.80, 0.08, '2024-11-12 23:30:30-06:00', true),
('2024-11-09', 1014, 9, 500.60, 56.80, 0.09, '2024-11-09 16:20:55-06:00', true),
('2024-11-10', 1015, 5, 329.85, 33.51, 0.08, '2024-11-10 11:45:30-06:00', true);
select * from "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders_stg;

Figure 2: orders_stg result set

CREATE TABLE dev.public.customer_opt_out
(
customer_id bigint,
customer_name varchar,
opt_out_ind char(1),
cust_rec_upd_ind char(1)
);
INSERT INTO dev.public.customer_opt_out VALUES
(9, 'Customer9 Martinez', 'Y', 'N'),
(12, 'Customer12 Thomas', 'Y', 'N'),
(13, 'Customer13 Albon', 'N', 'Y'),
(14, 'Customer14 Oscar', 'N', 'Y');
select * from dev.public.customer_opt_out;

Figure 3: customer_opt_out result set

You can now use the orders_stg and customer_opt_out tables to demonstrate data manipulation operations on the orders and customer tables created in the prerequisite section.

MERGE

MERGE conditionally inserts, updates, or deletes rows in a target table based on the results of a join with a source table. You can use MERGE to synchronize two tables by inserting, updating, or deleting rows in one table based on differences found in the other table.

To perform a MERGE operation:

Verify that the current data in the orders table for order IDs 1014, 1015, 1016, and 1017.You loaded this sample data in Part 1:

select * from "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
where order_id in (1014,1015,1016,1017);

Figure 4: orders data for existing orders for orders in orders_stg

The orders table contains existing rows for order IDs 1014 and 1015.

Run the following MERGE operation using order_id as the key column to match rows between the orders and orders_stg tables:

MERGE INTO "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders
USING "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orders_stg
ON orders.order_id = orders_stg.order_id
WHEN MATCHED THEN UPDATE 
SET
customer_id         = orders_stg.customer_id,
total_order_amt     = orders_stg.total_order_amt,
total_order_tax_amt = orders_stg.total_order_tax_amt,
tax_pct             = orders_stg.tax_pct,
order_date          = orders_stg.order_date,
order_created_at_tz = orders_stg.order_created_at_tz,
is_active_ind       = orders_stg.is_active_ind
WHEN NOT MATCHED THEN INSERT
VALUES 
(orders_stg.customer_id,orders_stg.order_id,orders_stg.total_order_amt,orders_stg.total_order_tax_amt,orders_stg.tax_pct,orders_stg.order_date,orders_stg.order_created_at_tz,orders_stg.is_active_ind);

The operation updates existing rows (1014 and 1015) and inserts new rows for order IDs that don’t exist in the orders table (1016 and 1017).

Verify the updated data in the orders table:

select * from "iceberg-write-blog@s3tablescatalog".iceberg_write_namespace.orderswhere order_id in (1014,1015,1016,1017);

Figure 5: merged data on orders from orders_stg

The MERGE operation performs the following changes:

Updates existing rows – Order IDs 1014 and 1015 have updated total_order_amt and total_order_tax_amt values from the orders_stg table
Inserts new rows – Order IDs 1016 and 1017 are inserted because they don’t exist in the orders table

This demonstrates the upsert pattern, where MERGE conditionally updates or inserts rows based on the matching key column.

UPDATE

UPDATE modifies existing rows in a table based on specified conditions or values from another table.

Update the customer Apache Iceberg table using data from the customer_opt_out Amazon Redshift native table. The UPDATE operation uses the cust_rec_upd_ind column as a filter, updating only rows where the value is ‘Y’.

To perform an UPDATE operation:

Verify the current customer_name values for customer IDs 13 and 14 in customer_opt_out and customer (loaded this sample data in Part 1) tables:

select * from dev.public.customer_opt_out
where cust_rec_upd_ind = 'Y';

Figure 6: verify existing customer data for customers from customer_opt_out

select customer_id,customer_name from dev.demo_iceberg.customer
where customer_id in(13,14);

Figure 7: verify existing customer name for customers from customer_opt_out

Run the following UPDATE operation to modify customer names based on the cust_rec_upd_ind from customer_opt_out:

UPDATE dev.demo_iceberg.customerSET customer_name = customer_opt_out.customer_name
FROM dev.public.customer_opt_out
WHERE customer_opt_out.cust_rec_upd_ind = 'Y'and customer.customer_id = customer_opt_out.customer_id;

Verify the changes for customer IDs 13 and 14:

select customer_id,customer_name from dev.demo_iceberg.customer where customer_id in(13,14) order by 1;

Figure 8: updated customer names in customer table

The UPDATE operation modifies the customer_name values based on the join condition with the customer_opt_out table. Customer IDs 13 and 14 now have updated names (Customer13 Albon and Customer14 Oscar).

DELETE

DELETE removes rows from a table based on specified conditions. Without a WHERE clause, DELETE removes all the rows from table.

Delete rows from the customer Apache Iceberg table using data from the customer_opt_out Amazon Redshift native table. The DELETE operation uses the opt_out_ind column as a filter, removing only rows where the value is ‘Y’.

To perform a DELETE operation:

Verify the opt-out indicator data in the customer_opt_out table:

select * from dev.public.customer_opt_out
where opt_out_ind = 'Y';

Figure 9: verify customer records for opt out

Verify the current customer data for customer IDs 9 and 12:

select * from dev.demo_iceberg.customerwhere customer_id in(9,12);

Figure 10: verify existing customers data in customer table for opt out

Review the query execution plan:

EXPLAINDELETE FROM demo_iceberg.customerUSING public.customer_opt_out
WHERE customer.customer_id = customer_opt_out.customer_id
AND customer_opt_out.opt_out_ind = 'Y';

Figure 11: query plan for the DELETE query. The execution plan shows Amazon S3 scans for Apache Iceberg format tables, indicating that Amazon Redshift removes rows directly from the Amazon S3 bucket.

Run the following DELETE operation:

DELETE FROM demo_iceberg.customer
USING public.customer_opt_out
WHERE customer.customer_id = customer_opt_out.customer_id
AND customer_opt_out.opt_out_ind = 'Y';

Verify that the rows were removed:

select * from dev.demo_iceberg.customer where customer_id in(9,12);

Figure 12: result set from customer table for opt out customer after delete

The query returns no rows, confirming that customer IDs 9 and 12 were successfully deleted from the customer table.

Best practices

After performing multiple UPDATE or DELETE operations, consider running table maintenance to optimize read performance:

For AWS Glue tables – Use AWS Glue table optimizers. For more information, see Table optimizers in the AWS Glue Developer Guide.
For S3 Tables – Use S3 Tables maintenance operations. For more information, see S3 Tables maintenance in the Amazon S3 User Guide.

Table maintenance merges and compacts deletion files generated by Merge-on-Read operations, improving query performance for subsequent reads.

Conclusion

You can use Amazon Redshift support for DELETE, UPDATE, and MERGE operations on Apache Iceberg tables to build data architectures that combine warehouse performance with data lake scalability. You can modify data at the row level while maintaining ACID compliance, giving you the same flexibility with Apache Iceberg tables as you have with native Amazon Redshift tables.

Get started:

Review the Amazon Redshift Iceberg integration documentation for complete syntax reference
Explore Writing to Apache Iceberg tables for detailed examples
Learn about table maintenance best practices for AWS Glue tables
Discover S3 Tables maintenance operations for improving query performance

About the authors

Get to insights faster using Notebooks in Amazon SageMaker Unified Studio

Praveen Kumar — Wed, 15 Apr 2026 21:24:36 +0000

In this post, we demonstrate how Notebooks in Amazon SageMaker Unified Studio help you get to insights faster by simplifying infrastructure configuration. You’ll see how to analyze housing price data, create scalable data tables, run distributed profiling, and train machine learning (ML) models within a single notebook environment.

Data scientists and analysts often spend days configuring infrastructure and managing authentication across multiple data sources before they can begin analysis. When working with data across Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Snowflake, and local files, teams face repeated authentication setup, manual compute scaling decisions, and tool-switching overhead that delays insights.

Notebooks in Amazon SageMaker Unified Studio provide instant access to 12+ data sources, compute scaling from local to distributed processing, and AI-powered code generation within a single browser-based environment. You’ll learn to use polyglot programming, multi-engine compute, and AI-assisted development to accelerate your path from question to insight.

What are Notebooks in Amazon SageMaker Unified Studio?

Notebooks in Amazon SageMaker Unified Studio provide an interactive environment for data analysis, exploration, engineering, and machine learning workflows. It delivers five integrated capabilities:

Polyglot programming: Write code in Python and SQL interchangeably within the same notebook environment
Unified data access: Connect instantly to data stored in Amazon S3, AWS Glue Data Catalog, Apache Iceberg tables, and third-party sources like Snowflake and BigQuery
Native visualization: Create charts directly from Python and SQL results for immersive data analytics
AI-powered development: Generate code through natural language prompts using SageMaker Data Agent, with an intelligent chat interface for data analytics, data science, and ML tasks
Flexible compute: Scale from basic instances to GPU-powered environments as your needs grow

Architecture

This section covers the architecture of Notebooks, which delivers enterprise-scale analytics with browser-based simplicity through a cloud-native architecture that integrates multiple compute engines, diverse data sources, and AI-powered assistance.

Presentation layer

You access the notebook interface through Amazon SageMaker Unified Studio, interacting with a familiar interface featuring code cells for execution, markdown cells for documentation, and visualization cells for charts and tables.

Compute layer

A dedicated notebook server manages your kernel lifecycle and session state. Key components include a Language Server for code completion, a Python 3.11 runtime with pre-loaded data science libraries, and a Polyglot Kernel that handles your Python, PySpark, and SQL execution within the same notebook. Persistent Amazon Elastic Block Store (Amazon EBS) storage backs each notebook you create.

Execution layer

Notebooks support multiple execution engines, automatically routing your code to the optimal processing engine. In-memory execution handles your smaller datasets and rapid prototyping. Apache Spark via Amazon Athena provides distributed processing for your large-scale analytics via Spark Connect. Native connectivity to Amazon Athena (Trino), Amazon Redshift, Snowflake, and BigQuery processes your SQL queries.

Data Integration

You get unified access to 12+ data sources including AWS-native (Amazon S3, AWS Glue, Amazon Athena, Amazon Redshift) and third-party (Snowflake, BigQuery, PostgreSQL, MySQL) data sources. For the latest supported data sources, see Connect to data sources .

AI layer

The SageMaker Data Agent operates in two modes to assist you: an Agent Panel for multi-step analytical workflows and Inline Assistance for focused, cell-level code generation. For a detailed overview, see Accelerate context-aware data analysis and ML workflows with Amazon SageMaker Data Agent .

Security is embedded throughout the architecture to protect your work. Data access respects your AWS Identity and Access Management (AWS IAM) permissions. The notebook and the agent can only access data sources you’re authorized to use. Communication between components uses encrypted channels, and your notebook storage is encrypted at rest. The AI agent includes built-in guardrails to help prevent destructive operations and logs interactions for your compliance and auditing purposes.

Prerequisites

Before you begin, you need:

An AWS account with appropriate permissions to create Amazon SageMaker Unified Studio resources. See Set up IAM-based domains for complete permission requirements.
Basic familiarity with Python programming and SQL queries
Understanding of data analysis concepts and ML workflows
Access to the sample housing dataset (provided in the walkthrough)

Getting started with Notebooks

To get started, open the Amazon SageMaker console and choose Get started.

You will be prompted either to select an existing AWS Identity and Access Management (AWS IAM) role that has access to your data and compute, or to create a new role. For this walkthrough, choose Create a new role and leave the other options at their defaults.

Choose Set up. It takes a few minutes to complete your environment.

Use case

In this post, you’ll use a Notebook and the SageMaker Data Agent to perform the following:

Working with dataset: Upload sample dataset housing.csv and explore with data explorer
Polyglot programming: Query dataframes with SQL via DuckDB
Multi-engine access via AWS Glue: Create an AWS Glue table to unlock Athena SQL/Spark engines for distributed processing
Advanced analytics: Use Athena Spark for data profiling
AI-assisted development: Generate profiling and ML code with Data Agent
ML workflow: Train Random Forest model and evaluate results

First, let’s walk through the interface and explore its core capabilities.

Understanding the interface

The Notebooks interface follows familiar notebook conventions with cells for code execution and markdown for documentation. Within the notebook, you’ll see your current programming environment (such as Python 3.11) and compute profile specifications. The interface allows you to:

Access your data by browsing files, exploring data catalogs, and managing third-party connections
Monitor variables created within your notebook context
Scale compute resources on demand by adjusting virtual CPUs and RAM based on your workload requirements, even scaling up to GPU instances
Manage packages by installing and configuring Python packages as needed

Working with the dataset

For this walkthrough, you’ll use the housing.csv sample dataset which you can download from this page. (the file is named canvas-sample-housing.csv on the linked page). Choose the Files icon in the left panel and choose the Local tab. Upload the CSV file to the notebook on the Local tab.

Notebooks provide you with instant access to your data assets. Using the data explorer, you can browse your AWS Glue Data Catalog, Amazon S3 table catalogs, Amazon S3 buckets, and configured third-party connections.

Choose the three-dot options menu.

Choose Read as dataframe, then run the inserted cell in the notebook to view the results.

import pandas as pd
<<df_csv_xxxx>> = pd.read_csv('housing.csv')
<<df_csv_xxxx>>

When you return a dataframe, Notebooks render it in a rich table format with automatic data profiling.

Polyglot programming: Python and SQL together

One of the most powerful features in Notebooks is the interoperability between Python and SQL. After you load data into a Python dataframe, you can immediately query it using SQL. For example, to calculate total population and household by ocean proximity, you can run:

select sum(population) ,sum(households),ocean_proximity 
from<<df_csv_xxxx>> 
group byocean_proximity

The notebook’s autocomplete functionality recognizes dataframes in your context, making SQL queries intuitive.

This SQL query runs on DuckDB (an in-memory SQL database engine), which requires no separate installation or server maintenance on your part. DuckDB’s lightweight design integrates into Python, Java, and other environments, making it ideal for your rapid interactive data analysis. For distributed processing needs, you can use engines such as Apache Spark or Trino after creating an AWS Glue table for this dataset.

Create an AWS Glue table for the dataset

After you create an AWS Glue table, you can query the dataset using various AWS Glue catalog-compatible engines, including Amazon Athena SQL (Trino) and Amazon Athena Spark. These engines deliver optimal price-performance for your specific workload requirements.

Start by creating an AWS Glue database. To do that, create a new cell in the notebook by choosing SQL and selecting Amazon Athena (SQL).

Run this SQL to create a database: create database demo;

Next, go to data explorer and choose +Add on the top left, then choose Create table. Choose the database you created earlier and enter a name for the table. Upload the housing.csv dataset file used earlier. Continue by choosing Next in the side panel to create the table.

Next, let’s run a sample SQL query in a new cell using Amazon Athena SQL:

select sum(population) , sum(households), ocean_proximity 
fromdemo.housing
group by ocean_proximity

Advanced capabilities with Athena Spark

Before you can build an ML model to predict house prices, let’s analyze the dataset further and run data profiling for additional insights. For advanced exploration, you can use Amazon Athena Spark within your notebook.To do that, you’ll create a new Python cell which has a built-in Spark session. Run the following code to check the Spark version:

# Verify Spark version
spark.version

Using the SageMaker Data Agent for data profiling

Instead of writing boilerplate code manually, you can use the built-in generative AI capability.

Prompt: “Perform data profiling and create visualization for housing table”

The AI assistant generates comprehensive profiling code for you, including basic statistics calculation, column-level profiling, data type analysis, and missing value detection.

The agent accessed your AWS Glue Data Catalog, understood your housing table structure, and generated profiling code tailored to your specific columns and data types. This context awareness reduces the trial-and-error cycle you’d normally face when adapting generic code snippets to your environment. Review the generated code and run it. The fast response times help you iterate on your analysis efficiently.

If you encounter an error, you can resolve it using Fix with AI as shown in the following figure. When errors occur during execution, the “Fix with AI” feature analyzes the traceback, diagnoses the root cause, and generates corrected code, so you can keep your analysis moving forward.

Training ML models

Next, you’ll use the data agent to generate code for training a model that predicts housing prices.

Prompt: “Generate code to train a model that predicts housing prices. Use table housing.”

The AI assistant generates end-to-end code for you that:

Reads housing data from AWS Glue catalog using Amazon Athena Spark and converts to pandas
Converts string columns to numeric, encodes using one-hot encoding and removes missing values
Trains a Random Forest model to predict median house values
Evaluates model performance (RMSE, MAE, R-square)
Displays top 10 most important features for predictions

This multi-step orchestration saves you hours of development time by handling the entire workflow from data access to model evaluation.

If you encounter an error, you can resolve it using Fix with AI available in the results traceback section.

This workflow showcased Notebooks’ unified capabilities: you uploaded files locally, created AWS Glue tables for multi-engine access, used Amazon Athena Spark for distributed profiling, and used AI-assisted ML development to predict housing prices. All of this happened within a single notebook environment without switching tools.

Key benefits and best practices

Notebooks in Amazon SageMaker Unified Studio deliver several advantages:

Faster time to insights: With traditional environments, you might spend hours on configuration before analysis begins. Notebooks bypass this overhead, so you can start work immediately.
Improved collaboration: You can share notebooks with consistent environments, supporting reproducibility and reducing “works on my machine” issues.
Reduced complexity: You can access multiple data sources and compute engines from one interface rather than navigating separate tools for each data source or processing engine.
AI-accelerated development: Generate task-specific code and receive intelligent suggestions, reducing time spent on repetitive coding tasks.
Scalable performance: Handle datasets from megabytes to petabytes with appropriate compute resources. The system scales automatically as data volumes grow.

Best practices

Start with appropriate compute profiles by beginning with smaller instances and scaling up as your needs grow.
Use AI assistance with natural language prompts for your repetitive tasks and complex operations.
Combine engines strategically by using Amazon Athena Spark for your large-scale processing, Amazon Redshift for data warehousing and other specialized engines for your specific workloads.
Document your work using markdown cells to create living documentation alongside your code.
Organize using multiple cells by breaking the complex workflows into logical steps for better readability and debugging.

Cleaning up

To avoid incurring future charges, delete the resources you created in this walkthrough:

In the Amazon SageMaker Unified Studio console, navigate to the Notebook page
Delete the notebook
Delete the demo database and housing table from the AWS Glue Data Catalog
Delete Amazon SageMaker Unified Studio domain created during this walkthrough
If you created a new IAM role specifically for this walkthrough, delete it from the IAM console

Conclusion

In this post, we demonstrated how Notebooks in Amazon SageMaker Unified Studio help you work more efficiently and deliver insights more quickly. By combining familiar notebook interfaces with enterprise-scale compute, multi-engine support, and generative AI assistance, teams can streamline data and AI workflows.

The integration of Python and SQL, instant access to diverse data sources, and intelligent code generation capabilities make Notebooks a valuable tool for modern data teams. Teams can perform exploratory data analysis, build complex data pipelines, or train ML models with the flexibility and power needed within a single, intuitive environment.

Ready to get started? Create your first notebook in Amazon SageMaker Unified Studio and begin analyzing data within minutes.

Explore additional capabilities:

Time series analysis workflows with seasonal decomposition and forecasting
Natural language processing pipelines for text classification and sentiment analysis
Integration with Amazon SageMaker Model Registry for ML model versioning
Advanced Spark optimization techniques for petabyte-scale processing

Learn more:

About the authors

How to use Parquet Column Indexes with Amazon Athena

Matt Wong — Mon, 13 Apr 2026 15:57:18 +0000

Amazon Athena recently added support for reading Parquet Column Indexes in Apache Iceberg tables on November 21, 2025. With this optimization, Athena can perform page-level data pruning to skip unnecessary data within Parquet row groups, potentially reducing the amount of data scanned and improving query runtime for queries with selective filters. For data teams, this may help enable faster insights and help reduce costs when analyzing large-scale data lakes.

Data teams building data lakes often choose Apache Iceberg for its ACID transactions, schema evolution, and metadata management capabilities. Athena is a serverless query engine that allows you to query Amazon S3-based data lakes using SQL, and you don’t need to manage infrastructure. Based on the type of data and query logic, Athena can apply multiple query optimizations to improve performance and reduce costs.

In this blog post, we use Athena and Amazon SageMaker Unified Studio to explore Parquet Column Indexes and demonstrate how they can improve Iceberg query performance. We explain what Parquet Column Indexes are, demonstrate their performance benefits, and show you how to use them in your applications.

Overview of Parquet Column Indexes

Parquet Column Indexes store metadata that query engines can use to skip irrelevant data with greater precision than row group statistics alone. To understand how they work, consider how data is structured within Parquet files and how engines like Athena process them.

Parquet files organize data hierarchically by dividing data into row groups (typically 128-512 MB each) and further subdividing them into pages (typically 1 MB each). Traditionally, Parquet maintains metadata on the contents of each row group level in the form of min/max statistics, allowing engines like Athena to skip row groups that don’t satisfy query predicates. Although this approach reduces the bytes scanned and query runtime, it has limitations. If even a single page within a row group overlaps with the values you are searching for, Athena scans all pages within the row group.

Parquet Column Indexes help address this problem by storing page-level min/max statistics in the Parquet file footer. Row group statistics provide coarse-grained filtering, but Parquet Column Indexes enable finer-grained filtering by allowing query engines like Athena to skip individual pages within a row group. Consider a Parquet file with a single row group containing 5 pages for a column. The row group has min/max statistics of (1, 20), and each page for that column has the following min/max statistics.

row-group-0: min=1, max=20
    page-0: min=1, max=10
    page-1: min=1, max=10
    page-2: min=5, max=15
    page-3: min=6, max=16
    page-4: min=10, max=20

When Athena runs a query filtering for values equal to 2, it first checks the row group statistics and confirms that 2 falls within the range (1, 20). Athena will then plan to scan the pages within that row group. Without Parquet Column Indexes, Athena scans each of the 5 pages in the row group. With Parquet Column Indexes, Athena examines the page-level statistics and determines that only page-0 and page-1 need to be read, skipping the remaining 3 pages.

How to use Parquet Column Indexes with Athena

Athena uses Parquet Column Indexes based on table type:

Amazon S3 Tables: Athena automatically uses Parquet Column Indexes by default when they are present.
Iceberg tables in S3 general purpose buckets: Athena does not use Parquet Column Indexes by default. To allow Athena to use Parquet Column Indexes, add an AWS Glue table property named use_iceberg_parquet_column_index and set it to true. Use the AWS Glue console or AWS Glue UpdateTable API to perform these actions.

Read more about how to use this feature in Use Parquet column indexing.

Measuring Athena performance gains when using Parquet Column Indexes

Now that we understand what Parquet Column Indexes are, we’ll demonstrate the performance benefits of using Parquet Column Indexes by analyzing the catalog_sales table from a 3TB TPC-DS dataset. This table contains ecommerce transaction data including order dates, sales amounts, customer IDs, and product information. This dataset is a good proxy for the types of business analysis that you might perform on your own data, such as identifying sales trends, analyzing customer purchasing patterns, and calculating revenue metrics. We compare query execution statistics with and without Parquet Column Indexes to quantify the performance improvement.

Prerequisites

Before you begin, you must have the following resources:

A SageMaker Unified Studio IAM-based domain.
An Execution IAM Role configured within the SageMaker Unified Studio IAM-based domain with access to S3, AWS Glue Data Catalog, and Athena.
An S3 bucket in your account to store Iceberg table data and Athena query results.

Create catalog_sales Iceberg table

Complete the following steps using SageMaker Unified Studio notebooks. There, you can use SageMaker Unified Studio’s multi-dialect notebook functionality to work with your data using the Athena SQL and Spark engines. To create a catalog_sales Iceberg table in your account, follow these steps:

Navigate to Amazon SageMaker in the AWS Management Console and choose Open under Get started with Amazon SageMaker Unified Studio.
From the side navigation, select Notebooks and choose Create Notebook. The subsequent steps in this post will execute scripts in this notebook.
Create a new SQL cell in the notebook and set the connection type to Athena (Spark). Execute the following query to create a database for the tables in this post.
```
CREATE DATABASE parquet_column_index_blog;
```

Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to create a Hive table pointing to the location of the TPC-DS catalog_sales table data at the public S3 bucket.

CREATE TABLE IF NOT EXISTS parquet_column_index_blog.catalog_sales_hive (
	  cs_sold_time_sk int,
	  cs_ship_date_sk int,
	  cs_bill_customer_sk int,
	  cs_bill_cdemo_sk int,
	  cs_bill_hdemo_sk int,
	  cs_bill_addr_sk int,
	  cs_ship_customer_sk int,
	  cs_ship_cdemo_sk int,
	  cs_ship_hdemo_sk int,
	  cs_ship_addr_sk int,
	  cs_call_center_sk int,
	  cs_catalog_page_sk int,
	  cs_ship_mode_sk int,
	  cs_warehouse_sk int,
	  cs_item_sk int,
	  cs_promo_sk int,
	  cs_order_number bigint,
	  cs_quantity int,
	  cs_wholesale_cost decimal(7, 2),
	  cs_list_price decimal(7, 2),
	  cs_sales_price decimal(7, 2),
	  cs_ext_discount_amt decimal(7, 2),
	  cs_ext_sales_price decimal(7, 2),
	  cs_ext_wholesale_cost decimal(7, 2),
	  cs_ext_list_price decimal(7, 2),
	  cs_ext_tax decimal(7, 2),
	  cs_coupon_amt decimal(7, 2),
	  cs_ext_ship_cost decimal(7, 2),
	  cs_net_paid decimal(7, 2),
	  cs_net_paid_inc_tax decimal(7, 2),
	  cs_net_paid_inc_ship decimal(7, 2),
	  cs_net_paid_inc_ship_tax decimal(7, 2),
	  cs_net_profit decimal(7, 2))
	USING parquet
	PARTITIONED BY (cs_sold_date_sk int)
	LOCATION 's3://blogpost-sparkoneks-us-east-1/blog/BLOG_TPCDS-TEST-3T-partitioned/catalog_sales/'
	TBLPROPERTIES (
	  'parquet.compression'='SNAPPY'
	);

Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to add the Hive partitions to the AWS Glue metadata.
```
MSCK REPAIR TABLE parquet_column_index_blog.catalog_sales_hive;
```
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Replace s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/catalog_sales/ with the S3 URI where you want to store your Iceberg table data, then execute the following query to create the catalog_sales Iceberg table from the Hive table.
```
CREATE TABLE parquet_column_index_blog.catalog_sales
	USING iceberg
	PARTITIONED BY (cs_sold_date_sk)
	LOCATION 's3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/catalog_sales/'
	AS
	SELECT * FROM parquet_column_index_blog.catalog_sales_hive;
```
Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the following query to delete the catalog_sales_hive table, which was only needed to create the catalog_sales Iceberg table.
```
DROP TABLE parquet_column_index_blog.catalog_sales_hive;
```

Run an Athena query without Parquet Column Indexes

After creating the catalog_sales Iceberg table in the preceding steps, we run a simple query that analyzes shipping delays of the top 10 most ordered items. This type of analysis could be critical for ecommerce and retail operations. By identifying which popular items experience the greatest delays, fulfillment teams can focus resources where they matter most. For example, you can adjust inventory placement, change warehouse assignments, or address carrier issues. Additionally, popular items with significant shipping delays are more likely to result in order cancellations or returns, so proactively identifying these issues helps protect revenue.

SELECT cs_item_sk,
    SUM(cs_quantity) as total_orders,
    AVG(cs_ship_date_sk - cs_sold_date_sk) as avg_ship_delay_days,
    MIN(cs_ship_date_sk - cs_sold_date_sk) as min_ship_delay,
    MAX(cs_ship_date_sk - cs_sold_date_sk) as max_ship_delay,
    SUM(
        CASE
            WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
        END
    ) as late_shipments,
    SUM(
        CASE
            WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
        END
    ) * 100.0 / COUNT(*) as late_shipment_pct,
    AVG(cs_ext_ship_cost) as avg_shipping_cost
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IN (
        SELECT cs_item_sk
        FROM parquet_column_index_blog.catalog_sales
        WHERE cs_item_sk IS NOT NULL
        GROUP BY cs_item_sk
        ORDER BY SUM(cs_quantity) DESC
        LIMIT 10
    )
    AND cs_ship_date_sk IS NOT NULL
    AND cs_sold_date_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY avg_ship_delay_days DESC;

Additionally, this query is a good candidate for demonstrating the effectiveness of using Parquet Column Indexes because it has a selective filter predicate on a single column cs_item_sk. When Athena executes this query, it first identifies row groups whose min/max ranges overlap with the top 10 most ordered items. Without using Parquet Column Indexes, Athena has to scan every page of data within those matched row groups. However, when using Parquet Column Indexes, Athena can prune data further by skipping individual pages within those row groups whose min/max ranges do not overlap with the ids. Complete the following steps to establish baseline query performance when Athena does not use Parquet Column Indexes during the query.

Create a new Python cell in the notebook. Replace s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/query_results/ with the S3 URI where you want to store your Athena query results, then execute the following script. Note the runtime and bytes scanned that will be printed. The script will run the query five times with query result reuse disabled and chooses the minimum runtime and the corresponding bytes scanned among those iterations. See our numbers in the Run Athena query with Parquet Column Indexes section.

import boto3
import time

# Configuration
DATABASE = "parquet_column_index_blog"
OUTPUT_LOCATION = "s3://amzn-s3-demo-bucket/athena_parquet_column_index_blog/query_results/"

def run_athena_query(query: str, database: str, output_location: str):
    athena_client = boto3.client('athena')
    
    response = athena_client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={'Database': database},
        ResultConfiguration={'OutputLocation': output_location}
    )
    
    query_execution_id = response['QueryExecutionId']
    
    while True:
        result = athena_client.get_query_execution(QueryExecutionId=query_execution_id)
        state = result['QueryExecution']['Status']['State']
        
        if state in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
            break
        
        time.sleep(5)
    
    if state != 'SUCCEEDED':
        raise Exception(f"Query failed with state: {state}")
    
    stats = result['QueryExecution']['Statistics']
    
    return {
        'execution_time_sec': stats['EngineExecutionTimeInMillis'] / 1000,
        'data_scanned_gb': stats['DataScannedInBytes'] / (1024 ** 3)
    }


def benchmark_query(query: str, database: str, output_location: str, num_runs: int = 5):
    results = []
    
    for i in range(num_runs):
        stats = run_athena_query(query, database, output_location)
        results.append(stats)
    
    best_run = min(results, key=lambda r: r['execution_time_sec'])
    
    execution_time = round(best_run['execution_time_sec'], 1)
    data_scanned = round(best_run['data_scanned_gb'], 1)
    
    print(f"Execution time: {execution_time} sec")
    print(f"Data scanned: {data_scanned} GB")


QUERY = """
SELECT cs_item_sk,
    SUM(cs_quantity) as total_orders,
    AVG(cs_ship_date_sk - cs_sold_date_sk) as avg_ship_delay_days,
    MIN(cs_ship_date_sk - cs_sold_date_sk) as min_ship_delay,
    MAX(cs_ship_date_sk - cs_sold_date_sk) as max_ship_delay,
    SUM(
        CASE
            WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
        END
    ) as late_shipments,
    SUM(
        CASE
            WHEN cs_ship_date_sk - cs_sold_date_sk > 7 THEN 1 ELSE 0
        END
    ) * 100.0 / COUNT(*) as late_shipment_pct,
    AVG(cs_ext_ship_cost) as avg_shipping_cost
FROM parquet_column_index_blog.catalog_sales
WHERE cs_item_sk IN (
        SELECT cs_item_sk
        FROM parquet_column_index_blog.catalog_sales
        WHERE cs_item_sk IS NOT NULL
        GROUP BY cs_item_sk
        ORDER BY SUM(cs_quantity) DESC
        LIMIT 10
    )
    AND cs_ship_date_sk IS NOT NULL
    AND cs_sold_date_sk IS NOT NULL
GROUP BY cs_item_sk
ORDER BY avg_ship_delay_days DESC;
"""

# Run benchmark
benchmark_query(QUERY, DATABASE, OUTPUT_LOCATION, num_runs=5)

Sort the catalog_sales table

Before rerunning the query with Athena using Parquet Column Indexes, you need to sort the catalog_sales table by the cs_item_sk column. In the preceding query, there is a dynamic filter as a subquery on the cs_item_sk column:

cs_item_sk IN (
        SELECT cs_item_sk
        FROM parquet_column_index_blog.catalog_sales
        WHERE cs_item_sk IS NOT NULL
        GROUP BY cs_item_sk
        ORDER BY SUM(cs_quantity) DESC
        LIMIT 10
    )

When executing this query, Athena pushes down the filter predicate to the data source level, fetching only rows that match the top 10 most ordered items. To maximize page pruning with Parquet Column Indexes, rows with the same cs_item_sk values should be stored near each other in the Parquet file. Without sorting, matching values could be scattered across many pages, forcing Athena to read more data. Sorting the table by cs_item_sk clusters similar values together, enabling Athena to read fewer pages.

Let’s examine the Parquet Column Indexes in one of the Parquet files to understand how the data in the catalog_sales table is currently organized. First, download the Parquet file from the cs_sold_date_sk = 2450815 partition and install the open-source parquet-cli tool on your local machine. Replace <local-path-to-parquet-file> with the path to the downloaded Parquet file, then run the following command on your local machine:

parquet column-index <local-path-to-parquet-file>

This displays Parquet Column Indexes for all columns. For brevity, only the first 11 pages of the cs_item_sk column from the first row group are shown in the following example:

row-group 0:
column index for column cs_item_sk:
Boundary order: UNORDERED
         null_count  min  max
page-0            0    4  359989
page-1            0    2  359996
page-2            0   10  359995
page-3            0   13  359996
page-4            0   22  359989
page-5            0   25  359984
page-6            0   13  359989
page-7            0   56  359990
page-8            0   14  359984
page-9            0    7  359978
page-10           0    1  359998

Notice that nearly every page contains a wide range of values. This overlap means Athena cannot eliminate pages when filtering with Parquet Column Indexes on cs_item_sk. For example, searching for cs_item_sk = 100 requires scanning each of the 11 pages because the value 100 falls within every page’s min/max range. With this overlap, enabling Athena to use Parquet Column Indexes would provide no performance benefit. Sorting the data by cs_item_sk eliminates this overlap, creating distinct, non-overlapping ranges for each page. To make Parquet Column Indexes more effective, sort the table by completing the following step:

Create a new SQL cell in the notebook and verify the connection type is Athena (Spark). Execute the query to sort the cs_item_sk column values of the catalog_sales table in ascending order and to put all the null values in the last few Parquet pages. New Iceberg data files will be generated from this query.
```
CALL spark_catalog.system.rewrite_data_files(
table => 'parquet_column_index_blog.catalog_sales', 
strategy => 'sort', 
sort_order => 'cs_item_sk ASC NULLS LAST', 
options => map('target-file-size-bytes', '1073741824', 
'rewrite-all', 'true', 'max-concurrent-file-group-rewrites', '200'));
```

Running the parquet column-index command on the sorted data file from the cs_sold_date_sk = 2450815 partition shows that the Parquet Column Indexes are now sorted and have non-overlapping ranges. The first 11 pages of the cs_item_sk column from the first row group are shown in the following example:

row-group 0:
column index for column cs_item_sk:
Boundary order: ASCENDING
         null_count  min    max
page-0           0      1   5282
page-1           0   5282  10556
page-2           0  10556  15842
page-3           0  15842  21154
page-4           0  21154  26434
page-5           0  26434  31669
page-6           0  31669  36916
page-7           0  36916  42205
page-8           0  42205  47528
page-9           0  47528  52808
page-10          0  52808  58189

Now when searching for cs_item_sk = 100, Athena only needs to read page-0, skipping the remaining 10 pages entirely.

Run Athena query with Parquet Column Indexes

Now that the data is sorted to eliminate overlapping pages within the row groups for the cs_item_sk column, we run two experiments on the sorted data. The first measures the impact of sorting alone, and the second measures the combined effect of sorting with Parquet Column Indexes.

Create a new Python cell in the notebook. Execute the same script in the section Run Athena query without Parquet Column Indexes and take note of the query runtime and bytes scanned results. This measures the performance of querying sorted data without using Parquet Column Indexes.

Create a new Python cell in the notebook. Execute the following Python script to set the use_iceberg_parquet_column_index table property to true for the catalog_sales table in the AWS Glue Data Catalog.

import boto3

def add_iceberg_parquet_column_index(database_name: str, table_name: str):
    glue_client = boto3.client('glue')
    
    # Get current table definition
    response = glue_client.get_table(DatabaseName=database_name, Name=table_name)
    table = response['Table']
    
    # Build TableInput with only allowed fields
    table_input = {'Name': table['Name']}
    
    allowed_fields = [
        'Description', 'Owner', 'LastAccessTime', 'LastAnalyzedTime',
        'Retention', 'StorageDescriptor', 'PartitionKeys', 'ViewOriginalText',
        'ViewExpandedText', 'TableType', 'Parameters', 'TargetTable'
    ]
    
    for field in allowed_fields:
        if field in table:
            table_input[field] = table[field]
    
    # Add the property
    if 'Parameters' not in table_input:
        table_input['Parameters'] = {}
    table_input['Parameters']['use_iceberg_parquet_column_index'] = 'true'
    
    # Update the table
    glue_client.update_table(DatabaseName=database_name, TableInput=table_input)

# Usage
add_iceberg_parquet_column_index("parquet_column_index_blog", "catalog_sales")

Create a new Python cell in the notebook. Execute the same script in the section Run Athena query without Parquet Column Indexes and take note of the query runtime and bytes scanned results. This measures the performance of querying sorted data using Parquet Column Indexes.

Athena query time and bytes scanned improvement

The following table summarizes the results from each experiment. The percentage improvements for the sorted experiments are measured against the unsorted baseline.

Experiment	Runtime (sec)	Bytes Scanned (GB)
Unsorted without Parquet Column Indexes	20.6	45.2
Sorted without Parquet Column Indexes	15.4 (25.2% faster)	27.8 (38.5% fewer bytes)
Sorted with Parquet Column Indexes	10.3 (50.0% faster)	13.0 (71.2% fewer bytes)

Recommendations

To maximize Athena’s ability to use Parquet Column Indexes and achieve optimal query performance, we recommend the following.

Sort data by frequently filtered columns. This allows Athena to efficiently read Parquet Column Indexes and skip irrelevant pages, potentially reducing scan time. When data is sorted by a filter column, similar values are clustered together within pages. Because Parquet Column Indexes store min/max values for each page, Athena can quickly determine which pages contain matching values and skip the rest.
Sort data by high-cardinality columns. This creates distinct value ranges between pages, maximizing the opportunity for Athena to skip pages during query execution. High-cardinality (many distinct values) columns produce non-overlapping min/max ranges across pages, allowing Athena to more effectively filter out irrelevant pages. In contrast, low-cardinality columns such as boolean or status fields result in overlapping ranges across many pages, reducing the number of skipped pages.

Clean up

When you have finished the steps in this post, complete the following cleanup actions to avoid incurring ongoing charges:

Create a new SQL cell in the notebook and set the connection type to Athena (Spark). Execute the following command to drop the parquet_column_index_blog database and the catalog_sales table.
```
DROP DATABASE parquet_column_index_blog CASCADE;
```
Delete the Iceberg table data and the Athena query results from your S3 bucket.
Delete the SageMaker Unified Studio IAM-based domain if it is no longer needed.

Conclusion

In this post, we showed you how Athena uses Parquet Column Indexes to speed up queries and reduce the number of bytes scanned. By using Parquet Column Indexes, Athena can skip irrelevant data pages to improve query performance, especially for queries with selective filters on sorted data. Refer to Optimize Iceberg tables to learn more about this feature and try it out on your own queries.

About the Author

Implementing Kerberos authentication for Apache Spark jobs on Amazon EMR on EKS to access a Kerberos-enabled Hive Metastore

Krishna Kumar Venkateswaran — Mon, 13 Apr 2026 15:51:31 +0000

Many organizations run their Apache Spark analytics platforms on Amazon EMR on Amazon Elastic Compute Cloud (Amazon EC2), using Kerberos authentication to secure connectivity between Spark jobs and a centralized shared Apache Hive Metastore (HMS). With Amazon EMR on Amazon EKS, they gained a new option for running Spark jobs with the benefits of Kubernetes-based container orchestration, improved resource utilization, and faster job startup times. However, an HMS deployment supports only one authentication mechanism at a time. This means that they must configure Kerberos authentication for their Spark jobs on Amazon EMR on EKS to connect to the existing Kerberos-enabled HMS.

In this post, we show how to configure Kerberos authentication for Spark jobs on Amazon EMR on EKS, authenticating against a Kerberos-enabled HMS so you can run both Amazon EMR on EC2 and Amazon EMR on EKS workloads against a single, secure HMS deployment.

Overview of solution

Consider an enterprise data platform team that’s been running Spark jobs on Amazon EMR on EC2 for several years. Their architecture includes a Kerberos-enabled standalone HMS that serves as the centralized data catalog, with Microsoft Active Directory functioning as the Key Distribution Center (KDC). As the team evaluates Amazon EMR on EKS for new workloads, their existing HMS must continue serving Amazon EMR on EC2, with both authenticating through the same Kerberos infrastructure. To address this, the platform team must configure their Spark jobs running on Amazon EMR on EKS to authenticate with the same KDC. This is so they can obtain valid Kerberos tickets and establish authenticated connections to the HMS while maintaining a unified security posture across their data platform.

Scope of Kerberos in this solution

Kerberos authentication in this solution secures the connection between Spark jobs and the HMS. Other components in the architecture use AWS and Kubernetes security mechanisms instead.

Solution architecture

Our solution implements Kerberos authentication to secure the connection between Spark jobs and the HMS. The architecture spans two Amazon Virtual Private Clouds (Amazon VPCs) connected using VPC peering, with distinct components handling identity management, compute, and metadata services.

Identity and Authentication layer

A self-managed Microsoft Active Directory Domain Controller is deployed in a dedicated VPC and serves as the KDC for Kerberos authentication. The Active Directory server hosts service principals for both the HMS service and Spark jobs. This separate VPC deployment mirrors real-world enterprise architectures where Active Directory is typically managed by identity teams in their own network boundary, whether on-premises or in AWS.

Data Platform layer

The data platform components reside in a separate VPC and includes an EKS cluster that hosts both the HMS service and Amazon EMR on EKS based Spark jobs persisting data in an Amazon Simple Storage Service (Amazon S3) bucket.

Hive Metastore service

The HMS is deployed in the EKS hive-metastore namespace and simulates a pre-existing, standalone Kerberos-enabled HMS, a common enterprise pattern where HMS is managed independently of any data processing platform. You can learn more about other enterprise design patterns in the post Design patterns for implementing Hive Metastore for Amazon EMR or EKS. The HMS service authenticates with the KDC using its service principal and keytab mounted from a Kubernetes secret.

Apache Spark Execution layer

Apache Spark jobs are deployed using the Spark Operator on EKS. The Spark driver and executor pods are configured with Kerberos credentials through mounted ConfigMaps containing krb5.conf and jaas.conf, along with keytab files from Kubernetes secrets. When a Spark job must access Hive tables, the driver authenticates with the KDC and establishes a secure Simple Authentication and Security Layer (SASL) connection to the HMS.

Authentication flow

The HMS runs as a long-running Kubernetes service that must be deployed and authenticated before Spark jobs can connect.

During HMS deployment:

HMS pod validates its Kerberos configuration. krb5.conf and jaas.conf are mounted from ConfigMaps
Service authenticates with KDC using its principal hive/hive-metastore-svc.hive-metastore.svc.cluster.local@CORP.KERBEROS
keytab is mounted from Kubernetes secret for credential access
Secure Thrift endpoint is established on port 9083 with SASL authentication enabled

When a Spark job must interact with the HMS:

Spark job submission:
1. User submits Spark job through Spark Operator
2. Driver and executor pods are created with Kerberos configuration mounted as volumes
3. krb5.conf ConfigMap provides KDC connection details including realm and server addresses
4. jaas.conf ConfigMap specifies a login module configuration with keytab path and principal
5. Keytab secret contains encrypted credentials for Spark service principal spark/analytics-team@CORP.KERBEROS
Authentication and connection:
1. Spark driver authenticates with KDC using its principal and keytab to obtain a Ticket Granting Ticket (TGT)
2. When connecting to HMS, Spark requests a service ticket from the KDC for the HMS principal hive/hive-metastore-svc.hive-metastore.svc.cluster.local@CORP.KERBEROS
3. KDC issues a service ticket encrypted with HMS’s secret key
4. Spark presents this service TGT to HMS over the Thrift connection on port 9083
5. HMS decrypts the ticket using its keytab, verifies Spark’s identity, and establishes the authenticated SASL session
6. Executor pods use the same configuration for authenticated operations
Data access:
1. Authenticated Spark job queries HMS for table metadata
2. HMS validates Kerberos tickets before serving metadata requests
3. Spark accesses underlying data in Amazon S3 using IRSA

Implementation workflow

The implementation involves three key stakeholders working together to establish the Kerberos-enabled communication:

Microsoft Active Directory Administrator

The Active Directory Administrator creates service accounts that are used for HMS and Spark jobs. This involves setting up the service principal names using the setspn utility and generating keytab files using ktpass for secure credential storage. The administrator configures the appropriate Active Directory permissions and Kerberos AES256 encryption type. Finally, the keytab files are uploaded to AWS Secrets Manager for secure distribution to Kubernetes workloads.

Data Platform Team

The platform team handles the Amazon EMR on EKS and Kubernetes configurations. They retrieve keytabs from Secrets Manager and create Kubernetes secrets for the workloads. They configure Helm charts for HMS deployment with Kerberos settings and set up ConfigMaps for krb5.conf, jaas.conf, and core-site.xml.

Data Engineering Operations

Data engineers submit jobs using the configured service account with Kerberos authentication. They monitor job execution and verify authenticated access to HMS.

Deploy the solution

In the remainder of this post, you will explore the implementation details for this solution. You can find the sample code in the AWS Samples GitHub repository. For additional details, including verification steps for each deployment stage, refer to the README in the repository.

Prerequisites

Before you deploy this solution, make sure that the following prerequisites are in place:

Access to a valid AWS account and permission to create AWS resources.
The AWS Command Line Interface (AWS CLI) is installed on your local machine.
Git, Docker, eksctl, kubectl, Helm, envsubst, jq, and yq utilities are installed on your local machine.
Familiarity with Kerberos, Apache Hive Metastore (HMS), Apache Spark, Kubernetes, Amazon EKS, and Amazon EMR on Amazon EKS.

Clone the repository and set up environment variables

Clone the repository to your local machine and set the two environment variables. Replace <AWS_REGION> with the AWS Region where you want to deploy these resources.

# Clone the Git repository
git clone https://github.com/aws-samples/sample-emr-eks-spark-kerberos-hms.git
cd sample-emr-eks-spark-kerberos-hms

# Set environment variables
export REPO_DIR=$(pwd)
export AWS_REGION=<AWS_REGION>

Setup Microsoft Active Directory infrastructure

In this section, we deploy a self-managed Microsoft Active Directory with KDC on a Windows Server EC2 instance into a dedicated VPC. This is an intentionally minimal implementation highlighting only the key components required for this blog post.

cd ${REPO_DIR}/microsoft-ad
./setup.sh

Setup EKS infrastructure

This section provisions the Amazon EMR on EKS infrastructure stack, including VPC, EKS cluster, Amazon Aurora PostgreSQL database, Amazon Elastic Container Registry (Amazon ECR), Amazon S3, Amazon EMR on EKS virtual clusters and the Spark Operator. Run the following script.

cd ${REPO_DIR}/data-infra
./setup.sh

Set up VPC peering

This section establishes network connectivity between the Active Directory VPC and EKS VPC for Kerberos authentication. Run the following script:

cd ${REPO_DIR}/vpc-peering
./setup.sh

Deploy Hive Metastore with Kerberos authentication

This section deploys a Kerberos-enabled HMS service on the EKS cluster. Complete the following steps:

Create Kerberos Service Principal for HMS service

cd ${REPO_DIR}/microsoft-ad/
# Create HMS service principal
./manage-ad-service-principals.sh create hive "hive/hive-metastore-svc.hive-metastore.svc.cluster.local"
# Verify the service principal was created
./manage-ad-service-principals.sh list

Deploy HMS service with Kerberos authentication

cd ${REPO_DIR}/hive-metastore
./deploy.sh

Set up Amazon EMR on Amazon EKS with Kerberos authentication

This section configures Spark jobs to authenticate with Kerberos-enabled HMS. This involves creating service principles for Spark jobs and generating the necessary configuration files. Complete the following steps:

Create Service Principal for Spark jobs

cd ${REPO_DIR}/microsoft-ad/
# Create Spark service principal
./manage-ad-service-principals.sh create spark "spark/analytics-team"
# Verify the service principal was created
./manage-ad-service-principals.sh list

Generate Kerberos configurations for Spark jobs

cd ${REPO_DIR}/spark-jobs/
./generate-spark-configs.sh --principal "spark/analytics-team@CORP.KERBEROS" --namespace emr

Submit Spark jobs

This section verifies Kerberos authentication by running a Spark job that connects to the Kerberized HMS. Complete the following steps:

Submit the test Spark job

cd ${REPO_DIR}/spark-jobs
kubectl apply -f spark-job.yaml

Monitor job execution

# Watch the SparkApplication status
kubectl get sparkapplications -n emr -w
# Check pod status
kubectl get pods -n emr | grep "spark-kerberos"

Verify Kerberos authentication and HMS connection

# Check Spark driver logs for successful authentication
kubectl logs spark-kerberos-job-driver -n emr

The logs should confirm successful authentication, along with a listing of sample databases and tables.

Understanding Kerberos configuration

The HMS requires specific configuration parameters to enable Kerberos authentication, applied through the previously mentioned steps. The key configurations are outlined in the following section.

HMS configuration (metastore-site.xml)

The following configurations are added to metastore-site.xml file.

Setting	Value	Purpose
`hive.metastore.sasl.enabled`	true	Enable SASL authentication
`hive.metastore.kerberos.principal`	`hive/hive-metastore-svc.hive-metastore.svc.cluster.local@CORP.KERBEROS`	HMS service principal
`hive.metastore.kerberos.keytab.file`	`/etc/security/keytab/hive.keytab`	Keytab path

Hadoop security (core-site.xml)

The following configurations are added to the core-site.xml file.

Setting	Value
`hadoop.security.authentication`	kerberos
`hadoop.security.authorization`	true

Spark configuration

Setting	Value	Purpose
`spark.security.credentials.kerberos.enabled`	true	Enable Kerberos for Spark
`spark.hadoop.hive.metastore.sasl.enabled`	true	SASL for HMS connection
`spark.kerberos.principal`	`spark/analytics-team@CORP.KERBEROS`	Spark service principal
`spark.kerberos.keytab`	`local:///etc/security/keytab/analytics-team.keytab`	Keytab path

Shared Kerberos files

Both HMS and Spark pods mount two common Kerberos configuration files: krb5.conf and jaas.conf, using ConfigMaps and Kubernetes secrets. The krb5.conf file is identical across both services and defines how each component connects to the KDC. The jaas.conf file follows the same structure but differs in the principal and keytab path for each service.

krb5 Configuration

[libdefaults]
	default_realm = CORP.KERBEROS
	dns_lookup_realm = false
	dns_lookup_kdc = false
	ticket_lifetime = 24h
	forwardable = true
	udp_preference_limit = 1
	default_tkt_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
	default_tgs_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96
	permitted_enctypes = aes256-cts-hmac-sha1-96 aes128-cts-hmac-sha1-96

[realms]
	CORP.KERBEROS = {
		kdc = <ad-server-ip>
		admin_server = <ad-server-ip>
	}

[domain_realm]
	.corp.kerberos = CORP.KERBEROS
	corp.kerberos = CORP.KERBEROS

For more information, see the online documentation for krb5.conf.

JAAS configuration

Client {
 com.sun.security.auth.module.Krb5LoginModule required
 useKeyTab=true
 keyTab="/etc/security/keytab/hive.keytab"
 principal="hive/hive-metastore-svc.hive-metastore.svc.cluster.local@CORP.KERBEROS"
 useTicketCache=false
 storeKey=true
 debug=false;
};

Additional security considerations

This post focuses on core Kerberos authentication mechanics between Spark and HMS. We recommend two additional security hardening steps based on your organization’s security posture and compliance requirements.

Protecting Keytabs at Rest with AWS KMS Envelope Encryption

Keytabs stored as Kubernetes Secrets are only base64-encoded by default, not encrypted at rest. We recommend enabling EKS envelope encryption using an AWS Key Management Service (AWS KMS) customer managed key. With envelope encryption, secret data is encrypted with a Data Encryption Key (DEK), which is encrypted by your customer managed key. This protects keytab content even if the etcd datastore is compromised. To enable this on an existing EKS cluster:

aws eks associate-encryption-config \
  --cluster-name <your-cluster> \
  --encryption-config '[{"resources":["secrets"],"provider":{"keyArn":"arn:aws:kms:<region>:<account-id>:key/<key-id>"}}]'

Refer to the Amazon EKS documentation on envelope encryption for full setup guidance.

Encrypting the Thrift Data Channel with TLS

SASL with Kerberos provides mutual authentication but doesn’t automatically encrypt data over the Thrift connection. Many deployments default to auth QoP, leaving the data channel unencrypted. We recommend either:

Set SASL QoP to auth-conf — enables SASL-layer encryption using Kerberos session keys
Layer TLS over Thrift (preferred) — enables transport-level encryption using modern cipher suites

Enabling TLS on HiveServer2 / Hive Metastore Thrift:

<property>
  <name>hive.server2.use.SSL</name>
  <value>true</value>
</property>
<property>
  <name>hive.server2.keystore.path</name>
  <value>/etc/tls/keystore.jks</value>
</property>

Refer to the Hive SSL/TLS configuration documentation for full details.

Cleaning up

To avoid incurring future charges, clean up all provisioned resources during this setup by executing the following cleanup script.

cd ${REPO_DIR}/
./cleanup.sh

Conclusion

In this post, we demonstrated how to implement Kerberos authentication for Amazon EMR on EKS to securely connect to a Kerberos-enabled HMS. This solution addresses a common challenge faced by organizations with existing Kerberos-enabled HMS deployments who want to adopt Amazon EMR on EKS while maintaining their Kerberos-enabled security posture.

This pattern applies whether you’re migrating from on-premises Hadoop, running hybrid Amazon EMR on EC2 or Amazon EMR on EKS environments, or building a new cloud-native platform. Any scenario where Spark jobs on Kerberos must authenticate with a shared, Kerberos-enabled HMS.

You can use this post as a starting point to implement this pattern and extend it further to suit your organization’s data platform needs.

About the authors

Introducing Amazon MSK Express Broker power for Kiro

Stephan Schiller — Thu, 09 Apr 2026 14:40:07 +0000

Developers working with Amazon Managed Streaming for Apache Kafka (Amazon MSK) regularly need to make decisions that require deep operational context—choosing the right instance type, diagnosing consumer lag, or planning for a traffic spike. Answering these questions means piecing together documentation, metrics, and operational know-how.

What if your IDE could guide you through that workflow with built-in domain expertise and tooling? Kiro is an AI-powered agentic IDE that lets you describe what you need in natural language. Whether it’s infrastructure configuration or operational troubleshooting, Kiro guides you through the solution.

In this post, we’ll show you how to use Kiro powers, a new capability that equips Kiro with contextual knowledge and tooling. You can simplify your MSK cluster management, from initial setup to diagnosing common issues, all through natural language conversations.

Challenges operating your MSK Express broker cluster

Amazon MSK Express Brokers are a fully managed offering where AWS handles much of the underlying infrastructure. However, platform teams still need to correctly size clusters based on throughput requirements. They also need to understand the right Amazon CloudWatch metrics during performance issues and investigate when CPU usage or replication lag is higher than expected. MSK best practices documentation spans multiple AWS guides. This makes it time-consuming to find relevant information during production incidents. New team members face a learning curve with MSK operations and can repeat common sizing and configuration mistakes.

Although Express Brokers simplify infrastructure management, you still face operational challenges that require deep Kafka expertise across three areas:

Cluster creation and sizing: You must still select the right instance type, configure networking, and choose authentication methods. These decisions impact cost and performance from day one.
Observability and troubleshooting: Effective operations require correlating broker, partition, and client metrics. Troubleshooting lag or replication issues still requires a solid understanding of Express Brokers’ architecture.
Capacity management: You must monitor CPU usage, understand per-broker throughput limits, and scale before hitting throttling thresholds.

These challenges mean that setting up an MSK cluster, analyzing slow-running clients, or investigating high-CPU load requires pulling together documentation, configuration details, CLI tooling, and operational know-how, which is often spread across multiple sources. Kiro powers address these challenges by bringing best practices, guided workflows, and tooling directly into your IDE, reducing the expertise barrier and the time spent context-switching between documentation, consoles, and the CLI.

Kiro powers

Kiro powers is a feature that combines best practices, specialized context, and tool integrations into a single capability. You can install powers with one click in the Kiro IDE or add them from a public GitHub URL. Each Power combines the following components:

Model Context Protocol (MCP) servers give your Kiro agent direct access to your infrastructure. The AWS MSK MCP server, for example, exposes tools to create clusters, monitor health, and optimize configurations.
Steering files provide persistent knowledge and workflow guides that Kiro loads based on the user’s task, such as monitoring best practices or troubleshooting workflows.
Optional hooks run automated actions when IDE events occur, such as validating configurations before deployment.

The key advantage of Kiro powers is that they load context dynamically based on the user’s task. Instead of configuring every MCP server upfront and re-providing context in each conversation, powers activate the right tools and knowledge on demand. This keeps your agent’s context focused and relevant. In the next section, we look at how these components work together specifically for MSK Express Broker operations.

The MSK Express broker power

The MSK Express broker power packages the AWS MSK MCP server with targeted streaming operations guidance, giving your Kiro agent expertise for MSK Express Broker operations and cluster management. You can use it to build Kafka-based streaming applications through Kiro while maintaining Express broker best practices throughout the development lifecycle.

For cluster operations, you can create Express broker clusters, monitor health metrics, and manage configurations through natural language. You can retrieve cluster metadata, check broker endpoints, and verify replication status. The Power also supports operational monitoring. You can track CPU utilization, throughput limits, partition distribution, and AWS Identity and Access Management (IAM) connection metrics.

To see how this works in practice, here’s what happens when you interact with the Power: When you ask Kiro to create an MSK cluster, the Power recommends appropriate instance sizes based on your throughput requirements. When you’re troubleshooting, it knows to check LeaderCount before diving into network metrics. When you’re troubleshooting authentication failures, it recommends client settings like reconnect.backoff.ms and group.instance.id to resolve connection churn and rebalancing issues against Express broker limits. Use cases include:

Cluster sizing and creation: Describe your throughput requirements (for example, “50 MBps ingress with 3x fan-out”) and the Power calculates the right instance type and broker count, then walks through cluster creation.
Proactive health monitoring: Ask Kiro to review your cluster. It checks CPU against the 60% threshold, compares throughput to instance limits, and flags partition imbalances and throughput bottlenecks before they become incidents.
Incident troubleshooting: Consumer lag spiking? The Power checks the relevant metrics, identifies the root cause (like skewed partition leadership), and guides you through resolution.
Capacity planning: Preparing for a traffic spike? The Power analyzes current utilization against instance limits and recommends whether to scale up or add brokers.

The MSK Express broker power brings together documentation, metrics, and operational context so your Kiro agent can correlate findings and help identify root causes specific to your infrastructure.

Getting started with the MSK Express broker power

Starting with Kiro powers takes only a few clicks in the Kiro IDE. You can install from the built-in marketplace or import from a public GitHub URL. Kiro packages all components and makes them available to the Kiro agent.

To set up the MSK Express broker power, follow these steps:

Choose the Powers icon in the Kiro sidebar
In the AVAILABLE panel, scroll down to Build and Operate MSK Express Broker
Choose Install
The power now appears in the INSTALLED panel.

You can also visit the Kiro powers marketplace to explore other powers.

Conclusion

The MSK Express broker power streamlines Kafka operations by combining Model Context Protocol (MCP) servers with operational guidance. With natural language interactions, you can create clusters, monitor health, optimize configurations, and troubleshoot issues without reviewing extensive documentation.

Learn more about Kiro and available Kiro powers.

About the authors

Introducing workload simulation workbench for Amazon MSK Express broker

Manu Mishra — Tue, 07 Apr 2026 16:49:49 +0000

Validating Kafka configurations before production deployment can be challenging. In this post, we introduce the workload simulation workbench for Amazon Managed Streaming for Apache Kafka (Amazon MSK) Express Broker. The simulation workbench is a tool that you can use to safely validate your streaming configurations through realistic testing scenarios.

Solution overview

Varying message sizes, partition strategies, throughput requirements, and scaling patterns make it challenging for you to predict how your Apache Kafka configurations will perform in production. The traditional approaches to test these variables create significant barriers: ad-hoc testing lacks consistency, manual set up of temporary clusters is time-consuming and error-prone, production-like environments require dedicated infrastructure teams, and team training often happens in isolation without realistic scenarios. You need a structured way to test and validate these configurations safely before deployment. The workload simulation workbench for MSK Express Broker addresses these challenges by providing a configurable, infrastructure as code (IaC) solution using AWS Cloud Development Kit (AWS CDK) deployments for realistic Apache Kafka testing. The workbench supports configurable workload scenarios, and real-time performance insights.

Express brokers for MSK Provisioned make managing Apache Kafka more streamlined, more cost-effective to run at scale, and more elastic with the low latency that you expect. Each broker node can provide up to 3x more throughput per broker, scale up to 20x faster, and recover 90% quicker compared to standard Apache Kafka brokers. The workload simulation workbench for Amazon MSK Express broker facilitates systematic experimentation with consistent, repeatable results. You can use the workbench for multiple use cases like production capacity planning, progressive training to prepare developers for Apache Kafka operations with increasing complexity, and architecture validation to prove streaming designs and compare different approaches before making production commitments.

Architecture overview

The workbench creates an isolated Apache Kafka testing environment in your AWS account. It deploys a private subnet where consumer and producer applications run as containers, connects to a private MSK Express broker and monitors for performance metrics and visibility. This architecture mirrors the production deployment pattern for experimentation. The following image describes this architecture using AWS services.

This architecture is deployed using the following AWS services:

Amazon Elastic Container Service (Amazon ECS) generate configurable workloads with Java-based producers and consumers, simulating various real-world scenarios through different message sizes and throughput patterns.

Amazon MSK Express Cluster runs Apache Kafka 3.9.0 on Graviton-based instances with hands-free storage management and enhanced performance characteristics.

Dynamic Amazon CloudWatch Dashboards automatically adapt to your configuration, displaying real-time throughput, latency, and resource utilization across different test scenarios.

Secure Amazon Virtual Private Cloud (Amazon VPC) Infrastructure provides private subnets across three Availability Zones with VPC endpoints for secure service communication.

Configuration-driven testing

The workbench provides different configuration options for your Apache Kafka testing environment, so you can customize instance types, broker count, topic distribution, message characteristics, and ingress rate. You can adjust the number of topics, partitions per topic, sender and receiver service instances, and message sizes to match your testing needs. These flexible configurations support two distinct testing approaches to validate different aspects of your Kafka deployment:

Approach 1: Workload validation (single deployment)

Test different workload patterns against the same MSK Express cluster configuration. This is useful for comparing partition strategies, message sizes, and load patterns.

// Fixed MSK Express Cluster Configuration
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // 1 broker per AZ = 3 total brokers
instanceType: 'express.m7g.large', // MSK Express instance type
};

// Multiple Concurrent Workload Tests
export const deploymentConfig: DeploymentConfig = { services: [
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, // High-throughput scenario
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 512 }, // Latency-optimized scenario
{ topics: 3, partitionsPerTopic: 4, instances: 2, messageSizeBytes: 4096 }, // Multi-topic scenario
]};

Approach 2: Infrastructure rightsizing (redeploy and compare)

Test different MSK Express cluster configurations by redeploying the workbench with different broker settings while keeping the same workload. This is recommended for rightsizing experiments and understanding the impact of vertical compared to horizontal scaling.

// Baseline: Deploy and test
export const mskBrokerConfig: MskBrokerConfig = { numberOfBrokers: 1, instanceType: 'express.m7g.large',};

// Vertical scaling: Redeploy with larger instances
export const mskBrokerConfig: MskBrokerConfig = { numberOfBrokers: 1,
instanceType: 'express.m7g.xlarge', // Larger instances
};

// Horizontal scaling: Redeploy with more brokers
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 2, // More brokers
instanceType: 'express.m7g.large',};

Each redeployment uses the same workload configuration, so you can isolate the impact of infrastructure changes on performance.

Workload testing scenarios (single deployment)

These scenarios test different workload patterns against the same MSK Express cluster:

Partition strategy impact testing

Scenario: You are debating the usage of fewer topics with many partitions compared to many topics with fewer partitions for your microservices architecture. You want to understand how partition count affects throughput and consumer group coordination before making this architectural decision.

const deploymentConfig = { services: [
{ topics: 1, partitionsPerTopic: 1, instances: 2, messageSizeBytes: 1024 }, // Baseline: minimal partitions
{ topics: 1, partitionsPerTopic: 10, instances: 2, messageSizeBytes: 1024 }, // Medium partitions
{ topics: 1, partitionsPerTopic: 20, instances: 2, messageSizeBytes: 1024 }, // High partitions
]};

Message size performance analysis

Scenario: Your application handles different types of events – small IoT sensor readings (256 bytes), medium user activity events (1 KB), and large document processing events (8KB). You must understand how message size impacts your overall system performance and if you should separate these into different topics or handle them together.

const deploymentConfig = { services: [
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 256 }, // IoT sensor data
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, // User events
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 8192 }, // Document events
]};

Load testing and scaling validation

Scenario: You expect traffic to vary significantly throughout the day, with peak loads requiring 10× more processing capacity than off-peak hours. You want to validate how your Apache Kafka topics and partitions handle different load levels and understand the performance characteristics before production deployment.

const deploymentConfig = { services: [
{ topics: 2, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, // Off-peak load simulation
{ topics: 2, partitionsPerTopic: 6, instances: 5, messageSizeBytes: 1024 }, // Medium load simulation
{ topics: 2, partitionsPerTopic: 6, instances: 10, messageSizeBytes: 1024 }, // Peak load simulation
]};

Infrastructure rightsizing experiments (redeploy and compare)

These scenarios help you understand the impact of different MSK Express cluster configurations by redeploying the workbench with different broker settings:

MSK broker rightsizing analysis

Scenario: You deploy a cluster with basic configuration and put load on it to establish baseline performance. Then you want to experiment with different broker configurations to see the effect of vertical scaling (larger instances) and horizontal scaling (more brokers) to find the right cost-performance balance for your production deployment.

Step 1: Deploy with baseline configuration

// Initial deployment: Basic configuration
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // 3 total brokers (1 per AZ)
instanceType: 'express.m7g.large',};export const deploymentConfig: DeploymentConfig = { services: [ { topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, ]};

Step 2: Redeploy with vertical scaling

// Redeploy: Test vertical scaling impact
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // Same broker count
instanceType: 'express.m7g.xlarge', // Larger instances
};

// Keep same workload configuration to compare results

Step 3: Redeploy with horizontal scaling

// Redeploy: Test horizontal scaling impact
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 2, // 6 total brokers (2 per AZ)
instanceType: 'express.m7g.large', // Back to original size
};

// Keep same workload configuration to compare results

This rightsizing approach helps you understand how broker configuration changes affect the same workload, so you can improve both performance and cost for your specific requirements.

Performance insights

The workbench provides detailed insights into your Apache Kafka configurations through monitoring and analytics, creating a CloudWatch dashboard that adapts to your configuration. The dashboard starts with a configuration summary showing your MSK Express cluster details and workbench service configurations, helping you to understand what you’re testing. The following image shows the dashboard configuration summary:

The second section of dashboard shows real-time MSK Express cluster metrics including:

Broker performance: CPU utilization and memory usage across brokers in your cluster
Network activity: Monitor bytes in/out and packet counts per broker to understand network utilization patterns
Connection monitoring: Displays active connections and connection patterns to help identify potential bottlenecks
Resource utilization: Broker-level resource tracking provides insights into overall cluster health

The following image shows the MSK cluster monitoring dashboard:

The third section of the dashboard shows the Intelligent Rebalancing and Cluster Capacity insights showing:

Intelligent rebalancing: in progress: Shows whether a rebalancing operation is currently in progress or has occurred in the past. A value of 1 indicates that rebalancing is actively running, while 0 means that the cluster is in a steady state.
Cluster under-provisioned: Indicates whether the cluster has insufficient broker capacity to perform partition rebalancing. A value of 1 means that the cluster is under-provisioned and Intelligent Rebalancing can’t redistribute partitions until more brokers are added or the instance type is upgraded.
Global partition count: Displays the total number of unique partitions across all topics in the cluster, excluding replicas. Use this to track partition growth over time and validate your deployment configuration.
Leader count per broker: Shows the number of leader partitions assigned to each broker. An uneven distribution indicates partition leadership skew, which can lead to hotspots where certain brokers handle disproportionate read/write traffic.
Partition count per broker: Shows the total number of partition replicas hosted on each broker. This metric includes both leader and follower replicas and is key to identifying replica distribution imbalances across the cluster.

The following image shows the Intelligent Rebalancing and Cluster Capacity section of the dashboard:

The fourth section of the dashboard shows the application-level insights showing:

System throughput: Displays the total number of messages per second across services, giving you a complete view of system performance
Service comparisons: Performs side-by-side performance analysis of different configurations to understand which approaches fit
Individual service performance: Each configured service has dedicated throughput tracking widgets for detailed analysis
Latency analysis: The end-to-end message delivery times and latency comparisons across different service configurations
Message size impact: Performance analysis across different payload sizes helps you understand how message size affects overall system behavior

The following image shows the application performance metrics section of the dashboard:

Getting started

This section walks you through setting up and deploying the workbench in your AWS environment. You will configure the necessary prerequisites, deploy the infrastructure using AWS CDK, and customize your first test.

Prerequisites

You can deploy the solution from the GitHub Repo. You can clone it and run it on your AWS environment. To deploy the artifacts, you will require:

AWS account with administrative credentials configured for creating AWS resources.
AWS Command Line Interface (AWS CLI) must be configured with appropriate permissions for AWS resource management.
AWS Cloud Development Kit (AWS CDK) should be installed globally using npm install -g aws-cdk for infrastructure deployment.
Node.js version 20.9 or higher is required, with version 22+ recommended.
Docker engine must be installed and running locally as the CDK builds container images during deployment. Docker daemon should be running and accessible to CDK for building the workbench application containers.

Deployment

# Clone the workbench repository
git clone https://github.com/aws-samples/sample-simulation-workbench-for-msk-express-brokers.git

# Install dependencies and build
npm install 
npm run build

# Bootstrap CDK (first time only per account/region)
cd cdk 
npx cdk bootstrap

# Synthesize CloudFormation template (optional verification step)
npx cdk synth

# Deploy to AWS (creates infrastructure and builds containers)
npx cdk deploy

After deployment is completed, you will receive a CloudWatch dashboard URL to monitor the workbench performance in real-time.You can also deploy multiple isolated instances of the workbench in the same AWS account for different teams, environments, or testing scenarios. Each instance operates independently with its own MSK cluster, ECS services, and CloudWatch dashboards.To deploy additional instances, modify the Environment Configuration in cdk/lib/config.ts:

// Instance 1: Development team
export const AppPrefix = 'mske';export const EnvPrefix = 'dev';

// Instance 2: Staging environment (separate deployment)
export const AppPrefix = 'mske';export const EnvPrefix = 'staging';

// Instance 3: Team-specific testing (separate deployment)
export const AppPrefix = 'team-alpha';export const EnvPrefix = 'test';

Each combination of AppPrefix and EnvPrefix creates completely isolated AWS resources so that multiple teams or environments can use the workbench simultaneously without conflicts.

Customizing your first test

You can edit the configuration file located at folder “cdk/lib/config-types.ts” to define your testing scenarios and run the deployment. It is preconfigured with the following configuration:

export const deploymentConfig: DeploymentConfig = { services: [
// Start with a simple baseline test
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 },

// Add a comparison scenario
{ topics: 1, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, ]};

Best practices

Following a structured approach to benchmarking ensures that your results are reliable and actionable. These best practices will help you isolate performance variables and build a clear understanding of how each configuration change affects your system’s behavior. Begin with single-service configurations to establish baseline performance:

const deploymentConfig = { services: [ { topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 } ]};

After you understand the baseline, add comparison scenarios.

Change one variable at a time

For clear insights, modify only one parameter between services:

const deploymentConfig = { services: [
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 }, // Baseline
{ topics: 1, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, // More partitions
{ topics: 1, partitionsPerTopic: 12, instances: 1, messageSizeBytes: 1024 }, // Even more partitions
]};

This approach helps you understand the impact of specific configuration changes.

Important considerations and limitations

Before relying on workbench results for production decisions, it is important to understand the tool’s intended scope and boundaries. The following considerations will help you set appropriate expectations and make the most effective use of the workbench in your planning process.

Performance testing disclaimer

The workbench is designed as an educational and sizing estimation tool to help teams prepare for MSK Express production deployments. While it provides valuable insights into performance characteristics:

Results can vary based on your specific use cases, network conditions, and configurations
Use workbench results as guidance for initial sizing and planning
Conduct comprehensive performance validation with your actual workloads in production-like environments before final deployment

Recommended usage approach

Production readiness training – Use the workbench to prepare teams for MSK Express capabilities and operations.

Architecture validation – Test streaming architectures and performance expectations using MSK Express enhanced performance characteristics.

Capacity planning – Use MSK Express streamlined sizing approach (throughput-based rather than storage-based) for initial estimates.

Team preparation – Build confidence and expertise with production Apache Kafka implementations using MSK Express.

Conclusion

In this post, we showed how the workload simulation workbench for Amazon MSK Express Broker supports learning and preparation for production deployments through configurable, hands-on testing and experiments. You can use the workbench to validate configurations, build expertise, and improve performance before production deployment. If you’re preparing for your first Apache Kafka deployment, training a team, or improving existing architectures, the workbench provides practical experience and insights needed for success. Refer to Amazon MSK documentation – Complete MSK Express documentation, best practices, and sizing guidance for more information.

About the authors

Proactive monitoring for Amazon Redshift Serverless using AWS Lambda and Slack alerts

Cristian Restrepo Lopez — Tue, 07 Apr 2026 16:27:14 +0000

Performance issues in analytics environments often remain invisible until they disrupt dashboards, delay ETL jobs, or impact business decisions. For teams running Amazon Redshift Serverless, unmonitored query queues, long-running queries, or unexpected spikes in compute capacity can degrade performance and increase costs if left undetected.

Amazon Redshift Serverless streamlines running analytics at scale by removing the need to provision or manage infrastructure. However, even in a serverless environment, maintaining visibility into performance and usage is essential for efficient operation and predictable costs. While Amazon Redshift Serverless provides advanced built-in dashboards for monitoring performance metrics, delivering notifications directly to platforms like Slack, brings another level of agility. Real-time alerts in the team’s workflow enable faster response times and more informed decision-making without requiring constant dashboard monitoring.

In this post, we show you how to build a serverless, low-cost monitoring solution for Amazon Redshift Serverless that proactively detects performance anomalies and sends actionable alerts directly to your selected Slack channels. This approach helps your analytics team identify and address issues early, often before your users notice a problem.

Solution overview

The solution presented in this post uses AWS services to collect key performance metrics from Amazon Redshift Serverless, evaluate them against thresholds that you can flexibly configure, and notify you when anomalies are detected.

The workflow operates as follows:

Scheduled execution – An Amazon EventBridge rule triggers an AWS Lambda function on a configurable schedule (by default, every 15 minutes during business hours).
Metric collection – The AWS Lambda function gathers metrics including queued queries, running queries, compute capacity (RPUs), data storage usage, table count, database connections, and slow-running queries using Amazon CloudWatch and the Amazon Redshift Data API.
Threshold evaluation – Collected metrics are compared against your predefined thresholds that reflect acceptable performance and usage limits.
Alerting – When a threshold is exceeded, the Lambda function publishes a notification to an Amazon SNS topic.
Slack notification – Amazon Q Developer in Chat applications (formerly AWS Chatbot) delivers the alert to your designated Slack channel.
Observability – Lambda execution logs are stored in Amazon CloudWatch Logs for troubleshooting and auditing.

This architecture is fully serverless and requires no changes to your existing Amazon Redshift Serverless workloads. To simplify deployment, we provide an AWS CloudFormation template that provisions all required resources.

Prerequisites

Before deploying this solution, you must collect information about your existing Amazon Redshift Serverless workgroup and namespace that you want to monitor. To identify your Amazon Redshift Serverless resources:

Open the Amazon Redshift console.
In the navigation pane, choose Serverless dashboard.
Note down your workgroup and namespace names. You will use these values when launching this blog’s AWS CloudFormation template.

Deploy the solution

You can launch the CloudFormation stack and deploy the solution via the provided link.

GitHub Repo

When launching the CloudFormation stack, complete the following steps in the AWS CloudFormation Console:

For Stack name, enter a descriptive name such as redshift-serverless-monitoring.
Review and modify the parameters as needed for your environment.
Acknowledge that AWS CloudFormation may create IAM resources with custom names.
Choose Submit.

CloudFormation parameters

Amazon Redshift Serverless Workgroup configuration

Provide details for your existing Amazon Redshift Serverless environment. These values connect the monitoring solution to your Redshift environment. Some parameters come with the default values that you can replace with your actual configuration.

Parameter	Default value	Description
Amazon Redshift Workgroup Name		Your Amazon Redshift Serverless workgroup name.
Amazon Redshift Namespace Name		Your Amazon Redshift Serverless namespace name.
Amazon Redshift Workgroup ID		Workgroup ID (UUID) of the Amazon Redshift Serverless workgroup to monitor. Must follow the UUID format: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` (lowercase hexadecimal with hyphens).
		Namespace ID (UUID) of the Amazon Redshift Serverless namespace. Must follow the UUID format: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` (lowercase hexadecimal with hyphens).
Database Name	`dev`	Target Amazon Redshift database for SQL-based diagnostic and monitoring queries.

Monitoring schedule

The default schedule runs diagnostic SQL queries every 15 minutes during business hours, balancing responsiveness and cost efficiency. Running more frequently might increase costs, while less frequent monitoring could delay detection of performance issues. You can adjust this schedule to your actual need.

Parameter	Default value	Description
Schedule Expression	cron(0/15 8-17 ? * MON-FRI *)	EventBridge schedule expression for Lambda function execution. Default runs every 15 minutes, Monday through Friday, 8 AM to 5 PM UTC.

Threshold configuration

Thresholds should be tuned based on your workload characteristics.

Parameter	Default value	Description
Queries Queued Threshold	20	Alerts threshold for queued queries.
Queries Running Threshold	20	Alerts threshold for running queries.
Compute Capacity Threshold (RPUs)	64	Alert threshold for compute capacity (RPUs).
Data Storage Threshold (MB)	5242880	Threshold for data storage in MB (default 5 TB).
Table Count Threshold (MB)	1000	Alerts threshold for total table count.
Database Connections Threshold	50	Alert threshold for database connections.
Slow Query Threshold (seconds)	10	Thresholds in seconds for slow query detection.
Query Timeout (Seconds)	30	Timeout for SQL diagnostics queries.

Tip: Start with conservative thresholds and refine them after observing baseline behavior for one to two weeks.

Lambda configuration

Configure the AWS Lambda function settings. The selected default values are appropriate for most monitoring scenarios. You may want to change them only in case of troubleshooting.

Parameter	Default value	Description
Lambda Memory Size (MB)	256	Lambda function memory size in MB.
Lambda Time Out (Seconds)	240	Lambda function timeout in seconds.

Security Configuration – Amazon Virtual Private Cloud (VPC)

If your organization has network isolation requirements, you can optionally enable VPC deployment for the Lambda function. When enabled, the Lambda function runs within your specified VPC subnets, providing network isolation and allowing access to VPC-only resources.

Parameter	Default value	Description
VPC ID		VPC ID for Lambda deployment (required if `EnableVPC` is true). The Lambda function will be deployed in this VPC. Ensure that the VPC has appropriate routing (NAT Gateway or VPC Endpoints) to allow Lambda to access AWS services like CloudWatch, Amazon Redshift, and Amazon SNS.
VPC Subnet IDs		Comma-separated list of subnet IDs for Lambda deployment (required if `EnableVPC` is true).
Security Group IDs		Comma-separated list of security group IDs for Lambda (optional). If not provided and `EnableVPC` is true, a default security group will be created with outbound HTTPS access. Custom security groups must allow outbound HTTPS (port 443) to AWS service endpoints.

Note that VPC deployment might increase cold start times and requires an NAT Gateway or VPC endpoints for AWS service access. We recommend provisioning interface VPC endpoints (through AWS PrivateLink) for the five services the Lambda function calls which keeps all traffic private without the recurring cost of a NAT Gateway.

Security configuration – Encryption

If your organization requires encryption of data at rest, you can optionally enable AWS Key Management Service (AWS KMS) encryption for the Lambda function’s environment variables, CloudWatch Logs, and SNS topic. When enabled, the template encrypts each resource using the AWS KMS keys that you provide, either a single shared key for all three services, or individual keys for granular key management and audit separation.

Parameter	Default value	Description
Shared KMS Key ARN		AWS KMS key ARN to use for all encryption (Lambda, Logs, and SNS) unless service-specific keys are provided. This streamlines key management by using a single key for all services. The key policy must grant encrypt/decrypt permissions to Lambda, CloudWatch Logs, and SNS.
Lambda KMS Key ARN		AWS KMS key ARN for Lambda environment variable encryption (optional, overrides `SharedKMSKeyArn`). Use this for separate key management per service. The key policy must grant decrypt permissions to the Lambda execution role. If not provided, `SharedKMSKeyArn` will be used when `EnableKMSEncryption` is true.
CloudWatch Logs KMS Key ARN		AWS KMS key ARN for CloudWatch Logs encryption (optional, overrides `SharedKMSKeyArn`). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to the CloudWatch Logs service. If not provided, `SharedKMSKeyArn` will be used when `EnableKMSEncryption` is true.
SNS Topic KMS Key ARN		AWS KMS key ARN for SNS topic encryption (optional, overrides `SharedKMSKeyArn`). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to SNS service and the Lambda execution role. If not provided, `SharedKMSKeyArn` will be used when `EnableKMSEncryption` is true.
Enable Dead Letter Queue	False	Optionally enable Dead Letter Queue (DLQ) for failed Lambda invocations to improve reliability and security monitoring. When enabled, events that fail after all retry attempts will be sent to an SQS queue for investigation and potential replay. This helps prevent data loss, provides visibility into failures, and enables security audit trails for monitoring anomalies. The DLQ retains messages for 14 days.

Note that AWS KMS encryption requires the key policy to grant appropriate permissions to each consuming service (Lambda, CloudWatch Logs, and SNS).

On the review page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names.
Choose Submit.

Resources created

The CloudFormation stack creates the following resources:

EventBridge rule for scheduled execution
AWS Lambda function (Python 3.12 runtime)
Amazon SNS topic for alerts
IAM role with permissions for CloudWatch, Amazon Redshift Data API, and SNS
CloudWatch Log Group for Lambda logs

Note: CloudFormation deployment typically takes 10–15 minutes to complete. You can monitor progress in real time under the Events tab of your CloudFormation stack.

Post-deployment configuration

After the CloudFormation stack has been successfully created, complete the following steps.

Step 1: Record CloudFormation outputs

Navigate to the AWS CloudFormation console.
Select your stack and choose the Outputs tab.
Note the values for LambdaRoleArn and SNSTopicArn. You will need these in subsequent steps.

Step 2: Grant Amazon Redshift permissions

Grant permissions to the Lambda function to query Amazon Redshift system tables for monitoring data. Complete the following steps to grant the necessary access:

Navigate to the Amazon Redshift console.
In the left navigation pane, choose Query Editor V2.
Connect to your Amazon Redshift Serverless workgroup.
Execute the following SQL commands, replacing <IAM Role ARN> with the LambdaRoleArn value from your CloudFormation outputs:

CREATE USER "IAMR:<IAM Lambda Role>" WITH PASSWORD DISABLE;

GRANT ROLE "sys:monitor" TO "IAMR:<IAM Role>";

These commands create an AmazonRedshift user associated with the Lambda IAM role and grant it the sys:monitor Amazon Redshift role. This role provides read-only access to catalog and system tables without granting permissions to user data tables.

Step 3: Configure Slack notifications

Amazon Q Developer in chat applications provides native AWS integration and managed authentication, removing custom webhook code and reducing setup complexity. To receive alerts in Slack, configure Amazon Q Developer in Chat Applications to connect your SNS topic to your preferred Slack channel:

Navigate to Amazon Q Developer in chat applications (formerly AWS Chatbot) in the AWS console.
Follow the instructions in the Slack integration documentation to authorize AWS access to your Slack workspace.
When configuring the Slack channel, ensure that you select the correct AWS Region where you deployed the CloudFormation stack.
In the Notifications section, select the SNS topic created by your CloudFormation stack (refer to the SNSTopicArn output value).
Keep the default IAM read-only permissions for the channel configuration.

After configured, alerts automatically appear in Slack whenever thresholds are exceeded.

Cost considerations

With the default configuration, this solution incurs minimal ongoing costs. The Lambda function executes approximately 693 times per month (every 15 minutes during an 8-hour business day, Monday through Friday), resulting in a monthly cost of approximately $0.33 USD. This includes Lambda compute costs ($0.26) and CloudWatch GetMetricData API calls ($0.07). All other services (EventBridge, SNS, CloudWatch Logs, and Amazon Redshift Data API). The Amazon Redshift Data API has no additional charges beyond the minimal Amazon Redshift Serverless RPU consumption for the Amazon Redshift Serverless system table query execution. You can reduce costs by decreasing the monitoring frequency (such as, every 30 minutes) or increase responsiveness by running more frequently (such as, every 5 minutes) with a proportional cost increase.

All costs are estimates and may vary based on your environment. Variations often occur because queries scanning system tables may take longer or require additional resources depending on the system complexity

Security best practices

This solution implements the following security controls:

IAM policies scoped to specific resource ARNs for the Amazon Redshift workgroup, namespace, SNS topic, and log group.
Data API statement access restricted to the Lambda function’s own IAM user ID.
Read-only sys:monitor database role for operational metadata access. Limit to the role created by the CloudFormation template.
Reserved concurrent executions capped at five.

To further strengthen your security posture, consider the following enhancements:

Enable EnableKMSEncryption to encrypt environment variables, logs, and SNS messages at rest.
Enable EnableVPC to deploy the function within a VPC for network isolation.
Audit access through AWS CloudTrail.

Important: This is sample code for non-production usage. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment. This solution demonstrates monitoring capabilities but requires additional security hardening for production environments, including encryption configuration, IAM policy scoping, VPC deployment, and comprehensive testing.

Clean up

To remove all resources and avoid ongoing charges if you don’t want to use the solution anymore:

Delete the CloudFormation stack.
Remove the Slack integration from Amazon Q Developer in chat applications.

Troubleshooting

If no metrics or incomplete SQL diagnostics are returned, verify that the Amazon Redshift Serverless workgroup is active with recent query activity, and ensure the database user has the sys:monitor role (GRANT ROLE sys:monitor TO <user>) in the query editor. Without this role, queries execute successfully but only return data visible to that user’s permissions rather than the full cluster activity.
For VPC-deployed functions that fail to reach AWS services, confirm that VPC endpoints or a NAT Gateway are configured for CloudWatch, Amazon Redshift Data API, Amazon Redshift Serverless, SNS, and CloudWatch Logs.
If the Lambda function times out, increase the LambdaTimeout and QueryTimeoutSeconds parameters. The default timeout of 240 seconds accommodates most workloads, but clusters with many active queries may require additional time for SQL diagnostics to complete.

Conclusion

In this post, we showed how you can build a proactive monitoring solution for Amazon Redshift Serverless using AWS Lambda, Amazon CloudWatch, and Amazon SNS with Slack integration. By automatically collecting metrics, evaluating thresholds, and delivering alerts in near real time to Slack or your preferred collaborative platform, this solution helps detect performance and cost issues early. Because the solution itself is serverless, it aligns with the operational simplicity goals of Amazon Redshift Serverless—scaling automatically, requiring minimal maintenance, and delivering high value at low cost. You can extend this foundation with additional metrics, diagnostic logic, or alternative notification channels to meet your organization’s needs.

To learn more, see the Amazon Redshift documentation on monitoring and performance optimization.

About the authors

Modernize business intelligence workloads using Amazon Quick

Satesh Sonti — Mon, 06 Apr 2026 17:56:35 +0000

Traditional business intelligence (BI) integration with enterprise data warehouses has been the established pattern for years. With generative AI, you can now modernize BI workloads with capabilities like interactive chat agents, automated business processes, and using natural language to generate dashboards.

In this post, we provide implementation guidance for building integrated analytics solutions that combine the generative BI features of Amazon Quick with Amazon Redshift and Amazon Athena SQL analytics capabilities. Use this post as a reference for proof-of-concept implementations, production deployment planning, or as a learning resource for understanding Quick integration patterns with Amazon Redshift and Athena.

Common use cases

You can use this integrated approach across several scenarios. The following are some of the most common use cases.

Traditional BI reporting benefits from bundled data warehouse and BI tool pricing, making generative BI the primary use case with significant cost advantages.
- Insurance: Automates Solvency II and IFRS 17 regulatory reporting, replacing manual spreadsheet consolidation.
- Banking: Accelerates FDIC call report generation and capital adequacy dashboards, cutting month-end close from days to hours.
Interactive dashboards with contextual chat agents give BI teams conversational interfaces alongside their visual metrics.
- Gaming: Live ops teams query player retention and monetization KPIs in plain English—no SQL needed.
- Financial Services: Trading analysts chat with real-time P&L dashboards to surface anomalies and drill into positions on demand.
Domain-specific analytics workspaces democratize enterprise data exploration through Quick Spaces and natural language queries.
- Insurance: Actuarial and underwriting teams query claims and risk data without waiting on data engineering.
- Banking: Risk and compliance teams explore credit, market, and operational data through a single natural language interface.
Workflow automation removes repetitive tasks and accelerates self-service analytics.
- Financial Services: Automated AR reconciliation flows replace manual ledger matching, shrinking close cycle effort significantly.
- Gaming: Telemetry ingestion pipelines trigger reporting refreshes automatically, freeing data engineers from routine work.

Let us examine an end-to-end solution combining these technologies.

Solution flow

AWS offers two native SQL analytics engines for building analytics workloads. Amazon Redshift provides a fully managed data warehouse with columnar storage and massively parallel processing. Amazon Athena delivers serverless interactive query capabilities directly against data in Amazon S3.

You can use either Amazon Redshift or Amazon Athena as a SQL engine while implementing the steps in this post. The following are the steps involved in building an end-to-end solution.

Figure1: Solution steps to integrate SQL Analytics engines with Amazon Quick

Set up your SQL analytics engines: Amazon Redshift or Amazon Athena.
Load data and create business views designed for analytics workloads.
Configure integration between SQL analytics engines and Amazon Quick.
Create data sources in Amazon Quick.
Create datasets and dashboards for visual analytics.
Use Topics and Spaces to provide natural language interfaces to your data.
Deploy chat agents to deliver conversational AI experiences for business users.
Implement business flows to automate repetitive workflows and processes.

Let’s start by walking through steps 1–4 for Amazon Redshift. We then describe the same four steps for Amazon Athena before explaining the Amazon Quick steps 5–8.

Configure and create datasets in Amazon Redshift

Amazon Redshift offers two deployment options to meet your data warehousing needs. Provisioned clusters provide traditional deployment where you manage compute resources by selecting node types and cluster size. Serverless automatically scales compute capacity based on workload demands with pay-per-use pricing. Both options are supported by Amazon Quick. For this walkthrough, we use Redshift Serverless.

Set up SQL analytics engine

To create a Redshift Serverless namespace and workgroup:

Open the Amazon Redshift console.
On the left navigation pane, select Redshift Serverless.
Follow the steps described in the Creating a workgroup with a namespace documentation page to create a workgroup and a namespace. Note the username and password provided. You will use these details for configuring connections in Amazon Redshift and Quick.
You should see the status as Available for both the workgroup and namespace in the Serverless dashboard.

Figure 2: Amazon Redshift Serverless Workgroup and NamespacesThe deployment will be completed in approximately 3–5 minutes.

Load data and create business views

Now you can load data using the industry-standard TPC-H benchmark dataset, which provides realistic customer, order, and product data for analytics workloads.To load data into Amazon Redshift:

Open the Amazon Redshift Query Editor V2 from the console.
Run the TPC H DDL statements to create TPC-H tables.
Run the following COPY commands to load data from the public S3 bucket: s3://redshift-downloads/TPC-H/.

Ensure that the IAM role attached to the namespace is set as the default IAM role. If you didn’t set up the default IAM role at the time of namespace creation, you can refer to the Creating an IAM role as default for Amazon Redshift documentation page to set it now.

copy customer from 's3://redshift-downloads/TPC-H/2.18/100GB/customer/' iam_role default delimiter '|' region 'us-east-1'; 

copy orders from 's3://redshift-downloads/TPC-H/2.18/100GB/orders/' iam_role default delimiter '|' region 'us-east-1'; 

copy lineitem from 's3://redshift-downloads/TPC-H/2.18/100GB/lineitem/' iam_role default delimiter '|' region 'us-east-1';

Run the following query to validate load status. The status column should show as completed. You can also review the information in other columns to see details about the loads such as record counts, duration, and data source.

select * from  SYS_LOAD_HISTORY  Where table_name in ('customer','orders','lineitem');

Figure 3: Output of SYS_LOAD_HISTORY showing successful completion of COPY Jobs

Create a materialized view to improve query performance:

Run the following SQL to create a materialized view that pre-compute results set for customer revenues and order volumes by market segment.

CREATE MATERIALIZED VIEW mv_customer_revenue AS 
SELECT 
c.c_custkey, 
c.c_name, 
c.c_mktsegment, 
SUM(l.l_extendedprice * (1 - l.l_discount)) as total_revenue, 
COUNT(DISTINCT o.o_orderkey) as order_count 
FROM customer c 
JOIN orders o ON c.c_custkey = o.o_custkey
JOIN lineitem l ON o.o_orderkey = l.l_orderkey
GROUP BY c.c_custkey, c.c_name, c.c_mktsegment;

Run the following SQL to review the data in the materialized view.

select * from mv_customer_revenue limit 10;

Configure integration with Amazon Quick

Amazon Quick auto discovers the Amazon Redshift provisioned clusters that are associated with your AWS account. These resources must be in the same AWS Region as your Amazon Quick account. For Amazon Redshift clusters in other accounts or Amazon Redshift Serverless, we recommend that you add a VPC connection following the steps in Enabling access to an Amazon Redshift cluster in a VPC documentation. Usually, these steps are performed by your organization’s cloud security administration team.

For serverless, you will apply the same steps in the workgroup instead of the cluster. You can find the VPC and Security Group settings in the Data Access tab of a workgroup.

Figure 4: Amazon Redshift Serverless workgroup VPC and Security groups

You can also refer to How do I privately connect Quick to an Amazon Redshift or RDS data source in a private subnet? for a demonstration.

Create data source

To create a dataset connecting to Amazon Redshift, complete the following steps.

In the Quick left navigation pane, go to Datasets.
Choose the Data sources tab and select Create data source.
Select Amazon Redshift and enter the following:
- Data Source Name: Provide customer-rev-datasource as data source name.
- Connection type: Select the VPC connection created in the previous step.
- Database server: Enter the Amazon Redshift workgroup endpoint (for example, quick-demo-wg.123456789.us-west-2.redshift-serverless.amazonaws.com).
- Port: 5439 (default).
- Database: dev.
- Username/Password: Amazon Redshift credentials with access to the database.
Choose Validate connection. The validation should be successful.

Figure 5: Amazon Redshift data source configuration

Choose Create Data Source to create a data source.

Now let’s explore how to perform all these four steps to configure Athena in Amazon Quick.

Configure and create datasets in Amazon Athena

Amazon Athena provides immediate query capabilities against petabytes of data with automatic scaling to handle concurrent users. Let’s go through the steps to configure connections between Amazon Quick and Amazon Athena.

Set up SQL analytics engine

To create an Athena workgroup:

Open the Amazon Athena console.
In the navigation pane, choose Workgroups.
Choose Create workgroup.
For Workgroup name, enter quick-demo.
For Query result configuration, select Athena managed.
Choose Create workgroup.

Your workgroup is ready immediately for querying data.

Load data and create business views

For Athena, you create tables using the TPC-H benchmark dataset that AWS provides in a public S3 bucket. This approach gives you 1.5 million customer records already optimized in Parquet format without requiring data loading.

To create tables and views in Athena:

Open the Athena Query Editor from the console.

Create a database for your analytics (create S3 bucket if it exists already):

CREATE DATABASE IF NOT EXISTS athena_demo_db 
COMMENT 'Analytics database for customer insights' 
LOCATION 's3://my-analytics-data-lake-[account-id]/';

Create an external table pointing to the TPC-H public dataset:

CREATE EXTERNAL TABLE IF NOT EXISTS athena_demo_db.customer_csv ( 
  C_CUSTKEY INT, 
  C_NAME STRING, 
  C_ADDRESS STRING, 
  C_NATIONKEY INT, 
  C_PHONE STRING, 
  C_ACCTBAL DOUBLE, 
  C_MKTSEGMENT STRING, 
  C_COMMENT STRING 
) 

ROW FORMAT DELIMITED 
FIELDS TERMINATED BY '|' 
STORED AS TEXTFILE 
LOCATION 's3://redshift-downloads/TPC-H/2.18/100GB/customer/'

Create a business-friendly view for analytics:

Run the following SQL to create a view that aggregates customer account balances grouped by market segments.

CREATE VIEW athena_demo_db.customer_deep_analysis AS 
SELECT 
    c_custkey AS customer_id, 
    c_name AS customer_name, 
    c_mktsegment AS market_segment, 
    c_nationkey, 
    ROUND(c_acctbal, 2) AS account_balance, 
    CASE 
        WHEN c_acctbal < 0    THEN 'At-Risk' 
        WHEN c_acctbal < 2500 THEN 'Low' 
        WHEN c_acctbal < 5000 THEN 'Mid' 
        WHEN c_acctbal < 8000 THEN 'High' 
        ELSE 'Premium' 
    END                                                              
AS balance_tier, 

    ROUND(AVG(c_acctbal) OVER (PARTITION BY c_mktsegment), 2)        AS segment_avg, 
    ROUND(c_acctbal - AVG(c_acctbal) OVER (PARTITION BY c_mktsegment), 2) AS vs_segment_avg, 
    ROUND((c_acctbal - AVG(c_acctbal) OVER (PARTITION BY c_mktsegment)) 
          / NULLIF(STDDEV(c_acctbal) OVER (PARTITION BY c_mktsegment), 0), 2) AS segment_z_score, 
    RANK() OVER (PARTITION BY c_mktsegment ORDER BY c_acctbal DESC)  AS rank_in_segment, 
    NTILE(5) OVER (ORDER BY c_acctbal DESC)                          AS global_quintile 

FROM athena_demo_db.customer_csv 
ORDER BY c_acctbal DESC;

Verify your view from Athena with:

SELECT * FROM athena_demo_db.customer_deep_analysis limit 5;

Figure 6: Output from the SELECT query

Configure integration with Amazon Quick

To connect to Amazon Athena in Amazon Quick, follow these steps, consolidated from official AWS documentation and authorizing connections to Amazon Athena.

Authorize Quick to Access Athena, S3 Bucket for data, and S3 bucket for Athena Results.

Open the Amazon Quick Security Settings

Sign in to the Amazon Quick console as an administrator.
In the top-right corner, choose your profile icon, then select Manage account.
Under Permissions, choose AWS resources.
Figure 7: AWS resource permissions

Enable Athena Access

Under Quick access to AWS services, choose Manage.
Locate Amazon Athena in the list of AWS services.
If Athena is already selected but access issues persist, clear the checkbox and re-select it to re-enable Athena.
Under Amazon S3, select S3 buckets.
Check the boxes next to each S3 bucket that Amazon Quick needs to access—including buckets used for Athena query results and any Redshift COPY source buckets.
Enable Write permission for Athena Workgroup to allow Amazon Quick to write Athena query results to S3 and choose Finish.
Choose Save to update the configuration.

The final step is to grant your Amazon Quick author permissions to query your database, Athena tables, and views. Configuration depends on whether AWS Lake Formation is enabled.

If AWS Lake Formation is not enabled

Permissions are managed at the Quick service role level through standard IAM-based S3 access control. Ensure that the Quick service role (for example, aws-quick-service-role-v0) has the appropriate IAM permissions for the relevant S3 buckets and Athena resources. No additional Lake Formation configuration is required.

If AWS Lake Formation is enabled

Lake Formation acts as the central authorization layer, overriding standard IAM-based S3 permissions. Grant permissions directly to the Amazon Quick author or IAM role.

To grant data permissions:

Open the AWS Lake Formation console.
Choose Permissions, then Data permissions, then Grant.
Select the IAM user or role.
Choose the required databases, tables, and columns.
Grant SELECT at minimum; add DESCRIBE for dataset creation.
Repeat for each user or role that requires access.

Create data source

Follow these steps to create an Athena data source on Amazon Quick.

In the Amazon Quick console, navigate to Datasets and choose Data sources tab.
Choose Create data source, then select the Amazon Athena card.
Enter a Data source name (you can give any name of your choice), select your Athena workgroup (like quick-demo), and choose Validate connection.

Figure 8: Athena data source creation

Choose Create data source.

Your Athena data source is now available for building datasets, dashboards, and Topics.

Use Amazon Quick generative AI features

The next steps, from 5–8, demonstrate Amazon Quick generative AI capabilities using Amazon Redshift as a data source. While we use Amazon Redshift in this example, you can substitute with Amazon Athena based on your specific requirements.

Create dashboards

Let’s start by creating datasets from the Amazon Redshift data source.

In the left navigation pane, choose Datasets.
On the Datasets page, choose Create Dataset.
For the data source, select Amazon Redshift data source customer-rev-datasource.
From the menu, choose mv_customer_revenue.

Figure 9: Select table to visualize

You can choose one of the following query modes. For this post, select Directly query your data option and choose Visualize.
- Import to SPICE for quicker analytics – Quick loads a snapshot into its in-memory engine for faster dashboard performance.
- Directly query your data– Quick runs queries on demand against your query engine.
Select Build icon to open a chat window. Enter “Show me orders by market segments” as the prompt. Note that you need Author Pro access to use this feature.

Figure 10: Build visualization using generative BI feature

You can change the visual type to a pie chart and add it to the analysis.

Figure 10: Change visual type

To publish your analysis as a dashboard

After you add the visuals, choose Publish.
Enter a name for the dashboard. For this post, use the Market Segment Dashboard.
Choose Publish dashboard. Your dashboard is now available for viewing and sharing.

Create topics and spaces

To fully maximize enterprise data with AI, we must provide the right structure and context. That’s where Topics and Spaces come in. Topics act as natural language interfaces to your structured datasets, automatically analyzing your data, mapping fields, and adding synonyms. Business users can ask “What are total revenues by market segment?” and receive instant, visualized answers without writing a single line of SQL. Spaces bring together all of your related assets into a single collaborative workspace that democratizes data access, reduces context-switching, accelerates team onboarding, so everyone is working from the same trusted, AI-ready data sources.

To create a Quick topic

From the Amazon Quick homepage, choose Topics, then choose Create topic.
Enter a name for your topic. For this post, use Customer Revenue Analytics.
Enter a description. For example:

The Customer Revenue Analytics topic is designed for business users (including analysts, sales operations teams, finance, and market segment owners who need to explore customer and revenue data without SQL expertise. It serves as a natural language interface over the mv_customer_revenue Amazon Redshift dataset, allowing users to ask plain-English questions like “What are total revenues by market segment?” and receive instant, visualized answers. By automatically mapping business language to the underlying schema, it democratizes access to revenue insights across the organization.

Under Dataset, select mv_customer_revenue.
Choose Create. The topic can take 15–30 minutes to enable depending on the data. During this time, Amazon Quick automatically analyzes your data, selects relevant fields, and adds synonyms.
After the topic is enabled, take a few minutes to review and enrich it. The following are some example enrichments.
1. Add column descriptions to clarify field meaning for business users.
2. Define preferred aggregations (for example, sum compared to average for revenue fields).
3. Confirm which fields are Dimensions and which are Measures.
(Optional) To further refine how your topic interprets and responds to queries, add multiple datasets (for example, a customer CSV combined with a database view), custom instructions, filters, and calculated fields.

After your topic is created, its columns are available to add to a Space or to an Agent by selecting it as a data source.

Figure 11: Create a Quick Topic

Create a Space for your team

Spaces bring together dashboards, topics, datasets, documents, and other resources into organized, collaborative workspaces. By centralizing related assets in a single workspace, Spaces reduce context-switching, accelerate onboarding, so everyone is working from the same trusted data sources.

What to include in your Quick Space

Dashboard – Add the dashboard Market Segment Dashboard published from your mv_customer_revenue analysis. This gives team members instant access to visualizations such as revenue by market segment, top customers by order volume, and revenue distribution.
Topic – Connect the Customer Revenue Analytics (built on the mv_customer_revenue materialized view) to enable natural language queries directly against your Amazon Redshift data.
Optionally, you can upload supporting context to ground your team’s analysis:
- Data dictionary or field definitions for mv_customer_revenue
- Market segment definitions (AUTOMOBILE, BUILDING, FURNITURE, MACHINERY, HOUSEHOLD)
- Business rules for revenue calculation (for example, how discounts are applied in the TPC-H model)
- This implementation guide, so new team members can onboard quickly

To create the Quick Space

From the left navigation menu, choose Spaces, then choose Create space.
Enter a name, for example, Customer Revenue & Segmentation.
Enter a description. For example:

Centralized workspace for customer revenue analysis powered by Amazon Redshift includes interactive dashboards, natural language query access to customer and segment data, and supports documentation for the TPC-H revenue model.

Add knowledge by connecting the Market Segment Dashboard and topic Customer Revenue Analytics.
You can invite team members, such as finance, sales operations, and segment owners, and set appropriate permissions.

Your Space is now ready for collaborative data exploration.

Figure 12: Create a Quick Space

Build chat agents

A custom chat agent delivers conversational AI experiences that understand business context and provide intelligent, grounded responses to user queries. These agents go beyond question-and-answer interactions. They synthesize knowledge from your dashboards, topics, datasets, and documents to explain trends, surface anomalies, guide users through complex analytics workflows, and recommend next steps.

Rather than requiring users to navigate multiple tools or write SQL queries, agents serve as a single conversational interface to your entire analytics environment. Agents can also connect to Actions, pre-built integrations with enterprise tools such as Slack, Microsoft Teams, Outlook, and SharePoint, enabling them to answer questions and trigger real-world workflows, send notifications, create tasks, and interact with external systems directly from the conversation. Custom agents can be tailored to specific business domains, teams, or use cases so that responses align with organizational terminology, data definitions, and business processes. After created, agents can be shared across teams, enabling consistent, actionable, AI-powered data access at scale. For teams working with the mv_customer_revenue dataset, we recommend creating a dedicated Customer Revenue Analysis Agent. This is a purpose-built conversational assistant grounded in your Amazon Redshift data, dashboards, and the Customer Revenue & Segmentation Space.

Create a Quick chat agent

There are two ways that you can use Amazon Quick to create a Quick agent. You can use the navigation menu or directly from Space. The following steps walk you through creating one from the navigation menu.

To create a Quick chat agent

From the left navigation menu, choose Agents, then choose Create agent.
Enter a name for your agent, for example, Customer Revenue Analyst.
Enter a description. For example:

An AI assistant for analyzing customer revenue, market segment performance, and order trends using our Amazon Redshift or data warehouse.

Under Knowledge Sources, add the Customer Revenue & Segmentation Space as a data source. This gives your agent access to the dashboards, topics, and reference documents you’ve already built.
(Optional) Define custom persona instructions to align the agent’s responses with your business context. For example, specifying preferred terminology, response style, or the types of questions it should prioritize.
Choose Launch chat agent.
Start having a conversation with your data. You are welcome to ask any questions. The following are some examples.
- Which market segment generated most revenue?
- Show me order trends

Figure 13: Create a Quick Chat agent

To share your Quick chat agent

After your agent is published, choose Share and invite team members or share it across your organization. Custom agents can be tailored to specific business contexts so that different teams can get AI assistance that speaks their language, without needing to configure anything themselves.

Create Quick Flows

Quick Flows automate repetitive tasks and orchestrate multi-step workflows across your entire analytics environment. This removes manual effort, reducing human error, and ensuring consistent execution of critical business processes. Flows can be triggered on a schedule or launched on demand, giving you flexible control over when and how automation runs.

You can build flows that span the full analytics lifecycle: monitoring data quality and flagging anomalies, generating and distributing scheduled reports to stakeholders, and triggering downstream actions in integrated systems such as Slack, Outlook. Amazon Quick gives you three ways to create a flow, so whether you prefer a no-code conversation or a visual step-by-step builder, there’s an option that fits how you work.

To create a flow from chat

While conversing with My Assistant or a custom agent, describe the workflow that you want to automate in plain English.
Amazon Quick generates the flow and offers to create it directly from your conversation — no configuration screens required.

To create a flow from a natural language description

From the left navigation menu, choose Flows, then choose Create flow.
Enter a plain-English description of your workflow. For example:

” Query revenue data by market segments. Filter by order count and all dates. Search web for comparable relevant market trends. Generate formatted summary reports providing market summary and look ahead per segment. ”

Amazon Quick automatically generates the complete workflow with all the necessary steps.
Optionally, you can add additional steps.
Choose Run Mode to test the Flow.
After your flow is created, share it with team members or publish it to your organization’s flow library, so everyone benefits from the same automation without having to rebuild it independently.

Figure 14: Create a Quick Flow to generate summaries and publish dashboards

For more complex flow, review weekly customer revenue summary flow as an example.

Queries the mv_customer_revenue materialized view in Amazon Redshift for the latest weekly revenue figures by market segment.
Compares results against the prior week to calculate segment-level variance.
Generates a formatted summary report and publishes it to the Customer Revenue & Segmentation Space.
Sends a notification through email or Slack to finance, sales operations, and segment owners with a direct link to the updated dashboard.
Flags any segment where revenue has declined more than a defined threshold, routing an alert to the appropriate owner for follow-up.

This flow transforms what might otherwise be a manual, multi-step reporting process into a fully automated pipeline, so stakeholders receive consistent, timely revenue insights without analyst intervention and saving analysts an estimated 3–5 hours per week. For detailed guidance on creating and managing flows, see Using Amazon Quick Flows. Also review Create workflows for routine tasks demo.

Cleanup

Consider deleting the following resources created while following this post to avoid incurring costs. We encourage you to use the trials at no cost as much as possible to familiarize yourself with the features described.

Delete the Amazon Redshift Serverless workgroup and namespace.
Delete Athena workgroup and S3 Buckets.
Delete the Amazon Quick account used while following this post. If you used an existing account, delete the data sets, dashboards, topics, spaces, agents and flows created.

Conclusion

This integrated approach to business intelligence combines the power of AWS SQL analytics engines with Amazon Quick generative AI capabilities to deliver comprehensive analytics solutions. By following these implementation steps, you establish a foundation for traditional BI reporting, interactive dashboards, natural language data exploration, and intelligent workflow automation. The architecture scales from proof-of-concept implementations to production deployments, transforming how organizations access and act on data insights. For more information about Amazon Quick features and capabilities, see the Amazon Quick documentation. To learn more about Amazon Redshift, visit the Amazon Redshift product page. For Amazon Athena details, see the Amazon Athena product page.

About the authors

Agentic AI for observability and troubleshooting with Amazon OpenSearch Service

Muthu Pitchaimani — Thu, 02 Apr 2026 21:44:17 +0000

Amazon OpenSearch Service powers observability workflows for organizations, giving their Site Reliability Engineering (SRE) and DevOps teams a single pane of glass to aggregate and analyze telemetry data. During incidents, correlating signals and identifying root causes demand deep expertise in log analytics and hours of manual work. Identifying the root cause remains largely manual. For many teams, this is the bottleneck that delays service recovery and burns engineering resources.

We recently showed how to build an Observability Agent using Amazon OpenSearch Service and Amazon Bedrock to reduce Mean time to Resolution (MTTR). Now, Amazon OpenSearch Service brings many of these functions to the OpenSearch UI—no additional infrastructure required. Three new agentic AI features are offered to streamline and accelerate MTTR:

An Agentic Chatbot that can access the context and the underlying data that you’re looking at, apply agentic reasoning, and use tools to query data and generate insights on your behalf.
An Investigation Agent that deep-dives across signal data with hypothesis-driven analysis, explaining its reasoning at every step.
An Agentic Memory that supports both agents, so their accuracy and speed improve the more you use them.

In this post, we show how these capabilities work together to help engineers go from alert to root cause in minutes. We also walk through a sample scenario where the Investigation Agent automatically correlates data across multiple indices to surface a root cause hypothesis.

How the agentic AI capabilities work together

These AI capabilities are accessible from OpenSearch UI through an Ask AI button, as shown in the following diagram, which gives an entry point for the Agentic Chatbot.

Agentic Chatbot

To open the chatbot interface, choose Ask AI.

The chatbot understands the context of the current page, so it understands what you’re looking at before you ask a question. You can ask questions about your data, initiate an investigation, or ask the chatbot to explain a concept. After it understands your request, the chatbot plans and uses tools to access data, including generating and running queries in the Discover page, and applies reasoning to produce a data-driven answer. You can also use the chatbot in the Dashboard page, initiating conversations from a particular visualization to get a summary as shown in the following image.

Investigation agent

Many incidents are too complex to resolve with one or two queries. Now you can get the help of the investigation agent to handle these complex situations. The investigation agent uses the plan-execute-reflect agent, which is designed for solving complex tasks that require iterative reasoning and step-by-step execution. It uses a Large Language Model (LLM) as a planner and another LLM as an executor. When an engineer identifies a suspicious observation, like an error rate spike or a latency anomaly, they can ask the investigation agent to investigate. One of the important steps the investigation agent performs is re-evaluation. The agent, after executing each step, reevaluates the plan using the planner and the intermediate results. The planner can adjust the plan if necessary or skip a step or dynamically add steps based on this new information. Using the planner, the agent generates a root cause analysis report led by the most likely hypothesis and recommendations, with full agent traces showing every reasoning step, all findings, and how they support the final hypotheses. You can provide feedback, add your own findings, iterate on the investigation goal, and review and validate each step of the agent’s reasoning. This approach mirrors how experienced incident responders work, but completes automatically in minutes. You can also use the “/investigate” slash command to initiate an investigation directly from the chatbot, building on an ongoing conversation or starting with a different investigation goal.

Agent in action

Automatic query generation

Consider a situation where you’re an SRE or DevOps engineer and received an alert that a key service is experiencing elevated latency. You log in to the OpenSearch UI, navigate to the Discover page, and select the Ask AI button. Without any expertise in the Piped Processing Language (PPL) query language, you enter the question “find all requests with latency greater than 10 seconds”. The chatbot understands the context and the data that you’re looking at, thinks through the request, generates the right PPL command, and updates it in the query bar to get you the results. And if the query runs into any errors, the chatbot can learn about the error, self-correct, and iterate on the query to get the results for you.

Investigation and investigation management

For complex incidents that normally require manually analyzing and correlating multiple logs for the possible root cause, you can choose Start Investigation to initiate the investigation agent. You can provide a goal for the investigation, along with any context or hypothesis that you want to instruct the investigation. For example, “identify the root cause of widespread high latency across services. Use TraceIDs from slow spans to correlate with detailed log entries in the related log indices. Analyze affected services, operations, error patterns, and any infrastructure or application-level bottlenecks without sampling”.

The agent, as part of the conversation, will offer to investigate any issue that you’re trying to debug.

The agent sets goals for itself along with any other relevant information like indices, associated time range, and other, and asks for your confirmation before creating a Notebook for this investigation. A Notebook is a way within the OpenSearch UI to develop a rich report that’s live and collaborative. This helps with the management of the investigation and allows for reinvestigation at a later date if necessary.

After the investigation starts, the agent will perform a quick analysis by log sequence and data distribution to surface outliers. Then, it will plan for the investigation into a series of actions, and then performs each action, such as query for a specific log type and time range. It will reflect on the results at every step, and iterate on the plan until it reaches the most likely hypotheses. Intermediate results will appear on the same page as the agent works so that you can follow the reasoning in real time. For example, you find that the Investigation Agent accurately mapped out the service topology and used it as a key intermediary steps for the investigation.

As the investigation completes, the investigation agent concludes that the most likely hypothesis is a fraud detection timeout. The associated finding shows a log entry from the payment service: “currency amount is too big, waiting for fraud detection”. This matches a known system design where large transactions trigger a fraud detection call that blocks the request until the transaction is scored and assessed. The agent arrived at this finding by correlating data across two separate indices, a metrics index where the original duration data lived, and a correlated log index where the payment service entries were stored. The agent linked these indices using trace IDs, connecting the latency measurement to the specific log entry that explained it.

After reviewing the hypothesis and the supporting evidence, you find the result reasonable and aligns with your domain knowledge and past experiences with similar issues. You can now accept the hypothesis and review the request flow topology for the affected traces that were provided as part of the hypothesis investigation.

Alternatively, if you find that the initial hypothesis wasn’t helpful, you can review the alternative hypothesis at the bottom of the report and select any of the alternative hypotheses if there’s one that’s more accurate. You can also trigger a re-investigation with additional inputs, or corrections from previous input so that the Investigation Agent can rework it.

Getting started

You can use any of the new agentic AI features (limits apply) in the OpenSearch UI at no cost. You will find the new agentic AI features ready to use in your OpenSearch UI applications, unless you have previously disabled AI features in any OpenSearch Service domains in your account. To enable or disable the AI features, you can navigate to the details page of the OpenSearch UI application in AWS Management Console and update the AI settings from there. Alternatively, you can also use the registerCapability API to enable the AI features or use the deregisterCapability API to disable them. Learn more at Agentic AI in Amazon OpenSearch Services.

The agentic AI feature uses the identity and permissions of the logged in users for authorizing access to the connected data sources. Make sure that your users have the necessary permissions to access the data sources. For more information, see Getting Started with OpenSearch UI.

The investigation results are saved in the metadata system of OpenSearch UI and encrypted with a service managed key. Optionally, you can configure a customer managed key to encrypt all of the metadata with your own key. For more information, see Encryption and Customer Managed Key with OpenSearch UI.

The AI features are powered by Claude Sonnet 4.6 model in Amazon Bedrock. Learn more at Amazon Bedrock Data Protection.

Conclusion

The new agentic AI capabilities announced for Amazon OpenSearch Service help reduce Mean Time to Resolution by providing context-aware agentic chatbot for assistance, hypothesis-driven investigations with full explainability, and agentic memory for context consistency. With the new agentic AI capabilities, your engineering team can spend less time writing queries and correlating signals, and more time acting on confirmed root causes. We invite you to explore these capabilities and experiment with your applications today.

About the authors

Streamline Apache Kafka topic management with Amazon MSK

Swapna Bandla — Thu, 02 Apr 2026 15:32:26 +0000

If you manage Apache Kafka today, you know the effort required to manage topics. Whether you use infrastructure as code (IaC) solutions or perform operations with admin clients, setting up topic management takes valuable time that could be spent on building streaming applications.

Amazon Managed Streaming for Apache Kafka (Amazon MSK) now streamlines topic management by supporting new topic APIs and console integration. You can programmatically create, update, and delete Apache Kafka topics using familiar interfaces including AWS Command Line Interface (AWS CLI), AWS SDKs, and AWS CloudFormation. With these APIs, you can define topic properties such as replication factor and partition count and configuration settings like retention and cleanup policies. The Amazon MSK console integrates these APIs, bringing all topic operations to one place. You can now create or update topics with a few selections using guided defaults while gaining comprehensive visibility into topic configurations, partition-level information, and metrics. You can browse for topics within a cluster, review replication settings and partition counts, and go into individual topics to examine detailed configuration, partition-level information, and metrics. A unified dashboard consolidates partition topics and metrics in one view.

In this post, we show you how to use the new topic management capabilities of Amazon MSK to streamline your Apache Kafka operations. We demonstrate how to manage topics through the console, control access with AWS Identity and Access Management (IAM), and bring topic provisioning into your continuous integration and continuous delivery (CI/CD) pipelines.

Prerequisites

To get started with topic management, you need:

An active AWS account with appropriate IAM permissions for Amazon MSK.
An existing Amazon MSK Express or Standard cluster using Apache Kafka version 3.6 and above.
Basic familiarity with Apache Kafka concepts like topics, partitions, and replication.
AWS CLI installed and configured (for command line examples).

Creating topics

The MSK console provides a guided experience with sensible defaults while still offering advanced configuration options when you need them.

Navigate to the Amazon MSK console and select your cluster.
Choose the Topics tab, then choose Create topic.
Enter a topic name (for example, customer-orders).
Specify the number of partitions (use the guided defaults or customize based on your needs).
Set the replication factor. Note that Express brokers improve the availability and durability of your Amazon MSK clusters by setting values for critical configurations and protecting them from common misconfiguration. If you try to create a topic with a replication factor value other than 3, Amazon MSK Express will create the topic with a replication factor of 3 by default.
(Optional) Configure advanced settings like retention period or message size limits.
Choose Create topic.

The console validates your configuration and creates the topic. You can create multiple topics simultaneously with the same configuration settings. These topic API responses reflect data that updates approximately every minute. For the most current topic state after making changes, wait approximately one minute before querying.

Configuration considerations

When choosing configuration options, consider your workload requirements:

You can configure more partitions to achieve higher throughput but this requires more broker resources. Refer to Best practices for Standard brokers and Amazon MSK Express broker quota for more information on partition limits.
With Standard brokers you can improve durability by configuring higher replication factors, though this will increase your storage costs. Refer to “Build highly available clusters” in Best practices for Standard brokers for more information on replication factors.
Standard brokers support the full range of Apache Kafka topic configurations.
Express brokers offer a curated set of the most important settings. Refer to Topic-level configurations on Express Brokers for more information.

Viewing and monitoring topics

After you create topics, the MSK console provides comprehensive visibility into their configuration. When you select a specific topic, you will see detailed information:

Partitions tab: Shows the distribution of partitions across brokers, including leader assignments and in-sync replica status showcasing Broker IDs for leader and replicas.
Configuration tab: Displays all topic-level configuration settings.
Monitoring tab: Integrates with Amazon CloudWatch to show metrics like bytes in/out, message rates, and consumer lag.

Updating topic configurations

As your workload requirements evolve, you might need to adjust topic configurations. You can modify various topic settings depending on your cluster type. For example:

Retention settings: Adjust retention.ms (time-based) or retention.bytes (size-based) to control how long messages are retained.
Message size limits: Modify max.message.bytes to accommodate larger or smaller messages.
Compression: Change compression.type to optimize storage and network usage.

Configuration changes take effect immediately for new messages. Existing messages remain subject to the previous configuration until they age out or are consumed.

Deleting topics

Amazon MSK also provides APIs for deleting topics that are no longer in use. Before deleting a topic, verify that:

No active producers are writing to the topic
All consumers have finished processing messages
You have backups if you need to retain the data
Downstream applications won’t be impacted

Important: Topic deletion permanently removes all messages in the topic.

Control access with IAM

Beyond streamlining topic operations, you also need appropriate access controls. Access control uses IAM, so you define permissions using the same model that you apply to other AWS resources. Amazon MSK uses a two-level permission model:

Resource-level permissions: An IAM policy that enforces which operations the cluster will allow
Principal-level permissions: IAM policies attached to Roles or Users that enforce which operations a principal is allowed to perform on a cluster

With this separation, you can control access depending on your organizational needs and access patterns for your cluster. Refer to the IAM permissions documentation for IAM permissions required for topic management for the Amazon MSK cluster.

You can grant your operations team broad access to manage all topics and restrict application teams to manage only their own topics. The permission granularity that you need is available through standard IAM policies. If you’ve already configured IAM permissions for Apache Kafka topics, they work immediately with the new functionality without any migration or reconfiguration.

Here is a sample IAM policy definition that allows Describe Topic API

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:111111111111:cluster/iam-auth-acl-test/a6b5c6d5-f74f-4dbc-ad14-63fb5e87fe4f-2"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:DescribeTopicDynamicConfiguration"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:111111111111:topic/iam-auth-acl-test/a6b5c6d5-f74f-4dbc-ad14-63fb5e87fe4f-2/*"
            ]
        }
    ]
}

This IAM policy grants the necessary permissions to describe Kafka topics in your Amazon MSK cluster. The policy includes three key permissions:

kafka-cluster:Connect – Allows connection to the specified MSK cluster
kafka-cluster:DescribeTopic – Enables viewing topic details
kafka-cluster:DescribeTopicDynamicConfiguration – Enables viewing topic dynamic configuration

The policy is scoped to a specific cluster ARN and applies to all topics within that cluster using the wildcard pattern /*. Replace the placeholder Amazon MSK cluster ARN with your MSK cluster ARN.

Infrastructure as Code

If you manage infrastructure as code (IaC), you can now define topics alongside clusters in your CloudFormation templates:

Resources:
    OrdersTopic:
      Type: AWS::MSK::Topic
      Properties:
        ClusterArn: !GetAtt MyMSKCluster.Arn
        TopicName: orders
        NumPartitions: 6
        ReplicationFactor: 3
        Config:
          retention.ms: "604800000"

This approach brings topic provisioning into your CI/CD pipelines.

Availability and pricing

The new Amazon MSK topic management experience is available today for Standard and Express Amazon MSK clusters using Apache Kafka version 3.6 and above in all AWS Regions where Amazon MSK is offered, at no additional cost.

Cleanup

To avoid incurring additional charges to your AWS account, ensure you delete all resources created during this tutorial, including:

Amazon MSK cluster
Any Kafka topics created
Associated AWS resources (security groups, VPCs, etc., if created specifically for this blog)

Remember to verify that all resources have been successfully removed to prevent ongoing costs.

Conclusion

Topic management has been a persistent pain point for Apache Kafka operations. The new integrated experience in Amazon MSK now reduces operational friction by bringing topic operations into the AWS tools that you use every day. You now have a consistent, streamlined way to handle these operations for all Apache Kafka topics across multiple MSK clusters. This capability reflects our commitment to reducing operational complexity in Apache Kafka. You get the reliability and performance of Apache Kafka without the operational overhead that traditionally comes with it. Your team spends less time on infrastructure maintenance and more time building streaming applications that drive your business forward.

Ready to start streamlining your topic management? Start managing your topics today through the Amazon MSK console or by visiting the Amazon MSK documentation.

About the authors

How to set up a network-isolated VPC for Amazon SageMaker Unified Studio

Rohit Vashishtha — Thu, 02 Apr 2026 15:28:23 +0000

Organizations are finding significant value using an integrated experience for all your data and AI with Amazon SageMaker Unified Studio. However, many organizations require strict network control to meet security and regulatory compliance requirements like HIPAA or FedRAMP for their data and AI initiatives, while maintaining operational efficiency.

In this post, we explore scenarios where customers need more control over their network infrastructure when building their unified data and analytics strategic layer. We’ll show how you can bring your own Amazon Virtual Private Cloud (Amazon VPC) and set up Amazon SageMaker Unified Studio for strict network control.

Solution overview

The solution covers complete technical know-how of a fully private network architecture using Amazon VPC with no public internet exposure. The approach leverages AWS PrivateLink through VPC endpoints to provide a secure communication between SageMaker Unified Studio and essential AWS services entirely over the AWS backbone network.

The architecture consists of three core components: a custom VPC named airgapped with multiple private subnets distributed across at least three Availability Zones for high availability, a comprehensive set of VPC interface and gateway endpoints for service connectivity, and the SageMaker Unified Studio domain configured to operate exclusively within this isolated environment. This design helps ensure that sensitive data never traverses the public internet while maintaining full functionality for data cataloging, query execution, and machine learning workflows.

By implementing this network-isolated VPC configuration, organizations gain granular control over network traffic, simplified compliance auditing, and the ability to integrate SageMaker Unified Studio with existing private data sources through controlled network pathways. The solution supports both immediate operational needs and long-term scalability through careful IP address planning and modular endpoint architecture.

Prerequisites

The set up requires you to have an existing VPC (for this post, we’ll refer to the name as airgapped but in reality, it refers to the VPC you would like to securely set up SageMaker Unified Studio). If you don’t have an existing VPC, you can follow SageMaker Unified Studio domain quick create administrator guide to get started.

The high level steps to create a VPC meeting minimum requirements for SageMaker Unified Studio are as follows:

In the AWS Management Console, navigate to the VPC console.
Choose Create VPC.
Select the VPC and more radio button.
For Name tag auto-generation, enter airgapped or a name of your choice.
Keep the default values for IPv4 CIDR block, IPv6 CIDR block, Tenancy, NAT gateways, VPC endpoints, and DNS options.
Select 3 for Number of Availability Zones (AZs).
Select 0 for Number of public subnets.
Choose Create VPC.

This produces the following VPC resource map:

Figure 1 – VPC configuration

Set up SageMaker Unified Studio

Now, we will set up SageMaker Unified Studio in an existing VPC, named airgapped-vpc.

Navigate to the SageMaker console, choose Domains in the navigation pane.
Choose Create Domain.
For How do you want to set up your domain?, select Quick set up.
Expand the Quick set up settings
Provide a name for your domain, such as airgapped-domain.
For Virtual private cloud (VPC), select airgapped-vpc.
For subnets, select a minimum of two private subnets.
Choose Continue.
Enter an email address to create a user in AWS IAM Identity Center.
Choose Create domain.
Once the domain is created, choose Open unified studio or use SageMaker Unified Studio URL under Domain details to access SageMaker Unified Studio.

Figure 2 – Amazon SageMaker Unified Studio URL Welcome Page
After logging in to SageMaker Unified Studio, create a project using the guided wizard.
Once the project is created, we need to add the necessary VPC endpoints to allow traffic from the project to communicate to AWS services.
S3 Gateway VPC endpoint was already selected as part of VPC creation step 5 in prerequisites and thus created by default. Now we must add two more VPC endpoints for Amazon DataZone and AWS Security Token Service as illustrated in following step.

These are the minimum set of VPC endpoints to allow using the tooling within SageMaker Unified Studio. For a list of other mandatory and non-mandatory VPC endpoints refer to the tables in the latter part of this post.

Create an interface endpoint

To create an interface endpoint, complete following steps:

Go to the SageMaker Unified Studio Project details page and copy the Project ID.
Figure 3 – SageMaker Unifed Studio Project Details Page
Go to the VPC console and choose Endpoints.
Choose Create Endpoint.
Enter a name for the endpoint, for example, DataZone endpoint for SageMaker Unified Studio.
For AWS Services, enter DataZone.

Figure 4 – Interface Endpoint creation wizard for AWS Service datazone
Select Service Name = com.amazonaws.us-east-1.datazone from the available options.

Figure 5 – Interface Endpoint creation wizard network settings
Select the subnets in the airgapped-vpc that you created earlier.
Filter the Security Groups by pasting the copied Project ID.
Select the security group with Group Name datazone-<project-id>-dev.
Choose Create Endpoint.
Repeat the same steps to create a VPC endpoint for AWS STS.
Once the VPC endpoints are created, validate connectivity in the SageMaker project by running a SQL query or using a Jupyterlab notebook.

For a successful domain and project which does not get into any service level usage, the mandatory VPC endpoints to be created are: S3 Gateway, DataZone, and STS interface endpoints. For other service usage dependent operations like authentication, data preview and working with compute, you would require other mandatory service specific endpoints explained later in this post.

Best practices for VPC set up for various use cases

When setting up SageMaker Unified Studio domain and project profiles, you need to specify the VPC network, subnets, and security groups. Here are some best practices around IP allocation, usage volume and expected growth to consider for different use cases within enterprises.

Production and enterprise use cases

If your organization require strict network control to meet security and compliance requirements for data and AI initiatives, consider following best practices in your production environment.

Use the bring-your-own (BYO) VPC approach to comply with company-specific networking and security requirements.
Implement private networking using VPC endpoints to keep traffic within the AWS backbone.
Use at least two private subnets across different Availability Zones.
Enable DNS hostnames and DNS Support.
Disable auto-assign public IP on subnets.
Plan IP capacity for at least 5 years. A prescriptive guidance for SageMaker Unified Studio is shared in VPC and Networking details section later in this post. Consider the following:
- Number of users
- Number of apps per user
- Number of unique instance types per user
- Average number of training instances
- Expected growth percentage

Testing and non-production use cases

For development, testing, non-prod environment where use cases don’t have stringent security and compliance requirements, use automated setup for quick experiments. Use sample CloudFormation github templates as part of the SageMaker Unified Studio express set up, to automate domain and project creation. However, this includes an Internet Gateway which may not be suitable for security-sensitive environments.

Private networking use cases

VPCs with private subnets require essential service endpoints to allow client resources like Amazon EC2 instances to securely access AWS services. The traffic between your VPC and AWS services remains within AWS network avoiding public internet exposure.

Implement all mandatory VPC endpoints for core services (SageMaker, DataZone, Glue, and more).
Add optional endpoints based on specific service needs, like IPv4 endpoints, dual-stack endpoints, and FIPS endpoints to programmatically connect to an AWS service.
Work with network administrators for:
- Preinstalling needed resources through secure channels like private subnets and self-referencing inbound rules in security groups to enable limited access.
- Allowlisting only necessary external connections like NAT gateway IP and bastion host access in firewall rules.
- Setting up appropriate proxy configurations if required.

External data source access use cases

Consider the following when working with external systems like third-party SaaS platforms, on-premises databases, partner APIs, legacy systems, or external vendors.

Consult with network administrators for appropriate connection methods.
Consider AWS PrivateLink integration where available.
Implement appropriate security measures for non-AWS data your source documents.
For High Availability:
- Deploy across at least three different Availability Zones (at least two for AWS Regions with only two AZs).
- Verify there’s a minimum of three free IPs per subnet.
- Consider larger CIDR blocks (/16 recommended) for future scalability.

VPC and networking details

In this section, we provide details of each networking aspect starting with choice of VPCs, network connectivity details for integrated services to work, the basis of VPC and subnet requirements, and finally the VPC endpoints required for private service access.

VPC

At a high level, you have two options to supply VPCs and subnets:

Bring-your-own (BYO) VPC. This is typically the case for most customers, as most have company specific networking and security requirements to reuse an existing VPC, or to create a VPC that are compliant with those requirements.
Create VPC with the SageMaker quick set up template. When creating a SageMaker Unified Studio domain (DataZone V2 domain in CloudFormation) through the automated quick set up, you will be shown a Quick create stack wizard in CloudFormation which creates VPCs and subnets used to configure your domain.Note:The quick create stack using template URL is not intended for production use. The template creates an Internet Gateway, which is not allowed in many enterprise settings. This is only appropriate if you are either trying out SageMaker Unified Studio or, running SageMaker Unified Studio for use cases that don’t have stringent security requirements.If you choose this option, you start with SageMaker console, navigate to domains and click Create domain button, followed by Create VPC button. You will navigate to CloudFormation and click on Create stack button to create a sample VPC named SageMakerUnifiedStudio-VPC with just one-click for trying out SageMaker Unified Studio.

Figure 6 – Create VPC button in SageMaker Unified Studio Create Domain Wizard

Cost estimation for recommended VPC set up

The exact cost depends on the configuration of your VPC. For more complex networking set ups (multi-VPC), you may need to use additional networking components such as a Transit Gateway, Network Firewall, and VPC Lattice. These components may incur charges, and cost depends on usage and AWS Region. Interface VPC endpoints are charged per availability zone. They also have a fixed and a variable component in the pricing structure. Use the AWS Pricing Calculator for a detailed estimate.

Network Connectivity

With regards to connectivity to the underlying AWS services integrated within SageMaker Unified Studio, there are two ways to enable connectivity (these are not Studio specific, these are standard ways to enable network connectivity within a VPC). This is an important security consideration that depends on your organization’s security policies.

Through the public Internet. Your traffic will traverse over the public Internet through an Internet Gateway in your VPC.
1. Your VPC must have an Internet Gateway attached to it.
2. Your public subnet must have a NAT Gateway. In addition, your public subnet’s route table must have a default route (0.0.0.0 for IPv4) to the Internet Gateway. This route is what makes the subnet public.
3. Your private subnets must have a default route to the public subnet’s NAT Gateway.
Through the AWS backbone. Your traffic will remain within the private AWS backbone through PrivateLink (by provisioning Interface and Gateway endpoints for the necessary AWS services in each Availability Zone).
1. A list of all the AWS services integrated into Studio and the VPC endpoints required can be found in section VPC Endpoints covered later in this post.
2. For non-AWS resources, certain external providers of these services may offer PrivateLink integration. Check with each provider’s documentation and your network administrator to understand the most suitable way to connect to these external providers.

In a private networking scenario, you will need to consider whether you need connectivity to non-AWS resources in a way that’s compliant with your organization’s security policies. A few examples include the following:

If you need to download software in your remote IDE host (for example, command line programs, such as Ping and Traceroute)
If you have code that connects to external APIs.
If you use software (such as JupyterLab or Code Editor extensions) that rely on external APIs.
If you depend on software dependencies hosted in the public domain (such as Maven, PyPi, npm)
If you need cross-Region access to certain resources (such as access to S3 buckets in a different Region)
If you need functionality whose underlying AWS services do not have VPC endpoints in all Regions or any Region.
1. Amazon Q (powers Q and code suggestions)
2. SQL Workbench (powers Query Editor)
3. IAM (powers Glue connections)

If you need to connect to data sources outside of AWS (such as Snowflake, Microsoft SQL Server, Google BigQuery)
Enterprise network administrators must also complete either of the following prerequisites to handle private networking scenarios:

Preinstall needed resources through secure channels if possible. An example would be to customize your SageMaker AI image by installing dependencies, after they are code scanned, vetted technically and legally by your organization.
If AWS PrivateLink integration is not available for external providers, allowlist network connections to these external sources. Allow firewall egress rules, directly or indirectly, through a proxy in your organization’s network. Check with your network administrator to understand the most appropriate option for your organization.

VPC Requirements

When setting up a new SageMaker Unified Studio Domain, it’s necessary to supply a VPC. It’s important to note that these VPC requirements are a union of all the requirements from the respective compute services integrated into Studio, some of which are reinforced by validation checks during the corresponding blueprint’s deployment. If these requirements that have validation checks are not fulfilled, the resource(s) contained in that blueprint may fail to create on project creation (on-create), or when creating the compute resource (on-demand). This section will present a summary of these requirements, as well as relevant documentation links from which they originate.

Subnet requirements for specific compute in a VPC

This section lists the compute services integrated in SageMaker Unified Studio that require VPC/subnets when provisioning the respective compute resources.

Compute Connections

Other Services

Requirements

Number of subnets: At least two private subnets. This requirement comes from Redshift Serverless.
Availability zones (AZs): At least two different AZs (for Regions with two AZs, two subnets are sufficient). This requirement comes from Redshift Serverless. For workgroups with Enhanced VPC Routing (EVR), you need three AZs.
Free IPs per subnet: At least three Ips per subnet. This requirement comes from Redshift Serverless without EVR. For detailed IP addresses requirement with EVR enabled workgroups, refer to Serverless usage considerations. Three is a minimum and may not be enough for your needs. For example, EMR cluster creation will fail if no subnets with enough IPs are found in the VPC. We recommend doing a forward-looking capacity planning exercise based on your use cases (for example, growth rate, users, compute needs) to project at least 5 years into the future. This helps to determine how many IPs are needed by the team using Studio and other services that use this VPC and come up with a ceiling for the CIDR block size.
Private or public subnets: We enforce that at least three private subnets be supplied, and recommend that only private subnets are chosen, with a few nuances. This requirement comes from SageMaker AI domain. A new SageMaker AI domain, when set up with VpcOnly mode, requires that all subnets in the VPC be private. This is the default networking mode in the Tooling blueprint. If you choose to use PublicInternetOnly mode, this restriction does not apply, you may choose public subnets from your VPC. To change the mode, modify the Tooling Blueprint parameter sagemakerDomainNetworkType.
Enable DNS hostname and DNS Support: Both must be enabled. This requirement comes from EMR. Without these VPC settings, enableDnsHostname and enableDnsSupport, connecting to the EMR Cluster using the private DNS name through the Livy Endpoint will fail. SSL Verification, which can only be done when connecting using the DNS name, not the IP.
Auto assign public IP: Disable. We recommend that this EC2 subnet setting (mapPublicIpOnLaunch) be disabled when using private subnets, because public IPs come at a cost and are a scarce resource in the total addressable IPv4 space.

VPC endpoints

If you choose to run SageMaker Unified Studio without public internet access, VPC endpoints are required for all services SageMaker Unified Studio needs to access. These endpoints provide secure, private connectivity between your VPC and AWS services without traversing the public internet. The following table lists the required endpoints, their types, and what each is used for.

Some endpoints may not show up directly in your browser’s network tab. The reason is that some of these services (such as CloudWatch) are transitively invoked by other services.

Mandatory endpoints

The following are required endpoints for SageMaker Unified Studio and supporting services to function properly. Gateway endpoints can be used where available, you can use interface endpoints for all other AWS services.

AWS service	Endpoint	Type	Purpose
Glue	`com.amazonaws.${region}.glue`	Interface	For Data Catalog and metadata management
STS	`com.amazonaws.${region}.sts`	Interface	Required for assuming IAM roles
S3	`com.amazonaws.${region}.s3`	Gateway	Required for datasets, Git backups, notebooks, and Git sync
SageMaker	`com.amazonaws.${region}.sagemaker.api`	Interface	Required for calling SageMaker APIs
SageMaker	`com.amazonaws.${region}.sagemaker.runtime`	Interface	For invoking deployed inference endpoints
DataZone	`com.amazonaws.${region}.datazone`	Interface	For data catalog and governance
Secrets Manager	`com.amazonaws.${region}.secretsmanager`	Interface	To securely access secrets
SSM	`com.amazonaws.${region}.ssm`	Interface	For secure command execution
SSM	`com.amazonaws.${region}.ssmmessages`	Interface	Enables live SSM sessions
KMS	`com.amazonaws.${region}.kms`	Interface	For decrypting data (volumes, S3, secrets)
EC2	`com.amazonaws.${region}.ec2`	Interface	For subnet and ENI management
EC2	`com.amazonaws.${region}.ec2messages`	Interface	Required for SSM messaging
Athena	`com.amazonaws.${region}.athena`	Interface	Required to run SQL queries
Amazon Q	`com.amazonaws.${region}.q`	Interface	Used by SageMaker Notebooks for enhanced productivity

Optional Endpoints

Only create these if the corresponding service is used in your environment.

AWS service	Endpoint	Type	Purpose
EMR	`com.amazonaws.${region}.emr-serverless`	Interface	Serverless Spark/Hive jobs
	`com.amazonaws.${region}.emr-serverless-services.livy`	Interface	Required for Livy job submission (EMR Serverless)
	`com.amazonaws.${region}.elasticmapreduce`	Interface	Classic EMR (EC2-based)
	`com.amazonaws.${region}.emr-containers`	Interface	EMR on EKS workloads
Redshift	`com.amazonaws.${region}.redshift`	Interface	For provisioned Redshift clusters
	`com.amazonaws.${region}.redshift-serverless`	Interface	For Redshift Serverless
	`com.amazonaws.${region}.redshift-data`	Interface	Required for running SQL against Redshift
Amazon Bedrock	`com.amazonaws.${region}.bedrock-runtime`	Interface	Invoke Bedrock models at runtime
	`com.amazonaws.${region}.bedrock-agent`	Interface	For Bedrock knowledge agents
	`com.amazonaws.${region}.bedrock-agent-runtime`	Interface	For running knowledge agent workloads
CloudWatch	`com.amazonaws.${region}.logs`	Interface	Application and notebook logs
RDS	`com.amazonaws.${region}.rds`	Interface	Connect to Amazon RDS and Aurora
CodeCommit	`com.amazonaws.${region}.codecommit`	Interface	Git integration with CodeCommit
CodeCommit	`com.amazonaws.${region}.git-codecommit`	Interface	Alternative endpoint for CodeCommit
CodeConnections and CodeStar	`com.amazonaws.${region}.codeconnections.api`	Interface	GitHub and GitLab repo integration
CodeConnections and CodeStar	`com.amazonaws.${region}.codestar-connections.api`	Interface	Alias of CodeConnections

Clean up

AWS resources provisioned in your AWS accounts may incur costs based on the resources consumed. Make sure you do not leave any unintended resources provisioned. If you created a VPC and subsequent resources as part of this post, make sure you delete them.

The following service resources provisioned during this blog post need to be deleted:

IAM Identity Center users and groups.
Resources provisioned within your project using tooling configuration and blueprints within your domain.
The airgapped VPC.

Conclusion

In this post, we walked through the process of using your own existing VPC when creating domains and projects in SageMaker Unified Studio. This approach benefits customers by giving them greater control over their network infrastructure while using the comprehensive data, analytics, and AI/ML capabilities of Amazon SageMaker. We also explored the critical role of VPC endpoints in this set up. You now understand when these become necessary components of your architecture, particularly in scenarios requiring enhanced security, compliance with data residency requirements, or improved network performance.

While using a custom VPC requires more initial set up than the Quick Create option, it provides the flexibility and control many organizations need for their data science and analytics workflows. This approach provides a mechanism for your SageMaker environment to integrate with your existing infrastructure and adheres to your organization’s networking policies. Custom VPC configurations are a powerful tool in your arsenal for building secure, compliant, and efficient data science environments.

To learn more, visit Amazon SageMaker Unified Studio – Administrator Guide and User Guide.

AWS Big Data Blog

Unified observability in Amazon OpenSearch Service: metrics, traces, and AI agent debugging in a single interface

Scenario 1: An underperforming AI agent

Why this matters for AI agents

Instrumenting AI agents

Scenario 2: Investigating a microservice issue

Checking the infrastructure with PromQL

How it’s wired together

Getting started

About the authors

Migrate to Apache Flink 2.2 on Amazon Managed Service for Apache Flink

What’s new in Amazon Managed Service for Apache Flink 2.2

Runtime and performance

SQL and Table API highlights

Migrating from Flink 1.x to 2.2

In-place version upgrades

Auto-rollback

Unsupported open source features

Prerequisites

Step 1: Update your application code

Step 2: Check state compatibility

Step 3: Turn on auto-rollback and automatic snapshots

Step 4: Take a manual snapshot (recommended)

Step 5: Run the upgrade

Step 6: Monitor the upgrade

Step 7: Validate application behavior

Step 8: Rollback (if needed)

Next steps

Conclusion

About the authors

Using Apache Sedona with AWS Glue to process billions of daily points from a geospatial dataset

Introduction to geospatial data

Core concepts of Apache Sedona

Use case

Solution overview

Prerequisites

Solution walkthrough

Upload the Apache Sedona libraries to Amazon S3

Download and upload the geospatial data to Amazon S3

Create an AWS Glue job and set up the job

Processing the geospatial flights data

Performance insights

Visualizing and analyzing geospatial data with Kepler.gl

Downloading the geospatial files

Visualizing the data in a graph

Clean up

Conclusion

About the authors

Analyzing your data catalog: Query SageMaker Catalog metadata with SQL

How metadata export works

Solution overview

What metadata is exposed?

Prerequisites

Enable data export using the AWS CLI

Access the exported asset table

Queries for observability and analytics

Clean up resources

Conclusion

About the Authors

Configure a custom domain name for your Amazon MSK cluster enabled with IAM authentication

Solution overview

Certificate management

Network Load Balancer

Domain Name System (DNS)

Amazon MSK

Solution Deployment

Cleanup

Frequently Asked Question about Custom Domain Name

Are there any limitations for this solution on MSK?

How the custom domain name solution scales when we add new brokers?

What changes are required when we remove brokers?

Is there any change in configuration required if there is any broker failure?

How to allow public access to a private MSK cluster using the custom domain name solution?

In the first part of the blog, two patterns have been highlighted. How to decide which pattern to use and why?

Option 1: Only bootstrap connection through NLB

Option 2: All connections through NLB

Is Amazon Route 53 required to use a custom domain name with Amazon MSK?

We don’t use Amazon Certificate Manager (ACM), can NLB integrate with other 3rd party certificate managers?

Getting connection to node terminated during authentication after setting advertised.listeners , what could be the issue?

Getting “unexpected broker id, expected 2 or empty string, but received 1”, what is causing this error?

Getting connection to node terminated during authentication after setting `advertised.listeners` , what could be the issue?