AWS Big Data Blog

Introducing workload simulation workbench for Amazon MSK Express broker

Manu Mishra — Tue, 07 Apr 2026 16:49:49 +0000

Validating Kafka configurations before production deployment can be challenging. In this post, we introduce the workload simulation workbench for Amazon Managed Streaming for Apache Kafka (Amazon MSK) Express Broker. The simulation workbench is a tool that you can use to safely validate your streaming configurations through realistic testing scenarios.

Solution overview

Varying message sizes, partition strategies, throughput requirements, and scaling patterns make it challenging for you to predict how your Apache Kafka configurations will perform in production. The traditional approaches to test these variables create significant barriers: ad-hoc testing lacks consistency, manual set up of temporary clusters is time-consuming and error-prone, production-like environments require dedicated infrastructure teams, and team training often happens in isolation without realistic scenarios. You need a structured way to test and validate these configurations safely before deployment. The workload simulation workbench for MSK Express Broker addresses these challenges by providing a configurable, infrastructure as code (IaC) solution using AWS Cloud Development Kit (AWS CDK) deployments for realistic Apache Kafka testing. The workbench supports configurable workload scenarios, and real-time performance insights.

Express brokers for MSK Provisioned make managing Apache Kafka more streamlined, more cost-effective to run at scale, and more elastic with the low latency that you expect. Each broker node can provide up to 3x more throughput per broker, scale up to 20x faster, and recover 90% quicker compared to standard Apache Kafka brokers. The workload simulation workbench for Amazon MSK Express broker facilitates systematic experimentation with consistent, repeatable results. You can use the workbench for multiple use cases like production capacity planning, progressive training to prepare developers for Apache Kafka operations with increasing complexity, and architecture validation to prove streaming designs and compare different approaches before making production commitments.

Architecture overview

The workbench creates an isolated Apache Kafka testing environment in your AWS account. It deploys a private subnet where consumer and producer applications run as containers, connects to a private MSK Express broker and monitors for performance metrics and visibility. This architecture mirrors the production deployment pattern for experimentation. The following image describes this architecture using AWS services.

This architecture is deployed using the following AWS services:

Amazon Elastic Container Service (Amazon ECS) generate configurable workloads with Java-based producers and consumers, simulating various real-world scenarios through different message sizes and throughput patterns.

Amazon MSK Express Cluster runs Apache Kafka 3.9.0 on Graviton-based instances with hands-free storage management and enhanced performance characteristics.

Dynamic Amazon CloudWatch Dashboards automatically adapt to your configuration, displaying real-time throughput, latency, and resource utilization across different test scenarios.

Secure Amazon Virtual Private Cloud (Amazon VPC) Infrastructure provides private subnets across three Availability Zones with VPC endpoints for secure service communication.

Configuration-driven testing

The workbench provides different configuration options for your Apache Kafka testing environment, so you can customize instance types, broker count, topic distribution, message characteristics, and ingress rate. You can adjust the number of topics, partitions per topic, sender and receiver service instances, and message sizes to match your testing needs. These flexible configurations support two distinct testing approaches to validate different aspects of your Kafka deployment:

Approach 1: Workload validation (single deployment)

Test different workload patterns against the same MSK Express cluster configuration. This is useful for comparing partition strategies, message sizes, and load patterns.

// Fixed MSK Express Cluster Configuration
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // 1 broker per AZ = 3 total brokers
instanceType: 'express.m7g.large', // MSK Express instance type
};

// Multiple Concurrent Workload Tests
export const deploymentConfig: DeploymentConfig = { services: [
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, // High-throughput scenario
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 512 }, // Latency-optimized scenario
{ topics: 3, partitionsPerTopic: 4, instances: 2, messageSizeBytes: 4096 }, // Multi-topic scenario
]};

Approach 2: Infrastructure rightsizing (redeploy and compare)

Test different MSK Express cluster configurations by redeploying the workbench with different broker settings while keeping the same workload. This is recommended for rightsizing experiments and understanding the impact of vertical compared to horizontal scaling.

// Baseline: Deploy and test
export const mskBrokerConfig: MskBrokerConfig = { numberOfBrokers: 1, instanceType: 'express.m7g.large',};

// Vertical scaling: Redeploy with larger instances
export const mskBrokerConfig: MskBrokerConfig = { numberOfBrokers: 1,
instanceType: 'express.m7g.xlarge', // Larger instances
};

// Horizontal scaling: Redeploy with more brokers
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 2, // More brokers
instanceType: 'express.m7g.large',};

Each redeployment uses the same workload configuration, so you can isolate the impact of infrastructure changes on performance.

Workload testing scenarios (single deployment)

These scenarios test different workload patterns against the same MSK Express cluster:

Partition strategy impact testing

Scenario: You are debating the usage of fewer topics with many partitions compared to many topics with fewer partitions for your microservices architecture. You want to understand how partition count affects throughput and consumer group coordination before making this architectural decision.

const deploymentConfig = { services: [
{ topics: 1, partitionsPerTopic: 1, instances: 2, messageSizeBytes: 1024 }, // Baseline: minimal partitions
{ topics: 1, partitionsPerTopic: 10, instances: 2, messageSizeBytes: 1024 }, // Medium partitions
{ topics: 1, partitionsPerTopic: 20, instances: 2, messageSizeBytes: 1024 }, // High partitions
]};

Message size performance analysis

Scenario: Your application handles different types of events – small IoT sensor readings (256 bytes), medium user activity events (1 KB), and large document processing events (8KB). You must understand how message size impacts your overall system performance and if you should separate these into different topics or handle them together.

const deploymentConfig = { services: [
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 256 }, // IoT sensor data
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, // User events
{ topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 8192 }, // Document events
]};

Load testing and scaling validation

Scenario: You expect traffic to vary significantly throughout the day, with peak loads requiring 10× more processing capacity than off-peak hours. You want to validate how your Apache Kafka topics and partitions handle different load levels and understand the performance characteristics before production deployment.

const deploymentConfig = { services: [
{ topics: 2, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, // Off-peak load simulation
{ topics: 2, partitionsPerTopic: 6, instances: 5, messageSizeBytes: 1024 }, // Medium load simulation
{ topics: 2, partitionsPerTopic: 6, instances: 10, messageSizeBytes: 1024 }, // Peak load simulation
]};

Infrastructure rightsizing experiments (redeploy and compare)

These scenarios help you understand the impact of different MSK Express cluster configurations by redeploying the workbench with different broker settings:

MSK broker rightsizing analysis

Scenario: You deploy a cluster with basic configuration and put load on it to establish baseline performance. Then you want to experiment with different broker configurations to see the effect of vertical scaling (larger instances) and horizontal scaling (more brokers) to find the right cost-performance balance for your production deployment.

Step 1: Deploy with baseline configuration

// Initial deployment: Basic configuration
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // 3 total brokers (1 per AZ)
instanceType: 'express.m7g.large',};export const deploymentConfig: DeploymentConfig = { services: [ { topics: 2, partitionsPerTopic: 6, instances: 3, messageSizeBytes: 1024 }, ]};

Step 2: Redeploy with vertical scaling

// Redeploy: Test vertical scaling impact
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 1, // Same broker count
instanceType: 'express.m7g.xlarge', // Larger instances
};

// Keep same workload configuration to compare results

Step 3: Redeploy with horizontal scaling

// Redeploy: Test horizontal scaling impact
export const mskBrokerConfig: MskBrokerConfig = {
numberOfBrokers: 2, // 6 total brokers (2 per AZ)
instanceType: 'express.m7g.large', // Back to original size
};

// Keep same workload configuration to compare results

This rightsizing approach helps you understand how broker configuration changes affect the same workload, so you can improve both performance and cost for your specific requirements.

Performance insights

The workbench provides detailed insights into your Apache Kafka configurations through monitoring and analytics, creating a CloudWatch dashboard that adapts to your configuration. The dashboard starts with a configuration summary showing your MSK Express cluster details and workbench service configurations, helping you to understand what you’re testing. The following image shows the dashboard configuration summary:

The second section of dashboard shows real-time MSK Express cluster metrics including:

Broker performance: CPU utilization and memory usage across brokers in your cluster
Network activity: Monitor bytes in/out and packet counts per broker to understand network utilization patterns
Connection monitoring: Displays active connections and connection patterns to help identify potential bottlenecks
Resource utilization: Broker-level resource tracking provides insights into overall cluster health

The following image shows the MSK cluster monitoring dashboard:

The third section of the dashboard shows the Intelligent Rebalancing and Cluster Capacity insights showing:

Intelligent rebalancing: in progress: Shows whether a rebalancing operation is currently in progress or has occurred in the past. A value of 1 indicates that rebalancing is actively running, while 0 means that the cluster is in a steady state.
Cluster under-provisioned: Indicates whether the cluster has insufficient broker capacity to perform partition rebalancing. A value of 1 means that the cluster is under-provisioned and Intelligent Rebalancing can’t redistribute partitions until more brokers are added or the instance type is upgraded.
Global partition count: Displays the total number of unique partitions across all topics in the cluster, excluding replicas. Use this to track partition growth over time and validate your deployment configuration.
Leader count per broker: Shows the number of leader partitions assigned to each broker. An uneven distribution indicates partition leadership skew, which can lead to hotspots where certain brokers handle disproportionate read/write traffic.
Partition count per broker: Shows the total number of partition replicas hosted on each broker. This metric includes both leader and follower replicas and is key to identifying replica distribution imbalances across the cluster.

The following image shows the Intelligent Rebalancing and Cluster Capacity section of the dashboard:

The fourth section of the dashboard shows the application-level insights showing:

System throughput: Displays the total number of messages per second across services, giving you a complete view of system performance
Service comparisons: Performs side-by-side performance analysis of different configurations to understand which approaches fit
Individual service performance: Each configured service has dedicated throughput tracking widgets for detailed analysis
Latency analysis: The end-to-end message delivery times and latency comparisons across different service configurations
Message size impact: Performance analysis across different payload sizes helps you understand how message size affects overall system behavior

The following image shows the application performance metrics section of the dashboard:

Getting started

This section walks you through setting up and deploying the workbench in your AWS environment. You will configure the necessary prerequisites, deploy the infrastructure using AWS CDK, and customize your first test.

Prerequisites

You can deploy the solution from the GitHub Repo. You can clone it and run it on your AWS environment. To deploy the artifacts, you will require:

AWS account with administrative credentials configured for creating AWS resources.
AWS Command Line Interface (AWS CLI) must be configured with appropriate permissions for AWS resource management.
AWS Cloud Development Kit (AWS CDK) should be installed globally using npm install -g aws-cdk for infrastructure deployment.
Node.js version 20.9 or higher is required, with version 22+ recommended.
Docker engine must be installed and running locally as the CDK builds container images during deployment. Docker daemon should be running and accessible to CDK for building the workbench application containers.

Deployment

# Clone the workbench repository
git clone https://github.com/aws-samples/sample-simulation-workbench-for-msk-express-brokers.git

# Install dependencies and build
npm install 
npm run build

# Bootstrap CDK (first time only per account/region)
cd cdk 
npx cdk bootstrap

# Synthesize CloudFormation template (optional verification step)
npx cdk synth

# Deploy to AWS (creates infrastructure and builds containers)
npx cdk deploy

After deployment is completed, you will receive a CloudWatch dashboard URL to monitor the workbench performance in real-time.You can also deploy multiple isolated instances of the workbench in the same AWS account for different teams, environments, or testing scenarios. Each instance operates independently with its own MSK cluster, ECS services, and CloudWatch dashboards.To deploy additional instances, modify the Environment Configuration in cdk/lib/config.ts:

// Instance 1: Development team
export const AppPrefix = 'mske';export const EnvPrefix = 'dev';

// Instance 2: Staging environment (separate deployment)
export const AppPrefix = 'mske';export const EnvPrefix = 'staging';

// Instance 3: Team-specific testing (separate deployment)
export const AppPrefix = 'team-alpha';export const EnvPrefix = 'test';

Each combination of AppPrefix and EnvPrefix creates completely isolated AWS resources so that multiple teams or environments can use the workbench simultaneously without conflicts.

Customizing your first test

You can edit the configuration file located at folder “cdk/lib/config-types.ts” to define your testing scenarios and run the deployment. It is preconfigured with the following configuration:

export const deploymentConfig: DeploymentConfig = { services: [
// Start with a simple baseline test
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 },

// Add a comparison scenario
{ topics: 1, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, ]};

Best practices

Following a structured approach to benchmarking ensures that your results are reliable and actionable. These best practices will help you isolate performance variables and build a clear understanding of how each configuration change affects your system’s behavior. Begin with single-service configurations to establish baseline performance:

const deploymentConfig = { services: [ { topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 } ]};

After you understand the baseline, add comparison scenarios.

Change one variable at a time

For clear insights, modify only one parameter between services:

const deploymentConfig = { services: [
{ topics: 1, partitionsPerTopic: 3, instances: 1, messageSizeBytes: 1024 }, // Baseline
{ topics: 1, partitionsPerTopic: 6, instances: 1, messageSizeBytes: 1024 }, // More partitions
{ topics: 1, partitionsPerTopic: 12, instances: 1, messageSizeBytes: 1024 }, // Even more partitions
]};

This approach helps you understand the impact of specific configuration changes.

Important considerations and limitations

Before relying on workbench results for production decisions, it is important to understand the tool’s intended scope and boundaries. The following considerations will help you set appropriate expectations and make the most effective use of the workbench in your planning process.

Performance testing disclaimer

The workbench is designed as an educational and sizing estimation tool to help teams prepare for MSK Express production deployments. While it provides valuable insights into performance characteristics:

Results can vary based on your specific use cases, network conditions, and configurations
Use workbench results as guidance for initial sizing and planning
Conduct comprehensive performance validation with your actual workloads in production-like environments before final deployment

Recommended usage approach

Production readiness training – Use the workbench to prepare teams for MSK Express capabilities and operations.

Architecture validation – Test streaming architectures and performance expectations using MSK Express enhanced performance characteristics.

Capacity planning – Use MSK Express streamlined sizing approach (throughput-based rather than storage-based) for initial estimates.

Team preparation – Build confidence and expertise with production Apache Kafka implementations using MSK Express.

Conclusion

In this post, we showed how the workload simulation workbench for Amazon MSK Express Broker supports learning and preparation for production deployments through configurable, hands-on testing and experiments. You can use the workbench to validate configurations, build expertise, and improve performance before production deployment. If you’re preparing for your first Apache Kafka deployment, training a team, or improving existing architectures, the workbench provides practical experience and insights needed for success. Refer to Amazon MSK documentation – Complete MSK Express documentation, best practices, and sizing guidance for more information.

About the authors

Proactive monitoring for Amazon Redshift Serverless using AWS Lambda and Slack alerts

Cristian Restrepo Lopez — Tue, 07 Apr 2026 16:27:14 +0000

Performance issues in analytics environments often remain invisible until they disrupt dashboards, delay ETL jobs, or impact business decisions. For teams running Amazon Redshift Serverless, unmonitored query queues, long-running queries, or unexpected spikes in compute capacity can degrade performance and increase costs if left undetected.

Amazon Redshift Serverless streamlines running analytics at scale by removing the need to provision or manage infrastructure. However, even in a serverless environment, maintaining visibility into performance and usage is essential for efficient operation and predictable costs. While Amazon Redshift Serverless provides advanced built-in dashboards for monitoring performance metrics, delivering notifications directly to platforms like Slack, brings another level of agility. Real-time alerts in the team’s workflow enable faster response times and more informed decision-making without requiring constant dashboard monitoring.

In this post, we show you how to build a serverless, low-cost monitoring solution for Amazon Redshift Serverless that proactively detects performance anomalies and sends actionable alerts directly to your selected Slack channels. This approach helps your analytics team identify and address issues early, often before your users notice a problem.

Solution overview

The solution presented in this post uses AWS services to collect key performance metrics from Amazon Redshift Serverless, evaluate them against thresholds that you can flexibly configure, and notify you when anomalies are detected.

The workflow operates as follows:

Scheduled execution – An Amazon EventBridge rule triggers an AWS Lambda function on a configurable schedule (by default, every 15 minutes during business hours).
Metric collection – The AWS Lambda function gathers metrics including queued queries, running queries, compute capacity (RPUs), data storage usage, table count, database connections, and slow-running queries using Amazon CloudWatch and the Amazon Redshift Data API.
Threshold evaluation – Collected metrics are compared against your predefined thresholds that reflect acceptable performance and usage limits.
Alerting – When a threshold is exceeded, the Lambda function publishes a notification to an Amazon SNS topic.
Slack notification – Amazon Q Developer in Chat applications (formerly AWS Chatbot) delivers the alert to your designated Slack channel.
Observability – Lambda execution logs are stored in Amazon CloudWatch Logs for troubleshooting and auditing.

This architecture is fully serverless and requires no changes to your existing Amazon Redshift Serverless workloads. To simplify deployment, we provide an AWS CloudFormation template that provisions all required resources.

Prerequisites

Before deploying this solution, you must collect information about your existing Amazon Redshift Serverless workgroup and namespace that you want to monitor. To identify your Amazon Redshift Serverless resources:

Open the Amazon Redshift console.
In the navigation pane, choose Serverless dashboard.
Note down your workgroup and namespace names. You will use these values when launching this blog’s AWS CloudFormation template.

Deploy the solution

You can launch the CloudFormation stack and deploy the solution via the provided link.

GitHub Repo

When launching the CloudFormation stack, complete the following steps in the AWS CloudFormation Console:

For Stack name, enter a descriptive name such as redshift-serverless-monitoring.
Review and modify the parameters as needed for your environment.
Acknowledge that AWS CloudFormation may create IAM resources with custom names.
Choose Submit.

CloudFormation parameters

Amazon Redshift Serverless Workgroup configuration

Provide details for your existing Amazon Redshift Serverless environment. These values connect the monitoring solution to your Redshift environment. Some parameters come with the default values that you can replace with your actual configuration.

Parameter	Default value	Description
Amazon Redshift Workgroup Name		Your Amazon Redshift Serverless workgroup name.
Amazon Redshift Namespace Name		Your Amazon Redshift Serverless namespace name.
Amazon Redshift Workgroup ID		Workgroup ID (UUID) of the Amazon Redshift Serverless workgroup to monitor. Must follow the UUID format: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` (lowercase hexadecimal with hyphens).
		Namespace ID (UUID) of the Amazon Redshift Serverless namespace. Must follow the UUID format: `xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` (lowercase hexadecimal with hyphens).
Database Name	`dev`	Target Amazon Redshift database for SQL-based diagnostic and monitoring queries.

Monitoring schedule

The default schedule runs diagnostic SQL queries every 15 minutes during business hours, balancing responsiveness and cost efficiency. Running more frequently might increase costs, while less frequent monitoring could delay detection of performance issues. You can adjust this schedule to your actual need.

Parameter	Default value	Description
Schedule Expression	cron(0/15 8-17 ? * MON-FRI *)	EventBridge schedule expression for Lambda function execution. Default runs every 15 minutes, Monday through Friday, 8 AM to 5 PM UTC.

Threshold configuration

Thresholds should be tuned based on your workload characteristics.

Parameter	Default value	Description
Queries Queued Threshold	20	Alerts threshold for queued queries.
Queries Running Threshold	20	Alerts threshold for running queries.
Compute Capacity Threshold (RPUs)	64	Alert threshold for compute capacity (RPUs).
Data Storage Threshold (MB)	5242880	Threshold for data storage in MB (default 5 TB).
Table Count Threshold (MB)	1000	Alerts threshold for total table count.
Database Connections Threshold	50	Alert threshold for database connections.
Slow Query Threshold (seconds)	10	Thresholds in seconds for slow query detection.
Query Timeout (Seconds)	30	Timeout for SQL diagnostics queries.

Tip: Start with conservative thresholds and refine them after observing baseline behavior for one to two weeks.

Lambda configuration

Configure the AWS Lambda function settings. The selected default values are appropriate for most monitoring scenarios. You may want to change them only in case of troubleshooting.

Parameter	Default value	Description
Lambda Memory Size (MB)	256	Lambda function memory size in MB.
Lambda Time Out (Seconds)	240	Lambda function timeout in seconds.

Security Configuration – Amazon Virtual Private Cloud (VPC)

If your organization has network isolation requirements, you can optionally enable VPC deployment for the Lambda function. When enabled, the Lambda function runs within your specified VPC subnets, providing network isolation and allowing access to VPC-only resources.

Parameter	Default value	Description
VPC ID		VPC ID for Lambda deployment (required if `EnableVPC` is true). The Lambda function will be deployed in this VPC. Ensure that the VPC has appropriate routing (NAT Gateway or VPC Endpoints) to allow Lambda to access AWS services like CloudWatch, Amazon Redshift, and Amazon SNS.
VPC Subnet IDs		Comma-separated list of subnet IDs for Lambda deployment (required if `EnableVPC` is true).
Security Group IDs		Comma-separated list of security group IDs for Lambda (optional). If not provided and `EnableVPC` is true, a default security group will be created with outbound HTTPS access. Custom security groups must allow outbound HTTPS (port 443) to AWS service endpoints.

Note that VPC deployment might increase cold start times and requires an NAT Gateway or VPC endpoints for AWS service access. We recommend provisioning interface VPC endpoints (through AWS PrivateLink) for the five services the Lambda function calls which keeps all traffic private without the recurring cost of a NAT Gateway.

Security configuration – Encryption

If your organization requires encryption of data at rest, you can optionally enable AWS Key Management Service (AWS KMS) encryption for the Lambda function’s environment variables, CloudWatch Logs, and SNS topic. When enabled, the template encrypts each resource using the AWS KMS keys that you provide, either a single shared key for all three services, or individual keys for granular key management and audit separation.

Parameter	Default value	Description
Shared KMS Key ARN		AWS KMS key ARN to use for all encryption (Lambda, Logs, and SNS) unless service-specific keys are provided. This streamlines key management by using a single key for all services. The key policy must grant encrypt/decrypt permissions to Lambda, CloudWatch Logs, and SNS.
Lambda KMS Key ARN		AWS KMS key ARN for Lambda environment variable encryption (optional, overrides `SharedKMSKeyArn`). Use this for separate key management per service. The key policy must grant decrypt permissions to the Lambda execution role. If not provided, `SharedKMSKeyArn` will be used when `EnableKMSEncryption` is true.
CloudWatch Logs KMS Key ARN		AWS KMS key ARN for CloudWatch Logs encryption (optional, overrides `SharedKMSKeyArn`). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to the CloudWatch Logs service. If not provided, `SharedKMSKeyArn` will be used when `EnableKMSEncryption` is true.
SNS Topic KMS Key ARN		AWS KMS key ARN for SNS topic encryption (optional, overrides `SharedKMSKeyArn`). Use this for separate key management per service. The key policy must grant encrypt/decrypt permissions to SNS service and the Lambda execution role. If not provided, `SharedKMSKeyArn` will be used when `EnableKMSEncryption` is true.
Enable Dead Letter Queue	False	Optionally enable Dead Letter Queue (DLQ) for failed Lambda invocations to improve reliability and security monitoring. When enabled, events that fail after all retry attempts will be sent to an SQS queue for investigation and potential replay. This helps prevent data loss, provides visibility into failures, and enables security audit trails for monitoring anomalies. The DLQ retains messages for 14 days.

Note that AWS KMS encryption requires the key policy to grant appropriate permissions to each consuming service (Lambda, CloudWatch Logs, and SNS).

On the review page, select I acknowledge that AWS CloudFormation might create IAM resources with custom names.
Choose Submit.

Resources created

The CloudFormation stack creates the following resources:

EventBridge rule for scheduled execution
AWS Lambda function (Python 3.12 runtime)
Amazon SNS topic for alerts
IAM role with permissions for CloudWatch, Amazon Redshift Data API, and SNS
CloudWatch Log Group for Lambda logs

Note: CloudFormation deployment typically takes 10–15 minutes to complete. You can monitor progress in real time under the Events tab of your CloudFormation stack.

Post-deployment configuration

After the CloudFormation stack has been successfully created, complete the following steps.

Step 1: Record CloudFormation outputs

Navigate to the AWS CloudFormation console.
Select your stack and choose the Outputs tab.
Note the values for LambdaRoleArn and SNSTopicArn. You will need these in subsequent steps.

Step 2: Grant Amazon Redshift permissions

Grant permissions to the Lambda function to query Amazon Redshift system tables for monitoring data. Complete the following steps to grant the necessary access:

Navigate to the Amazon Redshift console.
In the left navigation pane, choose Query Editor V2.
Connect to your Amazon Redshift Serverless workgroup.
Execute the following SQL commands, replacing <IAM Role ARN> with the LambdaRoleArn value from your CloudFormation outputs:

CREATE USER "IAMR:<IAM Lambda Role>" WITH PASSWORD DISABLE;

GRANT ROLE "sys:monitor" TO "IAMR:<IAM Role>";

These commands create an AmazonRedshift user associated with the Lambda IAM role and grant it the sys:monitor Amazon Redshift role. This role provides read-only access to catalog and system tables without granting permissions to user data tables.

Step 3: Configure Slack notifications

Amazon Q Developer in chat applications provides native AWS integration and managed authentication, removing custom webhook code and reducing setup complexity. To receive alerts in Slack, configure Amazon Q Developer in Chat Applications to connect your SNS topic to your preferred Slack channel:

Navigate to Amazon Q Developer in chat applications (formerly AWS Chatbot) in the AWS console.
Follow the instructions in the Slack integration documentation to authorize AWS access to your Slack workspace.
When configuring the Slack channel, ensure that you select the correct AWS Region where you deployed the CloudFormation stack.
In the Notifications section, select the SNS topic created by your CloudFormation stack (refer to the SNSTopicArn output value).
Keep the default IAM read-only permissions for the channel configuration.

After configured, alerts automatically appear in Slack whenever thresholds are exceeded.

Cost considerations

With the default configuration, this solution incurs minimal ongoing costs. The Lambda function executes approximately 693 times per month (every 15 minutes during an 8-hour business day, Monday through Friday), resulting in a monthly cost of approximately $0.33 USD. This includes Lambda compute costs ($0.26) and CloudWatch GetMetricData API calls ($0.07). All other services (EventBridge, SNS, CloudWatch Logs, and Amazon Redshift Data API). The Amazon Redshift Data API has no additional charges beyond the minimal Amazon Redshift Serverless RPU consumption for the Amazon Redshift Serverless system table query execution. You can reduce costs by decreasing the monitoring frequency (such as, every 30 minutes) or increase responsiveness by running more frequently (such as, every 5 minutes) with a proportional cost increase.

All costs are estimates and may vary based on your environment. Variations often occur because queries scanning system tables may take longer or require additional resources depending on the system complexity

Security best practices

This solution implements the following security controls:

IAM policies scoped to specific resource ARNs for the Amazon Redshift workgroup, namespace, SNS topic, and log group.
Data API statement access restricted to the Lambda function’s own IAM user ID.
Read-only sys:monitor database role for operational metadata access. Limit to the role created by the CloudFormation template.
Reserved concurrent executions capped at five.

To further strengthen your security posture, consider the following enhancements:

Enable EnableKMSEncryption to encrypt environment variables, logs, and SNS messages at rest.
Enable EnableVPC to deploy the function within a VPC for network isolation.
Audit access through AWS CloudTrail.

Important: This is sample code for non-production usage. Work with your security and legal teams to meet your organizational security, regulatory, and compliance requirements before deployment. This solution demonstrates monitoring capabilities but requires additional security hardening for production environments, including encryption configuration, IAM policy scoping, VPC deployment, and comprehensive testing.

Clean up

To remove all resources and avoid ongoing charges if you don’t want to use the solution anymore:

Delete the CloudFormation stack.
Remove the Slack integration from Amazon Q Developer in chat applications.

Troubleshooting

If no metrics or incomplete SQL diagnostics are returned, verify that the Amazon Redshift Serverless workgroup is active with recent query activity, and ensure the database user has the sys:monitor role (GRANT ROLE sys:monitor TO <user>) in the query editor. Without this role, queries execute successfully but only return data visible to that user’s permissions rather than the full cluster activity.
For VPC-deployed functions that fail to reach AWS services, confirm that VPC endpoints or a NAT Gateway are configured for CloudWatch, Amazon Redshift Data API, Amazon Redshift Serverless, SNS, and CloudWatch Logs.
If the Lambda function times out, increase the LambdaTimeout and QueryTimeoutSeconds parameters. The default timeout of 240 seconds accommodates most workloads, but clusters with many active queries may require additional time for SQL diagnostics to complete.

Conclusion

In this post, we showed how you can build a proactive monitoring solution for Amazon Redshift Serverless using AWS Lambda, Amazon CloudWatch, and Amazon SNS with Slack integration. By automatically collecting metrics, evaluating thresholds, and delivering alerts in near real time to Slack or your preferred collaborative platform, this solution helps detect performance and cost issues early. Because the solution itself is serverless, it aligns with the operational simplicity goals of Amazon Redshift Serverless—scaling automatically, requiring minimal maintenance, and delivering high value at low cost. You can extend this foundation with additional metrics, diagnostic logic, or alternative notification channels to meet your organization’s needs.

To learn more, see the Amazon Redshift documentation on monitoring and performance optimization.

About the authors

Modernize business intelligence workloads using Amazon Quick

Satesh Sonti — Mon, 06 Apr 2026 17:56:35 +0000

Traditional business intelligence (BI) integration with enterprise data warehouses has been the established pattern for years. With generative AI, you can now modernize BI workloads with capabilities like interactive chat agents, automated business processes, and using natural language to generate dashboards.

In this post, we provide implementation guidance for building integrated analytics solutions that combine the generative BI features of Amazon Quick with Amazon Redshift and Amazon Athena SQL analytics capabilities. Use this post as a reference for proof-of-concept implementations, production deployment planning, or as a learning resource for understanding Quick integration patterns with Amazon Redshift and Athena.

Common use cases

You can use this integrated approach across several scenarios. The following are some of the most common use cases.

Traditional BI reporting benefits from bundled data warehouse and BI tool pricing, making generative BI the primary use case with significant cost advantages.
- Insurance: Automates Solvency II and IFRS 17 regulatory reporting, replacing manual spreadsheet consolidation.
- Banking: Accelerates FDIC call report generation and capital adequacy dashboards, cutting month-end close from days to hours.
Interactive dashboards with contextual chat agents give BI teams conversational interfaces alongside their visual metrics.
- Gaming: Live ops teams query player retention and monetization KPIs in plain English—no SQL needed.
- Financial Services: Trading analysts chat with real-time P&L dashboards to surface anomalies and drill into positions on demand.
Domain-specific analytics workspaces democratize enterprise data exploration through Quick Spaces and natural language queries.
- Insurance: Actuarial and underwriting teams query claims and risk data without waiting on data engineering.
- Banking: Risk and compliance teams explore credit, market, and operational data through a single natural language interface.
Workflow automation removes repetitive tasks and accelerates self-service analytics.
- Financial Services: Automated AR reconciliation flows replace manual ledger matching, shrinking close cycle effort significantly.
- Gaming: Telemetry ingestion pipelines trigger reporting refreshes automatically, freeing data engineers from routine work.

Let us examine an end-to-end solution combining these technologies.

Solution flow

AWS offers two native SQL analytics engines for building analytics workloads. Amazon Redshift provides a fully managed data warehouse with columnar storage and massively parallel processing. Amazon Athena delivers serverless interactive query capabilities directly against data in Amazon S3.

You can use either Amazon Redshift or Amazon Athena as a SQL engine while implementing the steps in this post. The following are the steps involved in building an end-to-end solution.

Figure1: Solution steps to integrate SQL Analytics engines with Amazon Quick

Set up your SQL analytics engines: Amazon Redshift or Amazon Athena.
Load data and create business views designed for analytics workloads.
Configure integration between SQL analytics engines and Amazon Quick.
Create data sources in Amazon Quick.
Create datasets and dashboards for visual analytics.
Use Topics and Spaces to provide natural language interfaces to your data.
Deploy chat agents to deliver conversational AI experiences for business users.
Implement business flows to automate repetitive workflows and processes.

Let’s start by walking through steps 1–4 for Amazon Redshift. We then describe the same four steps for Amazon Athena before explaining the Amazon Quick steps 5–8.

Configure and create datasets in Amazon Redshift

Amazon Redshift offers two deployment options to meet your data warehousing needs. Provisioned clusters provide traditional deployment where you manage compute resources by selecting node types and cluster size. Serverless automatically scales compute capacity based on workload demands with pay-per-use pricing. Both options are supported by Amazon Quick. For this walkthrough, we use Redshift Serverless.

Set up SQL analytics engine

To create a Redshift Serverless namespace and workgroup:

Open the Amazon Redshift console.
On the left navigation pane, select Redshift Serverless.
Follow the steps described in the Creating a workgroup with a namespace documentation page to create a workgroup and a namespace. Note the username and password provided. You will use these details for configuring connections in Amazon Redshift and Quick.
You should see the status as Available for both the workgroup and namespace in the Serverless dashboard.

Figure 2: Amazon Redshift Serverless Workgroup and NamespacesThe deployment will be completed in approximately 3–5 minutes.

Load data and create business views

Now you can load data using the industry-standard TPC-H benchmark dataset, which provides realistic customer, order, and product data for analytics workloads.To load data into Amazon Redshift:

Open the Amazon Redshift Query Editor V2 from the console.
Run the TPC H DDL statements to create TPC-H tables.
Run the following COPY commands to load data from the public S3 bucket: s3://redshift-downloads/TPC-H/.

Ensure that the IAM role attached to the namespace is set as the default IAM role. If you didn’t set up the default IAM role at the time of namespace creation, you can refer to the Creating an IAM role as default for Amazon Redshift documentation page to set it now.

copy customer from 's3://redshift-downloads/TPC-H/2.18/100GB/customer/' iam_role default delimiter '|' region 'us-east-1'; 

copy orders from 's3://redshift-downloads/TPC-H/2.18/100GB/orders/' iam_role default delimiter '|' region 'us-east-1'; 

copy lineitem from 's3://redshift-downloads/TPC-H/2.18/100GB/lineitem/' iam_role default delimiter '|' region 'us-east-1';

Run the following query to validate load status. The status column should show as completed. You can also review the information in other columns to see details about the loads such as record counts, duration, and data source.

select * from  SYS_LOAD_HISTORY  Where table_name in ('customer','orders','lineitem');

Figure 3: Output of SYS_LOAD_HISTORY showing successful completion of COPY Jobs

Create a materialized view to improve query performance:

Run the following SQL to create a materialized view that pre-compute results set for customer revenues and order volumes by market segment.

CREATE MATERIALIZED VIEW mv_customer_revenue AS 
SELECT 
c.c_custkey, 
c.c_name, 
c.c_mktsegment, 
SUM(l.l_extendedprice * (1 - l.l_discount)) as total_revenue, 
COUNT(DISTINCT o.o_orderkey) as order_count 
FROM customer c 
JOIN orders o ON c.c_custkey = o.o_custkey
JOIN lineitem l ON o.o_orderkey = l.l_orderkey
GROUP BY c.c_custkey, c.c_name, c.c_mktsegment;

Run the following SQL to review the data in the materialized view.

select * from mv_customer_revenue limit 10;

Configure integration with Amazon Quick

Amazon Quick auto discovers the Amazon Redshift provisioned clusters that are associated with your AWS account. These resources must be in the same AWS Region as your Amazon Quick account. For Amazon Redshift clusters in other accounts or Amazon Redshift Serverless, we recommend that you add a VPC connection following the steps in Enabling access to an Amazon Redshift cluster in a VPC documentation. Usually, these steps are performed by your organization’s cloud security administration team.

For serverless, you will apply the same steps in the workgroup instead of the cluster. You can find the VPC and Security Group settings in the Data Access tab of a workgroup.

Figure 4: Amazon Redshift Serverless workgroup VPC and Security groups

You can also refer to How do I privately connect Quick to an Amazon Redshift or RDS data source in a private subnet? for a demonstration.

Create data source

To create a dataset connecting to Amazon Redshift, complete the following steps.

In the Quick left navigation pane, go to Datasets.
Choose the Data sources tab and select Create data source.
Select Amazon Redshift and enter the following:
- Data Source Name: Provide customer-rev-datasource as data source name.
- Connection type: Select the VPC connection created in the previous step.
- Database server: Enter the Amazon Redshift workgroup endpoint (for example, quick-demo-wg.123456789.us-west-2.redshift-serverless.amazonaws.com).
- Port: 5439 (default).
- Database: dev.
- Username/Password: Amazon Redshift credentials with access to the database.
Choose Validate connection. The validation should be successful.

Figure 5: Amazon Redshift data source configuration

Choose Create Data Source to create a data source.

Now let’s explore how to perform all these four steps to configure Athena in Amazon Quick.

Configure and create datasets in Amazon Athena

Amazon Athena provides immediate query capabilities against petabytes of data with automatic scaling to handle concurrent users. Let’s go through the steps to configure connections between Amazon Quick and Amazon Athena.

Set up SQL analytics engine

To create an Athena workgroup:

Open the Amazon Athena console.
In the navigation pane, choose Workgroups.
Choose Create workgroup.
For Workgroup name, enter quick-demo.
For Query result configuration, select Athena managed.
Choose Create workgroup.

Your workgroup is ready immediately for querying data.

Load data and create business views

For Athena, you create tables using the TPC-H benchmark dataset that AWS provides in a public S3 bucket. This approach gives you 1.5 million customer records already optimized in Parquet format without requiring data loading.

To create tables and views in Athena:

Open the Athena Query Editor from the console.

Create a database for your analytics (create S3 bucket if it exists already):

CREATE DATABASE IF NOT EXISTS athena_demo_db 
COMMENT 'Analytics database for customer insights' 
LOCATION 's3://my-analytics-data-lake-[account-id]/';

Create an external table pointing to the TPC-H public dataset:

CREATE EXTERNAL TABLE IF NOT EXISTS athena_demo_db.customer_csv ( 
  C_CUSTKEY INT, 
  C_NAME STRING, 
  C_ADDRESS STRING, 
  C_NATIONKEY INT, 
  C_PHONE STRING, 
  C_ACCTBAL DOUBLE, 
  C_MKTSEGMENT STRING, 
  C_COMMENT STRING 
) 

ROW FORMAT DELIMITED 
FIELDS TERMINATED BY '|' 
STORED AS TEXTFILE 
LOCATION 's3://redshift-downloads/TPC-H/2.18/100GB/customer/'

Create a business-friendly view for analytics:

Run the following SQL to create a view that aggregates customer account balances grouped by market segments.

CREATE VIEW athena_demo_db.customer_deep_analysis AS 
SELECT 
    c_custkey AS customer_id, 
    c_name AS customer_name, 
    c_mktsegment AS market_segment, 
    c_nationkey, 
    ROUND(c_acctbal, 2) AS account_balance, 
    CASE 
        WHEN c_acctbal < 0    THEN 'At-Risk' 
        WHEN c_acctbal < 2500 THEN 'Low' 
        WHEN c_acctbal < 5000 THEN 'Mid' 
        WHEN c_acctbal < 8000 THEN 'High' 
        ELSE 'Premium' 
    END                                                              
AS balance_tier, 

    ROUND(AVG(c_acctbal) OVER (PARTITION BY c_mktsegment), 2)        AS segment_avg, 
    ROUND(c_acctbal - AVG(c_acctbal) OVER (PARTITION BY c_mktsegment), 2) AS vs_segment_avg, 
    ROUND((c_acctbal - AVG(c_acctbal) OVER (PARTITION BY c_mktsegment)) 
          / NULLIF(STDDEV(c_acctbal) OVER (PARTITION BY c_mktsegment), 0), 2) AS segment_z_score, 
    RANK() OVER (PARTITION BY c_mktsegment ORDER BY c_acctbal DESC)  AS rank_in_segment, 
    NTILE(5) OVER (ORDER BY c_acctbal DESC)                          AS global_quintile 

FROM athena_demo_db.customer_csv 
ORDER BY c_acctbal DESC;

Verify your view from Athena with:

SELECT * FROM athena_demo_db.customer_deep_analysis limit 5;

Figure 6: Output from the SELECT query

Configure integration with Amazon Quick

To connect to Amazon Athena in Amazon Quick, follow these steps, consolidated from official AWS documentation and authorizing connections to Amazon Athena.

Authorize Quick to Access Athena, S3 Bucket for data, and S3 bucket for Athena Results.

Open the Amazon Quick Security Settings

Sign in to the Amazon Quick console as an administrator.
In the top-right corner, choose your profile icon, then select Manage account.
Under Permissions, choose AWS resources.
Figure 7: AWS resource permissions

Enable Athena Access

Under Quick access to AWS services, choose Manage.
Locate Amazon Athena in the list of AWS services.
If Athena is already selected but access issues persist, clear the checkbox and re-select it to re-enable Athena.
Under Amazon S3, select S3 buckets.
Check the boxes next to each S3 bucket that Amazon Quick needs to access—including buckets used for Athena query results and any Redshift COPY source buckets.
Enable Write permission for Athena Workgroup to allow Amazon Quick to write Athena query results to S3 and choose Finish.
Choose Save to update the configuration.

The final step is to grant your Amazon Quick author permissions to query your database, Athena tables, and views. Configuration depends on whether AWS Lake Formation is enabled.

If AWS Lake Formation is not enabled

Permissions are managed at the Quick service role level through standard IAM-based S3 access control. Ensure that the Quick service role (for example, aws-quick-service-role-v0) has the appropriate IAM permissions for the relevant S3 buckets and Athena resources. No additional Lake Formation configuration is required.

If AWS Lake Formation is enabled

Lake Formation acts as the central authorization layer, overriding standard IAM-based S3 permissions. Grant permissions directly to the Amazon Quick author or IAM role.

To grant data permissions:

Open the AWS Lake Formation console.
Choose Permissions, then Data permissions, then Grant.
Select the IAM user or role.
Choose the required databases, tables, and columns.
Grant SELECT at minimum; add DESCRIBE for dataset creation.
Repeat for each user or role that requires access.

Create data source

Follow these steps to create an Athena data source on Amazon Quick.

In the Amazon Quick console, navigate to Datasets and choose Data sources tab.
Choose Create data source, then select the Amazon Athena card.
Enter a Data source name (you can give any name of your choice), select your Athena workgroup (like quick-demo), and choose Validate connection.

Figure 8: Athena data source creation

Choose Create data source.

Your Athena data source is now available for building datasets, dashboards, and Topics.

Use Amazon Quick generative AI features

The next steps, from 5–8, demonstrate Amazon Quick generative AI capabilities using Amazon Redshift as a data source. While we use Amazon Redshift in this example, you can substitute with Amazon Athena based on your specific requirements.

Create dashboards

Let’s start by creating datasets from the Amazon Redshift data source.

In the left navigation pane, choose Datasets.
On the Datasets page, choose Create Dataset.
For the data source, select Amazon Redshift data source customer-rev-datasource.
From the menu, choose mv_customer_revenue.

Figure 9: Select table to visualize

You can choose one of the following query modes. For this post, select Directly query your data option and choose Visualize.
- Import to SPICE for quicker analytics – Quick loads a snapshot into its in-memory engine for faster dashboard performance.
- Directly query your data– Quick runs queries on demand against your query engine.
Select Build icon to open a chat window. Enter “Show me orders by market segments” as the prompt. Note that you need Author Pro access to use this feature.

Figure 10: Build visualization using generative BI feature

You can change the visual type to a pie chart and add it to the analysis.

Figure 10: Change visual type

To publish your analysis as a dashboard

After you add the visuals, choose Publish.
Enter a name for the dashboard. For this post, use the Market Segment Dashboard.
Choose Publish dashboard. Your dashboard is now available for viewing and sharing.

Create topics and spaces

To fully maximize enterprise data with AI, we must provide the right structure and context. That’s where Topics and Spaces come in. Topics act as natural language interfaces to your structured datasets, automatically analyzing your data, mapping fields, and adding synonyms. Business users can ask “What are total revenues by market segment?” and receive instant, visualized answers without writing a single line of SQL. Spaces bring together all of your related assets into a single collaborative workspace that democratizes data access, reduces context-switching, accelerates team onboarding, so everyone is working from the same trusted, AI-ready data sources.

To create a Quick topic

From the Amazon Quick homepage, choose Topics, then choose Create topic.
Enter a name for your topic. For this post, use Customer Revenue Analytics.
Enter a description. For example:

The Customer Revenue Analytics topic is designed for business users (including analysts, sales operations teams, finance, and market segment owners who need to explore customer and revenue data without SQL expertise. It serves as a natural language interface over the mv_customer_revenue Amazon Redshift dataset, allowing users to ask plain-English questions like “What are total revenues by market segment?” and receive instant, visualized answers. By automatically mapping business language to the underlying schema, it democratizes access to revenue insights across the organization.

Under Dataset, select mv_customer_revenue.
Choose Create. The topic can take 15–30 minutes to enable depending on the data. During this time, Amazon Quick automatically analyzes your data, selects relevant fields, and adds synonyms.
After the topic is enabled, take a few minutes to review and enrich it. The following are some example enrichments.
1. Add column descriptions to clarify field meaning for business users.
2. Define preferred aggregations (for example, sum compared to average for revenue fields).
3. Confirm which fields are Dimensions and which are Measures.
(Optional) To further refine how your topic interprets and responds to queries, add multiple datasets (for example, a customer CSV combined with a database view), custom instructions, filters, and calculated fields.

After your topic is created, its columns are available to add to a Space or to an Agent by selecting it as a data source.

Figure 11: Create a Quick Topic

Create a Space for your team

Spaces bring together dashboards, topics, datasets, documents, and other resources into organized, collaborative workspaces. By centralizing related assets in a single workspace, Spaces reduce context-switching, accelerate onboarding, so everyone is working from the same trusted data sources.

What to include in your Quick Space

Dashboard – Add the dashboard Market Segment Dashboard published from your mv_customer_revenue analysis. This gives team members instant access to visualizations such as revenue by market segment, top customers by order volume, and revenue distribution.
Topic – Connect the Customer Revenue Analytics (built on the mv_customer_revenue materialized view) to enable natural language queries directly against your Amazon Redshift data.
Optionally, you can upload supporting context to ground your team’s analysis:
- Data dictionary or field definitions for mv_customer_revenue
- Market segment definitions (AUTOMOBILE, BUILDING, FURNITURE, MACHINERY, HOUSEHOLD)
- Business rules for revenue calculation (for example, how discounts are applied in the TPC-H model)
- This implementation guide, so new team members can onboard quickly

To create the Quick Space

From the left navigation menu, choose Spaces, then choose Create space.
Enter a name, for example, Customer Revenue & Segmentation.
Enter a description. For example:

Centralized workspace for customer revenue analysis powered by Amazon Redshift includes interactive dashboards, natural language query access to customer and segment data, and supports documentation for the TPC-H revenue model.

Add knowledge by connecting the Market Segment Dashboard and topic Customer Revenue Analytics.
You can invite team members, such as finance, sales operations, and segment owners, and set appropriate permissions.

Your Space is now ready for collaborative data exploration.

Figure 12: Create a Quick Space

Build chat agents

A custom chat agent delivers conversational AI experiences that understand business context and provide intelligent, grounded responses to user queries. These agents go beyond question-and-answer interactions. They synthesize knowledge from your dashboards, topics, datasets, and documents to explain trends, surface anomalies, guide users through complex analytics workflows, and recommend next steps.

Rather than requiring users to navigate multiple tools or write SQL queries, agents serve as a single conversational interface to your entire analytics environment. Agents can also connect to Actions, pre-built integrations with enterprise tools such as Slack, Microsoft Teams, Outlook, and SharePoint, enabling them to answer questions and trigger real-world workflows, send notifications, create tasks, and interact with external systems directly from the conversation. Custom agents can be tailored to specific business domains, teams, or use cases so that responses align with organizational terminology, data definitions, and business processes. After created, agents can be shared across teams, enabling consistent, actionable, AI-powered data access at scale. For teams working with the mv_customer_revenue dataset, we recommend creating a dedicated Customer Revenue Analysis Agent. This is a purpose-built conversational assistant grounded in your Amazon Redshift data, dashboards, and the Customer Revenue & Segmentation Space.

Create a Quick chat agent

There are two ways that you can use Amazon Quick to create a Quick agent. You can use the navigation menu or directly from Space. The following steps walk you through creating one from the navigation menu.

To create a Quick chat agent

From the left navigation menu, choose Agents, then choose Create agent.
Enter a name for your agent, for example, Customer Revenue Analyst.
Enter a description. For example:

An AI assistant for analyzing customer revenue, market segment performance, and order trends using our Amazon Redshift or data warehouse.

Under Knowledge Sources, add the Customer Revenue & Segmentation Space as a data source. This gives your agent access to the dashboards, topics, and reference documents you’ve already built.
(Optional) Define custom persona instructions to align the agent’s responses with your business context. For example, specifying preferred terminology, response style, or the types of questions it should prioritize.
Choose Launch chat agent.
Start having a conversation with your data. You are welcome to ask any questions. The following are some examples.
- Which market segment generated most revenue?
- Show me order trends

Figure 13: Create a Quick Chat agent

To share your Quick chat agent

After your agent is published, choose Share and invite team members or share it across your organization. Custom agents can be tailored to specific business contexts so that different teams can get AI assistance that speaks their language, without needing to configure anything themselves.

Create Quick Flows

Quick Flows automate repetitive tasks and orchestrate multi-step workflows across your entire analytics environment. This removes manual effort, reducing human error, and ensuring consistent execution of critical business processes. Flows can be triggered on a schedule or launched on demand, giving you flexible control over when and how automation runs.

You can build flows that span the full analytics lifecycle: monitoring data quality and flagging anomalies, generating and distributing scheduled reports to stakeholders, and triggering downstream actions in integrated systems such as Slack, Outlook. Amazon Quick gives you three ways to create a flow, so whether you prefer a no-code conversation or a visual step-by-step builder, there’s an option that fits how you work.

To create a flow from chat

While conversing with My Assistant or a custom agent, describe the workflow that you want to automate in plain English.
Amazon Quick generates the flow and offers to create it directly from your conversation — no configuration screens required.

To create a flow from a natural language description

From the left navigation menu, choose Flows, then choose Create flow.
Enter a plain-English description of your workflow. For example:

” Query revenue data by market segments. Filter by order count and all dates. Search web for comparable relevant market trends. Generate formatted summary reports providing market summary and look ahead per segment. ”

Amazon Quick automatically generates the complete workflow with all the necessary steps.
Optionally, you can add additional steps.
Choose Run Mode to test the Flow.
After your flow is created, share it with team members or publish it to your organization’s flow library, so everyone benefits from the same automation without having to rebuild it independently.

Figure 14: Create a Quick Flow to generate summaries and publish dashboards

For more complex flow, review weekly customer revenue summary flow as an example.

Queries the mv_customer_revenue materialized view in Amazon Redshift for the latest weekly revenue figures by market segment.
Compares results against the prior week to calculate segment-level variance.
Generates a formatted summary report and publishes it to the Customer Revenue & Segmentation Space.
Sends a notification through email or Slack to finance, sales operations, and segment owners with a direct link to the updated dashboard.
Flags any segment where revenue has declined more than a defined threshold, routing an alert to the appropriate owner for follow-up.

This flow transforms what might otherwise be a manual, multi-step reporting process into a fully automated pipeline, so stakeholders receive consistent, timely revenue insights without analyst intervention and saving analysts an estimated 3–5 hours per week. For detailed guidance on creating and managing flows, see Using Amazon Quick Flows. Also review Create workflows for routine tasks demo.

Cleanup

Consider deleting the following resources created while following this post to avoid incurring costs. We encourage you to use the trials at no cost as much as possible to familiarize yourself with the features described.

Delete the Amazon Redshift Serverless workgroup and namespace.
Delete Athena workgroup and S3 Buckets.
Delete the Amazon Quick account used while following this post. If you used an existing account, delete the data sets, dashboards, topics, spaces, agents and flows created.

Conclusion

This integrated approach to business intelligence combines the power of AWS SQL analytics engines with Amazon Quick generative AI capabilities to deliver comprehensive analytics solutions. By following these implementation steps, you establish a foundation for traditional BI reporting, interactive dashboards, natural language data exploration, and intelligent workflow automation. The architecture scales from proof-of-concept implementations to production deployments, transforming how organizations access and act on data insights. For more information about Amazon Quick features and capabilities, see the Amazon Quick documentation. To learn more about Amazon Redshift, visit the Amazon Redshift product page. For Amazon Athena details, see the Amazon Athena product page.

About the authors

Agentic AI for observability and troubleshooting with Amazon OpenSearch Service

Muthu Pitchaimani — Thu, 02 Apr 2026 21:44:17 +0000

Amazon OpenSearch Service powers observability workflows for organizations, giving their Site Reliability Engineering (SRE) and DevOps teams a single pane of glass to aggregate and analyze telemetry data. During incidents, correlating signals and identifying root causes demand deep expertise in log analytics and hours of manual work. Identifying the root cause remains largely manual. For many teams, this is the bottleneck that delays service recovery and burns engineering resources.

We recently showed how to build an Observability Agent using Amazon OpenSearch Service and Amazon Bedrock to reduce Mean time to Resolution (MTTR). Now, Amazon OpenSearch Service brings many of these functions to the OpenSearch UI—no additional infrastructure required. Three new agentic AI features are offered to streamline and accelerate MTTR:

An Agentic Chatbot that can access the context and the underlying data that you’re looking at, apply agentic reasoning, and use tools to query data and generate insights on your behalf.
An Investigation Agent that deep-dives across signal data with hypothesis-driven analysis, explaining its reasoning at every step.
An Agentic Memory that supports both agents, so their accuracy and speed improve the more you use them.

In this post, we show how these capabilities work together to help engineers go from alert to root cause in minutes. We also walk through a sample scenario where the Investigation Agent automatically correlates data across multiple indices to surface a root cause hypothesis.

How the agentic AI capabilities work together

These AI capabilities are accessible from OpenSearch UI through an Ask AI button, as shown in the following diagram, which gives an entry point for the Agentic Chatbot.

Agentic Chatbot

To open the chatbot interface, choose Ask AI.

The chatbot understands the context of the current page, so it understands what you’re looking at before you ask a question. You can ask questions about your data, initiate an investigation, or ask the chatbot to explain a concept. After it understands your request, the chatbot plans and uses tools to access data, including generating and running queries in the Discover page, and applies reasoning to produce a data-driven answer. You can also use the chatbot in the Dashboard page, initiating conversations from a particular visualization to get a summary as shown in the following image.

Investigation agent

Many incidents are too complex to resolve with one or two queries. Now you can get the help of the investigation agent to handle these complex situations. The investigation agent uses the plan-execute-reflect agent, which is designed for solving complex tasks that require iterative reasoning and step-by-step execution. It uses a Large Language Model (LLM) as a planner and another LLM as an executor. When an engineer identifies a suspicious observation, like an error rate spike or a latency anomaly, they can ask the investigation agent to investigate. One of the important steps the investigation agent performs is re-evaluation. The agent, after executing each step, reevaluates the plan using the planner and the intermediate results. The planner can adjust the plan if necessary or skip a step or dynamically add steps based on this new information. Using the planner, the agent generates a root cause analysis report led by the most likely hypothesis and recommendations, with full agent traces showing every reasoning step, all findings, and how they support the final hypotheses. You can provide feedback, add your own findings, iterate on the investigation goal, and review and validate each step of the agent’s reasoning. This approach mirrors how experienced incident responders work, but completes automatically in minutes. You can also use the “/investigate” slash command to initiate an investigation directly from the chatbot, building on an ongoing conversation or starting with a different investigation goal.

Agent in action

Automatic query generation

Consider a situation where you’re an SRE or DevOps engineer and received an alert that a key service is experiencing elevated latency. You log in to the OpenSearch UI, navigate to the Discover page, and select the Ask AI button. Without any expertise in the Piped Processing Language (PPL) query language, you enter the question “find all requests with latency greater than 10 seconds”. The chatbot understands the context and the data that you’re looking at, thinks through the request, generates the right PPL command, and updates it in the query bar to get you the results. And if the query runs into any errors, the chatbot can learn about the error, self-correct, and iterate on the query to get the results for you.

Investigation and investigation management

For complex incidents that normally require manually analyzing and correlating multiple logs for the possible root cause, you can choose Start Investigation to initiate the investigation agent. You can provide a goal for the investigation, along with any context or hypothesis that you want to instruct the investigation. For example, “identify the root cause of widespread high latency across services. Use TraceIDs from slow spans to correlate with detailed log entries in the related log indices. Analyze affected services, operations, error patterns, and any infrastructure or application-level bottlenecks without sampling”.

The agent, as part of the conversation, will offer to investigate any issue that you’re trying to debug.

The agent sets goals for itself along with any other relevant information like indices, associated time range, and other, and asks for your confirmation before creating a Notebook for this investigation. A Notebook is a way within the OpenSearch UI to develop a rich report that’s live and collaborative. This helps with the management of the investigation and allows for reinvestigation at a later date if necessary.

After the investigation starts, the agent will perform a quick analysis by log sequence and data distribution to surface outliers. Then, it will plan for the investigation into a series of actions, and then performs each action, such as query for a specific log type and time range. It will reflect on the results at every step, and iterate on the plan until it reaches the most likely hypotheses. Intermediate results will appear on the same page as the agent works so that you can follow the reasoning in real time. For example, you find that the Investigation Agent accurately mapped out the service topology and used it as a key intermediary steps for the investigation.

As the investigation completes, the investigation agent concludes that the most likely hypothesis is a fraud detection timeout. The associated finding shows a log entry from the payment service: “currency amount is too big, waiting for fraud detection”. This matches a known system design where large transactions trigger a fraud detection call that blocks the request until the transaction is scored and assessed. The agent arrived at this finding by correlating data across two separate indices, a metrics index where the original duration data lived, and a correlated log index where the payment service entries were stored. The agent linked these indices using trace IDs, connecting the latency measurement to the specific log entry that explained it.

After reviewing the hypothesis and the supporting evidence, you find the result reasonable and aligns with your domain knowledge and past experiences with similar issues. You can now accept the hypothesis and review the request flow topology for the affected traces that were provided as part of the hypothesis investigation.

Alternatively, if you find that the initial hypothesis wasn’t helpful, you can review the alternative hypothesis at the bottom of the report and select any of the alternative hypotheses if there’s one that’s more accurate. You can also trigger a re-investigation with additional inputs, or corrections from previous input so that the Investigation Agent can rework it.

Getting started

You can use any of the new agentic AI features (limits apply) in the OpenSearch UI at no cost. You will find the new agentic AI features ready to use in your OpenSearch UI applications, unless you have previously disabled AI features in any OpenSearch Service domains in your account. To enable or disable the AI features, you can navigate to the details page of the OpenSearch UI application in AWS Management Console and update the AI settings from there. Alternatively, you can also use the registerCapability API to enable the AI features or use the deregisterCapability API to disable them. Learn more at Agentic AI in Amazon OpenSearch Services.

The agentic AI feature uses the identity and permissions of the logged in users for authorizing access to the connected data sources. Make sure that your users have the necessary permissions to access the data sources. For more information, see Getting Started with OpenSearch UI.

The investigation results are saved in the metadata system of OpenSearch UI and encrypted with a service managed key. Optionally, you can configure a customer managed key to encrypt all of the metadata with your own key. For more information, see Encryption and Customer Managed Key with OpenSearch UI.

The AI features are powered by Claude Sonnet 4.6 model in Amazon Bedrock. Learn more at Amazon Bedrock Data Protection.

Conclusion

The new agentic AI capabilities announced for Amazon OpenSearch Service help reduce Mean Time to Resolution by providing context-aware agentic chatbot for assistance, hypothesis-driven investigations with full explainability, and agentic memory for context consistency. With the new agentic AI capabilities, your engineering team can spend less time writing queries and correlating signals, and more time acting on confirmed root causes. We invite you to explore these capabilities and experiment with your applications today.

About the authors

Streamline Apache Kafka topic management with Amazon MSK

Swapna Bandla — Thu, 02 Apr 2026 15:32:26 +0000

If you manage Apache Kafka today, you know the effort required to manage topics. Whether you use infrastructure as code (IaC) solutions or perform operations with admin clients, setting up topic management takes valuable time that could be spent on building streaming applications.

Amazon Managed Streaming for Apache Kafka (Amazon MSK) now streamlines topic management by supporting new topic APIs and console integration. You can programmatically create, update, and delete Apache Kafka topics using familiar interfaces including AWS Command Line Interface (AWS CLI), AWS SDKs, and AWS CloudFormation. With these APIs, you can define topic properties such as replication factor and partition count and configuration settings like retention and cleanup policies. The Amazon MSK console integrates these APIs, bringing all topic operations to one place. You can now create or update topics with a few selections using guided defaults while gaining comprehensive visibility into topic configurations, partition-level information, and metrics. You can browse for topics within a cluster, review replication settings and partition counts, and go into individual topics to examine detailed configuration, partition-level information, and metrics. A unified dashboard consolidates partition topics and metrics in one view.

In this post, we show you how to use the new topic management capabilities of Amazon MSK to streamline your Apache Kafka operations. We demonstrate how to manage topics through the console, control access with AWS Identity and Access Management (IAM), and bring topic provisioning into your continuous integration and continuous delivery (CI/CD) pipelines.

Prerequisites

To get started with topic management, you need:

An active AWS account with appropriate IAM permissions for Amazon MSK.
An existing Amazon MSK Express or Standard cluster using Apache Kafka version 3.6 and above.
Basic familiarity with Apache Kafka concepts like topics, partitions, and replication.
AWS CLI installed and configured (for command line examples).

Creating topics

The MSK console provides a guided experience with sensible defaults while still offering advanced configuration options when you need them.

Navigate to the Amazon MSK console and select your cluster.
Choose the Topics tab, then choose Create topic.
Enter a topic name (for example, customer-orders).
Specify the number of partitions (use the guided defaults or customize based on your needs).
Set the replication factor. Note that Express brokers improve the availability and durability of your Amazon MSK clusters by setting values for critical configurations and protecting them from common misconfiguration. If you try to create a topic with a replication factor value other than 3, Amazon MSK Express will create the topic with a replication factor of 3 by default.
(Optional) Configure advanced settings like retention period or message size limits.
Choose Create topic.

The console validates your configuration and creates the topic. You can create multiple topics simultaneously with the same configuration settings. These topic API responses reflect data that updates approximately every minute. For the most current topic state after making changes, wait approximately one minute before querying.

Configuration considerations

When choosing configuration options, consider your workload requirements:

You can configure more partitions to achieve higher throughput but this requires more broker resources. Refer to Best practices for Standard brokers and Amazon MSK Express broker quota for more information on partition limits.
With Standard brokers you can improve durability by configuring higher replication factors, though this will increase your storage costs. Refer to “Build highly available clusters” in Best practices for Standard brokers for more information on replication factors.
Standard brokers support the full range of Apache Kafka topic configurations.
Express brokers offer a curated set of the most important settings. Refer to Topic-level configurations on Express Brokers for more information.

Viewing and monitoring topics

After you create topics, the MSK console provides comprehensive visibility into their configuration. When you select a specific topic, you will see detailed information:

Partitions tab: Shows the distribution of partitions across brokers, including leader assignments and in-sync replica status showcasing Broker IDs for leader and replicas.
Configuration tab: Displays all topic-level configuration settings.
Monitoring tab: Integrates with Amazon CloudWatch to show metrics like bytes in/out, message rates, and consumer lag.

Updating topic configurations

As your workload requirements evolve, you might need to adjust topic configurations. You can modify various topic settings depending on your cluster type. For example:

Retention settings: Adjust retention.ms (time-based) or retention.bytes (size-based) to control how long messages are retained.
Message size limits: Modify max.message.bytes to accommodate larger or smaller messages.
Compression: Change compression.type to optimize storage and network usage.

Configuration changes take effect immediately for new messages. Existing messages remain subject to the previous configuration until they age out or are consumed.

Deleting topics

Amazon MSK also provides APIs for deleting topics that are no longer in use. Before deleting a topic, verify that:

No active producers are writing to the topic
All consumers have finished processing messages
You have backups if you need to retain the data
Downstream applications won’t be impacted

Important: Topic deletion permanently removes all messages in the topic.

Control access with IAM

Beyond streamlining topic operations, you also need appropriate access controls. Access control uses IAM, so you define permissions using the same model that you apply to other AWS resources. Amazon MSK uses a two-level permission model:

Resource-level permissions: An IAM policy that enforces which operations the cluster will allow
Principal-level permissions: IAM policies attached to Roles or Users that enforce which operations a principal is allowed to perform on a cluster

With this separation, you can control access depending on your organizational needs and access patterns for your cluster. Refer to the IAM permissions documentation for IAM permissions required for topic management for the Amazon MSK cluster.

You can grant your operations team broad access to manage all topics and restrict application teams to manage only their own topics. The permission granularity that you need is available through standard IAM policies. If you’ve already configured IAM permissions for Apache Kafka topics, they work immediately with the new functionality without any migration or reconfiguration.

Here is a sample IAM policy definition that allows Describe Topic API

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:Connect"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:111111111111:cluster/iam-auth-acl-test/a6b5c6d5-f74f-4dbc-ad14-63fb5e87fe4f-2"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "kafka-cluster:DescribeTopic",
                "kafka-cluster:DescribeTopicDynamicConfiguration"
            ],
            "Resource": [
                "arn:aws:kafka:us-east-1:111111111111:topic/iam-auth-acl-test/a6b5c6d5-f74f-4dbc-ad14-63fb5e87fe4f-2/*"
            ]
        }
    ]
}

This IAM policy grants the necessary permissions to describe Kafka topics in your Amazon MSK cluster. The policy includes three key permissions:

kafka-cluster:Connect – Allows connection to the specified MSK cluster
kafka-cluster:DescribeTopic – Enables viewing topic details
kafka-cluster:DescribeTopicDynamicConfiguration – Enables viewing topic dynamic configuration

The policy is scoped to a specific cluster ARN and applies to all topics within that cluster using the wildcard pattern /*. Replace the placeholder Amazon MSK cluster ARN with your MSK cluster ARN.

Infrastructure as Code

If you manage infrastructure as code (IaC), you can now define topics alongside clusters in your CloudFormation templates:

Resources:
    OrdersTopic:
      Type: AWS::MSK::Topic
      Properties:
        ClusterArn: !GetAtt MyMSKCluster.Arn
        TopicName: orders
        NumPartitions: 6
        ReplicationFactor: 3
        Config:
          retention.ms: "604800000"

This approach brings topic provisioning into your CI/CD pipelines.

Availability and pricing

The new Amazon MSK topic management experience is available today for Standard and Express Amazon MSK clusters using Apache Kafka version 3.6 and above in all AWS Regions where Amazon MSK is offered, at no additional cost.

Cleanup

To avoid incurring additional charges to your AWS account, ensure you delete all resources created during this tutorial, including:

Amazon MSK cluster
Any Kafka topics created
Associated AWS resources (security groups, VPCs, etc., if created specifically for this blog)

Remember to verify that all resources have been successfully removed to prevent ongoing costs.

Conclusion

Topic management has been a persistent pain point for Apache Kafka operations. The new integrated experience in Amazon MSK now reduces operational friction by bringing topic operations into the AWS tools that you use every day. You now have a consistent, streamlined way to handle these operations for all Apache Kafka topics across multiple MSK clusters. This capability reflects our commitment to reducing operational complexity in Apache Kafka. You get the reliability and performance of Apache Kafka without the operational overhead that traditionally comes with it. Your team spends less time on infrastructure maintenance and more time building streaming applications that drive your business forward.

Ready to start streamlining your topic management? Start managing your topics today through the Amazon MSK console or by visiting the Amazon MSK documentation.

About the authors

How to set up an air-gapped VPC for Amazon SageMaker Unified Studio

Rohit Vashishtha — Thu, 02 Apr 2026 15:28:23 +0000

Organizations are finding significant value using an integrated experience for all your data and AI with Amazon SageMaker Unified Studio. However, many organizations require strict network control to meet security and regulatory compliance requirements like HIPAA or FedRAMP for their data and AI initiatives, while maintaining operational efficiency.

In this post, we explore scenarios where customers need more control over their network infrastructure when building their unified data and analytics strategic layer. We’ll show how you can bring your own Amazon Virtual Private Cloud (Amazon VPC) and set up Amazon SageMaker Unified Studio for strict network control.

Solution overview

The solution covers complete technical know-how of a fully private network architecture using Amazon VPC with no public internet exposure. The approach leverages AWS PrivateLink through VPC endpoints to provide a secure communication between SageMaker Unified Studio and essential AWS services entirely over the AWS backbone network.

The architecture consists of three core components: a custom VPC named airgapped with multiple private subnets distributed across at least three Availability Zones for high availability, a comprehensive set of VPC interface and gateway endpoints for service connectivity, and the SageMaker Unified Studio domain configured to operate exclusively within this isolated environment. This design helps ensure that sensitive data never traverses the public internet while maintaining full functionality for data cataloging, query execution, and machine learning workflows.

By implementing this air-gapped configuration, organizations gain granular control over network traffic, simplified compliance auditing, and the ability to integrate SageMaker Unified Studio with existing private data sources through controlled network pathways. The solution supports both immediate operational needs and long-term scalability through careful IP address planning and modular endpoint architecture.

Prerequisites

The set up requires you to have an existing VPC (for this post, we’ll refer to the name as airgapped but in reality, it refers to the VPC you would like to securely set up SageMaker Unified Studio). If you don’t have an existing VPC, you can follow SageMaker Unified Studio domain quick create administrator guide to get started.

The high level steps to create a VPC meeting minimum requirements for SageMaker Unified Studio are as follows:

In the AWS Management Console, navigate to the VPC console.
Choose Create VPC.
Select the VPC and more radio button.
For Name tag auto-generation, enter airgapped or a name of your choice.
Keep the default values for IPv4 CIDR block, IPv6 CIDR block, Tenancy, NAT gateways, VPC endpoints, and DNS options.
Select 3 for Number of Availability Zones (AZs).
Select 0 for Number of public subnets.
Choose Create VPC.

This produces the following VPC resource map:

Figure 1 – VPC configuration

Set up SageMaker Unified Studio

Now, we will set up SageMaker Unified Studio in an existing VPC, named airgapped-vpc.

Navigate to the SageMaker console, choose Domains in the navigation pane.
Choose Create Domain.
For How do you want to set up your domain?, select Quick set up.
Expand the Quick set up settings
Provide a name for your domain, such as airgapped-domain.
For Virtual private cloud (VPC), select airgapped-vpc.
For subnets, select a minimum of two private subnets.
Choose Continue.
Enter an email address to create a user in AWS IAM Identity Center.
Choose Create domain.
Once the domain is created, choose Open unified studio or use SageMaker Unified Studio URL under Domain details to access SageMaker Unified Studio.

Figure 2 – Amazon SageMaker Unified Studio URL Welcome Page
After logging in to SageMaker Unified Studio, create a project using the guided wizard.
Once the project is created, we need to add the necessary VPC endpoints to allow traffic from the project to communicate to AWS services.
S3 Gateway VPC endpoint was already selected as part of VPC creation step 5 in prerequisites and thus created by default. Now we must add two more VPC endpoints for Amazon DataZone and AWS Security Token Service as illustrated in following step.

These are the minimum set of VPC endpoints to allow using the tooling within SageMaker Unified Studio. For a list of other mandatory and non-mandatory VPC endpoints refer to the tables in the latter part of this post.

Create an interface endpoint

To create an interface endpoint, complete following steps:

Go to the SageMaker Unified Studio Project details page and copy the Project ID.
Figure 3 – SageMaker Unifed Studio Project Details Page
Go to the VPC console and choose Endpoints.
Choose Create Endpoint.
Enter a name for the endpoint, for example, DataZone endpoint for SageMaker Unified Studio.
For AWS Services, enter DataZone.

Figure 4 – Interface Endpoint creation wizard for AWS Service datazone
Select Service Name = com.amazonaws.us-east-1.datazone from the available options.

Figure 5 – Interface Endpoint creation wizard network settings
Select the subnets in the airgapped-vpc that you created earlier.
Filter the Security Groups by pasting the copied Project ID.
Select the security group with Group Name datazone-<project-id>-dev.
Choose Create Endpoint.
Repeat the same steps to create a VPC endpoint for AWS STS.
Once the VPC endpoints are created, validate connectivity in the SageMaker project by running a SQL query or using a Jupyterlab notebook.

For a successful domain and project which does not get into any service level usage, the mandatory VPC endpoints to be created are: S3 Gateway, DataZone, and STS interface endpoints. For other service usage dependent operations like authentication, data preview and working with compute, you would require other mandatory service specific endpoints explained later in this post.

Best practices for VPC set up for various use cases

When setting up SageMaker Unified Studio domain and project profiles, you need to specify the VPC network, subnets, and security groups. Here are some best practices around IP allocation, usage volume and expected growth to consider for different use cases within enterprises.

Production and enterprise use cases

If your organization require strict network control to meet security and compliance requirements for data and AI initiatives, consider following best practices in your production environment.

Use the bring-your-own (BYO) VPC approach to comply with company-specific networking and security requirements.
Implement private networking using VPC endpoints to keep traffic within the AWS backbone.
Use at least two private subnets across different Availability Zones.
Enable DNS hostnames and DNS Support.
Disable auto-assign public IP on subnets.
Plan IP capacity for at least 5 years. A prescriptive guidance for SageMaker Unified Studio is shared in VPC and Networking details section later in this post. Consider the following:
- Number of users
- Number of apps per user
- Number of unique instance types per user
- Average number of training instances
- Expected growth percentage

Testing and non-production use cases

For development, testing, non-prod environment where use cases don’t have stringent security and compliance requirements, use automated setup for quick experiments. Use sample CloudFormation github templates as part of the SageMaker Unified Studio express set up, to automate domain and project creation. However, this includes an Internet Gateway which may not be suitable for security-sensitive environments.

Private networking use cases

VPCs with private subnets require essential service endpoints to allow client resources like Amazon EC2 instances to securely access AWS services. The traffic between your VPC and AWS services remains within AWS network avoiding public internet exposure.

Implement all mandatory VPC endpoints for core services (SageMaker, DataZone, Glue, and more).
Add optional endpoints based on specific service needs, like IPv4 endpoints, dual-stack endpoints, and FIPS endpoints to programmatically connect to an AWS service.
Work with network administrators for:
- Preinstalling needed resources through secure channels like private subnets and self-referencing inbound rules in security groups to enable limited access.
- Allowlisting only necessary external connections like NAT gateway IP and bastion host access in firewall rules.
- Setting up appropriate proxy configurations if required.

External data source access use cases

Consider the following when working with external systems like third-party SaaS platforms, on-premises databases, partner APIs, legacy systems, or external vendors.

Consult with network administrators for appropriate connection methods.
Consider AWS PrivateLink integration where available.
Implement appropriate security measures for non-AWS data your source documents.
For High Availability:
- Deploy across at least three different Availability Zones (at least two for AWS Regions with only two AZs).
- Verify there’s a minimum of three free IPs per subnet.
- Consider larger CIDR blocks (/16 recommended) for future scalability.

VPC and networking details

In this section, we provide details of each networking aspect starting with choice of VPCs, network connectivity details for integrated services to work, the basis of VPC and subnet requirements, and finally the VPC endpoints required for private service access.

VPC

At a high level, you have two options to supply VPCs and subnets:

Bring-your-own (BYO) VPC. This is typically the case for most customers, as most have company specific networking and security requirements to reuse an existing VPC, or to create a VPC that are compliant with those requirements.
Create VPC with the SageMaker quick set up template. When creating a SageMaker Unified Studio domain (DataZone V2 domain in CloudFormation) through the automated quick set up, you will be shown a Quick create stack wizard in CloudFormation which creates VPCs and subnets used to configure your domain.
Note: The quick create stack using template URL is not intended for production use. The template creates an Internet Gateway, which is not allowed in many enterprise settings. This is only appropriate if you are either trying out SageMaker Unified Studio or, running SageMaker Unified Studio for use cases that don’t have stringent security requirements.If you choose this option, you start with SageMaker console, navigate to domains and click Create domain button, followed by Create VPC button. You will navigate to CloudFormation and click on Create stack button to create a sample VPC named SageMakerUnifiedStudio-VPC with just one-click for trying out SageMaker Unified Studio.

Figure 6 – Create VPC button in SageMaker Unified Studio Create Domain Wizard

Cost estimation for recommended VPC set up

The exact cost depends on the configuration of your VPC. For more complex networking set ups (multi-VPC), you may need to use additional networking components such as a Transit Gateway, Network Firewall, and VPC Lattice. These components may incur charges, and cost depends on usage and AWS Region. Interface VPC endpoints are charged per availability zone. They also have a fixed and a variable component in the pricing structure. Use the AWS Pricing Calculator for a detailed estimate.

Network Connectivity

With regards to connectivity to the underlying AWS services integrated within SageMaker Unified Studio, there are two ways to enable connectivity (these are not Studio specific, these are standard ways to enable network connectivity within a VPC). This is an important security consideration that depends on your organization’s security policies.

Through the public Internet. Your traffic will traverse over the public Internet through an Internet Gateway in your VPC.
1. Your VPC must have an Internet Gateway attached to it.
2. Your public subnet must have a NAT Gateway. In addition, your public subnet’s route table must have a default route (0.0.0.0 for IPv4) to the Internet Gateway. This route is what makes the subnet public.
3. Your private subnets must have a default route to the public subnet’s NAT Gateway.
Through the AWS backbone. Your traffic will remain within the private AWS backbone through PrivateLink (by provisioning Interface and Gateway endpoints for the necessary AWS services in each Availability Zone).
1. A list of all the AWS services integrated into Studio and the VPC endpoints required can be found in section VPC Endpoints covered later in this post.
2. For non-AWS resources, certain external providers of these services may offer PrivateLink integration. Check with each provider’s documentation and your network administrator to understand the most suitable way to connect to these external providers.

In a private networking scenario, you will need to consider whether you need connectivity to non-AWS resources in a way that’s compliant with your organization’s security policies. A few examples include the following:

If you need to download software in your remote IDE host (for example, command line programs, such as Ping and Traceroute)
If you have code that connects to external APIs.
If you use software (such as JupyterLab or Code Editor extensions) that rely on external APIs.
If you depend on software dependencies hosted in the public domain (such as Maven, PyPi, npm)
If you need cross-Region access to certain resources (such as access to S3 buckets in a different Region)
If you need functionality whose underlying AWS services do not have VPC endpoints in all Regions or any Region.
1. Amazon Q (powers Q and code suggestions)
2. SQL Workbench (powers Query Editor)
3. IAM (powers Glue connections)

If you need to connect to data sources outside of AWS (such as Snowflake, Microsoft SQL Server, Google BigQuery)
Enterprise network administrators must also complete either of the following prerequisites to handle private networking scenarios:

Preinstall needed resources through secure channels if possible. An example would be to customize your SageMaker AI image by installing dependencies, after they are code scanned, vetted technically and legally by your organization.
If AWS PrivateLink integration is not available for external providers, allowlist network connections to these external sources. Allow firewall egress rules, directly or indirectly, through a proxy in your organization’s network. Check with your network administrator to understand the most appropriate option for your organization.

VPC Requirements

When setting up a new SageMaker Unified Studio Domain, it’s necessary to supply a VPC. It’s important to note that these VPC requirements are a union of all the requirements from the respective compute services integrated into Studio, some of which are reinforced by validation checks during the corresponding blueprint’s deployment. If these requirements that have validation checks are not fulfilled, the resource(s) contained in that blueprint may fail to create on project creation (on-create), or when creating the compute resource (on-demand). This section will present a summary of these requirements, as well as relevant documentation links from which they originate.

Subnet requirements for specific compute in a VPC

This section lists the compute services integrated in SageMaker Unified Studio that require VPC/subnets when provisioning the respective compute resources.

Compute Connections

Other Services

Requirements

Number of subnets: At least two private subnets. This requirement comes from Redshift Serverless.
Availability zones (AZs): At least two different AZs (for Regions with two AZs, two subnets are sufficient). This requirement comes from Redshift Serverless. For workgroups with Enhanced VPC Routing (EVR), you need three AZs.
Free IPs per subnet: At least three Ips per subnet. This requirement comes from Redshift Serverless without EVR. For detailed IP addresses requirement with EVR enabled workgroups, refer to Serverless usage considerations. Three is a minimum and may not be enough for your needs. For example, EMR cluster creation will fail if no subnets with enough IPs are found in the VPC. We recommend doing a forward-looking capacity planning exercise based on your use cases (for example, growth rate, users, compute needs) to project at least 5 years into the future. This helps to determine how many IPs are needed by the team using Studio and other services that use this VPC and come up with a ceiling for the CIDR block size.
Private or public subnets: We enforce that at least three private subnets be supplied, and recommend that only private subnets are chosen, with a few nuances. This requirement comes from SageMaker AI domain. A new SageMaker AI domain, when set up with VpcOnly mode, requires that all subnets in the VPC be private. This is the default networking mode in the Tooling blueprint. If you choose to use PublicInternetOnly mode, this restriction does not apply, you may choose public subnets from your VPC. To change the mode, modify the Tooling Blueprint parameter sagemakerDomainNetworkType.
Enable DNS hostname and DNS Support: Both must be enabled. This requirement comes from EMR. Without these VPC settings, enableDnsHostname and enableDnsSupport, connecting to the EMR Cluster using the private DNS name through the Livy Endpoint will fail. SSL Verification, which can only be done when connecting using the DNS name, not the IP.
Auto assign public IP: Disable. We recommend that this EC2 subnet setting (mapPublicIpOnLaunch) be disabled when using private subnets, because public IPs come at a cost and are a scarce resource in the total addressable IPv4 space.

VPC endpoints

If you choose to run SageMaker Unified Studio without public internet access, VPC endpoints are required for all services SageMaker Unified Studio needs to access. These endpoints provide secure, private connectivity between your VPC and AWS services without traversing the public internet. The following table lists the required endpoints, their types, and what each is used for.

Some endpoints may not show up directly in your browser’s network tab. The reason is that some of these services (such as CloudWatch) are transitively invoked by other services.

Mandatory endpoints

The following are required endpoints for SageMaker Unified Studio and supporting services to function properly. Gateway endpoints can be used where available, you can use interface endpoints for all other AWS services.

AWS service	Endpoint	Type	Purpose
Glue	`com.amazonaws.${region}.glue`	Interface	For Data Catalog and metadata management
STS	`com.amazonaws.${region}.sts`	Interface	Required for assuming IAM roles
S3	`com.amazonaws.${region}.s3`	Gateway	Required for datasets, Git backups, notebooks, and Git sync
SageMaker	`com.amazonaws.${region}.sagemaker.api`	Interface	Required for calling SageMaker APIs
SageMaker	`com.amazonaws.${region}.sagemaker.runtime`	Interface	For invoking deployed inference endpoints
DataZone	`com.amazonaws.${region}.datazone`	Interface	For data catalog and governance
Secrets Manager	`com.amazonaws.${region}.secretsmanager`	Interface	To securely access secrets
SSM	`com.amazonaws.${region}.ssm`	Interface	For secure command execution
SSM	`com.amazonaws.${region}.ssmmessages`	Interface	Enables live SSM sessions
KMS	`com.amazonaws.${region}.kms`	Interface	For decrypting data (volumes, S3, secrets)
EC2	`com.amazonaws.${region}.ec2`	Interface	For subnet and ENI management
EC2	`com.amazonaws.${region}.ec2messages`	Interface	Required for SSM messaging
Athena	`com.amazonaws.${region}.athena`	Interface	Required to run SQL queries
Amazon Q	`com.amazonaws.${region}.q`	Interface	Used by SageMaker Notebooks for enhanced productivity

Optional Endpoints

Only create these if the corresponding service is used in your environment.

AWS service	Endpoint	Type	Purpose
EMR	`com.amazonaws.${region}.emr-serverless`	Interface	Serverless Spark/Hive jobs
	`com.amazonaws.${region}.emr-serverless-services.livy`	Interface	Required for Livy job submission (EMR Serverless)
	`com.amazonaws.${region}.elasticmapreduce`	Interface	Classic EMR (EC2-based)
	`com.amazonaws.${region}.emr-containers`	Interface	EMR on EKS workloads
Redshift	`com.amazonaws.${region}.redshift`	Interface	For provisioned Redshift clusters
	`com.amazonaws.${region}.redshift-serverless`	Interface	For Redshift Serverless
	`com.amazonaws.${region}.redshift-data`	Interface	Required for running SQL against Redshift
Amazon Bedrock	`com.amazonaws.${region}.bedrock-runtime`	Interface	Invoke Bedrock models at runtime
	`com.amazonaws.${region}.bedrock-agent`	Interface	For Bedrock knowledge agents
	`com.amazonaws.${region}.bedrock-agent-runtime`	Interface	For running knowledge agent workloads
CloudWatch	`com.amazonaws.${region}.logs`	Interface	Application and notebook logs
RDS	`com.amazonaws.${region}.rds`	Interface	Connect to Amazon RDS and Aurora
CodeCommit	`com.amazonaws.${region}.codecommit`	Interface	Git integration with CodeCommit
CodeCommit	`com.amazonaws.${region}.git-codecommit`	Interface	Alternative endpoint for CodeCommit
CodeConnections and CodeStar	`com.amazonaws.${region}.codeconnections.api`	Interface	GitHub and GitLab repo integration
CodeConnections and CodeStar	`com.amazonaws.${region}.codestar-connections.api`	Interface	Alias of CodeConnections

Clean up

AWS resources provisioned in your AWS accounts may incur costs based on the resources consumed. Make sure you do not leave any unintended resources provisioned. If you created a VPC and subsequent resources as part of this post, make sure you delete them.

The following service resources provisioned during this blog post need to be deleted:

IAM Identity Center users and groups.
Resources provisioned within your project using tooling configuration and blueprints within your domain.
The airgapped VPC.

Conclusion

In this post, we walked through the process of using your own existing VPC when creating domains and projects in SageMaker Unified Studio. This approach benefits customers by giving them greater control over their network infrastructure while using the comprehensive data, analytics, and AI/ML capabilities of Amazon SageMaker. We also explored the critical role of VPC endpoints in this set up. You now understand when these become necessary components of your architecture, particularly in scenarios requiring enhanced security, compliance with data residency requirements, or improved network performance.

While using a custom VPC requires more initial set up than the Quick Create option, it provides the flexibility and control many organizations need for their data science and analytics workflows. This approach provides a mechanism for your SageMaker environment to integrate with your existing infrastructure and adheres to your organization’s networking policies. Custom VPC configurations are a powerful tool in your arsenal for building secure, compliant, and efficient data science environments.

To learn more, visit Amazon SageMaker Unified Studio – Administrator Guide and User Guide.

About the authors

Navigating multi-account deployments in Amazon SageMaker Unified Studio: a governance-first approach

Ben Shafabakhsh — Wed, 01 Apr 2026 19:12:46 +0000

Amazon SageMaker Unified Studio brings together data engineering, analytics, and machine learning (ML) workflows into a cohesive, governed environment. This unified approach reduces traditional silos between data teams and ML practitioners, so organizations can advance their AI and ML initiatives with greater collaboration and efficiency.

As enterprises begin their SageMaker Unified Studio adoption, they must determine the best practices for implementing data federation principles when using SageMaker Unified Studio across the organization. The way that you structure your SageMaker Unified Studio deployment is more than a technical decision. It directly impacts your governance framework, security posture, operational scalability, and day-to-day team collaboration.

In this post, we explore SageMaker Unified Studio multi-account deployments in depth: what they entail, why they matter, and how to implement them effectively. We examine architecture patterns, evaluate trade-offs across security boundaries, operational overhead, and team autonomy. We also provide practical guidance to help you design a deployment that balances centralized control with distributed ownership across your organization.

The multi-account challenge: why organizations struggle

If you’re working in a large enterprise, a multi-account AWS environment is often your starting position. If you’re starting from scratch, consider whether to use a single-account for all SageMaker Unified Studio components or dedicate separate accounts for governance and business units. A multi-account architecture aligns with AWS best practices and proves valuable if you have:

Distributed teams with independent operations: multi-account architecture accommodates multiple teams or business units that maintain separate operations so that each team can manage their projects autonomously within isolated environments. Each unit can deploy and manage resources independently, implement team-specific security controls, and scale infrastructure without impacting others. This is achieved through a shared, unified integrated development environment (IDE) for collaboration and standardized tools across the organization.
Compliance and data governance requirements: For regulatory mandates like GDPR, HIPAA, or data sovereignty needs, you will benefit from this setup, because sensitive data remains segregated in business-unit specific accounts. This reduces risk exposure, streamlines audits, and maintains compliance boundaries without compromising access to centralized collaboration tools.
Centralized governance: A multi-account architecture maintains visibility across all projects and business units from a single control plane. The Domain account can enforce security policies and compliance requirements across the entire organization and provide centralized monitoring, audit logging, and user access management.
Clear cost visibility and accountability: Multi-account architecture enables granular billing tracking, with each account generating separate bills that clearly attribute costs to specific teams or business units. This transparency streamlines budgeting and financial accountability, removing the complexity of cost allocation tags and manual reporting that’s typically required in single-account models where multiple teams share the same infrastructure and resources.

Overall, this approach improves efficiency, security, and scalability for you, whether you’re managing a few teams or coordinating across a larger organizational structure.

Understanding the core constructs of SageMaker Unified Studio

Before diving into multi-account strategies, it’s important to understand the foundational constructs of SageMaker Unified Studio. Each is elaborated in greater detail in the Administrator Guide.

Domain: The top-level administrative boundary where governance lives. In a multi-account setup, this is your centralized control plane for catalog, policies, and user access.
Project: A collaborative workspace for developing data, AI, and ML initiatives. In multi-account deployments, a Project’s metadata lives in the Domain account and compute and data resources deploy into associated business unit accounts. This separation is central to the pattern that we explore.
Project Profile: A template that standardizes how Projects are created. For multi-account setups, this is where administrators define which accounts and AWS Regions Projects can target.
Blueprints: Infrastructure as code (IaC) components that define what gets provisioned inside a Project. Each associated account must enable its relevant Blueprints before Projects can deploy there.

The following diagram illustrates how these key constructs interact. Within a Domain, users create Projects organized through a single Project Profile. The Project Profile defines and configures a collection of Blueprints. When a project is created, the infrastructure specified in those Blueprints is automatically provisioned and becomes available within the project workspace.

Figure 1: Amazon SageMaker Unified Studio Core Constructs

Multi-account setup in SageMaker Unified Studio

To illustrate these concepts in practice, we demonstrate with a sample enterprise organization that exemplifies enterprise environments with several AWS accounts belonging to different business units:

Central Data Governance team: Owns and manages governance and access control across the organization. They plan to build a data solution in a dedicated AWS account using SageMaker Unified Studio. The platform must provide an integrated development environment (IDE) to work with data and ML use cases and connect to multiple business unit’s AWS accounts (Finance and Marketing).
Finance Business Unit: Owns datasets for fraud analysis and churn prediction in their own AWS account.
Marketing Business Unit: Maintains customer sentiment data and campaign analytics in their own AWS account.

In the following diagram we show the Data platform constructs provided by SageMaker Unified Studio in each AWS account showing the clear separation between centralized governance and distributed resource deployment.

Figure 2: Sample organization architecture in Amazon SageMaker Unified Studio

The Central Data Governance Account contains the SageMaker Unified Studio Domain. This contains the shared platform resources (Catalog, shared infrastructure), governance constructs (Domain units, metadata forms), and governance policies (authorization policies, enforcement rules). These configuration elements define the standards and capabilities available across the organization. They’re the Service-level configuration data: Metadata, policies, and governance rules that define how resources should be provisioned.

In contrast, the Associated Accounts (Marketing and Finance) contain the actual AWS infrastructure, compute/storage (purple cubes) and data stores (cylinders), provisioned when Projects are created. The diagram shows how Marketing Projects and Finance Projects ultimately deploy their runtime resources into their respective business unit accounts. The separation keeps the governance centralized and consistent while allowing business unit dependent resources to be isolated, billed separately, and managed according to each business unit’s specific requirements.

To understand the core constructs of SageMaker Unified Studio, we listed the core components of SageMaker Unified Studio and explained how they relate to each other. Taking the same diagram as the basis, we will now represent how these constructs are created in our multi-account sample scenario.

Construct	Deployment Location	Deployed Resources
Domain	Central Data Governance Account	Portal Catalog Metadata Forms Authorization Policies
Project	Central Data Governance Account + Associated Account	In Central Data Governance Account: Project configuration and metadata In Associated Account: Project Infrastructure resources such as Compute Project Data
Project Profile	Central Data Governance Account	Project Profile Configuration
Blueprint	Associated Account	Blueprints are enabled in each associated account

Core construct deployment locations and resources in SageMaker Unified Studio

Implementing multi-account deployments

To enable production-ready data science and analytics workflows across multiple AWS accounts governed by a SageMaker Unified Studio Domain, organizations must establish a structured cross-account configuration. This setup allows each business unit (BU) to retain ownership of its Projects and AWS resources while using centralized governance provided by the Domain. The process involves four key steps: account association, Blueprint enablement, Project Profile configuration, and Project creation.

Note: The following steps provide a high-level overview of the multi-account deployment process. For a more detailed, step-by-step guide, refer to How to associate an account when using Amazon SageMaker Unified Studio.

Step 1: Account association to a domain

The Domain administrator associates each AWS account with the SageMaker Unified Studio Domain for seamless cross-account functionality by providing the AWS account number for the targeted accounts. This association lets the Domain publish and consume data from associated accounts, create resources within them, maintain cross-account access for the SageMaker Catalog, and deploy Projects directly into business unit accounts. Account association is a critical prerequisite for cross-account Project deployment. Behind the scenes, SageMaker Unified Studio uses AWS Resource Access Manager (AWS RAM) to make this cross-account functionality happen.

Step 2: Enabling blueprints

Each associate account administrator must enable the relevant Blueprints before creating Project Profiles. This important step verifies that Projects can provision the necessary tools and resources that users need to run their workloads. Blueprints serve as standardized infrastructure templates that administrators can use to enforce organizational standards, security controls, and best practices across all Projects. Through Blueprints, administrators configure essential resources including AWS Identity and Access Management (IAM) roles, AWS Key Management Service (AWS KMS) keys, Amazon Simple Storage Service (Amazon S3) buckets, Amazon Virtual Private Cloud (Amazon VPC) settings, and security groups. This centralized approach helps maintain consistency, compliance, and governance at scale while preventing users from creating Projects with misconfigured or non-compliant infrastructure.

Step 3: Configuring project profile

With the accounts successfully associated and the Blueprints enabled, the next step is to configure a Project Profile that determines where your Project resources will be deployed. Your choice of Project Profile strategy impacts both operational flexibility and governance.

Domain administrators control which Blueprints are included in each Project Profile and can specify the target AWS Regions and accounts for deployment, providing the governance foundation to standardize Project creation. Administrators can use pre-created Project Profiles like “All Capabilities” or “SQL Analytics”, or create custom Project Profiles tailored to specific organizational needs.

When configuring Project Profiles, you can choose between two deployment models:

Static (Pre-Defined): Profile specifies a fixed account and Region. Projects by default deploy to the same location. This is recommended for strict governance controls and compliance requirements where production resources must remain in designated accounts or Regions.
Dynamic (Parameterized): Users select from available associated accounts and Regions during Project creation (configured through Account Pools). This is recommended for multi-environment workflows (Dev/Test/Prod) and reducing administrative overhead by maintaining fewer profile templates.

Dynamic profiles balance governance with agility: Administrators define standards once, while users retain deployment flexibility aligned with their business needs.

Step 4: Project creation

With Project Profiles configured, you can now create a new Project from any of the associated accounts using the profile created in the previous step.

Defining project boundaries: when to create a new project

One of the common challenges that you will face is determining when to create a new Project. The answer significantly impacts collaboration effectiveness, resource isolation, cost tracking, and governance. Here’s a practical framework to guide your decision-making.

A Project should represent a distinct business initiative with a defined scope, a dedicated team, and measurable outcomes. Think of Projects as team workspaces organized around business outcomes, not technical components.

Create a new Project when you need clear separation across multiple dimensions: cost allocation, access control, and data governance. If Finance and Marketing teams require separate budget tracking, distinct data access policies, and different governance controls, they should have separate Projects. For example, “Customer Churn Prediction” and “Fraud Detection” might use similar tools, but if they have different stakeholders, budget owners, and data sensitivity requirements, so they warrant separation. Similarly, create separate Projects when facing different compliance or regulatory requirements (like HIPAA versus PCI-DSS) or when initiatives have independent deployment lifecycles. Experimental ML research Projects shouldn’t share workspaces with production recommendation engines that require stricter change controls and availability guarantees.

However, avoid fragmenting related work into unnecessary silos:

Don’t create separate Projects for individual workflows or pipelines, a single “Marketing Campaign Optimization” Project should contain audience segmentation, propensity modeling, and campaign attribution workflows together.
Don’t separate different data processing stages; keep data ingestion, transformation, and analysis within one Project to maintain clear lineage and enable seamless collaboration.
Projects are team workspaces, not personal sandboxes, so use shared Projects with role-based access control rather than creating individual Projects per team member.
Small proof of concepts (POC) or temporary experiments should be conducted within the parent Project, with successful initiatives promoted to dedicated production Projects only when they mature into full-scale capabilities requiring independent governance.

Conclusion

Throughout this post, we explored how the separation of governance and working accounts forms the foundation of a scalable, secure, and compliant data and AI platform.

With centralized governance in the Domain account, organizations can enforce consistent security policies, compliance requirements, and cost management, while allowing sub-accounts the autonomy over their own resources. This approach enhances security and compliance, and fosters collaboration and innovation within teams by providing them with the flexibility that they need to operate effectively. Ultimately, this governance-first strategy supports keeping data remains protected and accessible in a controlled manner, empowering teams to drive business outcomes efficiently. To implement a multi-account deployment for your organization, get started by creating your first SageMaker Unified Studio Domain and follow the step-by-step guidance to establish your governance-first architecture.

About the authors

Improve the discoverability of your unstructured data in Amazon SageMaker Catalog using generative AI

Nishchai JM — Wed, 01 Apr 2026 19:09:48 +0000

Every day, businesses generate massive amounts of unstructured data such as PDFs, images, emails, customer feedback. Although this data holds valuable business insights, extracting meaningful value from it remains a significant challenge. Its lack of proper context and searchability often keeps it siloed and underutilized, limiting data-driven decision making. Financial reports, legal documents, and customer feedback are prime examples. They contain the answers that your business needs, yet they frequently go unanalyzed due to these barriers.The sheer volume of unstructured content requires scalable infrastructure and automated processing tools, while sensitive information embedded within demands sophisticated classification and protection strategies. Without proper management, organizations face operational inefficiencies, high costs, and increased regulatory risks and reduced AI effectiveness.

What if business context from your PDFs, images, and emails could be automatically extracted and surfaced wherever your teams search for information? In this post, we show you how to implement this. By combining Amazon SageMaker Catalog with generative AI capabilities, you can make unstructured data searchable and queryable through the same interfaces that your teams use for structured data analysis. Success requires balancing advanced AI techniques with governance frameworks so that your data is discoverable and secure for better decision making.

This is a two-part series post. In the first part, we walk you through how to set up the automated processing for unstructured documents, extract and enrich metadata using AI, and make your data discoverable through SageMaker Catalog. The second part is currently in the works and will show you how to discover and access the enriched unstructured data assets as a data consumer. By the end of this post, you will understand how to combine Amazon Textract and Anthropic Claude through Amazon Bedrock to extract key business terms and enrich metadata using Amazon SageMaker Catalog to transform unstructured data into a governed, discoverable asset.

Solutions overviewYou will transform unstructured data into an interactive knowledge base through automated processing within the Amazon SageMaker AI environment. Here is how it works:

You will set up an Amazon SageMaker Unified Studio Data Notebook Jupyter-based workspace where you manage your entire processing pipeline, add metadata to unstructured documents like PDFs, photos, emails, or audio recordings stored in Amazon Simple Storage Service (Amazon S3).
You will add your files to the SageMaker Project.
Amazon Textract extracts information and insights from text, removing manual transcription. This extracted content instantly populates your asset’s README.
Amazon Bedrock turns the text into business terms that provide SageMaker Catalog assets the correct business context to help with semantic search or business query search.
You will use a publish method to publish the enriched data to the Amazon SageMaker Catalog, making it available to your organization.

The architecture sets up a pipeline from processing raw documents to enabling end users interaction, with the Amazon SageMaker Catalog serving as the central hub for sending and receiving data. Amazon SageMaker Catalog includes generative AI features that automatically develop and add business descriptions for structured data assets. This capability streamlines documentation processes and provides greater consistency across data assets. You can further enhance this solution to create summaries by also reading and incorporating S3 metadata. This will add more context, such as object properties, access patterns, and storage characteristics, to the extracted document content, streamlining the process to find and catalog data.

Prerequisites

To implement the solution, you must complete the following prerequisites:

Create an AWS account – Required to access all AWS services (Amazon SageMaker Catalog, Amazon S3, Amazon Textract, Amazon Bedrock) used in this solution.
Create an Amazon SageMaker Unified Studio domain: This provides a collaborative environment for connecting your assets, users, and their projects.
Create an SageMaker Project with all capabilities: Your collaborative workspace where you will upload documents, run processing notebooks, and manage permissions for your data enrichment pipeline. Team members added to this project gain immediate access to all shared resources.
- Producer Project (project name: $(-your-project-name) use “unstructured-producer-project”, project profile: All capabilities)

Solution deployment

Now let’s complete the following steps to deploy and verify the solution.

Prepare source datasets

In this section you will use the following sample datasets by downloading them to your local machine. We will upload these files into your SageMaker Project S3 bucket created in the prerequisite step.

ED_DistributionToothDisorder.png: The dataset shows emergency department visits for tooth disorders in the US from 2020–2022, broken down by age group, gender, and race/ethnicity.
analysisDentalEDvsts.pdf: This report shows emergency department visits analysis for dental conditions across the United States between 2016–2019, showing that among non-traumatic dental visits.
s3_document_processor_unstructured.ipynb notebook (keep in local environment and will be used at a later stage)

With your sample datasets ready, let’s log in to SageMaker Unified Studio and upload them to your project.

Log in to Amazon SageMaker Unified Studio as a data producer

Log in to the SageMaker Unified Studio URL using your username and password. In the portal UI, select the producer project (unstructured-producer-project) that you created in the project selector (at the top center of the screen).
Under Data, do the following:
- Choose the default project bucket created amazon-sagemaker-12*********-us-west-2-51271642b525/dzd_*********/c3jl67qvxbic9c/.
- Next, choose the three dots and upload the downloaded files (1&2) from the prepared dataset section.
- After adding the files, choose Publish to Catalog to publish your asset.

Your files are now in the catalog. Before we process them, your project needs permission to access the services. Let’s add those permissions.

Add permissions to an IAM role for the Amazon SageMaker Project role.
- Go to the Project overview tab and find the Project role ARN. It can be found in the Project details section.
- Go to the AWS IAM service and choose Roles. Search for the role as highlighted in the preceding image and add the following permissions. The following policies use full-access managed policies to keep things straightforward for this walkthrough. We don’t recommend this for production environments. Instead, we encourage you to take a moment to review each policy with your security team and scope them down to the least-privilege permissions that your workload needs:
  - Add an AmazonBedrockFullAccess managed policy.
  - Add an AmazonTextractFullAccess managed policy.
  - Add an AmazonS3FullAccess managed policy.
  - Add this inline policy to project policy
```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "datazone:Search",
                "datazone:GetAsset",
                "datazone:CreateAssetRevision",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}
```

With permissions configured, let’s set up the governance framework that will classify your documents. We will create glossary terms to tag sensitive and non-sensitive data.

Add Glossary and Glossary terms.
- Choose Glossaries and CREATE GLOSSARY.
- Add the following:
- Name of the glossary and descriptions.
- Toggle on the Enable button as shown in the following screenshot.
- Create an appropriate glossary term. You will add this term to your business metadata.
- Navigate to the Discover menu in the top navigation bar.
- Choose Glossaries, and then select Create term.
- Ensure that you’re creating the term under the Confidentiality glossary that you created in step 4.
- Create two terms (sensitive and non-sensitive) and add a description. Make sure that the Enabled toggle is on to enable the new term.

Now that we have configured the necessary permissions and created our glossary terms, let’s proceed with building the business metadata.

Build business metadata

In this section, you will use Amazon Textract and Amazon Bedrock to automatically build and curate business metadata for your assets using a SageMaker Unified Studio Notebook.

From the Project Overview page, access the Compute section in the left menu.
- Navigate to the Spaces tab.
- Choose the default space created by your project(default-985-) to begin”.

Open the space details page by selecting the Name (default-****).
- Under Actions, Choose Open space to be taken to the Data Notebook workspace (Jupyter based).

After connected, upload the downloaded notebook from the prerequisite step in your JupyterLab interface by either dragging it into the File browser or using the upload icon.

Before running the notebook, let’s understand what each cell does and how they work together to transform your documents into discoverable assets.

The notebook contains code for processing your documents. It begins by setting up AWS service connections using Boto3, the SDK for Python, importing necessary libraries, and initializing clients for Amazon S3, Amazon Textract, and Amazon Bedrock. The code configures an S3 bucket for processing medical documents.

Now, proceed to run through the individual cells:

This cell searches an S3 bucket for files with specific extensions (.pdf, .jpg, .jpeg, .png, .tiff), collects them into a list called ‘documents’, and prints the total count and names of found files. It uses the list_objects_v2 method to fetch the contents and filters them based on their file extensions.

The documents extracted from the S3 bucket are processed using Amazon Textract. It loops through each document, starts an Amazon Textract job, monitors its progress, and when successful, extracts text from all pages. The extracted text is stored in a list along with its document identifier. The code handles pagination, errors, and includes delays between API calls to prevent throttling.

Output [1] The following screenshot shows the output of a Jupyter notebook cell after you run the Amazon Textract API.

This code takes all previously extracted document text, combines it into one string (2,002 characters total), and uses the Anthropic Claude 3 Sonnet model (available through Amazon Bedrock) to generate a concise summary. The code configures the AI model with specific parameters, sends the combined text for analysis, and returns a summarized version of all document content.

Output [2] The following screenshot shows the proceeding output result.

This code detects whether document text contains sensitive PII data (names, emails, addresses, financial details) and returns a Boolean true/false result.

Finally, the code completes the document processing pipeline by classifying the asset based on sensitivity, updating its metadata with an AI-generated summary, and assigning the appropriate glossary term. The script retrieves existing asset details to preserve all metadata forms, updates the README field with the summary content, and creates a new revision. This record of the classification and documentation is stored in the data catalog, accessible alongside your source assets for ongoing governance.

Output: The following screenshot shows the output result.

After generating metadata with Amazon Textract and Amazon Bedrock, republish the data to make it discoverable to users. Note the README and the Glossary terms have already been added based on the previous script.

To republish unstructured data with enriched metadata, go to your producer project and choose Re-publish asset.

To search for the asset that was published, choose one of the keywords from the README sections. For this example, we search using these keywords high percent of ED visits.

Go to Home in the search bar to enter the keyword. Then choose the asset as displayed in the following screenshot:

As shown in the preceding image, the search results display the asset name along with the project it belongs to. Select the asset to view its details, where you will find rich business metadata, lineage information, and more, as shown in the following screenshot.

With the asset published and metadata enriched, data assets are ready to be used.

Clean up

To avoid ongoing charges, make sure to delete the resources immediately after completing the tutorial:

Stop Studio Resources – Close all running notebooks – Stop any running notebook instances – Shut down unused kernels. Running instances continue to incur charges even when not actively used.
Clean S3 Storage – Delete any temporary files created during processing – Remove uploaded test documents if no longer needed. While Amazon S3 costs are minimal, large volumes of unneeded data can accumulate charges.

Conclusion

In this post we showed you how you can transform unstructured data into valuable business assets through seamless integration with AWS services. You can efficiently process documents using Amazon Textract for text extraction, harness the capabilities of Amazon Bedrock for intelligent term identification, and use Amazon SageMaker Catalog for metadata management—all within a secure, governed framework.

Additional resources

To continue your Amazon SageMaker AI journey, see the following resources:

1. - Explore the Amazon SageMaker Developer Guide.
  - Documentation – You can find complete setup guides in Amazon SageMaker Unified Studio Documentation.
  - Get started with these new integrations through the Amazon s Unified Studio console.

About the authors

Automated tag-based DAG permission management in Amazon MWAA

Amey Ramakant Mhadgut — Tue, 31 Mar 2026 15:57:06 +0000

Amazon Managed Workflows for Apache Airflow (Amazon MWAA) provides robust orchestration capabilities for data workflows, but managing DAG permissions at scale presents significant operational challenges. As organizations grow their workflow environments and teams, manually assigning and maintaining user permissions becomes a bottleneck that can impact both security and productivity.

Traditional approaches require administrators to manually configure role-based access control (RBAC) for each DAG, leading to:

Inconsistent permission assignments across teams
Delayed access provisioning for new team members
Increased risk of human error in permission management
Significant operational overhead that doesn’t scale

There is another way of doing it by defining custom RBAC roles as mentioned in this Amazon MWAA User Guide. However, it doesn’t use Airflow tags to do so.

In this post, we show you how to use Apache Airflow tags to systematically manage DAG permissions, reducing operational burden while maintaining robust security controls that complement infrastructure-level security measures.

Prerequisites

To implement this solution, you need:

AWS resources:

An Amazon MWAA environment (version 2.7.2 or later, not supported in Airflow 3.0)
IAM roles configured for Amazon MWAA access with appropriate trust relationships
Amazon Simple Storage Service (Amazon S3) bucket for Amazon MWAA DAG storage with proper permissions

Permissions:

IAM permissions to create and modify Amazon MWAA web login tokens
Amazon MWAA execution role with permissions to access the Apache Airflow metadata database
Administrative access to configure Apache Airflow roles and permissions

Solution overview

The automated permission management system consists of four key components that work together to provide scalable, secure access control.The following diagram shows the workflow of how the solution works.

IAM integration layer – AWS IAM roles map directly to Apache Airflow roles. Then, users authenticate through AWS IAM and are automatically assigned corresponding Airflow roles. This supports both individual user roles and group-based access patterns.
Note:
- IAM Based access control to Amazon MWAA works for Apache Airflow default roles. For custom roles, the Admin user can assign the custom role using the Apache Airflow UI as mentioned in the Knowledge Center post and in the Amazon MWAA User Guide.
- If using other authenticators, the tag-based DAG permissions continue to work as stated in the AWS Big Data Blog post.
Tag-based configuration – Apache Airflow tags defined in DAGs are used to declare access requirements. It supports read-only, edit, and delete permissions.
Automated synchronization engine – Scheduled DAG scans all active DAGs for permission tags based on CRON schedule. It then processes tags and updates Apache Airflow RBAC permissions accordingly. Then, it provides a configuration based to control the clean-up of existing permissions.
Role-based access control enforcement – Apache Airflow RBAC enforces the configured permissions by storing on Apache Airflow role and permissions metadata tables. Users see only the DAGs that they have access to. They have granular control over read compared to edit permissions.

Data flow

Amazon MWAA User assumes an IAM role to access the Amazon MWAA UI.
DAG developer adds relevant tags to the DAG definition.
manage_dag_permissions DAG deployed to the Amazon MWAA environment runs on a CRON schedule, for example, daily.
The DAG updates the respective role permissions to the DAG by updating the Apache Airflow metadata on the Apache Airflow DB.
Users gain or lose access based on their assigned roles.

Our solution builds upon the existing IAM integration of Amazon MWAA, while extending functionality through custom automation:

Authentication and role mapping – Users authenticate through AWS IAM roles that map directly to corresponding Airflow roles.
Automated user creation – Upon first login, users are automatically created in the Apache Airflow metadata database with appropriate role assignments.
Tag-based permission control – Each Apache Airflow role contains specific DAG permissions based on tags defined in the DAGs.
Automated synchronization – A scheduled script maintains permissions as DAGs are added or modified.

Step 1: Configure IAM to Airflow role mapping

First, establish the mapping between your IAM principals and Apache Airflow roles. To grant permission using the AWS Management Console, complete the following steps:

Sign in to your AWS account and open the IAM console.
In the left navigation pane, choose Users, then choose your Amazon MWAA IAM user from the users table.
On the user details page, under Summary, choose the Permissions tab, then choose Permissions policies to expand the card and choose Add permissions.
In the Grant permissions section, choose Attach existing policies directly, then choose Create policy to create and attach your own custom permissions policy.
On the Create policy page, choose JSON, then copy and paste the following JSON permissions policy in the policy editor. This policy grants web server access to the user with the default Public Apache Airflow role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "airflow:CreateWebLoginToken",
      "Resource": "arn:aws:airflow:region:account-id:environment/your-environment-name"
    }
  ]
}

Step 2: Create the automated permission management DAG

Now, create a DAG that will automatically manage permissions based on tags.

from airflow import DAG, settings
from airflow.operators.python import PythonOperator
from sqlalchemy import text
import pendulum
import logging

dag_id = "manage_dag_permissions"

class Constants:
    """
    Constants class to hold constant values used throughout the code.
    """
    AB_VIEW_MENU = "ab_view_menu"
    AB_PERMISSION = "ab_permission"
    AB_ROLE = "ab_role"
    AB_PERMISSION_VIEW = "ab_permission_view"
    AB_PERMISSION_VIEW_ROLE = "ab_permission_view_role"
    DAG_TAG = "dag_tag"

    CAN_READ = "can_read"
    CAN_EDIT = "can_edit"
    CAN_DELETE = "can_delete"


def _execute_query(sql_text, params=None, fetch=True):
    """
    Execute a parameterized SQL query against the Airflow metadata DB.
    All queries use SQLAlchemy text() with bind parameters to prevent SQL injection.

    Parameters:
        sql_text: SQL string with :named bind parameters
        params: dict of parameter values
        fetch: If True, return list of first-column values; if False, commit and return None
    Returns:
        List of values (first column) if fetch=True, else None
    Raises:
        Re-raises any exception after rollback and logging
    """
    session = settings.Session()
    try:
        stmt = text(sql_text)
        if fetch:
            result = session.execute(stmt, params or {}).fetchall()
            return [row[0] for row in result]
        else:
            session.execute(stmt, params or {})
            session.commit()
            return None
    except Exception as e:
        session.rollback()
        logging.error(f"DB query error (fetch={fetch}): {type(e).__name__}: {e}")
        raise
    finally:
        session.close()

def fetch_airflow_role_id(role_name):
    """
    Fetch role id of a given role name using parameterized query.
    """
    result = _execute_query(
        "SELECT id FROM ab_role WHERE name = :role_name",
        {"role_name": role_name},
    )
    if not result:
        raise ValueError(f"Airflow role not found: {role_name}")
    logging.info("Fetched role ID successfully")
    return result[0]

def fetch_airflow_permission_id(permission_name):
    """
    Fetch permission id of a given permission using parameterized query.
    """
    result = _execute_query(
        "SELECT id FROM ab_permission WHERE name = :perm_name",
        {"perm_name": permission_name},
    )
    if not result:
        raise ValueError(f"Airflow permission not found: {permission_name}")
    logging.info("Fetched permission ID successfully")
    return result[0]

def fetch_airflow_menu_object_ids(dag_names):
    """
    Fetch view_menu IDs for a list of DAG resource names.
    Uses parameterized IN-clause via individual bind params.

    Parameters:
        dag_names: list of DAG resource names (e.g. ['DAG:my_dag1', 'DAG:my_dag2'])
    Returns:
        list of view_menu IDs
    """
    if not dag_names:
        return []
    # Build parameterized IN clause: :p0, :p1, :p2, ...
    param_names = [f":p{i}" for i in range(len(dag_names))]
    params = {f"p{i}": name for i, name in enumerate(dag_names)}
    in_clause = ", ".join(param_names)
    result = _execute_query(
        f"SELECT id FROM ab_view_menu WHERE name IN ({in_clause})",
        params,
    )
    logging.info(f"Fetched {len(result)} view menu IDs")
    return result

def fetch_perms_obj_association_ids(perm_id, view_menu_ids):
    """
    Fetch permission_view IDs for a permission and list of view_menu IDs.
    Uses parameterized query.
    """
    if not view_menu_ids:
        return []
    param_names = [f":vm{i}" for i in range(len(view_menu_ids))]
    params = {f"vm{i}": vm_id for i, vm_id in enumerate(view_menu_ids)}
    params["perm_id"] = perm_id
    in_clause = ", ".join(param_names)
    result = _execute_query(
        f"SELECT id FROM ab_permission_view WHERE permission_id = :perm_id AND view_menu_id IN ({in_clause})",
        params,
    )
    logging.info(f"Fetched {len(result)} permission-view association IDs")
    return result

def fetch_dag_ids_by_tag(tag_name):
    """
    Fetch DAG IDs with a given tag name using parameterized query.
    """
    result = _execute_query(
        "SELECT DISTINCT dag_id FROM dag_tag WHERE name = :tag_name",
        {"tag_name": tag_name},
    )
    logging.info(f"Fetched {len(result)} DAG IDs for tag")
    return result

def associate_permission_to_object(perm_id, view_menu_ids):
    """
    Associate permission to view_menu objects (DAGs) using parameterized INSERT.
    """
    session = settings.Session()
    try:
        for vm_id in view_menu_ids:
            session.execute(
                text(
                    "INSERT INTO ab_permission_view (permission_id, view_menu_id) "
                    "VALUES (:perm_id, :vm_id) "
                    "ON CONFLICT (permission_id, view_menu_id) DO NOTHING"
                ),
                {"perm_id": perm_id, "vm_id": vm_id},
            )
        session.commit()
        logging.info(f"Associated permission to {len(view_menu_ids)} view menus")
    except Exception as e:
        session.rollback()
        logging.error(f"Error associating permission to objects: {type(e).__name__}: {e}")
        raise
    finally:
        session.close()

def associate_permission_to_role(permission_view_ids, role_id):
    """
    Associate permission_view entries to a role using parameterized INSERT.
    """
    session = settings.Session()
    try:
        for pv_id in permission_view_ids:
            session.execute(
                text(
                    "INSERT INTO ab_permission_view_role (permission_view_id, role_id) "
                    "VALUES (:pv_id, :role_id) "
                    "ON CONFLICT (permission_view_id, role_id) DO NOTHING"
                ),
                {"pv_id": pv_id, "role_id": role_id},
            )
        session.commit()
        logging.info(f"Associated {len(permission_view_ids)} permissions to role")
    except Exception as e:
        session.rollback()
        logging.error(f"Error associating permissions to role: {type(e).__name__}: {e}")
        raise
    finally:
        session.close()

def validate_if_permission_granted(permission_view_ids, role_id):
    """
    Validate if given permissions are associated to given role using parameterized query.
    """
    if not permission_view_ids:
        return []
    param_names = [f":pv{i}" for i in range(len(permission_view_ids))]
    params = {f"pv{i}": pv_id for i, pv_id in enumerate(permission_view_ids)}
    params["role_id"] = role_id
    in_clause = ", ".join(param_names)
    result = _execute_query(
        f"SELECT id FROM ab_permission_view_role "
        f"WHERE permission_view_id IN ({in_clause}) AND role_id = :role_id",
        params,
    )
    logging.info(f"Validated {len(result)} permission grants")
    return result

def clean_up_existing_dag_permissions_for_role(role_id):
    """
    Clean up existing DAG permissions for a given role using parameterized query.
    Note: this creates a brief window where the role has no DAG permissions.
    """
    _execute_query(
        "DELETE FROM ab_permission_view_role WHERE id IN ("
        "  SELECT pvr.id"
        "  FROM ab_permission_view_role pvr"
        "  INNER JOIN ab_permission_view pv ON pvr.permission_view_id = pv.id"
        "  INNER JOIN ab_view_menu vm ON pv.view_menu_id = vm.id"
        "  WHERE pvr.role_id = :role_id AND vm.name LIKE :dag_prefix"
        ")",
        {"role_id": role_id, "dag_prefix": "DAG:%"},
        fetch=False,
    )
    logging.info("Cleaned up existing DAG permissions for role")

def sync_permission(config_data):
    """
    Sync permissions based on the config.

    Parameters:
        config_data: dict with keys:
            - airflow_role_name: name of the custom Airflow role
            - managed_dags: list of DAG IDs to grant full management permissions on
              (can_read, can_edit, can_delete)
            - do_cleanup: if True, remove all existing DAG:* permissions first
    """
    # Get the role ID for role name
    role_id = fetch_airflow_role_id(config_data["airflow_role_name"])

    # Clean up existing DAG level permissions if requested
    if config_data.get("do_cleanup", False):
        clean_up_existing_dag_permissions_for_role(role_id)

    managed_dags = config_data.get("managed_dags", [])
    if not managed_dags:
        logging.info("No managed DAGs found, skipping permission sync")
        return

    # Determine which permissions to grant (default: can_read only)
    permissions = config_data.get("permissions", [Constants.CAN_READ])

    # Build DAG resource names (e.g. ["DAG:my_dag1", "DAG:my_dag2"])
    dag_resource_names = [f"DAG:{dag.strip()}" for dag in managed_dags]

    # Get IDs for DAG view_menu objects
    vm_ids = fetch_airflow_menu_object_ids(dag_resource_names)
    if not vm_ids:
        logging.info("No view_menu entries found for managed DAGs")
        return

    # Grant the configured permissions on each managed DAG
    all_perm_view_ids = []
    for perm_name in permissions:
        perm_id = fetch_airflow_permission_id(perm_name)
        associate_permission_to_object(perm_id, vm_ids)
        all_perm_view_ids += fetch_perms_obj_association_ids(perm_id, vm_ids)

    # Associate permission_view entries with the role and validate
    if all_perm_view_ids and role_id:
        associate_permission_to_role(all_perm_view_ids, role_id)
        validate_if_permission_granted(all_perm_view_ids, role_id)

def sync_permissions_with_tags(role_mappings):
    """
    For each role mapping, fetch DAG IDs by tag and sync permissions.
    """
    for role_map in role_mappings:
        username = list(role_map.keys())[0]
        airflow_role = role_map[username]["airflow_role"]
        edit_tag_name = role_map[username]["airflow_edit_tag"]

        config_data = {
            "airflow_role_name": airflow_role,
            "managed_dags": fetch_dag_ids_by_tag(edit_tag_name),
            "permissions": role_map[username].get("permissions", [Constants.CAN_READ]),
            "do_cleanup": role_map[username].get("do_cleanup", True),
        }
        logging.info(f"Syncing permissions for airflow role")
        sync_permission(config_data)
        logging.info("Completed permission sync for role")

"""
    Add new roles and permissions here.
    Format:
    {
       "<role_key>": {
            "airflow_role": <Custom Airflow role name to grant permissions to>,
            "airflow_edit_tag": <Airflow Tag Name - DAGs with this tag will be managed>,
            "permissions": <List of permissions to grant on each tagged DAG.
                Options: "can_read", "can_edit", "can_delete"
                Default: ["can_read"] if omitted>,
            "do_cleanup": <Set to True (recommended) to clean up existing DAG permissions>
        }
    },

    IMPORTANT - ROLE SETUP:
    When creating a new custom role (e.g. "analytics_reporting", "marketing_analyst")
    in the Airflow UI (Security > List Roles), you MUST copy the Viewer role's
    permissions into the new role. The Viewer permissions provide base UI access
    (browse DAGs, view logs, menu access, etc.). --or-- Assign the viewer role as well.
    Without them, users assigned to
    the custom role will not be able to log in to the Airflow UI.

    This DAG manages DAG-level permissions on DAG:xxx resources.
    Which permissions are granted is controlled by the "permissions" list
    in each config entry (options: can_read, can_edit, can_delete).
    It does NOT manage base UI permissions — those must be set up manually
    when creating the role.

    Steps to create a new custom role:
    1. Go to Security > List Roles > + (Add)
    2. Name it to match the "airflow_role" value in the config below
    3. Copy all permissions from the "Viewer" role into the new role
    4. Save — this DAG will then automatically add DAG-specific permissions
       (as configured in the "permissions" list) for each tagged DAG
"""
role_mappings = [
    {
        "analytics_reporting": {
            "airflow_role": "analytics_reporting",
            "airflow_edit_tag": "analytics_reporting_edit",
            "permissions": ["can_read", "can_edit", "can_delete"],
            "do_cleanup": True,
        }
    },
    {
        "marketing_analyst": {
            "airflow_role": "marketing_analyst",
            "airflow_edit_tag": "marketing_analyst_edit",
            "permissions": ["can_read", "can_edit", "can_delete"],
            "do_cleanup": True,
        },
    },
]

with DAG(
    dag_id=dag_id,
    schedule="*/15 * * * *",
    catchup=False,
    start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
) as dag:
    sync_dag_permissions_task = PythonOperator(
        task_id="sync_dag_permissions",
        python_callable=sync_permissions_with_tags,
        op_kwargs={"role_mappings": role_mappings},
    )

Step 3: Tag your DAGs for access control

Add appropriate tags to your DAGs to specify which roles should have access. Tags are used to define which roles have access to tagged DAGs.

# Example DAG for analytics_reporting
with DAG(
    "analytics_reporting_dag",
    description="Daily analytics reporting pipeline",
    schedule_interval="@daily",
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
    catchup=False,
    tags=["reporting", "analytics", "analytics_reporting_edit"]
) as dag:
    # DAG tasks here
    pass
    
    
# Example DAG for marketing_analyst
with DAG(
    "marketing_analyst_dag",
    description="Daily marketing lead analysis pipeline",
    schedule_interval="@daily",
    start_date=pendulum.datetime(2023, 1, 1, tz="UTC"),
    catchup=False,
    tags=["marketing", "analytics", "marketing_analyst_edit"]
) as dag:
    # DAG tasks here
    pass

In this example:

The analytics_reporting custom role will have read, edit, and delete access to the DAG analytics_reporting_dag (and other DAGs tagged with analytics_reporting_edit)
The marketing_analyst custom role will have read, edit, and delete access to the DAG marketing_analyst_dag (and other DAGs tagged with marketing_analyst_edit)

The exact permissions granted (can_read, can_edit, can_delete) are configurable per role in the role_mappings config inside the permission management DAG:

role_mappings = [
    {
        "analytics_reporting": {
            "airflow_role": "analytics_reporting",
            "airflow_edit_tag": "analytics_reporting_edit",
            "permissions": ["can_read", "can_edit", "can_delete"],
            "do_cleanup": True,
        }
    },
    {
        "marketing_analyst": {
            "airflow_role": "marketing_analyst",
            "airflow_edit_tag": "marketing_analyst_edit",
            "permissions": ["can_read", "can_edit", "can_delete"],
            "do_cleanup": True,
        },
    },
]

Note: Before this DAG can manage permissions for a custom role, the role must be created manually in the Apache Airflow UI (Security > List Roles) with the Viewer role’s permissions copied in. See Step 2 for details.

Step 4: Deploy and test

Upload both the permission management DAG and your tagged DAGs to your Amazon MWAA environment’s S3 bucket.
Wait for Amazon MWAA to detect and process the new DAGs.
Verify that the permission management DAG runs successfully.
Test access with different user roles to confirm proper permission enforcement.
Users can also integrate this with their CI/CD processes.

Troubleshooting

In this section, we cover some common issues and how to troubleshoot them.

Permission sync failures

Symptom: Permission sync DAG fails with database errors.

Cause: Insufficient permissions on MWAA execution role.

Solution: Ensure that the execution role has airflow:CreateWebLoginToken permission and database access.

Tags not being processed

Symptom: DAG tags are present but permissions aren’t updated.

Solution: Check that DAG is active and parsed successfully – Review permission sync DAG logs for processing errors.

Users cannot access expected DAGs

Symptom: Users with correct IAM roles cannot see DAGs

Solution: Confirm that IAM to Apache Airflow role mapping is correct. Verify that the permission sync DAG has run successfully. Check Amazon CloudWatch Logs for permission assignment errors.

Performance issues

Symptom: Permission sync takes too long or times out.

Solution: Reduce sync frequency for large environments. Consider batching permission updates. Monitor DAG execution time and optimize accordingly.

Debugging steps

Check Amazon MWAA environment health and connectivity
Review permission sync DAG execution logs
Verify IAM role configurations and trust relationships
Test with a single DAG to isolate issues
Monitor CloudWatch Logs for detailed error messages

Benefits and considerations

Automated permission management offers you significant operational advantages while enhancing your security. You will benefit from reduced administrative overhead as manual permission assignments are removed, so you can scale seamlessly without additional burden. Your security improves through consistent application of least-privilege principles and reduced human error. You will enhance your developer experience with automatic access provisioning that shortens onboarding time, while your system supports environments with over 500 DAGs without performance degradation.

When you implement these systems, you must adhere to key security practices. You should apply the principle of least privilege, validate tags to make sure that you’re only processing authorized tags, and establish comprehensive audit mechanisms including CloudTrail logging. Your access control measures should restrict permission management functions to administrators while you utilize appropriate role separation for different user personas.

You will need to consider several technical limitations during your implementation. IAM-based access control to Amazon MWAA works only with Apache Airflow default roles, not custom ones, though your tag-based permissions function with alternative authenticators. Permission changes propagate based on DAG schedules, potentially causing delays. You should establish approval processes for your production changes, maintain version control for permissions, and document your rollback procedures to ensure your system’s resilience and security.

Clean up

Clean up resources after your experimentation:

Delete the Amazon MWAA environments using the console or AWS CLI.
Update the IAM role policy or delete the IAM role if not needed.

Conclusion

In this post, you learned how to automate DAG permission management in Amazon MWAA using Apache Airflow’s tagging system. You saw how to implement tag-based access control that scales efficiently, reduces manual errors, and maintains least-privilege security principles across hundreds of DAGs. You also explored the key security practices and technical considerations that you need to keep in mind during implementation.

Try out this solution in your Amazon MWAA environment to streamline your permission management. Start by implementing the tagging system in a development environment, then gradually roll it out to production as your team becomes comfortable with the approach.

About the authors

Secure multi-warehouse Amazon Redshift access behind a Network Load Balancer using Microsoft Entra ID

Raghu Kuppala — Mon, 30 Mar 2026 17:46:40 +0000

As data analytics workloads scale, organizations face two challenges. First, they must deliver high-performance analytics at massive scale while maintaining secure access across diverse tools. Second, they must manage high-concurrency workloads while integrating with existing identity management systems.

You can address these challenges by using Amazon Redshift Serverless endpoints behind an AWS Network Load Balancer with Microsoft Entra ID federation. This architecture can authenticate while helping to streamline identity management across your data environment. Amazon Redshift Serverless provides petabyte-scale analytics with auto scaling capabilities, enabling high-concurrency workloads while streamlining user authentication and authorization.

In this post, we show you how to configure a native identity provider (IdP) federation for Amazon Redshift Serverless using Network Load Balancer. You will learn how to enable secure connections from tools like DBeaver and Power BI while maintaining your enterprise security standards.

Solution overview

The following diagram shows the architecture.

Figure 1: Sample architecture diagram

In this architecture:

A central Amazon Redshift ETL data warehouse shares data to multiple Amazon Redshift Serverless workgroups using Amazon Redshift data sharing.
Each workgroup has a dedicated managed Amazon Virtual Private Cloud (Amazon VPC) endpoint.
A Network Load Balancer sits in front of all VPC endpoints, providing a single connection point.
Users connect from DBeaver or Power BI through the Network Load Balancer and authenticate using their Microsoft Entra ID credentials.

This setup works whether you’re validating the concept with a single workgroup today or planning to scale to multiple workgroups in the future.

Prerequisites

Before you begin, make sure that you have completed these prerequisites.

Create Amazon Redshift Serverless endpoints.
Set up datashare from producer to Amazon Redshift Serverless endpoints.
Create Amazon Redshift-managed VPC endpoints.
Create a Network Load Balancer.
Configure a domain name.
Set up Amazon Redshift native IdP federation with Microsoft Entra ID.
Gather the following from your registered application in Microsoft Entra ID:
1. Scope (API-Scope)
2. Azure Client ID (AppID from App Registration Details)
3. IdP Tenant (Tenant ID from App Registration Details)
Download and install the latest Amazon Redshift JDBC and ODBC drivers.

This solution uses the following AWS services.

Implementation steps

This section covers configuring the Network Load Balancer, setting up an ACM certificate, creating custom domain names in Amazon Redshift, configuring DNS records in Amazon Route 53, and connecting your JDBC and ODBC clients using Microsoft Entra ID authentication.

1. Configure the Network Load Balancer

First, collect the private IP addresses for your Amazon Redshift-managed VPC endpoints:

Open the Amazon Redshift Serverless console.
Choose your workgroup.
Note the private IP address of your Redshift-managed VPC endpoint.
Repeat for each Amazon Redshift Serverless endpoint that you want to add to the Network Load Balancer.

Figure 2: Amazon Redshift managed VPC endpoint

Next, create a target group for your endpoints:

Open the Amazon Elastic Compute Cloud (Amazon EC2) console.
Choose Target Groups.
Choose Create target group.
Configure the target group:
- For Target type, choose IP addresses.
- For Target group name, enter rs-multicluster-tg.
- For Protocol, choose TCP.
- For Port, enter 5439 (Note: You can find your specific port number in the Redshift endpoint connection details. If you haven’t modified it, use the default port 5439.).
- For VPC, select your VPC.
- Choose Next.
Figure 3: create target group in NLB

Figure 4: NLB target group creation

Add a listener to your Network Load Balancer:

In the EC2 console, choose Load Balancers.
Select your Network Load Balancer.
In the Listeners tab, choose Add listener.
Configure the listener:
- For Protocol, choose TCP.
- For Port, enter 5439.
- For Default action, choose rs-multicluster-tg.
Choose Add listener.

Figure 5: NLB listener properties.

2. Configure AWS Certificate Manager (ACM)

For this example, we use myexampledomain.com as a custom domain. Replace it with your own domain name before you begin.Follow these steps to request and configure your certificate:

Request a certificate in AWS Certificate Manager (ACM):
- Open the AWS Certificate Manager console.
- Choose Request Certificate.
- Choose Request Public certificate.
- Choose Next.
Configure the certificate:
- Add two domain names:
  - Network Load Balancer CNAME: dev-redshift.myexampledomain.com
  - Wildcard domain: *.redshift.myexampledomain.com
- For Validation method, choose DNS validation.
- Choose Request.
For enhanced security, we recommend adding individual Amazon Redshift Serverless CNAMEs instead of using wildcards (*). This example uses DNS validation in AWS Certificate Manager, which requires creating CNAME records to prove domain control.

Figure 6: AWS Certificate Manager (ACM) certificate creation
Validate the certificate:
- Your AWS Certificate Manager (ACM) certificate initially shows a ‘Pending validation’ status.
- Wait for the status to change to ‘Issued’ before proceeding.
- You must have an ‘Issued’ status before creating Amazon Redshift custom domain names.
Figure 7: Sample issued AWS Certificate Manager (ACM) certificate

3. Configure Amazon Redshift custom domain names

Create a custom domain name:
- Open the Amazon Redshift Serverless console.
- Select your workgroup.
- From Actions, choose Create custom domain name.
Figure 8: Amazon Redshift custom domain name creation
Configure the domain settings:
- For Custom domain name, enter cluster-02.redshift.myexampledomain.com.
- For ACM certificate, select the certificate you created for dev-redshift.myexampledomain.com.
- Choose Create.
Figure 9: Amazon Redshift custom domain name creation
Verify that the custom domain name appears in your workgroup.

Figure 10: Amazon Redshift custom domain name
Repeat steps 1–3 for each remaining Amazon Redshift Serverless endpoint that you want to add to the Network Load Balancer. Use a unique custom domain name for each endpoint (for example, cluster-03.redshift.myexampledomain.com, cluster-04.redshift.myexampledomain.com) and select the same ACM certificate that you created earlier.

4. Configure Amazon Route 53

Amazon Route 53 maps your custom domain name to the correct Amazon Redshift endpoint, making it reachable by name rather than a system-generated address. Without it, clients have no way to resolve your custom domain and AWS Certificate Manager can’t verify domain ownership to enable secure connections.First, create a CNAME record for your Network Load Balancer:

Get the Network Load Balancer DNS name:
- Open the Amazon EC2 console.
- Choose Load Balancers.
- Select your Network Load Balancer.
- Copy the DNS name.
Figure 11: NLB DNS name
Create Route 53 records:
- Open the Amazon Route 53 console.
- Choose Hosted Zones.
- Select myexampledomain.com.
- Choose Create record.
- Configure the record:
  - For Record name, enter dev-redshift.myexampledomain.com.
  - For Record type, choose A – Routes traffic to an IPv4 address and some AWS resources.
  - For Alias, choose Yes.
  - For Route traffic to, choose Alias to Network Load Balancer.
  - Select your AWS Region and Network Load Balancer DNS name.
  - For Routing policy, choose Simple routing.
  - Choose Create records.
Figure 12: NLB – A record in route 53

Figure 13: NLB – A record in Route 53
Create the AWS Certificate Manager (ACM) validation CNAME:
- Open AWS Certificate Manager.
- Select your certificate for dev-redshift.myexampledomain.com.
- Copy the CNAME name and CNAME value.
- Return to Route 53.
- Create a CNAME record in your myexampledomain.com hosted zone using the values from AWS Certificate Manager (ACM).
- Choose Create records.
Figure 14: NLB – CNAME record in Route 53

5. Configure Amazon Redshift JDBC and ODBC drivers with native IdP

The JDBC and ODBC driver configuration connects your client applications to Amazon Redshift through the Network Load Balancer using your Microsoft Entra ID credentials for authentication. Configuring both drivers allows any tool, whether DBeaver using JDBC or Power BI using ODBC, to authenticate through the same identity provider and reach the correct Amazon Redshift endpoint through a single connection point.

JDBC driver setup in DBeaver

Create a new Amazon Redshift connection:
- Host: dev-redshift.myexampledomain.com (NLB CNAME).
- Database: dev.
- Authentication: Database Native.
- Username: login id for a user account.
Figure 15: Amazon Redshift JDBC driver setup
Configure driver properties:
- plugin_name: com.amazon.redshift.plugin.BrowserAzureOAuth2CredentialsProvider.
- sslmode: verify-ca.
Add user driver properties:
- client_id: [Your Microsoft Entra ID application client ID].
- idp_tenant: [Your Microsoft Entra ID tenant].
- listen_port: 7890.
- loginTimeout: 60.
- scope: [Your Microsoft Entra ID application scope].
Figure 16: Amazon Redshift JDBC driver user properties

ODBC driver setup

Configure the system DSN:
- Open ODBC Data Source Administrator (64-bit).
- Choose System DSN.
- Choose Add.
- Select Amazon Redshift ODBC Driver (x64) 2.01.04.00.
- Choose Finish.
Configure connection settings:
- Data Source Name: dev-redshift.
- Server: dev-redshift.myexampledomain.com.
- Port: 5439.
- Database: dev.
- Auth type: Identity Provider: Browser Azure AD OAUTH2.
- Scope: [Your Microsoft Entra ID application scope].
- Azure Client ID: [Your Microsoft Entra ID application client ID].
- IdP Tenant: [Your Microsoft Entra ID application tenant].
Figure 17: Amazon Redshift ODBC driver properties
Configure SSL settings:
- SSL Mode: verify-ca.
- Choose Save.
Figure 18: Amazon Redshift ODBC driver properties

6. Validate connectivity

Test DBeaver connection

After configuring the JDBC driver properties, choose Test Connection.
Authenticate through the Microsoft login in your browser.
Verify that you receive a success message.
Confirm successful connection using Native IdP through the Network Load Balancer.

Figure 19: Microsoft Entra id authentication

Figure 20: Successful Microsoft Entra id authentication

Figure 21: Successful Amazon Redshift authentication

Test power BI desktop connection

Launch Power BI Desktop:
- Choose Get data.
- Choose More.
- Under Other, select ODBC.
- Choose Connect.
Figure 22: Power BI desktop connectivity using Amazon Redshift ODBC driver

Figure 23: Power BI desktop connectivity using Amazon Redshift ODBC driver
Configure the connection:
- Select dev-redshift from the Data source name.
- Choose OK.
- Complete Microsoft Entra ID authentication in your browser.
Figure 24: Power bi desktop connectivity using Amazon Redshift odbc driver

Figure 25: Successful Microsoft Entra id authentication
Test the connection:
- From Navigator, choose schema tpcds.
- Select date_dim.
- Choose Load.
- Verify that you can analyze your Amazon Redshift data in Power BI Desktop.
Figure26: Power BI desktop connected to Amazon Redshift and schema browsing

Figure 27: Power BI desktop fetching data from date_dim table

Cleaning up

To avoid ongoing charges, delete the following resources:

Delete the Amazon Redshift data warehouses (provisioned cluster or serverless workgroup and namespace) and the VPC endpoints that you created.
Delete the certificate that you created in AWS Certificate Manager (ACM).
Delete the Network Load Balancer.

Conclusion

In this post, we showed you how to integrate Amazon Redshift Serverless with Microsoft Entra ID using an AWS Network Load Balancer as a single connection endpoint across multiple workgroups. As your data analytics use cases grow, you can continue to scale horizontally by adding new workgroups behind the same Network Load Balancer without changing your users’ connection settings or authentication experience.

For more information about extending and scaling this solution, see the following resources:

AWS Blogs

About the authors

Securely connect Kafka client applications to your Amazon MSK Serverless cluster from different VPCs and AWS accounts

Subham Rakshit — Mon, 30 Mar 2026 17:44:56 +0000

Amazon MSK Serverless is a cluster type for Amazon MSK that you can use to run Apache Kafka without having to manage and scale cluster capacity. It automatically provisions and scales capacity while managing the partitions in your topics, so you can stream data without thinking about right-sizing or scaling clusters. MSK Serverless is fully compatible with Apache Kafka, so you can use any compatible client applications to produce and consume data.

MSK Serverless uses AWS PrivateLink to provide private connectivity up to five virtual private clouds (VPCs) within the same AWS account. However, if you need cross-VPC connectivity beyond five VPCs or cross-account connectivity, you typically need VPC peering or AWS Transit Gateway, as explained in Secure connectivity patterns for Amazon MSK Serverless cross-account access.

Aklivity Zilla Plus for Amazon MSK is a stateless Kafka-native edge proxy that enables authorized Kafka clients deployed across VPCs (even cross-account) to securely connect, publish messages, and subscribe to topics in your MSK Serverless cluster using a custom domain name.

For more details on supporting SASL/SCRAM authentication with a custom domain, see Configure a custom domain name for your Amazon MSK cluster.

In this post, we show you how Kafka clients can use Zilla Plus to securely access your MSK Serverless clusters through Identity and Access Management (IAM) authentication over PrivateLink, from as many different AWS accounts or VPCs as needed. We also show you how the solution provides a way to support a custom domain name for your MSK Serverless cluster.

Secure private access to one MSK Serverless cluster

Network Load Balancers (NLBs) provide a convenient way to define remote connectivity to MSK Serverless clusters from other VPCs. In the following architecture diagram, Zilla Plus is deployed in an auto scaling group, reachable as a target group behind an NLB. Zilla Plus connects to an MSK Serverless cluster through the (rightmost) VPC endpoint associated directly with the MSK Serverless cluster. Zilla Plus is configured to use an AWS Certificate Manager (ACM) wildcard certificate for your custom domain. By creating a Zilla Plus VPC Endpoint Service, you make the MSK Serverless cluster reachable from other VPCs through Zilla Plus.

As shown in the preceding figure, the client VPC has minimal configuration, consisting of a Zilla Plus VPC endpoint to reach the Zilla Plus VPC Endpoint Service, and an Amazon Route 53 local zone mapping your custom domain name to the Zilla Plus VPC endpoint.

How the custom domain works across VPCs for MSK Serverless

When an MSK Serverless cluster is created, it is associated with a bootstrap broker address like this:boot-xxxxxxxx.yy.kafka-serverless.region.amazonaws.com:9098. However, this address is only resolvable within the originating VPC.

To access the cluster from another VPC or account, Kafka clients connect to a custom domain exposed by Zilla Plus, such as boot.my.custom.domain:9098. The Route 53 DNS in the client VPC maps this custom domain to a VPC endpoint (NLB), while the NLB forwards traffic to Zilla Plus, which presents the appropriate ACM wildcard certificate. When a Kafka client needs to bootstrap connectivity to a Kafka cluster (such as an MSK Serverless cluster), the client must follow a two-step discovery process to learn the specific addresses of the brokers in the cluster, so it can then connect to each broker directly as needed.

For example, if the client needs to produce messages to a specific Kafka topic such as my-messages, then the client first uses a bootstrap server address to connect to any broker in the Kafka cluster, requesting topic metadata that includes the address of each broker responsible for storage of messages in the my-messages topic. In the second step, the client connects directly to the corresponding brokers for the my-messages topic to produce messages. The sequence of connection flow between Kafka client and broker is shown below.

When the Kafka client connection for the custom domain bootstrap server arrives at the Zilla Plus VPC NLB, it’s routed to any of the Zilla Plus instances in the target group. Zilla Plus presents the wildcard TLS certificate for the custom domain and completes the TLS handshake before establishing connectivity to the MSK Serverless bootstrap server. Kafka protocol requests flow from the client through Zilla Plus to the MSK Serverless bootstrap server. When the metadata request is made by the Kafka client, Zilla Plus intercepts the metadata response and rewrites the discovered broker addresses advertised to the client, mapping them to the custom domain.

When the Kafka client connections for each individual broker address arrive at Zilla Plus, the broker-specific custom domain address is mapped to the broker-specific MSK Serverless address so that the client connects to the requested broker in the cluster. Even though the MSK Serverless cluster can have any number of advertised broker addresses, the number of instances in the Zilla Plus target group isn’t required to match. Each Zilla Plus instance can relay broker-specific custom domain connectivity for any broker in the MSK Serverless cluster. Because no configuration changes are required at the MSK Serverless cluster to enable the Zilla Plus custom domain mapping, there’s no impact on other Kafka clients already connecting directly to the MSK Serverless cluster using the AWS-generated bootstrap server.

Follow the guided steps in the Aklivity Zilla Plus documentation to deploy this solution using the AWS Cloud Development Kit (AWS CDK). This automates the setup for you, including the client VPC configuration to create the VPC endpoint and Route 53 DNS entries.

After the secure private access and secure private access client scenarios have been deployed successfully, you can verify remote access to the MSK Serverless cluster from any Kafka client using your custom domain bootstrap server.

Secure private access to multiple MSK Serverless clusters

When a Kafka client needs to bootstrap to multiple different custom domain MSK Serverless clusters, the approach described previously keeps the client VPC configuration relatively straightforward.

As shown in the preceding figure, each custom domain has a single Route53-hosted zone wildcard DNS record aliased to the corresponding local VPC endpoint for the corresponding remote MSK Serverless cluster. When the Kafka client performs bootstrap, local DNS resolution for the custom domain bootstrap server hostname routes connectivity to the correct VPC endpoint and the TLS certificate presented validates trust for the custom domain hostname too. Connectivity to individual broker addresses in the same custom domain are routed and trusted in the same way.

Secure private to MSK Serverless clusters through AWS Client VPN

When on-premises Kafka clients need to access an MSK Serverless cluster, the client VPC can be associated with an AWS Client VPN endpoint to connect through AWS Client VPN, as shown in the following figure.

By configuring the AWS Client VPN endpoint to use the client VPC DNS server, the AWS Client VPN connections will automatically resolve the custom domain bootstrap server hostname and connect through Zilla Plus to MSK Serverless.

Conclusion

You can use Amazon MSK Serverless clusters to run Apache Kafka without having to manage and scale cluster capacity. With Zilla Plus for Amazon MSK, you can access one or more of your Amazon MSK Serverless clusters from one or more remote client VPCs using a custom domain for each MSK Serverless cluster. The remote client VPCs can also belong to different AWS accounts, while still enforcing fine-grained AWS Identity and Access Management (IAM) authorization for topics and consumer groups. On-premises clients can also use this approach to connect to an MSK Serverless cluster through AWS Client VPN from a different AWS account.

Zilla Plus requires no configuration changes to your MSK Serverless cluster, so adding a custom domain for remote Kafka clients has no impact on existing Kafka clients—including MSK Connect, MSK Replicator, or other MSK Integrations—that connect directly to your MSK Serverless cluster.

Learn more about Zilla Plus for Amazon MSK on AWS Marketplace and the Aklivity Zilla Plus documentation.

About the authors

Build AWS Glue Data Quality pipeline using Terraform

Viquar Khan — Thu, 26 Mar 2026 15:35:21 +0000

AWS Glue Data Quality is a feature of AWS Glue that helps maintain trust in your data and support better decision-making and analytics across your organization. It allows users to define, monitor, and enforce data quality rules across their data lakes and data pipelines. With AWS Glue Data Quality, you can automatically detect anomalies, validate data against predefined rules, and generate quality scores for your datasets. This feature provides flexibility in how you validate your data – you can incorporate quality checks into your ETL processes for transformation-time validation, or validate data directly against cataloged tables for ongoing data lake monitoring. By leveraging machine learning, it can also suggest data quality rules based on your data patterns.

You can use Terraform, an open source Infrastructure as Code (IaC) tool developed by HashiCorp, to deploy AWS Glue Data Quality pipelines.

It allows developers and operations teams to define, provision, and manage cloud infrastructure using a declarative language. With Terraform, you can version, share, and reuse your infrastructure code across multiple cloud providers and services. Its powerful state management and planning capabilities enable teams to collaborate efficiently and maintain consistent infrastructure across different environments.

Using Terraform to deploy AWS Glue Data Quality pipeline enables IaC best practices to ensure consistent, version controlled and repeatable deployments across multiple environments, while fostering collaboration and reducing errors due to manual configuration.

In this post, we explore two complementary methods for implementing AWS Glue Data Quality using Terraform:

ETL-based Data Quality – Validates data during ETL (Extract, Transform, Load) job execution, generating detailed quality metrics and row-level validation outputs
Catalog-based Data Quality – Validates data directly against Glue Data Catalog tables without requiring ETL execution, ideal for monitoring data at rest

Solution overview

This post demonstrates how to implement AWS Glue Data Quality pipelines using Terraform using two complementary approaches mentioned above to ensure comprehensive data quality across your data lake.

We’ll use the NYC yellow taxi trip data, a real-world public dataset, to illustrate data quality validation and monitoring capabilities. The pipeline ingests parquet-formatted taxi trip data from Amazon Simple Storage Service (Amazon S3) and applies comprehensive data quality rules that validate data completeness, accuracy, and consistency across various trip attributes.

Method 1: ETL-based Data Quality

ETL-based Data Quality validates data during Extract, Transform, Load (ETL) job execution. This approach is ideal for:

Validating data as it moves through transformation pipelines
Applying quality checks during data processing workflows
Generating row-level validation outputs alongside transformed data

The pipeline generates two key outputs:

Data Quality Results – Detailed quality metrics and rule evaluation outcomes stored in the dqresults/ folder, providing insights into data quality trends and anomalies
Row-Level Validation – Individual records with their corresponding quality check results written to the processed/ folder, enabling granular analysis of data quality issues

Method 2: Catalog-based Data Quality

Catalog-based Data Quality validates data quality rules directly against AWS Glue Data Catalog tables without requiring ETL job execution. This approach is ideal for:

Validating data at rest in the data lake
Running scheduled data quality checks independent of ETL pipelines
Monitoring data quality across multiple tables in a database

Architecture overview

The following diagram illustrates how both approaches work together to provide comprehensive data quality validation:

Source data stored in Amazon S3 (Yellow Taxi Data)
AWS Glue ETL processes data with quality checks
ETL validation results are stored in S3
AWS Glue Crawler discovers schema
Metadata is stored in AWS Glue Catalog
AWS Glue Data Quality validates catalog tables
Catalog validation results are stored in S3
Amazon CloudWatch monitors all operations

By using AWS Glue’s serverless ETL capabilities and Terraform’s infrastructure-as-code approach, this solution provides a scalable, maintainable, and automated framework for ensuring data quality in your analytics pipeline.

Prerequisites:

An AWS account with AWS CLI installed and configured
HashiCorp Terraform installed and configured

Solution Implementation

Complete the following steps to build AWS Glue Data Quality pipeline using Terraform:

Clone the Repository

This post includes a GitHub repository that generates the following resources when deployed. To clone the repository, run the following command in your terminal:

git clone https://github.com/aws-samples/sample-build-aws-glue-data-quality-pipeline-using-terraform.git
cd sample-build-aws-glue-data-quality-pipeline-using-terraform

Core Infrastructure:

Amazon S3 bucket: glue-data-quality-{AWS AccountID}-{env} with AES256 encryption
Sample NYC taxi dataset (sample-data.parquet) automatically uploaded to the data/ folder
AWS Identity and Access Management (IAM) role: aws-glue-data-quality-role-{env} with Glue execution permissions and S3 read/write access
CloudWatch dashboard: glue-data-quality-{env} for monitoring job execution and data quality metrics
CloudWatch Log Groups for job logging with configurable retention

ETL-Based Data Quality Resources:

AWS Glue ETL job: data-quality-pipeline with 8 comprehensive validation rules
Python script: GlueDataQualityDynamicRules.py stored in glue-scripts/ folder
Results storage in dqresults/ folder with detailed rule outcomes
Row-level validation outputs in processed/ folder
Optional scheduled triggers for automated execution
CloudWatch alarm: etl-glue-data-quality-failure-{env} for monitoring job failures

Catalog-Based Data Quality Resources (Optional – when catalog_dq_enabled = true):

Glue Database: {catalog_database_name} for catalog table management
Glue Crawler: {job_name}-catalog-crawler for automatic schema discovery from S3 data
Crawler schedule trigger for automated execution (default: daily at 4 AM)
Glue Catalog Tables automatically discovered and created by the crawler
Catalog Data Quality job: {job_name}-catalog with 7 catalog-specific validation rules
Python script: CatalogDataQuality.py for catalog validation
Results storage in catalog-dq-results/ folder partitioned by table name
Catalog DQ schedule trigger for automated validation (default: daily at 6 AM)
CloudWatch alarm: catalog-glue-data-quality-failure-{env} for monitoring catalog job failures
Enhanced CloudWatch dashboard widgets for crawler status and catalog metrics

Review the Glue Data Quality Job Script

Review the Glue Data Quality job script GlueDataQualityDynamicRules.py located in the folder scripts, which has the following rules:

Rules = [
    CustomSql "select vendorid from primary where passenger_count > 0" with threshold > 0.9,
    Mean "trip_distance" < 150,
    Sum "total_amount" between 1000 and 100000,
    RowCount between 1000 and 1000000,
    Completeness "fare_amount" > 0.9,
    DistinctValuesCount "ratecodeid" between 3 and 10,
    DistinctValuesCount "pulocationid" > 100,
    ColumnCount = 19
]

Brief explanation of rules for NY Taxi data is as follows:

Rule Type	Condition	Description
CustomSql	“select vendorid from primary where passenger_count > 0” with threshold > 0.9	Checks if at least 90% of rides have at least one passenger
Mean	“trip_distance” < 150	Ensures the average trip distance is less than 150 miles
Sum	“total_amount” between 1000 and 100000	Verifies that total revenue from all trips falls within this range
RowCount	between 1000 and 1000000	Checks if the dataset has between 1,000 and 1 million records
Completeness	“fare_amount” > 0.9	Ensures over 90% of records have a fare amount
DistinctValuesCount	“ratecodeid” between 3 and 10	Verifies rate codes fall between 3-10 unique values
DistinctValuesCount	“pulocationid” > 100	Checks if there are over 100 unique pickup locations
ColumnCount	19	Validates that dataset has exactly 19 columns

These rules together ensure data quality by validating volume, completeness, reasonable values and proper structure of the taxi trip data.

Configure Terraform Variables

Before deploying the infrastructure, configure your Terraform variables in the terraform.tfvars file located in the examples directory. This configuration determines which features will be deployed – ETL-based Data Quality only, or both ETL-based and Catalog-based Data Quality.

Basic Configuration

The solution uses default values for most settings, but you can customize the following in your terraform.tfvars file:

AWS Region – The AWS region where resources will be deployed
Environment – Environment identifier (such as, “dev”, “prod”) used in resource naming
Job Name – Name for the Glue job (default: data-quality-pipeline)

Enable Catalog-Based Data Quality

By default, the solution deploys only ETL-based Data Quality. To enable Catalog-based Data Quality validation, add the following configuration to your terraform.tfvars file:

# Enable Catalog-based Data Quality
catalog_dq_enabled = true

# Glue Database name for catalog tables
catalog_database_name = "taxi_data_catalog"

# S3 paths containing parquet data for catalog table creation
s3_data_paths = ["data/"]

# Optional: Specific table names to validate (empty = all tables in database)
catalog_table_names = []

# Data quality rules for catalog validation
catalog_dq_rules = <<EOF
Rules = [
  RowCount > 0,
  Completeness "vendorid" > 0.9,
  Completeness "passenger_count" > 0.95,
  Mean "trip_distance" < 150,
  ColumnCount > 5
]
EOF

# Enable scheduled execution for catalog data quality
catalog_enable_schedule = false
catalog_schedule_expression = "cron(0 6 * * ? *)"

# Crawler schedule for automatic table discovery
catalog_crawler_schedule = "cron(0 4 * * ? *)"

Configuration Notes:

catalog_dq_enabled – Set to true to enable Catalog-based validation alongside ETL-based validation,which will deploy both ETL and Catalog validation
catalog_database_name – Name of the Glue database that will be created for catalog tables
s3_data_paths – S3 folders containing parquet data that the Glue Crawler will discover
catalog_table_names – Leave empty to validate all tables, or specify specific table names
catalog_dq_rules – Define validation rules specific to catalog tables (can differ from ETL rules)
catalog_enable_schedule – Set to true to enable automatic scheduled execution
Schedule expressions – Use cron format for automated execution (crawler runs before DQ job)

Once you’ve configured your variables, save the terraform.tfvars file and proceed to the next step.

Set Up AWS CLI Authentication

Before you can interact with AWS services using the command line, you need to set up and authenticate the AWS CLI. This section guides you through the process of configuring your AWS CLI and verifying your authentication. Follow these steps to ensure you have the necessary permissions to access AWS resources.

Open your terminal or command prompt.
Set up authentication in the AWS CLI. You need administrator permissions to set up this environment.
```
aws configure
```

To test if your AWS CLI is working and you’re authenticated, run the following command:

aws sts get-caller-identity --output json

The output should look similar to the following:

{
   "UserId": "UUID123123:your_user",
  "Account": "111122223333",
"Arn": "arn:aws:sts::111122223333:assumed-role/some-role/your_user"
 }

Deploy with Terraform

Follow these steps to deploy your infrastructure using Terraform. This process will initialize your working directory, review planned changes, and apply your infrastructure configuration to AWS.

To deploy with Terraform, navigate to the examples folder by running the following command in your CLI from inside the repository

cd .\examples

Run the following bash commands:

terraform init

Initializes a Terraform working directory, downloads required provider plugins, and sets up the backend for storing state.

On success you will receive output Terraform has been successfully initialized!

terraform plan

Creates an execution plan, shows what changes Terraform will make to your infrastructure. This command doesn’t make any changes.

terraform apply

Deploys infrastructure and code to the AWS Account. By default, it asks for confirmation before making any changes. Use ‘terraform apply -auto-approve’ to skip the confirmation step.

When prompted with ‘Do you want to perform these actions?’, type ‘yes’ and press Enter to confirm and allow Terraform to execute the described actions.

Upon successful execution, the system will display ‘Apply complete!’ message.

Run the AWS Glue Data Quality Pipeline

After deploying the infrastructure with Terraform, you can validate data quality using two methods – ETL-based and Catalog-based. Each method serves different use cases and can be run independently or together.

Method 1: Run the ETL-Based Data Quality Job

ETL-based data quality validates data during the transformation process, making it ideal for catching issues early in your data pipeline.

Steps to execute:

Navigate to the AWS Glue Console and select ETL Jobs from the left navigation panel
Locate and select the job named data-quality-pipeline
Choose Run to start the job execution
Monitor the job status – it typically completes in 2-3 minutes

Review the results:

Once completed, click on the Data Quality tab to review the validation results.
The following screenshot shows the results.

Understanding AWS Glue Data Quality Results: NYC Taxi Data Example.

Rule Results Summary

We had 8 total rules
7 rules passed
1 rule failed

Rule	Rule Condition	Status	Pass/Fail Reason
Passenger Count Check	At least 90% of rides should have at least one passenger	Passed	95% of rides had passengers, exceeding 90% threshold
Trip Distance	Average trip < 150 miles	Passed	Average was 5.94 miles, well below 150-mile limit
Row Count	Between 1,000 and 1,000,000 records	Passed	63,441 records fell within required range
Fare Amount Completeness	90% of records should have fare amounts	Passed	100% completeness exceeded 90% requirement
Rate Code Variety	Between 3-10 different rate codes	Passed	7 unique codes fell within acceptable range
Pickup Locations	More than 100 different pickup locations	Passed	205 locations exceeded minimum requirement
Column Count	Exactly 19 columns	Passed	Exact match at 19 columns
Total Amount Range	Sum of all fares between $1,000 and $100,000	Failed	Total of $130,638.29 exceeded maximum limit

Check the S3 bucket for detailed outputs:
- Data Quality metrics: s3://glue-data-quality-{AccountID}-{env}/dqresults/
- Row-level validation: s3://glue-data-quality-{AccountID}-{env}/processed/

The job processes the NYC taxi data and applies all 8 validation rules during the ETL execution. You’ll see a quality score along with detailed metrics for each rule.

Method 2: Run the Catalog-Based Data Quality Pipeline

Catalog-based data quality validates data at rest in your data lake, independent of ETL processing. This method requires the Glue Crawler to first discover and catalog your data.

Run the Glue Crawler (first-time setup or when schema changes):
- Navigate to AWS Glue Console and select Crawlers
- Locate data-quality-pipeline-catalog-crawler
- Select data-quality-pipeline-catalog-crawler checkbox and click Run and wait for completion (1-2 minutes)
- Verify the table was created in your Glue database
Run the Catalog Data Quality Job:
- Navigate to the AWS Glue Console and select ETL Jobs from the left navigation panel
- Select the job named data-quality-pipeline-catalog
- Click Run job to execute the validation
- Monitor the job status until completion

Review the results:

Once completed, click on the Data Quality tab to review the validation results.

The following screenshot shows the results.

Rule Results Summary

We had 7 total rules
6 rules passed
1 rule failed

Rule	Rule Condition	Status	Pass/Fail Reason
Row Count	Row count should be greater than zero	Passed	63441 rows present in the source data file
Completeness “vendorid”	90% of records should have vendorid	Passed	100% completeness exceeded 90% requirement
Completeness “passenger_count”	95% of records should have vendorid	Passed	96% completeness exceeded 95% requirement
Mean “trip_distance”	Mean “trip_distance” < 150	Passed	trip_distance.Mean: 5.94 which is less than threshold 150
Sum “total_amount”	Sum “total_amount” between 1000 and 100000	Failed	total_amount.Sum: 1330638.29 which does not satisfy condition

Distinct vale count “ratecodeid”	DistinctValuesCount “ratecodeid” between 3 and 10	Passed	ratecodeid.DistinctValuesCount: 7, satisfies condition
Column Count	Greater than 5 columns	Passed	ColumnCount: 19, satisfies condition

Check the S3 bucket for detailed outputs s3://glue-data-quality-{AccountID}-{env}/catalog-dq-results/

Catalog vs ETL Data Quality Comparison

Feature	ETL Data Quality	Catalog Data Quality
Execution Context	Validates data during ETL job processing	Validates data against catalog tables at rest
Data Source	Reads directly from S3 files (parquet format)	Queries Glue Data Catalog tables
Results Location	s3://…/dqresults/	s3://…/catalog-dq-results/
Primary Use Case	Validate data quality during transformation pipelines	Monitor data lake quality independent of ETL workflows
Execution Trigger	Runs as part of Glue ETL job execution	Runs independently as scheduled Data Quality job
Scheduling	Configured via Glue job schedule or on-demand	Configured via Data Quality job schedule or on-demand
Table Discovery	Manual – requires explicit S3 path configuration	Automatic – Glue Crawler discovers schema and creates tables
Schema Management	Defined in ETL job script	Managed by Glue Data Catalog
Output Format	Data Quality metrics + row-level validation outputs	Data Quality metrics only
Best For	Catching issues early in data pipelines	Ongoing monitoring of data at rest in data lakes
Dependencies	Requires ETL job execution	Requires Glue Crawler to run first
CloudWatch Integration	Job-level metrics and logs	Data Quality-specific metrics and logs

Monitoring and Troubleshooting

Both data quality methods automatically send metrics and logs to Amazon CloudWatch. You can set up alarms to notify you when quality scores drop below acceptable thresholds.

Clean up

To avoid incurring unnecessary AWS charges, make sure to delete all resources created during this tutorial. Ensure you have backed up any important data before running these commands, as this will permanently delete the resources and their associated data. To destroy all resources created as part of this blog, run following command in your terminal:

terraform destroy

Conclusion

In this blog post, we demonstrated how to build and deploy a scalable data quality pipeline using AWS Glue Data Quality and Terraform. The solution implements two validation methods:

ETL-based Data Quality – Integrated validation during ETL job execution for transformation pipeline quality assurance
Catalog-based Data Quality – Independent validation against Glue Data Catalog tables for data lake quality monitoring

By implementing data quality checks on NYC taxi trip data, we showed how organizations can automate their data validation processes and maintain data integrity at scale. The combination of AWS Glue’s serverless architecture and Terraform’s infrastructure-as-code capabilities provides a powerful framework for implementing reproducible, version-controlled data quality solutions. This approach not only helps teams catch data issues early but also enables them to maintain consistent data quality standards across different environments. Whether you’re dealing with small datasets or processing massive amounts of data, this solution can be adapted to meet your organization’s specific data quality requirements. As data quality continues to be a crucial aspect of successful data initiatives, implementing automated quality checks using AWS Glue Data Quality and Terraform sets a strong foundation for reliable data analytics and decision-making.

To learn more about AWS Glue Data Quality, refer to the following:

About the authors

Automating data classification in Amazon SageMaker Catalog using an AI agent

Ramesh H Singh — Tue, 24 Mar 2026 21:44:32 +0000

If you’re struggling with manual data classification in your organization, the new Amazon SageMaker Catalog AI agent can automate this process for you. Most large organizations face challenges with the manual tagging of data assets, which doesn’t scale and is unreliable. In some cases, business terms aren’t applied consistently across teams. Different groups name and tag data assets based on local conventions. This creates a fragmented catalog where discovery becomes unreliable and governance teams spend more time normalizing metadata than governing.

In this post, we show you how to implement this automated classification to help reduce the manual tagging effort and improve metadata consistency across your organization.

Amazon SageMaker Catalog provides automated data classification that suggests business glossary terms during data publishing. This helps to reduce the manual tagging effort and improve metadata consistency across organizations. This capability analyzes table metadata and schema information using Amazon Bedrock language models to recommend relevant terms from organizational business glossaries. Data producers receive AI-generated suggestions for business terms defined within their glossaries. These suggestions include both functional terms and sensitive data classifications such as PII and PHI, making it straightforward to tag their datasets with standardized vocabulary. Producers can accept or modify these suggestions before publishing, facilitating consistent terminology across data assets and improving data discoverability for business users.

The problem with manual classification

Manual tagging doesn’t scale effectively. Data producers interpret business terms differently, especially across domains. Critical labels like PII and PHI get missed because the publishing workflow is already complex. After assets enter the catalog with inconsistent terminology, search functionality and access controls quickly degrade.The solution isn’t only better training—it’s making the classification process predictable and consistent.

How automated classification works

The capability runs directly inside the publish workflow:

The catalog looks at the table’s structure—column names, types, whatever metadata exists.
That structure is sent to an Amazon Bedrock model that matches patterns against the organization’s glossary.
Producers receive a set of suggestions from the defined business glossary terms for classification that might include both functional and sensitive-data glossary terms.
They accept or adjust the suggestions before publishing.
The final list is written into the asset’s metadata using the controlled vocabulary.

The model evaluates column names, data types, schema patterns, and existing metadata. It maps those signals to the terms defined in the organization’s glossary. The suggestions are generated inline during publishing, with no separate Extract, Transform and Load (ETL) or batch processes to maintain. The accepted terms become part of the asset’s metadata and flow into downstream catalog operations immediately.

Under the hood: intelligent agent-based classification

Automated business glossary assignment goes beyond simple metadata lookups using a reasoning-driven approach. The AI agent functions like a virtual data steward, following human-like reasoning patterns such as:

Reviews asset details and context
Searches the catalog for relevant terms
Evaluates whether results make sense
Refines strategy if initial searches don’t surface appropriate terms
Learns from each step to improve recommendations

Key approaches:

Reasoning over static queries – The agent interprets asset attributes and context rather than treating metadata as a fixed index, generating dynamic search intents instead of relying on predefined queries.
Iterative adaptive search – When initial results are weak, the agent automatically adjusts queries—broadening, narrowing, or shifting terms through a feedback loop that helps improve discovery quality.
Structured semantic search – The agent performs semantic querying across entity types, applies filtering and relevance scoring, and conducts multi-directional exploration until strong matches are found.

This allows the agent to explore multiple directions until strong matches are found, improving recall and precision over static methods like direct vector search when asset metadata is incomplete or ambiguous.

Things to keep in mind

This feature is only as strong as the glossary it sits on top of. If the glossary is incomplete or inconsistent, the suggestions reflect that. Producers should still review each recommendation, especially for regulatory labels. Governance teams should monitor how often suggestions are accepted or overridden to understand model accuracy and glossary gaps.

Prerequisites

To follow along, you must have an Amazon SageMaker Unified Studio domain set up with a domain owner or domain unit owner permissions. You must have a project that you can use to publish assets. For instructions on setting up a new domain, refer to the SageMaker Unified Studio Getting started guide. We will also use Amazon Redshift to catalog data. If you are not familiar, read Learn Amazon Redshift concepts to learn more.

Step 1: Define business glossary and terms

AI recommendations suggest terms only from glossaries and definitions already present in the system. As a first step we create high-quality, well-described glossary entries so the AI can return accurate and meaningful suggestions.

We create the following business glossaries in our domain. For information about how to create a business glossary, see Create a business glossary in Amazon SageMaker Unified Studio.

Domain: Terms – Customer Profile, Policy, Order, Invoice.

The following is the view of ‘Domain’ business glossary with all terms added.

Data sensitivity: Terms – PII, PHI, Confidential, Internal.

The following is the view of ‘Data sensitivity’ business glossary with all terms added.

Business Unit: Terms – KYC, Credit Risk, Marketing Analytics

The following is the view of ‘Business Unit’ business glossary with all terms added.

We recommend that you use glossary descriptions to make terms unambiguous. Ambiguous or overlapping definitions confuse AI models and humans equally.

Step 2: Create data assets

Create the following table in Amazon Redshift. For information about how to bring Amazon Redshift data to Amazon SageMaker Catalog, see Amazon Redshift compute connections in Amazon SageMaker Unified Studio.

CREATE TABLE  dev.public.customer_analytics_data (
    customer_id VARCHAR(50) NOT NULL,
    customer_full_name VARCHAR(200),
    customer_email VARCHAR(255),
    customer_phone VARCHAR(20),
    customer_dob DATE,
    customer_tax_id VARCHAR(256),
    policy_id VARCHAR(50),
    policy_type VARCHAR(100),
    policy_start_date DATE,
    policy_end_date DATE,
    policy_coverage_amount DECIMAL(18,2),
    order_id VARCHAR(50),
    order_date TIMESTAMP,
    order_status VARCHAR(50),
    order_total DECIMAL(18,2),
    invoice_id VARCHAR(50),
    invoice_date DATE,
    invoice_amount DECIMAL(18,2),
    invoice_payment_status VARCHAR(50),
    customer_profile_created_timestamp TIMESTAMP DEFAULT GETDATE(),
    customer_profile_updated_timestamp TIMESTAMP DEFAULT GETDATE(),

    PRIMARY KEY (customer_id, order_id)
)
DISTSTYLE KEY
DISTKEY (customer_id)
SORTKEY (customer_id, order_date);

Once the Redshift is onboarded with above steps, navigate to Project catalog from left navigation menu and choose Data sources. Run the Data Source to add the table to Project inventory assets.

‘customer_analytics_data’ should be Project Assets inventory.

Verify navigating to ‘Project catalog’ menu on the left and choose ‘Assets’.

Step 3: Generate classification recommendations

To automatically generate terms, select GENERATE TERMS in ‘GLOSSARY TERMS’ section of the asset.

AI recommendations for glossary terms automatically analyze asset metadata and context to determine the most relevant business glossary terms for each asset and its columns. Instead of relying on manual tagging or static rules, it reasons about the data and performs iterative searches across what already exists in the environment to identify the most relevant glossary term concepts.

After recommendations are generated, review the terms both at table and column level. Table level suggested terms can be viewed as shown in the following image:

Select the SCHEMA tab to review column level tags as shown in the following image:

Review and accept individually by selecting the AI icon shown in below image.

In this case, we select ACCEPT ALL and then select PUBLISH ASSET as shown below.

The tags are now added to the asset and columns without manual search and addition. Select PUBLISH ASSET.

The asset is now published to the catalog as shown in the following image in the upper left corner.

Step 4: Improve data discovery

Users can now experience enhanced search results and find assets in the catalog based on the associated terms.

Browse by TermsUsers can now explore the catalog and filter by terms as shown in left navigation “APPLY FILTER” section

Search and FilterUsers can also search assets by glossary terms as shown below:

Cleanup

To delete SageMaker domain, follow instructions at Delete domains
To delete your Redshift cluster follow steps at Shutting down and deleting a cluster

Conclusion

By standardizing terminology at publication, organizations can reduce metadata drift and improve discovery reliability. The feature integrates with existing workflows, requiring minimal process changes while helping deliver immediate catalog consistency improvements.

By tagging data at publication rather than correcting it later, data teams can spend less time fixing metadata and more time using it. For more information on SageMaker capabilities, see the Amazon SageMaker Catalog User Guide.

About the authors

Designing centralized and distributed network connectivity patterns for Amazon OpenSearch Serverless – Part 2

Ankush Goyal — Tue, 24 Mar 2026 21:42:53 +0000

This post is Part 2 of our two-part series on hybrid multi-account access patterns for Amazon OpenSearch Serverless. In Part 1, we explored a centralized architecture where a single account hosts multiple OpenSearch Serverless collections and a shared VPC endpoint. This approach works well when a single business unit or team manages collections on behalf of the organization.

However, many enterprises have multiple business units that need independent ownership of their OpenSearch Serverless infrastructure. When each business unit wants to manage their own collections, security policies, and VPC endpoints within their own AWS accounts, the centralized model from Part 1 no longer fits.

In this post, we address this multi-business unit scenario by introducing a pattern where the central networking account manages a custom private hosted zone (PHZ) with CNAME records pointing to each business unit’s VPC endpoint. This approach maintains centralized DNS management and connectivity while giving each business unit full autonomy over their collections and infrastructure.

The challenge with multiple business units

When multiple business units independently manage their own OpenSearch Serverless collections in separate AWS accounts, each account has its own VPC endpoint with its own private hosted zones. These private hosted zones only work within their respective VPCs, creating DNS fragmentation across the organization. Consumers in spoke accounts and on-premises environments can’t resolve collection endpoints in other accounts without additional DNS configuration.

Managing individual PHZ associations for each consumer VPC doesn’t scale, and asking each business unit to coordinate DNS with every consumer creates operational overhead. You need a network architecture that gives each team autonomy while keeping DNS management and connectivity centralized.

Solution overview

This architecture solves the problem by centralizing DNS management in the networking account while leaving collection and VPC endpoint ownership with each business unit. The networking account maintains a custom PHZ with CNAME records that map each collection endpoint to the regional DNS name of its corresponding VPC endpoint. This custom PHZ is associated with a Route 53 Profile and shared through AWS Resource Access Manager (AWS RAM) to spoke accounts. On-premises DNS resolution flows through the Route 53 Resolver inbound endpoint in the central networking VPC, which uses the same custom PHZ.

We cover two complementary patterns: Pattern 1 for on-premises access to collections across multiple business unit accounts, and Pattern 2 for spoke account access to those same collections. Both patterns rely on the centralized custom PHZ managed by your networking team.

Pattern 1: On-premises access to OpenSearch Serverless collections across multiple business unit accounts

With this pattern, your on-premises clients can privately access OpenSearch Serverless collections hosted across multiple business unit accounts, each with its own VPC endpoint. The following diagram illustrates this multi-business-unit architecture. It shows how on-premises DNS queries are resolved through the custom PHZ in the central networking account and routed to the correct business unit’s VPC endpoint through AWS PrivateLink.

(A) The Route 53 Profiles are created in the central networking account and shared through AWS Resource Access Manager (AWS RAM) with the central OpenSearch Serverless account.

(B) The central networking account has a custom PHZ with domain us-east-1.aoss.amazonaws.com and associated with central networking account VPC and with Route 53 Profiles.

(C) This PHZ contains CNAME records pointing to each business unit’s VPC endpoints.

(D) Each business unit account has Private DNS enabled on its VPC endpoint.

(E) This automatically creates the following PHZs during VPC endpoint creation and associates them with the business unit’s VPC, so DNS resolution works locally within that VPC without depending on the Route 53 Profiles.

us-east-1.aoss.amazonaws.com
us-east-1.opensearch.amazonaws.com
us-east-1.aoss-fips.amazonaws.com
privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

DNS resolution flow

Your on-premises client initiates a request to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com.
The on-premises DNS resolver has a conditional forwarder for us-east-1.aoss.amazonaws.com and forwards the query over AWS Direct Connect or AWS Site-to-Site VPN to the Route 53 Resolver inbound endpoint IPs in the central networking VPC.
The inbound Resolver endpoint passes the query to the Route 53 VPC Resolver in the central networking VPC.
The VPC Resolver finds the custom PHZ (bu-1-collection-id-1.us-east-1.aoss.amazonaws.com) associated with the central networking VPC. The CNAME record for bu-1-collection-id-1.us-east-1.aoss.amazonaws.com resolves to the regional DNS name of BU1’s VPC endpoint (for example, vpce-1234567890abcdefghi.a2oselk.vpce-svc-0c3ebf9a1a3ad247b.us-east-1.vpce.amazonaws.com).
The VPC Resolver then resolves the VPC endpoint regional DNS name to its elastic network interfaces (ENIs) private IP addresses.
The traffic reaches BU1’s VPC endpoint elastic network interfaces (ENIs) through private network connectivity because the on-premises client connects over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.

Data flow

Your on-premises client sends an HTTPS request to the resolved IP address with the TLS Server Name Indication (SNI) header set to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com, over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.
Traffic reaches the VPC endpoint ENIs in BU1’s OpenSearch Serverless VPC.
The VPC endpoint forwards the request to the OpenSearch Serverless service, which inspects the hostname and routes to BU1 Collection 1.

To access a collection in BU2, your client follows the same flow using bu-2-collection-id-1.us-east-1.aoss.amazonaws.com. The custom PHZ contains a separate CNAME record pointing to BU2’s VPC endpoint, and the OpenSearch Serverless service routes to the correct collection based on the hostname.

Pattern 2: Spoke account access to OpenSearch Serverless collections across multiple business unit accounts

While Pattern 1 addresses on-premises access, you might also need to provide access from compute resources and distributed applications in spoke accounts to OpenSearch Serverless collections across multiple business unit accounts. With this pattern, compute resources in spoke account VPCs can privately access OpenSearch Serverless collections across multiple business unit accounts through the centralized private hosted zone (PHZ) in the networking account.

In the following example, we use an Amazon Elastic Compute Cloud (Amazon EC2) instance as a compute resource to illustrate the pattern. However, the same approach applies to any compute resource within the spoke VPC. The following diagram illustrates this multi-business-unit, multi-spoke architecture, showing how spoke VPCs resolve DNS through the shared Route 53 Profile and custom PHZ, then route traffic to the correct business unit’s OpenSearch Serverless collections through AWS PrivateLink.

(A) The Route 53 Profiles, created in the central networking account, are shared through AWS RAM with all spoke accounts and with the central OpenSearch Serverless account.

(B) The central networking account has a custom PHZ with domain us-east-1.aoss.amazonaws.com and associated with central networking account VPC and with Route 53 Profiles.

(C) This PHZ contains CNAME records pointing to each business unit’s VPC endpoints.

(D) Each business unit account has Private DNS enabled on its VPC endpoint.

us-east-1.aoss.amazonaws.com
us-east-1.opensearch.amazonaws.com
us-east-1.aoss-fips.amazonaws.com
privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

DNS resolution flow

An Amazon EC2 instance in BU1 Spoke VPC 1 initiates a request to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com and sends a DNS query to the Route 53 VPC Resolver.
The VPC Resolver finds the Route 53 Profiles associated with the spoke VPC.
The Profiles reference the custom PHZ (us-east-1.aoss.amazonaws.com) managed in the central networking account. The CNAME record for bu-1-collection-id-1.us-east-1.aoss.amazonaws.com resolves to the regional DNS name of BU1’s VPC endpoint.
The VPC Resolver then resolves the VPC endpoint regional DNS name to its elastic network interfaces (ENIs) private IP addresses.
Traffic reaches the VPC endpoint ENIs through private network connectivity because the spoke VPC connects to BU1’s VPC through AWS Transit Gateway or AWS Cloud WAN.

Data flow

Your Amazon EC2 instance sends an HTTPS request to the resolved IP address with the TLS SNI header set to bu-1-collection-id-1.us-east-1.aoss.amazonaws.com, routed through AWS Transit Gateway or AWS Cloud WAN to BU1’s OpenSearch Serverless VPC.
The request arrives at the VPC endpoint ENIs in BU1’s OpenSearch Serverless VPC.
The VPC endpoint forwards the request to the OpenSearch Serverless service, which inspects the hostname and routes to BU1 Collection 1.

To access a collection in BU2, the same flow applies using bu-2-collection-id-1.us-east-1.aoss.amazonaws.com. The custom PHZ resolves to BU2’s VPC endpoint, and routes traffic through the transit gateway to BU2’s VPC. The same applies to resources in other spoke accounts with the Route 53 Profiles associated.

Custom PHZ record structure

The custom PHZ in the central networking account uses the domain us-east-1.aoss.amazonaws.com and contains CNAME records that map each collection endpoint to the regional DNS name of its corresponding VPC endpoint. Note that collections within the same business unit account share the same VPC endpoint, so their CNAME records point to the same regional DNS name. Collections in different business unit accounts point to different VPC endpoints.

Custom PHZ management

Unlike Part 1, where the auto-created PHZs from the VPC endpoint handle DNS resolution, this pattern requires your networking team to manually maintain the custom PHZ. When a business unit adds a new collection, the networking team must add a corresponding CNAME record to the custom PHZ.

Cost considerations

The architecture patterns described in this post use several AWS services that can contribute to your overall costs, including Amazon Route 53 (hosted zones, DNS queries, and Resolver endpoints), and Route 53 Profiles. We recommend reviewing the official AWS pricing pages for the most current rates:

For a full cost estimate tailored to your workload, use the AWS Pricing Calculator.

Conclusion

In this post, we showed how you can give on-premises clients and spoke account resources private access to OpenSearch Serverless collections distributed across multiple business unit accounts. By centralizing DNS management through a custom PHZ in the networking account and sharing it through Route 53 Profiles, you avoid coordinating PHZ associations across accounts while giving each business unit full ownership of their collections and VPC endpoints.

Combined with Part 1, you now have two architectural approaches for hybrid multi-account access to OpenSearch Serverless: a centralized model where one account owns all collections and a shared VPC endpoint, and a distributed model where multiple business units each manage their own collections and VPC endpoints. Choose the centralized model when a single team manages collections on behalf of the organization. Choose the distributed model when business units need independent ownership of their OpenSearch Serverless infrastructure.

For additional details, refer to the Amazon OpenSearch Serverless VPC endpoint documentation and Route 53 Profiles documentation.

About the authors

Designing centralized and distributed network connectivity patterns for Amazon OpenSearch Serverless – Part 1

Ankush Goyal — Tue, 24 Mar 2026 21:42:17 +0000

Amazon OpenSearch Serverless is a fully managed, serverless option for Amazon OpenSearch Service that removes the operational complexity of provisioning, configuring, and tuning OpenSearch clusters. When you run OpenSearch Serverless collections in a central account and need secure, private access from both on-premises environments and multiple AWS accounts, network architecture becomes critical. In this post, we explore two patterns to help you achieve this connectivity securely.

Solution overview

Working with customers implementing OpenSearch Serverless, we published blog posts addressing various network connectivity patterns to meet their evolving requirements:

In Network connectivity patterns for Amazon OpenSearch Serverless, we covered foundational patterns such as public access via internet gateway, private access through interface VPC endpoints, multi-VPC connectivity, and on-premises access using Route 53 VPC Resolver inbound endpoints.
In Designing centralized and distributed network connectivity patterns for Amazon OpenSearch Serverless, we introduced two advanced multi-account patterns: a centralized interface VPC endpoint model using Amazon Route 53 Profiles and a distributed endpoint model with centralized DNS management through a self-managed private hosted zone (PHZ).

In this post, we build on those patterns to address an additional enterprise requirement. When you manage many OpenSearch Serverless collections centrally but need access from multiple accounts and on-premises, you face several key challenges:

Coordinating VPC endpoints across accounts: managing endpoint provisioning and lifecycle across many consumer accounts adds operational overhead
Managing DNS configurations for each consumer: each new account or on-premises environment requires its own DNS setup, increasing complexity
Separating networking responsibilities from application ownership: without clear boundaries, networking and application teams become tightly coupled, slowing down both

This architecture solves these challenges with a clear separation of responsibilities. The central networking account shares Route 53 Profiles to manage DNS propagation across spoke accounts. The OpenSearch Serverless account owner maintains full control over their VPC endpoint and the associated private hosted zones (PHZs). Application owners retain autonomy over DNS configuration and collection management.

A single VPC endpoint handles multiple OpenSearch Serverless collections in an AWS Region, which reduces complexity and cost. Your networking team manages connectivity infrastructure while your application teams independently manage their OpenSearch Serverless collections, data access policies, and collection-specific DNS configurations. This gives you connectivity from on-premises networks (through AWS Direct Connect or AWS Site-to-Site VPN) and from compute resources across multiple AWS accounts through a unified network path.

This separation means that your network administrators and application teams can work independently. The result is a governance model that scales with your organization. We cover two complementary patterns that together give you complete hybrid access coverage, Pattern 1 for on-premises access and Pattern 2 for multi-account access, both using centralized interface VPC endpoints and Route 53 Profiles.

Before proceeding, you should be familiar with OpenSearch Serverless interface VPC endpoint DNS resolution. When creating an OpenSearch Serverless interface VPC endpoint, AWS automatically provisions four private hosted zones. The zones are three visible private hosted zones (for collections, dashboards, and FIPS endpoints) and one hidden internal private hosted zone that work together to resolve collection endpoints to private IP addresses. For more details on this DNS resolution mechanism, customers can review our previous blog post.

Pattern 1: Accessing multiple OpenSearch Serverless collections from on-premises through a centralized VPC endpoint and Route 53 Profiles in a multi-account architecture

The architecture spans three components:

A central OpenSearch Serverless account that hosts the collections and interface VPC endpoint.
A central networking account that owns the Route 53 Profiles and Inbound Resolver.
An on-premises environment connected using AWS Direct Connect or AWS Site-to-Site VPN.

The following diagram illustrates the architecture across these three components. It shows how DNS queries from on-premises clients are resolved through the Route 53 Profile and Inbound Resolver, and how data traffic reaches the OpenSearch Serverless collections using AWS PrivateLink.

(A) The Route 53 Profiles are created in the central networking account and shared through AWS Resource Access Manager (AWS RAM) with the central OpenSearch Serverless account. The central OpenSearch Serverless account associates the PHZs and interface VPC endpoint association with the shared Route 53 Profiles because that’s needed for end-to-end private DNS resolution.

(B) The central AOSS VPC contains an interface VPC endpoint for OpenSearch Serverless with Private DNS enabled. There are four Private Hosted Zones (PHZs):

us-east-1.aoss.amazonaws.com
us-east-1.opensearch.amazonaws.com
us-east-1.aoss-fips.amazonaws.com
privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

(C) The first three PHZs must be manually associated with the Route 53 Profile.

(D) The fourth is automatically associated when the VPC endpoint is associated with the profile.

(E) On the on-premises side, the DNS resolver is configured with conditional forwarding for us-east-1.aoss.amazonaws.com, directing queries to the Route 53 Resolver Inbound Endpoint in the central networking account.

DNS resolution flow

Your on-premises client initiates a request to collection-id-1.us-east-1.aoss.amazonaws.com.
The on-premises DNS resolver has a conditional forwarder for us-east-1.aoss.amazonaws.com and forwards the query over AWS Direct Connect or AWS Site-to-Site VPN to the Route 53 Resolver inbound endpoint IPs in the central networking VPC.
The inbound resolver receives the query and passes it to the Route 53 VPC Resolver.
The VPC Resolver checks the Route 53 Profiles associated with the central networking VPC. The Profiles provide access to the visible and hidden PHZs from the central OpenSearch Serverless VPC.
The VPC Resolver uses the visible PHZ to match the wildcard CNAME *.us-east-1.aoss.amazonaws.com to the interface VPC endpoint (VPCE) DNS name. It then uses the hidden PHZ to resolve the VPCE DNS name to the private elastic network interface (ENI) IP addresses of the interface VPC endpoint in the central OpenSearch Serverless VPC.
The private ENI IP addresses are returned through the inbound resolver endpoint to the on-premises DNS resolver and back to the on-premises client.

Data flow

The on-premises client sends an HTTPS request to the resolved private ENI IP address with the TLS Server Name Indication (SNI) header set to collection-id-1.us-east-1.aoss.amazonaws.com, over AWS Direct Connect or AWS Site-to-Site VPN through AWS Transit Gateway or AWS Cloud WAN.
Traffic reaches the interface VPC endpoint ENIs in the central OpenSearch Serverless VPC.
The interface VPC endpoint forwards the request to the OpenSearch Serverless service, which routes to Collection 1.

To access Collection 2, the client follows the same flow using collection-id-2.us-east-1.aoss.amazonaws.com. The wildcard DNS resolves to the same interface VPC endpoint, and the OpenSearch Serverless service routes to the correct collection based on the hostname.

Pattern 2: Accessing multiple OpenSearch Serverless collections from spoke accounts using a centralized VPC endpoint and Route 53 Profiles

While Pattern 1 addresses on-premises access, you might also need to provide access from compute resources and distributed applications across multiple AWS accounts. With this pattern, any compute resource running within spoke account VPCs can privately access multiple OpenSearch Serverless collections hosted in a central OpenSearch Serverless VPC through a single shared interface VPC endpoint.

The following diagram illustrates this multi-account architecture, showing how spoke account VPCs resolve DNS and route data traffic to the central OpenSearch Serverless collections through the shared Route 53 Profile and AWS PrivateLink, alongside the on-premises access path from Pattern 1.

(A) The Route 53 Profiles, created in the central networking account, are shared via AWS RAM with both the OpenSearch Serverless account and the spoke accounts. The OpenSearch Serverless account associates its interface VPC endpoint and private hosted zones (PHZs) with the Profiles, while each spoke account associates the Profiles with its VPC. Spoke VPCs get full DNS resolution for OpenSearch Serverless collection endpoints without requiring their own interface VPC endpoints, PHZs, or manual DNS configuration.

(B) The central AOSS VPC contains an interface VPC endpoint for OpenSearch Serverless with Private DNS enabled. There are four Private Hosted Zones (PHZs):

us-east-1.aoss.amazonaws.com
us-east-1.opensearch.amazonaws.com
us-east-1.aoss-fips.amazonaws.com
privatelink.c0X.sgw.iad.prod.aoss.searchservices.aws.dev

(C) The first three PHZs must be manually associated with the Route 53 Profile.

(D) The fourth is automatically associated when the VPC endpoint is associated with the profile.

DNS resolution flow

An Amazon EC2 instance in Spoke VPC 1 initiates a request to collection-id-1.us-east-1.aoss.amazonaws.com and sends a DNS query to the Route 53 VPC Resolver.
The VPC Resolver finds the Route 53 Profiles associated with the spoke VPC, which carries the PHZs from the central OpenSearch Serverless account.
The visible PHZ matches the wildcard CNAME *.us-east-1.aoss.amazonaws.com to the VPCE DNS name. The hidden PHZ resolves the VPCE DNS name to the private ENI IP addresses of the interface VPC endpoint in the central OpenSearch Serverless VPC.
Route 53 returns the private ENI IP addresses to the Amazon EC2 instance.

Data flow

Your Amazon EC2 instance sends an HTTPS request to the resolved private ENI IP address with the TLS SNI header set to collection-id-1.us-east-1.aoss.amazonaws.com. This is routed through AWS Transit Gateway or AWS Cloud WAN to the central OpenSearch Serverless VPC.
The request arrives at the interface VPC endpoint ENIs in the central OpenSearch Serverless VPC.
The interface VPC endpoint forwards the request to the OpenSearch Serverless service, which routes to Collection 1.

To access Collection 2, the same flow applies using collection-id-2.us-east-1.aoss.amazonaws.com. The same applies to resources in Spoke Account 2 or other spoke accounts with the Route 53 Profiles associated.

AWS RAM Permission Configuration for Resource Association

When the central networking account shares the Route 53 Profiles through AWS RAM, the default AWS managed permissions policy (AWSRAMPermissionRoute53ProfileAllowAssociationActions) only grants actions for associating and disassociating the Profile with VPCs, viewing Profiles details, and listing associations. It does not include the route53profiles:AssociateResourceToProfile or route53profiles:DisassociateResourceFromProfile actions required for the OpenSearch Serverless account to associate its interface VPC endpoint and PHZs with the shared Profiles.

To enable this, the central networking account must create a custom managed permission in AWS RAM with the following actions:

route53profiles:AssociateProfile
route53profiles:AssociateResourceToProfile
route53profiles:DisassociateProfile
route53profiles:DisassociateResourceFromProfile
route53profiles:GetProfile
route53profiles:GetProfileResourceAssociation
route53profiles:ListProfileAssociations
route53profiles:ListProfileResourceAssociations
route53profiles:ListProfiles

This custom permission must be attached to the RAM resource share before the central OpenSearch Serverless account can associate its PHZs and interface VPC endpoint with the Profiles.

Cost considerations

The architecture patterns described in this post use several AWS services that may contribute to your overall costs, including Amazon Route 53 (hosted zones, DNS queries, and Resolver endpoints), and Route 53 Profiles. We recommend reviewing the official AWS pricing pages for the most current rates:

For a full cost estimate tailored to your workload, use the AWS Pricing Calculator.

Conclusion

In this post, we showed how organizations can provide secure, private access to multiple OpenSearch Serverless collections from both on-premises environments and distributed AWS accounts using a single centralized interface VPC endpoint and Route 53 Profiles. This architecture centralizes OpenSearch Serverless collections and network infrastructure in a dedicated account, using Route 53 Profiles to propagate DNS across accounts. This eliminates per-account VPC endpoints, manual PHZ associations, and custom DNS configuration in spoke accounts.

This pattern is a good fit when a single team or business unit manages OpenSearch Serverless collections centrally. However, many enterprises have business units that need to independently manage their own OpenSearch Serverless collections in separate AWS accounts, each with their own interface VPC endpoints, security policies, and collection lifecycle. In Part 2, we explore how the architecture changes to support this distributed ownership model, where each business unit runs OpenSearch Serverless in their own account while still relying on centralized network connectivity and DNS management through Route 53 Profiles. We will cover how DNS resolution and the Route 53 Profiles configuration adapt when interface VPC endpoints and collections are spread across multiple accounts.

For additional details, refer to the OpenSearch Serverless VPC endpoint documentation and Route 53 Profiles documentation.

About the authors

Extract data from Amazon Aurora MySQL to Amazon S3 Tables in Apache Iceberg format

Kunal Ghosh — Mon, 23 Mar 2026 18:38:10 +0000

If you manage data in Amazon Aurora MySQL-Compatible Edition and want to make it available for analytics, machine learning (ML), or cross-service querying in a modern lakehouse format, you’re not alone.

Organizations often need to run analytics, build ML models, or join data across multiple sources. These are examples of workloads that can be resource-intensive and impractical to run directly against a transactional database. By extracting your Aurora MySQL data into Amazon S3 Tables in Apache Iceberg format, you can offload analytical queries from your production database without impacting its performance, while storing data in a fully managed Iceberg table store optimized for analytics. Built on the open Apache Iceberg standard, Amazon Simple Storage Service (Amazon S3) Table data is queryable from engines like Amazon Athena, Amazon Redshift Spectrum, and Apache Spark without additional data copies. You can also combine relational data with other datasets already in your data lake, enabling richer cross-domain insights.

Apache Iceberg and Amazon S3 Tables

Apache Iceberg is a widely adopted open table format that offers Atomicity, Consistency, Isolation, Durability (ACID) transactions, schema evolution, and time travel capabilities. It enables multiple engines to work concurrently on the same dataset, making it a popular choice for building open lakehouse architectures.

Amazon S3 Tables is a purpose-built, fully managed Apache Iceberg table store designed for analytics workloads. It delivers up to 3x faster query performance and up to 10x more transactions per second compared to self-managed Iceberg tables. It also automatically compacts data and removes unreferenced files to optimize storage and performance.

In this post, you learn how to set up an automated, end-to-end solution that extracts tables from Amazon Aurora MySQL Serverless v2 and writes them to Amazon S3 Tables in Apache Iceberg format using AWS Glue. The entire infrastructure is deployed using a single AWS CloudFormation stack.

Requirements

AWS offers zero-ETL integrations from Amazon Aurora to Amazon Redshift and Amazon SageMaker AI, enabling seamless data flow for analytics and machine learning workloads.

However, there isn’t yet a native zero-ETL integration between Amazon Aurora and Amazon S3 Tables. This means that organizations looking to use Amazon S3 Tables for their Lakehouse architecture currently face several requirements:

Setting up ETL pipelines to extract data from Amazon Aurora and transform it into Apache Iceberg format
Configuring networking and security for AWS Glue jobs to access Amazon Aurora databases in private subnets
Coordinating the provisioning of source databases, ETL pipelines, and target table stores
Managing the end-to-end workflow without native automation

Solution overview

In this solution, you automate the extraction of relational database tables from Amazon Aurora MySQL Serverless v2 to Amazon S3 Tables in Apache Iceberg format using AWS Glue 5.0. To help you get started and test this solution, a CloudFormation template is provided. This template provisions the required infrastructure, loads sample data, and configures the Extract, Transform, Load (ETL) pipeline. You can adapt this template for your own scenario.

Sample data

This solution uses the TICKIT sample database, a well-known dataset used in Amazon Redshift documentation. The TICKIT data models a fictional ticket sales system with seven interrelated tables: users, venue, category, date, event, listing, and sales. The dataset is publicly available as mentioned in the Amazon Redshift Getting Started Guide.

Solution flow

The solution flow as shown in the previous architecture diagram:

An AWS Lambda function downloads the TICKIT sample dataset (a fictional ticket sales system used in Amazon Redshift documentation) from a public Amazon S3 bucket to a staging S3 bucket.
A second Lambda function, using PyMySQL (a Python MySQL client library), loads the staged data files into the Aurora MySQL Serverless v2 database using LOAD DATA LOCAL INFILE.
The AWS Glue job reads seven TICKIT tables from Aurora MySQL through a native MySQL connection and writes them to Amazon S3 Tables in Apache Iceberg format using the S3 Tables REST catalog endpoint with SigV4 authentication.
You can query the migrated data in S3 Tables using Amazon Athena.

The solution consists of the following key components:

Amazon Aurora MySQL Serverless v2 as the source relational database containing the TICKIT sample dataset (users, venue, category, date, event, listing, and sales tables)
AWS Secrets Manager to store the Aurora MySQL database credentials securely
Amazon S3 staging bucket for the TICKIT sample data files downloaded from the public redshift-downloads S3 bucket
AWS Lambda functions using PyMySQL to load data into Aurora MySQL
AWS Glue 5.0 job (PySpark) to read tables from Aurora MySQL and write them to S3 Tables in Apache Iceberg format
Amazon S3 Tables as the target storage for the migrated Iceberg tables
Amazon VPC with private subnets and VPC endpoints for Amazon S3, S3 Tables, AWS Glue, Secrets Manager, AWS Security Token Service (AWS STS), CloudWatch Logs, and CloudFormation

Here are some advantages of this architecture:

Fully automated setup: A single CloudFormation stack provisions the required infrastructure, loads sample data, and configures the ETL pipeline.
Serverless and cost-efficient: Aurora MySQL Serverless v2 and AWS Glue both scale based on demand, minimizing idle costs.
Apache Iceberg table format: Data is stored in Apache Iceberg format, enabling ACID transactions, schema evolution, and time travel queries.
Network isolation and credential management: The resources run within private subnets with Virtual Private Cloud (VPC) endpoints, and database credentials are managed through AWS Secrets Manager.
Extensible pattern: The same approach can be adapted for other relational databases (PostgreSQL, SQL Server) and other target formats supported by AWS Glue.

Prerequisites

To follow along, you need an AWS account. If you don’t yet have an AWS account, you must create one. The CloudFormation stack deployment takes approximately 30-45 minutes to complete and requires familiarity with Amazon S3 Tables, AWS CloudFormation, Apache Iceberg, AWS Glue, Amazon Aurora. This solution will incur AWS costs. The main cost drivers are AWS Glue ETL job runs (billed per DPU-hour, proportional to data volume) and Amazon S3 Tables storage and request charges. Remember to clean up resources when you are done to avoid unnecessary charges.

CloudFormation parameters

You can configure the following parameters before deploying the CloudFormation stack:

Parameter	Description	Default	Required
S3TableBucketName	Name of the S3 Tables bucket to create (or use existing)		Yes
DatabaseName	Name of the initial Aurora MySQL database	tickit	No
MasterUsername	Master username for Aurora MySQL	admin	No
VpcCidr	CIDR block for the VPC	10.1.0.0/16	No
S3TableNamespace	Namespace for S3 Tables	tickit	No

Implementation walkthrough

The following steps walk you through the implementation. These steps are to deploy and test an end-to-end solution from scratch. If you are already running some of these components, you may skip to the relevant step. You can also refer to the aws-samples repository, sample-to-write-aurora-mysql-to-s3tables-using-glue for the entire solution.

Step 1: Deploy the CloudFormation stack

Deploy the CloudFormation template scripts/aurora-mysql-to-s3tables-stack.yaml using the AWS Console or the AWS Command Line Interface (AWS CLI). Provide a name for the S3 Tables bucket; the stack will create it automatically (or use an existing one if it already exists).

To deploy using the AWS Console (recommended), navigate to the AWS CloudFormation Console and use the CloudFormation template. Alternatively, to deploy using the AWS CLI first upload the template to an S3 bucket (the template exceeds the 51,200 byte limit for inline –template-body), then create the stack.

# Upload the template to S3
aws s3 cp scripts/aurora-mysql-to-s3tables-stack.yaml \
  s3://<your-s3-bucket>/aurora-mysql-to-s3tables-stack.yaml \
  --region <your-region>

# Create the stack using the S3 template URL
aws cloudformation create-stack \
  --stack-name aurora-mysql-tickit-stack \
  --template-url https://<your-s3-bucket>.s3.<your-region>.amazonaws.com/aurora-mysql-to-s3tables-stack.yaml \
  --parameters \
    ParameterKey=S3TableBucketName,ParameterValue=<your-s3-table-bucket-name> \
  --capabilities CAPABILITY_NAMED_IAM \
  --region <your-region>

The stack will automatically:

Create the S3 Tables bucket (or use existing if it already exists)
Create a VPC with private subnets and VPC endpoints
Provision an Aurora MySQL Serverless v2 cluster
Download TICKIT sample data from the public Amazon S3 bucket
Load the sample data into Aurora MySQL via a Lambda function using PyMySQL
Create a Glue job configured to migrate data to S3 Tables in Iceberg format

Note: The S3 Tables bucket is retained when the stack is deleted to preserve your data.

Step 2: Verify the Aurora MySQL data

Retrieve the AuroraClusterEndpoint, DatabaseName, and SecretArn values from the CloudFormation stack, make a note of the AuroraClusterEndpoint, DatabaseName, and SecretArn. You can navigate to the Amazon Aurora Console, choose the Query Editor, and enter the values from the CloudFormation stack to connect. You can also choose your preferred method of connecting to an Amazon Aurora DB cluster.

Use the AWS CLI to retrieve the stack outputs: –

aws cloudformation describe-stacks --stack-name aurora-mysql-tickit-stack --region <your-region> --query "Stacks[0].Outputs"

Then run the following SQL commands to verify the data load:

-- Verify if the tables are created
SELECT * FROM information_schema.tables WHERE table_schema = 'tickit';

-- Verify if the data is loaded
SELECT 'users' AS table_name, COUNT(*) AS record_count FROM tickit.users
UNION ALL SELECT 'venue', COUNT(*) FROM tickit.venue
UNION ALL SELECT 'category', COUNT(*) FROM tickit.category
UNION ALL SELECT 'date', COUNT(*) FROM tickit.date
UNION ALL SELECT 'event', COUNT(*) FROM tickit.event
UNION ALL SELECT 'listing', COUNT(*) FROM tickit.listing
UNION ALL SELECT 'sales', COUNT(*) FROM tickit.sales;

Step 3: Run the Glue job

Navigate to the AWS Glue Console, choose ETL jobs under Data Integration and ETL from the left panel. Select the AWS Glue job mysql-tickit-to-iceberg-job and choose Run job to start execution. You can also start the ETL job using the AWS CLI:

aws glue start-job-run --job-name mysql-tickit-to-iceberg-job --region <your-region>

The AWS Glue job performs the following operations for each of the seven TICKIT tables:

Reads the table from Aurora MySQL through the native MYSQL Glue connection
Converts the data to a Spark DataFrame
Creates the Iceberg table in the S3 Tables namespace using CREATE TABLE IF NOT EXISTS with the USING ICEBERG clause
Inserts the data using INSERT INTO (or INSERT OVERWRITE if the table already exists)
Verifies the record count and displays sample data

Step 4: Verify the results

After the AWS Glue job completes, verify that the tables have been created in your S3 Table bucket by navigating to the Amazon S3 Console. Choose Table buckets under Buckets and select your S3 Table bucket. You can also verify using the AWS CLI:

aws s3tables list-tables \
  --table-bucket-arn arn:aws:s3tables:<your-region>:<your-account-id>:bucket/<your-s3-table-bucket-name> \
  --namespace tickit \
  --region <your-region>

Select a table from the tickit namespace and choose Preview to inspect the data.

You can also query the migrated tables using Amazon Athena to validate the data.

Clean up resources

Remember to clean up resources when you no longer need them to avoid unnecessary charges.

Navigate to the CloudFormation console, search for your stack and choose Delete. Alternatively, use the AWS CLI:

aws cloudformation delete-stack --stack-name aurora-mysql-tickit-stack --region <your-region>

The S3 Tables bucket is retained by default. To delete it, use the Amazon S3 console or the AWS CLI to remove the table bucket separately. The staging S3 bucket will be automatically emptied and deleted as part of the stack deletion.

aws s3tables delete-table-bucket --table-bucket-arn arn:aws:s3tables:<your-region>:<your-account-id>:bucket/<your-s3-table-bucket-name> --region <your-region>

Summary

In this post, we showed you how to extract data from Amazon Aurora MySQL Serverless v2 and write it to Amazon S3 Tables in Apache Iceberg format using AWS Glue 5.0. By using the native Iceberg support of AWS Glue and the S3 Tables REST catalog endpoint, you can bridge the gap between relational databases and modern lakehouse storage formats. By automating the entire pipeline through CloudFormation, you can quickly set up and replicate this pattern across multiple environments.

As AWS Glue and Amazon S3 Tables continue to evolve, you can take advantage of future enhancements while maintaining this automated migration pattern.

If you have questions or suggestions, leave us a comment.

About the authors

Simplifying Kafka operations with Amazon MSK Express brokers

Mazrim Mehrtens — Mon, 23 Mar 2026 18:36:26 +0000

In this post, we show you how Amazon Managed Streaming for Apache Kafka (Amazon MSK) Express brokers brokers streamline the end-to-end activities for Kafka administration. Apache Kafka has become the de facto standard for real-time data streaming, powering mission-critical applications across industries worldwide. Its popularity stems from its ability to handle high-throughput, fault-tolerant data pipelines at scale. Given its central role in modern data architectures, managing Apache Kafka with high resilience and reliability is essential for business success.

To maintain this level of resilience, administrators need to handle several important operational tasks. Apache Kafka is a distributed stateful system, whose state management requires constant communication and data movement in dynamic cloud environments. Administrators need to carefully size clusters by calculating complex compute, storage, and network requirements. They must provision storage volumes upfront and monitor utilization constantly to avoid disruptions. When workloads grow, scaling the cluster requires hours or days of effort using multiple tools to provision capacity and rebalance load.

With these operational requirements in mind, many administrators ask: is there an easier way to manage Apache Kafka at scale while maintaining the high resilience their applications demand?

Amazon MSK Express addresses these challenges directly. In this post, we show you how MSK Express brokers streamline the end-to-end activities for Kafka administration, including:

Sizing Kafka clusters for optimal performance and cost
Scaling cluster storage up and down with workload changes
Scaling cluster compute in and out over time
Monitoring cluster health
Managing cluster security
Ensuring high availability with fast and automatic broker recovery

What are Amazon MSK Express brokers?

Amazon MSK Express brokers are a transformative breakthrough for customers needing high-throughput Kafka clusters that scale faster and cost less. Express brokers reimagine Kafka’s compute and storage, decoupling to unlock performance and elasticity benefits. Express brokers deliver performance improvements that directly impact your operations:

Up to 3x more throughput per broker, allowing you to handle more data with fewer resources and lower costs
Rebalance partitions across brokers 180x faster, reducing scaling from hours to minutes
Scale up to 20x faster, enabling you to respond to demand spikes without lengthy planning cycles
Recover 90% quicker compared to standard Apache Kafka brokers, minimizing workload disruption and maintaining business continuity

To learn more about the technical details, see Express brokers for Amazon MSK: Turbo-charged Kafka scaling with up to 20 times faster performance. For a comprehensive overview of Express broker capabilities, see the MSK Express brokers documentation.

Let’s explore how MSK Express brokers simplify Apache Kafka management.

Sizing an Express cluster

Sizing a traditional Apache Kafka cluster is complex. Working backwards from your ingress and egress load, you need to consider every dimension of your cluster compute, storage, and network limitations. Each node must be carefully sized to handle:

Ingress and egress traffic from your clients
Internal Kafka operations like replication and rebalancing (the process of redistributing partitions across brokers to maintain balance)
High availability with node and Availability Zone failures
Client operations like backfill procedures when reading historical data

These activities impact your cluster storage I/O limits, network ingress/egress limits, and CPU and memory constraints. Beyond this, you need to consider the number of partitions required and determine whether your cluster can scale to handle partition management for your use case.

MSK Express brokers simplify this calculus. Rather than considering these complex variables, you can focus on what matters:

Your ingress throughput
Your egress throughput
Your partition needs

MSK documents the Express broker throughput throttle and partition limits by broker size. MSK pre-calculates these to consider all cluster limits. They include multi-Availability Zone high availability to handle rare events like node failures or AZ impairment.

Notice we did not discuss storage in sizing an Express cluster. That is because storage in Express scales nearly infinitely. You pay for storage as you go rather than sizing storage up front.

Scaling Express cluster storage

With sizing simplified by focusing on throughput and partitions, storage management becomes the next operational consideration.

Normally, Apache Kafka clusters need storage volumes pre-provisioned to handle all retained data. You must allocate all storage up-front and pay for that storage no matter what your actual data retention is.

Example: If you store 7 days of data at 1 MB/sec ingress, that’s 600+ GB of storage. This does not include data replication across nodes and buffers for growth and workload variability. This workload requires over 3 TB of storage, allocated up-front, to handle replicas and storage buffers.

As your workload evolves, careful monitoring of storage utilization becomes essential. Adding storage capacity prevents workload disruptions. Often, you cannot reclaim this storage. Once you increase the volume size, you continue paying for additional storage even if your workload scales down and no longer requires additional capacity.

With Express brokers, there is no need for sizing and provisioning storage volumes. You pay for what you use with no provisioning: the data ingested to the cluster and data stored in the cluster per-GB-per-hour. All data stored in the cluster is replicated across 3 Availability Zones for high availability. This pay-as-you-go model eliminates wasted capacity costs and reduces your total infrastructure spend.

As workloads scale up, the cluster uses more storage with no changes needed from you
When workloads scale down, the cluster uses less storage, reducing storage charges automatically
Storage management for Apache Kafka becomes simpler with Express. You focus on ensuring that your per-topic retention is right-sized for each use case. That is the only consideration. Once you set up topic retention, MSK Express automatically manages and cost-optimizes storage on your behalf.

Storage management in MSK Express brokers is far simpler than in a traditional Apache Kafka cluster. So is scaling the compute capacity for an Express-based cluster.

Scaling Express cluster compute

Just as storage scales automatically with your workload, compute capacity can also adapt to changing demands.

As your workload grows and changes, you may find that you exceed your initial sizing estimates. For a traditional Apache Kafka cluster, scaling the cluster capacity is a significant event. Scaling takes effort to provision capacity and rebalance load, it requires using multiple tools to manage the scaling process (compute, storage, DNS, rebalancing, client configs, and more). The scaling process can take hours or days to complete, which can exacerbate application impact. This means you need to plan well ahead to ensure your Kafka cluster is prepared for any load changes.

With MSK Express clusters, this process becomes much simpler and requires little to no upfront planning. It has near zero disruption to your existing workload, allowing your team to focus on building features rather than managing infrastructure.

To scale up an MSK Express cluster, you simply add brokers to the cluster. Once new brokers come online, Express Intelligent Rebalancing automatically rebalances topic partitions to the new nodes. Thanks to the Express storage architecture, the new nodes automatically have almost all the data they need. There is no significant inter-broker communication for rebalancing. This causes no disruption to existing brokers.

The cluster then elects new broker leaders for each partition, enabling producers to direct traffic to the new nodes. The same applies to consumer groups.

Express broker DNS design keeps this in mind. Express broker connection strings abstract away from the nodes themselves. Clients connect to the active broker nodes with one connection string. No changes to DNS, load balancing, or client configurations are needed.

Deciding when to scale in an Express cluster is also simpler than in a traditional Apache Kafka cluster. The simplified Express architecture means less to monitor and manage for long-term cluster operations.

Monitoring Express clusters

With simplified scaling decisions comes simplified monitoring. Express brokers reduce the number of metrics you need to track for cluster health. The below image demonstrates a dashboard which highlights the key metrics for monitoring MSK Express broker health.

In a traditional Apache Kafka cluster, you need to consider dozens of metrics to understand overall cluster health. Express brokers simplify this operational process. They highlight ingress and egress throughput as two critical metrics for workload sizing and scaling. This streamlined monitoring approach reduces the expertise required to operate Kafka clusters and allows smaller teams to manage larger deployments effectively.

Other factors, like poorly designed clients, can incur additional overhead on a cluster. This can cause symptoms such as high CPU utilization without high ingress throughput. It is still important to monitor a variety of metrics with MSK Express brokers.

For Express brokers, the following table shows the critical metrics you must monitor and alert on for cluster health:

Metric Name	Description	Recommended Alarm
BytesInPerSec	Ingress throughput to the cluster	When > broker limit for > 5 minutes
BytesOutPerSec	Egress throughput to the cluster	When > broker limit for > 5 minutes
CpuUser + CpuSystem	CPU utilization percentage	When greater than 60% for 15 minutes
NetworkProcessorAvgIdlePercent	Network processor thread idle time	When less than 0.5 for > 5 minutes
RequestHandlerAvgIdlePercent	Request processor thread idle time	When less than 0.4 for > 15 minutes
FetchThrottleByteRate	Consumer fetch throttling rate	When < 0 for > 15 minutes
ProduceThrottleByteRate	Producer ingress throttling rate	When < 0 for > 15 minutes

For more information on monitoring Amazon MSK, see Monitoring Amazon MSK with Amazon CloudWatch.

Managing Express cluster access

Beyond monitoring, cluster management is another area where MSK Express brokers reduce operational complexity.

Express brokers simplify the internal management of Kafka clusters. In a traditional Kafka environment, you use schemes like SASL/SCRAM (username and password-based authentication) or mutual TLS (certificate-based authentication) for client authentication. Once authenticated, you configure complex Kafka ACLs (Access Control Lists—permissions that define who can access which topics) inside the Kafka cluster to authorize client access to topics and data.

These paradigms require you to manage all topics, authentication, and authorization inside Apache Kafka. This includes credential management, rotation, and other operational activities surrounding cluster access.

MSK simplifies this process by integrating with AWS Identity and Access Management (IAM) for access control. Clients can use IAM Roles that clearly specify cluster access boundaries. They also provide topic-level authorization to read and write data to a cluster with Kafka APIs.

Finally, clients can use MSK APIs to directly manage Kafka cluster configurations and Kafka topics, including creating new topics, updating topic configurations and partition counts, and deleting topics. Configurations and topics can be managed with the AWS Console, AWS CLI, and AWS SDK. For more information, refer to Amazon MSK simplifies Kafka topic management with new APIs and console integration.

You can focus only on your existing enterprise standards for IAM access controls, and your existing AWS CloudFormation and AWS CDK automation to manage your cluster with Infrastructure as Code (IaC). This integration reduces the operational overhead of cluster management and accelerates your time to production by leveraging existing security infrastructure.

MSK also supports using SASL/SCRAM and mutual TLS authentication modes alongside IAM access control. This gives you the flexibility to authorize applications outside of AWS. You can also provide access to legacy applications without the need for code changes.

For more information, see IAM access control for Amazon MSK and Security in Amazon MSK.

Building highly available Express brokers

With security simplified through IAM integration, high availability is the final piece of the operational puzzle.

Many of the same considerations we discussed in scaling Express cluster compute align with high availability considerations for MSK Express brokers.

Based on internal testing, MSK Express broker storage improvements enable faster recovery when broker nodes fail—90% faster than standard brokers. The new node can simply start up with almost no disruption to the rest of the cluster without needing to perform significant rebalancing. This contrasts with standard Kafka clusters, where the cluster needs to rebalance partitions to new nodes after recovery.

In addition to these improvements, MSK Express brokers are highly available by default. The service manages critical cluster and topic configurations for high availability and performance on your behalf. This eliminates the need for managing most cluster configurations.

Express fully manages configurations like min.insync.replicas, num.io.threads, and others described in Express brokers’ read-only configurations. This gives you a highly available and performant cluster out of the box.

You no longer need to worry about most cluster-level configurations of an Apache Kafka cluster. You can simply:

Start an MSK Express cluster
Configure topics and retention
Proceed without the fine tuning normally needed to ensure a highly available cluster

Conclusion

In this post, we showed how MSK Express brokers simplify cluster operations for Apache Kafka clusters. They lower the Total Cost of Ownership (TCO) of running an Apache Kafka cluster by simplifying sizing, storage management, compute management, high availability, and access control, while providing high performance, reliability, and cost-efficiency. These simplifications reduce the specialized expertise needed for cluster administration and accelerate your deployment timeline.

With this in mind, we recommend MSK Express brokers for almost all MSK workloads. If you are starting out with a new Kafka cluster or optimizing an existing one, MSK Express brokers provide a strong combination of simplicity, performance, and cost-efficiency.

Ready to simplify your Kafka operations? Get started using Amazon MSK to create your first Express cluster today. You can provision a fully managed, highly available Kafka cluster in minutes and start experiencing the operational benefits immediately. For pricing details, see Amazon MSK pricing.

For comprehensive information about Amazon MSK capabilities and features, visit the Amazon MSK product page and the Amazon MSK Developer Guide.

About the authors

Filter catalog assets using custom metadata search filters in Amazon SageMaker Unified Studio

Ramesh H Singh — Wed, 18 Mar 2026 23:18:11 +0000

Finding the right data assets in large enterprise catalogs can be challenging, especially when thousands of datasets are cataloged with organization-specific metadata. Amazon SageMaker Unified Studio now supports custom metadata search filters. You can filter catalog assets using your own metadata form fields like therapeutic area, data sensitivity, or geographic region rather than relying only on free-text search. Custom metadata forms are structured templates that define additional attributes that can be attached to catalog assets.

In this post, you learn how to create custom metadata forms, publish assets with metadata values, and use structured filters to discover those assets. We explore a healthcare and life sciences use case. A research organization catalogs metrics in Amazon SageMaker Catalog using custom metadata forms with fields such as Therapeutic Area and Sample Size. Researchers building Machine learning models can now search datasets based on custom filters across hundreds of cataloged assets to identify the best datasets to train their models.

Key capabilities

Custom metadata search filters in SageMaker Unified Studio offer the following key capabilities:

Custom metadata form filters – You can filter search results using any custom metadata form fields defined in their catalog. For example, a researcher can filter by Therapeutic Area = Oncology and Data Sensitivity = Confidential to locate specific datasets.
Name and description filters – You can add filters that target asset names or descriptions using a text search operator, enabling targeted discovery without scanning full search results.
Date range filters – You can filter assets by date using on, before, after, and between operators, making it straightforward to locate recently updated or historically relevant assets.
Combinable filters – You can combine multiple filters to construct precise queries. For example, filtering by AWS Region = US AND Classification = PII AND Updated after 2026-01-01 returns only assets matching all three criteria.
Persistent filter selections – You can filter configurations stored in your browser and are not shared across devices or other users. You can later return to the catalog and find your previously defined filters.

Solution overview

In the following sections, we demonstrate how to set up custom metadata forms, publish assets with metadata values, and use custom metadata search filters to discover those assets.We complete the following three steps for the demonstration.

Create a custom metadata form
Create and publish assets with metadata
Use custom metadata search filters

Prerequisites

To follow along with this post, you should have:

An Amazon SageMaker Unified Studio domain setup
An existing project

For instructions on setting up a domain and project, see the Getting started guide.

To create a custom metadata form

Complete the following steps to create a custom metadata form with filterable fields:

In SageMaker Unified Studio, choose Project overview from the navigation pane.
Under Project catalog, choose Metadata entities.
Choose Create metadata form.
To create a new metadata form ‘research_metadata’ use the following details, then choose Create metadata form.
Define the form fields. For this demo, we add the following fields:
Create first field Therapeutic Area (String) – Mark as Searchable

Create second field Subject Count (Integer) – Mark as Filterable by range
Mark the form as ‘Enabled’ so the form is visible and can be used.

Create and publish with metadata

In this section, you create a custom asset and attach the research_metadata form created in the previous step.

Under Project catalog in the navigation pane, choose Metadata entities. Choose the ‘ASSET TYPES’ tab and select “CREATE ASSET TYPE’.
Create a new asset type and attach the metadata form that we created in the previous step.

A new asset type ‘metric’ is created.
Next, we will create two metrics. Under Project catalog in the navigation pane, choose Assets. On the Asset page, choose CREATE, and then choose Create asset from the menu.
In this demo, you create two metrics.

For the first metric ‘drug_1_treatment’, provide the following asset name and description.

Add the following values for the metadata form.

Validate all fields and choose CREATE.

Publish the asset to the catalog.

Next, we will create the second metric ‘drug_1_treatment’. Repeat the steps from the previous procedure and enter the values shown.

Subject Count = 450
Therapeutic Area = Oncology

Use custom metadata search filters

After publishing assets with custom metadata, go to the Browse Assets page to use the filters.

To browse assets and view filters

In SageMaker Unified Studio, choose Discover from the navigation bar, then select Catalog, Browse Assets.
The search page displays with the filter sidebar on the left. You can see the existing system filters (Data type, Glossary terms, Asset type, Owning project, Source Region, Source account, Domain unit) along with the new Date range and Add Filter sections.

Add a custom filter

Choose + Add Filter at the bottom of the filter sidebar. For Filter type, select Metadata form. For Metadata form, select research_metadata and add a filter as shown in the following image. Choose Apply when you’re done.

The search results update to show only assets where ‘subject_count’ is greater than 50.

To combine multiple filters

Choose + Add Filter again. For Filter type, select Metadata form. For Metadata form, select research_metadata and add a filter as shown in the following image. Choose Apply when you’re done.

Manage custom filters

Filter configurations are stored in the user’s browser and are not shared across devices or users.

To customize search, you could:

Toggle filters – Use the checkboxes next to each custom filter to enable or disable them without deleting.
Edit or delete – Choose the kebab menu (⋮) next to any custom filter to edit its values or delete it.
Clear all – Choose CLEAR next to the Custom filters header to deselect all custom filters at once.
Persistence – Your custom filters persist across browser sessions. When you return to the Browse Assets page, your previously defined filters are still listed in the sidebar, ready to be activated.

Using the SearchListings API

To search catalog assets programmatically, you can use the SearchListings API in Amazon DataZone, which supports the same filtering capabilities as the SageMaker Unified Studio UI. The following example filters assets where a custom string field contains a specific value and a numeric field is within a range:

aws datazone search-listings \
    --domain-identifier "dzd_your_domain_id" \
    --filters '{ "and": [
        { "filter": { "attribute": "research_metadata.TherapeuticArea", "value": "Oncology", "operator": "TEXT_SEARCH" } },
        { "filter": { "attribute": "research_metadata.SubjectCount", "intValue": 100, "operator": "GT" } }
    ] }'

For more details, see the SearchListings API documentation in the Amazon DataZone API Reference.

Best practices

Consider the following best practices when using custom metadata search filters:

Define your metadata forms before publishing assets at scale. If you publish assets before the forms are finalized, you might need to re-tag existing assets, which is a time-consuming process in large catalogs.
Define metadata forms aligned with your organization’s discovery needs (therapeutic areas, data classifications, geographic regions) before publishing assets at scale.
Use specific, consistent values in metadata fields to get precise filter results. For example, use standardized values (for example, use “Oncology” consistently rather than “oncology” or “Onc”) across all assets.
Combine multiple filters to narrow results efficiently rather than scanning through broad result sets.
Use the date range filter alongside custom metadata filters to locate assets within specific time windows.

Clean up resources

For instructions on deleting the added assets, see Delete an Amazon SageMaker Unified Studio asset.
For instructions on deleting the metadata forms, see Delete a metadata form in Amazon SageMaker Unified Studio.

Conclusion

Custom metadata search filters in Amazon SageMaker Unified Studio give data consumers the ability to find exact assets using structured filters based on their organization’s own metadata fields. By combining multiple filters across custom metadata forms, asset names, descriptions, and date ranges, data consumers can construct precise queries that surface the right datasets without scanning through broad search results. Filter persistence across browser sessions further streamlines repeated discovery workflows.

Custom metadata search filters are now available in AWS Regions where Amazon SageMaker is supported.

To learn more about Amazon SageMaker, see the Amazon SageMaker documentation. To get started with this capability, refer to the Amazon SageMaker Unified Studio User Guide.

About the authors

Best practices for Amazon Redshift Lambda User-Defined Functions

Sergey Konoplev — Wed, 18 Mar 2026 18:09:14 +0000

While working with Lambda User-Defined Functions (UDFs) in Amazon Redshift, knowing best practices may help you streamline the respective feature development and reduce common performance bottlenecks and unnecessary costs.

You wonder what programming language could improve your UDF performance, how else can you use batch processing benefits, what concurrency management considerations might be applicable in your case? In this post, we answer these and other questions by providing a consolidated view of practices to improve your Lambda UDF efficiency. We explain how to choose a programming language, use existing libraries effectively, minimize payload sizes, manage return data, and batch processing. We discuss scalability and concurrency considerations at both the account and per-function levels. Finally, we examine the benefits and nuances of using external services with your Lambda UDFs.

Background

Amazon Redshift is a fast, petabyte-scale cloud data warehouse service that makes it simple and cost-effective to analyze data using standard SQL and existing business intelligence tools.

AWS Lambda is a compute service that lets you run code without provisioning or managing servers, supporting a wide variety of programming languages, automatically scaling your applications.

Amazon Redshift Lambda UDFs allows you to run Lambda functions directly from SQL, which unlock such capabilities like external API integration, unified code deployment, better compute scalability, cost separation.

Prerequisites

AWS account setup requirements
Basic Lambda function creation knowledge
Amazon Redshift cluster access and UDF permissions.

Performance optimization best practices

The following diagram contains necessary visual references from the best practices description.

Use efficient programming languages

You can choose from Lambda’s wide variety of runtime environments and programming languages. This choice affects both the performance and billing. More performant code may help reduce the cost of Lambda compute and improve SQL query speed. Faster SQL queries could also help reduce costs for Redshift Serverless and potentially improve throughput for Provisioned clusters depending on your specific workload and configuration.

When choosing a programming language for your Lambda UDFs, benchmarks may help predict performance and cost implications. The famous Debian’s Benchmarks Game Team provides publicly available insights for different languages in their micro-benchmark results. For example, their Python vs Golang comparison shows up to 2 orders of magnitude run time improvement and twice memory consumption reduction if you could use Golang instead of Python. That may positively reflect on both Lambda UDF performance and Lambda costs for the respective scenarios.

Use existing libraries efficiently

For every language provided by Lambda, you can explore the whole collection of libraries to help you implement tasks better from the speed and resource consumption point of view. When transitioning to Lambda UDFs, review this aspect carefully.

For instance, if your Python function manipulates datasets, it might be worth considering using the Pandas library.

Avoid unnecessary data in payloads

Lambda limits request and response payload size to 6 MB for synchronous invocations. Considering that, Redshift is doing best effort to batch the values so that the number of batches (and hence the Lambda calls) would be minimal which reduces the communication overhead. So, the unnecessary data, like one added for future use but not immediately actionable, may reduce efficiency of this effort.

Keep in mind returning data size

Because, from the point of view of Redshift, each Lambda function is a closed system, it is impossible to know what size the returned data can possibly be before executing the function. In this case, if the returned payload is higher than the Lambda payload limit, Redshift will have to retry with the outbound batch of a lower size. That will continue until a fit return payload will be achieved. While it is the best effort, the process might bring a notable overhead.

In order to avoid this overhead, you might use the knowledge of your Lambda code, to directly set the maximum batch size on the Redshift side using the MAX_BATCH_SIZE clause in your Lambda UDF definition.

Use benefits of processing values in batches

Batched calls provide new optimization opportunities to your UDFs. Having a batch of many values passed to the function at once, allows to use various optimization techniques.

For example, memoization (result caching), when your function can avoid running the same logic on the same values, hence reducing the total execution time. The standard Python library functools provides convenient caching and Least Recently Used (LRU) caching decorators implementing exactly that.

Scalability and concurrency management

Increase the account-level concurrency

Redshift uses advanced congestion control to provide the best performance in a highly competitive environment. Lambda provides a default concurrency limit of 1,000 concurrent execution per AWS Region for an account. However, if the latter is not enough, you can always request the account level quota increase for Lambda concurrency, which might be as high as tens of thousands.

Note that even with a restricted concurrency space, our Lambda UDF implementation will do the best effort to minimize the congestion and equalize the chances for function calls across Redshift clusters in your account.

Restrict function concurrency with reserved concurrency

If you want to isolate some of the Lambda functions in a restricted concurrency scope, for example you have a data science team experimenting with embedding generation using Lambda UDFs and you don’t want them to affect your account’s Lambda concurrency much, you might want to set a reserved concurrency for their specific functions to operate with.

Learn more about reserved concurrency in Lambda.

Integration and external services

Call existing external services for optimal execution

In some cases, it might be worth considering using existing external services or components of your application instead of re-implementing the same tasks yourself in the Lambda code. For example, you can use Open Policy Agent (OPA) for policy checking, a managed service Protegrity to protect your sensitive data, there are also a variety of services providing hardware acceleration for computationally heavy tasks.

Note that some services have their own batching control with a limited batch size. For that we implemented a per-function batch row count setting MAX_BATCH_ROWS as a clause in the Lambda UDF definition.

To learn more on the external service interaction using Lambda UDFs refer the following links:

Conclusion

Lambda UDFs provide a way to extend your data warehouse capabilities. By implementing the best practices from this post, you may help optimize your Lambda UDFs for performance and cost efficiency.The key takeaways from this post are:

performance optimization, showing how to choose efficient programming languages and tools, minimize payload sizes, and leverage batch processing to reduce execution time and costs
scalability management, showing how to configure appropriate concurrency settings at both account and function levels to handle varying workloads effectively
integration efficiency, explaining how to benefit from external services to avoid reinventing functionality while maintaining optimal performance.

For more information, visit the Redshift documentation and explore the integration examples referenced in this post.

About the author

How Vanguard transformed analytics with Amazon Redshift multi-warehouse architecture

Alex Rabinovich — Wed, 18 Mar 2026 18:07:54 +0000

This is a guest post by Alex Rabinovich, Anindya Dasgupta, and Vijesh Chandran from Vanguard, Financial Advisor Services division, in partnership with AWS.

Vanguard stands as one of the world’s leading investment companies, serving more than 50 million investors globally. The company offers an extensive selection of low-cost mutual funds and ETFs with over 450 funds/ETFs along with comprehensive investment advice and related financial services. With a workforce of approximately 20,000 crew members, Vanguard has built its reputation on providing low-cost, high-quality investment solutions that help investors achieve their long-term financial goals.

Within this massive organization, Vanguard’s Financial Advisor Services (FAS) division stands as one of the most prominent B2B operations in the financial services industry. Operating at an extraordinary scale, FAS oversees a broad range and diverse range of assets through the intermediary channel while supporting a vast network of advisory firms and financial advisors across the country. This division delivers a full suite of investment products, model portfolios, research capabilities, and technology-driven support services designed to help financial advisors serve their clients more effectively.

Business use cases and initial architecture

The scale and complexity of FAS operations generate enormous amounts of data that require sophisticated analytics capabilities to drive business insights, regulatory compliance, and operational efficiency. To address this, Vanguard launched the FAS 360 initiative. This initiative aims to empower Financial Advisor Services (FAS) with a centralized cloud data warehouse that integrates both internal and external data sources into a unified, intelligent system.

Key business use cases:

Business operations – Enables sales goal setting, tracking, and compensation management to drive operational excellence. It delivers insights on product usage patterns across financial advisor clients.
Data science – Powers customer segmentation models and call transcription analytics to drive strategic insights. It also supports marketing campaign preparation and customer insights for sales call preparation.
Exploratory analytics – Enables ad-hoc leadership questions, what-if scenario analysis, and sales trend analysis for channel managers competitor comparative analysis.

By consolidating these use cases into a centralized system, FAS 360 enables consistent reporting and data-driven decision-making across Vanguard’s Financial Advisor Services division.

Centralized data warehouse FAS 360:

Vanguard’s first wave of modernization established FAS 360 as a centralized enterprise data warehouse, migrating from a fragmented “data swamp” of Parquet files on Amazon Simple Storage Service (Amazon S3) to a structured, unified system.

The following architecture diagram leverages Amazon S3 for raw data storage with Amazon Redshift serving as the core processing engine, providing integrated access for BI tools, analyst exploration, and data science workloads.

Here are the key benefits achieved with this architecture:

Single source of truth – Consolidated fragmented data sources into a unified system, minimizing multiple versions of truth and establishing consistent reporting practices across the organization
10x faster query performance – Dramatically improved query response times compared to the previous solution, helping enhance analyst productivity and enabling more complex analytical workloads
Seamless data lake integration – Maintained connectivity with the broader data lake environment while providing structured warehouse capabilities
Enhanced business agility – Increased trust in metrics and unlocked new use cases that were previously untenable, directing the new migration efforts toward the FAS360 system

This centralized architecture successfully addressed the limitations of Vanguard’s previous approach, where data was scattered across individuals with limited governance, and established a foundation for their subsequent architectural evolution.

Significant growth and expanding use cases

Vanguard FAS experienced remarkable growth in their data analytics requirements over a two-year period, demonstrating the rapid evolution of modern data needs:

Initial State:

20 AWS Glue ETL jobs processing daily data loads
Approximately 100 tables in their data warehouse
20 Tableau dashboards serving business users
Around 60 analysts accessing the system

Two Years Later:

20 TB in data volume in Amazon Redshift and another 150 TB in S3 data lake
600+ AWS Glue ETL jobs (a 30x increase) handling complex data transformations
300+ tables (3x growth) storing diverse business data
250+ Amazon Redshift materialized views optimizing query performance
Over 500 Tableau dashboards (25x expansion) serving various business functions
500,000+ user queries/months

This exponential growth reflected FAS’s increasing reliance on data-driven decision making across the business functions, from risk management and compliance to client service optimization and operational efficiency improvements.

Resource contention and performance bottlenecks

As Vanguard FAS’s data environment expanded, their initial architecture, a single Amazon Redshift provisioned cluster with 2 nodes (ra3.4xlarge), began experiencing severe performance challenges that threatened business operations:

ETL performance issues:

Frequent ETL SLA failures disrupting critical business processes
Tableau extract failures resulting in stale dashboard data
Resource conflicts between data ingestion and transformation workloads

End-user experience degradation:

Poor query performance during peak usage periods
Table and object locking issues preventing concurrent access
Frustrated analysts unable to perform deep data exploration
Limited ability to run long-running analytical queries

Operational challenges:

Resource contention between ETL workloads and interactive analytics
Inability to scale compute resources independently for different workload types
Single point of failure affecting the data operations
Difficulty in workload prioritization and resource allocation

These challenges were fundamentally limiting FAS’s ability to leverage their data assets effectively, impacting everything from daily operational reporting to strategic business analysis.

Solution overview

To address these critical challenges, Vanguard FAS implemented following multi-warehouse architecture that leverages the advanced data sharing capabilities of Amazon Redshift for workload isolation and independent scaling.

Producer – Amazon Redshift Provisioned Cluster

The central hub consists of the original Amazon Redshift provisioned cluster with RA3 nodes, optimized for consistent, predictable workloads:

Dedicated ETL processing: Handles data ingestion, transformation, and loading operations
Write workload optimization: Manages data writes and updates without interference
Cost optimization: Utilizes reserved instances for predictable, steady-state workloads
Data governance: Serves as the single source of truth for the enterprise data

Consumer – Amazon Redshift Serverless Workgroups

Multiple Amazon Redshift Serverless instances serve as specialized consumer endpoints which auto-scales compute resources based on demand:

Analyst Exploration: Dedicated environment for analyst data discovery and experimentation
BI Tools: Instance optimized specifically for Tableau dashboard and visualization workloads
Data Science: For complex and long running machine learning workloads in completely isolated environment

The solution leverages the native data sharing capabilities of Amazon Redshift to enable secure connectivity between the producer and consumers instances. Consumer clusters can access live data from the producer without data movement, providing real-time access to the most current information available. This zero-copy sharing approach alleviates the need for data duplication or complex synchronization processes, helping reduce both storage costs and operational complexity.

Results

The implementation of the multi-warehouse architecture delivered significant improvements across the key performance indicators:

Predictable Performance

Nightly ETL cycles now consistently complete before the 9 AM SLA, eliminating the previous SLA failures that disrupted business operations and ensuring fresh data is available for morning business activities. Dashboards and reports now reflect the most current data available, providing teams with up-to-date insights for decision-making.

Improved Analyst Productivity and Experience

The new architecture removed the restrictive 10-minute query timeout that previously prevented deep ad hoc exploratory queries. Analysts can now run complex analytical workloads exceeding 30 minutes in a fully isolated environment without impacting other users or ETL processes. This change, combined with significantly faster query response times, has led to higher analyst satisfaction and productivity across the team.

New Analytical Capabilities

The architecture introduced a dedicated “Data Lab” environment where analysts have write access to experiment with data using CREATE TABLE AS SELECT (CTAS) commands. Each workload type can now scale independently based on demand, with different consumer clusters optimized for specific use cases, enabling more sophisticated analytical approaches.

Operational Excellence

The separation of workloads enabled efficient utilization of compute resources across different patterns, leading to better cost control through appropriate sizing, serverless pay-as-you-go pricing, and reserved instance usage. The cleaner separation of concerns between ETL and analytics workloads has simplified overall management of the data platform.

Ongoing modernization: Evolution toward data mesh architecture

As Vanguard’s data environment matured and their success with the multi-warehouse architecture enabled broader adoption across the organization, they recognized an opportunity to evolve their architecture to match their organizational growth. The expanding portfolio of data products and increasing number of teams leveraging the system created new opportunities for innovation.

As Vanguard’s data environment grew, three key challenges emerged:

Centralized ownership bottleneck – Single-team data ownership couldn’t scale with the growing number of data products
Write workload contention – Resource contention persisted for write operations on shared endpoints
Cross-domain dependencies – Data object interdependencies across business domains slowed data product development

Rationale for Data Mesh

Vanguard’s decision to adopt Data Mesh was driven by the need to:

Decentralize data ownership by establishing data domains with dedicated stewards
Remove write contention by isolating each domain’s data loads to separate endpoints
Enable autonomous development allowing stewards to own the complete data product lifecycle and governance
Leverage modern data lake capabilities using AWS Glue and Apache Iceberg format for data product curation

This evolution supports Vanguard’s ability to scale organizationally while building on the technical foundation and operational excellence achieved with their multi-warehouse architecture. Building on the success of their Amazon Redshift multi-warehouse implementation, Vanguard FAS is now exploring on the next phase of their data architecture evolution, implementing following data mesh approach.

This new data mesh architecture has several key components that work together to enable scalable, domain-oriented data management.

Domain-Oriented Data Ownership

Vanguard is establishing distinct data domains aligned with business functions and assigning dedicated data stewards to each domain for clear ownership and accountability. This strategy shifts from centralized data management to a decentralized model where data ownership and responsibility can be distributed across business domains, enabling teams closer to the data to make informed decisions about their domain-specific needs.

Distributed Data Architecture

The new architecture isolates domain-specific data loads to separate compute endpoints and creates independent data processing pipelines for each domain. This approach helps reduce cross-domain dependencies and conflicts that previously slowed development cycles, allowing teams to iterate and deploy changes without waiting for coordination across the entire organization.

Data Product Approach

Vanguard is curating data products on the data lake using Apache Iceberg format and leveraging AWS Glue for metrics computation and data lake integration. This approach treats data as products with defined SLAs and quality metrics, helping facilitate reliable, high-quality data delivery that downstream consumers can depend on with confidence.

Self-Service Analytics

The implementation enables domain teams to manage their complete data product lifecycle independently while maintaining enterprise governance standards. Vanguard provides comprehensive tools and systems for independent data management, allowing teams to innovate quickly without compromising data quality or security, ultimately accelerating time-to-insight across the organization.This evolution represents a natural progression from centralized data warehouse to multi-warehouse architecture, and finally to a fully distributed, domain-oriented data mesh that can scale with Vanguard’s continued growth.

Conclusion

Vanguard Financial Advisor Services’ journey demonstrates that scaling analytics is no longer about scaling a single warehouse bigger, but about architecting for workload isolation, independent scaling, and organizational growth.

By evolving from a single 2-node RA3 provisioned cluster to a multi-warehouse architecture using Amazon Redshift Serverless and Provisioned, Vanguard achieved measurable, production-grade outcomes:

500,000+ monthly queries supported without ETL or dashboard contention
100% ETL SLA adherence, with nightly pipelines completing before 9 AM
25x growth in BI consumption (20 → 500+ Tableau dashboards) without performance degradation
8x growth in analyst population (60 → 500+) enabled through workload isolation
30x increase in ETL pipelines (20 → 600+) without re-architecting ingestion logic
Zero-copy Amazon Redshift data sharing across producer and consumer warehouses, minimizing data duplication and synchronization costs
Removal of 10-minute query limits, unlocking advanced exploratory and long-running analytics

Critically, these gains were not achieved by over-provisioning compute, but by right-sizing and specializing compute per workload, reserving capacity where demand was predictable (ETL) and using Amazon Redshift Serverless auto-scaling where demand was bursty (BI and ad-hoc analysis).

As Vanguard now progresses toward a domain-oriented data mesh, their experience reinforces a key lesson: Multi-warehouse architecture is a foundational enabler for organizational scale, data product ownership, and autonomous analytics.For organizations experiencing exciting growth in their data analytics requirements, Vanguard’s approach showcases the tremendous possibilities that await. With the right architecture and the help of AWS services, organizations can transform their data infrastructure to achieve remarkable improvements in performance, significant cost reductions, and unlock powerful new analytical capabilities that accelerate business value creation.

AWS encourages you to connect with your AWS Account Team to engage an AWS analytics specialist who can provide expert architectural guidance and tailored recommendations to help you achieve your data transformation goals.

© 2026 The Vanguard Group, Inc. and Amazon Web Services, Inc. All rights reserved. This material is provided for informational purposes only and is not intended to be investment advice or a recommendation to take any particular investment action.